Crawly logo Crawly

Site Settings


Introduction

When you add a website to Crawly, we name it as 'Site'. You can have many settings about a site that you control on Crawly.

Authentication Settings

If you want to control your protected pages where login is needed. In this case, you can use Crawly to crawl those pages in better way.

In order to reach authentication settings of your site, go to 'Edit' on the left sidebar.

image

  1. Form URL is the url address for your login for refers to. For example, your login accept route is "https://example.com/login" in this case you need to type your route here.
  2. Input Name is first input of your login form, generally this is a username or email input of the form. Define your input name that you require from users to login.
  3. Input Value is a username/email address for Crawly to login to your site. You should create a account for Crawly to allow it.
  4. Input Name is second input of your login form, generally this is password. Define your input name of your password input.
  5. Input Value is password of the account that you created for Crawly.
  6. Token Input Name stands for Cross-Site Request Forgery (CSRF). is a type of malicious exploit of a website where unauthorized commands are transmitted from a user that the web application trusts. So, if your site has CSRF protection on login form, you need to define token's input name here. Generally, it is token or _token.