Skip to main content

Add / Edit Crawl Site

View Add Crawl Site Page

This page is used to add/edit the details of the crawl site configuration.

Form Fields

  • Display Name: Enter a unique name to identify this crawl site.
  • Crawl Interval (Days): Our crawl will revisit the site every X days.
  • Max Pages: Our crawler supports between 1 and 1000 pages per crawl site.
  • Seed URLs (one per line): Enter a list of URLs, preferably the home page of each website to include in this crawl site.
  • Exclude Patterns (one per line): If you wish to exclude certain URLs from being called, you can enter individual urls here. You may also use "." for matching urls with a wildcard, such as /contact. will match and exclude all urls starting with /contact such as: /contact/form and /contact/submit
  • Crawl Delay (ms): Enter a value between 1000 and 60000 that the crawler must wait between requests. This ensures our crawler does not exceed the limits of the target server.
  • Use Jitter: Jitter randomizes the delay between requests slightly to look less like a robot.
  • Status: Active enabled the crawler to download and index the site | Inactive will stop the crawler from further indexing. | Error means the last crawl failed and may need further attention.
  • Only Crawl If Home Page Changed: If set to yes, the child links of the website will only be crawled if the home page has changed since the last crawl.