Search Results for:
*
How To Crawl A Staging Website
Find out how to crawl a staging or development website, considering robots.txt, authentication, and the SEO Spider configuration.
How To Use List Mode
Upload a list of URLs using 'list mode', and control what other elements are crawled, such as external links, or images for laser-focused crawling.
What Is Link Score?
Find out more about our Link Score algorithm which calculates the relative importance of URLs based on the internal linking of a site.
Resolving Google Analytics / Google Search Console Connection Issues
If you've experienced a 'Failed to Connect To Google Analytics / Search Console' security message, this guide will help you debug and resolve the issue.
How To Crawl Large Websites
Crawl large websites by switching to database storage mode, increasing memory and by configuring the crawl to extract the data you need.
Search function
The search box in the top right of the interface allows you to search all visible columns. It defaults to regular text search of the ‘Address’ column, but allows you to switch to regex, choose from a variety of predefined...
Robots.txt Testing In The SEO Spider
View URLs blocked by robots.txt, the disallow lines & use the custom robots.txt to check & validate a site's robots.txt thoroughly, and at scale.
Crawling Password Protected Websites
Crawl websites that require a login, using web forms authentication using our inbuilt Chrome browser.
Exclude
Configuration > Exclude The exclude configuration allows you to exclude URLs from a crawl by using partial regex matching. A URL that matches an exclude is not crawled at all (it’s not just ‘hidden’ in the interface). This will mean...
URL rewriting
Configuration > URL Rewriting The URL rewriting feature allows you to rewrite URLs on the fly. For the majority of cases, the ‘remove parameters’ and common options (under ‘options’) will suffice. However, we do also offer an advanced regex replace...
