Screaming Frog SEO Spider Update – Version 1.70

I am excited to announce another update to the Screaming Frog SEO spider, to version 1.70.

So lets get straight to it, what new features are there in version 1.70 of the tool?

• Include (REGEX Crawling) – This new feature allows you to control URLs the spider crawls using REGEX. It narrows the default search by only crawling the URLs that match the REGEX which is particularly useful for larger sites, or sites with less intuitive URL structures.

• User Agent Switcher – The SEO spider now have a user-agent switcher with inbuilt preset user agents for Googlebot, Bingbot, Yahoo! Slurp, various browsers and more. We also have a custom user-agent feature which allows you to specify your own user agent. This feature is only available to licensed users and we ask everyone to use this responsibly.

• X-Robots-Tag & Canonical HTTP Headers – The SEO spider now offers support for both the X-robots-tag and canonical http headers.

• Re-Spider URLs – You can now re-spider urls with the right click function. This is particularly useful if you’ve had connection time outs, unexpected responses or you wish to update some URLs when editing the site itself. You can use the control and shift functions to highlight multiple URLs to re-spider. So for example, you could simply go to the ‘response codes’ tab, click the ‘No response’ filter, press ‘control a’ to highlight all the URLs, right click and re-spider to crawl them all again.

We also fixed a few bugs around around case sensitivity of content type, a meta refresh display bug, crawl control for some domain extensions (.com.au etc) and the trailing slash on the root page between host and file path.

We hope everyone enjoys the new features in the tool, as always please do just let us know if you have any feedback or encounter any bugs. To download the new version of the Screaming Frog SEO spider, simply navigate to ‘help’ and ‘check for updates’ in the interface or download it here.