Screaming Frog SEO Spider Update – Version 1.20

We have just released a new version of the Screaming Frog SEO Spider with a couple of very exciting new features.

• Upload URLs –  We have added a ‘list mode’ which allows you to upload a list of URLs to the spider for greater control over your crawl. You can upload URLs as a .txt file, .csv or a unicode .csv file direct from Adwords Editor. The tool does not require you to format the file types (so you don’t have to amend your Adwords Editor downloads with ads etc in the slightest for example). Simply upload anything in the file types above and the Spider will do the rest and find all the URLs for you regardless of what you have contained within.

• Custom Source Code Search – The spider now has the ability to find anything you want in the source code of a website! We have added a custom query string search feature which will check the source code of every page you decide to crawl for what it is you wish to find. So, for example – You may wish to find all pages which have ‘Out of Stock’ on them. Or perhaps you wish to find any pages that do not contain your Google Analytics code. This is an extremely powerful feature that is only available in the licensed version of the tool!

You can read more about these features  in our configuration options FAQ. Alongside these two new features, we have also included the following (smaller) updates –

• Provided the option to independently crawl internal and external “nofollow” links.
• Like a search engine the tool now doesn’t crawl external links which have a meta “nofollow” on them as default. The tool always did this at link level, but now also the page level directive.
• Treats character case in the domain as lowercase (and the canonical). For example, www.Screamingfrog.co.uk (uppercase ‘S’) is treated the same as www.screamingfrog.co.uk rather than as two pages). This is only for the domain itself, URI (and pages) found with variations in capitialisation can still be seen as technically separate pages, hence the spider will continue to treat them this way. Thanks to Brian Chappell for spotting this.
• -1 Response code now shows “Invalid HTTP Response” for some website pages that just give back HTML, but no http header.
• Export and saving of files now automatically show directory level when choosing the save location without having to choose ‘all files’ to navigate.

If you haven’t already tried the Screaming Frog SEO Spider why not download it now? If you already have, simply ‘check for updates’ for the new version.