SEO Spider

How To Use List Mode

Introduction To List Mode

There are two fundamental crawling modes in the SEO Spider, the default ‘Spider’ mode which allows you to input and crawl a website, and ‘List’, which enables you to upload a list of URLs and crawl them.

It sounds simple, and it is – but there are some advanced uses of list mode, that are a bit more complex and make it really powerful, which this guide covers below.

First of all, let’s go over the basics. To switch to list mode, click ‘Mode > List’ in the top-level navigation.

‘List’ mode isn’t really that different from regular ‘Spider’ mode, there are two key differences:

The interface changes to show an upload button instead of an address bar.
The ‘Limit Crawl Depth‘ configuration is automatically enabled and set to ‘0’. This means that only the URLs you upload in list mode will be crawled.

The second point is the important difference, as it means only the URLs uploaded will be crawled. It won’t crawl these URLs and their images, or their external links etc. It will only crawl the URLs uploaded, nothing else*.

*Unless you’re using JavaScript rendering, where the SEO Spider needs to crawl the page resources to accurately render the page in its headless Chrome browser.

Check out our video guide on list mode, or continue to read our guide below.

Uploading A List

When you’re in list mode (Mode > List), just click the ‘Upload’ button and choose to upload from a file, enter into a dialog box, paste a list of URLs or download an XML Sitemap.

It’s that simple. But there are a couple of initial things you should be aware of in list mode when uploading URLs.

Protocol Required

If you don’t include either HTTP or HTTPS (e.g, just www.screamingfrog.co.uk/), the URL won’t be read and uploaded.

You’ll see a very sad message saying ‘found 0 URLs’. So always include the URL with protocol, for example –

https://www.screamingfrog.co.uk/

Normalisation & De-Duplication

The SEO Spider normalises URLs on upload, and de-dupes during the crawl. Let’s say you have the following 4 URLs to upload for example –

https://www.screamingfrog.co.uk/
https://www.screamingfrog.co.uk/#fragment-this-wont-be-crawled
https://www.screamingfrog.co.uk/seo-spider/
https://www.screamingfrog.co.uk/seo-spider/

The SEO Spider will automatically determine how many are unique URLs to be crawled.

For a small list, it’s easy to see (for most SEOs) that these 4 URLs are actually only 2 unique URLs, but with larger lists, it can be less obvious.

The SEO Spider page is duplicated, while the fragment URL (with a ‘#’) is not seen as a separate unique URL, so it’s normalised upon upload.

If these URLs are uploaded into the SEO Spider, it will report that it’s found the 4 URLs – and normalise them in the window dialog –

However, upon crawling it will only crawl the unique URLs (2 in this case).

Although it’s only crawled 2 unique URLs from the 4 uploaded, you are still able to export your original uploaded list in the same order.

Exporting

You can just use the ‘export button’ on any tab as usual to export data from list mode.

However, if you wish to export data in list mode in the same order it was uploaded to match back against other data, then use the ‘Export’ button which appears next to the ‘upload’ and ‘start’ buttons at the top of the user interface.

The data in the export will be in the same order and include all of the exact URLs in the original upload, including duplicates or any fix-ups performed.

The ‘Original URL’ is the URL uploaded, while the ‘Address’ is the URL the SEO Spider crawled.

Advanced Crawling In List Mode

List mode is really powerful with the right set-up. There are a few cool advanced uses that you should know to help you laser-focus your analysis and save time and effort.

Crawling A List of URLs, & Another Element

List mode can be super flexible and allow you to crawl the list of URLs you upload, and another element.

For example, if you wanted to crawl a list of URLs and its images. Or you needed to audit a list of URLs and their newly implemented canonicals, AMP, or hreflang rather than the whole site. Or you wanted to gather all external links from a list of URLs for broken link building. You can perform all of these in list mode, and the process is virtually the same.

Once you’re in list mode, remove the crawl depth limit that is automatically set to ‘0’. Go to ‘Config > Spider > Limits’ and untick the configuration.

This means the SEO Spider will now crawl your list of URLs – and all URLs on the same subdomain that they link to.

Therefore you need to control what is crawled specifically via the granular configuration options. Navigate to ‘Config > Spider > Crawl’. Disable all ‘Resource Links’ and ‘Page Links’ in the configuration menu for ‘Crawl’.

Then select the element(s) you wish to ‘Crawl’ alongside your list of URLs. For example, if you wanted to crawl the list of URLs and its images, the set-up would be.

And if you upload a single URL, such as the SEO Spider page, you can see both the page and its images are crawled.

Crawling a List of URLs & A Specific Element

This advanced configurability allows for laser-focused auditing of precisely the link elements you require.

Auditing Redirects

If you’re auditing redirects in a site migration it can be particularly useful to crawl their target URLs and any redirect chains encountered. This saves having to upload multiple lists of target URLs each time to get to the end.

In this scenario, we recommend using the ‘always follow redirects’ configuration under ‘Config > Spider > Advanced’. Enabling this configuration means the ‘crawl depth limit’ is ignored and redirects will be followed until they reach a non-3XX response (or your until your ‘Max Redirects To Follow‘ limit is reached under ‘Config > Spider > Limits’).

If you then use the ‘All Redirects’ report, it will map out the full chain of redirects in a single report.

Please read our guide on How To Audit Redirects In A Site Migration for more detail on this process.

Connecting To APIs

In list mode you can connect to GA, GSC, PSI and backlink analysis tool APIs to pull in data. For example, you can connect to the Ahrefs API, and pull in data such as referring domains, keywords, traffic and value, which is then displayed in the ‘Link Metrics’ tab.