How To Compare Crawls

How To Perform A Crawl Comparison

This tutorial explains how to compare crawls using the Screaming Frog SEO Spider and analyse how crawl data has changed in tabs and filters over time.

Crawl comparison helps monitor the progress of SEO issues and opportunities and provides granular data about what has changed between crawls.

This can help save time when monitoring progress and identifying new issues, or changes that might impact organic performance. The crawl comparison feature allows you to –

1. Compare crawl data, issues and opportunities that have changed in any tabs and filters.
2. Compare site structure and changes in directories, pages and crawl depth between crawls.
3. Detect changes in elements and metrics, such as page titles, meta descriptions, crawl depth, word count, internal linking, content and more.

This tutorial provides an overview of how to compare crawls, interpret the data and various reports, and utilise some of the more advanced features such as change detection and URL Mapping in different scenarios.

---

1) Switch To Database Storage Mode

To compare crawls, you need to be in database storage mode and have a licence. You can switch to database storage by selecting ‘File > Settings > Storage Mode’ and ‘Database Storage’.

If you’re new to using database storage mode, any new crawls will now be auto stored and available to be opened via the ‘File > Crawls’ top-level menu. Existing crawls will not appear here, but you can import them via ‘File > Import’.

Database storage has enormous benefits over default memory storage, including improved crawling at scale, auto saving of crawls, rapid opening, and it can even help avoid lost crawls if your machine powers down unexpectedly.

Please see our guide on saving, opening, exporting & importing crawls.

---

2) Select Crawls To Compare

There are two ways you can select crawls to compare.

When in ‘Spider’ or ‘List’ modes, navigate to ‘File > Crawls’ and highlight two crawls by holding down ‘control’ on Windows, or the ‘command’ key on macOS.

Then click ‘Select To Compare’, which will switch you to ‘compare’ mode.

Alternatively, you can switch to ‘compare’ mode by clicking ‘Mode > Compare’ in the top-level menu. The interface will then update, so you can choose crawls for comparison.

Clicking the ‘Select Crawl’ buttons at the top will display the same ‘File > Crawls’ menu, where you can then pick two crawls you wish to compare.

Little tip – You can select two crawls at once (by holding down ‘control’ on Windows, or the ‘command’ key on macOS), you don’t need to click both ‘Select Crawl’ buttons for the current and previous crawl.

---

3) Adjust Comparison Config For Change Detection

The SEO Spider will compare all crawl data in tabs and filters in the right hand ‘Overview’ tab between the two crawls.

However, you can also choose to configure the tool to perform additional analysis around more granular changes to elements and key metrics – such as a page title, meta description, h1, word count, crawl depth, internal links, structured data, content and more.

This is useful when an element has changed, but it hasn’t triggered a filter. For example, a page title might change completely, but this wouldn’t be picked up if it didn’t trigger a filter change such as ‘Missing’, ‘Over X Characters’, ‘Duplicates’ etc. Change detection will let you know if there has been any change to an element or metric.

In ‘Compare’ mode, click on the compare configuration via ‘Config > Compare’ (or the ‘cog’ icon at the top) and select the elements and metrics you want to identify changes in.

Selecting these options will mean the crawl comparison analysis discussed in the next section will take longer to be performed. Once it has run, a ‘Change Detection’ tab will appear in the master view and in the Overview tab, containing filters for any elements and metrics selected with details of changes discovered.

Please note, the ‘Content’ change detection requires both crawls to store the HTML of the pages to analyse their similarity and change. This option takes longer to process, as it uses minhash to compare the similarity of pages and determine the change % between current and previous content.

---

4) Click ‘Compare’ To Perform The Analysis

Next up, you just need to click ‘Compare’. The crawl comparison analysis will then run.

This is often quick, but for larger crawls or with certain change detection elements and metrics selected, it can take a little longer. The progress bar will update, and data will appear when finished.

By default the latest crawl will appear as the ‘Current’ crawl, and the older crawl as ‘Previous’. But these can be switched around by clicking the arrow next to the cog icon at the top of the interface.

---

5) View The Right-Hand ‘Overview’ & ‘Issues’ Tabs

When the crawl comparison analysis has completed, the right-hand overview and issues tabs will populate to show current and previous crawl data and changes.

You can click on the tabs which update the charts below, as well as click on the filters and numbers in columns which update the master window data.

There are four columns (and filters on the master window view) that help segment URLs that have changed in tabs and filters.

1. Added – URLs in previous crawl that moved to filter of current crawl.
2. New – New URLs not in the previous crawl, that are in current crawl and filter.
3. Removed – URLs in filter for previous crawl, but not in filter for current crawl.
4. Missing – URLs not found in the current crawl, that previously were in filter.

In summary, ‘added’ and ‘removed’ are URLs that exist in both the current AND previous crawls. ‘New’ and ‘missing’ are URLs that only exist in one of the crawls.

This helps determine whether new issues or fixes are for existing URLs or new URLs.

You can click on the numbers in the columns to view which URLs have changed, and use the filter on the master window view to toggle between current and previous crawls, as well as added, new, removed or missing.

The overview data can be exported using the ‘Reports > Crawl Overview’ export.

All data can be exported in bulk via ‘Bulk Export’, and reports are also fully accessible via the usual ‘Reports’ menu.

---

6) Analyse Changes In The ‘Change Detection’ Tab

When you have adjusted the compare configuration and the analysis has run, the ‘Change Detection’ tab becomes available and shows the changes in elements and metrics selected.

The ‘Change Detection’ tab will appear as the last tab in the right-hand overview tab, and master view.

The ‘All’ filter in the Change detection tab informs you how many URLs in the current and previous crawl match, and have been analysed for changes.

In the above example, there are a total of 2,236 URLs that exist in both the current and previous crawl that have been analysed for changes.

URLs that don’t exist in either the current or previous crawl and are therefore not used in change detection analysis can be seen under ‘Internal > HTML’ and the ‘New’ and ‘Missing’ columns.

The numbers in the current, previous and change columns will always match up. This is because they are all communicating the number of URLs that have changed between the current and previous crawls. In the example above, 209 URLs have page title changes, 6 have changed in crawl depth etc.

You’re able to click on an element and view the changes in the master window. For example, for page titles you can view current and previous page titles side-by-side to analyse how they have changed.

You’re able to view which pages have changed in word count, to keep an eye on any changes being made by clients or team members.

Or be alerted to changes in crawl depth of pages. Have any key landing pages suddenly changed?

Or lost significant unique internal links, due to changes in internal linking.

And more. You’re able to sort these columns by the absolute change number, or percentage in the interface. You’re also able to click the ‘Changed’ filter and select ‘Not Changed’ if you would like to see URLs which have remained the same.

All of these items can be vital to how a page performs organically and this feature can help monitor and alert you to any changes. This feature can be helpful when reviewing the implementation of SEO recommendations (has the client really updated those page titles?), helping to save time from manually comparing in a spread sheet.

It can also be used in more advanced scenarios, like when performing parity checks, such as comparing a mobile Vs desktop crawl, raw HTML Vs JavaScript enabled crawl or different user-agents.

---

7) View ‘Site Structure’ Tab To Visualise Changes

The ‘Site Structure’ tab displays a directory tree overview of how architecture of the site has evolved between the crawls. You can view where on the site there has been changes at directory level from the previous to current crawl.

For example, you can see new files have been found within the media, animals and reviews directories below.

You’re also able to visualise how crawl depth has changed, which helps understand changes to internal linking, or how deep content sits within the architecture over time.

You can expand directories to see which specific URLs have changed, like in the reviews directory in this example. Green up arrows and numbers mean URLs added, while red down arrows and numbers mean URLs now missing from the crawl.

The site structure data can be exported using the ‘export’ button at the top of the tab.

---

8) Compare Data In Lower Window Tabs

The lower window tabs also allow you to compare data side-by-side between crawls.

If you’ve performed a JavaScript crawl on desktop and mobile to compare, you’re able to select the ‘Compare’ dropdown on the lower window rendered page tab to view the mobile and desktop render side-by-side.

The inlinks and outlinks tabs have an additional filter that is set to ‘Auto’ by default. This means the lower window tab will show link data based upon the upper window view.

In this case, it will show link data for the ‘current’ crawl as selected by the upper master window. But the lower window can be set independently to show data for the ‘previous’ crawl, even if the top window is selected for the current crawl.

The ‘Added’ and ‘Removed’ filters can also highlight changes between the two crawls, and show where internal links have changed.

For example, the Blobfish page shown above has dropped in crawl depth from 1 to 2.

But why is that?

By switching the filter to show ‘Removed’ URLs, you can see it used to have links direct from the homepage, which has been removed in the current crawl.

So this helps clarify why the page changed in crawl depth. The ‘Added’ and ‘Removed’ filter is also really helpful when analysing changes to internal or external unique link metrics and parity checks.

---

How To Compare Staging & Other URL Structures With URL Mapping

You’re able to compare two different URL structures using the ‘URL Mapping’ feature introduced in crawl comparison.

This is helpful in a number of scenarios, such as comparing a live site against staging, or a mobile website against its desktop equivalent. You can compare entirely different hostnames, directories, or more subtle changes to URLs.

To compare two different URL structures in crawl comparison, click on the compare configuration (‘Config > Compare’) and ‘URL Mapping’.

Input a regex to map the previous crawl URLs to the current crawl. For example, Screaming Frog changed host to Kinsta and tested the staging site pre to migrating with URL Mapping with the following.

The Kinsta staging site URLs on a different hostname are ‘mapped’ to match against the existing live site, so the equivalent URLs are compared against each other for overview tab data, issues, and opportunities, the site structure tab, and change detection.

You can switch between the current and previous crawls to view the different URL structures still.

Another scenario for URL Mapping might be comparing mobile and desktop websites.

The Chain Reactions Cycles website has a /mobile/ subfolder as an example, so to compare the mobile website against desktop, the mobile URLs can be mapped to the desktop equivalent with the following URL Mapping.

Finally, if a trailing slash has been removed (or added), you could use the following in URL mapping to strip the trailing slash at the end of a URL.

You’re also able to use a combination of all the above, there is no limit for URL Mapping.

A couple of important points to remember when using URL Mapping –

• You are mapping previous URLs to current URLs. If you’re struggling to map previous to current, you can switch which crawl is current and previous, using the arrows icon at the top of the compare interface.
• You can test your URL Mapping using the ‘Test’ tab within the ‘URL Mapping’ configuration before running the crawl comparison analysis to avoid making mistakes and wasting time.

URL Mapping is an extremely powerful feature that can really help save time during new site releases and migrations where often URL structures are different.

---

How To View Old Crawl Comparisons

Once you have compared crawls a small comparison file is stored within ‘File > Crawls’ alongside your usual stored crawls, which allows you to quickly open and view it, without requiring you to run the analysis again.

The mode column in the Crawls menu will show these files as ‘Compare’. You can use the ‘Show’ filter at the top to select any ‘Compare’ mode crawl files.

The compare file utilises the current and previous crawls stored and compared. Therefore, they are both required to be stored to view the comparison.

If you delete one or both of the crawls in the comparison, the comparison will not be accessible anymore either.

---

Summary

This tutorial will hopefully help you better monitor progress of technical SEO issues and quickly identify changes using crawl comparison in the Screaming Frog SEO Spider tool.

Most importantly, you’re able to dig into the data and see exactly what’s changed, and why.

If you experience any issues when comparing crawls or have any further questions or suggestions for this feature, please contact us via support and we can help.