Screaming Frog SEO Spider Update – Version 17.0
Dan Sharp
Posted 17 August, 2022 by Dan Sharp in Screaming Frog SEO Spider
Screaming Frog SEO Spider Update – Version 17.0
We’re pleased to announce Screaming Frog SEO Spider version 17.0, codenamed internally as ‘Lionesses’.
Since releasing the URL Inspection API integration in the SEO Spider and the launch of version 5 of the Log File Analyser, we’ve been busy working on the next round of prioritised features and enhancements.
Here’s what’s new in our latest update.
1) Issues Tab
There’s a new ‘Issues’ right-hand tab, which details issues, warnings and opportunities discovered.
This tab has been introduced to help better direct users to potential problems and areas for improvement, based upon existing filters from the ‘Overview’ tab.
An in-app explanation of each issue and potential actions is provided in English, German, Spanish, French and Italian.
Each issue has a ‘type’ and an estimated ‘priority’ based upon the potential impact.
- Issues are an error or issue that should ideally be fixed.
- Opportunities are potential areas for optimisation and improvement.
- Warnings are not necessarily an issue, but should be checked – and potentially fixed.
E.g – An ‘Internal URL Blocked by Robots.txt’ will be classed as a ‘warning’, but with a ‘High’ priority as it could potentially have a big impact if incorrectly disallowed.
For experienced users, the new Issues tab is a useful way to quickly identify top-level problems and dive straight into them as an alternative to the Overview tab. For users with less SEO expertise, it will help provide more direction and guidance on improving their website.
All Issues can be exported in bulk via ‘Bulk Export > Issues > All’. This will export each issue discovered (including their ‘inlinks’ variants for things like broken links) as a separate spreadsheet in a folder (as a CSV, Excel and Sheets).
The new Issues tab also works with crawl comparison similar to the Overview tab, to allow users to identify where issues have changed and monitor progress over time.
The SEO Spider has always been built for experts and we have been reticent to ever tell our users how to do SEO, as SEO requires context. The new issues tab does not replace SEO expertise and a professional who has that context of the business, strategy, objectives, resource, website and nuances related to SEO and prioritising what’s important.
The new issues tab should provide helpful hints and support for an SEO who can make sense of the data and interpret it into appropriate prioritised actions.
2) Links Tab
There’s a new ‘Links’ tab which helps better identify link-based issues, such as pages with a high crawl-depth, pages without any internal outlinks, pages using nofollow on internal links, or non-descriptive anchor text.
Filters such as ‘high’ internal and external outlinks and non-descriptive anchor text can be customised under ‘Config > Spider > Preferences’ to user-preferred limits.
All data can be exported alongside source pages via the ‘Bulk Export > Links’ menu as well.
3) New Limits
Users are now able to control the number of URLs crawled by URL Path for improved crawl control and sampling of template types.
Under ‘Config > Spider > Limits’ there’s now a ‘Limit by URL Path’ configuration to enter a list of URL patterns and the maximum number of pages to crawl for each.
In the example above a maximum of 1,000 product URLs will be crawled, which will be enough of a sample to make smarter decisions.
Users can also now ‘Limit URLs Per Crawl Depth’, which can help with better sampling in some scenarios.
4) ‘Multiple Properties’ Config For URL Inspection API
The URL Inspection API is limited to 2k queries per property a day by Google.
However, it’s possible to have multiple verified properties (subdomains or subfolders) for a website, where each individual property will have a 2k query limit.
Therefore, in the URL Inspection API configuration (‘Config > API Access > GSC > URL Inspection’) users can now select to use ‘multiple properties’ in a single crawl. The SEO Spider will automatically detect all relevant properties in the account, and use the most specific property to request data for the URL.
This means it’s now possible to get far more than 2k URLs with URL Inspection API data in a single crawl, if there are multiple properties set up – without having to perform multiple crawls.
Please use responsibly. This feature wasn’t built for the purpose to circumvent Google’s limits and motivate users to create many different properties to get indexing data for every URL on a website. Google may adjust this limit (to domain etc) if it’s abused.
5) Apple Silicon Version & RPM for Fedora
There’s now a native Apple Silicon version available for users on M1/2 macs, and an RPM for Fedora Linux users.
In limited internal testing of the native Silicon version we found that:
- Crawling a locally hosted site with very little latency resulted in the crawl running twice as quickly.
- Loading in a saved crawl was 4 times faster.
The native experience is just so much smoother overall in comparison to using the Rosetta2 emulation layer.
6) Detachable Tabs
All tabs are now detachable. Users can right-click and ‘detach’ any tab from the main UI and move to a preferred position (across multiple screens).
This is pretty cool for users that like to keep one eye on a crawl while doing other things, or analysing all the things at once.
There are options to ‘re-attach’, ‘pin’ – and reset all tabs back to normal when it’s all too much, too.
Other Updates
Version 17.0 also includes a number of smaller updates and bug fixes.
- The ‘Response Codes’ tab now has an additional ‘Internal’ and ‘External’ filter, which is reflected in the Overview tab and Crawl Overview Report. Thanks to Aleyda for the nudge on that one!
- Tabs can be re-ordered more efficiently by dragging, or via a new ‘configure tabs’ menu available via a right-click on a tab or the tab down arrow.
- The main scheduling UI has been updated with details of task name, next run, interval and has improved validation for potential issues that might stop it launching.
- There’s a new ‘Save & Open’ option for exports, which works for all formats.
- GA & GSC configs now have date pre-sets (last week, last month etc) which can be saved as part of the configuration and supplied in scheduled crawls.
- The PSI integration now has Interaction to Next Paint (INP) and Time to First Byte (TTFB) metrics available.
- The ‘Ignore Non-Indexable URLs for On-Page Filters’ configuration has been updated to ‘Ignore Non-Indexable URLs for Issues‘ and will by default, not flag non-indexable URLs for checks related to issues across a wider range of tabs and filters as detailed in-app and in the user guide.
- The hreflang tag limit has been increased from 200 to 500, for websites that have
lost the plotlots of alternative localised URLs. - All new tabs, filters and issues, warnings and opportunities counts are available in the ‘Export for Data Studio’ and Data Studio Crawl Report via scheduling.
- The SEO Spider has been updated to Java 17.
- Fixed lots of small bugs that nobody wants to read about, particularly when everyone is on holiday and nobody is reading this.
We hope the new update is useful.
A big thank you goes out to our beta testers who have helped with the different language versions of our new in-app issue descriptions. In particular, MJ Cachón for translation help, a new Spanish Google Data Studio crawl template – and generally being awesome.
As always, thanks to the SEO community for your continued support, feedback and feature requests. Please let us know if you experience any issues with version 17.0 of the SEO Spider via our support.
Small Update – Version 17.1 Released 23rd August 2022
We have just released a small update to version 17.1 of the SEO Spider. This release is mainly bug fixes and small improvements –
- Fix issue preventing scheduling working on Ubuntu.
- Fix issue preventing JavaScript crawling working on Ubuntu.
- Fix cookies issues around forms based authentication.
- Fix JavaScript crawling failing to render some pages.
- Fix crash when changing filters in the Response Codes tab.
- Fix crash crawling URLs with HTTP in the domain name.
- Fix crash selecting ‘Save & Open’.
Small Update – Version 17.2 Released 12th September 2022
We have just released a small update to version 17.2 of the SEO Spider. This release is mainly bug fixes and small improvements –
- Reinstate auto scrolling behaviour at the bottom of table view, so users can see URLs appear as they are dicovered. Thanks to Pete Mindenhall, Steve Morgan and a few others that spotted it.
- Add handling for webp files in bad content filter.
- Fix regression in cookie handling not showing all cookies.
- Fix issue showing incorrect values for Total Internal/External URLs when ignoring URLs blocked by robots.txt.
- Fix issue with last mode used not being persisted after restarts.
- Fix issue with new Issues tab graphs not updating during a crawl.
- Fix issue with new task dialog showing single digits for time.
- Fix issue with selected filter being lost when returning to a tab.
- Fix issue with old Response Codes 5XX export tab not working on upgrade.
- Fix issue with Spelling & Grammer crashing when used on Apple Silicon.
- Fix issue with Response Codes 4XX filtering showing 2XX responses after re-spider & reload.
- Fix issue with grabled fonts affecting users with their own version of Roboto font installed.
- Fix crash in view source panel.
- Fix crash crawling robots.txt with long Sitemap: line.
- Fix crash around detachable tabs and full screen on macOS.
- Fix crash showing ‘Visualisations > Force-Directed Crawl Diagram’.
- Fix crash removing/re-spidering URL.
- Fix crash viewing scheduling history.
I’ve just started using this free tool and I want to thank the Developers for a continuing to maintain and improve it
Thank you Dan. The Issues Tab looks great. Looking forward to using it.
Great update, again.
HenryV
Thx !..
Great update !
Alright! Every M1 Max user is in heaven now ;)
Awesome! It will be great if we can download straight as a pdf file and email to clients straight away. As most of SEO Agency offers free SEO audit and use random tools to create BS Audit report. Even me I don’t run screaming frog until we get contract signed but now as Screaming Frog itself offering this then would be nice to have these feature as well. Hope I made it clear and not asking more too much. Thanks for the update.
testing new versions all day well done guys
A very useful program to detect issues that affect user experience on websites. What I like best is that there is no forceful push towards one size fits all solutions that is so common with other automatic SEO tools.
As always, thank you for the update and excellent tool. I think the detachable tabs and separating internal/external response codes will be very helpful for my needs.
Thanks man, awesome to hear!
Hope all is good with you.
Dan
Game changer – issues tab was definitely needed. So glad to see it.
Game changer indeed. We saw it on other SEO tools since years and we are so glad we have it in Screaming Frog already. Thanks guys, it will be in great use.
Very nice Update. The Issue Tab is so helpfull.
Did you also change the Font of ScreamingFrog?
Hi Tom,
Good spot! Yes, we switched to use Roboto fairly recently (partly as some system fonts had become unreliable across platforms).
Cheers.
Dan
The new Issues tab is superb and is incredibly useful. A great addition to an already amazing tool.
Hey! We are reading this…and we’re not on holiday (yet) Great new stuff, loving it!
I am curious about how did you estimate impact number of each issue in the issues tab?
Thanks
Hi Salman,
It’s an estimated impact based upon our expertise and common SEO knowledge / best practice.
As per the post above, it doesn’t have context specific to each site or business.
And hence why it’s only a best estimate, to try and identify where you might want to check first.
An experienced SEO still needs to review and prioritise.
Cheers,
Dan
It’s amazing Update..Thanks a lot
Awesome ! Thanks Dan !
Best tool ever !
Been using this tool for a long time and never once I have been disappointed. New updates are fire as well.
Finally the ability to export all of the reports at once! This is awesome, thanks SF team!
Amazing update guys, long time SF fan. Detachable tabs will be a godsend.
After install. bad font – https://i.imgur.com/GfbuCts.jpeg
Pop us an email to support@screamingfrog.co.uk if you need help :-)
Cheers.
Dan
Hii
Can I ask you, how did you solve this problem?
Hi Mert,
It’s fixed in a beta, if you want to email in for it.
Cheers.
Dan
Links Tab and Detachable Tabs are one of the most anticipated improvements. Incredibly useful functionalities. It’s great that Screaming Frog is developing so intensively.
Great update, thank you!
Another great and useful update – thanks!
Been using this tool for a long time, I love new update, more stuff I can dig in. Thanks for all you done till today.
Another amazing update with amazing features!
If i may, I would love to be able to export all the new “Issues” in a unique Excel file divided by tabs, or at least be able to choose which issues to export!
This way users won’t be flooded by so many excel files!
I hope you’ll consider adding this feature in a future update :)
Thank you very much!
This update was so handy and eliminate some of bugs that i faced before it.
keep going and release useful updates like this, thanks.
I’d love to see some easier ways to filter internal links.
I.e. filter by only showing links that appear in the body of a page.
Excluding header, footer and sidebar links.
I understand it can be done already, but making it simpler would be great.
Hi Marc,
Thanks for the suggestion!
Yeah, you can do this already – https://www.screamingfrog.co.uk/how-to-analyse-link-position/
How would you make it simpler? Any thoughts appreciated, welcome to send through via email (support@screamingfrog.co.uk) if easier.
Always open to any new ideas!
Cheers.
Dan
Always new features rolling up. Thank you all team, you do a great job
Finally the ability to export all of the reports at once! This is awesome, thanks SF team!
I’m using this free tool since version 4 and I want to thank the Developers for a continuing to maintain and improve it
I would like to thank everyone who contributed to a great program that eased our workload.
Thank you for the update Dan!
Regards from Spain
well this one made me shout a bit out loud…. “Limit by URL Path”
Hey
Is it just me? I can’t install 17.2. There are write permission problems with some .dll . so i need to abort the installation and screaming frog isn’t usable.
Hi Oliver,
Restart your machine, don’t open the app, and give it another go.
It should work.
Cheers.
Dan
Sorry my fault.
An instance of screaming frog was running in the background….
That’s amazing updates. I am happy to see the new updates for beginners to easy way to find issues, warnings, opportunities,
Thanks, my best wishes to you.
Best regards
Ahmad Iqbal SEO
Epic as usual guys. Great update. Thanks for that.
Screaming Frog is great. Proud to recommend it in my list of the best SEO tools.
I am really happy about this latest update. Screaming Frog is one of the best SEO audit tools on the market. I’ve been using it for a few years now, and I’m so glad you keep adding new features! Thank you so much for the update!
Hi to all the SF team,
We wanted just to express our happiness and satisfaction about this new version of your SEO-Software . As a professional SEO Company working for lot of clients, it helps us a lot running audits and sorting relevant data that help us prioritize our actions.
Thank you for this update and your explanations.
Nice, thanks for the updates. Screaming Frig is still one of my favourite tools for OnPage analysis and the issues tab is very helpful.
great update – as always. Thanks!
Amazing Update Dan! Thanks a lot to you and your Developers team