Search Results for:
*
Screaming Frog SEO Spider Update – Version 9.0
I’m delighted to announce the release of Screaming Frog SEO Spider 9.0, codenamed internally as ‘8-year Monkey’. Our team have been busy in development working on exciting new features. In our last update, we released a new user interface, in...
Google Search Console robots.txt Tester Inconsistencies
We’ve had a few customers notice that our robots.txt tester tool, in version 8.1 of the SEO Spider, gives different results than the robots.txt tester in Google Search Console . This variation in behaviour is around the handling of non-ASCII...
Screaming Frog Drinking Bottles Giveaway
It’s time for another giveaway. This time, you can win some very cool Screaming Frog aluminium drinking bottles. You’ve probably been told you should drink more (water), so why not keep hydrated in SEOtyle? Sorry. The large capacity (770ml) and...
Search function
The search box in the top right of the interface allows you to search all visible columns. It defaults to regular text search of the ‘Address’ column, but allows you to switch to regex, choose from a variety of predefined...
How Accurate Are Website Traffic Estimators?
If you’ve worked at an agency for any significant amount of time, and particularly if you’ve been involved in forecasting, proposals or client pitches, you’ve likely been asked at least one of (or a combination or amalgamation of) the following...
Exclude
Configuration > Exclude The exclude configuration allows you to exclude URLs from a crawl by using partial regex matching. A URL that matches an exclude is not crawled at all (it’s not just ‘hidden’ in the interface). This will mean...
URL rewriting
Configuration > URL Rewriting The URL rewriting feature allows you to rewrite URLs on the fly. For the majority of cases, the ‘remove parameters’ and common options (under ‘options’) will suffice. However, we do also offer an advanced regex replace...
Robots.txt
The Screaming Frog SEO Spider is robots.txt compliant. It obeys robots.txt in the same way as Google. It will check the robots.txt of the subdomain(s) and follow (allow/disallow) directives specifically for the Screaming Frog SEO Spider user-agent, if not Googlebot...
Crawling
The Screaming Frog SEO Spider is free to download and use for crawling up to 500 URLs at a time. For £199 a year you can buy a licence, which removes the 500 URL crawl limit. A licence also provides...
How do I extract multiple matches of a regex?
If you want all the H1s from the following HTML: <html> <head> <title>2 h1s</title> </head> <body> <h1>h1-1</h1> <h1>h1-2</h1> </body> </html> Then we can use: <h1>(.*?)</h1>
