Search Results for:
*
How does the Spider treat robots.txt?
The SEO Spider is robots.txt compliant. It checks robots.txt in the same way as Google. It will check robots.txt of the (sub) domain and follow directives specifically any for Googlebot, or for all user-agents. You are able to adjust the...
Why isn’t my Include/Exclude function working?
The Include and Exclude are case sensitive, so any functions need to match the URL exactly as it appears. Please read both guides for more information. Functions will be applied to URLs that have not yet been discovered by the...
Web Scraping & Custom Extraction
Scrape any data from the HTML of a page using CSS Path, XPath and regex to enhance a crawl, such as author name, comments, shares or more.
Screaming Frog SEO Spider Update – Version 4.0
I’m really pleased to announce version 4.0 of the Screaming Frog SEO Spider, codenamed internally as ‘Ella’. We have been busy in development working on some significant improvements to the SEO Spider due for release later this year, which include...
Screaming Frog Changes For VAT In 2015
I’m writing this post on the eve of 2015, as I prepare to update 28 VAT rates for each country in the EU as they enter the New Year. We first found out about the EU VAT changes on the...
Screaming Frog SEO Spider Update – Version 2.30
I am really pleased to announce version 2.30 of the Screaming Frog SEO spider code-named ‘Weckl’. This update includes new features that provide greater control of crawls, as well as how the data is analysed. Thanks again to everyone for...
Negative SEO - Should You Monitor Your Backlinks?
Historically links were often either counted or ignored by Google in their scoring. However, as their algorithms evolved over the past 5 years and more recently with the introduction of Penguin in April 2012 and increase in manual actions, links...
The 11 Most Infamous Google Bombs In History
Google bombing is a practise whereby a specific web page is targeted to rank in 1st position in the SERPs for a particular search phrase, so that when that phrase is typed in Google it brings often humorous or controversial...
Screaming Frog at LinkLove London
Screaming Frog at LinkLove London 2012 Last Friday the Screaming Frog SEO team ventured out of our comfortable setting of sunny Henley-on-Thames, to head into the Big Smoke to attend Distilled’s much lauded LinkLove conference. Distilled have been running successful...
