Directives: Nofollow

URLs containing a ‘nofollow’ directive in either a robots meta tag or X-Robots-Tag in the HTTP header.

This is a ‘hint’ which tells the search engines not to follow any links on the page for crawling.

This is generally used by mistake in combination with ‘noindex’, when there is no need to include this directive as it stops ‘PageRank’ from being passed onwards.

How to Analyse in the SEO Spider

View URLs with this issue in the ‘Directives’ tab and ‘Nofollow’ filter, and export all URLs using the ‘Export’ button.

To crawl pages with a nofollow directive within the SEO Spider, enable ‘Follow Internal Nofollow’ via ‘Config > Spider > Crawl’.

What Triggers This Issue

This issue is triggered when URLs contain a ‘nofollow’ directive, either in a robots meta tag within the HTML or in an X-Robots-Tag in the HTTP header.

For example:

https://www.screamingfrog.co.uk/

Has the meta tag:

<meta name="robots" content="nofollow">

How To Fix

URLs with a ‘nofollow’ should be reviewed carefully to ensure that links shouldn’t be crawled and PageRank shouldn’t be passed on.

If outlinks should be crawled and PageRank should be passed onwards, then the ‘nofollow’ directive should be removed.