Directives: None

URLs containing a ‘none’ directive in either a robots meta tag or X-Robots-Tag in the HTTP header.

The ‘none’ directive is the equivalent to ‘noindex, nofollow’.

The page will still be crawled (to see the directive), but it will then be dropped from the index.

How to Analyse in the SEO Spider

View URLs with this issue in the ‘Directives’ tab and ‘None’ filter, and export all URLs using the ‘Export’ button.

To crawl pages with a nofollow directive within the SEO Spider, enable ‘Follow Internal Nofollow’ via ‘Config > Spider > Crawl’.

What Triggers This Issue

This issue is triggered when URLs contain a ‘none’ directive, either in a robots meta tag within the HTML or in an X-Robots-Tag in the HTTP header.

For example:

https://www.screamingfrog.co.uk/

Has the meta tag:

<meta name="robots" content="none">

How To Fix

URLs with a ‘none’ should be reviewed carefully to ensure they are correct and shouldn’t be indexed. If these pages should be indexed, then the ‘none’ directive should be removed.