How To Fetch & Render (Almost) Any Site
Dan Sharp
Posted 5 April, 2017 by Dan Sharp in SEO
How To Fetch & Render (Almost) Any Site
From time to time, we’ve struggled to get access to Google Search Console quickly from a client to perform a fetch and render. Or, we’ve just wanted to perform a fetch and render on a third party site that isn’t verified to us, to understand how it renders, or test if they are cloaking (by more than just user-agent).
We developed a fetch and render feature in the Screaming Frog SEO Spider, and Max Prin of Merkle has released a very useful web based fetch and render tool.
While these tools emulate how Google render, only Google’s own fetch and render will show you exactly how they see and respond to a page. So, internally we also use a little hack to render pages from unverified sites in Google Search Console.
The Simple iframe Hack
Essentially, all you need to do is create a page on a site you already have verified in Google Search Console, which contains an iframe of the page you wish to render using the desktop (1024) or mobile (411) width dimensions that Google use within fetch and render. Then set the height at 10,000 pixels, which means the scroll bars will be on the browser, rather than the iframe when you look at the results in Search Console.
A basic page, like the below, will do.
<html>
<head><title>Fetch & Render Proxy</title>
</head>
<body style="margin:0px;padding:0px">
<iframe src="http://www.example.com/" frameborder="0" style="overflow:hidden;height:10000;width:1024" height="10000" width="1024"></iframe>
</script>
</body>
</html>
Then you can just fetch and render that page within Google Search Console.
Making It Easier
Rather than having to edit HTML and upload the file to your server each time, you can use PHP to read the website you wish to perform a fetch and render on from a query string parameter. This is then used as the src for the iframe. You can also choose mobile or desktop rendering using the type parameter.
For example, you can see a desktop and mobile fetch and render of Techcrunch , on our screamingprojects.com domain. Here’s how it looks in Google Search Console after you perform a fetch and render there.
You can download our fetch and render code on Github. This means you can simply change the ‘example.com’ domain to perform a fetch and render with the following URL queries on yourverifieddomain.com.
http://www.yourverifieddomain.com/fetch-and-render/?url=http://www.example.com/&type=mobile
http://www.yourverifieddomain.com/fetch-and-render/?url=http://www.example.com/&type=desktop
A couple of things to note –
- You can’t load insecure (HTTP) iframes on a secure (HTTPS) domain. So, it’s best to set this up on an insecure domain.
- Not every website allows you to use it within an iframe in this way. On that note, John Mueller suggested it would be wise to remind everyone of the X-Frame-Options HTTP header which can be used to indicate whether or not a browser should be allowed to render a page in a frame, iframe or object.
That’s it! We hope this little hack is helpful. Enjoy.
Will be using this today on some single page apps with good rankings to see if Fetch and Render is irrelevant for SPAs.
What was your finding?
Doesn’t work for SPAs
Brilliant, just brilliant! A simple solutions for a complicated problem!
Hi,
very useful tool!! Congratulations!
I tried it for some sites and for one of this i receive a blank page.
How can i check what is the problem?
Thanks
I’m also getting the blank page wpseoit noted – I copy/pasted the HTML code example into a textpad, FTP uploaded, tried a couple different URLs, got blanks in GSC in both cases, any ideas to fix? Thanks!
@WPSEO.it & David,
It won’t work for every site, like Google, as the X-Frame-Options HTTP header (https://developer.mozilla.org/en/docs/Web/HTTP/Headers/X-Frame-Options) can be used etc.
Try it with the same example as us (Techcrunch), and if it doesn’t render then feel free to share your live URL and we can have a look.
Cheers.
Dan
Thanks, Dan – it kind of worked:
I used this code:
Fetch & Render Proxy
And here’s how it fetched/rendered in GSC: https://www.screencast.com/t/ThvVp5asDL
You able to share the full URL (feel free to send to support@ if confidential), so we can check the code?
Cheers.
Dan
Awesome! Thank you for providing the code on Github.
Thanks, Dan – just sent from my @3qdigital email. Cheers.
Cheers, mate – replied anyway!
What an ingenuous idea, although is there any real practical uses other than testing stuff out?
How do you automate this? Does google provide a search console API to run “fetch as google”? If not, what’s the point running inside Google search console?
Hey Angelo,
They don’t for fetch as Gbot. The point of fetching through Search Console, is to test how Google respond and render the content.
Cheers.
Dan
I was reading this post yesterday ( April 5, 2017) https://www.screamingfrog.co.uk/crawl-javascript-seo/ and the last paragraph http://nimb.ws/omevn4 got me thinking how can I use google’s fetch and render tool to test a third party site, and today I am testing this feature, it’s a good feeling, Thanks Dan!
No worries, thanks Russell.
Gold. Thanks!
Brilliant. Now Google is going to fix this. Thank you. Thank you very much.
Thanks for this post!
Thank you for this – good stuff :)
Hello,
This is a useful hack to fetch and render pages, especially on your competitor’s pages. Everything works properly.
Thanks for sharing!
Best regards,
Ivan
Thank you for sharing this.
One of the best. Amazing work guys.
A simple solutions for a complicated problem. Thanks for this solution.
It’s a nice feature. Let me try!
This is a very interesting approach. I guess iframes still have their place in society. Thanks!
Never would have thought it was this easy! Never would have thought of this solution myself either, haha.
Brilliant! Thank you!
It’s a nice feature. Let me try!
I am facing rendering issue where Googlebot cannot see even half of my website page, like this one https://www.stemjar.com/watch-free-movies-online/. You can see the snapshot here – https://www.screencast.com/t/lJXReHf8 . I tried to implement your solution but that does not improve my fetch and render result. I am getting the same result as before.
Can you help me in this?
Recently google has changed the search console tool and the new feature of Testing live URL is amazing. It can give you brief overview of Fetch & rendered results. Just put the URL in inspection box, click on request indexing (if not indexed in case of already indexed URL’s test live url option is available automatically) and See the HTML, Screenshot and can even check the page resource code, HTTP response and Java Script Console messages.
Thanks for this post, it’s a nice feature. Let me try!