I was browsing Webmaster Tools recently and noticed something was missing from the “Labs” drop down. There used to be three items but something was now missing and I couldn’t for the life of me work out what it was. I asked a colleague and he said it was the “Fetch as Googlebot” feature. He was right. After being in the Labs section for as long as I can remember, it looks like the Fetch as Googlebot feature is finally a completed and accepted feature.
Our Fetch as Googlebot section in Webmaster Tools already has some entries so we’ll take a look at those in a moment.
For now, let’s look at the options you get given. You get to enter the URL from your site to check and also get to select the type of page so Googlebot knows which crawler to use. These options are Web, Mobile XHML/WML and Mobile cHTML. For this demonstration we’ll be using Web.
Lets enter a URL to try and see what it gives us.
The Googlebot takes a few moments to crawl the page you asked for. If you refresh the page after a few seconds, it should say “Success”.
If you click the Success text, you will be given the page as Googlebot saw it in raw form.
As you can see here, I neglected to add a trailing forward slash at the end of the URL I wanted to check, so it’s given me the 301 redirect page. Along with the source code we have the document and server headers. The headers are a great source of information, as it can tell you the status code (in this case a 301 redirect), the content type, the location, the request time and date, and the server type along with its list of attributes.
The main reason we would need to use this tool is simple; checking for cloaked content. Cloaked content can be done on purpose or accidentally. If Google discovers cloaked content, it can and will penalise you for trying to hide your sites real content. This tool helps you discover if your site might be showing content to Google that you might not be seeing, perhaps based on user agent of the browser or because of IP restrictions.
Related Posts
