Why might GoogleBot get errors when trying to access my robots.txt file?.


A great thing about Google is, it gives webmasters all the help they need to get their websites into Google's index. There's a nice tool available in Google Webmaster Tools called 'Fetch as GoogleBot'. This tool, as we discussed in our SEO Tips for start-ups, can be a great help diagnosing errors and getting a website in Google's index faster. A robots.txt file is used for crawling efficiency, and preventing certain pages from being crawled etc. Sometimes though, GoogleBot might have difficulty fetching your robots.txt file. Here's a solution from Google to this problem.

The original question asked o the GWT forum had to with crawling inefficiency. The GoogleBot was unable to crawl a robots.txt file 50% of the time, even though the file could be fetched from other hosts with a 100% success rate. It is worth noting that this was on a plain nginx server and a mit.edu host, so that should have a pretty good up-time. So the problem seems to be with Google, right?

Sometimes, people try cloaking on their websites. Cloaking means hiding content from crawlers, so that different content is served to crawlers and users. So what a user might see on their websites might be a lot different than what crawlers such as GoogleBot see. Not only is this a bad SEO practice, it can also have consequences.

During cloaking, people sometimes make a mistake, and end-up reverse-cloaking. So while browsers and user agents see the website fine, crawlers don't see any content. Making such a mistake is like axing your own foot. So this could be one of the reasons to the problem.

As we discussed about at the start, the Fetch as GoogleBot feature in Google Webmaster Tools is a pretty awesome tool. You can use it to fetch your robots.txt file. t will tell you when there's a problem. Many people might not know this, but sometimes, their web hosts might alternate between different systems and hosts. So a 50% success rate might be accounted for one of the hosts being improperly configured. You might want to contact your hosting company about this.

These two could be the most probable causes for robot.txt crawling errors. Did this help? Please do let us know. And stay tuned for more SEO questions and their answers :)

Need Quick Help within 24 Hours? ASK NOW

If you don't want to get yourself into Serious Technical Trouble while editing your Blog Template then just sit back and relax and let us do the Job for you at a fairly reasonable cost. Submit your order details by Clicking Here »


We have Zero Tolerance to Spam. Chessy Comments and Comments with 'Links' will be deleted immediately upon our review.
  1. Really Nice information Thanks for sharing this post. :)

    Rahul Kashyap
    Blog: SDMMovies.com | Movies4u.US

  2. Really informative post i really like your way of writing. Please update more posts about improving webmasters skills

    Register .com and .info domain in 0.99$

  3. @Technononimous Tweaker

    Thanks for the post ,

    I have had issue for my site Anna University , in which i've set to hide particular Category posts on the homepage and in feeds, so , does this cause the problem ??

    " Google can't able to access your site robot.txt " error ??

    Tnea 2013 application

  4. thnx for nice post

  5. Very nice tutorial . I also face the same problem in past . Keep writing .

    WordPress And Blogspot Tutorial

  6. mera dost bahut assa article. mere km ayaga.
    my book site: bangla books

  7. Muhammad Mustafa Bhai...
    My site is www.ccolumns.com before 2 days ago my site traffic is contantly increase day by day like 2500 per day but recently my site traffic is totally down. It going to below 200 or 300.
    What happening this please tell me i really want a help from you.
    God Bless You

  8. please any body reply of my above question please.

  9. This comment has been removed by the author.

  10. This comment has been removed by the author.

  11. Your blog post GoogleBot feature in Google Webmaster Tools is a pretty awesome tool. You can use it to fetch your robots.txt file is excellent Actually, i have seen your post and That was very educational and very entertaining for me. gratitude for blog posting Really Such Things. I should suggest your site to my friends.

    msc dissertation

  12. maybe I false use robots.txt , so my blog don't index by google :(

  13. My blog is 8 months old, but it take upto 3 to 4 days to index my posts on Google. Please tell me the solution. I will be very thankful to you.

    Blog: www.bloggerzhelp.com

  14. I faced the same problem with my blog. Once google messaged me that they couldn't access site. So, I felt really depressed. I was looking for solution and finally came here.

    The problem was inappropriate robots.txt. So, I solved the problem and the google bot could again access the site. I also make an idea to aware people about it. So, I decided to write a post on how to optimize wordpress robots.txt. It is especially for wordpress users.