November 28, 2012

Why might GoogleBot get errors when trying to access my robots.txt file?.



GoogleBot

A great thing about Google is, it gives webmasters all the help they need to get their websites into Google's index. There's a nice tool available in Google Webmaster Tools called 'Fetch as GoogleBot'. This tool, as we discussed in our SEO Tips for start-ups, can be a great help diagnosing errors and getting a website in Google's index faster. A robots.txt file is used for crawling efficiency, and preventing certain pages from being crawled etc. Sometimes though, GoogleBot might have difficulty fetching your robots.txt file. Here's a solution from Google to this problem.


The original question asked o the GWT forum had to with crawling inefficiency. The GoogleBot was unable to crawl a robots.txt file 50% of the time, even though the file could be fetched from other hosts with a 100% success rate. It is worth noting that this was on a plain nginx server and a mit.edu host, so that should have a pretty good up-time. So the problem seems to be with Google, right?


Sometimes, people try cloaking on their websites. Cloaking means hiding content from crawlers, so that different content is served to crawlers and users. So what a user might see on their websites might be a lot different than what crawlers such as GoogleBot see. Not only is this a bad SEO practice, it can also have consequences.

During cloaking, people sometimes make a mistake, and end-up reverse-cloaking. So while browsers and user agents see the website fine, crawlers don't see any content. Making such a mistake is like axing your own foot. So this could be one of the reasons to the problem.

As we discussed about at the start, the Fetch as GoogleBot feature in Google Webmaster Tools is a pretty awesome tool. You can use it to fetch your robots.txt file. t will tell you when there's a problem. Many people might not know this, but sometimes, their web hosts might alternate between different systems and hosts. So a 50% success rate might be accounted for one of the hosts being improperly configured. You might want to contact your hosting company about this.

These two could be the most probable causes for robot.txt crawling errors. Did this help? Please do let us know. And stay tuned for more SEO questions and their answers :)




About Author:

Qasim Zaib is a passionate blogger who enjoys writing articles on Technology, Make Money Online and Blogging niche. He is our Gold Star Contributor and co-author of all blogs under our network.

Follow him @ Twitter | Facebook

If you don't want to get yourself into Serious Technical Trouble while editing your Blog Template then just sit back and relax and let us do the Job for you at a fairly reasonable cost. Submit your order details by Clicking Here »

15 comments : Post Yours! Read Comment Policy ▼
PLEASE NOTE:
We have Zero Tolerance to Spam. Chessy Comments and Comments with Links will be deleted immediately upon our review.

  1. Really Nice information Thanks for sharing this post. :)

    Regards
    Rahul Kashyap
    Blog: SDMMovies.com | Movies4u.US

    ReplyDelete
  2. Really informative post i really like your way of writing. Please update more posts about improving webmasters skills

    Register .com and .info domain in 0.99$

    ReplyDelete
  3. @Technononimous Tweaker

    Thanks for the post ,

    I have had issue for my site Anna University , in which i've set to hide particular Category posts on the homepage and in feeds, so , does this cause the problem ??

    " Google can't able to access your site robot.txt " error ??

    Tnea 2013 application

    ReplyDelete
  4. thnx for nice post
    http://stufftop.blogspot.com/

    ReplyDelete
  5. Very nice tutorial . I also face the same problem in past . Keep writing .

    WordPress And Blogspot Tutorial

    ReplyDelete
  6. mera dost bahut assa article. mere km ayaga.
    my book site: bangla books

    ReplyDelete
  7. Muhammad Mustafa Bhai...
    My site is www.ccolumns.com before 2 days ago my site traffic is contantly increase day by day like 2500 per day but recently my site traffic is totally down. It going to below 200 or 300.
    What happening this please tell me i really want a help from you.
    God Bless You

    ReplyDelete
  8. please any body reply of my above question please.

    ReplyDelete
  9. This comment has been removed by the author.

    ReplyDelete
  10. This comment has been removed by the author.

    ReplyDelete
  11. Your blog post GoogleBot feature in Google Webmaster Tools is a pretty awesome tool. You can use it to fetch your robots.txt file is excellent Actually, i have seen your post and That was very educational and very entertaining for me. gratitude for blog posting Really Such Things. I should suggest your site to my friends.

    msc dissertation

    ReplyDelete
  12. maybe I false use robots.txt , so my blog don't index by google :(

    ReplyDelete
  13. My blog is 8 months old, but it take upto 3 to 4 days to index my posts on Google. Please tell me the solution. I will be very thankful to you.

    Blog: www.bloggerzhelp.com

    ReplyDelete
  14. I faced the same problem with my blog. Once google messaged me that they couldn't access site. So, I felt really depressed. I was looking for solution and finally came here.

    The problem was inappropriate robots.txt. So, I solved the problem and the google bot could again access the site. I also make an idea to aware people about it. So, I decided to write a post on how to optimize wordpress robots.txt. It is especially for wordpress users.

    ReplyDelete

 
  • MBT Icons and buttons

    Icons and Buttons

    Our resources have been successfully downloaded over 10K times and found almost every where. Get yours!

  • choosing webhost for a blog

    Why HostGator?

    Learn Why we chose HostGator as our Web Host and find discount coupons to kick start your blog today!

  • SEO Settings for blogger

    ALL IN ONE SEO PACK 2012

    Learn every single SEO tip that will boost your blog's ranking and organic traffic. We got them all!

  • Blogger widgets and plugins

    Visit MBT's Blogger LAB

    Why not take a tour of all great Blogger widgets published so far? You Name it we have it!

  • become a six figure blogger!

    Become a SIX FIGURE BLOGGER

    Learn what it takes to become a successful entrepreneur and build a living online!