09 Jul 09 SEO SIMPLIFIED – Robot.txt

Robot.txt Protocol, also known as Robot Exclusion Protocol or Robot exclusion standard is used by the webmasters to instruct the web spiders about their web sites. Robots or web spiders are the programs that search engines use to scan through each and every page of the website to categorize and archive the web site.

Let’s understand this by an example. Let’s say we have a website, http://www.seosimplified.com, which we want to get indexed by every major search engines. To assist the search engines in indexing the web site properly, we create an XML sitemap (Click here to read more about XML Sitemaps). So far, so good, but there are pages that we do not want search engines to see, what should we do? We write an exclusion protocol for the web robots, and that exclusion protocol is commonly known as Robot.txt.

Now, you will be thinking, why we will not like search engines to see some of the pages? Well, there can be two reasons: (1) the information on the page is too critical to be shared freely on the Internet, and (2), the pages are restricted for paid or registered users.

How Robot.txt Works

When a search engine spider comes to our http://www.seosimplified.com to index its web pages, it goes to Robot.txt see what protocol has been set for it before going forward with indexing anything that comes its way. Suppose a search spiders sees the following items written in Robot.txt:

User-agent: *

Disallow: /

Then it will interpret this message in the following way:

“User-agent: *” will say to the search spiders that the protocol written below is applicable to all types of web robots and

“Disallow: /” means the robots are instructed not to visit any page on the website.

Two Caveats of Robot.txt

I have heard many a newbie exclaim WOW on the first exposure to Robot.txt. They just get thrilled by the idea of controlling search engine spiders, but let me tell you do not use Robot.txt casually to hide information from the search engine spiders. And these are the reasons why I am instructing you so:

1. The Robot.txt file is publically available, so any evil web master or malicious code will have access to it, which they can use to understand what parts of your web server you are not allowing them to see. This information will be enough to put you in uncomfortable position vis-à-vis hackers.
2. Not all the robots follow the instruction written in a Robot.txt file. Malicious spiders do not look at it because their intension is not to index your web pages but to exploit the security vulnerabilities.
3. If you only use Robot.txt to prohibit access to certain part of your web page then they come to know where to target, and they already know how.

I hope this discussion on Robot.txt must have helped you understand Robot.txt and its application.

Tags: Caveats, First Exposure, Major Search Engines, Newbie, Robot Exclusion Protocol, Robot Exclusion Standard, Robot Txt, Search Engine Spider, Search Engine Spiders, Search Spiders, SEO, Sitemaps, Web Pages, Web Robots, Web Spiders, Xml Sitemap