How to Create Robot.txt File
Robots.txt file is very important for search engine optimization. It is a file placed in your server to notify various of search engine’s spiders not to crawl or index certain sections of your site. All search engines now look for a robots.txt file as soon their spiders or bots arrive on your site. There are a number of situations where you may wish to exclude spiders from some or all of your site. Do not want the unfinished pages to be appeared in search engines.
- Sensitive and security pages would prefer not to appear in search engines.
- Some directories they would prefer were not crawled such as cgi-bin or a directory that simply contains thank you or error pages.
- Doorway pages to avoid being penalized for spamming a search engine with a series of overly similar pages.
- To exclude some spiders from search engines you do not want to appear in collecting email addresses.
- Some webmaster would like to exclude blog comments to be index by search engines.
Creating the Robots.txt File
Format for robot.txt file will be :
| http://www.example.com/robot.txt |
There is nothing difficult about creating a basic robots.txt file. It can be created using notepad or whatever is your favorite text editor. Each entry has just two lines:-
User-Agent: [Spider or Bot name]
Disallow: [Directory or File Name]
1. Allow all spiders to index everything
| User-agent: * Disallow: |
2. Allow Google spider to index everything
| User-agent: Googlebot Disallow: |
3. Allow Lycos spider to index everything
| User-agent: T-Rex Disallow: |
4. Allow Altavista spider to index everything
| User-agent: Scooter Disallow: |
5. Ban all search engines from index everthing
| User-agent: * Disallow: / |
5. Disallow all spiders to index cgi-bin directory.
| User-agent: * Disallow: /cgi-bin/ |
Some users would like to create Robot.txt file using generator such as Submit Corner. Anyway, it will be easier if we install wordpress plugin using KB-Robot.txt. There is an example collection robot.txt for your reference if you do not what file or directories to be excluded from spiders.
If you like this post, please consider buying me a beer or donation to support my project fund. ThanksIf you enjoyed this post, make sure you subscribe to my RSS feed!
October 5th, 2008 at 2:17 am
I have bookmarked this site. Thanks Kfz Versicherung