Pages

Advertisement

Tuesday, July 10, 2007

ROBOTS.TXT Primer - ROBOTS.TXT Analysis

User-agent: EmailCollector
Disallow: /
If you were to copy and paste the above into notepad, save the file as robots.txt and then upload it to the root directory of your server (where you will find your home page)what you have done, is told a nasty email collector to keep out of your website. Which is good news as it may mean less spam!
I do not have the space here for a fully fledged robots.txt tutorial, however there is a good one at
http://www.robotstxt.org/wc/exclusion-admin.html
Or simply use the robotsbeispiel.txt I have uploaded for you. Simply copy and paste it into notepad, save it as robots.txt and upload it to your server root directory.
http://www.abakus-internet-marketing.de/robotsbeispiel.txt

No comments: