Quote:
|
Originally Posted by xris
Here's part of a robots.txt file I have
Code:
# Hello little robots
user-agent: *
disallow: /tiles
disallow: /status.htm
disallow: /preorder.htm
disallow: /updates.htm
It tells them not to search those directories / files listed (because I think they get updated to often to warrant repeated searches of them). |
status.htm - updated 27th January 2002 3:15 PM BST
preorder.htm - updated 27th January 2002 3:15 PM GMT
updates.htm - updated 1st August 2003
Sorry. Couldn't resist...
