# this file is to guide automated search engine robots to the areas of the website we want them # to index and find key words # see http://www.searchengineworld.com/robots/robots_tutorial.htm for more info # --------------------------------------------------------------------------------------- # "*' means to allow all robots access to the website where this robot.txt file resides # ie http://www.mydomain.com/robots.txt User-agent: * # --------------------------------------------------------------------------------------- # default directory access is to all directories in the web site # to limit access, use disallow statements: Disallow: /cgi-bin/ Disallow: /logs/ Disallow: /report/