Crawlers, Robots, and Spiders
Directory: Computers » Internet » Searching the Net » Crawlers, Robots, and Spiders

• Search Engine IP Addresses
  Lists IP addresses of search engine spiders. Can be searched by IP address. More Info....
  http://www.iplists.com

• Psychedelix.com
  Andreas Staeding's large list of search engine spiders, similar Web robots, and Web browsers: their web-log identification and links to their originators. More Info....
  http://www.psychedelix.com/agents.html

• Search engine robots
  This page lists the search engine robots known to JafSoft Limited More Info....
  http://www.jafsoft.com/searchengines/webbots.html

• robotstxt.org
  Information about robots.txt, the web robots exclusion standard, and writing well-behaved web robots. More Info....
  http://www.robotstxt.org