How Deep into Your Site Are the Spiders Crawling?
By Robin Nobles
What you're doing in steps #1-3 is creating your robots.txt file. You'll tell the program which areas or pages that you want to keep the spiders from crawling, and the software will create the robots.txt file for you and upload it to the root directory of your Web site.
Keep in mind that you don't have to create a robots.txt file if you don't want or need to. You can simply use the spider monitoring portion by itself.
Now Let's Get to the "Fun" Stuff
As I stated previously, the most exciting part of the program to me is the way it monitors spider traffic.
The first time you download your log files to analyze your traffic, it won't be quite as interesting as it gets each additional time you use the program.
Every time you use the program, you'll open the initial file you created, hit "refresh" to download the latest log files, and let the program update your spider visits.
If we were to click on one of the spiders, such as Googlebot, we'd see how many pages from this site that Google has spidered.
Google is a deep crawling engine, and it really can find most of your Web pages if you provide text links to them.
If we were to click on any of the actual Web pages, we'd see a screen that tells us how often the spider has visited the page.
Can you see the value in having this type of information?
* If an engine hasn't found one of your important pages, maybe you need to add more links to that page from pages that are getting spidered more frequently.
* When you create new pages, you'll be able to see at a glance when the spiders find and crawl those pages.
* If you use pay inclusion, you'll be able to monitor whether the spiders are visiting like they're supposed to.
* You'll know exactly which pages of your Web site are being spidered by which search engines, and when.
If you're interested in tracking spider visits to your site, Robot Manager Pro is a "must have" software program for you. It's fast and easy to use, and it provides key information about how the spiders are crawling your Web site.
Visit the Robot Manager Pro Web site at http://www.websitemanagementtools.com/robot.html to learn more or to download a free trial version. They also offer a regular version of Robot Manager, but it doesn't monitor spider traffic.
Robin Nobles with Search Engine Workshops teaches SEO strategies the "stress free" way through hands-on, search engine marketing workshops (http://www.searchengineworkshops.com) in locations across the globe and online search engine marketing courses (http://www.onlinewebtraining.com). Visit the World Resource Center, a new networking community for search engine marketers. (http://www.sew-wrc.com)
Back to Articles Directory
Copyright © 2002-2003 Web World Index