Software to help you create a robots.txt file
By Sumantra Roy
Step 2: In the left pane of the main Robogen window, click on the small button present to the right of the drop-down combo box at the top. In the menu that appears, select "Popular Search Engines".
Step 3: Now, click on the drop-down combo box and choose the search engine that you are targeting (say, AltaVista). In the right pane, select a page which is NOT meant for AltaVista. Now click on the "<<<" button. You will find that the file which you had selected in the right pane is now present in the left pane. In this way, click on each of the pages which are not meant for AltaVista and press the "<<<" button.
Repeat step 3 for each of the search engines that you are targeting. Just select the search engine that you are targeting from the drop-down combo box, select the pages which are not meant for the search engine one by one from the right pane, and click on the "<<<" button after clicking on each age. Repeat this process for all the search engines that you are targeting.
Step 4: Upload the file to your server using the "Upload Robots.Txt" button. If you want, you can also use the Save menu item from the File menu to save a copy of the robots.txt file in your hard drive.
Article by Sumantra Roy. Sumantra is one of the most respected search engine positioning specialists on the Internet. To have Sumantra's company place your site at the top of the search engines, go to http://www.1stSearchRanking.com/t.cgi?4021 For more advice on how you can take your web site to the top of the search engines, subscribe to his FREE newsletter by going to http://www.1stSearchRanking.com/t.cgi?4021&newsletter.htm
Back to Articles Directory
Copyright © 2002-2003 Web World Index