Step 2 - Disallow Access

Robot-Manager Standard and Professional Editions

This is where you set the disallow rules for the spiders. The spiders selected in Step 1 - Spiders are displayed here so that you may select the disallowed files and folders. Dragging and dripping spiders on top of one another allows you to set rules for groups of spiders thereby shrinking the overall size of the final robots.txt file. Robot-Manager can read the web site directory and file structure from either your local hard drive or over the internet using FTP.