Robot-Manager
feed icon

Web Wisdom

Our new Podcast and Weblog with topics on Web site design, promotion, and marketing. Click Web Wisdom to visit the Weblog.

podcast mp3 icon

Click a screen shot below to see a full size image and a brief description.

Step 4 - Spiders

Determine Search Engine Spider Visits

Spider Visit Report

Disallow Access for Search Engine Spiders

Indexed Pages Report

Log File Analysis

Get all the great features of the Standard Edition, plus get complete control over visiting search engine spiders and robots through log file analysis. Know what search engine spiders are accessing your site, what pages they are indexing and when. All of this is available through standard web server log files.

Loaded with the information from your log files, you can make informed decisions about what pages to submit. If a web page has changed and was indexed by a spider, you can go to that search engine and determine your new page rank and if your changes were successful.

Supported Log Files

Robot-Manager Professional Edition supports a wide range of log files from different types of web servers. Log file analysis can be a complex task, that's why the Professional Edition supports compressed log files as well—eliminating the tedious task of decompressing log files before analyzing them.

Key Features of the Professional Edition

  • Uses log file analysis to detect spider visits and pages indexed
  • Informative reports showing spider visits and pages indexed by month, plus a logical tree view to show relationships
  • Whois dialog box for looking up the owners of host names and ip addresses
  • Builtin editor for adding spiders that don't normally request the robots.txt file
  • Support for Windows, Linux (Unix), and Mac ftp servers
  • Indicates visually the pages requested by spiders in error (i.e., 404 Not Found)
  • Can access log files locally or download them from web servers
  • Telnet to the web server and extract spider visits directly from the server
  • Can add any new spiders found on your web site to Robot-Manager
  • Exports spider and page visits to both CSV (Comma Separated Values) and XML formats
  • Exports raw log entries of just spider visits
  • Exports all spider IP addresses so they can be used by cloaking software
  • Commandline options for running multiple projects unattended
  • Provides log file analysis for muliple log files at the same time
  • Support for compressed log files (both Windows and Linux)
  • Auto detects your log file format, and in the rare case where it can't, you can specifiy the format manually