Index of /ports_www/p5-WWW-RobotRules

Icon  Name                    Last modified      Size  
[DIR] Parent Directory                             -   
[   ] Makefile                19-Apr-2011 21:50  622   
[   ] distinfo                14-Apr-2011 17:18  144   
[TXT] pkg-descr               14-Apr-2011 17:18  303   
[   ] pkg-plist               14-Apr-2011 17:18  306   

This module parses /robots.txt files which are used to forbid conforming
robots from accessing parts of a web site. The parsed files are kept in
a WWW::RobotRules object, and this object provides methods to check if
access to a given URL is prohibited.

WWW: http://search.cpan.org/dist/WWW-RobotRules/