User-agent: * Disallow: /cgi-bin/ Disallow: /wp-content/plugins/ User-agent: Googlebot Allow: /wp-content/plugins/wptouch/themes/ User-agent: Googlebot Allow: /wp-content/plugins/wptouch/resources/ # http://www.webmasterworld.com/robots.txt has a long list of active robots you might want to block. # Some of these (and many others) ignore robots.txt, and are forcibly blocked in .htaccess. # User-agent: sitecheck.internetseer.com User-agent: MarcoPolo User-agent: Nutch User-agent: Zao User-agent: semanticdiscovery User-agent: PubCrawl User-agent: TurnitinBot User-agent: NPbot User-agent: psbot User-agent: baiduspider User-agent: larbin User-agent: ia_archiver User-agent: NationalDirectory User-agent: LNSpiderguy User-agent: Teleport User-agent: MIIxpc User-agent: asterias User-agent: lwp-trivial User-agent: LinkWalker User-agent: cosmos User-agent: MSIECrawler User-agent: pompos User-agent: Generic User-agent: WebSearchBench User-agent: almaden User-agent: k2spider User-agent: curl User-agent: Wget User-agent: AhrefsBot Disallow: /