# Inspiration from http://inchoo.net/ecommerce/ultimate-magento-robots-txt-file-examples/ # @author James Spittal # Attempt to block Baiduspider User-agent: Baiduspider Disallow: / # Everything else User-agent: * # Attempt to enforce some delays to minimize bandwidth usage/server load Crawl-delay: 10 # Robots seem to love getting below incessantly Disallow: /wishlist/index/add/product/ # No need for spiders to cache this #Disallow: /downloader/ # Alkaline World's Sitemap Sitemap: http://alkalineworld.com.au/pub/sitemap.xml