# Base robots.txt file. # Use this to control access to the site by web spiders. User-agent: * Disallow: /companies/ Disallow: /deals/ Disallow: /individuals/ Disallow: /intelligence/ Disallow: /siteincludes/ Disallow: /sitenav/ Disallow: /templates/