# All robots will spider the domain User-agent: * Disallow: # Disallow directory knife User-agent: * Disallow: knife # Disallow directory nurpel User-agent: * Disallow: nurpel # Disallow directory test User-agent: * Disallow: test # Disallow directory log User-agent: * Disallow: log