User-agent: * Disallow: / User-agent: Screaming Frog SEO Spider Allow: / User-agent: Tag Inspector Spider Allow: / User-agent: * Disallow: /jss-test05/ User-agent: * Disallow: /jss-test06/ User-agent: * Disallow: /jss-test07/ User-agent: * Disallow: /jss-preprod05/ User-agent: * Disallow: /jss-preprod06/ User-agent: * Disallow: /jss-preprod07/ User-agent: * Disallow: /jss-preprod08/ User-agent: * Disallow: /jss-prod05/ User-agent: * Disallow: /jss-prod06/ User-agent: * Disallow: /jss.sitecore/