1 2 3 4 5 6 7 8 9 10 11
User-agent: * Disallow: / User-agent: Googlebot Allow: / User-agent: Bingbot Allow: / User-agent: DuckDuckBot Allow: / User-agent: archive.org_bot Allow: /