elk/public/robots.txt

User-agent: *
Allow: /

# Disallow authenticated pages

Disallow: /intent
Disallow: /settings
Disallow: /blocks
Disallow: /bookmarks
Disallow: /compose
Disallow: /conversations
Disallow: /domain_blocks
Disallow: /favourites
Disallow: /home
Disallow: /mutes
Disallow: /notifications
Disallow: /pinned
Disallow: /search
Disallow: /settings
Disallow: /share-target

# Wait 1 second between successive requests.
Crawl-delay: 1
feat: pwa with push notifications (#337) 2022-12-18 00:29:16 +01:00			`User-agent: *`
			`Allow: /`
fix: disallow all indexing on canary site and indexing of auth-only pages in prod (#1175) 2023-01-15 14:34:23 +00:00
			`# Disallow authenticated pages`

fix: syntax for robots.txt disallow rules (#1925) 2023-03-26 17:11:22 +02:00			`Disallow: /intent`
			`Disallow: /settings`
			`Disallow: /blocks`
			`Disallow: /bookmarks`
			`Disallow: /compose`
			`Disallow: /conversations`
			`Disallow: /domain_blocks`
			`Disallow: /favourites`
			`Disallow: /home`
			`Disallow: /mutes`
			`Disallow: /notifications`
			`Disallow: /pinned`
			`Disallow: /search`
			`Disallow: /settings`
			`Disallow: /share-target`
fix: disallow all indexing on canary site and indexing of auth-only pages in prod (#1175) 2023-01-15 14:34:23 +00:00
			`# Wait 1 second between successive requests.`
			`Crawl-delay: 1`