elk/public/robots.txt

User-agent: *
Allow: /

# Disallow authenticated pages

Disallow /intent
Disallow /settings
Disallow /blocks
Disallow /bookmarks
Disallow /compose
Disallow /conversations
Disallow /domain_blocks
Disallow /favourites
Disallow /home
Disallow /mutes
Disallow /notifications
Disallow /pinned
Disallow /search
Disallow /settings
Disallow /share-target

# Wait 1 second between successive requests.
Crawl-delay: 1
feat: pwa with push notifications (#337) 2022-12-18 00:29:16 +01:00			`User-agent: *`
			`Allow: /`
fix: disallow all indexing on canary site and indexing of auth-only pages in prod (#1175) 2023-01-15 14:34:23 +00:00
			`# Disallow authenticated pages`

			`Disallow /intent`
			`Disallow /settings`
			`Disallow /blocks`
			`Disallow /bookmarks`
			`Disallow /compose`
			`Disallow /conversations`
			`Disallow /domain_blocks`
			`Disallow /favourites`
			`Disallow /home`
			`Disallow /mutes`
			`Disallow /notifications`
			`Disallow /pinned`
			`Disallow /search`
			`Disallow /settings`
			`Disallow /share-target`

			`# Wait 1 second between successive requests.`
			`Crawl-delay: 1`