pyfedi/app/templates/robots.txt

User-Agent: *
  Disallow: /community/add_local
  Disallow: /community/add_remote
  Disallow: /auth/register
  Disallow: /auth/reset_password_request
  Disallow: /auth/login
  Disallow: /admin
  Disallow: /inbox
  Disallow: /search/
  Disallow: /modlog
  Disallow: /u/
  Disallow: /d/
  Disallow: /static/media/users/
  Disallow: /post/*/options
delete account, with federation 2023-12-29 17:32:35 +13:00			`User-Agent: *`
			`Disallow: /community/add_local`
			`Disallow: /community/add_remote`
			`Disallow: /auth/register`
			`Disallow: /auth/reset_password_request`
			`Disallow: /auth/login`
			`Disallow: /admin`
			`Disallow: /inbox`
			`Disallow: /search/`
			`Disallow: /modlog`
update robots.txt 2024-01-23 20:21:41 +13:00			`Disallow: /u/`
			`Disallow: /d/`
update robots.txt 2024-01-24 16:42:15 +13:00			`Disallow: /static/media/users/`
			`Disallow: /post/*/options`