michael/pyfedi
1
0
Fork 0
mirror of https://codeberg.org/rimu/pyfedi synced 2025-01-23 19:36:56 -08:00
Code Issues Projects Releases Packages Wiki Activity Actions

block various bots using robots.txt

Browse source
This commit is contained in:
rimu 2024-04-09 19:43:56 +12:00
parent 2729539d8c
commit f777f97006
1 changed files with 15 additions and 0 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

15
app/templates/robots.txt
View file

@ -12,3 +12,18 @@ User-Agent: *
Disallow: /d/
Disallow: /static/media/users/
Disallow: /post/*/options
User-agent: GPTBot
Disallow: /
User-agent: AhrefsBot
Disallow: /
User-agent: SemrushBot
Disallow: /
User-agent: DotBot
Disallow: /
User-agent: SeznamBot
Disallow: /