Stop bots scraping mediawiki special pages/diffs/history

This commit is contained in:
Felix Albrigtsen 2023-06-14 23:24:16 +02:00
parent 57055c9c6f
commit 134b203d92
1 changed files with 18 additions and 0 deletions

18
www/robots.txt Normal file
View File

@ -0,0 +1,18 @@
# This file set up by fredj according to proposed Standard of Robot
# Exclusion at http://web.nexor.co.uk/mak/doc/robots/norobots.html
#
# Currently: allow all robots.
# (An empty 'Disallow' line, looking like this:)
# User-agent: * # Means: All robots.
# Disallow: # Means: Disallow nothing.
User-agent: *
Disallow: /usage
Disallow: /news_archive
Disallow: /cgi
Disallow: /finger
Disallow: /cgi-bin
Disallow: /~pallo/usage/
Disallow: /~pallo/historic/
Disallow: /~meldahl/
Disallow: /w/index.php?