Stop bots scraping mediawiki special pages/diffs/history #45

Merged
felixalbrigtsen merged 1 commits from robots-mediawiki into master 2023-06-14 23:26:28 +02:00
1 changed files with 18 additions and 0 deletions
Showing only changes of commit 134b203d92 - Show all commits

18
www/robots.txt Normal file
View File

@ -0,0 +1,18 @@
# This file set up by fredj according to proposed Standard of Robot
# Exclusion at http://web.nexor.co.uk/mak/doc/robots/norobots.html
#
# Currently: allow all robots.
# (An empty 'Disallow' line, looking like this:)
# User-agent: * # Means: All robots.
# Disallow: # Means: Disallow nothing.
User-agent: *
Disallow: /usage
Disallow: /news_archive
Disallow: /cgi
Disallow: /finger
Disallow: /cgi-bin
Disallow: /~pallo/usage/
Disallow: /~pallo/historic/
Disallow: /~meldahl/
Disallow: /w/index.php?