mirror of
https://pagure.io/fedora-infra/ansible.git
synced 2026-04-30 21:41:53 +08:00
aeea2c1d06
Turns out back in 2015 (in 623be6ff73) I added a robots.txt on the koji
hubs to disallow crawling, but since it's behind the proxies, no one
ever saw that robots.txt. Instead they got the default one that just had
a 'crawl-delay 1' in it.
So, lets tell robots to go away for real now, as they are causing load
problems on the koji database and I don't think it's particularly useful
for koji to be indexed. It's a lot of dynamic content anyhow.
Signed-off-by: Kevin Fenzi <kevin@scrye.com>