mirror of
https://pagure.io/fedora-infra/ansible.git
synced 2026-05-04 20:30:41 +08:00
7e426dbf37
I went through the last couple of logs afer the first round of 'turn off the spiders' went out. I looked at the areas which the /robots.txt disregard and then looked for the bots which ignored it and still looked up stuff in 'accounts'. This may cut down CPU spikes as these are looking at dynamic data which can 'blow' things up. It might be good to add similar tooling to pagure and src since they seem to be hit a lot in the logs also. Signed-off-by: Stephen Smoogen <ssmoogen@redhat.com>