Due to ordering, these config files don't get copied over before the
first time httpd gets restarted. Setting them as optional should allow
initial runs to work better.
Signed-off-by: Kevin Fenzi <kevin@scrye.com>
This will unify all the handlers to use first uppercase letter for
ansible-lint to stop complaining.
I went through all `notify:` occurrences and fixed them by running
```
set TEXT "text_to_replace"; set REPLACEMENT "replacement_text"; git grep
-rlz "$TEXT" . | xargs -0 sed -i "s/$TEXT/$REPLACEMENT/g"
```
Then I went through all the changes and removed the ones that wasn't
expected to be changed.
Fixes https://pagure.io/fedora-infrastructure/issue/12391
Signed-off-by: Michal Konecny <mkonecny@redhat.com>
I went through the last couple of logs afer the first round of 'turn
off the spiders' went out. I looked at the areas which the /robots.txt
disregard and then looked for the bots which ignored it and still
looked up stuff in 'accounts'. This may cut down CPU spikes as these
are looking at dynamic data which can 'blow' things up.
It might be good to add similar tooling to pagure and src since they
seem to be hit a lot in the logs also.
Signed-off-by: Stephen Smoogen <ssmoogen@redhat.com>
This crawler is doing about 2M requests a day to
lists.fedoraproject.org. This is causing db load on db01, causing
services to have issues.
So, block them here for now, but we may want to block them elsewhere
too.
Signed-off-by: Kevin Fenzi <kevin@scrye.com>
alt.fpo/en/cloud is still the first link to pop up on search engine and
the current redirect only works with /cloud.
This commit fix the redirect, and prevent this old page to be indexed.
Turns out back in 2015 (in 623be6ff73) I added a robots.txt on the koji
hubs to disallow crawling, but since it's behind the proxies, no one
ever saw that robots.txt. Instead they got the default one that just had
a 'crawl-delay 1' in it.
So, lets tell robots to go away for real now, as they are causing load
problems on the koji database and I don't think it's particularly useful
for koji to be indexed. It's a lot of dynamic content anyhow.
Signed-off-by: Kevin Fenzi <kevin@scrye.com>
This reverts commit 4430178b29.
It's time to put this back before the cert expires and before we go into
Beta freeze. Hopefully the odd issue with armv7 qemu guests having a
time behind real time is not still happening.
This reverts commit 57f0d4fdb6.
For an anoying reason, armv7 image builds come up with the time as 10
days ago, which makes this cert invalid. So, move back to the old cert
for a week or so and then switch to the new one again. ;(
This permit to move, let's say, blog to a provider
that do support LE natively, but without needing to
transfer the certificate or anything.
Just set "enable_certgetter: false", run the playbook,
do the http01 negociation, and then switch DNS.
The various openshift tools get hit by various crawlers and do not send
a robots.txt. This seems to be due to the balancer code used to send
back to the nodes. This forces the proxy robots.txt to be honored
always.
The actual cache is only set in the proxy HTTP config.
While we could set the cache path in the other servers' configs as well,
that would be a significantly larger change.
Signed-off-by: Patrick Uiterwijk <patrick@puiterwijk.org>
