The blame endpoint gets crawled a lot by scrapers and it's very
intensive for pagure. It has to run the blame in the git repo, which
can take a while and requests pile up.
While right now load is fine with it, it's might be causing
sporadic 503's that we are seeing cloing things. If there's a bunch
of blame requests cloging up the queue.
Users can git clone and run a blame locally if they need to.
Signed-off-by: Kevin Fenzi <kevin@scrye.com>
Scrapers are crawling these endpoints and pkgs01 takes a while to call
git on the backend and return data to them. This causes latency to
increase a bunch because it's got all those blame and history requests
it's processing so it can't process more important things.
So, lets just block these for now. Any users who need them can easily
git clone locally and run history/blame just fine.
Signed-off-by: Kevin Fenzi <kevin@scrye.com>
These fonts are kind of hit a lot, no need to send to the pkgs01 backend
for them, we can just cache them in varnish for clients.
Signed-off-by: Kevin Fenzi <kevin@scrye.com>
This will increase some values to allow for handling
scrapers better. In any case it doesn't seem to hurt.
Signed-off-by: Kevin Fenzi <kevin@scrye.com>
The python productmd library hits kojipkgs to find recent
composes/compose information. It's not easy for it to adjust on the
client end, but it does send a user-agent header like "productmd/1.49"
So, lets just allowlist it for now.
Signed-off-by: Kevin Fenzi <kevin@scrye.com>
Bodhi messaging queue on staging has around 1 000 000 messages in it and
after some investigation it seems that the processing is blocked on
database. Let's bump the amount of CPUs to 8.
Signed-off-by: Michal Konecny <mkonecny@redhat.com>
/ and /var/log are the same filesystem on log01, so it makes little
sense to monitor both. Just monitor /var/log and increase it's limits.
We are going to archive things, but likely in january.
Signed-off-by: Kevin Fenzi <kevin@scrye.com>
Checking for the presence of dead.package by trying to download the
dead.package file and checking for an HTTP 404 or 200 is an easy way to
check if a package is retied in distgit.
This practice is used in the wild in tools like fedora-misc-package-utilities.
I am the main technical point of contact for folks working within
the Fedora Ready program, so it makes sense to have me on this alias.
Signed-off-by: Neal Gompa <ngompa@fedoraproject.org>
This was an attempt to block the scrapers, but sadly it breaks humans.
We cannot challenge everything under forks because people need to push
pr's and those get blocked.
So, go back to just weighting those higher...
Signed-off-by: Kevin Fenzi <kevin@scrye.com>
