I am not sure these are even ai scrapers. If they are, they are broken
and unfit for scraping. They just hit these forks (and nothing else)
over and over via a Distributed pile of ips. They pass anubis
challenges, so probibly residential users who they don't care about.
Anyhow, on high load on pkgs01, see if more blocks need to be added
here.
Signed-off-by: Kevin Fenzi <kevin@scrye.com>
there's about... 7million hits a day from sites passing a referrer
of forks/kernel or forks/firefox where they are fetching static content
over and over and over. This may be because before they were blocked
from the forks themselves they were also downloading the js and static
content, and now they are just too dumb to see the 403 and still
want to fetch the old static content. Fortunately, they send a
referrer we can match on.
So, this should cut load another chunk.
Signed-off-by: Kevin Fenzi <kevin@scrye.com>
Some scraper(s) were very very agressively crawling kernel fork repos
and causing all kinds of problems for koji and src.
Signed-off-by: Kevin Fenzi <kevin@scrye.com>
This apache config is needed in order to allow auth to upload to the
candidate registry. Without it, skopeo just gives a perm denied.
Anytime the datacenter networks change this will also need updated.
Signed-off-by: Kevin Fenzi <kevin@scrye.com>
So, we renamed the cluster with the ocp-rdu3 name, so drop all this
special handling. All the proxies should be able to reach it by that
name and via the vpn endpoints it has.
Signed-off-by: Kevin Fenzi <kevin@scrye.com>
We can't use easily the existing hostname/site, as that goes to the
current iad2 cluster, so setup a -rdu3 version for now.
After we switch we can drop this and repoint the main one to the new
cluster.
Hopefully this all works and does the right thing.
Signed-off-by: Kevin Fenzi <kevin@scrye.com>
These have to be in "s in order to do a string comparison, since
they were not, they were never matching anything. ;(
Fix them all up, and also block a few more repos on pagure that are
getting heavily crawled.
Signed-off-by: Kevin Fenzi <kevin@scrye.com>
Bots are htiting these at a massive level.
Since these cause koji to do db queries it basically swamps it and it
stops processing at all.
Hopefully with enough of these 403's the bots will go away.
Signed-off-by: Kevin Fenzi <kevin@scrye.com>
Bots are hitting the wiki pretty hard and we don't particularly
care about indexing it anymore, as most real docs should have moved
to docs.fedoraproject.org. Also, many of these bots ignore robots.txt
or do other things we don't want.
Signed-off-by: Kevin Fenzi <kevin@scrye.com>
This will unify all the handlers to use first uppercase letter for
ansible-lint to stop complaining.
I went through all `notify:` occurrences and fixed them by running
```
set TEXT "text_to_replace"; set REPLACEMENT "replacement_text"; git grep
-rlz "$TEXT" . | xargs -0 sed -i "s/$TEXT/$REPLACEMENT/g"
```
Then I went through all the changes and removed the ones that wasn't
expected to be changed.
Fixes https://pagure.io/fedora-infrastructure/issue/12391
Signed-off-by: Michal Konecny <mkonecny@redhat.com>
Users don't need to use this and so restrict it to admins by ip for now.
Down the road we should be able to do this much better once we can set a
policy for access here.
Signed-off-by: Kevin Fenzi <kevin@scrye.com>
The mirrormanager application moved over to
mirrormanager.fedoraproject.org from
admin.fedoraproject.org/mirrormanager. So we need to change this
redirect to point things to the new place.
Signed-off-by: Kevin Fenzi <kevin@scrye.com>
This removes osbs and allmost all it's associated playbooks and files.
It served long and well, but we no longer need it.
flatpaks are building with a koji-flatpak plugin.
base/minimal/toolbox containers are building with kiwi.
We aren't building any other containers right now, and we did they could
be added to kiwi.
This is the end of an era... I look with nostolga on
ansible-ansible-openshift-ansible (a role to setup ansible on a control
host and run it from our ansible).
Good bye osbs!
Signed-off-by: Kevin Fenzi <kevin@scrye.com>
New ipa checks the referrer to avoid CSRF issues.
We need to have the proxy edit requests for the right internal hostname
for it to be able to work.
Signed-off-by: Kevin Fenzi <kevin@scrye.com>
