This tweaks the Forgejo redirect-to-pagure-for-attachments stuff
to work for prod as well as staging, since we proved it out in
staging and we do want it to actually work for prod migrations.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
This is a bit ugly, but can't think of a better way right now.
Since @kparal is using this migration of testdays-web from prod
pagure to staging forgejo as a test case for prod migrations in
general, let's make sure it works properly by proxying attachment
requests to prod pague instead of staging.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
This turned out to be a bit complicated, see
https://codeberg.org/forgejo/forgejo/issues/6360#issuecomment-9010932 . Pagure
images in comments use root-relative Markdown links, like this:
but Forgejo renders those relative to *the repo*, not the server root, so
they get rendered as something like:
<img src="/forgejoorg/forgejorepo/group/repo/issue/raw/files/image.png">
However, it does *not* do this for *non-image* root-relative links, so those
aren't 'broken'. This means we need to handle *both* cases in the proxying,
and we also need to keep in mind that Pagure allows repos without a group.
So we can wind up with one, two, three or four folders before /issue.
I did some testing and I *think* this should cover all cases. I've tested
this does fix images, I haven't tested on a non-image attachment yet (need
to find one).
Signed-off-by: Adam Williamson <awilliam@redhat.com>
The attachment reverse proxy was failing for several reasons, one
of which is that SSL options weren't set up correctly so the SSL
connection to pagure.io failed. This adds a Proxy section for
stg.pagure.io with the appropriate settings to make it work.
There are still several other issues, but this at least fixes the
SSL problem.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
redirect attachment file requests from Fedora Forge to Pagure for migrated
projects, allowing attachments to remain on this existing instance without
needing to be migrated.
Signed-off-by: Akashdeep Dhar <akashdeep.dhar@gmail.com>
Scrapers are crawling these endpoints and pkgs01 takes a while to call
git on the backend and return data to them. This causes latency to
increase a bunch because it's got all those blame and history requests
it's processing so it can't process more important things.
So, lets just block these for now. Any users who need them can easily
git clone locally and run history/blame just fine.
Signed-off-by: Kevin Fenzi <kevin@scrye.com>
I am not sure these are even ai scrapers. If they are, they are broken
and unfit for scraping. They just hit these forks (and nothing else)
over and over via a Distributed pile of ips. They pass anubis
challenges, so probibly residential users who they don't care about.
Anyhow, on high load on pkgs01, see if more blocks need to be added
here.
Signed-off-by: Kevin Fenzi <kevin@scrye.com>
there's about... 7million hits a day from sites passing a referrer
of forks/kernel or forks/firefox where they are fetching static content
over and over and over. This may be because before they were blocked
from the forks themselves they were also downloading the js and static
content, and now they are just too dumb to see the 403 and still
want to fetch the old static content. Fortunately, they send a
referrer we can match on.
So, this should cut load another chunk.
Signed-off-by: Kevin Fenzi <kevin@scrye.com>
Some scraper(s) were very very agressively crawling kernel fork repos
and causing all kinds of problems for koji and src.
Signed-off-by: Kevin Fenzi <kevin@scrye.com>
This apache config is needed in order to allow auth to upload to the
candidate registry. Without it, skopeo just gives a perm denied.
Anytime the datacenter networks change this will also need updated.
Signed-off-by: Kevin Fenzi <kevin@scrye.com>
So, we renamed the cluster with the ocp-rdu3 name, so drop all this
special handling. All the proxies should be able to reach it by that
name and via the vpn endpoints it has.
Signed-off-by: Kevin Fenzi <kevin@scrye.com>
We can't use easily the existing hostname/site, as that goes to the
current iad2 cluster, so setup a -rdu3 version for now.
After we switch we can drop this and repoint the main one to the new
cluster.
Hopefully this all works and does the right thing.
Signed-off-by: Kevin Fenzi <kevin@scrye.com>
These have to be in "s in order to do a string comparison, since
they were not, they were never matching anything. ;(
Fix them all up, and also block a few more repos on pagure that are
getting heavily crawled.
Signed-off-by: Kevin Fenzi <kevin@scrye.com>
Bots are htiting these at a massive level.
Since these cause koji to do db queries it basically swamps it and it
stops processing at all.
Hopefully with enough of these 403's the bots will go away.
Signed-off-by: Kevin Fenzi <kevin@scrye.com>
Bots are hitting the wiki pretty hard and we don't particularly
care about indexing it anymore, as most real docs should have moved
to docs.fedoraproject.org. Also, many of these bots ignore robots.txt
or do other things we don't want.
Signed-off-by: Kevin Fenzi <kevin@scrye.com>
This will unify all the handlers to use first uppercase letter for
ansible-lint to stop complaining.
I went through all `notify:` occurrences and fixed them by running
```
set TEXT "text_to_replace"; set REPLACEMENT "replacement_text"; git grep
-rlz "$TEXT" . | xargs -0 sed -i "s/$TEXT/$REPLACEMENT/g"
```
Then I went through all the changes and removed the ones that wasn't
expected to be changed.
Fixes https://pagure.io/fedora-infrastructure/issue/12391
Signed-off-by: Michal Konecny <mkonecny@redhat.com>
