This tweaks the Forgejo redirect-to-pagure-for-attachments stuff
to work for prod as well as staging, since we proved it out in
staging and we do want it to actually work for prod migrations.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
This is a bit ugly, but can't think of a better way right now.
Since @kparal is using this migration of testdays-web from prod
pagure to staging forgejo as a test case for prod migrations in
general, let's make sure it works properly by proxying attachment
requests to prod pague instead of staging.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
This turned out to be a bit complicated, see
https://codeberg.org/forgejo/forgejo/issues/6360#issuecomment-9010932 . Pagure
images in comments use root-relative Markdown links, like this:
but Forgejo renders those relative to *the repo*, not the server root, so
they get rendered as something like:
<img src="/forgejoorg/forgejorepo/group/repo/issue/raw/files/image.png">
However, it does *not* do this for *non-image* root-relative links, so those
aren't 'broken'. This means we need to handle *both* cases in the proxying,
and we also need to keep in mind that Pagure allows repos without a group.
So we can wind up with one, two, three or four folders before /issue.
I did some testing and I *think* this should cover all cases. I've tested
this does fix images, I haven't tested on a non-image attachment yet (need
to find one).
Signed-off-by: Adam Williamson <awilliam@redhat.com>
The attachment reverse proxy was failing for several reasons, one
of which is that SSL options weren't set up correctly so the SSL
connection to pagure.io failed. This adds a Proxy section for
stg.pagure.io with the appropriate settings to make it work.
There are still several other issues, but this at least fixes the
SSL problem.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
redirect attachment file requests from Fedora Forge to Pagure for migrated
projects, allowing attachments to remain on this existing instance without
needing to be migrated.
Signed-off-by: Akashdeep Dhar <akashdeep.dhar@gmail.com>
Scrapers are crawling these endpoints and pkgs01 takes a while to call
git on the backend and return data to them. This causes latency to
increase a bunch because it's got all those blame and history requests
it's processing so it can't process more important things.
So, lets just block these for now. Any users who need them can easily
git clone locally and run history/blame just fine.
Signed-off-by: Kevin Fenzi <kevin@scrye.com>
I am not sure these are even ai scrapers. If they are, they are broken
and unfit for scraping. They just hit these forks (and nothing else)
over and over via a Distributed pile of ips. They pass anubis
challenges, so probibly residential users who they don't care about.
Anyhow, on high load on pkgs01, see if more blocks need to be added
here.
Signed-off-by: Kevin Fenzi <kevin@scrye.com>
there's about... 7million hits a day from sites passing a referrer
of forks/kernel or forks/firefox where they are fetching static content
over and over and over. This may be because before they were blocked
from the forks themselves they were also downloading the js and static
content, and now they are just too dumb to see the 403 and still
want to fetch the old static content. Fortunately, they send a
referrer we can match on.
So, this should cut load another chunk.
Signed-off-by: Kevin Fenzi <kevin@scrye.com>
Some scraper(s) were very very agressively crawling kernel fork repos
and causing all kinds of problems for koji and src.
Signed-off-by: Kevin Fenzi <kevin@scrye.com>
We need to tell apache to use the X-Real-Ip we set at the first proxy
level for remoteipheader. This will get it to log the correct remote ip
in logs.
We can also then re-enable the fix for lists/duplicate headers with no
problems.
Signed-off-by: Kevin Fenzi <kevin@scrye.com>
All the sites except list are fine with multiple hosts in the
x-forwarder-for. Lists breaks. This gets list working, but then
all the log entries are localhost instead of the real ip.
So, lets disable this for now and revisit a solution for lists
before enabling it.
Signed-off-by: Kevin Fenzi <kevin@scrye.com>
Do not add proxy headers here as they were already added when we
first got the request and sent it to anubis. This fixes some applications
(like mailman/django) that don't deal well with duplicate x-forwarded-for
entries and keeps the non anubis behavior.
Signed-off-by: Kevin Fenzi <kevin@scrye.com>
We are in freeze and don't want to change production, so
drop this blank line that was added outside a staging
conditional.
Signed-off-by: Kevin Fenzi <kevin@scrye.com>
This just reuses the existing copr anubis role and adds it into our
proxy setup (in staging only).
A new variable 'anubis' is set globally to false, but can be enabled on
a per site / app basis in the httpd/website role call.
I have set it for koji.stg.
The proxy playbook now should install anubis on staging proxies and then
only use it for the one site thats enabled in configuration.
Before moving to prod:
- testing in staging
- testing with some more staging apps
- perhaps moving the copr anubis role to a base role?
- adding some more bot policy
Signed-off-by: Kevin Fenzi <kevin@scrye.com>
This apache config is needed in order to allow auth to upload to the
candidate registry. Without it, skopeo just gives a perm denied.
Anytime the datacenter networks change this will also need updated.
Signed-off-by: Kevin Fenzi <kevin@scrye.com>
So, we renamed the cluster with the ocp-rdu3 name, so drop all this
special handling. All the proxies should be able to reach it by that
name and via the vpn endpoints it has.
Signed-off-by: Kevin Fenzi <kevin@scrye.com>
We can't use easily the existing hostname/site, as that goes to the
current iad2 cluster, so setup a -rdu3 version for now.
After we switch we can drop this and repoint the main one to the new
cluster.
Hopefully this all works and does the right thing.
Signed-off-by: Kevin Fenzi <kevin@scrye.com>
Due to ordering, these config files don't get copied over before the
first time httpd gets restarted. Setting them as optional should allow
initial runs to work better.
Signed-off-by: Kevin Fenzi <kevin@scrye.com>
These have to be in "s in order to do a string comparison, since
they were not, they were never matching anything. ;(
Fix them all up, and also block a few more repos on pagure that are
getting heavily crawled.
Signed-off-by: Kevin Fenzi <kevin@scrye.com>
