Commit Graph

4511 Commits

Author SHA1 Message Date
Jiri Kyjovsky
8dd7e55028 copr-hv: enable migrated hvs on copr-be-dev 2026-02-16 15:34:54 +01:00
Miroslav Suchý
23ff808fe3 copr: add x86_64 reserved powerful builders 2026-02-16 11:26:43 +01:00
Pavel Raiskup
d075c470de copr-hv: enable p09 01 2026-02-16 09:59:37 +01:00
Kevin Fenzi
c4bdfcc897 proxies: block a ip that was hitting release-monitoring.org a lot
This ip had hit release-monitoring.org like 5,000,000 times in the
course of a few hours and swamped it's web pod.

Lets block it for now and see if anyone complains.
If this is you: please add some rate limiting.

Signed-off-by: Kevin Fenzi <kevin@scrye.com>
2026-02-13 08:48:01 -08:00
Kevin Fenzi
991273d7f1 copr_hypervisors: enable nbde on all of them
The x86 ones are now in rdu3 and reinstalled with rhel10.
All the power9 ones are in rdu3 and reinstalled.

So, we should just enable nbde on all of them.

Signed-off-by: Kevin Fenzi <kevin@scrye.com>
2026-02-09 11:30:46 -08:00
Kevin Fenzi
ab01301f5c inventory: update to reflect machines that moved from f42 to f43
Signed-off-by: Kevin Fenzi <kevin@scrye.com>
2026-01-30 11:36:47 -08:00
Kevin Fenzi
5091fd4373 ocp-rdu3: retire this host/proxy/cert now that we are moved
There's no need to keep ocp-rdu3 around anymore, we only used
it when we were moving datacenters last year.

Signed-off-by: Kevin Fenzi <kevin@scrye.com>
2026-01-30 10:35:10 -08:00
James Antill
0dd9a4f2c4 proxies: Renewed stg.fp.o wildcard cert
Signed-off-by: James Antill <james@fedoraproject.org>
2026-01-29 15:30:33 -05:00
Adam Williamson
f07a93576f openqa: drop a stray - from a variable definition
Signed-off-by: Adam Williamson <awilliam@redhat.com>
2026-01-26 15:42:09 -08:00
Adam Williamson
ae2cd3530b roles/openqa/server: drop OpenID auth support
We've been using OAuth2 for prod and stg for some time now, so
let's clean this up.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2026-01-26 15:40:50 -08:00
Greg Sutcliffe
40cc20185a Zabbix: fix more hosts/groups for apache-status
Signed-off-by: Greg Sutcliffe <fedora@emeraldreverie.org>
2026-01-26 14:18:01 +00:00
Greg Sutcliffe
a72e5b2fbf Zabbix: fix different handling of /apache-status on proxy hosts
Seems like the proxies don't want to handle port 80 nicely, I get
errors in Zabbix for them using localhost:80/apache-status (which
works elsewhere, like sundries). However using https/443 seems to
work, so we'll do that instead.

Signed-off-by: Greg Sutcliffe <fedora@emeraldreverie.org>
2026-01-26 11:39:48 +00:00
Kevin Fenzi
27a81038b9 proxies: block a bunch of networks from an abusive asn
This asn is hammering koji links.

Signed-off-by: Kevin Fenzi <kevin@scrye.com>
2026-01-25 10:32:48 -08:00
Kevin Fenzi
896ab6360b download: block subnet thats using rclone against master mirrors
Signed-off-by: Kevin Fenzi <kevin@scrye.com>
2026-01-22 10:35:44 -08:00
Kevin Fenzi
a754144f19 Update infra pagure.io links to forge.fp.o (WIP)
This should update all the references we have to
https://pagure.io/fedora-infrastructure to the
new https://forge.fedoraproject.org/infra/tickets/ area.

Do not merge this before the migration on tuesday.

Signed-off-by: Kevin Fenzi <kevin@scrye.com>
2026-01-20 14:39:40 -08:00
Jakub Kadlcik
7d56f15586 copr-be: upgrade production builders to F43
Fix https://github.com/fedora-copr/copr/issues/4001

I updated the `copr_builder_images.hypervisor.x86_64` even though the image is
not uploaded there yet because the HV is currently down.

Somebody needs to run this when it gets back online:

    STAMP=$(date -I) \
    ARCHES=x86_64 \
    TARGETS=libvirt \
    copr-upload-builder-images /var/lib/copr/public_html/images/2026-01-14/
2026-01-20 15:50:20 +01:00
Jakub Kadlcik
82245389e8 copr-be-dev: upgrade builder HV ppc64le images to F43 2026-01-20 11:32:44 +01:00
Jakub Kadlcik
f68255f1ae copr-be-dev: upgrade builder images to F43
Except for HV because of https://github.com/fedora-copr/copr/issues/4040
2026-01-15 14:50:12 +01:00
Greg Sutcliffe
93ed0457e0 Nagios: remove first batch of services
This removes the known-good things we've had in Zabbix for a while -
RAID, disk space, processes, and mail queue. It also removes swap which
we've decided we don't need.

Also includes some FS overrides on the Zabbix side so the relevant
NFS mounts get monitored on the OCI, and pkgs hosts, as per Nagios had.

Signed-off-by: Greg Sutcliffe <fedora@emeraldreverie.org>
2026-01-14 10:25:15 +00:00
Jiri Kyjovsky
88102722cb copr-anubis: allow copr-related user agents to pass without challenge 2026-01-12 09:00:32 +00:00
Jiri Kyjovsky
3595436104 copr-anubis: loosen policies (go through defaults), allow essentials 2026-01-08 16:40:34 +01:00
Greg Sutcliffe
5765dd09b5 Zabbix: report warnings to #noc:fpo Matrix room
Signed-off-by: Greg Sutcliffe <fedora@emeraldreverie.org>
2026-01-08 13:56:38 +00:00
Jiri Kyjovsky
d79d6ca487 copr/anubis: fix configuration on frontend to protect only /coprs (same for distgit)
Anubis was accidentally enabled for all traffic (/) instead of just
the /coprs/ web UI. This caused unnecessary bot challenges for API
clients, dnf/yum, and other automated tools.

Use Anubis BASE_PREFIX to cleanly protect only specific endpoints:
- Frontend: /coprs/ (web UI)
- Dist-git: /{{ cgit_uri }}/ (package browser)

https://anubis.techaro.lol/docs/admin/installation#using-base-prefix
2026-01-07 13:58:11 +01:00
Miroslav Suchý
1becdd9e81 bump up number of reserved instances 2026-01-06 12:55:31 +01:00
Adam Williamson
37e2d34eb6 proxies: block IP attempting SQL injections on openQA (prod)
My staging effort seems to have worked, so let's do prod too.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2025-12-30 08:58:11 -08:00
Adam Williamson
3e8c446877 proxies: block IP attempting SQL injections on openQA
I hope I got this right. This IP is trying Little Bobby Tables
attacks on openQA and it's making the servers crash.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2025-12-30 08:46:28 -08:00
Greg Sutcliffe
9fae8c7d2d Zabbix: First try of the CMDB-in-Zabbix-Inventory idea (#12817)
Signed-off-by: Greg Sutcliffe <fedora@emeraldreverie.org>
2025-12-22 15:09:07 +00:00
Pavel Raiskup
cd95aa0095 copr-be: try to use x86 copr04 2025-12-20 15:54:12 +01:00
David Kirwan
1a18ff41fd communishift: mark project communishift-fedora-coreos-ai-helpers do not
clean up.

Signed-off-by: David Kirwan <davidkirwanirl@gmail.com>
2025-12-18 16:09:42 +00:00
Kevin Fenzi
c4f5d8ea2e proxies / proxies_stg: fix comment about max workers and set to 3200 both places
Signed-off-by: Kevin Fenzi <kevin@scrye.com>
2025-12-14 09:45:38 -08:00
Kevin Fenzi
e6e014780d proxies: sync up prod proxies mem/cpus and add more in stg
We manually added cpus and memory in the past to 01/10/101/110, so
update ansible so it matches whats deployed now.

On staging, 2 cpus isn't going to cut it anymore, so just bump them up
to be the same as prod.

Signed-off-by: Kevin Fenzi <kevin@scrye.com>
2025-12-13 12:34:17 -08:00
Kevin Fenzi
48d97a929d proxies: increase max workers to 3200
There's no log messages about it, but we have been seeing some odd
connection reset messages and collectd shows we are near the 2500 limit
we had.

So, bump this to 3200 (based on 8 cpus * 300 ).
If we need to bump this more, we probibly need to add cpus.

Signed-off-by: Kevin Fenzi <kevin@scrye.com>
2025-12-11 10:21:27 -08:00
Pavel Raiskup
e0f9b863d1 copr-be: boost the x86 power a bit more in AWS
(we still don't have x86 hypervisors re-reacked)
2025-12-11 08:37:11 +01:00
Pavel Raiskup
50223282e7 copr-fe: deploy anubis
Relates: https://pagure.io/fedora-infrastructure/issue/12971
Fixes: https://github.com/fedora-copr/copr/issues/4064

Co-authored-by: Jiri Kyjovsky <j1.kyjovsky@gmail.com>
2025-12-10 16:01:04 +01:00
Greg Sutcliffe
222ecbf573 Zabbix: apply load macro to all copr hosts instead of just the backend
Signed-off-by: Greg Sutcliffe <fedora@emeraldreverie.org>
2025-12-10 10:54:25 +00:00
Miroslav Suchý
360d90e907 bump up number of reserved instances 2025-12-09 14:31:52 +01:00
Pavel Raiskup
287960a5d6 copr: more power in AWS, since hypervisors are being moved to a different lab 2025-12-09 10:16:57 +01:00
Kevin Fenzi
737608a2e2 Revert "nagios / external: try and put pagure01 in rdu3_external to see if that makes noc02 happy"
This reverts commit 2d3797de65.

This just adds confusion, try reverting it for now.
2025-12-08 11:09:56 -08:00
Greg Sutcliffe
df6f219bcd Zabbix: Use new external API endpoints for COPR hosts
Signed-off-by: Greg Sutcliffe <fedora@emeraldreverie.org>
2025-12-08 16:46:04 +00:00
Greg Sutcliffe
6ed61b6b5a Zabbix: Bump load limits for copr-be and p10 hosts
Signed-off-by: Greg Sutcliffe <fedora@emeraldreverie.org>
2025-12-08 16:09:47 +00:00
Kevin Fenzi
557a9df3af pkgs: adjust procs and threads too
Signed-off-by: Kevin Fenzi <kevin@scrye.com>
2025-12-06 10:19:15 -08:00
Kevin Fenzi
f4e7495b01 virthost: increase nrpe proc limits
Signed-off-by: Kevin Fenzi <kevin@scrye.com>
2025-12-04 14:27:46 -08:00
Kevin Fenzi
2d3797de65 nagios / external: try and put pagure01 in rdu3_external to see if that makes noc02 happy
Signed-off-by: Kevin Fenzi <kevin@scrye.com>
2025-12-03 17:08:34 -08:00
Kevin Fenzi
9b02c8bf4e download: add some kernel.org mirrors (infra#12941)
Signed-off-by: Kevin Fenzi <kevin@scrye.com>
2025-12-03 11:26:58 -08:00
Kevin Fenzi
900a06c049 os-control: add sysadmin-openshift to be able to login and sudo
We want to add a few folks here so they can login to os-control and sudo
and debug cluster wide issues.

Signed-off-by: Kevin Fenzi <kevin@scrye.com>
2025-12-03 09:17:18 -08:00
Jiri Kyjovsky
6f94dc8907 copr: migration to f43, drop birthday, enable services 2025-11-26 10:57:16 +01:00
Jiri Kyjovsky
78a5b65cf0 copr: migrate to f43 step 1 2025-11-26 08:47:00 +01:00
Jiri Kyjovsky
8daa7e4eb1 copr: revert inventory changes, and birthday 2025-11-23 22:34:22 +01:00
Jiri Kyjovsky
f935ba42b7 copr-keygen-dev: specify the backend private ip 2025-11-23 20:12:57 +01:00
Jiri Kyjovsky
b22673576e copr-dist-git: allow anubis for f43 2025-11-23 20:08:51 +01:00