We only have two job groups, so the front page is a bit sad and
empty. Let's show 10 builds per group, not 3.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
it never worked anyway (ffmpeg always showed up, *somehow*) and
on the new workers it doesn't seem to be an issue.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
We weren't actually applying the quota we had defined (only) for
lab, so both lab and prod had the upstream default (which now
seems to be 200GB, not 100GB). Let's fix it so we do apply the
value, and set it to 250GB for both prod and stg, because we're
now aiming to have full parity in the update test sets between
aarch64 and x86_64, and we have the space on the rdu3 hosts.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
There's this annoying pattern where the NFS mount fails on boot
and then the worker services all start up and take jobs, but they
instafail because the share isn't there.
Ideally we could handle this very easily with Restart= directives
but systemd has...*opinions* about this:
https://github.com/systemd/systemd/issues/4468https://github.com/systemd/systemd/issues/1312
so we have to do some fairly awkward hacks to just express:
* Retry the NFS mount if it fails
* Don't start the workers unless the NFS mount is up
* Retry the workers after a while if they were blocked
It's ugly, but in testing this same config on one worker it seems
to work...
Signed-off-by: Adam Williamson <awilliam@redhat.com>
This reverts commit 4dc01bc892 and
a follow-up commit. I'm having trouble getting things to work
and want to see if it works if we go back to having the openQA
bridge be br0, and rename the bridge used for the system's bonded
network connection to something else instead.
I thought/assumed/knew/something? that resultsdb_conventions_fedora
required resultsdb_conventions, but right now it seems it doesn't.
It *should*, but I can't fix it right now as the buildsystem is
down, so let's just install it here...
Signed-off-by: Adam Williamson <awilliam@redhat.com>
On the new rdu3 worker hosts, br0 already exists and is the main
system 'interface' (it's a bridge on two bonded physical interfaces
connected to different switches, to make networking upgrades
easier). So we can't call our openvswitch bridge 'br0' any more.
Let's try calling it 'openqabr0' and see if anything explodes.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
OpenID support in FAS is going away. openQA has OAuth2 support.
I've tested this config to work with manual edits on lab, now
ansiblizing it (for lab only to start with).
Signed-off-by: Adam Williamson <awilliam@redhat.com>
This will unify all the handlers to use first uppercase letter for
ansible-lint to stop complaining.
I went through all `notify:` occurrences and fixed them by running
```
set TEXT "text_to_replace"; set REPLACEMENT "replacement_text"; git grep
-rlz "$TEXT" . | xargs -0 sed -i "s/$TEXT/$REPLACEMENT/g"
```
Then I went through all the changes and removed the ones that wasn't
expected to be changed.
Fixes https://pagure.io/fedora-infrastructure/issue/12391
Signed-off-by: Michal Konecny <mkonecny@redhat.com>
fix 1900 failures of the following case issue:
`name[casing]: All names should start with an uppercase letter.`
Signed-off-by: Ryan Lerch <rlerch@redhat.com>
This is an awful hack to deal with
https://github.com/os-autoinst/os-autoinst/issues/2549 while we
try and fix it properly. This finds stuck qemu processes by
parsing the journal messages of the workers, and kills them.
workers stuck in the broken state should then recover on the
next checkin with the server. I tested this manually on all the
worker hosts and it...seemed to work, mostly. I'll keep an eye on
things after deploying it.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
In various roles I maintain I use `python3 -m pip install` to
directly install a Python project (usually a fedora-messaging
consumer), to avoid the pointless bureaucracy of packaging them.
The roles install all the deps of these projects as packages
first, so pip doesn't have to install any deps, it only installs
the project itself. Well...that's the idea. It's possible for
this to go wrong (say I forget to update the roles when adding
a dep to the project), and in that case I think we'd rather have
things blow up (so I know something's wrong) than have pip
silently install some random upstream wheel system-wide to make
it work. The intent is that all the deps still come from proper
Fedora packages, only these projects themselves get installed
directly.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
This is a new feature in openQA that prevents worker hosts
picking up new jobs if their load average is above a certain
threshold. It defaults to 40. Our big worker hosts tend to run
above this, so let's bump it on those.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
network-scripts-openvswitch was removed in f40 and network-scripts
is going away in f41; we really need to get off using them.
This attempts to implement the same setup using NetworkManager,
based on a few different NM/ovs references, and the source of
openQA upstream's os-autoinst-setup-multi-machine . It might
need a bit of tweaking, so for now, we make it a separate task
and use it only on p09-worker01 for testing. This doesn't handle
tearing down the old network-scripts-based config as that's
pretty complex and will only need to happen once; I'll do it
manually before trying this out.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
We don't want it there - see earlier commits - but I didn't
notice it's actually explicitly listed here for all arches,
which breaks stuff on aarch64 now we told dnf to exclude it.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
This works around an annoying problem where, for some reason, we
sometimes just miss sending completed test results to resultsdb.
I've never been able to figure out why this happens, but this
should band-aid it by looking, daily, for updates stuck in
waiting gating status, checking for cases where a test finished
but we didn't send a result, and sending it.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
Using the same approach as we do for the tests and fedora_openqa.
I wish I'd done this *before* I ran the playbook on lab and it
wiped every...single...goddamn...disk image.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
Encoding with ffmpeg rather than os-autoinst's built-in encoder
gives us less broken videos, but on aarch64 it seems to cause
problems, especially on stg's old, busted worker hosts - I think
it's more CPU-intensive and they just can't handle the load. So,
let's block it.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
I'm adding this as a Recommends: for os-autoinst, but want to
get it on the workers now. Having it installed gives us better
videos of test runs (the internal video encoder is a bit wonky
and produces videos that have errors which make jumping around
within the video not work properly).
Signed-off-by: Adam Williamson <awilliam@redhat.com>
It's overall simpler and more idempotent to just use a side repo
maintained outside of ansible than re-create one on each system
on each run of the plays.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
It is extremely slow to run, and we figured out that the problem
on openqa01 was excessive space being used by Netapp snapshots,
so we don't need this any more. It was actually deleting old
jobs before their time, because it had already wiped every
video file and didn't know what else to do...
Signed-off-by: Adam Williamson <awilliam@redhat.com>
We're having issues with test results eating up all the disk
space we can throw at them (prod is over 4T, stg is over 2T -
I don't know why prod is bigger, that's odd, but it may be an
odd effect of having more arches on stg, maybe aarch64 and
ppc64le tests generally have smaller videos, or something).
This config setting should make openQA keep the space usage
on the partition at a max of 85%, by deleting videos from older
tests as required.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
nirik and I went around and around a bit today and ended up back
where we started, but with a clearer understanding of where that
this. This explains it a bit better, and makes what's actually
going on in various places clearer with the use of appropriate
shared variables. This should not actually *change* anything at
all when deployed.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
