OpenID support in FAS is going away. openQA has OAuth2 support.
I've tested this config to work with manual edits on lab, now
ansiblizing it (for lab only to start with).
Signed-off-by: Adam Williamson <awilliam@redhat.com>
This will unify all the handlers to use first uppercase letter for
ansible-lint to stop complaining.
I went through all `notify:` occurrences and fixed them by running
```
set TEXT "text_to_replace"; set REPLACEMENT "replacement_text"; git grep
-rlz "$TEXT" . | xargs -0 sed -i "s/$TEXT/$REPLACEMENT/g"
```
Then I went through all the changes and removed the ones that wasn't
expected to be changed.
Fixes https://pagure.io/fedora-infrastructure/issue/12391
Signed-off-by: Michal Konecny <mkonecny@redhat.com>
fix 1900 failures of the following case issue:
`name[casing]: All names should start with an uppercase letter.`
Signed-off-by: Ryan Lerch <rlerch@redhat.com>
This is an awful hack to deal with
https://github.com/os-autoinst/os-autoinst/issues/2549 while we
try and fix it properly. This finds stuck qemu processes by
parsing the journal messages of the workers, and kills them.
workers stuck in the broken state should then recover on the
next checkin with the server. I tested this manually on all the
worker hosts and it...seemed to work, mostly. I'll keep an eye on
things after deploying it.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
In various roles I maintain I use `python3 -m pip install` to
directly install a Python project (usually a fedora-messaging
consumer), to avoid the pointless bureaucracy of packaging them.
The roles install all the deps of these projects as packages
first, so pip doesn't have to install any deps, it only installs
the project itself. Well...that's the idea. It's possible for
this to go wrong (say I forget to update the roles when adding
a dep to the project), and in that case I think we'd rather have
things blow up (so I know something's wrong) than have pip
silently install some random upstream wheel system-wide to make
it work. The intent is that all the deps still come from proper
Fedora packages, only these projects themselves get installed
directly.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
This is a new feature in openQA that prevents worker hosts
picking up new jobs if their load average is above a certain
threshold. It defaults to 40. Our big worker hosts tend to run
above this, so let's bump it on those.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
network-scripts-openvswitch was removed in f40 and network-scripts
is going away in f41; we really need to get off using them.
This attempts to implement the same setup using NetworkManager,
based on a few different NM/ovs references, and the source of
openQA upstream's os-autoinst-setup-multi-machine . It might
need a bit of tweaking, so for now, we make it a separate task
and use it only on p09-worker01 for testing. This doesn't handle
tearing down the old network-scripts-based config as that's
pretty complex and will only need to happen once; I'll do it
manually before trying this out.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
We don't want it there - see earlier commits - but I didn't
notice it's actually explicitly listed here for all arches,
which breaks stuff on aarch64 now we told dnf to exclude it.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
This works around an annoying problem where, for some reason, we
sometimes just miss sending completed test results to resultsdb.
I've never been able to figure out why this happens, but this
should band-aid it by looking, daily, for updates stuck in
waiting gating status, checking for cases where a test finished
but we didn't send a result, and sending it.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
Using the same approach as we do for the tests and fedora_openqa.
I wish I'd done this *before* I ran the playbook on lab and it
wiped every...single...goddamn...disk image.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
Encoding with ffmpeg rather than os-autoinst's built-in encoder
gives us less broken videos, but on aarch64 it seems to cause
problems, especially on stg's old, busted worker hosts - I think
it's more CPU-intensive and they just can't handle the load. So,
let's block it.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
I'm adding this as a Recommends: for os-autoinst, but want to
get it on the workers now. Having it installed gives us better
videos of test runs (the internal video encoder is a bit wonky
and produces videos that have errors which make jumping around
within the video not work properly).
Signed-off-by: Adam Williamson <awilliam@redhat.com>
It's overall simpler and more idempotent to just use a side repo
maintained outside of ansible than re-create one on each system
on each run of the plays.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
It is extremely slow to run, and we figured out that the problem
on openqa01 was excessive space being used by Netapp snapshots,
so we don't need this any more. It was actually deleting old
jobs before their time, because it had already wiped every
video file and didn't know what else to do...
Signed-off-by: Adam Williamson <awilliam@redhat.com>
We're having issues with test results eating up all the disk
space we can throw at them (prod is over 4T, stg is over 2T -
I don't know why prod is bigger, that's odd, but it may be an
odd effect of having more arches on stg, maybe aarch64 and
ppc64le tests generally have smaller videos, or something).
This config setting should make openQA keep the space usage
on the partition at a max of 85%, by deleting videos from older
tests as required.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
nirik and I went around and around a bit today and ended up back
where we started, but with a clearer understanding of where that
this. This explains it a bit better, and makes what's actually
going on in various places clearer with the use of appropriate
shared variables. This should not actually *change* anything at
all when deployed.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
I'm going to try splitting the tap jobs across multiple worker
hosts. We have quite a lot of tap jobs, now, and I have seen
sometimes a situation where all non-tap jobs have been run and
the non-tap worker hosts are sitting idle, but the single tap
worker host has a long queue of tap jobs to get through.
We can't just put multiple hosts per instance into the tap
class, because then we might get a case where job A from a tap
group is run on one host and job B from a tap group is run on
a different host, and they can't communicate. It's actually
possible to set this up so it works, but it needs yet more
complex networking stuff I don't want to mess with. So instead
I'm just gonna split the tap job groups across two classes,
'tap' and 'tap2'. That way we can have one 'tap' worker host
and one 'tap2' worker host per instance and arch, and they will
each get about half the tap jobs.
Unfortunately since we only have one aarch64 worker for lab it
will still have to run all the jobs, but for all other cases we
do have at least two workers, so we can split the load.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
This reverts commit 892453da7e.
openQA still had problems with the very long request, so I just
did an ugly hack to get the request under the limit instead.
The openQA job scheduler was hitting 414 errors today because
an update has so many builds there are more than 8190 characters
(the default limit) in the POST request. Let's bump the limit
to 16000.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
It really needs to be called exactly 60-block-scheduler.rules
as it's overriding a file of the same name in `/usr`.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
This applies only within Fedora infra for now, as we're not sure
whether worker hosts on different hardware hit this bug. It's
intended to work around:
https://bugzilla.redhat.com/show_bug.cgi?id=2009585
a bug which results in the infra worker hosts hanging after a
short time when running kernels newer than 5.11.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
Upstream implemented a feature that we can use to do the same
thing using just a test variable, so we're switching to that.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
Ugh, we delegate for the assetsize stuff too and there's tons of
that, splitting it would be awful. Let's try a different approach
with a new optional variable for the delegate target.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
Using the machine's own hostname works for the ansible delegate
stuff but doesn't work for openQA itself (if you try and access
the DB by hostname like this, postgres denies access; you have
to use 'localhost' for postgres to allow it). Using 'localhost'
works for postgres but doesn't do the right thing for delegation.
Let's use 'localhost' and split the two play steps into
delegated and non-delegated versions.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
The config file should treat these as optional, not every openQA
instance wants to report results.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
We don't want to include this section if the vars aren't set.
Not every openQA server has to be an AMQP publisher.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
