* Add a script make-dist-tarball.py that runs 'setup.py sdist' and compares
what gets distributed with git and with a set of patterns of extra files
added during distribution, and with git files that we don't want to
distribute.
* Extend MANIFEST.in or add __init__.py files to distribute:
run-unittests.sh
tox.in
conf/client_secrets.json
tests/test_builder
tests/integration
* Don't include all of module_build_service - this is too prone to backup
files and development trash files; instead add just the files we need
that aren't *.py
* Move global-excludes to the end - they modify the existing list of files;
and add *.db - for some reason, .mbs_local_build.db files were
getting disted.
To clean things up and facilitate future changes to test cases, split
the method for creating a builder apart from the fixture so that it can
be used independently.
For local builds, required modules are not necessarily in the local
database, so the method of looking up the build to find the koji tag
doesn't work reliably. However, the caller has the koji_tag - so just
pass it in.
When downloading files from Koji to make a local repository, display
a temporary status of which files are being displayed to the console
appended after any log messages. Updates are done by erasing the status
that was written, adding a log message, then writing the status again.
When running in local mode, add a special handler that filters log
messages to the console to produce attractive output. Implemented
behaviors include:
- INFO level messages are only displayed if done through
MBSLogger.console() rather than MBSLogger.info().
- Timestamps and thread names are omitted unless the local build
is started with the -d option
- Warning/error messages have the level highlighted in red
- Special handling can be added to log messages, initially:
- Long running operations can be displayed to the console as
"Doing foo ... <pause>done"
Use dogpile.cache to avoid repeatedly making the same queries to the MBS.
We frequently query a list of modules, and then later in the build go
back and ask for details of those modules again by nsvc, so special case
querying a single module by nsvc and prime the cache for this from lists
of results.
Add new tests and modify existing tests so that test_mbs.py entirely
covers MBSResolver.py. To make the tests simpler, change from
explicitly expectations about the MBS API request => response sequence
to a stub local implementatin of the MBS API.
Fix some oddities in the DBResolver implementation of
get_compatible_base_module_modulemds() and make the MBSResolver version -
which was previously just buggy - match that. (Tests for the MBSResolver
version are added in a subsequent commit.)
* If an empty virtual_streams argument was passed in, *all* streams
were considered compatible. Throw an exception in this case - it
should be considered an error.
* If stream_version_lte=True, but the stream from the base module
wasn't in the form FOOx.y.z, then throw an exception. This was
previously treated like stream_version_lte=False, which is just
a recipe for confusion and mistakes.
test_get_reusable_module_use_latest_build() is rewritten to
comprehensively test all possibilities, including the case that changed
above.
test_add_default_modules_compatible_platforms() is changed to run
under allow_only_compatible_base_modules=False, since it expected
Fedora-style virtual streams (versions not in FOOx.y.z form, all
share the same stream), which doesn't make sense with
allow_only_compatible_base_modules=True.
Two problems occurred with the moksha/twisted handling of SIGINT:
* While KeyboardInterrupt caused the moksha loop to exit, it just
left the test in a confused state, instead of triggering standard
pytest behavior and aborting the entire test run.
* The handler was left-over for remaining tests that prevent Control-C
from working at all.
Fix that by using mock.patch to override moksha's signal handler with
our own signal handler that stores the KeyboardInterrupt in the
current EventTrap, and restores the default signal handler after
the loop ends.
Note that since the KeyboardInterrupt is always handled in the main thread,
we don't get a useful backtrace from the child thread.
Event handlers decorated with @celery_app_task don't raise an exception -
they just log the exception, leaving the Moksha hub running. This meant
that any failures in test_build would result in the test suite hanging
rather than failing usefully.
We solve this by adding a new class EventTrap which acts as a context
manager. Any exceptions that occur in event handlers are set on the
current EventTrap, and the test_build tests re-raise the exception.
SQLAlchemy objects can't be used from multiple threads - so when starting
threads for builds, pass the ComponentBuild id rather than the object.
(Note that despite the comment that the threads were sharing a session,
they weren't - what was passed to the thread was a scoped_session that
acts as a separate thread-local session per-thread.)
BUILD_COMPONENT_DB_SESSION_LOCK - a threading.Lock() object that was used
in a few places - but not nearly enough places to effectively lock usage
of a shared session - is removed.
Using a memory database causes tests/test_build to intermittently
fail, because using the same pysqlite3 connection object from multiple
threads - as was done so that the threads shared the same memory database
- is not, in the end, thread safe. One thread will stomp on the transaction
state of other threads, resulting in errors from starting a new transaction
when another is already in progress, or trying to commit a transaction
that is not in progress.
To avoid a significant speed penalty, the session-scope fixture sets up
a database in the pytest temporary directory, which will typically be on
tmpfs. Time to complete all tests:
memory backend: 38 seconds
file on tmpfs: 40 seconds
file on nvme ssd with btrfs: 137 seconds
MBSSQLAlchemy, which attempted to make the memory backend work, is removed.
Session hooks are installed on the Session class rather than on the
scoped_session instance - this works better when we're changing from
one database to another at test setup time.
The base module conflict generation was skipped for local builds
in 6b2e5be93a because libdnf wasn't ported to libmodulemd yet -
that was done in libdnf-0.45, so only warn and skip for versions of
dnf too old to require libdnf-0.45.
(Don't just unconditionally skip check/warning in case someone is
doing local module builds on RHEL 8.)
No implementations of MBS are using Greenwave, and there are no current plans
to do so. Koji Resolver will be sufficient for any usecase dependent on gating.
There is a need to rebuild the module builds done in CentOS 9 Stream
internally in MBS to include them in RHEL. This is currenly a hard task,
because the RPM components included in a module are usually
taken from HEAD of the branch defined by their `ref` value.
For the rebuild task, it means we would have to ensure that the HEAD
of all RPM components points to right commit hash right before we start
rebuilding CentOS 9 Stream module in internal MBS. This is very hard
and fragile thing to do, especially if there are two different modules
using the RPM component from the same branch. This is prone to race
condition and makes the rebuilds quite complex and in some cases
not possible to do without force pushes to RPM component repositories
which is not acceptable by internal dist-git policy.
This commit fixes it by allowing overriding the commit hash while
submitting the module build. This helps in the mentioned situation,
because we can keep internal RPM components branches in 1:1 sync with
CentOS 9 Stream branches and HEAD can always point to the same commit
in both internal and CentOS 9 Stream repositories.
When the module rebuild is submitted in internal MBS,
we can use this new feature to override the `ref` for each RPM component
so it points to particular commit and the requirement for HEAD to point
to this commit is no longer there.
The `ref` is overriden only internally in MBS (but it is recorded in logs
and in XMD section), so the input modulemd file is not altered. This is
the same logic as used for other overrides (`buildrequire_overrides` or
`side_tag`).
This does not bring any security problem, because it is already possible
to use commit hash in `ref`, so the package maintainer can already change
the commit hash to any particular commit by using this `ref` value.
Signed-off-by: Jan Kaluza <jkaluza@redhat.com>
