When downloading files from Koji to make a local repository, display
a temporary status of which files are being displayed to the console
appended after any log messages. Updates are done by erasing the status
that was written, adding a log message, then writing the status again.
When running in local mode, add a special handler that filters log
messages to the console to produce attractive output. Implemented
behaviors include:
- INFO level messages are only displayed if done through
MBSLogger.console() rather than MBSLogger.info().
- Timestamps and thread names are omitted unless the local build
is started with the -d option
- Warning/error messages have the level highlighted in red
- Special handling can be added to log messages, initially:
- Long running operations can be displayed to the console as
"Doing foo ... <pause>done"
Since the build log is already within the a build-specific directory, we
don't need to put the build ID (which ends up always being "2") into the
build log filename.
Logging during a build that occurs before the build directory is created
used to be logged to the console, but not retained in the build log
file. This made referring to the build log file confusing. Solve this
by buffering logs in memory until the log file is created and replaying
them.
A little bit of hackery is needed to avoid saving dangling references to
libsolv objects.
* Allow MBS_CONFIG_FILE="" to entirely suppress loading any configuration
file (useful for running tests and avoiding loading a system-wide
configuration file.)
* When loading the configuration file:
* If the default configuration file path doesn't exist, silently fall back
to the default configuration
* For any other OSError, print the exact error
* Let any other exception throw through, to allow people to debug their
configuration file
Fix some oddities in the DBResolver implementation of
get_compatible_base_module_modulemds() and make the MBSResolver version -
which was previously just buggy - match that. (Tests for the MBSResolver
version are added in a subsequent commit.)
* If an empty virtual_streams argument was passed in, *all* streams
were considered compatible. Throw an exception in this case - it
should be considered an error.
* If stream_version_lte=True, but the stream from the base module
wasn't in the form FOOx.y.z, then throw an exception. This was
previously treated like stream_version_lte=False, which is just
a recipe for confusion and mistakes.
test_get_reusable_module_use_latest_build() is rewritten to
comprehensively test all possibilities, including the case that changed
above.
test_add_default_modules_compatible_platforms() is changed to run
under allow_only_compatible_base_modules=False, since it expected
Fedora-style virtual streams (versions not in FOOx.y.z form, all
share the same stream), which doesn't make sense with
allow_only_compatible_base_modules=True.
The default LocalBuildConfiguration builds against Fedora - which has
ALLOW_ONLY_COMPATIBLE_BASE_MODULES = False on the server side. Match that
to avoid warnings that f33 isn't in <stream>x.y.z form.
config.allow_compatible_base_modules=False does different things for
build-requires selection and for module reuse. Clarify this in the config
key documentation.
(This config key is really: True: "do what RHEL expects"
False: "do what Fedora expects")
SQLAlchemy objects can't be used from multiple threads - so when starting
threads for builds, pass the ComponentBuild id rather than the object.
(Note that despite the comment that the threads were sharing a session,
they weren't - what was passed to the thread was a scoped_session that
acts as a separate thread-local session per-thread.)
BUILD_COMPONENT_DB_SESSION_LOCK - a threading.Lock() object that was used
in a few places - but not nearly enough places to effectively lock usage
of a shared session - is removed.
Using a memory database causes tests/test_build to intermittently
fail, because using the same pysqlite3 connection object from multiple
threads - as was done so that the threads shared the same memory database
- is not, in the end, thread safe. One thread will stomp on the transaction
state of other threads, resulting in errors from starting a new transaction
when another is already in progress, or trying to commit a transaction
that is not in progress.
To avoid a significant speed penalty, the session-scope fixture sets up
a database in the pytest temporary directory, which will typically be on
tmpfs. Time to complete all tests:
memory backend: 38 seconds
file on tmpfs: 40 seconds
file on nvme ssd with btrfs: 137 seconds
MBSSQLAlchemy, which attempted to make the memory backend work, is removed.
Session hooks are installed on the Session class rather than on the
scoped_session instance - this works better when we're changing from
one database to another at test setup time.
We don't need messages at all for local builds, so use a separate "drop"
backend and reserve the "in_memory" backend for tests, where we sometimes
want to inspect the messages. This avoids a warning for each published
message.
No implementations of MBS are using Greenwave, and there are no current plans
to do so. Koji Resolver will be sufficient for any usecase dependent on gating.
This will enable building modules from the new libmodulemd v3 format.
This format fixes major issue with modularity which enables clear
upgrade paths for multicontext modules.
Signed-off-by: Martin Curlej <mcurlej@redhat.com>
Since ComponentBuildTrace(s) get created with db_session.commit() call,
is is not possible to commit more items in bulk if they already have been flushed.
Current unit-tests' setup can be significantly sped up if items can be quickly
flushed on the fly and bulk-commited only once at the end. Moreover in general it
seems more appropriate/safer to handle this in before_flush as any implicit
or accidental flush could cause new build traces not to be created at all. As flush
is implicitly called before every commit anyway, this change shouldn't pose any harm.
In MBS, there are two cases to send a message when a module build moves
to a new state. One is to create a new module build, with
ModuleBuild.create particularly, when user submit a module build.
Another one is to transition a module build to a new state with
ModuleBuild.transition. This commit handles these two cases in a little
different ways.
For the former, existing code is refactored by moving the publish call
outside ModuleBuild.create.
For the latter, message is sent in a hook of SQLAlchemy ORM event
after_commit rather than immediately inside the ModuleBuild.transition.
Both of these changes ensure the message is sent after the changes are
committed into database successfully. Then, the backend can have
confidence that the database has the module build data when receive a
message.
Signed-off-by: Chenxiong Qi <cqi@redhat.com>
This also includes `from __future__ import absolute_import`
in every file so that the imports are consistent in Python 2 and 3.
The Python 2 tests fail without this.
