When running in local mode, add a special handler that filters log
messages to the console to produce attractive output. Implemented
behaviors include:
- INFO level messages are only displayed if done through
MBSLogger.console() rather than MBSLogger.info().
- Timestamps and thread names are omitted unless the local build
is started with the -d option
- Warning/error messages have the level highlighted in red
- Special handling can be added to log messages, initially:
- Long running operations can be displayed to the console as
"Doing foo ... <pause>done"
Use dogpile.cache to avoid repeatedly making the same queries to the MBS.
We frequently query a list of modules, and then later in the build go
back and ask for details of those modules again by nsvc, so special case
querying a single module by nsvc and prime the cache for this from lists
of results.
The code made some half-hearted attemps to handle a MBS verbose query
missing the modulemd data, but it didn't do anything logically
consistent - fix it to consistently ignore such modules.
Code in MBSResolver.resolve_requires() to compute filtered_rpms()
assumed the response from MBS has a "rpms" key with binary RPMS
from the build, but this was never implemented.
Fix some oddities in the DBResolver implementation of
get_compatible_base_module_modulemds() and make the MBSResolver version -
which was previously just buggy - match that. (Tests for the MBSResolver
version are added in a subsequent commit.)
* If an empty virtual_streams argument was passed in, *all* streams
were considered compatible. Throw an exception in this case - it
should be considered an error.
* If stream_version_lte=True, but the stream from the base module
wasn't in the form FOOx.y.z, then throw an exception. This was
previously treated like stream_version_lte=False, which is just
a recipe for confusion and mistakes.
test_get_reusable_module_use_latest_build() is rewritten to
comprehensively test all possibilities, including the case that changed
above.
test_add_default_modules_compatible_platforms() is changed to run
under allow_only_compatible_base_modules=False, since it expected
Fedora-style virtual streams (versions not in FOOx.y.z form, all
share the same stream), which doesn't make sense with
allow_only_compatible_base_modules=True.
This also includes `from __future__ import absolute_import`
in every file so that the imports are consistent in Python 2 and 3.
The Python 2 tests fail without this.
This moves the code used by the backend and API to common/submit.py,
the code used just by the API to web/submit.py, and the code used
just by the backend to scheduler/submit.py.
This puts backend specific code in either the builder or scheduler
subpackage. This puts API specific code in the new web subpackage.
Lastly, any code shared between the API and backend is placed in the
common subpackage.
Query Koji for the real stream name of each module and keep only those matching
requested `stream`.
This needs to be done, because MBS stores the stream name in the "version" field in Koji,
but the "version" field cannot contain "-" character. Therefore MBS replaces all "-"
with "_". This makes it impossible to reconstruct the original stream name from the
"version" field.
We therefore need to ask for real original stream name here and filter out modules based
on this real stream name.
If KojiResolver is enabled for buildrequired base module and
MBSResolver is used, then `MBSResolver.get_buildrequired_modulemds`
will use KojiResolver to get the list of buildrequired module builds.
Otherwise it uses the current behavior.
To implement this, the `KojiResolver.get_buildrequired_modules` was
split into two methods:
- `get_buildrequired_koji_builds` returns buildrequired Koji builds.
- `get_buildrequired_modules` calls `get_buildrequired_koji_builds`
and finds the corresponding ModuleBuilds in MBS DB.
In this commit, when component reuse code finds out that the base module uses
KojiResolver, it uses the `KojiResolver.get_buildrequired_modules` method
to find out possible modules to reuse and limits the original query just
by the IDs of these modules.
In order to do that, this commit splits the original
`KojiResolver.get_buildrequired_modulemds` into two methods:
- The `get_buildrequired_modules` returning the ModuleBuilds.
- The `get_buildrequired_modulemds` calling the `get_buildrequired_modules`
and returning modulemd metadata.
This also removes the outdated comments around authorship of each
file. If there is still interest in this information, one can just
look at the git history.
For KojiResolver, this method returns always an empty list. The compatible modules are
defined by the Koji tag inheritance, so there is no need to find out the compatible
base modules on MBS side.
This makes `mse.get_base_module_mmds` to ignore virtual streams and just use
the input base module as the only module without finding the compatible
base modules.
This commit:
- Adds KojiResolver class and KojiResolver tests.
- Changes the GenericResolver and its subclasses to pass base_module_mmds
instead of base_module_nsvc to get_buildrequired_modulemds. This is needed,
because KojiResolver needs to access XMD section of base module.
- Implements KojiResolver.get_buildrequired_modulemds to ask Koji for list of
modules tagged in the Koji tag and return their modulemds.
The `MBSResolver.get_buildrequired_modulemds` did not try to load
local module builds and always just queried the remote MBS instance.
This commit fixes it by using local module if available.
This patch separates the use of database session in different MBS components
and do not mix them together.
In general, MBS components could be separated as the REST API (implemented
based on Flask) and non-REST API including the backend build workflow
(implemented as a fedmsg consumer on top of fedmsg-hub and running
independently) and library shared by them. As a result, there are two kind of
database session used in MBS, one is created and managed by Flask-SQLAlchemy,
and another one is created from SQLAclhemy Session API directly. The goal of
this patch is to make ensure session object is used properly in the right
place.
All the changes follow these rules:
* REST API related code uses the session object db.session created and
managed by Flask-SQLAlchemy.
* Non-REST API related code uses the session object created with SQLAlchemy
Session API. Function make_db_session does that.
* Shared code does not created a new session object as much as possible.
Instead, it accepts an argument db_session.
The first two rules are applicable to tests as well.
Major changes:
* Switch tests back to run with a file-based SQLite database.
* make_session is renamed to make_db_session and SQLAlchemy connection pool
options are applied for PostgreSQL backend.
* Frontend Flask related code uses db.session
* Shared code by REST API and backend build workflow accepts SQLAlchemy session
object as an argument. For example, resolver class is constructed with a
database session, and some functions accepts an argument for database session.
* Build workflow related code use session object returned from make_db_session
and ensure db.session is not used.
* Only tests for views use db.session, and other tests use db_session fixture
to access database.
* All argument name session, that is for database access, are renamed to
db_session.
* Functions model_tests_init_data, reuse_component_init_data and
reuse_shared_userspace_init_data, which creates fixture data for
tests, are converted into pytest fixtures from original function
called inside setup_method or a test method. The reason of this
conversion is to use fixture ``db_session`` rather than create a
new one. That would also benefit the whole test suite to reduce the
number of SQLAlchemy session objects.
Signed-off-by: Chenxiong Qi <cqi@redhat.com>
This seems to be better behaviour than simply rejecting the module build
completely. MBS still shows warning in the log that it cannot find
any compatible module, but the build continues with the base module
requested in the submitted modulemd.
This commit fixes issue with following situation:
Module `foo` has been built with `buildrequires: platform: [f28]`
and `requires: platform: []`. It can therefore be used as a buildrequirement
on any platform stream. But if you want to build module `app` which
buildrequires `foo` on `platform:f30`, the MBS won't pull-in the `foo`
module build, because MBS currently limits the modules by the platform
they have been built for.
This commit adds new config option to allow including modules built
against any platform stream.
This also moves the methods load_mmd and load_mmd_file to
module_build_service.utils.general.
This also removes some MSE unit tests with a mix of positive and
negative streams since this is not supported in libmodulemd v2. The
user will be presented with a syntax error if they try to submit
such a modulemd file.
GenericResolver.extract_modulemd is not removed, but deprecated. Call of it
will result in a deprecation message printed. Any new code should call
load_mmd.
Signed-off-by: Chenxiong Qi <cqi@redhat.com>
