In this commit, when component reuse code finds out that the base module uses
KojiResolver, it uses the `KojiResolver.get_buildrequired_modules` method
to find out possible modules to reuse and limits the original query just
by the IDs of these modules.
In order to do that, this commit splits the original
`KojiResolver.get_buildrequired_modulemds` into two methods:
- The `get_buildrequired_modules` returning the ModuleBuilds.
- The `get_buildrequired_modulemds` calling the `get_buildrequired_modules`
and returning modulemd metadata.
This also removes the outdated comments around authorship of each
file. If there is still interest in this information, one can just
look at the git history.
This commit:
- Adds KojiResolver class and KojiResolver tests.
- Changes the GenericResolver and its subclasses to pass base_module_mmds
instead of base_module_nsvc to get_buildrequired_modulemds. This is needed,
because KojiResolver needs to access XMD section of base module.
- Implements KojiResolver.get_buildrequired_modulemds to ask Koji for list of
modules tagged in the Koji tag and return their modulemds.
An empty list [] is evaluated as a false value. So, it is not necessary
to compare like "if some_var == []:".
Signed-off-by: Chenxiong Qi <cqi@redhat.com>
This patch separates the use of database session in different MBS components
and do not mix them together.
In general, MBS components could be separated as the REST API (implemented
based on Flask) and non-REST API including the backend build workflow
(implemented as a fedmsg consumer on top of fedmsg-hub and running
independently) and library shared by them. As a result, there are two kind of
database session used in MBS, one is created and managed by Flask-SQLAlchemy,
and another one is created from SQLAclhemy Session API directly. The goal of
this patch is to make ensure session object is used properly in the right
place.
All the changes follow these rules:
* REST API related code uses the session object db.session created and
managed by Flask-SQLAlchemy.
* Non-REST API related code uses the session object created with SQLAlchemy
Session API. Function make_db_session does that.
* Shared code does not created a new session object as much as possible.
Instead, it accepts an argument db_session.
The first two rules are applicable to tests as well.
Major changes:
* Switch tests back to run with a file-based SQLite database.
* make_session is renamed to make_db_session and SQLAlchemy connection pool
options are applied for PostgreSQL backend.
* Frontend Flask related code uses db.session
* Shared code by REST API and backend build workflow accepts SQLAlchemy session
object as an argument. For example, resolver class is constructed with a
database session, and some functions accepts an argument for database session.
* Build workflow related code use session object returned from make_db_session
and ensure db.session is not used.
* Only tests for views use db.session, and other tests use db_session fixture
to access database.
* All argument name session, that is for database access, are renamed to
db_session.
* Functions model_tests_init_data, reuse_component_init_data and
reuse_shared_userspace_init_data, which creates fixture data for
tests, are converted into pytest fixtures from original function
called inside setup_method or a test method. The reason of this
conversion is to use fixture ``db_session`` rather than create a
new one. That would also benefit the whole test suite to reduce the
number of SQLAlchemy session objects.
Signed-off-by: Chenxiong Qi <cqi@redhat.com>
This seems to be better behaviour than simply rejecting the module build
completely. MBS still shows warning in the log that it cannot find
any compatible module, but the build continues with the base module
requested in the submitted modulemd.
This commit fixes issue with following situation:
Module `foo` has been built with `buildrequires: platform: [f28]`
and `requires: platform: []`. It can therefore be used as a buildrequirement
on any platform stream. But if you want to build module `app` which
buildrequires `foo` on `platform:f30`, the MBS won't pull-in the `foo`
module build, because MBS currently limits the modules by the platform
they have been built for.
This commit adds new config option to allow including modules built
against any platform stream.
This also moves the methods load_mmd and load_mmd_file to
module_build_service.utils.general.
This also removes some MSE unit tests with a mix of positive and
negative streams since this is not supported in libmodulemd v2. The
user will be presented with a syntax error if they try to submit
such a modulemd file.
The original description of argument stream is inconsistent with the value
passed to function _expand_mse_streams. The actual passed-in value is
Modulemd.SimpleSet. To fix this inconsistency, just pass list of streams, each
of them is a string representing a stream.
Signed-off-by: Chenxiong Qi <cqi@redhat.com>
If base module name:stream is not found, consider `stream` as virtual stream
and return the latest (the one with highest stream_version) base module with
that virtual stream.
Before this commit, the compatible base modules for Module Stream Expansion
have been found without any limitation, just based on the stream version.
It was therefore possible that `platform:lp29.0.0` was found as compatible
module for `platform:f29.1.0` although those platform streams are not
compatible at all.
In this commit, the module can be treated as compatible only if it has
the same virtual stream as the input module. The idea behind this
is that both `platform:f29.0.0` and `platform:f29.1.0` should include
the `virtual_streams: [f29]` in their XMD section which tells MBS
that they are actually compatible. The `lp29` stream will not
have the same virtual stream (most likely it won't have any virtual
stream at all).
The `virtual_streams` is already used for this use-case in `MMDResolver`,
but it was not used to limit the inputs to `MMDResolver` which is what
this commit is doing.
This commit also fixes the issue in `get_last_builds_in_stream_version_lte`
which was simply broken if multiple stream_versions of single base module
existed and their builds had different version. In this case, only
builds with single (randomly chosen) version were returned.
Imagine we have "platform:f29.0.0" and "platform:f29.1.0" base modules.
We also have "DBI" module we want to build agaisnt "platform:f29.1.0".
This "DBI" module depends on "perl" module which is only build against
"platform:f29.0.0".
Currently, DBI build would fail to resolve the dependencies, because
it wouldn't find "perl" module, because it is built against different
platform stream.
This PR changes the MSE code to include buildrequired module builds built
against all the compatible platform streams.
It does so by introducing following changes:
- MSE code uses new get_base_module_mmds() method to find out all the
compatible platform modules. This needed new methods in DBResolver
and MBSResolver.
- For each buildrequired module defined by name:stream, the MSE code then
finds particular NSVC built against each compatible platform module.
Side effect of these code changes is that every module now must buildrequire
some base module.
_get_mmds_from_requires does not do anything against db.session. Hence,
remove parameter session from it and
get_mmds_required_by_module_recursively.
Signed-off-by: Chenxiong Qi <cqi@redhat.com>
system_resolver is created based on loaded configuration, which could
avoid calls like `GenericResolver.create(conf)` repeatedly in the code.
However, if some cases need to create a specific resolver explicitly,
`GenericResolver.create` could be called with addition argument, for
example db or mbs is passed to argument backend in tests.
Signed-off-by: Chenxiong Qi <cqi@redhat.com>
We can't rely on matching name-stream to distinguish which module is
currently being built as modules can buildrequire older versions of
the same module-stream.
Fixes#954
1. Changed ModuleBuild's context property to db column, it's
non-nullable and default value is '00000000' to keep it consistent
with previous behaviour.
2. Changed ModuleBuild.contexts_from_mmd to return a tuple of
(ref_build_context, build_context, runtime_context, context).
3. Updated tests affected by this change.
