Please note that this patch does not change the use of database session
in MBS. So, in the frontend, the database session is still managed by
Flask-SQLAlchemy, that is the db.session. And the backend, running event
handlers, has its own database session created from SQLAclehmy session
API directly.
This patch aims to reduce the number of scoped_session created when call
original function make_db_session. For technical detailed information,
please refer to SQLAlchemy documentation Contextual/Thread-local
Sessions.
As a result, a global scoped_session is accessible from the
code running inside backend, both the event handlers and functions
called from handlers. The library code shared by frontend and backend,
like resolvers, has no change.
Similarly, db.session is only used to recreate database for every test.
Signed-off-by: Chenxiong Qi <cqi@redhat.com>
Query Koji for the real stream name of each module and keep only those matching
requested `stream`.
This needs to be done, because MBS stores the stream name in the "version" field in Koji,
but the "version" field cannot contain "-" character. Therefore MBS replaces all "-"
with "_". This makes it impossible to reconstruct the original stream name from the
"version" field.
We therefore need to ask for real original stream name here and filter out modules based
on this real stream name.
In this commit, when component reuse code finds out that the base module uses
KojiResolver, it uses the `KojiResolver.get_buildrequired_modules` method
to find out possible modules to reuse and limits the original query just
by the IDs of these modules.
In order to do that, this commit splits the original
`KojiResolver.get_buildrequired_modulemds` into two methods:
- The `get_buildrequired_modules` returning the ModuleBuilds.
- The `get_buildrequired_modulemds` calling the `get_buildrequired_modules`
and returning modulemd metadata.
This also removes the outdated comments around authorship of each
file. If there is still interest in this information, one can just
look at the git history.
Before this commit, the base modules used in the `get_reusable_module` have
not been sorted and therefore when `get_reusable_module` tried to find out
the reusable module built against some base module, it could find a module
built against some old version of base module.
This could lead to situation when MBS tried to reuse components from quite
old module despite the fact that newer module build existed.
This commit fixes this by sorting the base modules by stream_version,
so MBS always tries to get the reusable module built against the latest
base module.
The goal here is to define certain branches from which only scratch
module builds can be submitted. The main use case is for "private-*"
branches which can be created and maintained by anyone, but there
must not be production-ready module build created from them.
This commit adds new `scratch_build_only_branches` config option
to define the list of regexes to match such branches.
* xmd/mbs is always set if it is not present in xmd, so move the code on
the top of function. This change is also helpful for accessing keys
under xmd/mbs.
* By setting xmd/mbs in the beginning, code is simplified to get
disttag_marking and virtual_streams. The result is much straightforwar
for getting a default value for them.
* Move disttag_marking validation code next to the line getting
disttag_marking from xmd/mbs. As a result, the code structure is
easier to read as getting disttag_marking and validate it, getting
virtual_streams and validate.
* Rewrite the part of code for check_buildrequires. Always set
xmd/mbs/buildrequires if it is not present and check_buildrequires is
set to True, as it is required by
ModuleBuild.get_buildrequired_base_modules.
* Using in operator instead of dict.get to check if key koji_tag exists.
Using dict.get would be ambiguous because even if koji_tag exists
under xmd/mbs, but due to its value is set to None occasionally, there
is still a message logged to tell koji_tag is not set.
* Rwrite all lines of code for updating virtual streams. A new method
update_virtual_streams is added to ModuleBuild. This also fixes
FACTORY-4561.
* Tests are added for the rewrite of virtual streams update.
Signed-off-by: Chenxiong Qi <cqi@redhat.com>
The original motivation for this refactor is to reuse make_module and
drop TestMMDResolver._make_mmd. Some tests require a modulemd created
and some tests also require those modulemd to be stored into database as
a module build. The problem is db_session has to be passed to
make_module even if no need to store into database.
Major changes in this patch:
* Argument db_session is optional.
* Arguments requires_list and build_requires_list are replaced by a
single argument dependencies which is a list of group of requires and
buildrequires
* A new make_module_in_db is created for creating and storing the new
modulemd into database conveniently.
* Tests are updated with the new make_module and make_module_in_db.
Signed-off-by: Chenxiong Qi <cqi@redhat.com>
When resubmitting a module build, if some component is found out that
its attributes have changed, MBS will raise an error to stop the work to
recording components. The problem is original code tells a module build
exists in database already rather than a component build, meanwhile
get_module_name() call on an ComponentRpm object is also incorrect.
Signed-off-by: Chenxiong Qi <cqi@redhat.com>
This patch separates the use of database session in different MBS components
and do not mix them together.
In general, MBS components could be separated as the REST API (implemented
based on Flask) and non-REST API including the backend build workflow
(implemented as a fedmsg consumer on top of fedmsg-hub and running
independently) and library shared by them. As a result, there are two kind of
database session used in MBS, one is created and managed by Flask-SQLAlchemy,
and another one is created from SQLAclhemy Session API directly. The goal of
this patch is to make ensure session object is used properly in the right
place.
All the changes follow these rules:
* REST API related code uses the session object db.session created and
managed by Flask-SQLAlchemy.
* Non-REST API related code uses the session object created with SQLAlchemy
Session API. Function make_db_session does that.
* Shared code does not created a new session object as much as possible.
Instead, it accepts an argument db_session.
The first two rules are applicable to tests as well.
Major changes:
* Switch tests back to run with a file-based SQLite database.
* make_session is renamed to make_db_session and SQLAlchemy connection pool
options are applied for PostgreSQL backend.
* Frontend Flask related code uses db.session
* Shared code by REST API and backend build workflow accepts SQLAlchemy session
object as an argument. For example, resolver class is constructed with a
database session, and some functions accepts an argument for database session.
* Build workflow related code use session object returned from make_db_session
and ensure db.session is not used.
* Only tests for views use db.session, and other tests use db_session fixture
to access database.
* All argument name session, that is for database access, are renamed to
db_session.
* Functions model_tests_init_data, reuse_component_init_data and
reuse_shared_userspace_init_data, which creates fixture data for
tests, are converted into pytest fixtures from original function
called inside setup_method or a test method. The reason of this
conversion is to use fixture ``db_session`` rather than create a
new one. That would also benefit the whole test suite to reduce the
number of SQLAlchemy session objects.
Signed-off-by: Chenxiong Qi <cqi@redhat.com>
Most of the issues are caused by the use of SQLAlchemy database session. Some
inline comments describe the issues in detail.
Signed-off-by: Chenxiong Qi <cqi@redhat.com>
There was a race condition, when 2 builds where build in the same time.
There was an issue that after one has finished the other reused the new build
for reuse and not the one it started with.
Ticket-ID: FACTORY-3862
Signed-off-by: Martin Curlej <mcurlej@redhat.com>
Currently, we are using just `conf.arches` and `conf.base_module_arches`
to define the list of arches for which the RPMs in a submitted module are
built. This is not enough, because RCM needs to generate modules based
on the base modules which should use different arches.
This commit changes the MBS to take the list of arches from the buildrequired
module build. It checks the buildrequires for "privileged" module or base
module and if it finds such module, it queries the Koji to find out the list
of arches to set for the module.
The "privileged" module is a module which can override base module arches
or disttag. Previously, these modules have been defined by
`allowed_disttag_marking_module_names` config option. In this commit,
this has been renamed to `allowed_privileged_module_names`.
The list of arches are stored per module build in new table represented
by ModuleArch class and are m:n mapped to ModuleBuild.
This also moves the methods load_mmd and load_mmd_file to
module_build_service.utils.general.
This also removes some MSE unit tests with a mix of positive and
negative streams since this is not supported in libmodulemd v2. The
user will be presented with a syntax error if they try to submit
such a modulemd file.
GenericResolver.extract_modulemd is not removed, but deprecated. Call of it
will result in a deprecation message printed. Any new code should call
load_mmd.
Signed-off-by: Chenxiong Qi <cqi@redhat.com>
Currently, the PLATFORM_ID is parsed from the `/etc/os-release`. This
is good default value, but sometimes you want to build module locally
against the different platform stream.
For example building on platform:f29 against the platform:f30 modules. In
that case, we need to be able to override the host PLATFORM_ID and
set it manually chosen value.
When importing modules for offline local builds from local repositories,
the XMD section does not have to be set at all - it gets removed during
the compose and is also MBS specific.
We need to be able to import such MMDs using the `import_mmd` method
in order to make --offline local builds working.
This commit adds new `check_buildrequires` bool kwarg in `import_mmd`
method to disable `xmd["buildrequires"]` checks to fix this.
These module builds will basically act as metadata-only module builds.
This will be more useful as additional features stem from these types
of builds.
Previously MockModuleBuilder was checking the module state to see if
it should run a final createrepo, but since eafa93037f, finalize() is
called before changing the module state; add an explicit boolean to
GenericBuilder.finalize() to avoid worrying about ordering.
There are following changes introduced in this commit:
- The `koji_tag` of module builds imported from the local repositories
is now in `repofile:///etc/yum.repos.d/some.repo` format to store the
repository from which the module was imported to local MBS DB.
- The `koji_tag` of fake base module is set to empty `repofile://`
and in `MockModuleBuilder` the `conf.base_module_repofiles` list
is used as source for the repositories defining platform. We can't
simply use single repository, because there might be fedora.repo
and fedora-update.repo and so on.
- The list of default .repo files for platform are passed using the
`-r` switch in `build_module_locally` `mbs-manager` command.
- The LocalResolver (subclass of DBResolver) is added which is used
to resolve the build dependencies when building modules offline
locally.
- The `MockModuleBuilder` enables the buildrequired modules and
repositories from which they come in the mock config.
With this commit, it is possible to build testmodule locally
without any external infra.
This is the first PR in many for Offline local builds. This PR:
- Adds --offline flag to build_module_locally mbs-manager command to enable
offline local builds.
- If this flag is used, new `import_builds_from_local_dnf_repos` method is
called which uses DNF API to get all the available installable modulemd
files and imports each module into MBS local SQLite database.
- It also adds fake "platform:stream" module based on the /etc/os-release,
so the buildrequirements of the imported modules are satisfied.
The idea here is that in upcoming commits, I will create LocalResolver
which will be similar to DBResolver with some extra rules to resolve
local module builds. This new LocalResolver will still be based on
the models.ModuleBuild methods and therefore we need the modules
imported in database.
Since the required parameters vary based on if the modulemd
comes from SCM or a direct submission, the concept of optional
parameters doesn't really apply.
When importing a base module, we must ensure the value that will be
used in the RPM disttags doesn't contain a dash since a dash isn't
allowed in the release field of the NVR.
MBS uses the base module's stream that was buildrequired by the module
in the RPM disttags for that module build. The stream name may not be
ideal for all situations, so now this is customizable by setting the
xmd['mbs']['disttag_marking'] in the base module's modulemd.
