This patch separates the use of database session in different MBS components
and do not mix them together.
In general, MBS components could be separated as the REST API (implemented
based on Flask) and non-REST API including the backend build workflow
(implemented as a fedmsg consumer on top of fedmsg-hub and running
independently) and library shared by them. As a result, there are two kind of
database session used in MBS, one is created and managed by Flask-SQLAlchemy,
and another one is created from SQLAclhemy Session API directly. The goal of
this patch is to make ensure session object is used properly in the right
place.
All the changes follow these rules:
* REST API related code uses the session object db.session created and
managed by Flask-SQLAlchemy.
* Non-REST API related code uses the session object created with SQLAlchemy
Session API. Function make_db_session does that.
* Shared code does not created a new session object as much as possible.
Instead, it accepts an argument db_session.
The first two rules are applicable to tests as well.
Major changes:
* Switch tests back to run with a file-based SQLite database.
* make_session is renamed to make_db_session and SQLAlchemy connection pool
options are applied for PostgreSQL backend.
* Frontend Flask related code uses db.session
* Shared code by REST API and backend build workflow accepts SQLAlchemy session
object as an argument. For example, resolver class is constructed with a
database session, and some functions accepts an argument for database session.
* Build workflow related code use session object returned from make_db_session
and ensure db.session is not used.
* Only tests for views use db.session, and other tests use db_session fixture
to access database.
* All argument name session, that is for database access, are renamed to
db_session.
* Functions model_tests_init_data, reuse_component_init_data and
reuse_shared_userspace_init_data, which creates fixture data for
tests, are converted into pytest fixtures from original function
called inside setup_method or a test method. The reason of this
conversion is to use fixture ``db_session`` rather than create a
new one. That would also benefit the whole test suite to reduce the
number of SQLAlchemy session objects.
Signed-off-by: Chenxiong Qi <cqi@redhat.com>
Most of the issues are caused by the use of SQLAlchemy database session. Some
inline comments describe the issues in detail.
Signed-off-by: Chenxiong Qi <cqi@redhat.com>
Currently, we are using just `conf.arches` and `conf.base_module_arches`
to define the list of arches for which the RPMs in a submitted module are
built. This is not enough, because RCM needs to generate modules based
on the base modules which should use different arches.
This commit changes the MBS to take the list of arches from the buildrequired
module build. It checks the buildrequires for "privileged" module or base
module and if it finds such module, it queries the Koji to find out the list
of arches to set for the module.
The "privileged" module is a module which can override base module arches
or disttag. Previously, these modules have been defined by
`allowed_disttag_marking_module_names` config option. In this commit,
this has been renamed to `allowed_privileged_module_names`.
The list of arches are stored per module build in new table represented
by ModuleArch class and are m:n mapped to ModuleBuild.
In certain use-cases, a module's buildrequires may remain the same
in the modulemd, but a different support stream of the buildrequired
base module should be used. For example, since RHEL 8.0.0 is GA, any
modules that buildrequire platform:el8.0.0 should buildrequire
platform:el8.0.0z instead.
This also moves the methods load_mmd and load_mmd_file to
module_build_service.utils.general.
This also removes some MSE unit tests with a mix of positive and
negative streams since this is not supported in libmodulemd v2. The
user will be presented with a syntax error if they try to submit
such a modulemd file.
GenericResolver.extract_modulemd is not removed, but deprecated. Call of it
will result in a deprecation message printed. Any new code should call
load_mmd.
Signed-off-by: Chenxiong Qi <cqi@redhat.com>
These module builds will basically act as metadata-only module builds.
This will be more useful as additional features stem from these types
of builds.
Before this commit, the compatible base modules for Module Stream Expansion
have been found without any limitation, just based on the stream version.
It was therefore possible that `platform:lp29.0.0` was found as compatible
module for `platform:f29.1.0` although those platform streams are not
compatible at all.
In this commit, the module can be treated as compatible only if it has
the same virtual stream as the input module. The idea behind this
is that both `platform:f29.0.0` and `platform:f29.1.0` should include
the `virtual_streams: [f29]` in their XMD section which tells MBS
that they are actually compatible. The `lp29` stream will not
have the same virtual stream (most likely it won't have any virtual
stream at all).
The `virtual_streams` is already used for this use-case in `MMDResolver`,
but it was not used to limit the inputs to `MMDResolver` which is what
this commit is doing.
This commit also fixes the issue in `get_last_builds_in_stream_version_lte`
which was simply broken if multiple stream_versions of single base module
existed and their builds had different version. In this case, only
builds with single (randomly chosen) version were returned.
- Keep scratch module builds in the 'done' state.
- Make koji tagging for scratch modules unique so the same
commit can be resubmitted.
- Use alternate prefix for scratch module build components so they can
be identified later.
- Prevent scratch build components from being reused.
- Assorted code and comment cleanup.
Signed-off-by: Merlin Mathesius <mmathesi@redhat.com>
This is required for monitoring use-cases, where we can have a Kerberos principal for a
service account but no associated account in LDAP to check group membership.
This commit introduces new to_text_type helper method and calls it
for return value of all mmd.dumps() calls. That way, we always
end up with proper unicode string represntation on both python
major versions.
This commit also adds unicode character to description of all
the yaml files we use in the tests so we can be sure MBS can
handle unicode characters properly.
This might be temporary fix, depending on the result of discussion
at https://github.com/fedora-modularity/libmodulemd/issues/184.
This commit includes the backport of the changes to `krb_login` in
https://pagure.io/koji/pull-request/1187. This change is required
for our separate threads to use a separate Kerberos context per thread.
* Fix failure test after rebasing on master branch.
* Fix some grammar issues.
* Only check stream collision modules on new created module build.
* Logging message properly.
Signed-off-by: Chenxiong Qi <cqi@redhat.com>
This resolve the stream collision by adding specific RPMs to
module-build-macros SRPM as Conflicts.
For more information about module stream collision, please refer to
docstring in utils/ursine.py
Signed-off-by: Chenxiong Qi <cqi@redhat.com>
ModuleBuild.json now includes xmd/mbs/buildrequires in the output JSON
data. As a result, these APIs will show buildrequires by default.
* /module-builds/
* /module-builds/?verbose=True
* /module-builds/$build_id
Buildrequires is accessible like this:
br_module_names = response_json['buildrequires'].keys()
br_module_dict = response_json['buildrequires'][module name]
Fixes: FACTORY-2201
Signed-off-by: Chenxiong Qi <cqi@redhat.com>
This will allow for tooling to automatically override certain buildrequires and requires
based on the branch name the modulemd is built form.
Addresses FACTORY-3414
