PR #1331 made the assumption that the Ursa Major ursine RPMs were at
xmd["mbs"]["ursine_rpms"], but they are actually at
xmd["mbs"]["buildrequires"]["platform"]["ursine_rpms"]. This commit
handles the ursine RPMs generated by handle_collisions_with_base_module_rpms
separately since the base module the RPMs came from are not tracked in
that function.
This patch attemps to make code shorter and easier to read by naming
component builds for different states.
Signed-off-by: Chenxiong Qi <cqi@redhat.com>
Currently, we generate `Conflicts` for ursine RPMs conflicting with
modular RPMs only when Ursa Prime is used for the base module. This
commit changes it, so these Conflicts are generated everytime.
The reason is that modular RPMs should always be preferred in the
buildroot over the ursine RPMs no matter what is their NVR. So far,
this has been guarded on Koji side by using external repos, but
we need to move away from external repo or at least use "bare"
merge mode which basically means we won't get this feature for free
from Koji.
The reason why we need to move away from external repos or use "bare"
merge mode is that without this, the Koji removes RPMs sharing the same
name but different version/release from the buildroot and only keeps
the latest one. This is an issue in situation when you need two
versions of single RPM in a buildroot comming from two modules.
This removes support for default_modules_url in the Platform XMD and
gets the list of default name:stream combinations to include in the buildroot
from https://pagure.io/releng/fedora-module-defaults.
Addresses #1402
A full modulemd is usually a loooon text. This change would be much
easier for reading logs at INFO level only.
Signed-off-by: Chenxiong Qi <cqi@redhat.com>
This patch separates the use of database session in different MBS components
and do not mix them together.
In general, MBS components could be separated as the REST API (implemented
based on Flask) and non-REST API including the backend build workflow
(implemented as a fedmsg consumer on top of fedmsg-hub and running
independently) and library shared by them. As a result, there are two kind of
database session used in MBS, one is created and managed by Flask-SQLAlchemy,
and another one is created from SQLAclhemy Session API directly. The goal of
this patch is to make ensure session object is used properly in the right
place.
All the changes follow these rules:
* REST API related code uses the session object db.session created and
managed by Flask-SQLAlchemy.
* Non-REST API related code uses the session object created with SQLAlchemy
Session API. Function make_db_session does that.
* Shared code does not created a new session object as much as possible.
Instead, it accepts an argument db_session.
The first two rules are applicable to tests as well.
Major changes:
* Switch tests back to run with a file-based SQLite database.
* make_session is renamed to make_db_session and SQLAlchemy connection pool
options are applied for PostgreSQL backend.
* Frontend Flask related code uses db.session
* Shared code by REST API and backend build workflow accepts SQLAlchemy session
object as an argument. For example, resolver class is constructed with a
database session, and some functions accepts an argument for database session.
* Build workflow related code use session object returned from make_db_session
and ensure db.session is not used.
* Only tests for views use db.session, and other tests use db_session fixture
to access database.
* All argument name session, that is for database access, are renamed to
db_session.
* Functions model_tests_init_data, reuse_component_init_data and
reuse_shared_userspace_init_data, which creates fixture data for
tests, are converted into pytest fixtures from original function
called inside setup_method or a test method. The reason of this
conversion is to use fixture ``db_session`` rather than create a
new one. That would also benefit the whole test suite to reduce the
number of SQLAlchemy session objects.
Signed-off-by: Chenxiong Qi <cqi@redhat.com>
When using default modules, this feature will add conflicts to
module-build-macros for every RPM in a buildrequired base module
that overlaps with RPMs in the buildrequired modules. This will
prevent them from being available in the buildroot, and thus
ensure that the RPMs from the buildrequired modules (non-base
modules) are used even if they have a lower NVR.
A base module can set xmd.mbs.default_modules_url, which contains a
URL to a list of modules in the format of name:stream separated by
new lines. When a module buildrequires this base module, the list
of default modules are added as buildrequires of the module automatically
unless there are conflicting streams or the default module is not
in the MBS database.
Currently, we are using just `conf.arches` and `conf.base_module_arches`
to define the list of arches for which the RPMs in a submitted module are
built. This is not enough, because RCM needs to generate modules based
on the base modules which should use different arches.
This commit changes the MBS to take the list of arches from the buildrequired
module build. It checks the buildrequires for "privileged" module or base
module and if it finds such module, it queries the Koji to find out the list
of arches to set for the module.
The "privileged" module is a module which can override base module arches
or disttag. Previously, these modules have been defined by
`allowed_disttag_marking_module_names` config option. In this commit,
this has been renamed to `allowed_privileged_module_names`.
The list of arches are stored per module build in new table represented
by ModuleArch class and are m:n mapped to ModuleBuild.
This also moves the methods load_mmd and load_mmd_file to
module_build_service.utils.general.
This also removes some MSE unit tests with a mix of positive and
negative streams since this is not supported in libmodulemd v2. The
user will be presented with a syntax error if they try to submit
such a modulemd file.
Previously MockModuleBuilder was checking the module state to see if
it should run a final createrepo, but since eafa93037f, finalize() is
called before changing the module state; add an explicit boolean to
GenericBuilder.finalize() to avoid worrying about ordering.
There are following changes introduced in this commit:
- The `koji_tag` of module builds imported from the local repositories
is now in `repofile:///etc/yum.repos.d/some.repo` format to store the
repository from which the module was imported to local MBS DB.
- The `koji_tag` of fake base module is set to empty `repofile://`
and in `MockModuleBuilder` the `conf.base_module_repofiles` list
is used as source for the repositories defining platform. We can't
simply use single repository, because there might be fedora.repo
and fedora-update.repo and so on.
- The list of default .repo files for platform are passed using the
`-r` switch in `build_module_locally` `mbs-manager` command.
- The LocalResolver (subclass of DBResolver) is added which is used
to resolve the build dependencies when building modules offline
locally.
- The `MockModuleBuilder` enables the buildrequired modules and
repositories from which they come in the mock config.
With this commit, it is possible to build testmodule locally
without any external infra.
The `KojiContentGenerator.finalize()` needs the `ModuleBuild.time_completed`.
We currently set `time_completed` once the module build transitions
into `done` state. But we have moved the `KojiContentGenerator` call
to end of `build` state, so right now it is called before the `time_completed`
is set. This leads to traceback.
In this commit, the `time_completed` is set before the `KojiContentGenerator`
call, so it is defined properly.
- Keep scratch module builds in the 'done' state.
- Make koji tagging for scratch modules unique so the same
commit can be resubmitted.
- Use alternate prefix for scratch module build components so they can
be identified later.
- Prevent scratch build components from being reused.
- Assorted code and comment cleanup.
Signed-off-by: Merlin Mathesius <mmathesi@redhat.com>
We encountered an issue where start_next_batch_build was called twice on a
module build and it caused the module build to fail because Brew failed
one of the components since it was submitted twice. This will help us narrow
down the issue if it happens again.
This commit introduces new to_text_type helper method and calls it
for return value of all mmd.dumps() calls. That way, we always
end up with proper unicode string represntation on both python
major versions.
This commit also adds unicode character to description of all
the yaml files we use in the tests so we can be sure MBS can
handle unicode characters properly.
This might be temporary fix, depending on the result of discussion
at https://github.com/fedora-modularity/libmodulemd/issues/184.
Adds new meethod checking the "tagged" and "tagged_in_final" attributes of
"complete" ComponentBuilds in the current batch of module builds
in "building" state against the Koji.
In case the Koji shows the build as tagged/tagged_in_final,
the fake "tagged" message is added to work queue.
Client tool like module-build-watch and `module-build --watch' could
output failed reason. However, original message "Some components failed
to buld" is not informative enough. Instead, list package names of
failed builds would be helpful.
Signed-off-by: Chenxiong Qi <cqi@redhat.com>
This fixes issues when UMB message delivery from frontend to backend fails
for whatever reason...
We do the same thing for 'wait' state already, so this commit just extends
it to 'init' state too.
