PR #1331 made the assumption that the Ursa Major ursine RPMs were at
xmd["mbs"]["ursine_rpms"], but they are actually at
xmd["mbs"]["buildrequires"]["platform"]["ursine_rpms"]. This commit
handles the ursine RPMs generated by handle_collisions_with_base_module_rpms
separately since the base module the RPMs came from are not tracked in
that function.
Update test test_build_module_locally_set_stream accordingly by not
mocking database session object in order to test against the real
SQLAlchemy Session scope.
Resolves: RHBZ#1752075
Signed-off-by: Chenxiong Qi <cqi@redhat.com>
Currently, we generate `Conflicts` for ursine RPMs conflicting with
modular RPMs only when Ursa Prime is used for the base module. This
commit changes it, so these Conflicts are generated everytime.
The reason is that modular RPMs should always be preferred in the
buildroot over the ursine RPMs no matter what is their NVR. So far,
this has been guarded on Koji side by using external repos, but
we need to move away from external repo or at least use "bare"
merge mode which basically means we won't get this feature for free
from Koji.
The reason why we need to move away from external repos or use "bare"
merge mode is that without this, the Koji removes RPMs sharing the same
name but different version/release from the buildroot and only keeps
the latest one. This is an issue in situation when you need two
versions of single RPM in a buildroot comming from two modules.
This removes support for default_modules_url in the Platform XMD and
gets the list of default name:stream combinations to include in the buildroot
from https://pagure.io/releng/fedora-module-defaults.
Addresses #1402
Before this commit, the base modules used in the `get_reusable_module` have
not been sorted and therefore when `get_reusable_module` tried to find out
the reusable module built against some base module, it could find a module
built against some old version of base module.
This could lead to situation when MBS tried to reuse components from quite
old module despite the fact that newer module build existed.
This commit fixes this by sorting the base modules by stream_version,
so MBS always tries to get the reusable module built against the latest
base module.
The `MBSResolver.get_buildrequired_modulemds` did not try to load
local module builds and always just queried the remote MBS instance.
This commit fixes it by using local module if available.
The goal here is to define certain branches from which only scratch
module builds can be submitted. The main use case is for "private-*"
branches which can be created and maintained by anyone, but there
must not be production-ready module build created from them.
This commit adds new `scratch_build_only_branches` config option
to define the list of regexes to match such branches.
* xmd/mbs is always set if it is not present in xmd, so move the code on
the top of function. This change is also helpful for accessing keys
under xmd/mbs.
* By setting xmd/mbs in the beginning, code is simplified to get
disttag_marking and virtual_streams. The result is much straightforwar
for getting a default value for them.
* Move disttag_marking validation code next to the line getting
disttag_marking from xmd/mbs. As a result, the code structure is
easier to read as getting disttag_marking and validate it, getting
virtual_streams and validate.
* Rewrite the part of code for check_buildrequires. Always set
xmd/mbs/buildrequires if it is not present and check_buildrequires is
set to True, as it is required by
ModuleBuild.get_buildrequired_base_modules.
* Using in operator instead of dict.get to check if key koji_tag exists.
Using dict.get would be ambiguous because even if koji_tag exists
under xmd/mbs, but due to its value is set to None occasionally, there
is still a message logged to tell koji_tag is not set.
* Rwrite all lines of code for updating virtual streams. A new method
update_virtual_streams is added to ModuleBuild. This also fixes
FACTORY-4561.
* Tests are added for the rewrite of virtual streams update.
Signed-off-by: Chenxiong Qi <cqi@redhat.com>
The issue is that users don't get feedback from MBS about why a
component was not reused. There was added logic which enables to
store log messages in the database and can be viewed through the
REST api of MBS.
Ticket-ID: #1284
Signed-off-by: Martin Curlej <mcurlej@redhat.com>
When recover_orphaned_artifact is called for module-build-macros
and the module-build-macros is not tagged in the -build Koji tag,
then the tag_artifacts() is called to tag it there.
This is correct, but the issue is that module-build-macros need
to be added to "build" and "srpm-build" Koji tag groups, otherwise
it is not installed in the buildroot by default.
The original motivation for this refactor is to reuse make_module and
drop TestMMDResolver._make_mmd. Some tests require a modulemd created
and some tests also require those modulemd to be stored into database as
a module build. The problem is db_session has to be passed to
make_module even if no need to store into database.
Major changes in this patch:
* Argument db_session is optional.
* Arguments requires_list and build_requires_list are replaced by a
single argument dependencies which is a list of group of requires and
buildrequires
* A new make_module_in_db is created for creating and storing the new
modulemd into database conveniently.
* Tests are updated with the new make_module and make_module_in_db.
Signed-off-by: Chenxiong Qi <cqi@redhat.com>
When resubmitting a module build, if some component is found out that
its attributes have changed, MBS will raise an error to stop the work to
recording components. The problem is original code tells a module build
exists in database already rather than a component build, meanwhile
get_module_name() call on an ComponentRpm object is also incorrect.
Signed-off-by: Chenxiong Qi <cqi@redhat.com>
This patch separates the use of database session in different MBS components
and do not mix them together.
In general, MBS components could be separated as the REST API (implemented
based on Flask) and non-REST API including the backend build workflow
(implemented as a fedmsg consumer on top of fedmsg-hub and running
independently) and library shared by them. As a result, there are two kind of
database session used in MBS, one is created and managed by Flask-SQLAlchemy,
and another one is created from SQLAclhemy Session API directly. The goal of
this patch is to make ensure session object is used properly in the right
place.
All the changes follow these rules:
* REST API related code uses the session object db.session created and
managed by Flask-SQLAlchemy.
* Non-REST API related code uses the session object created with SQLAlchemy
Session API. Function make_db_session does that.
* Shared code does not created a new session object as much as possible.
Instead, it accepts an argument db_session.
The first two rules are applicable to tests as well.
Major changes:
* Switch tests back to run with a file-based SQLite database.
* make_session is renamed to make_db_session and SQLAlchemy connection pool
options are applied for PostgreSQL backend.
* Frontend Flask related code uses db.session
* Shared code by REST API and backend build workflow accepts SQLAlchemy session
object as an argument. For example, resolver class is constructed with a
database session, and some functions accepts an argument for database session.
* Build workflow related code use session object returned from make_db_session
and ensure db.session is not used.
* Only tests for views use db.session, and other tests use db_session fixture
to access database.
* All argument name session, that is for database access, are renamed to
db_session.
* Functions model_tests_init_data, reuse_component_init_data and
reuse_shared_userspace_init_data, which creates fixture data for
tests, are converted into pytest fixtures from original function
called inside setup_method or a test method. The reason of this
conversion is to use fixture ``db_session`` rather than create a
new one. That would also benefit the whole test suite to reduce the
number of SQLAlchemy session objects.
Signed-off-by: Chenxiong Qi <cqi@redhat.com>
This just simplifies the code. It doesn't change the functionality
at all. The tests had to be changed because it assumed that the
xmd.mbs.buildrequires section would be filled out for the input
module which isn't true.
When using default modules, this feature will add conflicts to
module-build-macros for every RPM in a buildrequired base module
that overlaps with RPMs in the buildrequired modules. This will
prevent them from being available in the buildroot, and thus
ensure that the RPMs from the buildrequired modules (non-base
modules) are used even if they have a lower NVR.
