When importing a base module, we must ensure the value that will be
used in the RPM disttags doesn't contain a dash since a dash isn't
allowed in the release field of the NVR.
MBS uses the base module's stream that was buildrequired by the module
in the RPM disttags for that module build. The stream name may not be
ideal for all situations, so now this is customizable by setting the
xmd['mbs']['disttag_marking'] in the base module's modulemd.
- Keep scratch module builds in the 'done' state.
- Make koji tagging for scratch modules unique so the same
commit can be resubmitted.
- Use alternate prefix for scratch module build components so they can
be identified later.
- Prevent scratch build components from being reused.
- Assorted code and comment cleanup.
Signed-off-by: Merlin Mathesius <mmathesi@redhat.com>
In case there is lot of components in a module build or there are some
networking issues and we need to retry the "git clone" commands, the
`format_mmd` method can take long time.
If it takes more than 10 minutes, the poller can produce fake event,
because it seems the module build is stuck. This is wrong, because
it would lead to another unexpected init handler call.
In this commit, the `format_mmd` updates the `ModuleBuild.time_modified`
method regularly to prevent poller from sending the unexpected fake
init message.
This commit introduces new to_text_type helper method and calls it
for return value of all mmd.dumps() calls. That way, we always
end up with proper unicode string represntation on both python
major versions.
This commit also adds unicode character to description of all
the yaml files we use in the tests so we can be sure MBS can
handle unicode characters properly.
This might be temporary fix, depending on the result of discussion
at https://github.com/fedora-modularity/libmodulemd/issues/184.
We always set the "context" to DEFAULT_MODULE_CONTEXT from the historical
reasons - the context was not stored in the database before and we stored
just the build_context/runtime_context. But this is no longer true for
some time.
In this commit, the context is respected and stored in the database when
importing module using the `import_mmd` method. If the context is not set
in the imported MMD, the DEFAULT_MODULE_CONTEXT is used.
This is needed for Fedora branching, but it is generally useful.
For example, there is a module buildrequiring "platform:[]". In the time
this module has been built, only platform:f29 existed, so it has been built
just against platform:f29.
After a while, the platform:f30 is released and the maintainer needs
to rebuild the module against platform:f30. Right now, he needs to create
new commit in the module and submit the build, but this will result in useless
rebuild of the module also against platform:f29.
In this commit, MBS allows to resubmit the module build in a case
there are new MSE builds to build. MBS will send all the module builds
back to the user - so the existing builds will be already marked as
"ready" and the newly submitted builds will have the "init" state in
the REST API response.
However, in case when there are no new MSE builds to build, MBS still
sends back the Conflict error as it used to. This is done for backwards
compatibility and also to not confuse the users in case when no new build
has been submitted.
Due to a logic error, if record_filtered_rpms() was called on a module which
already had filtered_rpms set, the buildrequires xmd information was discarded.
This broke all local builds.
MBS calls some read-only Koji APIs which does not require to log into a
session. This patch makes it optional to choose whether to login a
session and use anonymous session properly to call those read-only APIs.
Signed-off-by: Chenxiong Qi <cqi@redhat.com>
* Fix failure test after rebasing on master branch.
* Fix some grammar issues.
* Only check stream collision modules on new created module build.
* Logging message properly.
Signed-off-by: Chenxiong Qi <cqi@redhat.com>
_record_ursine_rpms needs to get each collision module's koji_tag and
then get built RPMs from that koji_tag eventually. _get_module on each
individual resolver is called to get module metadata accordingly. So,
when running a local build, module metadata is got from remote MBS, and
for a MBS instance, connected database is queried.
Signed-off-by: Chenxiong Qi <cqi@redhat.com>
Not both Fedora and internal Brew uses config topurl as the external
repo's URL prefix. Hence, make it configurable to fulfill this
difference.
Signed-off-by: Chenxiong Qi <cqi@redhat.com>
This resolve the stream collision by adding specific RPMs to
module-build-macros SRPM as Conflicts.
For more information about module stream collision, please refer to
docstring in utils/ursine.py
Signed-off-by: Chenxiong Qi <cqi@redhat.com>
This will allow for tooling to automatically override certain buildrequires and requires
based on the branch name the modulemd is built form.
Addresses FACTORY-3414
Imagine we have "platform:f29.0.0" and "platform:f29.1.0" base modules.
We also have "DBI" module we want to build agaisnt "platform:f29.1.0".
This "DBI" module depends on "perl" module which is only build against
"platform:f29.0.0".
Currently, DBI build would fail to resolve the dependencies, because
it wouldn't find "perl" module, because it is built against different
platform stream.
This PR changes the MSE code to include buildrequired module builds built
against all the compatible platform streams.
It does so by introducing following changes:
- MSE code uses new get_base_module_mmds() method to find out all the
compatible platform modules. This needed new methods in DBResolver
and MBSResolver.
- For each buildrequired module defined by name:stream, the MSE code then
finds particular NSVC built against each compatible platform module.
Side effect of these code changes is that every module now must buildrequire
some base module.
submit_module_build creates new module build and set parameter url to
ModuleBuild.scmurl. It looks scm.url could be passed to scmurl as well,
but scm is not used through out whole method. url is enough for the
purpose, hence scm is removed.
Signed-off-by: Chenxiong Qi <cqi@redhat.com>
_get_mmds_from_requires does not do anything against db.session. Hence,
remove parameter session from it and
get_mmds_required_by_module_recursively.
Signed-off-by: Chenxiong Qi <cqi@redhat.com>
Modularity team needs to be able to define multiple streams for single "platform"
module. They need it to express that "platform:el8.1.0" also provides "platform:el8".
This needs changes in a way how MMDResolver resolves dependencies between modules.
For example, if we are building module against platform:el8.1.0, the MMDResolver must
not return buildrequired module built against platform > el8.1.0, but it can for example
return (cherry-pick) buildrequired module from platform:el8.0.0 if there is no such
module built for platform:el8.1.0.
The way how it is implemented is following:
- MMDResolver reads list of virtual streams from xmd["mbs"]["virtual_streams"] when
creating Solvable from MMD and adds additional Provides for these virtual streams.
We expect these to be set mainly on base modules. The versions of such provides
are based on "stream version" number, so we can compare them.
- The base module ("platform") buildrequires of MMDs added to MMDResolver are overriden
to mark particular platform "stream version". For example, if module "foo" buildrequires
"platform:el8" in MMD file, but was in fact built against platform:el8.1.0, it will
be treated as if it would really buildrequire "platform:el8.1.0". This is needed
to not include buildrequired modules built against newer platform than what we want.
- MMDResolver resolves all the valid combinations of buildrequires, but we are only
interested in the one with latest versions of buildrequired module. Therefore the
solve() method is changed to find out combinations which have the latest versions
for each alternative name:stream buildrequires.
Future use cases will require the ability to find compatible module builds
to buildrequire based on the base module the module used to build. This
commit adds an association table that will contain module build IDs
and the base module they buildrequire.
Addresses FACTORY-3353
system_resolver is created based on loaded configuration, which could
avoid calls like `GenericResolver.create(conf)` repeatedly in the code.
However, if some cases need to create a specific resolver explicitly,
`GenericResolver.create` could be called with addition argument, for
example db or mbs is passed to argument backend in tests.
Signed-off-by: Chenxiong Qi <cqi@redhat.com>
During I'm reading through the code base to learn MBS, I found out some
typos and minor issues in some other docstrings, and I also found some
docstrings with extra informative description could make it easier to
understand the code.
Signed-off-by: Chenxiong Qi <cqi@redhat.com>
This bug was reported by @nphilipp here https://bodhi.fedoraproject.org/updates/FEDORA-2018-c9c3a37d96
The issue is that the `record_filtered_rpms` method returns a new xmd
buildrequires block, but whole buildrequires entries were omitted in the return
value for local builds. This one-liner includes them again.
