* Fix failure test after rebasing on master branch.
* Fix some grammar issues.
* Only check stream collision modules on new created module build.
* Logging message properly.
Signed-off-by: Chenxiong Qi <cqi@redhat.com>
_record_ursine_rpms needs to get each collision module's koji_tag and
then get built RPMs from that koji_tag eventually. _get_module on each
individual resolver is called to get module metadata accordingly. So,
when running a local build, module metadata is got from remote MBS, and
for a MBS instance, connected database is queried.
Signed-off-by: Chenxiong Qi <cqi@redhat.com>
For tracking error easily when custom exception is raised, log exception
as well before transitioning to failed state.
Signed-off-by: Chenxiong Qi <cqi@redhat.com>
Not both Fedora and internal Brew uses config topurl as the external
repo's URL prefix. Hence, make it configurable to fulfill this
difference.
Signed-off-by: Chenxiong Qi <cqi@redhat.com>
This resolve the stream collision by adding specific RPMs to
module-build-macros SRPM as Conflicts.
For more information about module stream collision, please refer to
docstring in utils/ursine.py
Signed-off-by: Chenxiong Qi <cqi@redhat.com>
If an RPM is not included, its correspnding -debug* RPMs should also not
be included.
Also ensure that source RPMs are only ever added to -devel modules if
a binary RPM has been completely excluded from non-devel module.
Internal ref: FACTORY-3263
Signed-off-by: Luiz Carvalho <lucarval@redhat.com>
There is a need to ship the unchanged source file which was used to build
a module build from legal reasons. The KojiContentGenerator is extended
in this commit to fetch it from SCM URL using the `scm` module and later
attach it as `modulemd.src.txt` to Koji CG build.
This commit introduces KojiContentGenerator._sanitize_mmd method to:
- remove `repository` and `cache` from ComponentRPM in Modulemd.
- remove `mbs` section from `xmd`.
This is done to not leak internal build-only information to final
modulemd files.
The value referenced by variable dependencies is a list of koji_tags.
So, it would be more straightforward to use an alternative name like
dep_koji_tags than dependencies which is too general.
Signed-off-by: Chenxiong Qi <cqi@redhat.com>
Both tag and dependencies are set directly by calling private method
_get_deps_and_tag. There is no need to set initial value for them
separately.
Signed-off-by: Chenxiong Qi <cqi@redhat.com>
git-ls-remote accepts paramter refs to just return refs for specific
ones. This refactor uses this parameter to avoid handling full list of
refs from remote repository.
In original code, option --exit-code is passed to git-ls-remote, which
would cause command return exit code 2 when no ref is found from remote
repository. SCM._run raises an error if that happens, but the raised
error is not handled. This refactor catches this error to ensure the
original behavior happens, that is if a ref is not found, treat it as a
commit hash and call SCM.get_full_commit_hash.
Signed-off-by: Chenxiong Qi <cqi@redhat.com>
For an explanation of transitive stream collision, please refer to the
source code docstring.
When such collision is detected, error is raised with proper error
message.
Signed-off-by: Chenxiong Qi <cqi@redhat.com>
For some modules, modularity-wg wants to be able to ship the "-devel" modules containing
the RPMs which are normally filtered out from the module.
This PR generates second Koji CG module with -devel suffix in a name with final modulemd
files containing the filtered out RPMs.
This will allow for tooling to automatically override certain buildrequires and requires
based on the branch name the modulemd is built form.
Addresses FACTORY-3414
Imagine we have "platform:f29.0.0" and "platform:f29.1.0" base modules.
We also have "DBI" module we want to build agaisnt "platform:f29.1.0".
This "DBI" module depends on "perl" module which is only build against
"platform:f29.0.0".
Currently, DBI build would fail to resolve the dependencies, because
it wouldn't find "perl" module, because it is built against different
platform stream.
This PR changes the MSE code to include buildrequired module builds built
against all the compatible platform streams.
It does so by introducing following changes:
- MSE code uses new get_base_module_mmds() method to find out all the
compatible platform modules. This needed new methods in DBResolver
and MBSResolver.
- For each buildrequired module defined by name:stream, the MSE code then
finds particular NSVC built against each compatible platform module.
Side effect of these code changes is that every module now must buildrequire
some base module.
The test tests installation of multiple streams of single module.
For example:
- "app:1" requires "foo:1" and "gtk:1".
- "foo:1" requires "bar:1".
- "gtk:1" requires "bar:2".
"bar:1" and "bar:2" cannot be installed in the same time and therefore
there need to be conflict defined between them.
submit_module_build creates new module build and set parameter url to
ModuleBuild.scmurl. It looks scm.url could be passed to scmurl as well,
but scm is not used through out whole method. url is enough for the
purpose, hence scm is removed.
Signed-off-by: Chenxiong Qi <cqi@redhat.com>
_get_mmds_from_requires does not do anything against db.session. Hence,
remove parameter session from it and
get_mmds_required_by_module_recursively.
Signed-off-by: Chenxiong Qi <cqi@redhat.com>
Modularity team needs to be able to define multiple streams for single "platform"
module. They need it to express that "platform:el8.1.0" also provides "platform:el8".
This needs changes in a way how MMDResolver resolves dependencies between modules.
For example, if we are building module against platform:el8.1.0, the MMDResolver must
not return buildrequired module built against platform > el8.1.0, but it can for example
return (cherry-pick) buildrequired module from platform:el8.0.0 if there is no such
module built for platform:el8.1.0.
The way how it is implemented is following:
- MMDResolver reads list of virtual streams from xmd["mbs"]["virtual_streams"] when
creating Solvable from MMD and adds additional Provides for these virtual streams.
We expect these to be set mainly on base modules. The versions of such provides
are based on "stream version" number, so we can compare them.
- The base module ("platform") buildrequires of MMDs added to MMDResolver are overriden
to mark particular platform "stream version". For example, if module "foo" buildrequires
"platform:el8" in MMD file, but was in fact built against platform:el8.1.0, it will
be treated as if it would really buildrequire "platform:el8.1.0". This is needed
to not include buildrequired modules built against newer platform than what we want.
- MMDResolver resolves all the valid combinations of buildrequires, but we are only
interested in the one with latest versions of buildrequired module. Therefore the
solve() method is changed to find out combinations which have the latest versions
for each alternative name:stream buildrequires.
Future use cases will require the ability to find compatible module builds
to buildrequire based on the base module the module used to build. This
commit adds an association table that will contain module build IDs
and the base module they buildrequire.
Addresses FACTORY-3353
