Commit Graph

87 Commits

Author SHA1 Message Date
Chenxiong Qi
db4dbe3b25 Remove unused parameter session
_get_mmds_from_requires does not do anything against db.session. Hence,
remove parameter session from it and
get_mmds_required_by_module_recursively.

Signed-off-by: Chenxiong Qi <cqi@redhat.com>
2018-10-22 11:48:04 +08:00
Chenxiong Qi
20de3d2e94 Make _make_module reusable
TestUtilsModuleStreamExpansion._make_module is moved to
tests/__init__.py so that other tests could reuse this to create a
module build.

Signed-off-by: Chenxiong Qi <cqi@redhat.com>
2018-10-19 15:58:47 +08:00
Jan Kaluza
192eaae012 Support Virtual Streams in MMDResolver.
Modularity team needs to be able to define multiple streams for single "platform"
module. They need it to express that "platform:el8.1.0" also provides "platform:el8".

This needs changes in a way how MMDResolver resolves dependencies between modules.

For example, if we are building module against platform:el8.1.0, the MMDResolver must
not return buildrequired module built against platform > el8.1.0, but it can for example
return (cherry-pick) buildrequired module from platform:el8.0.0 if there is no such
module built for platform:el8.1.0.

The way how it is implemented is following:

  - MMDResolver reads list of virtual streams from xmd["mbs"]["virtual_streams"] when
    creating Solvable from MMD and adds additional Provides for these virtual streams.
    We expect these to be set mainly on base modules. The versions of such provides
    are based on "stream version" number, so we can compare them.
  - The base module ("platform") buildrequires of MMDs added to MMDResolver are overriden
    to mark particular platform "stream version". For example, if module "foo" buildrequires
    "platform:el8" in MMD file, but was in fact built against platform:el8.1.0, it will
    be treated as if it would really buildrequire "platform:el8.1.0". This is needed
    to not include buildrequired modules built against newer platform than what we want.
  - MMDResolver resolves all the valid combinations of buildrequires, but we are only
    interested in the one with latest versions of buildrequired module. Therefore the
    solve() method is changed to find out combinations which have the latest versions
    for each alternative name:stream buildrequires.
2018-10-18 15:06:04 +02:00
Valerij Maljulin
8ee7168017 get_reusable_component now checks the architecture
Signed-off-by: Valerij Maljulin <vmaljuli@redhat.com>
2018-10-16 13:44:18 +02:00
Valerij Maljulin
7f60db545a renaming koji_arches to arches
Signed-off-by: Valerij Maljulin <vmaljuli@redhat.com>
2018-10-15 15:29:50 +02:00
Valerij Maljulin
924a0951fb add architectures support for format_mmd 2018-10-15 15:29:50 +02:00
mprahl
117791ee16 Prefix the component disttag with the platform stream 2018-10-04 07:19:07 -04:00
mprahl
cfb75b4d0f Prefix the module version based on the platform it buildrequires 2018-10-04 07:19:07 -04:00
Ralph Bean
0aff640ef3 Optionally, block building modules with EOL streams.
Requested by @mboddu in:

- https://pagure.io/fm-orchestrator/issue/960
- https://pagure.io/modularity/issue/102
2018-08-31 15:23:04 -04:00
Jan Kaluza
dad03d3742 Copy the buildrequires which are not in requires list to expanded MMD. 2018-08-28 15:15:58 +02:00
mprahl
5197eb4912 Revert "Use the libmodulemd API properly when setting values on existing objects"
This reverts commit 8173040ea6. Since it didn't
provide any real benefit after talking with sgallagh, it seems like we'd better
off not introduce more risk to the next deployment.
2018-08-07 12:03:44 -04:00
mprahl
8173040ea6 Use the libmodulemd API properly when setting values on existing objects
MBS has been using the libmodulemd API incorrectly by assuming that
methods like `get_rpm_components` return the actual object in memory
and not a copy. This has been true but wasn't something sgallagh
intended. We found this out after he had me test his new version of
libmodulemd. This PR removes those assumptions.
2018-07-17 09:00:42 -04:00
Ralph Bean
6981449853 Fix local builds.
I discovered that local builds have been broken by recent (good) changes in
dnf.  At the end of every batch, we regenerate the local repo with createrepo
and we also call modifyrepo to include the modulemd file as we progress.  This
was added in #467.

What we really want, is for the modulemd file to be present at the *end* of the
build.

Recently, dnf started respecting the modulemd file natively, such that builds
we built in batch0 would not show up in batch1.  They would be present in the
repo, but they would be marked as belonging to a module which was not enabled,
and so could not be pulled in.  Every module build would fail because
module-srpm-macros was in a disabled module (the module being built at the
time).

This change makes it so that the module metadata is only added to the repo at
the very end of the build.  I moved it into a `finalize` method on the builder
which the copr builder was using, and for symmetry's sake I moved the koji
content generator code to the same method on that builder.

There was a circular import issue to solve between the koji module builder and
the koji content generator modules that generated a larger diff, but is mostly
cosmetic and mock changes.
2018-06-25 17:05:22 -04:00
Mikolaj Izdebski
a22f663d4a Add reproducer for #954 2018-06-18 14:17:13 +02:00
Qixiang Wan
d83e6897ca Change ModuleBuild.context to db column
1. Changed ModuleBuild's context property to db column, it's
non-nullable and default value is '00000000' to keep it consistent
with previous behaviour.

2. Changed ModuleBuild.contexts_from_mmd to return a tuple of
(ref_build_context, build_context, runtime_context, context).

3. Updated tests affected by this change.
2018-04-28 11:46:31 +08:00
Qixiang Wan
8b807a9fcd Generate koji tag from MBS and use informative name
koji now supports tags with max length of 256, we can use
more informative tag name instead of the hash one.

The new format of koji tag name is:

    module-<name>-<stream>-<version>-<context>

However when the generated tag's length > (256 - len('build')), we
fallback to the old way of name in hash format (module-<hash>).

In this change, koji tag is always generated from MBS itself, even
with pdc resolver.

FIXES: #918 #925
2018-04-26 22:04:16 +08:00
Matt Prahl
8351713828 Merge #922 Generate 'context' from hash based on buildrequires/requires stream. Reuse components only from module with the same stream_build_context. 2018-04-24 22:50:02 +00:00
Jan Kaluza
8d1d439737 Generate 'context' from hash based on buildrequires/requires stream. Reuse components only from module with the same stream_build_context. 2018-04-24 15:45:26 +02:00
Jan Kaluza
836fb9db10 Keep the 'module_name:[]' in requires unexpanded in generate_expanded_mmds. 2018-04-06 10:48:59 +02:00
mprahl
c2824138d3 Fix a bug in MSE when a module requires a module that isn't also a buildrequire 2018-04-05 10:58:29 -04:00
Jan Kaluza
65513afd1a Share the same in the dist tag between all MSE builds. 2018-04-05 10:42:56 +02:00
Jan Kaluza
9b6fd2ba39 Split utils.py to multiple files based on the methods use-case. 2018-04-03 09:58:57 -04:00
Jan Kaluza
9503e4624e Support local build of modules with MSE. 2018-04-03 09:58:57 -04:00
Jan Kaluza
e5e9a24fce Allow adding bool to XMD. Fix flake8 issues. 2018-04-03 09:58:57 -04:00
Jan Kaluza
2482e47c07 Fix issues from review. Drop xmd['mbs']['requires']. Do not allow building MSE builds against non-MSE builds. 2018-04-03 09:58:57 -04:00
Jan Kaluza
31ddbe51a5 Add generate_expanded_mmds which returns list of MMDs which can be submitted as MSE builds and submit them. 2018-04-03 09:58:57 -04:00
mprahl
eb0b2e1c38 Add the database resolver plugin 2018-04-03 09:58:57 -04:00
Jan Kaluza
253d29bcef Rewrite ModuleBuild methods to get all streams and all builds using the latest version, not build id. 2018-04-03 09:58:57 -04:00
Jan Kaluza
e22cbbab31 Add get_modules_build_required_by_module_recursively to get the input for libsolv solver. 2018-04-03 09:58:57 -04:00
mprahl
f0852d9009 Port to libmodulemd and support v2 modulemd without module stream expansion 2018-04-03 09:58:57 -04:00
Yash Nanavati
ed25afd9ec Code changes for Python 3 support 2018-03-04 03:22:38 -05:00
mprahl
f5530d9b16 Fix flake8 errors made by jkaluza 2018-02-09 09:12:00 -05:00
Jan Kaluza
6a5d7267fd Save another 20s in tests. 2018-02-09 11:46:12 +01:00
mprahl
8f024e6b04 Remove the dependency on vcrpy in unit tests and modernize some of the test data 2018-02-05 22:17:18 -05:00
mprahl
e820fe655c Remove duplicate test that tests a module resubmission 2018-02-05 22:11:31 -05:00
mprahl
e91d09f7ca Change the format of the unit tests to pytest 2018-01-31 16:34:21 -05:00
Ralph Bean
01c8295269 Add an incrementing prefix to the dist_hash.
For #823.
2018-01-17 16:54:02 -05:00
mprahl
430ac4ba31 Use the "build_context" field when evaluating if a previous module's buildrequires changed 2018-01-16 10:36:09 -05:00
mprahl
a67b53f5c3 Add a "context" field on component and module releases in Koji for uniqueness for when Module Stream Expansion is implemented 2018-01-16 10:36:09 -05:00
mprahl
c1dbe9be27 Remove unused yaml argument in utils.submit_module_build 2017-12-11 14:14:39 -05:00
Jan Kaluza
7de120c17b Get the weight of Koji tasks and use it to sort the components in a batch instead of builds time. 2017-11-28 10:52:17 -05:00
mprahl
30ef85af01 Fix "instant complete" builds and reusing all components 2017-11-27 16:20:28 -05:00
mprahl
78fea1678b Flake8 fix 2017-11-27 16:20:11 -05:00
Martin Curlej
a1fd9daa73 Moved skiptests cli option 2017-11-21 09:07:32 +01:00
Matt Prahl
ea26e0281a Merge #788 Fix bug and clean up scm.get_latest 2017-11-14 13:38:23 +00:00
mprahl
1622494b17 Replace the name of the "branch" parameter in "scm.get_latest" to "ref" 2017-11-13 09:20:20 -05:00
Ralph Bean
8431d2698b Explicit imports of builders in the test suite. 2017-11-10 16:03:14 -05:00
mprahl
d8f126e978 Allow resuming from a failure during the "init" phase
This also shifts build failure transitions from individual functions
in utils.py to the init module handler.
2017-11-06 14:15:12 -05:00
mprahl
58d160cce4 Transition a failed module to wait instead of init upon resuming 2017-11-03 15:12:39 -04:00
mprahl
45cce653eb Add the ability to specify different rebuild methods 2017-11-01 15:39:43 -04:00