Commit Graph

146 Commits

Author SHA1 Message Date
mprahl
9651891d74 Support base modules with x.y.z versioning and no virtual streams 2019-03-25 15:59:06 +00:00
Jan Kaluza
780ed117b4 Find compatible base modules based on the virtual streams.
Before this commit, the compatible base modules for Module Stream Expansion
have been found without any limitation, just based on the stream version.
It was therefore possible that `platform:lp29.0.0` was found as compatible
module for `platform:f29.1.0` although those platform streams are not
compatible at all.

In this commit, the module can be treated as compatible only if it has
the same virtual stream as the input module. The idea behind this
is that both `platform:f29.0.0` and `platform:f29.1.0` should include
the `virtual_streams: [f29]` in their XMD section which tells MBS
that they are actually compatible. The `lp29` stream will not
have the same virtual stream (most likely it won't have any virtual
stream at all).

The `virtual_streams` is already used for this use-case in `MMDResolver`,
but it was not used to limit the inputs to `MMDResolver` which is what
this commit is doing.

This commit also fixes the issue in `get_last_builds_in_stream_version_lte`
which was simply broken if multiple stream_versions of single base module
existed and their builds had different version. In this case, only
builds with single (randomly chosen) version were returned.
2019-03-25 16:53:54 +01:00
mprahl
a8a8913bbc Add the ability to override a buildrequired module stream based on a module's branch
This moves the functionality that was in rpkg to MBS. See the following
PRs for more context:
https://pagure.io/rpkg/pull-request/390
https://pagure.io/rpkg/pull-request/420
2019-03-19 16:45:00 -04:00
mprahl
998084b441 Don't default the module name to "unnamed" on a direct modulemd submission
The module name should either be set in the modulemd file or provided
by the `module_name` parameter.
2019-03-19 08:31:33 -04:00
Merlin Mathesius
a46a4ec470 Accept modulemd for scratch module builds as a parameter in the submitted JSON.
Signed-off-by: Merlin Mathesius <mmathesi@redhat.com>
2019-03-19 11:56:47 +00:00
mprahl
b570948484 Don't allow building modules with the same name as a base module (e.g. platform)
A user managed to build a module called "platform", which stopped dependency
resolution from working. This should stop that from happening again.
2019-03-12 14:58:28 -04:00
Merlin Mathesius
244d873130 Always allow submitting yaml for scratch module builds.
Signed-off-by: Merlin Mathesius <mmathesi@redhat.com>
2019-03-08 11:58:46 -06:00
Merlin Mathesius
152419f376 Module scratch build fixups per PR review feedback:
- Keep scratch module builds in the 'done' state.
- Make koji tagging for scratch modules unique so the same
  commit can be resubmitted.
- Use alternate prefix for scratch module build components so they can
  be identified later.
- Prevent scratch build components from being reused.
- Assorted code and comment cleanup.

Signed-off-by: Merlin Mathesius <mmathesi@redhat.com>
2019-03-01 10:27:04 -06:00
Merlin Mathesius
838feb4ecd Add scratch module build submit tests.
Signed-off-by: Merlin Mathesius <mmathesi@redhat.com>
2019-03-01 10:27:04 -06:00
Merlin Mathesius
2bf7813446 Add 'scratch' and 'srpms' attributes to data model for module scratch builds,
and update tests accordingly.

Signed-off-by: Merlin Mathesius <mmathesi@redhat.com>
2019-03-01 10:27:04 -06:00
mprahl
d5f9c5872f Return an exception to the user if no dependency combination is determined 2019-02-27 14:08:15 +00:00
Valerij Maljulin
f6a4befefe Change exception information for errors when parsing modulemd file.
This fixes #1149

Signed-off-by: Valerij Maljulin <vmaljuli@redhat.com>
2019-02-26 14:09:57 +01:00
mprahl
776330a2bb Add the allowed_users configuration for service accounts to bypass the group membership check
This is required for monitoring use-cases, where we can have a Kerberos principal for a
service account but no associated account in LDAP to check group membership.
2019-02-14 11:11:18 -05:00
Jan Kaluza
40f23b65e4 Fix the handling of modulemd files with unicode characters.
This commit introduces new to_text_type helper method and calls it
for return value of all mmd.dumps() calls. That way, we always
end up with proper unicode string represntation on both python
major versions.

This commit also adds unicode character to description of all
the yaml files we use in the tests so we can be sure MBS can
handle unicode characters properly.

This might be temporary fix, depending on the result of discussion
at https://github.com/fedora-modularity/libmodulemd/issues/184.
2019-02-14 13:42:47 +00:00
Mike Bonnet
e01b3fd077 handle different string representations of set() 2019-02-08 17:43:43 +00:00
mprahl
9f9a025958 Use the https protocol to clone dist-git repos instead of the git protocol
You can read about this at:
https://fedoraproject.org/wiki/Infrastructure/HTTPS-commits

The bug was reported at:
https://bugzilla.redhat.com/show_bug.cgi?id=1666257
2019-01-16 19:00:25 +00:00
mprahl
25122cb53e Modify MBS to use a separate Kerberos context per thread so both threads can use the thread keyring to store the Kerberos cache
This commit includes the backport of the changes to `krb_login` in
https://pagure.io/koji/pull-request/1187. This change is required
for our separate threads to use a separate Kerberos context per thread.
2018-12-18 16:05:55 -05:00
Chenxiong Qi
cbb8156311 Add tests for use of anonymous koji session
This patch also clean and rewrite some tests that are relative to use
anonymous koji session.

Signed-off-by: Chenxiong Qi <cqi@redhat.com>
2018-12-07 18:37:36 +08:00
Chenxiong Qi
5579a0e72b Init event handler handles module stream collision
Fixes: FACTORY-3616

Signed-off-by: Chenxiong Qi <cqi@redhat.com>
2018-12-05 18:25:14 +08:00
Matt Prahl
8d36d97a5d Merge #1085 Show expanded buildrequires in API output 2018-12-04 16:57:12 +00:00
Chenxiong Qi
1cd64d336a Fix some minor issues
* Fix failure test after rebasing on master branch.
* Fix some grammar issues.
* Only check stream collision modules on new created module build.
* Logging message properly.

Signed-off-by: Chenxiong Qi <cqi@redhat.com>
2018-11-27 11:06:34 +08:00
Chenxiong Qi
917c06ad0c Resolve stream collision with modules added to ursine content
This resolve the stream collision by adding specific RPMs to
module-build-macros SRPM as Conflicts.

For more information about module stream collision, please refer to
docstring in utils/ursine.py

Signed-off-by: Chenxiong Qi <cqi@redhat.com>
2018-11-27 11:06:34 +08:00
Chenxiong Qi
04b5a9a2a7 Show expanded buildrequires in API output
ModuleBuild.json now includes xmd/mbs/buildrequires in the output JSON
data. As a result, these APIs will show buildrequires by default.

* /module-builds/
* /module-builds/?verbose=True
* /module-builds/$build_id

Buildrequires is accessible like this:

    br_module_names = response_json['buildrequires'].keys()
    br_module_dict = response_json['buildrequires'][module name]

Fixes: FACTORY-2201

Signed-off-by: Chenxiong Qi <cqi@redhat.com>
2018-11-23 17:40:23 +08:00
mprahl
f7e7765686 Add the ability to override buildrequires and requires when submitting a module build
This will allow for tooling to automatically override certain buildrequires and requires
based on the branch name the modulemd is built form.

Addresses FACTORY-3414
2018-10-29 17:08:27 -04:00
mprahl
85401c2e1e Surface an error to the user when no base module could be found in the module's buildrequires 2018-10-26 14:47:01 -04:00
Jan Kaluza
f2a236bc74 Pass buildrequired modules built against all compatible base module streams to MMDResolver.
Imagine we have "platform:f29.0.0" and "platform:f29.1.0" base modules.
We also have "DBI" module we want to build agaisnt "platform:f29.1.0".
This "DBI" module depends on "perl" module which is only build against
"platform:f29.0.0".

Currently, DBI build would fail to resolve the dependencies, because
it wouldn't find "perl" module, because it is built against different
platform stream.

This PR changes the MSE code to include buildrequired module builds built
against all the compatible platform streams.

It does so by introducing following changes:

- MSE code uses new get_base_module_mmds() method to find out all the
  compatible platform modules. This needed new methods in DBResolver
  and MBSResolver.
- For each buildrequired module defined by name:stream, the MSE code then
  finds particular NSVC built against each compatible platform module.

Side effect of these code changes is that every module now must buildrequire
some base module.
2018-10-26 14:02:36 +02:00
mprahl
599c881714 Add the ability to query by the base modules a module build buildrequires
Future use cases will require the ability to find compatible module builds
to buildrequire based on the base module the module used to build. This
commit adds an association table that will contain module build IDs
and the base module they buildrequire.

Addresses FACTORY-3353
2018-10-17 07:47:31 -04:00
mprahl
cfb75b4d0f Prefix the module version based on the platform it buildrequires 2018-10-04 07:19:07 -04:00
Filip Valder
598347e1b6 Import module API 2018-09-07 17:12:01 +02:00
mprahl
8a54b26fd7 Expose the NVR of component builds in the API to help future debugging 2018-08-07 10:33:14 -04:00
mprahl
a78c564073 Revert "Store the component's build ID and use that to identify the build when acting on a tag message"
This reverts commit 9bd16beeef.
2018-08-07 10:32:50 -04:00
mprahl
9bd16beeef Store the component's build ID and use that to identify the build when acting on a tag message
This resolves an issue where the component name is different than the
package being tagged due to a macro such as those used for SCLs.
2018-08-03 20:45:00 -04:00
Martin Curlej
0aa9f014ce Added search for modules by binary rpm.
Signed-off-by: Martin Curlej <mcurlej@redhat.com>

Updated PR according to review.

Signed-off-by: Martin Curlej <mcurlej@redhat.com>

Searching by multiple koji tags + tests

Signed-off-by: Martin Curlej <mcurlej@redhat.com>
2018-06-26 15:52:37 +02:00
Matt Prahl
764c467c1d Merge #950 New: Support querying of modules/components with multiple state value filtering 2018-06-15 00:36:33 +00:00
Filip Valder
f1fc7ed467 New: Support querying of modules/components with multiple state value filtering 2018-06-13 09:42:38 +02:00
Martin Curlej
e2e804b1d6 Added the ability to search mbs builds by the whole NSVC string.
Signed-off-by: Martin Curlej <mcurlej@redhat.com>

Found about the magical properties of zip.

Signed-off-by: Martin Curlej <mcurlej@redhat.com>
2018-06-13 09:30:52 +02:00
mprahl
00e46c205b Show siblings and component_builds in the standard modules API 2018-05-10 11:48:00 -04:00
Qixiang Wan
062e92e209 Add test case for desc order by context
This is a test case for #868
2018-05-03 22:02:54 +08:00
Qixiang Wan
1ca6b61790 Add test for filtering module builds by context 2018-04-28 11:46:36 +08:00
Qixiang Wan
d83e6897ca Change ModuleBuild.context to db column
1. Changed ModuleBuild's context property to db column, it's
non-nullable and default value is '00000000' to keep it consistent
with previous behaviour.

2. Changed ModuleBuild.contexts_from_mmd to return a tuple of
(ref_build_context, build_context, runtime_context, context).

3. Updated tests affected by this change.
2018-04-28 11:46:31 +08:00
mprahl
05f34598eb Surface the error to the user when a module dependency can't be found 2018-04-05 14:37:08 -04:00
mprahl
5b278211e6 Add API v2 which will return a list of modules on build submissions based on module stream expansion 2018-04-03 09:58:57 -04:00
mprahl
41814b42de Add siblings to the extended JSON output of module builds 2018-04-03 09:58:57 -04:00
Jan Kaluza
e5e9a24fce Allow adding bool to XMD. Fix flake8 issues. 2018-04-03 09:58:57 -04:00
Jan Kaluza
31ddbe51a5 Add generate_expanded_mmds which returns list of MMDs which can be submitted as MSE builds and submit them. 2018-04-03 09:58:57 -04:00
mprahl
eb0b2e1c38 Add the database resolver plugin 2018-04-03 09:58:57 -04:00
Jan Kaluza
1bbe8d69f1 Add ModuleBuild methods for getting list of streams, last build in a name:stream and all builds in name:stream:version. 2018-04-03 09:58:57 -04:00
mprahl
f0852d9009 Port to libmodulemd and support v2 modulemd without module stream expansion 2018-04-03 09:58:57 -04:00
Yash Nanavati
ed25afd9ec Code changes for Python 3 support 2018-03-04 03:22:38 -05:00
mprahl
f5530d9b16 Fix flake8 errors made by jkaluza 2018-02-09 09:12:00 -05:00