Commit Graph

2550 Commits

Author SHA1 Message Date
Mike McLean
120681806b update script options in CONTRIBUTING doc 2019-10-15 11:17:51 -04:00
Mike McLean
0d8af77ce4 run-unittests.sh: docker pull needed images
Related: https://pagure.io/fm-orchestrator/issue/1463
2019-10-15 11:14:10 -04:00
Jan Kaluza
eeb65c97da KojiResolver: Filter module builds based on the real stream name.
Query Koji for the real stream name of each module and keep only those matching
requested `stream`.

This needs to be done, because MBS stores the stream name in the "version" field in Koji,
but the "version" field cannot contain "-" character. Therefore MBS replaces all "-"
with "_". This makes it impossible to reconstruct the original stream name from the
"version" field.

We therefore need to ask for real original stream name here and filter out modules based
on this real stream name.
2019-10-14 14:13:34 +02:00
Jan Kaluža
578d6f1e33 Merge #1461 Add KojiResolver doc. 2019-10-11 07:36:29 +00:00
Jan Kaluza
3fedb14483 Add KojiResolver doc. 2019-10-11 09:34:19 +02:00
Matt Prahl
74bf638cdc Merge #1458 Only use default modules that were built with the same base module 2019-10-09 12:44:52 +00:00
Jan Kaluža
514e4c115a Merge #1457 Make reuse_shared_userspace_init_data faster 2019-10-09 05:24:00 +00:00
mprahl
27f4d71392 Only use default modules that were built with the same base module 2019-10-08 11:38:39 -04:00
Chenxiong Qi
800f606b83 Make reuse_shared_userspace_init_data faster
* A significant performance improvement is to avoid repeating call of
  mmd.get_xmd() in a looping.
* Function is refactored to avoid call of mmd_to_str twice for a module
  build.
* Test using this fixture already has a setup_method to clean database,
  so no need to call clean_database again.
* Calculate build context directly without calculating other contexts
  together.

Signed-off-by: Chenxiong Qi <cqi@redhat.com>
2019-10-08 11:30:50 +08:00
Jan Kaluza
339c1ef98d Add KojiResolver to config.SUPPORTED_RESOLVERS. 2019-10-07 12:52:13 +02:00
Jan Kaluza
5ffdeb4649 Fallback to DBResolver in KojiResolver.get_compatible_base_module_modulemds.
In case KojiResolver is not enabled for the base module, fallback to
DBResolver in `KojiResolver.get_compatible_base_module_modulemds`.
2019-10-04 15:22:35 +02:00
Jan Kaluža
9e38493112 Merge #1452 Support KojiResolver in MBSResolver. 2019-10-04 06:45:39 +00:00
Jan Kaluza
72d2c3daf4 Support KojiResolver in MBSResolver.
If KojiResolver is enabled for buildrequired base module and
MBSResolver is used, then `MBSResolver.get_buildrequired_modulemds`
will use KojiResolver to get the list of buildrequired module builds.

Otherwise it uses the current behavior.

To implement this, the `KojiResolver.get_buildrequired_modules` was
split into two methods:

- `get_buildrequired_koji_builds` returns buildrequired Koji builds.
- `get_buildrequired_modules` calls `get_buildrequired_koji_builds`
  and finds the corresponding ModuleBuilds in MBS DB.
2019-10-04 07:43:11 +02:00
Mike McLean
e9946fcd00 Release v2.29.1 v2.29.1 2019-10-03 16:37:02 -04:00
mprahl
540e2ceac3 Fix a fork in the database migration scripts 2019-10-03 16:29:47 -04:00
Mike McLean
bbe10fd2da Release 2.29.0 v2.29.0 2019-10-03 15:10:54 -04:00
Valerij Maljulin
2c0291987f Raise Unathorized exception instead of general one whenever OIDC auth fails
This fixes #1147
2019-10-03 17:02:33 +02:00
Valerij Maljulin
689d949a48 buildonly support
This fixes #1307

Signed-off-by: Valerij Maljulin <vmaljuli@redhat.com>
2019-10-03 16:20:04 +02:00
Jan Kaluža
6e34e50c18 Merge #1451 Add support for KojiResolver in component reuse code. 2019-10-03 13:16:00 +00:00
Jan Kaluza
486dc39898 Add support for KojiResolver in component reuse code.
In this commit, when component reuse code finds out that the base module uses
KojiResolver, it uses the `KojiResolver.get_buildrequired_modules` method
to find out possible modules to reuse and limits the original query just
by the IDs of these modules.

In order to do that, this commit splits the original
`KojiResolver.get_buildrequired_modulemds` into two methods:

- The `get_buildrequired_modules` returning the ModuleBuilds.
- The `get_buildrequired_modulemds` calling the `get_buildrequired_modules`
  and returning modulemd metadata.
2019-10-03 15:14:29 +02:00
mprahl
8c6cfb702d Use small license headers in the Python files
This also removes the outdated comments around authorship of each
file. If there is still interest in this information, one can just
look at the git history.
2019-10-03 08:47:24 -04:00
Matt Prahl
df295516e3 Merge #1449 Use double quotes instead of single quotes to match the style guide 2019-10-02 13:57:36 +00:00
Matt Prahl
73e2d1a6f7 Merge #1427 Allow running an individual test using a relative path 2019-10-02 12:09:30 +00:00
mprahl
f60b292c9c Use double quotes instead of single quotes to match the style guide 2019-10-02 07:42:20 -04:00
mprahl
164f592c1d Make the DNF minrate setting configurable when loading repos 2019-10-02 07:37:12 -04:00
Matt Prahl
2796b9e6a1 Merge #1448 Load the DNF repos in parallel 2019-10-02 11:33:21 +00:00
Jan Kaluza
fc39df9487 Store the build_context without base modules in a database.
For KojiResolver, we need to be able to find out the module to reuse
components from based only on the non-base-module buildrequires. The
base module buildrequirement will be defined by the Koji tag inheritance.

For this, we need to store build_context computed without the base
modules.
2019-10-02 07:16:55 +02:00
mprahl
8d76977434 Load the DNF repos in parallel
In production, loading each repo can be quite slow. This approach
loads the repos in parallel.
2019-10-01 15:29:00 -04:00
Matt Prahl
d4b7a019c7 Merge #1444 Return an empty list in KojiResolver.get_compatible_base_module_modulemds. 2019-09-30 15:12:31 +00:00
Matt Prahl
db7696eafc Merge #1439 Disable handling collisions on local builds due to RHBZ#1693683 2019-09-30 14:58:40 +00:00
Jan Kaluza
a28df5edb7 Return an empty list in KojiResolver.get_compatible_base_module_modulemds.
For KojiResolver, this method returns always an empty list. The compatible modules are
defined by the Koji tag inheritance, so there is no need to find out the compatible
base modules on MBS side.

This makes `mse.get_base_module_mmds` to ignore virtual streams and just use
the input base module as the only module without finding the compatible
base modules.
2019-09-30 09:04:21 +02:00
Jan Kaluža
f6c2a88cc0 Merge #1435 Add initial code for KojiResolver class. 2019-09-30 06:25:55 +00:00
Jan Kaluza
a7540452cd Add initial code for KojiResolver class.
This commit:

- Adds KojiResolver class and KojiResolver tests.
- Changes the GenericResolver and its subclasses to pass base_module_mmds
  instead of base_module_nsvc to get_buildrequired_modulemds. This is needed,
  because KojiResolver needs to access XMD section of base module.
- Implements KojiResolver.get_buildrequired_modulemds to ask Koji for list of
  modules tagged in the Koji tag and return their modulemds.
2019-09-30 07:56:07 +02:00
Stephen Gallagher
8e3ebe5df2 Doc improvements for virtual modules
Signed-off-by: Stephen Gallagher <sgallagh@redhat.com>
2019-09-27 08:48:05 -04:00
mprahl
6b2e5be93a Disable handling collisions on local builds due to RHBZ#1693683 2019-09-26 11:43:30 -04:00
mprahl
453d871e03 Make the log message in execute_cmd easier to read for local builds 2019-09-25 17:30:21 -04:00
Jan Kaluza
1385a1dea3 default_modules: Convert arch to canon_arch.
Convert arch to canon_arch. This handles cases where Koji "i686" arch is mapped to
"i386" when generating RPM repository.
2019-09-23 13:18:38 +02:00
Jan Kaluža
3d5dd91cf7 Merge #1433 __all__ should be in type list[str] 2019-09-23 05:41:43 +00:00
Chenxiong Qi
8e54f05428 __all__ should be in type list[str]
Signed-off-by: Chenxiong Qi <cqi@redhat.com>
2019-09-20 21:47:04 +08:00
Luiz Carvalho
97be1c4737 Document how to generate DB migration scripts
Signed-off-by: Luiz Carvalho <lucarval@redhat.com>
2019-09-19 16:26:26 -04:00
mprahl
550b266961 Release v2.28.2 v2.28.2 2019-09-19 10:22:01 -04:00
mprahl
749f186524 Add conflicts in module-build-macros for NEVRAs found in handle_collisions_with_base_module_rpms
PR #1331 made the assumption that the Ursa Major ursine RPMs were at
xmd["mbs"]["ursine_rpms"], but they are actually at
xmd["mbs"]["buildrequires"]["platform"]["ursine_rpms"]. This commit
handles the ursine RPMs generated by handle_collisions_with_base_module_rpms
separately since the base module the RPMs came from are not tracked in
that function.
2019-09-19 13:31:10 +00:00
Mike McLean
e2939c80a1 use patch.object 2019-09-18 17:22:59 -04:00
Mike McLean
e1cf07c038 extend unit test 2019-09-18 20:21:14 +00:00
Mike McLean
bbb1d4bb5b Use utc date for compare and correct operator 2019-09-18 20:21:14 +00:00
Mike McLean
fafff9f5d0 use date comparison rather than datetime 2019-09-18 20:21:14 +00:00
mprahl
2fc439c860 Allow running an individual test using a relative path
Since the tests now run in a different location in the container
since PR #1378, the documentation needed to be updated. Instead,
we should just allow the user to pass a relative path instead.
2019-09-18 16:01:05 -04:00
Matt Prahl
902d887a5e Merge #1423 option to use sudo when running docker for unit tests 2019-09-18 19:27:53 +00:00
mprahl
235d4cd457 Fix the test_handle_collisions_with_base_module_rpms unit test 2019-09-18 12:04:18 -04:00
mprahl
099ee8cbd1 Call record_module_build_arches before add_default_modules
record_module_build_arches needs to be called before since
record_module_build_arches requires the arches to be set to know
which DNF repos to query.
2019-09-18 11:36:01 -04:00