Commit Graph

1300 Commits

Author SHA1 Message Date
mprahl
164f592c1d Make the DNF minrate setting configurable when loading repos 2019-10-02 07:37:12 -04:00
Matt Prahl
2796b9e6a1 Merge #1448 Load the DNF repos in parallel 2019-10-02 11:33:21 +00:00
Jan Kaluza
fc39df9487 Store the build_context without base modules in a database.
For KojiResolver, we need to be able to find out the module to reuse
components from based only on the non-base-module buildrequires. The
base module buildrequirement will be defined by the Koji tag inheritance.

For this, we need to store build_context computed without the base
modules.
2019-10-02 07:16:55 +02:00
mprahl
8d76977434 Load the DNF repos in parallel
In production, loading each repo can be quite slow. This approach
loads the repos in parallel.
2019-10-01 15:29:00 -04:00
Matt Prahl
d4b7a019c7 Merge #1444 Return an empty list in KojiResolver.get_compatible_base_module_modulemds. 2019-09-30 15:12:31 +00:00
Matt Prahl
db7696eafc Merge #1439 Disable handling collisions on local builds due to RHBZ#1693683 2019-09-30 14:58:40 +00:00
Jan Kaluza
a28df5edb7 Return an empty list in KojiResolver.get_compatible_base_module_modulemds.
For KojiResolver, this method returns always an empty list. The compatible modules are
defined by the Koji tag inheritance, so there is no need to find out the compatible
base modules on MBS side.

This makes `mse.get_base_module_mmds` to ignore virtual streams and just use
the input base module as the only module without finding the compatible
base modules.
2019-09-30 09:04:21 +02:00
Jan Kaluža
f6c2a88cc0 Merge #1435 Add initial code for KojiResolver class. 2019-09-30 06:25:55 +00:00
Jan Kaluza
a7540452cd Add initial code for KojiResolver class.
This commit:

- Adds KojiResolver class and KojiResolver tests.
- Changes the GenericResolver and its subclasses to pass base_module_mmds
  instead of base_module_nsvc to get_buildrequired_modulemds. This is needed,
  because KojiResolver needs to access XMD section of base module.
- Implements KojiResolver.get_buildrequired_modulemds to ask Koji for list of
  modules tagged in the Koji tag and return their modulemds.
2019-09-30 07:56:07 +02:00
mprahl
6b2e5be93a Disable handling collisions on local builds due to RHBZ#1693683 2019-09-26 11:43:30 -04:00
mprahl
453d871e03 Make the log message in execute_cmd easier to read for local builds 2019-09-25 17:30:21 -04:00
Jan Kaluza
1385a1dea3 default_modules: Convert arch to canon_arch.
Convert arch to canon_arch. This handles cases where Koji "i686" arch is mapped to
"i386" when generating RPM repository.
2019-09-23 13:18:38 +02:00
Chenxiong Qi
8e54f05428 __all__ should be in type list[str]
Signed-off-by: Chenxiong Qi <cqi@redhat.com>
2019-09-20 21:47:04 +08:00
mprahl
749f186524 Add conflicts in module-build-macros for NEVRAs found in handle_collisions_with_base_module_rpms
PR #1331 made the assumption that the Ursa Major ursine RPMs were at
xmd["mbs"]["ursine_rpms"], but they are actually at
xmd["mbs"]["buildrequires"]["platform"]["ursine_rpms"]. This commit
handles the ursine RPMs generated by handle_collisions_with_base_module_rpms
separately since the base module the RPMs came from are not tracked in
that function.
2019-09-19 13:31:10 +00:00
Mike McLean
bbb1d4bb5b Use utc date for compare and correct operator 2019-09-18 20:21:14 +00:00
Mike McLean
fafff9f5d0 use date comparison rather than datetime 2019-09-18 20:21:14 +00:00
mprahl
099ee8cbd1 Call record_module_build_arches before add_default_modules
record_module_build_arches needs to be called before since
record_module_build_arches requires the arches to be set to know
which DNF repos to query.
2019-09-18 11:36:01 -04:00
Chenxiong Qi
0899e470f0 Convert value to correct type in order to pass to _get_rpms_from_tags
Fixes #1420

Signed-off-by: Chenxiong Qi <cqi@redhat.com>
2019-09-18 16:48:39 +08:00
Chenxiong Qi
065abe3c45 Fix DetachedInstanceError in command module_build_locally
Update test test_build_module_locally_set_stream accordingly by not
mocking database session object in order to test against the real
SQLAlchemy Session scope.

Resolves: RHBZ#1752075

Signed-off-by: Chenxiong Qi <cqi@redhat.com>
2019-09-17 23:09:14 +08:00
Chenxiong Qi
c36bd7ebac Name component builds by states
This patch attemps to make code shorter and easier to read by naming
component builds for different states.

Signed-off-by: Chenxiong Qi <cqi@redhat.com>
2019-09-16 15:29:21 +08:00
Jan Kaluža
b33fc6adf1 Merge #1410 Don't check for compatibile modules during component reuse if allow_only_compatible_base_modules is false 2019-09-16 06:14:10 +00:00
Jan Kaluža
0bf84f557e Merge #1407 Do not ignore building components to reset state when submit an existing module build 2019-09-16 06:11:04 +00:00
mprahl
d2e7c0cf90 Don't check for compatibile modules during component reuse if allow_only_compatible_base_modules is false
Addresses #1409
2019-09-13 13:06:35 +00:00
Jan Kaluza
52e88ba3ff Handle the conflicts between base module modular Koji tags everytime.
Currently, we generate `Conflicts` for ursine RPMs conflicting with
modular RPMs only when Ursa Prime is used for the base module. This
commit changes it, so these Conflicts are generated everytime.

The reason is that modular RPMs should always be preferred in the
buildroot over the ursine RPMs no matter what is their NVR. So far,
this has been guarded on Koji side by using external repos, but
we need to move away from external repo or at least use "bare"
merge mode which basically means we won't get this feature for free
from Koji.

The reason why we need to move away from external repos or use "bare"
merge mode is that without this, the Koji removes RPMs sharing the same
name but different version/release from the buildroot and only keeps
the latest one. This is an issue in situation when you need two
versions of single RPM in a buildroot comming from two modules.
2019-09-12 11:08:57 +02:00
mprahl
6b484b7f49 Use str(e) instead of e.message to support Python 2 and 3 2019-09-11 12:38:02 -04:00
mprahl
00e78494d9 Refactor handling of default buildroot modules (Ursa Prime)
This removes support for default_modules_url in the Platform XMD and
gets the list of default name:stream combinations to include in the buildroot
from https://pagure.io/releng/fedora-module-defaults.

Addresses #1402
2019-09-11 12:20:35 -04:00
Chenxiong Qi
c54c3a288b Do not ignore building components to reset state when submit an existing module build
koji.BUILD_STATE["BUILDING"] equals to 0. So, checking if
component.state is None explicitly to avoid ignoring the components
which are in building state.

Signed-off-by: Chenxiong Qi <cqi@redhat.com>
2019-09-11 09:55:10 +08:00
Qixiang Wan
47809d530a Set build logs dir to mock results dir for mock builder
When build module with MockModulerBuilder, set build logs dir to
the main results dir of mock builder before build_logs start.
2019-09-03 22:27:03 +08:00
Qixiang Wan
478709e25d Change console log level of local build to INFO
1. Add option "-d|--debug" to build_module_locally command

   Note: there is also global debug option to Manager, so we need to use
a different dest name ("log_debug" in this change).

2. When debug option is not specified for local build command, set log
level of console to INFO.
2019-09-03 22:26:45 +08:00
Chenxiong Qi
6d19a23c17 Refactor buildroot_add_artifacts
* Reuse tag_artifacts.
* Split unblocking packages and adding packages to groups into
  separate methods.

Signed-off-by: Chenxiong Qi <cqi@redhat.com>
2019-09-02 20:30:49 +08:00
Chenxiong Qi
6c3721ef75 Update KojiModuleBuilder *_artifacts methods docstring
Signed-off-by: Chenxiong Qi <cqi@redhat.com>
2019-09-02 17:37:13 +08:00
Jan Kaluža
457b16daff Merge #1395 Reuse the latest module build found. 2019-08-30 05:48:25 +00:00
Jan Kaluza
00ad20dfcd Reuse the latest module build found.
Before this commit, the base modules used in the `get_reusable_module` have
not been sorted and therefore when `get_reusable_module` tried to find out
the reusable module built against some base module, it could find a module
built against some old version of base module.

This could lead to situation when MBS tried to reuse components from quite
old module despite the fact that newer module build existed.

This commit fixes this by sorting the base modules by stream_version,
so MBS always tries to get the reusable module built against the latest
base module.
2019-08-29 15:35:21 +02:00
Valerij Maljulin
c4a95d0673 Add a check for non-ready builds (siblings) in requires resolver
Signed-off-by: Valerij Maljulin <vmaljuli@redhat.com>
2019-08-28 15:34:46 +02:00
Jan Kaluža
9d40a9b9cb Merge #1390 Fix --add-local-build with MBS Resolver. 2019-08-26 05:33:40 +00:00
Jan Kaluza
cc5aaf90c1 Fix --add-local-build with MBS Resolver.
The `MBSResolver.get_buildrequired_modulemds` did not try to load
local module builds and always just queried the remote MBS instance.

This commit fixes it by using local module if available.
2019-08-23 14:58:15 +02:00
mprahl
f3db9a0ac2 Fix a typo in an error message 2019-08-22 17:15:55 -04:00
Valerij Maljulin
4b4428a9e3 Fix locale issues with date representations
This fixes #1214
Also fixes tests running on different locales

Signed-off-by: Valerij Maljulin <vmaljuli@redhat.com>
2019-08-22 17:06:26 +02:00
mprahl
a7099d8431 Make the timestamps consistent in import_mmd
This will keep tests from failing occasionally.
2019-08-21 16:21:05 -04:00
Jan Kaluža
13a18d1d5a Merge #1384 Add "scratch_build_only_branches" configuration options. 2019-08-21 11:18:34 +00:00
Jan Kaluza
80fca557af Do not check Greenwave gating status for scratch builds.
Scratch builds cannot be gated. They stay in the `done` state forever.
Therefore it is useless to query Greenwave for its status in the Poller.
2019-08-21 13:17:04 +02:00
Chenxiong Qi
e1342d8ffc Use dict literal to create dict
Some code create a dict in this way:

some_var = {}
some_var["a"] = 100
some_var["b"] = 200

Using dict literal could make these lines a little bit simpler.

Signed-off-by: Chenxiong Qi <cqi@redhat.com>
2019-08-15 21:28:27 +08:00
Chenxiong Qi
e6aa47e02a Use set literal to create a set
Signed-off-by: Chenxiong Qi <cqi@redhat.com>
2019-08-15 21:14:02 +08:00
Chenxiong Qi
dbced8668b Do not compare with a empty list
An empty list [] is evaluated as a false value. So, it is not necessary
to compare like "if some_var == []:".

Signed-off-by: Chenxiong Qi <cqi@redhat.com>
2019-08-15 17:48:33 +08:00
Jan Kaluza
796a367457 Add "scratch_build_only_branches" configuration options.
The goal here is to define certain branches from which only scratch
module builds can be submitted. The main use case is for "private-*"
branches which can be created and maintained by anyone, but there
must not be production-ready module build created from them.

This commit adds new `scratch_build_only_branches` config option
to define the list of regexes to match such branches.
2019-08-15 10:49:03 +02:00
Chenxiong Qi
6f1d0b3ad9 Return empty result if short=true is specified on empty list of builds
Fixes #1376

Signed-off-by: Chenxiong Qi <cqi@redhat.com>
2019-08-11 22:18:38 +08:00
Chenxiong Qi
0917d6aa4d Start build logs from init event handler
Signed-off-by: Chenxiong Qi <cqi@redhat.com>
2019-08-09 16:47:46 +08:00
Valerij Maljulin
e5735efc76 Skip prefix validation for modules in allowed_privileged_module_names and base_module_names
Signed-off-by: Valerij Maljulin <vmaljuli@redhat.com>
2019-08-07 14:17:39 +02:00
Jan Kaluža
f759a4ec1c Merge #1374 Remove the “All” MMDResolver policy 2019-08-07 06:29:38 +00:00
Jan Kaluža
758cf9c112 Merge #1365 Allow components to be reused from module builds even if the buildrequires commit hashes changed for the changed-and-after rebuild strategy 2019-08-06 11:08:46 +00:00