Commit Graph

82 Commits

Author SHA1 Message Date
mprahl
14098cea08 Migrate to libmodulemd v2
This also moves the methods load_mmd and load_mmd_file to
module_build_service.utils.general.

This also removes some MSE unit tests with a mix of positive and
negative streams since this is not supported in libmodulemd v2. The
user will be presented with a syntax error if they try to submit
such a modulemd file.
2019-05-13 13:40:37 -04:00
mprahl
66c3f82160 Format the coding style across the codebase using "black" and manual tweaks
The main benefit of this commit is that the use of double quotes
is now consistent.
2019-04-26 00:32:13 -04:00
Jan Kaluza
67a5a9d1b0 Allow building module in --offline module with dependencies from local repositories.
There are following changes introduced in this commit:

- The `koji_tag` of module builds imported from the local repositories
  is now in `repofile:///etc/yum.repos.d/some.repo` format to store the
  repository from which the module was imported to local MBS DB.
- The `koji_tag` of fake base module is set to empty `repofile://`
  and in `MockModuleBuilder` the `conf.base_module_repofiles` list
  is used as source for the repositories defining platform. We can't
  simply use single repository, because there might be fedora.repo
  and fedora-update.repo and so on.
- The list of default .repo files for platform are passed using the
  `-r` switch in `build_module_locally` `mbs-manager` command.
- The LocalResolver (subclass of DBResolver) is added which is used
  to resolve the build dependencies when building modules offline
  locally.
- The `MockModuleBuilder` enables the buildrequired modules and
  repositories from which they come in the mock config.

With this commit, it is possible to build testmodule locally
without any external infra.
2019-04-01 13:13:08 +00:00
Jan Kaluza
765e640129 Allow setting the Koji tag extra options using the conf.koji_tag_extra_opts.
The Koji tag extra options used to be hard-coded and to change them,
we had to release new MBS version.

We do not change them often, but right now fedora-infra is requesting
to use new `mock.new_chroot` option and we need to release new MBS
because of that.

This commit makes such changes easier in the future.
2019-03-18 17:27:57 +00:00
Jan Kaluza
9a6646f27c Use the scrmod prefix also for build targets for scratch builds.
Currently, the Koji tags have properly set the scrmod- prefix, but
the build target still sets module- prefix even for scratch builds.

In this commit, build target has the scrmod- prefix too.
2019-03-14 13:10:47 +01:00
Valerij Maljulin
687a78242e Change xmlrpclib importing way
Signed-off-by: Valerij Maljulin <vmaljuli@redhat.com>
2019-02-25 16:56:30 +01:00
Valerij Maljulin
eafa93037f Make finalization before changing state to done
Signed-off-by: Valerij Maljulin <vmaljuli@redhat.com>
2019-02-15 15:19:59 +01:00
Jan Kaluza
40f23b65e4 Fix the handling of modulemd files with unicode characters.
This commit introduces new to_text_type helper method and calls it
for return value of all mmd.dumps() calls. That way, we always
end up with proper unicode string represntation on both python
major versions.

This commit also adds unicode character to description of all
the yaml files we use in the tests so we can be sure MBS can
handle unicode characters properly.

This might be temporary fix, depending on the result of discussion
at https://github.com/fedora-modularity/libmodulemd/issues/184.
2019-02-14 13:42:47 +00:00
Mike Bonnet
dcd38db5e0 hide the missing krbV module 2019-02-08 17:43:43 +00:00
mprahl
9f9a025958 Use the https protocol to clone dist-git repos instead of the git protocol
You can read about this at:
https://fedoraproject.org/wiki/Infrastructure/HTTPS-commits

The bug was reported at:
https://bugzilla.redhat.com/show_bug.cgi?id=1666257
2019-01-16 19:00:25 +00:00
mprahl
25122cb53e Modify MBS to use a separate Kerberos context per thread so both threads can use the thread keyring to store the Kerberos cache
This commit includes the backport of the changes to `krb_login` in
https://pagure.io/koji/pull-request/1187. This change is required
for our separate threads to use a separate Kerberos context per thread.
2018-12-18 16:05:55 -05:00
mprahl
96f82443aa Remove the unused "owner" parameter from KojiModuleBuilder.get_session 2018-12-10 09:59:42 -05:00
Chenxiong Qi
cbb8156311 Add tests for use of anonymous koji session
This patch also clean and rewrite some tests that are relative to use
anonymous koji session.

Signed-off-by: Chenxiong Qi <cqi@redhat.com>
2018-12-07 18:37:36 +08:00
mprahl
25b6a93a07 Make the devel Koji CG build optional using a config option 2018-12-03 12:00:30 -05:00
Chenxiong Qi
917c06ad0c Resolve stream collision with modules added to ursine content
This resolve the stream collision by adding specific RPMs to
module-build-macros SRPM as Conflicts.

For more information about module stream collision, please refer to
docstring in utils/ursine.py

Signed-off-by: Chenxiong Qi <cqi@redhat.com>
2018-11-27 11:06:34 +08:00
Valerij Maljulin
f3c86541b6 Do not retreive rpm info if there is an empty rpm list.
Fixes #1078

Signed-off-by: Valerij Maljulin <vmaljuli@redhat.com>
2018-11-20 10:45:17 +01:00
mprahl
599c881714 Add the ability to query by the base modules a module build buildrequires
Future use cases will require the ability to find compatible module builds
to buildrequire based on the base module the module used to build. This
commit adds an association table that will contain module build IDs
and the base module they buildrequire.

Addresses FACTORY-3353
2018-10-17 07:47:31 -04:00
Valerij Maljulin
7f60db545a renaming koji_arches to arches
Signed-off-by: Valerij Maljulin <vmaljuli@redhat.com>
2018-10-15 15:29:50 +02:00
Chenxiong Qi
520a979d3d Add singleton system_resolver
system_resolver is created based on loaded configuration, which could
avoid calls like `GenericResolver.create(conf)` repeatedly in the code.

However, if some cases need to create a specific resolver explicitly,
`GenericResolver.create` could be called with addition argument, for
example db or mbs is passed to argument backend in tests.

Signed-off-by: Chenxiong Qi <cqi@redhat.com>
2018-09-28 11:00:52 +08:00
Jan Kaluza
56fe201c33 Fill in the 'filterer_rpms' in backend to not access Koji from frontend. 2018-08-21 07:58:52 +02:00
Jan Kaluza
f0b35be067 Add BASE_MODULE_KOJI_ARCHES to override default Koji arches. 2018-08-14 13:18:56 +02:00
mprahl
d54422adeb Don't filter RPMs of reused components from a module that depends on itself
If a module has filters set and it buildrequires itself, issues occur
during component reuse. In that event, the filtered RPMs from the
previous module build will not be available in the buildroot because the NVRs
in the filters will match the RPMs from the reused component.

Co-authored-by: Jan Kaluza <jkaluza@redhat.com>
2018-08-13 11:13:40 -04:00
Jan Kaluza
6d2e4bbffa Fix filters generation 2018-08-09 15:16:13 +02:00
Martin Curlej
dfe7660519 Removed PDC dependency from MBS and replaced it with MBS prod instance.
Signed-off-by: Martin Curlej <mcurlej@redhat.com>

Removed pdc from comments.

Signed-off-by: Martin Curlej <mcurlej@redhat.com>

Adding missing files.

Signed-off-by: Martin Curlej <mcurlej@redhat.com>

Updated PR according to review.

Signed-off-by: Martin Curlej <mcurlej@redhat.com>

Local modules builds now

Signed-off-by: Martin Curlej <mcurlej@redhat.com>

Removed copr from config

Signed-off-by: Martin Curlej <mcurlej@redhat.com>

Fixed bugs

Signed-off-by: Martin Curlej <mcurlej@redhat.com>
2018-07-04 10:13:18 +02:00
Mikolaj Izdebski
0d29633da9 Make repo_include_all setting configurable in xmd
Previously MBS configured all Koji build tags with
repo_include_all=True extra option.  But for some modules it was
desired to be able to have tags with repo_include_all=False.

Fixes #957
2018-06-29 17:06:58 +02:00
mprahl
74e4c99984 Support the modulemd buildopts.rpms.whitelist option 2018-06-27 07:05:06 -04:00
Owen W. Taylor
ddee4f840b MockModuleBuilder: add artifacts to artifacts not components
The built RPM artifacts were being added to the components list (the
RPMs to build) instead of the proper artifacts section. Also, they
need to be in NEVRA format, so use 'rpm -qf' to query the epochs from
the built artifacts.
2018-05-30 15:47:14 -04:00
mprahl
15a7e322a6 Fallback to the old Koji tag format for the target when the tag name is too long 2018-05-08 09:11:22 -04:00
Jan Kaluza
c1c95fc963 Allow defining list of packages which are blocked in the -build tag until they are built in a module. 2018-04-13 12:57:27 +02:00
Jan Kaluza
5d91c97386 Allow defining list of packages which are blocked in the -build tag until they are built in a module. 2018-04-12 09:45:57 +02:00
mprahl
caadb13f0e Remove the COPR tests
The COPR tests were causing failures and since most of them were
skipped and haven't been updated in a long time, let's just delete
them.
2018-04-03 09:58:57 -04:00
Jan Kaluza
e5e9a24fce Allow adding bool to XMD. Fix flake8 issues. 2018-04-03 09:58:57 -04:00
mprahl
eb0b2e1c38 Add the database resolver plugin 2018-04-03 09:58:57 -04:00
mprahl
f0852d9009 Port to libmodulemd and support v2 modulemd without module stream expansion 2018-04-03 09:58:57 -04:00
Yash Nanavati
ed25afd9ec Code changes for Python 3 support 2018-03-04 03:22:38 -05:00
Yash Nanavati
1d3fd2af64 Minor changes towards Python 3 Compatability 2018-02-28 09:01:51 -05:00
mprahl
f5530d9b16 Fix flake8 errors made by jkaluza 2018-02-09 09:12:00 -05:00
Jan Kaluza
6a5d7267fd Save another 20s in tests. 2018-02-09 11:46:12 +01:00
mprahl
e91d09f7ca Change the format of the unit tests to pytest 2018-01-31 16:34:21 -05:00
mprahl
df594da8a5 Remove the Koji proxyuser functionality 2018-01-30 11:49:03 -05:00
Filip Valder
5b4a8f6240 New: Resolver API for pluggable resolvers 2018-01-29 15:54:19 +01:00
Ralph Bean
01c8295269 Add an incrementing prefix to the dist_hash.
For #823.
2018-01-17 16:54:02 -05:00
mprahl
a67b53f5c3 Add a "context" field on component and module releases in Koji for uniqueness for when Module Stream Expansion is implemented 2018-01-16 10:36:09 -05:00
mprahl
c0ec0f82c1 Remove urlgrabber as a dependency 2018-01-03 10:21:37 -05:00
mprahl
a54045e518 Don't run KojiModuleBuilder.buildroot_connect in the stale module builds poller 2017-12-11 13:27:30 -05:00
Jan Kaluza
7de120c17b Get the weight of Koji tasks and use it to sort the components in a batch instead of builds time. 2017-11-28 10:52:17 -05:00
mprahl
e4be54a838 Skip COPR tests that now fail since PR #765 2017-11-28 09:56:51 -05:00
mprahl
30ef85af01 Fix "instant complete" builds and reusing all components 2017-11-27 16:20:28 -05:00
Ralph Bean
8431d2698b Explicit imports of builders in the test suite. 2017-11-10 16:03:14 -05:00
mprahl
a8421568f9 Fix Flake8 errors 2017-11-08 08:45:58 -05:00