2149 Commits

Author SHA1 Message Date
Jan Kaluza
40e00f8aa8 Release 2.16.0 v2.16.0 2019-03-04 10:13:33 +01:00
Jan Kaluza
ac0bd8ceb8 Set time_completed before calling KojiContentGenerator.
The `KojiContentGenerator.finalize()` needs the `ModuleBuild.time_completed`.

We currently set `time_completed` once the module build transitions
into `done` state. But we have moved the `KojiContentGenerator` call
to end of `build` state, so right now it is called before the `time_completed`
is set. This leads to traceback.

In this commit, the `time_completed` is set before the `KojiContentGenerator`
call, so it is defined properly.
2019-03-04 10:03:47 +01:00
Merlin Mathesius
152419f376 Module scratch build fixups per PR review feedback:
- Keep scratch module builds in the 'done' state.
- Make koji tagging for scratch modules unique so the same
  commit can be resubmitted.
- Use alternate prefix for scratch module build components so they can
  be identified later.
- Prevent scratch build components from being reused.
- Assorted code and comment cleanup.

Signed-off-by: Merlin Mathesius <mmathesi@redhat.com>
2019-03-01 10:27:04 -06:00
Merlin Mathesius
6daa4d3776 Revise command line parsing for local module builds to require SRPMs to be
specified individually using multiple '--srpm SRPM' options.

Signed-off-by: Merlin Mathesius <mmathesi@redhat.com>
2019-03-01 10:27:04 -06:00
Merlin Mathesius
a43d684859 Updates to handle including custom SRPMs for scratch module builds.
Signed-off-by: Merlin Mathesius <mmathesi@redhat.com>
2019-03-01 10:27:04 -06:00
Merlin Mathesius
dd950857ec Update koji tagging for scratch modules.
Signed-off-by: Merlin Mathesius <mmathesi@redhat.com>
2019-03-01 10:27:04 -06:00
Merlin Mathesius
7794950802 Add preliminary module scratch build details to documentation.
Signed-off-by: Merlin Mathesius <mmathesi@redhat.com>
2019-03-01 10:27:04 -06:00
Merlin Mathesius
838feb4ecd Add scratch module build submit tests.
Signed-off-by: Merlin Mathesius <mmathesi@redhat.com>
2019-03-01 10:27:04 -06:00
Merlin Mathesius
10e9dace9c Add database migration handling of new module_builds columns for module scratch builds.
Signed-off-by: Merlin Mathesius <mmathesi@redhat.com>
2019-03-01 10:27:04 -06:00
Merlin Mathesius
2bf7813446 Add 'scratch' and 'srpms' attributes to data model for module scratch builds,
and update tests accordingly.

Signed-off-by: Merlin Mathesius <mmathesi@redhat.com>
2019-03-01 10:27:04 -06:00
Merlin Mathesius
f999d1b8fc Import json.dumps and json.loads to avoid future namespace conflicts.
Signed-off-by: Merlin Mathesius <mmathesi@redhat.com>
2019-03-01 10:27:04 -06:00
Merlin Mathesius
f2e7a8db07 Add web request options 'scratch' and 'srpms' for module scratch builds.
Signed-off-by: Merlin Mathesius <mmathesi@redhat.com>
2019-03-01 10:27:04 -06:00
Merlin Mathesius
fcac146230 Separate multipart/form-data logic from YAML handling logic.
Signed-off-by: Merlin Mathesius <mmathesi@redhat.com>
2019-03-01 10:27:04 -06:00
Jan Kaluza
1258294984 Release 2.15.0 v2.15.0 2019-03-01 15:18:23 +01:00
Jan Kaluža
93ce2da037 Merge #1154 Add simple mbs-cli client tool. 2019-03-01 12:44:45 +00:00
Jan Kaluza
d5d64f8a8a Add simple mbs-cli client tool.
The goal of this client is to allow simple access to endpoints we use
to manage MBS - like "import" module or "retire" module.

It is not intended to contain tasks available in pyrpkg.

So far only "import" module is implemented, because that's the only
command we have REST API for.
2019-03-01 12:34:03 +01:00
mprahl
8cfb7c9f99 Add a log statement to know when the poller processes a paused module build
We encountered an issue where start_next_batch_build was called twice on a
module build and it caused the module build to fail because Brew failed
one of the components since it was submitted twice. This will help us narrow
down the issue if it happens again.
2019-02-28 14:46:46 -05:00
mprahl
d5f9c5872f Return an exception to the user if no dependency combination is determined 2019-02-27 14:08:15 +00:00
Valerij Maljulin
f6a4befefe Change exception information for errors when parsing modulemd file.
This fixes #1149

Signed-off-by: Valerij Maljulin <vmaljuli@redhat.com>
2019-02-26 14:09:57 +01:00
Valerij Maljulin
687a78242e Change xmlrpclib importing way
Signed-off-by: Valerij Maljulin <vmaljuli@redhat.com>
2019-02-25 16:56:30 +01:00
Valerij Maljulin
b4d7fc4230 Handle Koji communication errors
This fixes #1105

Signed-off-by: Valerij Maljulin <vmaljuli@redhat.com>
2019-02-25 16:56:30 +01:00
Jan Kaluža
9e56b60a1b Merge #1146 Update the ModuleBuild.time_modified in case the format_mmd takes long time. 2019-02-22 06:21:01 +00:00
Jan Kaluza
efaaea66e0 Update the ModuleBuild.time_modified in case the format_mmd takes long time.
In case there is lot of components in a module build or there are some
networking issues and we need to retry the "git clone" commands, the
`format_mmd` method can take long time.

If it takes more than 10 minutes, the poller can produce fake event,
because it seems the module build is stuck. This is wrong, because
it would lead to another unexpected init handler call.

In this commit, the `format_mmd` updates the `ModuleBuild.time_modified`
method regularly to prevent poller from sending the unexpected fake
init message.
2019-02-15 15:20:34 +01:00
Valerij Maljulin
eafa93037f Make finalization before changing state to done
Signed-off-by: Valerij Maljulin <vmaljuli@redhat.com>
2019-02-15 15:19:59 +01:00
mprahl
89ccef5e42 Release v2.14.0 v2.14.0 2019-02-14 13:39:32 -05:00
mprahl
776330a2bb Add the allowed_users configuration for service accounts to bypass the group membership check
This is required for monitoring use-cases, where we can have a Kerberos principal for a
service account but no associated account in LDAP to check group membership.
2019-02-14 11:11:18 -05:00
Jan Kaluza
40f23b65e4 Fix the handling of modulemd files with unicode characters.
This commit introduces new to_text_type helper method and calls it
for return value of all mmd.dumps() calls. That way, we always
end up with proper unicode string represntation on both python
major versions.

This commit also adds unicode character to description of all
the yaml files we use in the tests so we can be sure MBS can
handle unicode characters properly.

This might be temporary fix, depending on the result of discussion
at https://github.com/fedora-modularity/libmodulemd/issues/184.
2019-02-14 13:42:47 +00:00
Jan Kaluza
7052ea0a11 Make init handler idempotent.
This fixes the issue when module build is cancelled in init state.
2019-02-14 14:18:54 +01:00
mprahl
486d644d68 Update the module submission API documentation 2019-02-12 15:55:01 -05:00
Filip Valder
38c0ea6147 Basic monitoring w/-o MBS-specific metrics 2019-02-12 16:08:41 +00:00
mprahl
afd95f3043 Don't use more-itertools 6.0.0 in Python 2 tests 2019-02-12 15:04:16 +00:00
Jan Kaluza
73d71e11b8 Revert "KojiContentGenerator: Convert MMD to UTF-8"
This reverts commit ae79b711d8.

This breaks MBS with python3 and fixes only single occurence of this
issue. We want to fix this in libmodulemd or find a better way how
to fix this to work on both python2 and python3.

We have libmodulemd PR open to address this issue on libmodulemd
level: https://github.com/fedora-modularity/libmodulemd/issues/184.
2019-02-12 15:21:04 +01:00
Igor Gnatenko
ae79b711d8 KojiContentGenerator: Convert MMD to UTF-8
Signed-off-by: Igor Gnatenko <ignatenkobrain@fedoraproject.org>
2019-02-12 08:21:05 +01:00
Jan Kaluza
130911bb49 Do not retry on 'git ls-remote' failure, but instead fallback to 'git clone'. 2019-02-11 15:27:07 +01:00
Mike Bonnet
0b3d7809f9 test pull requests under Python 3 as well 2019-02-08 17:43:43 +00:00
Mike Bonnet
b4d3c51219 don't assign the exception to a variable to make flake8 happy 2019-02-08 17:43:43 +00:00
Mike Bonnet
8479014de2 reindent to 4 characters to make flake8 happy 2019-02-08 17:43:43 +00:00
Mike Bonnet
e01b3fd077 handle different string representations of set() 2019-02-08 17:43:43 +00:00
Mike Bonnet
dcd38db5e0 hide the missing krbV module 2019-02-08 17:43:43 +00:00
Mike Bonnet
0f76fd5591 fix unicode/str/bytes inconsistencies 2019-02-08 17:43:42 +00:00
Mike Bonnet
a44e1fe4b0 fix cases where dict views could not be used in place of lists 2019-02-08 17:43:42 +00:00
Mike Bonnet
6e62cf6867 libmodulemd does not require the branch value to be bytes 2019-02-06 16:56:33 -08:00
mprahl
fce77b48dc Release v2.13.1 v2.13.1 2019-01-28 09:14:52 -05:00
Jan Kaluza
642c8406ca Allow setting context in the imported MMD file.
We always set the "context" to DEFAULT_MODULE_CONTEXT from the historical
reasons - the context was not stored in the database before and we stored
just the build_context/runtime_context. But this is no longer true for
some time.

In this commit, the context is respected and stored in the database when
importing module using the `import_mmd` method. If the context is not set
in the imported MMD, the DEFAULT_MODULE_CONTEXT is used.
2019-01-28 12:16:07 +01:00
Jan Kaluza
26d707cb8e Allow resubmiting the very same module build in case it results in new MSE build.
This is needed for Fedora branching, but it is generally useful.

For example, there is a module buildrequiring "platform:[]". In the time
this module has been built, only platform:f29 existed, so it has been built
just against platform:f29.

After a while, the platform:f30 is released and the maintainer needs
to rebuild the module against platform:f30. Right now, he needs to create
new commit in the module and submit the build, but this will result in useless
rebuild of the module also against platform:f29.

In this commit, MBS allows to resubmit the module build in a case
there are new MSE builds to build. MBS will send all the module builds
back to the user - so the existing builds will be already marked as
"ready" and the newly submitted builds will have the "init" state in
the REST API response.

However, in case when there are no new MSE builds to build, MBS still
sends back the Conflict error as it used to. This is done for backwards
compatibility and also to not confuse the users in case when no new build
has been submitted.
2019-01-28 10:53:37 +01:00
mprahl
6817102506 Release v2.13.0 v2.13.0 2019-01-24 13:23:13 -05:00
mprahl
25cda8d2bd Shutdown fedmsg-hub when SQLAlchemy can no longer resolve DNS records 2019-01-24 10:48:28 -05:00
Luiz Carvalho
5921c109f7 Add retire command to mbs-manager
With this command, admins can retire module builds that should no longer
be used as a dependency for other module builds.

Fixes #1021

Signed-off-by: Luiz Carvalho <lucarval@redhat.com>
2019-01-18 10:58:26 -05:00
mprahl
a94ad0a20b Release v2.12.2 v2.12.2 2019-01-17 09:37:53 -05:00
mprahl
5b0cb105f8 Add a missing entry to the change log 2019-01-17 09:35:20 -05:00