Commit Graph

116 Commits

Author SHA1 Message Date
Jan Kaluza
4f6d683645 Fix #173 - Handle includedmodules by building their components recursively. 2016-11-23 06:38:16 +01:00
Jan Kaluza
c719e1d0c1 Fix the buildroot_connect calls in repos and models and set 'system' to 'mock' when invoked from manage.py using build_module_locally 2016-11-22 10:54:38 +01:00
Filip Valder
88e6ddf953 configuration logic goes to config.py 2016-11-21 22:54:18 +01:00
Filip Valder
f8496379eb cosmetic changes 2016-11-21 22:54:18 +01:00
Ralph Bean
439721da99 Gather build groups from module profiles.
Fixes #181.

When initializing the buildroot for a module build, we used to set up
some build 'groups' for the tag: `build` and `srpm-build`.  These are
the lists of RPMs that koji is supposed to install into the buildroot
before anything else is done.  Crucial stuff goes here, like `git` in
the `srpm-build` group so that koji can clone the repo in the first
place.

We had those lists hardcoded before.  This list changes that to use the
`buildroot` and `srpm-buildroot` profiles of the modules which are our
dependencies (recursively).

This will allow people like @psabata and the base-runtime to make
changes to the build groups for the generational core and work around
their own problems, instead of having to ask us to expand that list.

There were a couple ways to do this:

- I could've cloned the SCM repos for all dependencies and gotten their
  profiles from the modulemd source there.  This seemed flimsy because
  we only want to depend on the profiles of modules that were *really*
  built.
- We could modify PDC to stuff the modulemd contents in there.  We
  already get some dep and tag info from PDC.  My thought here was that
  it would be too heavyweight to store every copy of the modulemd file
  in PDC for every build ever.  We already have it in MBS.
- Lastly, and this is what I did here, I just referred to MBS' own
  database to get the profiles.  This seems to work just fine.

One side-effect is that we need the build profiles from the manually
bootstrapped modules that were put together by hand, and were never
built in the MBS.  In order to work around that, I added an alembic
upgrade script which pre-populates the database with one fake
bootstrapped base-runtime module.  We can expand on that in the future
if we need.
2016-11-21 13:25:52 -05:00
Ralph Bean
e278ffa5a4 build.buildroot_task_id is not a thing.
... and it never was, as far as I can tell from the gitlog.
2016-11-21 13:25:52 -05:00
Jan Kaluza
528b6ab20a Do not submit failed components for a build 2016-11-21 13:07:18 +01:00
Jan Kaluza
9aca8057a3 Handle the exception in builder.build(...) code and do not mark whole module as failed - mark just the particular component. 2016-11-18 19:09:03 +01:00
Ralph Bean
d83f210297 Get tests of the retry decorator working in the test suite. 2016-11-18 10:19:41 -05:00
Filip Valder
8c6a5f4919 Merge #196 Fix #94 - handle IOError 2016-11-18 14:11:20 +00:00
Filip Valder
99c3c1e717 catch on IOError and retry 2016-11-18 14:37:54 +01:00
Jan Kaluža
b2b63e6a1f Merge #204 Resolve component refs on build submission. 2016-11-18 12:25:57 +00:00
Filip Valder
d8942ba44a Load defaults from config 2016-11-18 10:24:34 +01:00
Filip Valder
ca198daebf Default network timeouts 2016-11-18 10:24:34 +01:00
Filip Valder
225d116d45 lint 2016-11-18 10:22:35 +01:00
Ralph Bean
bf9e854046 Store the reason for failure. 2016-11-17 16:14:22 -05:00
Ralph Bean
411da3def5 Resolve component refs on build submission.
Fixes #203.

The modulemd files we had used to all pin the commit hashes of the
components they were pulling in.  We changed this with modulemd-1.0 and
now modules can define their components in terms of their *stream*
(a.k.a. their dist-git *branch*).  This is awesome, and is a big
stepping stone towards greater things.

The problem is that we used to take that git ref literally from the
modulemd and pass it on to koji.  Now that instead we are passing on a
branch name to koji, it falls down.  Koji requires us to send it an
explicit commit hash.

This change takes that branch name and converts it to an explicit commit
hash when the module is first submitted.

Verified that things are working again:
- http://koji.stg.fedoraproject.org/koji/taskinfo?taskID=90084980
2016-11-17 14:49:48 -05:00
Matt Prahl
efd5d203b3 Merge #184 Add Build Rate Limiting 2016-11-16 20:51:07 +00:00
Ralph Bean
a45eba41e8 Merge #194 Allow configurable mock resultsdir and backup logs after each component build 2016-11-16 20:47:18 +00:00
Matt Prahl
244b77f54c Add component build throttling 2016-11-16 15:24:17 -05:00
Matt Prahl
67d07023c5 Rename function to start_build_batch for further flexibility in the future 2016-11-16 15:24:16 -05:00
Matt Prahl
2cb7ce99c6 Add NUM_CONSECUTIVE_BUILDS configuration option 2016-11-16 15:24:16 -05:00
Matt Prahl
fb043cd2c7 Add TODO comment 2016-11-16 15:24:16 -05:00
Matt Prahl
62654f871f Fixes bug where the build tag was not returned 2016-11-16 15:24:16 -05:00
Matt Prahl
28d684331f Remove stale code from previous modulemd version 2016-11-16 15:24:16 -05:00
Ralph Bean
d6418a7de5 Merge #195 This repo_from_tag belongs to copr builder 2016-11-16 14:53:28 +00:00
Jan Kaluza
2b7ebdfc6f Fix #172 - Set ComponentBuild.batch according to mmd.buildorder attribute 2016-11-16 12:41:06 +01:00
Jakub Kadlčík
f62ea2f82e This repo_from_tag belongs to copr builder 2016-11-16 11:23:48 +01:00
Jan Kaluza
c9aff6f9e7 Allow configurable mock resultsdir and backup logs after each 2016-11-16 10:24:39 +01:00
Matt Prahl
3073fc2859 Merge #190 New-style PDC deps. 2016-11-15 15:37:03 +00:00
Ralph Bean
82ec1cd4d9 Merge branch '97_mbs_config' 2016-11-15 09:51:09 -05:00
Ralph Bean
7cb1e7cf8f Move and reformat the defaults list as described in PR review. 2016-11-15 09:50:35 -05:00
Filip Valder
15e8903d39 add: desc for configuration items 2016-11-15 13:25:14 +01:00
Jan Kaluža
a9d83b1c83 Merge #189 Some better logging around PDC stuff. 2016-11-15 09:58:58 +00:00
Filip Valder
27b76daa65 more accurate comments/exceptions 2016-11-15 10:29:49 +01:00
Filip Valder
5c6cfd657a Merge branch '97_mbs_config' of ssh://pagure.io/forks/fivaldi/fm-orchestrator into 97_mbs_config 2016-11-15 10:13:10 +01:00
Ralph Bean
7a39a8f160 New-style PDC deps.
This is only one half of the modulemd-1.0 changes needed around PDC.

This was done to get module handling their dependencies when they are
defined in terms of streams instead of NVRs.

In PDC we used to store the dep as one whole string, i.e.::

    base-runtime-0.25-11

Now we're going to store them with some more structure, since we can't
assume monotonically increasing version values::

    dependency: base-runtime
    stream: master

Pull-requests to PDC and PDC-updater are forthcoming.
2016-11-14 19:34:39 -05:00
Ralph Bean
f9b2821f7f Some better logging around PDC stuff.
I spent a while debugging these interactions tonight and the log
statements definitely helped to figure it out.
2016-11-14 19:33:25 -05:00
Ralph Bean
83bc4eb077 Add a _module_build macro flag.
Fixes #185.
2016-11-14 14:08:04 -05:00
Matt Prahl
1db4797136 Merge #177 Add support for modulemd-1.0 backward-incompatible changes like commit being replaced by ref 2016-11-14 14:34:08 +00:00
Jan Kaluza
966a73251a Add support for modulemd-1.0 backward-incompatible changes like commit being replaced by ref 2016-11-14 15:32:51 +01:00
Filip Valder
ea99293919 fix: conf variable needn't be registered in DEFAULTS to be able to use _setifok_ handler 2016-11-11 16:04:57 +01:00
Filip Valder
051cbb1d0f fix: to be more specific 2016-11-11 14:54:13 +01:00
Filip Valder
3313cbb007 convention: avoid confusion with __setitem__ 2016-11-11 14:53:43 +01:00
Filip Valder
6a026e4d5b fix: helps identifying conf var name instead of generic TypeError 2016-11-11 14:24:55 +01:00
Filip Valder
5c3a46c194 fix: reasonable defaults or taken from runtime configuration (top-level ocnfig.py) 2016-11-11 14:22:32 +01:00
Filip Valder
b9382f8e9f refactor: registering conf items is not required 2016-11-11 14:05:57 +01:00
Jakub Kadlčík
9ea879b49f Add some TODOs 2016-11-10 13:27:46 -05:00
Jakub Kadlčík
864789d1d4 Refactor __init__ method of copr builder 2016-11-10 13:27:46 -05:00
Jakub Kadlčík
fbe362bbd7 Fix issue with creating project for group 2016-11-10 13:27:46 -05:00