modules/fm-orchestrator
1
0
Fork 0
mirror of https://pagure.io/fm-orchestrator.git synced 2026-04-04 03:08:21 +08:00
Code Issues Actions Packages Projects Releases Wiki Activity

Merge #648 Update and reformat README, CONTRIBUTING guide and sync code comments

Browse Source
This commit is contained in:
Jan Kaluža
2017-08-28 19:19:34 +00:00
parent 73031fd15b 5b7be25558
commit af6ef0d352
3 changed files with 194 additions and 58 deletions
Download Patch File Download Diff File Expand all files Collapse all files

203
README.rst
View File

@@ -14,16 +14,75 @@ tasks:
- Emitting bus messages about all state changes so that other infrastructure
services can pick up the work.
Development
===========
Supported build systems
=======================
For help on setting up a development environment, see `docs/CONTRIBUTING.rst`.
Koji
----
Koji is the software that builds RPM packages from source tarballs and
SPEC files. It uses its own Mock to create chroot environments to
perform builds.
MBS comes with its own ``koji.conf`` config file which allows configuring
for your custom Koji instance(s).
Mock
----
Mock is a simple program that will build source RPMs inside a chroot. It
doesn't do anything terribly fancy other than populate a chroot with the
contents specified by a configuration file, then build any input SRPM(s)
in that chroot.
MBS is able to perform local module builds by directing local Mock.
MBS supports threaded Mock builds which utilizes performance and
significantly speeds up local module builds.
Copr
----
Copr is yet another an easy-to-use automatic build system providing a
package repository as its output.
As with Koji, MBS comes with its own ``copr.conf`` config file which allows
altering your default Copr configuration.
_`Client tooling`
=================
``mbs-build``
-------------
This command submits and manages module builds.
The most frequently used subcommand would be 'submit'. After providing
access credentials, a module build is passed to a preconfigured
MBS instanace. When 'scm_url' or 'branch' is not set, it presumes you
are executing this command in a directory with a cloned git repository
containing a module prescript. The same approach is used in the case of
local module submission.
Other subcommands allow local module submission, watching module builds,
canceling them etc. For more info, there's an existing help available.
``mbs-manager``
---------------
This command controls the MBS instance itself.
There are subcommands for running the MBS server, performing database
migrations, generating certificates and submitting local module
builds. For more info, there's an existing help available.
Client-side API
===============
The orchestrator implements a RESTful interface for module build submission and
state querying. Not all REST methods are supported. See below for details.
The orchestrator implements a RESTful interface for module build
submission and state querying. Not all REST methods are supported. See
below for details. For client tooling which utilizes the API, please
refer to `Client tooling`_ section.
Module build submission
-----------------------
@@ -53,14 +112,15 @@ The response, in case of a successful submission, would include the task ID.
}
When ``YAML_SUBMIT_ALLOWED`` is enabled, it is also possible to submit raw modulemd yaml file by sending
``multipart/form-data`` request with input file named as ``yaml``.
When ``YAML_SUBMIT_ALLOWED`` is enabled, it is also possible to submit
raw modulemd yaml file by sending ``multipart/form-data`` request with
input file named as ``yaml``.
Module build state query
------------------------
Once created, the client can query the current build state by requesting the
build task's URL. Querying the BPO service might be preferred, however.
build task's URL. Querying the BPO service might be preferred, however.
::
@@ -98,9 +158,9 @@ about the referenced build task.
...
}
"id" is the ID of the task. "state" refers to the orchestrator module
"id" is the ID of the task. "state" refers to the orchestrator module
build state and might be one of "init", "wait", "build", "done", "failed" or
"ready". "tasks" is a dictionary of information about the individual component
"ready". "tasks" is a dictionary of information about the individual component
builds including their IDs in the backend buildsystem, their state, a reason
for their state, and the NVR (if known).
@@ -119,12 +179,12 @@ querying the "module-builds" resource.
There are a number of configurable GET parameters to change how the
module builds are displayed. These parameters are:
- `verbose` - Shows the builds with the same amount of detail as querying
them individually (i.e. verbose=True). This value defaults to False.
- `page` - Specifies which page should be displayed (e.g. page=3). This
- ``verbose`` - Shows the builds with the same amount of detail as querying
them individually (i.e. ``verbose=True``). This value defaults to False.
- ``page`` - Specifies which page should be displayed (e.g. ``page=3``). This
value defaults to 1.
- `per_page` - Specifies how many items per page should be displayed
(e.g. per_page=20). This value defaults to 10.
- ``per_page`` - Specifies how many items per page should be displayed
(e.g. ``per_page=20``). This value defaults to 10.
An example of querying the "module-builds" resource without any additional
parameters::
@@ -293,21 +353,26 @@ Filtering module builds
The module-builds can be filtered by a variety of GET parameters. These
paramters are:
- owner - Shows builds submitted by a particular user (e.g. owner=mprahl)
- state - Shows builds in a particular state (can be the state name or
the state ID) (e.g. state=done)
- submitted_before - Shows builds that were submitted before a particular
Zulu ISO 8601 timestamp (e.g. submitted_before=2016-08-23T09:40:07Z)
- submitted_after - Shows builds that were submitted after a particular
Zulu ISO 8601 timestamp (e.g. submitted_after=2016-08-22T09:40:07Z)
- modified_before - Shows builds that were modified before a particular
Zulu ISO 8601 timestamp (e.g. modified_before=2016-08-23T09:40:07Z)
- modified_after - Shows builds that were modified after a particular
Zulu ISO 8601 timestamp (e.g. modified_after=2016-08-22T09:40:07Z)
- completed_before - Shows builds that were completed before a particular
Zulu ISO 8601 timestamp (e.g. completed_before=2016-08-22T09:40:07Z)
- completed_after - Shows builds that were completed after a particular
Zulu ISO 8601 timestamp (e.g. completed_after=2016-08-23T09:40:07Z)
- ``name`` - Shows builds of modules with a particular name (e.g.
``name=testmodule``)
- ``koji_tag`` - Shows builds tagged with a particular Koji tag (e.g.
``koji_tag=module-984ed60dd37b9361``)
- ``owner`` - Shows builds submitted by a particular user (e.g.
``owner=mprahl``)
- ``state`` - Shows builds in a particular state (can be the state name or
the state ID) (e.g. ``state=done``)
- ``submitted_before`` - Shows builds that were submitted before a particular
Zulu ISO 8601 timestamp (e.g. ``submitted_before=2016-08-23T09:40:07Z``)
- ``submitted_after`` - Shows builds that were submitted after a particular
Zulu ISO 8601 timestamp (e.g. ``submitted_after=2016-08-22T09:40:07Z``)
- ``modified_before`` - Shows builds that were modified before a particular
Zulu ISO 8601 timestamp (e.g. ``modified_before=2016-08-23T09:40:07Z``)
- ``modified_after`` - Shows builds that were modified after a particular
Zulu ISO 8601 timestamp (e.g. ``modified_after=2016-08-22T09:40:07Z``)
- ``completed_before`` - Shows builds that were completed before a particular
Zulu ISO 8601 timestamp (e.g. ``completed_before=2016-08-22T09:40:07Z``)
- ``completed_after`` - Shows builds that were completed after a particular
Zulu ISO 8601 timestamp (e.g. ``completed_after=2016-08-23T09:40:07Z``)
An example of querying the "module-builds" resource with the "state",
and the "submitted_before" parameters::
@@ -344,6 +409,48 @@ and the "submitted_before" parameters::
"total": 3
}
Listing component builds
------------------------
An example of querying the "component-builds" resource without any additional
parameters::
GET /module-build-service/1/component-builds/
::
HTTP 200 OK
::
{
"items": [
{
"id": 854,
"state": 1
},
{
"id": 107,
"state": 1
},
{
"id": 104,
"state": 1
},
....
],
"meta": {
"first": "https://127.0.0.1:5000/module-build-service/1/component-builds/?per_page=10&page=1",
"last": "https://127.0.0.1:5000/module-build-service/1/component-builds/?per_page=10&page=4237",
"next": "https://127.0.0.1:5000/module-build-service/1/component-builds/?per_page=10&page=2",
"page": 1,
"pages": 4237,
"per_page": 10,
"prev": null,
"total": 42366
}
}
HTTP Response Codes
-------------------
@@ -352,6 +459,7 @@ Possible response codes are for various requests include:
- HTTP 200 OK - The task exists and the query was successful.
- HTTP 201 Created - The module build task was successfully created.
- HTTP 400 Bad Request - The client's input isn't a valid request.
- HTTP 401 Unauthorized - No 'authorization' header found.
- HTTP 403 Forbidden - The SCM URL is not pointing to a whitelisted SCM server.
- HTTP 404 Not Found - The requested URL has no handler associated with it or
the requested resource doesn't exist.
@@ -378,11 +486,11 @@ init
This is (obviously) the first state a module build enters.
When a user first submits a module build, it enters this state. We parse the
When a user first submits a module build, it enters this state. We parse the
modulemd file, learn the NVR, and create a record for the module build.
Then, we validate that the components are available, and that we can fetch
them. If this is all good, then we set the build to the 'wait' state. If
them. If this is all good, then we set the build to the 'wait' state. If
anything goes wrong, we jump immediately to the 'failed' state.
wait
@@ -395,7 +503,7 @@ builds for the build system to handle.
build
~~~~~
The scheduler works on builds in this state. We prepare the buildroot, submit
The scheduler works on builds in this state. We prepare the buildroot, submit
builds for all the components, and wait for the results to come back.
done
@@ -413,12 +521,18 @@ ready
~~~~~
This is a state to be set when a module is ready to be part of a
larger compose. perhaps it is set by an external service that knows
larger compose. perhaps it is set by an external service that knows
about the Grand Plan.
Bus messages
============
Supported messaging backends:
- fedmsg - Federated Messaging with ZeroMQ
- in_memory - Local/internal messaging only
- amq - Apache ActiveMQ
Message Topic
-------------
@@ -472,18 +586,23 @@ the following rules (all of them are evaluated from top to bottom):
they are used for configuration. Following environment variables are
recognized:
- `MBS_CONFIG_FILE`: Overrides default configuration file location,
typically `/etc/module-build-service/config.py`.
- `MBS_CONFIG_SECTION`: Overrides configuration section.
- ``MBS_CONFIG_FILE``: Overrides default configuration file location,
typically ``/etc/module-build-service/config.py``.
- ``MBS_CONFIG_SECTION``: Overrides configuration section.
It is possible to set these values in httpd using `SetEnv`,
anywhere in `/etc/profile.d/` etc.
It is possible to set these values in httpd using ``SetEnv``,
anywhere in ``/etc/profile.d/`` etc.
- If test-runtime environment is detected,
TestConfiguration is used, otherwise...
- if `MODULE_BUILD_SERVICE_DEVELOPER_ENV` is set to some reasonable
value, DevConfiguration is forced and `config.py` is used directly from the
MBS's develop instance. For more information see `docs/CONTRIBUTING.rst`.
- if ``MODULE_BUILD_SERVICE_DEVELOPER_ENV`` is set to some reasonable
value, DevConfiguration is forced and ``config.py`` is used directly from the
MBS's develop instance. For more information see ``docs/CONTRIBUTING.rst``.
Development
===========
For help on setting up a development environment, see ``docs/CONTRIBUTING.rst``.
License
=======

10
docs/CONTRIBUTING.rst
View File

@@ -196,11 +196,10 @@ Module Submission
You can submit a local test build with the `contrib/mbs-build` script,
which should submit an HTTP POST to the frontend, requesting a build::
$ ./contrib/mbs-build -s [server] submit [scm_url] [branch]
Here, `server` should specify the `hostname[:port]` port of the MBS instance
you want to submit to. For local development, try `https://127.0.0.1:5000`.
you want to submit to. For local development, try `https://127.0.0.1:5000`.
The `scmurl` should be a url to a dist-git repo of the module in question and
the `branch` should be the stream that you want to build. Note that
@@ -234,6 +233,13 @@ Create a directory to house the fedmsg cache::
Then uncomment the fedmsg signing configuration in
`fedmsg.d/module_build_service.py`.
PEP 8
=====
Following PEP 8 is highly recommended and all patches and future code
changes shall be PEP 8 compliant to keep at least constant or decreasing
number of PEP 8 violations.
Historical Names of Module Build Service
========================================

39
module_build_service/models.py
View File

@@ -41,26 +41,37 @@ from sqlalchemy.orm import lazyload
# Just like koji.BUILD_STATES, except our own codes for modules.
BUILD_STATES = {
# When you parse the modulemd file and know the nvr and you create a
# record in the db, and that's it.
# publish the message
# validate that components are available
# and that you can fetch them.
# if all is good, go to wait: telling module_build_service_daemon to take over.
# if something is bad, go straight to failed.
# This is (obviously) the first state a module build enters.
#
# When a user first submits a module build, it enters this state. We parse
# the modulemd file, learn the NVR, create a record for the module build.
# and publish the message.
#
# Then, we validate that the components are available, and that we can
# fetch them. If this is all good, then we set the build to the 'wait'
# state. If anything goes wrong, we jump immediately to the 'failed' state.
"init": 0,
# Here, the scheduler picks up tasks in wait.
# switch to build immediately.
# throttling logic (when we write it) goes here.
# Here, the scheduler picks up tasks in wait and switches to build
# immediately. Eventually, we'll add throttling logic here so we don't
# submit too many builds for the build system to handle
"wait": 1,
# Actively working on it.
# The scheduler works on builds in this state. We prepare the buildroot,
# submit builds for all the components, and wait for the results to come
# back.
"build": 2,
# All is good
# Once all components have succeeded, we set the top-level module build
# to 'done'.
"done": 3,
# Something failed
# If any of the component builds fail, then we set the top-level module
# build to 'failed' also.
"failed": 4,
# This is a state to be set when a module is ready to be part of a
# larger compose. perhaps it is set by an external service that knows
# larger compose. perhaps it is set by an external service that knows
# about the Grand Plan.
"ready": 5,
}