From 2e2856169ad82e31593996485410ae6a0ad80649 Mon Sep 17 00:00:00 2001 From: Filip Valder Date: Fri, 25 Aug 2017 14:45:42 +0200 Subject: [PATCH 1/2] Update and reformat README, CONTRIBUTING guide and sync code comments --- README.rst | 161 ++++++++++++++++++++++++--------- docs/CONTRIBUTING.rst | 10 +- module_build_service/models.py | 39 +++++--- 3 files changed, 152 insertions(+), 58 deletions(-) diff --git a/README.rst b/README.rst index 3b092e00..0aadb854 100644 --- a/README.rst +++ b/README.rst @@ -14,16 +14,75 @@ tasks: - Emitting bus messages about all state changes so that other infrastructure services can pick up the work. -Development -=========== +Supported build systems +======================= -For help on setting up a development environment, see `docs/CONTRIBUTING.rst`. +Koji +---- + +Koji is the software that builds RPM packages from source tarballs and +SPEC files. It uses its own Mock to create chroot environments to +perform builds. + +MBS comes with its own ``koji.conf`` config file which allows configuring +for your custom Koji instance(s). + +Mock +---- + +Mock is a simple program that will build source RPMs inside a chroot. It +doesn't do anything terribly fancy other than populate a chroot with the +contents specified by a configuration file, then build any input SRPM(s) +in that chroot. + +MBS is able to perform local module builds by directing local Mock. + +MBS supports threaded Mock builds which utilizes performance and +significantly speeds up local module builds. + +Copr +---- + +Copr is yet another an easy-to-use automatic build system providing a +package repository as its output. + +As with Koji, MBS comes with its own ``copr.conf`` config file which allows +altering your default Copr configuration. + +_`Client tooling` +================= + +``mbs-build`` +------------- + +This command submits and manages module builds. + +The most frequently used subcommand would be 'submit'. After providing +access credentials, a module build is passed to a preconfigured +MBS instanace. When 'scm_url' or 'branch' is not set, it presumes you +are executing this command in a directory with a cloned git repository +containing a module prescript. The same approach is used in the case of +local module submission. + +Other subcommands allow local module submission, watching module builds, +canceling them etc. For more info, there's an existing help available. + +``mbs-manager`` +--------------- + +This command controls the MBS instance itself. + +There are subcommands for running the MBS server, performing database +migrations, generating certificates and submitting local module +builds. For more info, there's an existing help available. Client-side API =============== -The orchestrator implements a RESTful interface for module build submission and -state querying. Not all REST methods are supported. See below for details. +The orchestrator implements a RESTful interface for module build +submission and state querying. Not all REST methods are supported. See +below for details. For client tooling which utilizes the API, please +refer to `Client tooling`_ section. Module build submission ----------------------- @@ -53,14 +112,15 @@ The response, in case of a successful submission, would include the task ID. } -When ``YAML_SUBMIT_ALLOWED`` is enabled, it is also possible to submit raw modulemd yaml file by sending -``multipart/form-data`` request with input file named as ``yaml``. +When ``YAML_SUBMIT_ALLOWED`` is enabled, it is also possible to submit +raw modulemd yaml file by sending ``multipart/form-data`` request with +input file named as ``yaml``. Module build state query ------------------------ Once created, the client can query the current build state by requesting the -build task's URL. Querying the BPO service might be preferred, however. +build task's URL. Querying the BPO service might be preferred, however. :: @@ -98,9 +158,9 @@ about the referenced build task. ... } -"id" is the ID of the task. "state" refers to the orchestrator module +"id" is the ID of the task. "state" refers to the orchestrator module build state and might be one of "init", "wait", "build", "done", "failed" or -"ready". "tasks" is a dictionary of information about the individual component +"ready". "tasks" is a dictionary of information about the individual component builds including their IDs in the backend buildsystem, their state, a reason for their state, and the NVR (if known). @@ -119,12 +179,12 @@ querying the "module-builds" resource. There are a number of configurable GET parameters to change how the module builds are displayed. These parameters are: -- `verbose` - Shows the builds with the same amount of detail as querying - them individually (i.e. verbose=True). This value defaults to False. -- `page` - Specifies which page should be displayed (e.g. page=3). This +- ``verbose`` - Shows the builds with the same amount of detail as querying + them individually (i.e. ``verbose=True``). This value defaults to False. +- ``page`` - Specifies which page should be displayed (e.g. ``page=3``). This value defaults to 1. -- `per_page` - Specifies how many items per page should be displayed - (e.g. per_page=20). This value defaults to 10. +- ``per_page`` - Specifies how many items per page should be displayed + (e.g. ``per_page=20``). This value defaults to 10. An example of querying the "module-builds" resource without any additional parameters:: @@ -293,21 +353,26 @@ Filtering module builds The module-builds can be filtered by a variety of GET parameters. These paramters are: -- owner - Shows builds submitted by a particular user (e.g. owner=mprahl) -- state - Shows builds in a particular state (can be the state name or - the state ID) (e.g. state=done) -- submitted_before - Shows builds that were submitted before a particular - Zulu ISO 8601 timestamp (e.g. submitted_before=2016-08-23T09:40:07Z) -- submitted_after - Shows builds that were submitted after a particular - Zulu ISO 8601 timestamp (e.g. submitted_after=2016-08-22T09:40:07Z) -- modified_before - Shows builds that were modified before a particular - Zulu ISO 8601 timestamp (e.g. modified_before=2016-08-23T09:40:07Z) -- modified_after - Shows builds that were modified after a particular - Zulu ISO 8601 timestamp (e.g. modified_after=2016-08-22T09:40:07Z) -- completed_before - Shows builds that were completed before a particular - Zulu ISO 8601 timestamp (e.g. completed_before=2016-08-22T09:40:07Z) -- completed_after - Shows builds that were completed after a particular - Zulu ISO 8601 timestamp (e.g. completed_after=2016-08-23T09:40:07Z) +- ``name`` - Shows builds of modules with a particular name (e.g. + ``name=testmodule``) +- ``koji_tag`` - Shows builds tagged with a particular Koji tag (e.g. + ``koji_tag=module-984ed60dd37b9361``) +- ``owner`` - Shows builds submitted by a particular user (e.g. + ``owner=mprahl``) +- ``state`` - Shows builds in a particular state (can be the state name or + the state ID) (e.g. ``state=done``) +- ``submitted_before`` - Shows builds that were submitted before a particular + Zulu ISO 8601 timestamp (e.g. ``submitted_before=2016-08-23T09:40:07Z``) +- ``submitted_after`` - Shows builds that were submitted after a particular + Zulu ISO 8601 timestamp (e.g. ``submitted_after=2016-08-22T09:40:07Z``) +- ``modified_before`` - Shows builds that were modified before a particular + Zulu ISO 8601 timestamp (e.g. ``modified_before=2016-08-23T09:40:07Z``) +- ``modified_after`` - Shows builds that were modified after a particular + Zulu ISO 8601 timestamp (e.g. ``modified_after=2016-08-22T09:40:07Z``) +- ``completed_before`` - Shows builds that were completed before a particular + Zulu ISO 8601 timestamp (e.g. ``completed_before=2016-08-22T09:40:07Z``) +- ``completed_after`` - Shows builds that were completed after a particular + Zulu ISO 8601 timestamp (e.g. ``completed_after=2016-08-23T09:40:07Z``) An example of querying the "module-builds" resource with the "state", and the "submitted_before" parameters:: @@ -352,6 +417,7 @@ Possible response codes are for various requests include: - HTTP 200 OK - The task exists and the query was successful. - HTTP 201 Created - The module build task was successfully created. - HTTP 400 Bad Request - The client's input isn't a valid request. +- HTTP 401 Unauthorized - No 'authorization' header found. - HTTP 403 Forbidden - The SCM URL is not pointing to a whitelisted SCM server. - HTTP 404 Not Found - The requested URL has no handler associated with it or the requested resource doesn't exist. @@ -378,11 +444,11 @@ init This is (obviously) the first state a module build enters. -When a user first submits a module build, it enters this state. We parse the +When a user first submits a module build, it enters this state. We parse the modulemd file, learn the NVR, and create a record for the module build. Then, we validate that the components are available, and that we can fetch -them. If this is all good, then we set the build to the 'wait' state. If +them. If this is all good, then we set the build to the 'wait' state. If anything goes wrong, we jump immediately to the 'failed' state. wait @@ -395,7 +461,7 @@ builds for the build system to handle. build ~~~~~ -The scheduler works on builds in this state. We prepare the buildroot, submit +The scheduler works on builds in this state. We prepare the buildroot, submit builds for all the components, and wait for the results to come back. done @@ -413,12 +479,18 @@ ready ~~~~~ This is a state to be set when a module is ready to be part of a -larger compose. perhaps it is set by an external service that knows +larger compose. perhaps it is set by an external service that knows about the Grand Plan. Bus messages ============ +Supported messaging backends: + +- fedmsg - Federated Messaging with ZeroMQ +- in_memory - Local/internal messaging only +- amq - Apache ActiveMQ + Message Topic ------------- @@ -472,18 +544,23 @@ the following rules (all of them are evaluated from top to bottom): they are used for configuration. Following environment variables are recognized: - - `MBS_CONFIG_FILE`: Overrides default configuration file location, - typically `/etc/module-build-service/config.py`. - - `MBS_CONFIG_SECTION`: Overrides configuration section. + - ``MBS_CONFIG_FILE``: Overrides default configuration file location, + typically ``/etc/module-build-service/config.py``. + - ``MBS_CONFIG_SECTION``: Overrides configuration section. - It is possible to set these values in httpd using `SetEnv`, - anywhere in `/etc/profile.d/` etc. + It is possible to set these values in httpd using ``SetEnv``, + anywhere in ``/etc/profile.d/`` etc. - If test-runtime environment is detected, TestConfiguration is used, otherwise... -- if `MODULE_BUILD_SERVICE_DEVELOPER_ENV` is set to some reasonable - value, DevConfiguration is forced and `config.py` is used directly from the - MBS's develop instance. For more information see `docs/CONTRIBUTING.rst`. +- if ``MODULE_BUILD_SERVICE_DEVELOPER_ENV`` is set to some reasonable + value, DevConfiguration is forced and ``config.py`` is used directly from the + MBS's develop instance. For more information see ``docs/CONTRIBUTING.rst``. + +Development +=========== + +For help on setting up a development environment, see ``docs/CONTRIBUTING.rst``. License ======= diff --git a/docs/CONTRIBUTING.rst b/docs/CONTRIBUTING.rst index 4f7215e9..629aa922 100644 --- a/docs/CONTRIBUTING.rst +++ b/docs/CONTRIBUTING.rst @@ -196,11 +196,10 @@ Module Submission You can submit a local test build with the `contrib/mbs-build` script, which should submit an HTTP POST to the frontend, requesting a build:: - $ ./contrib/mbs-build -s [server] submit [scm_url] [branch] Here, `server` should specify the `hostname[:port]` port of the MBS instance -you want to submit to. For local development, try `https://127.0.0.1:5000`. +you want to submit to. For local development, try `https://127.0.0.1:5000`. The `scmurl` should be a url to a dist-git repo of the module in question and the `branch` should be the stream that you want to build. Note that @@ -234,6 +233,13 @@ Create a directory to house the fedmsg cache:: Then uncomment the fedmsg signing configuration in `fedmsg.d/module_build_service.py`. +PEP 8 +===== + +Following PEP 8 is highly recommended and all patches and future code +changes shall be PEP 8 compliant to keep at least constant or decreasing +number of PEP 8 violations. + Historical Names of Module Build Service ======================================== diff --git a/module_build_service/models.py b/module_build_service/models.py index 0fe5f3a2..0fc7efe1 100644 --- a/module_build_service/models.py +++ b/module_build_service/models.py @@ -41,26 +41,37 @@ from sqlalchemy.orm import lazyload # Just like koji.BUILD_STATES, except our own codes for modules. BUILD_STATES = { - # When you parse the modulemd file and know the nvr and you create a - # record in the db, and that's it. - # publish the message - # validate that components are available - # and that you can fetch them. - # if all is good, go to wait: telling module_build_service_daemon to take over. - # if something is bad, go straight to failed. + # This is (obviously) the first state a module build enters. + # + # When a user first submits a module build, it enters this state. We parse + # the modulemd file, learn the NVR, create a record for the module build. + # and publish the message. + # + # Then, we validate that the components are available, and that we can + # fetch them. If this is all good, then we set the build to the 'wait' + # state. If anything goes wrong, we jump immediately to the 'failed' state. "init": 0, - # Here, the scheduler picks up tasks in wait. - # switch to build immediately. - # throttling logic (when we write it) goes here. + + # Here, the scheduler picks up tasks in wait and switches to build + # immediately. Eventually, we'll add throttling logic here so we don't + # submit too many builds for the build system to handle "wait": 1, - # Actively working on it. + + # The scheduler works on builds in this state. We prepare the buildroot, + # submit builds for all the components, and wait for the results to come + # back. "build": 2, - # All is good + + # Once all components have succeeded, we set the top-level module build + # to 'done'. "done": 3, - # Something failed + + # If any of the component builds fail, then we set the top-level module + # build to 'failed' also. "failed": 4, + # This is a state to be set when a module is ready to be part of a - # larger compose. perhaps it is set by an external service that knows + # larger compose. perhaps it is set by an external service that knows # about the Grand Plan. "ready": 5, } From 5b7be25558507e067a15031a7d802daa5b09f619 Mon Sep 17 00:00:00 2001 From: Filip Valder Date: Fri, 25 Aug 2017 15:01:17 +0200 Subject: [PATCH 2/2] Add 'Listing component builds' subsection to the API section --- README.rst | 42 ++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 42 insertions(+) diff --git a/README.rst b/README.rst index 0aadb854..8f672745 100644 --- a/README.rst +++ b/README.rst @@ -409,6 +409,48 @@ and the "submitted_before" parameters:: "total": 3 } +Listing component builds +------------------------ + +An example of querying the "component-builds" resource without any additional +parameters:: + + GET /module-build-service/1/component-builds/ + +:: + + HTTP 200 OK + +:: + + { + "items": [ + { + "id": 854, + "state": 1 + }, + { + "id": 107, + "state": 1 + }, + { + "id": 104, + "state": 1 + }, + .... + ], + "meta": { + "first": "https://127.0.0.1:5000/module-build-service/1/component-builds/?per_page=10&page=1", + "last": "https://127.0.0.1:5000/module-build-service/1/component-builds/?per_page=10&page=4237", + "next": "https://127.0.0.1:5000/module-build-service/1/component-builds/?per_page=10&page=2", + "page": 1, + "pages": 4237, + "per_page": 10, + "prev": null, + "total": 42366 + } + } + HTTP Response Codes -------------------