mirror of
https://pagure.io/fm-orchestrator.git
synced 2026-04-02 02:11:19 +08:00
Minor adjustments.
This commit is contained in:
Ralph Bean
@@ -1,7 +1,7 @@
|
||||
How does MBS build modules?
|
||||
===========================
|
||||
|
||||
This document describes how are modules built internally in MBS. The goal of this document is
|
||||
This document describes how modules are built internally in MBS. The goal of this document is
|
||||
to explain code-flow of module builds. It assumes everything goes as expected and does not
|
||||
mention any error handling or corner cases.
|
||||
|
||||
@@ -9,10 +9,10 @@ mention any error handling or corner cases.
|
||||
User submits module build request
|
||||
---------------------------------
|
||||
|
||||
There is MBS frontend providing REST API (See `views.py`). User sends POST request with JSON
|
||||
describing the module to build. There is mainly the URL to git repository (called `scmurl`)
|
||||
and branch name (called `branch`). The `scmurl` points to git repository containing the
|
||||
modulemd file defining the module.
|
||||
There is the MBS frontend, which provides a REST API (See `views.py`). A user sends a POST request
|
||||
with JSON describing the module to build. There is mainly the URL to the git repository (called
|
||||
`scmurl`) and branch name (called `branch`). The `scmurl` points to the git repository containing
|
||||
the modulemd file defining the module.
|
||||
|
||||
This JSON data is handled by `views.SCMHandler`, which validates the JSON and calls
|
||||
`utils.submit.submit_module_build_from_scm(...)` method. This goes down to
|
||||
@@ -24,15 +24,15 @@ Module Stream Expansion (MSE)
|
||||
|
||||
The first thing done in `submit_module_build(...)` is Module Stream Expansion (MSE).
|
||||
|
||||
The submitted modulemd file might have buildrequires and requires pairs defined in ambigous way.
|
||||
For example the module can buildrequire `platform:[28, 29]` modules, which means it should
|
||||
The submitted modulemd file might have buildrequires and requires pairs defined in an ambiguous way.
|
||||
For example the module can buildrequire `platform:[f28, f29]` modules, which means it should
|
||||
be built against the `f28` and `f29` streams of `platform` module.
|
||||
|
||||
The process of resolving these ambigous buildrequires and requires is called Module Stream
|
||||
The process of resolving these ambiguous buildrequires and requires is called Module Stream
|
||||
Expansion.
|
||||
|
||||
Input for this process is the submitted modulemd file with ambigous buildrequires/requires.
|
||||
Output of this process is list of multiple modulemd files with all the ambigous
|
||||
Input for this process is the submitted modulemd file with ambiguous buildrequires/requires.
|
||||
Output of this process is the list of multiple modulemd files with all the ambiguous
|
||||
buildrequires/requires resolved.
|
||||
|
||||
This all happens in `utils.mse.generate_expanded_mmds(...)` method.
|
||||
@@ -41,74 +41,81 @@ At first, this method finds out all the possible buildrequires/requires for the
|
||||
This is done using `DBResolver` which simply finds out the modules in the MBS database.
|
||||
In our case, it would list all the `platform:f28` and `platform:f29` modules.
|
||||
|
||||
It then uses `MMDResolver` class to find all the possible combinations of buildrequires/requires
|
||||
for which the input module can be built.
|
||||
It then uses the `MMDResolver` class to find all the possible combinations of buildrequires/requires
|
||||
against which the input module can be built.
|
||||
|
||||
In our case, it would generate two expanded modulemd files (one for each platform stream) which
|
||||
would be identical to input modulemd file with only following exceptions:
|
||||
would be identical to the input modulemd file with only the following exceptions:
|
||||
|
||||
- The buildrequires/requires pairs from input modulemd files will be replaced by the particular
|
||||
- The buildrequires/requires pairs from the input modulemd files will be replaced by the particular
|
||||
combination returned by `MMDResolver`
|
||||
- The `xmd` section of generated modulemd files will contain `buildrequires` list which lists all
|
||||
the modules required to build this expanded modulemd file. This is used later by MBS.
|
||||
the modules required to build this expanded modulemd file. Requirements are traversed to produce
|
||||
a transitive list that includes all `requires` of each `buildrequires` entry. This is used later
|
||||
by MBS.
|
||||
- The context is computed and filled for each expanded modulemd file. It is based on the
|
||||
expanded buildrequires and requires pairs. See `models.ModuleBuild.contexts_from_mmd(...)`.
|
||||
|
||||
Such expanded modulemd files are then added to database as next step in `submit_module_build(...)`
|
||||
and are handled as separate module builds later in MBS.
|
||||
Such expanded modulemd files are then added to database as the next step in
|
||||
`submit_module_build(...)` and are handled as separate module builds later in MBS.
|
||||
|
||||
The `submit_module_build(...)` then moves the module builds to "init" state and sends message to
|
||||
fedmsg hub or UMB for each submitted expanded module build
|
||||
The `submit_module_build(...)` then moves the module builds to "init" state and sends a message on
|
||||
the configured message bus.
|
||||
|
||||
|
||||
Backend handles module moved to "init" state
|
||||
--------------------------------------------
|
||||
Backend handles module moved to the "init" state
|
||||
------------------------------------------------
|
||||
|
||||
When module build is moved to "init" state, backend handles that in
|
||||
When module build is moved to the "init" state, the backend handles that in the
|
||||
`scheduler.handlers.modules.init(...)` method.
|
||||
|
||||
This method calls `utils.submit.record_component_builds` which reads the modulemd file
|
||||
stored in database by frontend and records all the components (future RPM packages) in the
|
||||
stored in the database by the frontend and records all the components (future RPM packages) in the
|
||||
database.
|
||||
|
||||
The components are divided into the **batches** based on their buildorder in the modulemd file.
|
||||
|
||||
Once the components which are supposed to be built as part of this module build are recorded,
|
||||
the module moves to "wait" state and another message it sent to the message bus.
|
||||
the module moves to the "wait" state and another message is sent on the message bus.
|
||||
|
||||
|
||||
Backend handles module moved to "wait" state
|
||||
--------------------------------------------
|
||||
Backend handles module moved to the "wait" state
|
||||
------------------------------------------------
|
||||
|
||||
When module build is moved to "wait" state, backend handles that in
|
||||
When the module build is moved to the "wait" state, the backend handles that in the
|
||||
`scheduler.handlers.modules.wait(...)` method.
|
||||
|
||||
At first, this method uses KojiModuleBuilder to generate Koji tag in which the components will be
|
||||
build. The Koji tag reflects the buildrequires of module by inheriting their Koji tags. In our
|
||||
case, the Koji tag would inherit just `platform:f28` or `platform:f29` Koji tag, because that's
|
||||
At first, this method uses KojiModuleBuilder to generate the Koji tag in which the components will be
|
||||
built. The Koji tag reflects the buildrequires of the module by inheriting their Koji tags. In our
|
||||
case, the Koji tag would inherit just the `platform:f28` or `platform:f29` Koji tag, because that's
|
||||
the only buildrequired module we have.
|
||||
The list of modules buildrequired by currently building module is get from `buildrequires` list in
|
||||
the `xmd` section of expanded modulemd file.
|
||||
The list of modules buildrequired by the currently building module is determined by the `buildrequires` list in
|
||||
the `xmd` section of the expanded modulemd file.
|
||||
|
||||
Once the Koji tag is ready, it tries to build `module-build-macros` package. This package contains
|
||||
special build macros which for example defines the dist-tag for built RPMs, ensures that filtered
|
||||
packages are not installed into the buildroot and so on.
|
||||
Once the Koji tag is ready, it tries to build a synthetic `module-build-macros` package. This
|
||||
package contains special build macros which for example, define the dist-tag for built RPMs, ensure
|
||||
that filtered packages are not installed into the buildroot and so on.
|
||||
|
||||
The module-build-macros is always built in first batch.
|
||||
The module-build-macros package is always built in the first batch.
|
||||
|
||||
|
||||
Module-build-macros package is built
|
||||
------------------------------------
|
||||
|
||||
Once the module-build-macros package is built, Koji sends message to message hub which is handled
|
||||
in `scheduler.handlers.components.complete(...)` method.
|
||||
Once the `module-build-macros` package is built, Koji sends a message on the message bus, which is
|
||||
handled in the `scheduler.handlers.components.complete(...)` method.
|
||||
|
||||
This method changes that state of component build in MBS database to "complete".
|
||||
This method changes the state of that component build in the MBS database to "complete".
|
||||
|
||||
It then checks if there are any other unbuilt components in current batch. Because the
|
||||
"module-build-macros" is the only component in batch 1, it can continue tagging it
|
||||
into the Koji tag representing the module, so the module-build-macros can be later
|
||||
installed during the build of next components and can influence them.
|
||||
It then checks if there are any other unbuilt components in the current batch. Because the
|
||||
`module-build-macros` package is the only component in batch 1, it can continue tagging it
|
||||
into the Koji tag representing the module, so the `module-build-macros` can be later
|
||||
installed during the build of the next batch of components and can influence them.
|
||||
|
||||
Note that the `module-build-macros` package is the only package in the course of a module build that
|
||||
*only* gets tagged into the build tag. All other builds are tagged both into the build tag (to
|
||||
influence subsequent component builds) and into the destination tag (to be delivered as a component
|
||||
in the module).
|
||||
|
||||
|
||||
Module-build-macros package is tagged into the Koji tag
|
||||
@@ -120,31 +127,31 @@ method is called.
|
||||
This simply waits until all the components in a currently built batch are tagged in a Koji tag.
|
||||
|
||||
Because module-build-macros is the only component in batch 1, it can continue by regenerating
|
||||
the Koji repository based on a tag, so the newly build packages (just module-build-macros
|
||||
in our case), can be installed from that repository when building next components in a module.
|
||||
the Koji repository based on a tag, so the newly built packages (just module-build-macros
|
||||
in our case), can be installed from that repository when building the next components in a module.
|
||||
|
||||
|
||||
Koji repository is regenerated
|
||||
------------------------------
|
||||
|
||||
Once the Koji repository containing packages from currently built batch is regenerated,
|
||||
Once the Koji repository containing packages from the currently built batch is regenerated,
|
||||
the `scheduler.handlers.repos.done(...)` method is called.
|
||||
|
||||
This verifies that all the packages from current batch (just module-build-macros for now)
|
||||
really appear in generated repository and if so, it starts building next batch by calling
|
||||
This verifies that all the packages from the current batch (just module-build-macros for now)
|
||||
really appear in the generated repository and if so, it starts building the next batch by calling
|
||||
`utils.batches.start_next_batch_build(...)`.
|
||||
|
||||
|
||||
Building next batch
|
||||
-------------------
|
||||
Building the next batch
|
||||
-----------------------
|
||||
|
||||
The `start_next_batch_build(...)` increases the `ModuleBuild.batch` counter to note that it
|
||||
is going to build next batch with next component builds.
|
||||
is going to build the next batch with the next component builds.
|
||||
|
||||
It then generates the list of unbuilt components in the batch and tries to reuse some from
|
||||
previous module builds. This can happen for example when the component is built from the
|
||||
same source as previously, no component builds in previous batches changed and the
|
||||
buildrequires/requires of current module build are still the same as previously.
|
||||
buildrequires/requires of the current module build are still the same as previously.
|
||||
|
||||
For components which cannot be reused, it submits them to Koji.
|
||||
|
||||
@@ -154,25 +161,25 @@ Build all components in all batches in a module
|
||||
|
||||
The process for every component build is the same as for module-build-macros.
|
||||
|
||||
MBS builds it in Koji. Once all the components in current batch are built, MBS tags them into
|
||||
MBS builds it in Koji. Once all the components in the current batch are built, MBS tags them into
|
||||
the Koji tag. Once they are tagged, it regenerates the Koji tag repository and then starts
|
||||
building next batch.
|
||||
|
||||
It all ends up when all batches are done.
|
||||
Rinse and repeat! This process is repeated until all the batches are complete.
|
||||
|
||||
|
||||
Last component is built
|
||||
-----------------------
|
||||
|
||||
Once the last component is built and the repository is regenerated, the
|
||||
`scheduler.handlers.repos.done(...)` method moves the module build to "done" state.
|
||||
`scheduler.handlers.repos.done(...)` method moves the module build to the "done" state.
|
||||
|
||||
|
||||
Importing module build to Koji
|
||||
------------------------------
|
||||
Importing the module build to Koji
|
||||
----------------------------------
|
||||
|
||||
The "done" state message is handled in `scheduler.handlers.modules.done(...)` method.
|
||||
The "done" state message is handled in the `scheduler.handlers.modules.done(...)` method.
|
||||
|
||||
This method imports the module build into the Koji using the `KojiContentGenerator` class.
|
||||
The module build in Koji points to Koji tag with module components and also contains the
|
||||
This method imports the module build into Koji using the `KojiContentGenerator` class.
|
||||
The module build in Koji points to the Koji tag with the module's components and also contains the
|
||||
final modulemd files generated for earch architecture the module is built for.
|
||||
|
||||
Reference in New Issue
Block a user