From f54073ebf72dd4a3586118cc0e56d2c86df22736 Mon Sep 17 00:00:00 2001 From: Ralph Bean Date: Wed, 10 Oct 2018 16:34:17 -0400 Subject: [PATCH] Minor adjustments. --- docs/HOW_MBS_BUILDS_MODULES.rst | 127 +++++++++++++++++--------------- 1 file changed, 67 insertions(+), 60 deletions(-) diff --git a/docs/HOW_MBS_BUILDS_MODULES.rst b/docs/HOW_MBS_BUILDS_MODULES.rst index 5b474abb..2cb51886 100644 --- a/docs/HOW_MBS_BUILDS_MODULES.rst +++ b/docs/HOW_MBS_BUILDS_MODULES.rst @@ -1,7 +1,7 @@ How does MBS build modules? =========================== -This document describes how are modules built internally in MBS. The goal of this document is +This document describes how modules are built internally in MBS. The goal of this document is to explain code-flow of module builds. It assumes everything goes as expected and does not mention any error handling or corner cases. @@ -9,10 +9,10 @@ mention any error handling or corner cases. User submits module build request --------------------------------- -There is MBS frontend providing REST API (See `views.py`). User sends POST request with JSON -describing the module to build. There is mainly the URL to git repository (called `scmurl`) -and branch name (called `branch`). The `scmurl` points to git repository containing the -modulemd file defining the module. +There is the MBS frontend, which provides a REST API (See `views.py`). A user sends a POST request +with JSON describing the module to build. There is mainly the URL to the git repository (called +`scmurl`) and branch name (called `branch`). The `scmurl` points to the git repository containing +the modulemd file defining the module. This JSON data is handled by `views.SCMHandler`, which validates the JSON and calls `utils.submit.submit_module_build_from_scm(...)` method. This goes down to @@ -24,15 +24,15 @@ Module Stream Expansion (MSE) The first thing done in `submit_module_build(...)` is Module Stream Expansion (MSE). -The submitted modulemd file might have buildrequires and requires pairs defined in ambigous way. -For example the module can buildrequire `platform:[28, 29]` modules, which means it should +The submitted modulemd file might have buildrequires and requires pairs defined in an ambiguous way. +For example the module can buildrequire `platform:[f28, f29]` modules, which means it should be built against the `f28` and `f29` streams of `platform` module. -The process of resolving these ambigous buildrequires and requires is called Module Stream +The process of resolving these ambiguous buildrequires and requires is called Module Stream Expansion. -Input for this process is the submitted modulemd file with ambigous buildrequires/requires. -Output of this process is list of multiple modulemd files with all the ambigous +Input for this process is the submitted modulemd file with ambiguous buildrequires/requires. +Output of this process is the list of multiple modulemd files with all the ambiguous buildrequires/requires resolved. This all happens in `utils.mse.generate_expanded_mmds(...)` method. @@ -41,74 +41,81 @@ At first, this method finds out all the possible buildrequires/requires for the This is done using `DBResolver` which simply finds out the modules in the MBS database. In our case, it would list all the `platform:f28` and `platform:f29` modules. -It then uses `MMDResolver` class to find all the possible combinations of buildrequires/requires -for which the input module can be built. +It then uses the `MMDResolver` class to find all the possible combinations of buildrequires/requires +against which the input module can be built. In our case, it would generate two expanded modulemd files (one for each platform stream) which -would be identical to input modulemd file with only following exceptions: +would be identical to the input modulemd file with only the following exceptions: -- The buildrequires/requires pairs from input modulemd files will be replaced by the particular +- The buildrequires/requires pairs from the input modulemd files will be replaced by the particular combination returned by `MMDResolver` - The `xmd` section of generated modulemd files will contain `buildrequires` list which lists all - the modules required to build this expanded modulemd file. This is used later by MBS. + the modules required to build this expanded modulemd file. Requirements are traversed to produce + a transitive list that includes all `requires` of each `buildrequires` entry. This is used later + by MBS. - The context is computed and filled for each expanded modulemd file. It is based on the expanded buildrequires and requires pairs. See `models.ModuleBuild.contexts_from_mmd(...)`. -Such expanded modulemd files are then added to database as next step in `submit_module_build(...)` -and are handled as separate module builds later in MBS. +Such expanded modulemd files are then added to database as the next step in +`submit_module_build(...)` and are handled as separate module builds later in MBS. -The `submit_module_build(...)` then moves the module builds to "init" state and sends message to -fedmsg hub or UMB for each submitted expanded module build +The `submit_module_build(...)` then moves the module builds to "init" state and sends a message on +the configured message bus. -Backend handles module moved to "init" state --------------------------------------------- +Backend handles module moved to the "init" state +------------------------------------------------ -When module build is moved to "init" state, backend handles that in +When module build is moved to the "init" state, the backend handles that in the `scheduler.handlers.modules.init(...)` method. This method calls `utils.submit.record_component_builds` which reads the modulemd file -stored in database by frontend and records all the components (future RPM packages) in the +stored in the database by the frontend and records all the components (future RPM packages) in the database. The components are divided into the **batches** based on their buildorder in the modulemd file. Once the components which are supposed to be built as part of this module build are recorded, -the module moves to "wait" state and another message it sent to the message bus. +the module moves to the "wait" state and another message is sent on the message bus. -Backend handles module moved to "wait" state --------------------------------------------- +Backend handles module moved to the "wait" state +------------------------------------------------ -When module build is moved to "wait" state, backend handles that in +When the module build is moved to the "wait" state, the backend handles that in the `scheduler.handlers.modules.wait(...)` method. -At first, this method uses KojiModuleBuilder to generate Koji tag in which the components will be -build. The Koji tag reflects the buildrequires of module by inheriting their Koji tags. In our -case, the Koji tag would inherit just `platform:f28` or `platform:f29` Koji tag, because that's +At first, this method uses KojiModuleBuilder to generate the Koji tag in which the components will be +built. The Koji tag reflects the buildrequires of the module by inheriting their Koji tags. In our +case, the Koji tag would inherit just the `platform:f28` or `platform:f29` Koji tag, because that's the only buildrequired module we have. -The list of modules buildrequired by currently building module is get from `buildrequires` list in -the `xmd` section of expanded modulemd file. +The list of modules buildrequired by the currently building module is determined by the `buildrequires` list in +the `xmd` section of the expanded modulemd file. -Once the Koji tag is ready, it tries to build `module-build-macros` package. This package contains -special build macros which for example defines the dist-tag for built RPMs, ensures that filtered -packages are not installed into the buildroot and so on. +Once the Koji tag is ready, it tries to build a synthetic `module-build-macros` package. This +package contains special build macros which for example, define the dist-tag for built RPMs, ensure +that filtered packages are not installed into the buildroot and so on. -The module-build-macros is always built in first batch. +The module-build-macros package is always built in the first batch. Module-build-macros package is built ------------------------------------ -Once the module-build-macros package is built, Koji sends message to message hub which is handled -in `scheduler.handlers.components.complete(...)` method. +Once the `module-build-macros` package is built, Koji sends a message on the message bus, which is +handled in the `scheduler.handlers.components.complete(...)` method. -This method changes that state of component build in MBS database to "complete". +This method changes the state of that component build in the MBS database to "complete". -It then checks if there are any other unbuilt components in current batch. Because the -"module-build-macros" is the only component in batch 1, it can continue tagging it -into the Koji tag representing the module, so the module-build-macros can be later -installed during the build of next components and can influence them. +It then checks if there are any other unbuilt components in the current batch. Because the +`module-build-macros` package is the only component in batch 1, it can continue tagging it +into the Koji tag representing the module, so the `module-build-macros` can be later +installed during the build of the next batch of components and can influence them. + +Note that the `module-build-macros` package is the only package in the course of a module build that +*only* gets tagged into the build tag. All other builds are tagged both into the build tag (to +influence subsequent component builds) and into the destination tag (to be delivered as a component +in the module). Module-build-macros package is tagged into the Koji tag @@ -120,31 +127,31 @@ method is called. This simply waits until all the components in a currently built batch are tagged in a Koji tag. Because module-build-macros is the only component in batch 1, it can continue by regenerating -the Koji repository based on a tag, so the newly build packages (just module-build-macros -in our case), can be installed from that repository when building next components in a module. +the Koji repository based on a tag, so the newly built packages (just module-build-macros +in our case), can be installed from that repository when building the next components in a module. Koji repository is regenerated ------------------------------ -Once the Koji repository containing packages from currently built batch is regenerated, +Once the Koji repository containing packages from the currently built batch is regenerated, the `scheduler.handlers.repos.done(...)` method is called. -This verifies that all the packages from current batch (just module-build-macros for now) -really appear in generated repository and if so, it starts building next batch by calling +This verifies that all the packages from the current batch (just module-build-macros for now) +really appear in the generated repository and if so, it starts building the next batch by calling `utils.batches.start_next_batch_build(...)`. -Building next batch -------------------- +Building the next batch +----------------------- The `start_next_batch_build(...)` increases the `ModuleBuild.batch` counter to note that it -is going to build next batch with next component builds. +is going to build the next batch with the next component builds. It then generates the list of unbuilt components in the batch and tries to reuse some from previous module builds. This can happen for example when the component is built from the same source as previously, no component builds in previous batches changed and the -buildrequires/requires of current module build are still the same as previously. +buildrequires/requires of the current module build are still the same as previously. For components which cannot be reused, it submits them to Koji. @@ -154,25 +161,25 @@ Build all components in all batches in a module The process for every component build is the same as for module-build-macros. -MBS builds it in Koji. Once all the components in current batch are built, MBS tags them into +MBS builds it in Koji. Once all the components in the current batch are built, MBS tags them into the Koji tag. Once they are tagged, it regenerates the Koji tag repository and then starts building next batch. -It all ends up when all batches are done. +Rinse and repeat! This process is repeated until all the batches are complete. Last component is built ----------------------- Once the last component is built and the repository is regenerated, the -`scheduler.handlers.repos.done(...)` method moves the module build to "done" state. +`scheduler.handlers.repos.done(...)` method moves the module build to the "done" state. -Importing module build to Koji ------------------------------- +Importing the module build to Koji +---------------------------------- -The "done" state message is handled in `scheduler.handlers.modules.done(...)` method. +The "done" state message is handled in the `scheduler.handlers.modules.done(...)` method. -This method imports the module build into the Koji using the `KojiContentGenerator` class. -The module build in Koji points to Koji tag with module components and also contains the +This method imports the module build into Koji using the `KojiContentGenerator` class. +The module build in Koji points to the Koji tag with the module's components and also contains the final modulemd files generated for earch architecture the module is built for.