How does MBS build modules? =========================== This document describes how modules are built internally in MBS. The goal of this document is to explain code-flow of module builds. It assumes everything goes as expected and does not mention any error handling or corner cases. User submits module build request --------------------------------- There is the MBS frontend, which provides a REST API (See ``views.py``). A user sends a POST request with JSON describing the module to build. There is mainly the URL to the git repository (called ``scmurl`` |---| which points to the git repository containing the modulemd file defining the module) and branch name (called ``branch``). This JSON data is handled by ``views.SCMHandler``, which validates the JSON and calls ``utils.submit.submit_module_build_from_scm(...)`` method. This goes down to ``submit_module_build(...)``. Alternatively, if submitting a YAML modulemd file is allowed (MBS setting ``YAML_SUBMIT_ALLOWED`` is ``True``), the user can include it in the JSON data (called ``modulemd`` and ``module_name``) or send a ``multipart/form-data`` POST request directly including the contents of a YAML modulemd file (called ``yaml``). In this case, the JSON data and YAML file are handled by ``views.YAMLFileHandler``, which validates the data and calls the ``utils.submit.submit_module_build_from_yaml(...)`` method which also goes down to ``submit_module_build(...)``. If module scratch builds are allowed (MBS setting ``MODULES_ALLOW_SCRATCH`` is ``True``), the user can request a scratch module build (called ``scratch``). With a scratch build request, the user can include a YAML modulemd file (see above) and also upload one or more source RPMs to Koji via calls to Koji's ``session.uploadWrapper(..)``, and supply the list of upload links to MBS (called ``srpms``). Such custom SRPMs will be used to override the git repository source for corresponding components. Module Stream Expansion (MSE) ----------------------------- The first thing done in ``submit_module_build(...)`` is Module Stream Expansion (MSE). The submitted modulemd file might have buildrequires and requires pairs defined in an ambiguous way. For example the module can buildrequire ``platform:[f28, f29]`` modules, which means it should be built against the ``f28`` and ``f29`` streams of ``platform`` module. The process of resolving these ambiguous buildrequires and requires is called Module Stream Expansion. Input for this process is the submitted modulemd file with ambiguous buildrequires/requires. Output of this process is the list of multiple modulemd files with all the ambiguous buildrequires/requires resolved. This all happens in ``utils.mse.generate_expanded_mmds(...)`` method. At first, this method finds out all the possible buildrequires/requires for the input module. This is done using ``DBResolver`` which simply finds out the modules in the MBS database. In our case, it would list all the ``platform:f28`` and ``platform:f29`` modules. It then uses the ``MMDResolver`` class to find all the possible combinations of buildrequires/requires against which the input module can be built. In our case, it would generate two expanded modulemd files (one for each platform stream) which would be identical to the input modulemd file with only the following exceptions: - The buildrequires/requires pairs from the input modulemd files will be replaced by the particular combination returned by ``MMDResolver`` - The ``xmd`` section of generated modulemd files will contain ``buildrequires`` list which lists all the modules required to build this expanded modulemd file. Requirements are traversed to produce a transitive list that includes all ``requires`` of each ``buildrequires`` entry. This is used later by MBS. - The context is computed and filled for each expanded modulemd file. It is based on the expanded buildrequires and requires pairs. See ``models.ModuleBuild.contexts_from_mmd(...)``. Such expanded modulemd files are then added to database as the next step in ``submit_module_build(...)`` and are handled as separate module builds later in MBS. The ``submit_module_build(...)`` then moves the module builds to "init" state and sends a message on the configured message bus. There is a build option for MBS which enables us to override stream expansion and define contexts directly. For more information see |docs/STATIC_CONTEXTS.rst|_. .. |docs/STATIC_CONTEXTS.rst| replace:: ``docs/STATIC_CONTEXTS.rst`` .. _docs/STATIC_CONTEXTS.rst: STATIC_CONTEXTS.rst Backend handles module moved to the "init" state ------------------------------------------------ When module build is moved to the "init" state, the backend handles that in the ``scheduler.handlers.modules.init(...)`` method. This method calls ``utils.submit.record_component_builds`` which reads the modulemd file stored in the database by the frontend and records all the components (future RPM packages) in the database. The components are divided into the **batches** based on their buildorder in the modulemd file. Once the components which are supposed to be built as part of this module build are recorded, the module moves to the "wait" state and another message is sent on the message bus. Backend handles module moved to the "wait" state ------------------------------------------------ When the module build is moved to the "wait" state, the backend handles that in the ``scheduler.handlers.modules.wait(...)`` method. At first, this method uses KojiModuleBuilder to generate the Koji tag in which the components will be built. The Koji tag reflects the buildrequires of the module by inheriting their Koji tags. In our case, the Koji tag would inherit just the ``platform:f28`` or ``platform:f29`` Koji tag, because that's the only buildrequired module we have. The list of modules buildrequired by the currently building module is determined by the ``buildrequires`` list in the ``xmd`` section of the expanded modulemd file. Once the Koji tag is ready, it tries to build a synthetic ``module-build-macros`` package. This package contains special build macros which for example, define the dist-tag for built RPMs, ensure that filtered packages are not installed into the buildroot and so on. The module-build-macros package is always built in the first batch. Module-build-macros package is built ------------------------------------ Once the ``module-build-macros`` package is built, Koji sends a message on the message bus, which is handled in the ``scheduler.handlers.components.complete(...)`` method. This method changes the state of that component build in the MBS database to "complete". It then checks if there are any other unbuilt components in the current batch. Because the ``module-build-macros`` package is the only component in batch 1, it can continue tagging it into the Koji tag representing the module, so the ``module-build-macros`` can be later installed during the build of the next batch of components and can influence them. Note that the ``module-build-macros`` package is the only package in the course of a module build that *only* gets tagged into the build tag. All other builds are tagged both into the build tag (to influence subsequent component builds) and into the destination tag (to be delivered as a component in the module). Module-build-macros package is tagged into the Koji tag ------------------------------------------------------- Once the module-build-macros package is tagged by Koji, the ``scheduler.handlers.tags.tagged(...)`` method is called. This simply waits until all the components in a currently built batch are tagged in a Koji tag. Because module-build-macros is the only component in batch 1, it can continue by regenerating the Koji repository based on a tag, so the newly built packages (just module-build-macros in our case), can be installed from that repository when building the next components in a module. Koji repository is regenerated ------------------------------ Once the Koji repository containing packages from the currently built batch is regenerated, the ``scheduler.handlers.repos.done(...)`` method is called. This verifies that all the packages from the current batch (just module-build-macros for now) really appear in the generated repository and if so, it starts building the next batch by calling ``module_build_service.scheduler.batches.start_next_batch_build(...)``. Building the next batch ----------------------- The ``start_next_batch_build(...)`` increases the ``ModuleBuild.batch`` counter to note that it is going to build the next batch with the next component builds. It then generates the list of unbuilt components in the batch and tries to reuse some from previous module builds. This can happen for example when the component is built from the same source as previously, no component builds in previous batches changed and the buildrequires/requires of the current module build are still the same as previously. For components which cannot be reused, it submits them to Koji. Build all components in all batches in a module ----------------------------------------------- The process for every component build is the same as for module-build-macros. MBS builds it in Koji. Once all the components in the current batch are built, MBS tags them into the Koji tag. Once they are tagged, it regenerates the Koji tag repository and then starts building next batch. Rinse and repeat! This process is repeated until all the batches are complete. Last component is built ----------------------- Once the last component is built and the repository is regenerated, the ``scheduler.handlers.repos.done(...)`` method moves the module build to the "done" state. Importing the module build to Koji ---------------------------------- The "done" state message is handled in the ``scheduler.handlers.modules.done(...)`` method. This method imports the module build into Koji using the ``KojiContentGenerator`` class. The module build in Koji points to the Koji tag with the module's components and also contains the final modulemd files generated for earch architecture the module is built for. .. |---| unicode:: U+2014 .. em dash, trimming surrounding whitespace :trim: