This also moves the methods load_mmd and load_mmd_file to
module_build_service.utils.general.
This also removes some MSE unit tests with a mix of positive and
negative streams since this is not supported in libmodulemd v2. The
user will be presented with a syntax error if they try to submit
such a modulemd file.
GenericResolver.extract_modulemd is not removed, but deprecated. Call of it
will result in a deprecation message printed. Any new code should call
load_mmd.
Signed-off-by: Chenxiong Qi <cqi@redhat.com>
Currently, the PLATFORM_ID is parsed from the `/etc/os-release`. This
is good default value, but sometimes you want to build module locally
against the different platform stream.
For example building on platform:f29 against the platform:f30 modules. In
that case, we need to be able to override the host PLATFORM_ID and
set it manually chosen value.
When importing modules for offline local builds from local repositories,
the XMD section does not have to be set at all - it gets removed during
the compose and is also MBS specific.
We need to be able to import such MMDs using the `import_mmd` method
in order to make --offline local builds working.
This commit adds new `check_buildrequires` bool kwarg in `import_mmd`
method to disable `xmd["buildrequires"]` checks to fix this.
These module builds will basically act as metadata-only module builds.
This will be more useful as additional features stem from these types
of builds.
Previously MockModuleBuilder was checking the module state to see if
it should run a final createrepo, but since eafa93037f, finalize() is
called before changing the module state; add an explicit boolean to
GenericBuilder.finalize() to avoid worrying about ordering.
There are following changes introduced in this commit:
- The `koji_tag` of module builds imported from the local repositories
is now in `repofile:///etc/yum.repos.d/some.repo` format to store the
repository from which the module was imported to local MBS DB.
- The `koji_tag` of fake base module is set to empty `repofile://`
and in `MockModuleBuilder` the `conf.base_module_repofiles` list
is used as source for the repositories defining platform. We can't
simply use single repository, because there might be fedora.repo
and fedora-update.repo and so on.
- The list of default .repo files for platform are passed using the
`-r` switch in `build_module_locally` `mbs-manager` command.
- The LocalResolver (subclass of DBResolver) is added which is used
to resolve the build dependencies when building modules offline
locally.
- The `MockModuleBuilder` enables the buildrequired modules and
repositories from which they come in the mock config.
With this commit, it is possible to build testmodule locally
without any external infra.
This is the first PR in many for Offline local builds. This PR:
- Adds --offline flag to build_module_locally mbs-manager command to enable
offline local builds.
- If this flag is used, new `import_builds_from_local_dnf_repos` method is
called which uses DNF API to get all the available installable modulemd
files and imports each module into MBS local SQLite database.
- It also adds fake "platform:stream" module based on the /etc/os-release,
so the buildrequirements of the imported modules are satisfied.
The idea here is that in upcoming commits, I will create LocalResolver
which will be similar to DBResolver with some extra rules to resolve
local module builds. This new LocalResolver will still be based on
the models.ModuleBuild methods and therefore we need the modules
imported in database.
Since the required parameters vary based on if the modulemd
comes from SCM or a direct submission, the concept of optional
parameters doesn't really apply.
When importing a base module, we must ensure the value that will be
used in the RPM disttags doesn't contain a dash since a dash isn't
allowed in the release field of the NVR.
MBS uses the base module's stream that was buildrequired by the module
in the RPM disttags for that module build. The stream name may not be
ideal for all situations, so now this is customizable by setting the
xmd['mbs']['disttag_marking'] in the base module's modulemd.
- Keep scratch module builds in the 'done' state.
- Make koji tagging for scratch modules unique so the same
commit can be resubmitted.
- Use alternate prefix for scratch module build components so they can
be identified later.
- Prevent scratch build components from being reused.
- Assorted code and comment cleanup.
Signed-off-by: Merlin Mathesius <mmathesi@redhat.com>
In case there is lot of components in a module build or there are some
networking issues and we need to retry the "git clone" commands, the
`format_mmd` method can take long time.
If it takes more than 10 minutes, the poller can produce fake event,
because it seems the module build is stuck. This is wrong, because
it would lead to another unexpected init handler call.
In this commit, the `format_mmd` updates the `ModuleBuild.time_modified`
method regularly to prevent poller from sending the unexpected fake
init message.
This commit introduces new to_text_type helper method and calls it
for return value of all mmd.dumps() calls. That way, we always
end up with proper unicode string represntation on both python
major versions.
This commit also adds unicode character to description of all
the yaml files we use in the tests so we can be sure MBS can
handle unicode characters properly.
This might be temporary fix, depending on the result of discussion
at https://github.com/fedora-modularity/libmodulemd/issues/184.
We always set the "context" to DEFAULT_MODULE_CONTEXT from the historical
reasons - the context was not stored in the database before and we stored
just the build_context/runtime_context. But this is no longer true for
some time.
In this commit, the context is respected and stored in the database when
importing module using the `import_mmd` method. If the context is not set
in the imported MMD, the DEFAULT_MODULE_CONTEXT is used.
This is needed for Fedora branching, but it is generally useful.
For example, there is a module buildrequiring "platform:[]". In the time
this module has been built, only platform:f29 existed, so it has been built
just against platform:f29.
After a while, the platform:f30 is released and the maintainer needs
to rebuild the module against platform:f30. Right now, he needs to create
new commit in the module and submit the build, but this will result in useless
rebuild of the module also against platform:f29.
In this commit, MBS allows to resubmit the module build in a case
there are new MSE builds to build. MBS will send all the module builds
back to the user - so the existing builds will be already marked as
"ready" and the newly submitted builds will have the "init" state in
the REST API response.
However, in case when there are no new MSE builds to build, MBS still
sends back the Conflict error as it used to. This is done for backwards
compatibility and also to not confuse the users in case when no new build
has been submitted.
_record_ursine_rpms needs to get each collision module's koji_tag and
then get built RPMs from that koji_tag eventually. _get_module on each
individual resolver is called to get module metadata accordingly. So,
when running a local build, module metadata is got from remote MBS, and
for a MBS instance, connected database is queried.
Signed-off-by: Chenxiong Qi <cqi@redhat.com>
This resolve the stream collision by adding specific RPMs to
module-build-macros SRPM as Conflicts.
For more information about module stream collision, please refer to
docstring in utils/ursine.py
Signed-off-by: Chenxiong Qi <cqi@redhat.com>
This reverts commit 8173040ea6. Since it didn't
provide any real benefit after talking with sgallagh, it seems like we'd better
off not introduce more risk to the next deployment.
MBS has been using the libmodulemd API incorrectly by assuming that
methods like `get_rpm_components` return the actual object in memory
and not a copy. This has been true but wasn't something sgallagh
intended. We found this out after he had me test his new version of
libmodulemd. This PR removes those assumptions.
I discovered that local builds have been broken by recent (good) changes in
dnf. At the end of every batch, we regenerate the local repo with createrepo
and we also call modifyrepo to include the modulemd file as we progress. This
was added in #467.
What we really want, is for the modulemd file to be present at the *end* of the
build.
Recently, dnf started respecting the modulemd file natively, such that builds
we built in batch0 would not show up in batch1. They would be present in the
repo, but they would be marked as belonging to a module which was not enabled,
and so could not be pulled in. Every module build would fail because
module-srpm-macros was in a disabled module (the module being built at the
time).
This change makes it so that the module metadata is only added to the repo at
the very end of the build. I moved it into a `finalize` method on the builder
which the copr builder was using, and for symmetry's sake I moved the koji
content generator code to the same method on that builder.
There was a circular import issue to solve between the koji module builder and
the koji content generator modules that generated a larger diff, but is mostly
cosmetic and mock changes.
koji now supports tags with max length of 256, we can use
more informative tag name instead of the hash one.
The new format of koji tag name is:
module-<name>-<stream>-<version>-<context>
However when the generated tag's length > (256 - len('build')), we
fallback to the old way of name in hash format (module-<hash>).
In this change, koji tag is always generated from MBS itself, even
with pdc resolver.
FIXES: #918#925
