This also moves the methods load_mmd and load_mmd_file to
module_build_service.utils.general.
This also removes some MSE unit tests with a mix of positive and
negative streams since this is not supported in libmodulemd v2. The
user will be presented with a syntax error if they try to submit
such a modulemd file.
Our current code has following issues with -debuginfo/-debugsource handling:
- The foo-debuginfo is included in the MMD only when foo is included there,
but some special packages contain custom -debuginfo sub-packages like
foo-common-debuginfo without matching foo-common sub-package. With our
current code, the foo-common-debuginfo is never included in the final MMD.
- The foo-debugsource is included in the MMD only when foo.src.rpm,
but some special packages contain custom -debuginfo sub-package.
This commit changes the handling of -debuginfo/-debugsource like this:
- The RPMs to include in the final MMD are evaluated in particular order now.
At first we evaluate non-debug RPMs and then debug RPMs.
- When handling the foo-debuginfo/foo-debugsource RPM, we include it in
final MMD only in one of these cases:
- The "foo" is included in the MMD file (it means it is not filtered out).
- The "foo" package does not exist at all (it means only foo-debuginfo exists
and we need to include this package unless filtered out) and in the same time
the SRPM from which this -debuginfo/-debugsource RPM has been built is included
in a final MMD (it means that there is at least some package from this build
included - this handles case when only foo.src.rpm and foo-debugsource.rpm
would be included in a final MMD, which would be wrong.)
We also respect filters here, so it is possible to explicitely filter out also
-debuginfo/-debugsource packages.
GenericResolver.extract_modulemd is not removed, but deprecated. Call of it
will result in a deprecation message printed. Any new code should call
load_mmd.
Signed-off-by: Chenxiong Qi <cqi@redhat.com>
The current code reads the data, converts them to unicode string and
then uses the `len()` of that string as filesize. This is wrong,
because Koji expects filesize to be really number of bytes, not number
of characters.
Therefore, in this commit, the filesize is computed from raw data (bytes).
This commit introduces new to_text_type helper method and calls it
for return value of all mmd.dumps() calls. That way, we always
end up with proper unicode string represntation on both python
major versions.
This commit also adds unicode character to description of all
the yaml files we use in the tests so we can be sure MBS can
handle unicode characters properly.
This might be temporary fix, depending on the result of discussion
at https://github.com/fedora-modularity/libmodulemd/issues/184.
This reverts commit ae79b711d8.
This breaks MBS with python3 and fixes only single occurence of this
issue. We want to fix this in libmodulemd or find a better way how
to fix this to work on both python2 and python3.
We have libmodulemd PR open to address this issue on libmodulemd
level: https://github.com/fedora-modularity/libmodulemd/issues/184.
When encoutering a Windows end of line (^M), io.open and open in Python 3
will convert those to UNIX end of lines by default. When reading logs
to compute the checksum, it's important those new lines aren't converted,
to ensure the checksum is correct. This caused issues in Fedora staging
because when cloning down a repo, the repoSpanner output had Windows end
of lines, and this would end up in build.log. The solution is to just read
it as binary so that Python doesn't perform these conversions.
In case the module did not contain any RPM, _koji_rpms_in_tag still
tried to call the session.getRPMHeaders with an empty list. This results
in None being returned, which is something _koji_rpms_in_tag did
not count with and failed with traceback when trying to iterate
that None value.
In this commit, the _koji_rpms_in_tag returns an empty list early
if Koji tag does not contain any RPM. Therefore the
session.getRPMHeaders is not called at all in this case, because it
does not make any sense to get RPM Headers when Koji tag does not
contain any RPM.
Current code presumes SRPM name always matches the RPM name built out of this SRPM
and only includes it together with the main package in this case. This is wrong
assumption, because usually there are multiple binary RPMs built from single SRPM.
This commit fixes that by including the SRPM NEVRA in `non_devel_source_rpms`
no matter what RPM name is.
The test RPMs are reworked in this commit to match the reality better - especially
the relations between SRPM and RPMs. The case with different SRPM name and RPM
name is also included in the reworked test - dhcp-libs binary RPM built from
dhcp SRPM.
MBS calls some read-only Koji APIs which does not require to log into a
session. This patch makes it optional to choose whether to login a
session and use anonymous session properly to call those read-only APIs.
Signed-off-by: Chenxiong Qi <cqi@redhat.com>
If an RPM is not included, its correspnding -debug* RPMs should also not
be included.
Also ensure that source RPMs are only ever added to -devel modules if
a binary RPM has been completely excluded from non-devel module.
Internal ref: FACTORY-3263
Signed-off-by: Luiz Carvalho <lucarval@redhat.com>
There is a need to ship the unchanged source file which was used to build
a module build from legal reasons. The KojiContentGenerator is extended
in this commit to fetch it from SCM URL using the `scm` module and later
attach it as `modulemd.src.txt` to Koji CG build.
This commit introduces KojiContentGenerator._sanitize_mmd method to:
- remove `repository` and `cache` from ComponentRPM in Modulemd.
- remove `mbs` section from `xmd`.
This is done to not leak internal build-only information to final
modulemd files.
For some modules, modularity-wg wants to be able to ship the "-devel" modules containing
the RPMs which are normally filtered out from the module.
This PR generates second Koji CG module with -devel suffix in a name with final modulemd
files containing the filtered out RPMs.
The original Pungi code, on which MBS code is based on, always passed only RPMs
with valid architectures to further decide if their subset should end up in a
final modulemd file.
In MBS, we pass RPMs with all architecture and there was no code to actually filter
out the RPMs which are from architectures which should never end up in a final MMD.
This commit checks that RPMs for completely different architectures will never
be considered to be included in a final MMD.
The list of arches we compare against should not include multilib
arches. Otherwise excluding does not behave correctly.
Example:
A build has ExcludeArch: i686 (because it only works on 64 bit arches).
A noarch package is built there, but it would be excluded from x86_64.
Relates: https://pagure.io/pungi/pull-request/1050
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
During I'm reading through the code base to learn MBS, I found out some
typos and minor issues in some other docstrings, and I also found some
docstrings with extra informative description could make it easier to
understand the code.
Signed-off-by: Chenxiong Qi <cqi@redhat.com>
I discovered that local builds have been broken by recent (good) changes in
dnf. At the end of every batch, we regenerate the local repo with createrepo
and we also call modifyrepo to include the modulemd file as we progress. This
was added in #467.
What we really want, is for the modulemd file to be present at the *end* of the
build.
Recently, dnf started respecting the modulemd file natively, such that builds
we built in batch0 would not show up in batch1. They would be present in the
repo, but they would be marked as belonging to a module which was not enabled,
and so could not be pulled in. Every module build would fail because
module-srpm-macros was in a disabled module (the module being built at the
time).
This change makes it so that the module metadata is only added to the repo at
the very end of the build. I moved it into a `finalize` method on the builder
which the copr builder was using, and for symmetry's sake I moved the koji
content generator code to the same method on that builder.
There was a circular import issue to solve between the koji module builder and
the koji content generator modules that generated a larger diff, but is mostly
cosmetic and mock changes.
