Please note that this patch does not change the use of database session
in MBS. So, in the frontend, the database session is still managed by
Flask-SQLAlchemy, that is the db.session. And the backend, running event
handlers, has its own database session created from SQLAclehmy session
API directly.
This patch aims to reduce the number of scoped_session created when call
original function make_db_session. For technical detailed information,
please refer to SQLAlchemy documentation Contextual/Thread-local
Sessions.
As a result, a global scoped_session is accessible from the
code running inside backend, both the event handlers and functions
called from handlers. The library code shared by frontend and backend,
like resolvers, has no change.
Similarly, db.session is only used to recreate database for every test.
Signed-off-by: Chenxiong Qi <cqi@redhat.com>
This also removes the outdated comments around authorship of each
file. If there is still interest in this information, one can just
look at the git history.
This patch separates the use of database session in different MBS components
and do not mix them together.
In general, MBS components could be separated as the REST API (implemented
based on Flask) and non-REST API including the backend build workflow
(implemented as a fedmsg consumer on top of fedmsg-hub and running
independently) and library shared by them. As a result, there are two kind of
database session used in MBS, one is created and managed by Flask-SQLAlchemy,
and another one is created from SQLAclhemy Session API directly. The goal of
this patch is to make ensure session object is used properly in the right
place.
All the changes follow these rules:
* REST API related code uses the session object db.session created and
managed by Flask-SQLAlchemy.
* Non-REST API related code uses the session object created with SQLAlchemy
Session API. Function make_db_session does that.
* Shared code does not created a new session object as much as possible.
Instead, it accepts an argument db_session.
The first two rules are applicable to tests as well.
Major changes:
* Switch tests back to run with a file-based SQLite database.
* make_session is renamed to make_db_session and SQLAlchemy connection pool
options are applied for PostgreSQL backend.
* Frontend Flask related code uses db.session
* Shared code by REST API and backend build workflow accepts SQLAlchemy session
object as an argument. For example, resolver class is constructed with a
database session, and some functions accepts an argument for database session.
* Build workflow related code use session object returned from make_db_session
and ensure db.session is not used.
* Only tests for views use db.session, and other tests use db_session fixture
to access database.
* All argument name session, that is for database access, are renamed to
db_session.
* Functions model_tests_init_data, reuse_component_init_data and
reuse_shared_userspace_init_data, which creates fixture data for
tests, are converted into pytest fixtures from original function
called inside setup_method or a test method. The reason of this
conversion is to use fixture ``db_session`` rather than create a
new one. That would also benefit the whole test suite to reduce the
number of SQLAlchemy session objects.
Signed-off-by: Chenxiong Qi <cqi@redhat.com>
The context column was added in d83e6897ca. The
migration set `nullable=False`, but the model kept the default of `nullable=True`.
This caused `mbs-manager db migrate` to create a migration for setting the context
collumn to `nullable=True`.
This also moves the methods load_mmd and load_mmd_file to
module_build_service.utils.general.
This also removes some MSE unit tests with a mix of positive and
negative streams since this is not supported in libmodulemd v2. The
user will be presented with a syntax error if they try to submit
such a modulemd file.
Our current code has following issues with -debuginfo/-debugsource handling:
- The foo-debuginfo is included in the MMD only when foo is included there,
but some special packages contain custom -debuginfo sub-packages like
foo-common-debuginfo without matching foo-common sub-package. With our
current code, the foo-common-debuginfo is never included in the final MMD.
- The foo-debugsource is included in the MMD only when foo.src.rpm,
but some special packages contain custom -debuginfo sub-package.
This commit changes the handling of -debuginfo/-debugsource like this:
- The RPMs to include in the final MMD are evaluated in particular order now.
At first we evaluate non-debug RPMs and then debug RPMs.
- When handling the foo-debuginfo/foo-debugsource RPM, we include it in
final MMD only in one of these cases:
- The "foo" is included in the MMD file (it means it is not filtered out).
- The "foo" package does not exist at all (it means only foo-debuginfo exists
and we need to include this package unless filtered out) and in the same time
the SRPM from which this -debuginfo/-debugsource RPM has been built is included
in a final MMD (it means that there is at least some package from this build
included - this handles case when only foo.src.rpm and foo-debugsource.rpm
would be included in a final MMD, which would be wrong.)
We also respect filters here, so it is possible to explicitely filter out also
-debuginfo/-debugsource packages.
GenericResolver.extract_modulemd is not removed, but deprecated. Call of it
will result in a deprecation message printed. Any new code should call
load_mmd.
Signed-off-by: Chenxiong Qi <cqi@redhat.com>
The current code reads the data, converts them to unicode string and
then uses the `len()` of that string as filesize. This is wrong,
because Koji expects filesize to be really number of bytes, not number
of characters.
Therefore, in this commit, the filesize is computed from raw data (bytes).
This commit introduces new to_text_type helper method and calls it
for return value of all mmd.dumps() calls. That way, we always
end up with proper unicode string represntation on both python
major versions.
This commit also adds unicode character to description of all
the yaml files we use in the tests so we can be sure MBS can
handle unicode characters properly.
This might be temporary fix, depending on the result of discussion
at https://github.com/fedora-modularity/libmodulemd/issues/184.
This commit includes the backport of the changes to `krb_login` in
https://pagure.io/koji/pull-request/1187. This change is required
for our separate threads to use a separate Kerberos context per thread.
In case the module did not contain any RPM, _koji_rpms_in_tag still
tried to call the session.getRPMHeaders with an empty list. This results
in None being returned, which is something _koji_rpms_in_tag did
not count with and failed with traceback when trying to iterate
that None value.
In this commit, the _koji_rpms_in_tag returns an empty list early
if Koji tag does not contain any RPM. Therefore the
session.getRPMHeaders is not called at all in this case, because it
does not make any sense to get RPM Headers when Koji tag does not
contain any RPM.
Current code presumes SRPM name always matches the RPM name built out of this SRPM
and only includes it together with the main package in this case. This is wrong
assumption, because usually there are multiple binary RPMs built from single SRPM.
This commit fixes that by including the SRPM NEVRA in `non_devel_source_rpms`
no matter what RPM name is.
The test RPMs are reworked in this commit to match the reality better - especially
the relations between SRPM and RPMs. The case with different SRPM name and RPM
name is also included in the reworked test - dhcp-libs binary RPM built from
dhcp SRPM.
If an RPM is not included, its correspnding -debug* RPMs should also not
be included.
Also ensure that source RPMs are only ever added to -devel modules if
a binary RPM has been completely excluded from non-devel module.
Internal ref: FACTORY-3263
Signed-off-by: Luiz Carvalho <lucarval@redhat.com>
There is a need to ship the unchanged source file which was used to build
a module build from legal reasons. The KojiContentGenerator is extended
in this commit to fetch it from SCM URL using the `scm` module and later
attach it as `modulemd.src.txt` to Koji CG build.
This commit introduces KojiContentGenerator._sanitize_mmd method to:
- remove `repository` and `cache` from ComponentRPM in Modulemd.
- remove `mbs` section from `xmd`.
This is done to not leak internal build-only information to final
modulemd files.
For some modules, modularity-wg wants to be able to ship the "-devel" modules containing
the RPMs which are normally filtered out from the module.
This PR generates second Koji CG module with -devel suffix in a name with final modulemd
files containing the filtered out RPMs.
The original Pungi code, on which MBS code is based on, always passed only RPMs
with valid architectures to further decide if their subset should end up in a
final modulemd file.
In MBS, we pass RPMs with all architecture and there was no code to actually filter
out the RPMs which are from architectures which should never end up in a final MMD.
This commit checks that RPMs for completely different architectures will never
be considered to be included in a final MMD.
I discovered that local builds have been broken by recent (good) changes in
dnf. At the end of every batch, we regenerate the local repo with createrepo
and we also call modifyrepo to include the modulemd file as we progress. This
was added in #467.
What we really want, is for the modulemd file to be present at the *end* of the
build.
Recently, dnf started respecting the modulemd file natively, such that builds
we built in batch0 would not show up in batch1. They would be present in the
repo, but they would be marked as belonging to a module which was not enabled,
and so could not be pulled in. Every module build would fail because
module-srpm-macros was in a disabled module (the module being built at the
time).
This change makes it so that the module metadata is only added to the repo at
the very end of the build. I moved it into a `finalize` method on the builder
which the copr builder was using, and for symmetry's sake I moved the koji
content generator code to the same method on that builder.
There was a circular import issue to solve between the koji module builder and
the koji content generator modules that generated a larger diff, but is mostly
cosmetic and mock changes.
1. Changed ModuleBuild's context property to db column, it's
non-nullable and default value is '00000000' to keep it consistent
with previous behaviour.
2. Changed ModuleBuild.contexts_from_mmd to return a tuple of
(ref_build_context, build_context, runtime_context, context).
3. Updated tests affected by this change.
