This also includes `from __future__ import absolute_import`
in every file so that the imports are consistent in Python 2 and 3.
The Python 2 tests fail without this.
This puts backend specific code in either the builder or scheduler
subpackage. This puts API specific code in the new web subpackage.
Lastly, any code shared between the API and backend is placed in the
common subpackage.
Please note that this patch does not change the use of database session
in MBS. So, in the frontend, the database session is still managed by
Flask-SQLAlchemy, that is the db.session. And the backend, running event
handlers, has its own database session created from SQLAclehmy session
API directly.
This patch aims to reduce the number of scoped_session created when call
original function make_db_session. For technical detailed information,
please refer to SQLAlchemy documentation Contextual/Thread-local
Sessions.
As a result, a global scoped_session is accessible from the
code running inside backend, both the event handlers and functions
called from handlers. The library code shared by frontend and backend,
like resolvers, has no change.
Similarly, db.session is only used to recreate database for every test.
Signed-off-by: Chenxiong Qi <cqi@redhat.com>
This also removes the outdated comments around authorship of each
file. If there is still interest in this information, one can just
look at the git history.
This patch separates the use of database session in different MBS components
and do not mix them together.
In general, MBS components could be separated as the REST API (implemented
based on Flask) and non-REST API including the backend build workflow
(implemented as a fedmsg consumer on top of fedmsg-hub and running
independently) and library shared by them. As a result, there are two kind of
database session used in MBS, one is created and managed by Flask-SQLAlchemy,
and another one is created from SQLAclhemy Session API directly. The goal of
this patch is to make ensure session object is used properly in the right
place.
All the changes follow these rules:
* REST API related code uses the session object db.session created and
managed by Flask-SQLAlchemy.
* Non-REST API related code uses the session object created with SQLAlchemy
Session API. Function make_db_session does that.
* Shared code does not created a new session object as much as possible.
Instead, it accepts an argument db_session.
The first two rules are applicable to tests as well.
Major changes:
* Switch tests back to run with a file-based SQLite database.
* make_session is renamed to make_db_session and SQLAlchemy connection pool
options are applied for PostgreSQL backend.
* Frontend Flask related code uses db.session
* Shared code by REST API and backend build workflow accepts SQLAlchemy session
object as an argument. For example, resolver class is constructed with a
database session, and some functions accepts an argument for database session.
* Build workflow related code use session object returned from make_db_session
and ensure db.session is not used.
* Only tests for views use db.session, and other tests use db_session fixture
to access database.
* All argument name session, that is for database access, are renamed to
db_session.
* Functions model_tests_init_data, reuse_component_init_data and
reuse_shared_userspace_init_data, which creates fixture data for
tests, are converted into pytest fixtures from original function
called inside setup_method or a test method. The reason of this
conversion is to use fixture ``db_session`` rather than create a
new one. That would also benefit the whole test suite to reduce the
number of SQLAlchemy session objects.
Signed-off-by: Chenxiong Qi <cqi@redhat.com>
This also moves the methods load_mmd and load_mmd_file to
module_build_service.utils.general.
This also removes some MSE unit tests with a mix of positive and
negative streams since this is not supported in libmodulemd v2. The
user will be presented with a syntax error if they try to submit
such a modulemd file.
Our current code has following issues with -debuginfo/-debugsource handling:
- The foo-debuginfo is included in the MMD only when foo is included there,
but some special packages contain custom -debuginfo sub-packages like
foo-common-debuginfo without matching foo-common sub-package. With our
current code, the foo-common-debuginfo is never included in the final MMD.
- The foo-debugsource is included in the MMD only when foo.src.rpm,
but some special packages contain custom -debuginfo sub-package.
This commit changes the handling of -debuginfo/-debugsource like this:
- The RPMs to include in the final MMD are evaluated in particular order now.
At first we evaluate non-debug RPMs and then debug RPMs.
- When handling the foo-debuginfo/foo-debugsource RPM, we include it in
final MMD only in one of these cases:
- The "foo" is included in the MMD file (it means it is not filtered out).
- The "foo" package does not exist at all (it means only foo-debuginfo exists
and we need to include this package unless filtered out) and in the same time
the SRPM from which this -debuginfo/-debugsource RPM has been built is included
in a final MMD (it means that there is at least some package from this build
included - this handles case when only foo.src.rpm and foo-debugsource.rpm
would be included in a final MMD, which would be wrong.)
We also respect filters here, so it is possible to explicitely filter out also
-debuginfo/-debugsource packages.
GenericResolver.extract_modulemd is not removed, but deprecated. Call of it
will result in a deprecation message printed. Any new code should call
load_mmd.
Signed-off-by: Chenxiong Qi <cqi@redhat.com>
The current code reads the data, converts them to unicode string and
then uses the `len()` of that string as filesize. This is wrong,
because Koji expects filesize to be really number of bytes, not number
of characters.
Therefore, in this commit, the filesize is computed from raw data (bytes).
This commit introduces new to_text_type helper method and calls it
for return value of all mmd.dumps() calls. That way, we always
end up with proper unicode string represntation on both python
major versions.
This commit also adds unicode character to description of all
the yaml files we use in the tests so we can be sure MBS can
handle unicode characters properly.
This might be temporary fix, depending on the result of discussion
at https://github.com/fedora-modularity/libmodulemd/issues/184.
This reverts commit ae79b711d8.
This breaks MBS with python3 and fixes only single occurence of this
issue. We want to fix this in libmodulemd or find a better way how
to fix this to work on both python2 and python3.
We have libmodulemd PR open to address this issue on libmodulemd
level: https://github.com/fedora-modularity/libmodulemd/issues/184.
When encoutering a Windows end of line (^M), io.open and open in Python 3
will convert those to UNIX end of lines by default. When reading logs
to compute the checksum, it's important those new lines aren't converted,
to ensure the checksum is correct. This caused issues in Fedora staging
because when cloning down a repo, the repoSpanner output had Windows end
of lines, and this would end up in build.log. The solution is to just read
it as binary so that Python doesn't perform these conversions.
In case the module did not contain any RPM, _koji_rpms_in_tag still
tried to call the session.getRPMHeaders with an empty list. This results
in None being returned, which is something _koji_rpms_in_tag did
not count with and failed with traceback when trying to iterate
that None value.
In this commit, the _koji_rpms_in_tag returns an empty list early
if Koji tag does not contain any RPM. Therefore the
session.getRPMHeaders is not called at all in this case, because it
does not make any sense to get RPM Headers when Koji tag does not
contain any RPM.
Current code presumes SRPM name always matches the RPM name built out of this SRPM
and only includes it together with the main package in this case. This is wrong
assumption, because usually there are multiple binary RPMs built from single SRPM.
This commit fixes that by including the SRPM NEVRA in `non_devel_source_rpms`
no matter what RPM name is.
The test RPMs are reworked in this commit to match the reality better - especially
the relations between SRPM and RPMs. The case with different SRPM name and RPM
name is also included in the reworked test - dhcp-libs binary RPM built from
dhcp SRPM.
MBS calls some read-only Koji APIs which does not require to log into a
session. This patch makes it optional to choose whether to login a
session and use anonymous session properly to call those read-only APIs.
Signed-off-by: Chenxiong Qi <cqi@redhat.com>
If an RPM is not included, its correspnding -debug* RPMs should also not
be included.
Also ensure that source RPMs are only ever added to -devel modules if
a binary RPM has been completely excluded from non-devel module.
Internal ref: FACTORY-3263
Signed-off-by: Luiz Carvalho <lucarval@redhat.com>
There is a need to ship the unchanged source file which was used to build
a module build from legal reasons. The KojiContentGenerator is extended
in this commit to fetch it from SCM URL using the `scm` module and later
attach it as `modulemd.src.txt` to Koji CG build.
This commit introduces KojiContentGenerator._sanitize_mmd method to:
- remove `repository` and `cache` from ComponentRPM in Modulemd.
- remove `mbs` section from `xmd`.
This is done to not leak internal build-only information to final
modulemd files.
For some modules, modularity-wg wants to be able to ship the "-devel" modules containing
the RPMs which are normally filtered out from the module.
This PR generates second Koji CG module with -devel suffix in a name with final modulemd
files containing the filtered out RPMs.
The original Pungi code, on which MBS code is based on, always passed only RPMs
with valid architectures to further decide if their subset should end up in a
final modulemd file.
In MBS, we pass RPMs with all architecture and there was no code to actually filter
out the RPMs which are from architectures which should never end up in a final MMD.
This commit checks that RPMs for completely different architectures will never
be considered to be included in a final MMD.
The list of arches we compare against should not include multilib
arches. Otherwise excluding does not behave correctly.
Example:
A build has ExcludeArch: i686 (because it only works on 64 bit arches).
A noarch package is built there, but it would be excluded from x86_64.
Relates: https://pagure.io/pungi/pull-request/1050
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
During I'm reading through the code base to learn MBS, I found out some
typos and minor issues in some other docstrings, and I also found some
docstrings with extra informative description could make it easier to
understand the code.
Signed-off-by: Chenxiong Qi <cqi@redhat.com>
