This puts backend specific code in either the builder or scheduler
subpackage. This puts API specific code in the new web subpackage.
Lastly, any code shared between the API and backend is placed in the
common subpackage.
The following handler arguments are not used at all:
1. `build_id` in handlers/components.py:build_task_finalize
2. `build_name` in handlers/tags.py:tagged
To support multiple backend, we need to get rid of `further_work` concept
which is used in multiple places in the MBS code. Before this commit, if
one handler wanted to execute another handler, it planned this work by
constructing fake message and returning it. MBSConsumer then planned
its execution by adding it into the event loop.
In this commit, the new `events.scheduler` instance of new Scheduler
class is used to acomplish this. If handler wants to execute another
handler, it simply schedules it using `events.scheduler.add` method.
In the end of each handler, the `events.scheduler.run` method is
executed which calls all the scheduled handlers.
The idea is that when Celery is enabled, we can change the
`Scheduler.run` method to execute the handlers using the Celery, while
during the local builds, we could execute them directly without Celery.
Use of Scheduler also fixes the issue with ordering of such calls. If
we would call the handlers directly, they could have been executed
in the middle of another handler leading to behavior incompatible
with the current `further_work` concept. Using the Scheduler, these
calls are executed always in the end of the handler no matter when
they have been scheduled.
This patch drops message objects, defined by class BaseMessage and its
subclasses, and pass event info arguments to event handler directly.
Different event handler requires different arguments to handle a kind of
specific event. The event info is parsed from the raw message received
from message bus.
Signed-off-by: Chenxiong Qi <cqi@redhat.com>
Message classes and FedmsgMessageParser are moved into dedicated Python module
under scheduler/ directory.
FedmsgMessageParser is decoupled from messaging.py by initializing a parser
object with known fedmsg services. This decouple avoids cycle import between
parser.py and messaging.py.
Signed-off-by: Chenxiong Qi <cqi@redhat.com>
Please note that this patch does not change the use of database session
in MBS. So, in the frontend, the database session is still managed by
Flask-SQLAlchemy, that is the db.session. And the backend, running event
handlers, has its own database session created from SQLAclehmy session
API directly.
This patch aims to reduce the number of scoped_session created when call
original function make_db_session. For technical detailed information,
please refer to SQLAlchemy documentation Contextual/Thread-local
Sessions.
As a result, a global scoped_session is accessible from the
code running inside backend, both the event handlers and functions
called from handlers. The library code shared by frontend and backend,
like resolvers, has no change.
Similarly, db.session is only used to recreate database for every test.
Signed-off-by: Chenxiong Qi <cqi@redhat.com>
The original code before changing to is_unbuilt is to check if any
component builds in current batch has state None. This patch fixes that.
Signed-off-by: Chenxiong Qi <cqi@redhat.com>
In this commit, when component reuse code finds out that the base module uses
KojiResolver, it uses the `KojiResolver.get_buildrequired_modules` method
to find out possible modules to reuse and limits the original query just
by the IDs of these modules.
In order to do that, this commit splits the original
`KojiResolver.get_buildrequired_modulemds` into two methods:
- The `get_buildrequired_modules` returning the ModuleBuilds.
- The `get_buildrequired_modulemds` calling the `get_buildrequired_modules`
and returning modulemd metadata.
This also removes the outdated comments around authorship of each
file. If there is still interest in this information, one can just
look at the git history.
For KojiResolver, we need to be able to find out the module to reuse
components from based only on the non-base-module buildrequires. The
base module buildrequirement will be defined by the Koji tag inheritance.
For this, we need to store build_context computed without the base
modules.
This commit:
- Adds KojiResolver class and KojiResolver tests.
- Changes the GenericResolver and its subclasses to pass base_module_mmds
instead of base_module_nsvc to get_buildrequired_modulemds. This is needed,
because KojiResolver needs to access XMD section of base module.
- Implements KojiResolver.get_buildrequired_modulemds to ask Koji for list of
modules tagged in the Koji tag and return their modulemds.
This patch attemps to make code shorter and easier to read by naming
component builds for different states.
Signed-off-by: Chenxiong Qi <cqi@redhat.com>
koji.BUILD_STATE["BUILDING"] equals to 0. So, checking if
component.state is None explicitly to avoid ignoring the components
which are in building state.
Signed-off-by: Chenxiong Qi <cqi@redhat.com>
Before this commit, the base modules used in the `get_reusable_module` have
not been sorted and therefore when `get_reusable_module` tried to find out
the reusable module built against some base module, it could find a module
built against some old version of base module.
This could lead to situation when MBS tried to reuse components from quite
old module despite the fact that newer module build existed.
This commit fixes this by sorting the base modules by stream_version,
so MBS always tries to get the reusable module built against the latest
base module.
An empty list [] is evaluated as a false value. So, it is not necessary
to compare like "if some_var == []:".
Signed-off-by: Chenxiong Qi <cqi@redhat.com>
The goal here is to define certain branches from which only scratch
module builds can be submitted. The main use case is for "private-*"
branches which can be created and maintained by anyone, but there
must not be production-ready module build created from them.
This commit adds new `scratch_build_only_branches` config option
to define the list of regexes to match such branches.
* xmd/mbs is always set if it is not present in xmd, so move the code on
the top of function. This change is also helpful for accessing keys
under xmd/mbs.
* By setting xmd/mbs in the beginning, code is simplified to get
disttag_marking and virtual_streams. The result is much straightforwar
for getting a default value for them.
* Move disttag_marking validation code next to the line getting
disttag_marking from xmd/mbs. As a result, the code structure is
easier to read as getting disttag_marking and validate it, getting
virtual_streams and validate.
* Rewrite the part of code for check_buildrequires. Always set
xmd/mbs/buildrequires if it is not present and check_buildrequires is
set to True, as it is required by
ModuleBuild.get_buildrequired_base_modules.
* Using in operator instead of dict.get to check if key koji_tag exists.
Using dict.get would be ambiguous because even if koji_tag exists
under xmd/mbs, but due to its value is set to None occasionally, there
is still a message logged to tell koji_tag is not set.
* Rwrite all lines of code for updating virtual streams. A new method
update_virtual_streams is added to ModuleBuild. This also fixes
FACTORY-4561.
* Tests are added for the rewrite of virtual streams update.
Signed-off-by: Chenxiong Qi <cqi@redhat.com>
The issue is that users don't get feedback from MBS about why a
component was not reused. There was added logic which enables to
store log messages in the database and can be viewed through the
REST api of MBS.
Ticket-ID: #1284
Signed-off-by: Martin Curlej <mcurlej@redhat.com>
When resubmitting a module build, if some component is found out that
its attributes have changed, MBS will raise an error to stop the work to
recording components. The problem is original code tells a module build
exists in database already rather than a component build, meanwhile
get_module_name() call on an ComponentRpm object is also incorrect.
Signed-off-by: Chenxiong Qi <cqi@redhat.com>
This patch separates the use of database session in different MBS components
and do not mix them together.
In general, MBS components could be separated as the REST API (implemented
based on Flask) and non-REST API including the backend build workflow
(implemented as a fedmsg consumer on top of fedmsg-hub and running
independently) and library shared by them. As a result, there are two kind of
database session used in MBS, one is created and managed by Flask-SQLAlchemy,
and another one is created from SQLAclhemy Session API directly. The goal of
this patch is to make ensure session object is used properly in the right
place.
All the changes follow these rules:
* REST API related code uses the session object db.session created and
managed by Flask-SQLAlchemy.
* Non-REST API related code uses the session object created with SQLAlchemy
Session API. Function make_db_session does that.
* Shared code does not created a new session object as much as possible.
Instead, it accepts an argument db_session.
The first two rules are applicable to tests as well.
Major changes:
* Switch tests back to run with a file-based SQLite database.
* make_session is renamed to make_db_session and SQLAlchemy connection pool
options are applied for PostgreSQL backend.
* Frontend Flask related code uses db.session
* Shared code by REST API and backend build workflow accepts SQLAlchemy session
object as an argument. For example, resolver class is constructed with a
database session, and some functions accepts an argument for database session.
* Build workflow related code use session object returned from make_db_session
and ensure db.session is not used.
* Only tests for views use db.session, and other tests use db_session fixture
to access database.
* All argument name session, that is for database access, are renamed to
db_session.
* Functions model_tests_init_data, reuse_component_init_data and
reuse_shared_userspace_init_data, which creates fixture data for
tests, are converted into pytest fixtures from original function
called inside setup_method or a test method. The reason of this
conversion is to use fixture ``db_session`` rather than create a
new one. That would also benefit the whole test suite to reduce the
number of SQLAlchemy session objects.
Signed-off-by: Chenxiong Qi <cqi@redhat.com>
