We started using `events.scheduler` to plan fake events instead of internal
queue implemented by fedmsg-hub. Thanks to that, we can stop using fedmsg-hub
completely for local builds.
In this commit, the `scheduler.local.main` is rewritten to not use fedmsg-hub.
The original `scheduler.local.main` is still needed by test_build tests and
therefore it is moved there.
It turns out we still have some backend code relies on the flask app,
so revert the change to init_config, and make minor change to create
config object for frontend and backend with the same function.
1. init_web_config: create Config object for frontend, load
configuration from `web_config.py`.
2. init_backend_config: create Config for backend, load configuration
from `backend_config.py`.
And two new classes inherit from `Config` in config.py:
1. WebConfig: representing the orchestrator frontend web configuration
2. BackendConfig: representing the orchestrator backend workers
configuration
Before calling init_{web,backend}_config, check sys.argv, if
"fedmsg-hub*", "celery" or "build_module_locally" is present, it's
running as backend.
To support multiple backend, we need to get rid of `further_work` concept
which is used in multiple places in the MBS code. Before this commit, if
one handler wanted to execute another handler, it planned this work by
constructing fake message and returning it. MBSConsumer then planned
its execution by adding it into the event loop.
In this commit, the new `events.scheduler` instance of new Scheduler
class is used to acomplish this. If handler wants to execute another
handler, it simply schedules it using `events.scheduler.add` method.
In the end of each handler, the `events.scheduler.run` method is
executed which calls all the scheduled handlers.
The idea is that when Celery is enabled, we can change the
`Scheduler.run` method to execute the handlers using the Celery, while
during the local builds, we could execute them directly without Celery.
Use of Scheduler also fixes the issue with ordering of such calls. If
we would call the handlers directly, they could have been executed
in the middle of another handler leading to behavior incompatible
with the current `further_work` concept. Using the Scheduler, these
calls are executed always in the end of the handler no matter when
they have been scheduled.
This patch allows to schedule tasks and launch workers with basic
config. To launch a worker:
celery -A module_build_service.celery_app -l info
Signed-off-by: Chenxiong Qi <cqi@redhat.com>
This patch drops message objects, defined by class BaseMessage and its
subclasses, and pass event info arguments to event handler directly.
Different event handler requires different arguments to handle a kind of
specific event. The event info is parsed from the raw message received
from message bus.
Signed-off-by: Chenxiong Qi <cqi@redhat.com>
For the purpose of migrating to Celery to run event handler inside a
worker, Config object is not serializable. And from the usage of config
argument, every event handler can just access module_build_service.conf
directly. This removal would make the migration easier.
Signed-off-by: Chenxiong Qi <cqi@redhat.com>
The accessible configured messaging backend is sigleton. This patch make
it possible to access the configured backend and avoid accessing a
"private" variable from module messaging.
Signed-off-by: Chenxiong Qi <cqi@redhat.com>
Message classes and FedmsgMessageParser are moved into dedicated Python module
under scheduler/ directory.
FedmsgMessageParser is decoupled from messaging.py by initializing a parser
object with known fedmsg services. This decouple avoids cycle import between
parser.py and messaging.py.
Signed-off-by: Chenxiong Qi <cqi@redhat.com>
MBS will iterate through all the builds in buildrequires to determine
the expected list of arches on the associated Koji tag. In some cases,
these builds do not have a Koji tag. They should be skipped for this
operation.
Signed-off-by: Luiz Carvalho <lucarval@redhat.com>
Please note that this patch does not change the use of database session
in MBS. So, in the frontend, the database session is still managed by
Flask-SQLAlchemy, that is the db.session. And the backend, running event
handlers, has its own database session created from SQLAclehmy session
API directly.
This patch aims to reduce the number of scoped_session created when call
original function make_db_session. For technical detailed information,
please refer to SQLAlchemy documentation Contextual/Thread-local
Sessions.
As a result, a global scoped_session is accessible from the
code running inside backend, both the event handlers and functions
called from handlers. The library code shared by frontend and backend,
like resolvers, has no change.
Similarly, db.session is only used to recreate database for every test.
Signed-off-by: Chenxiong Qi <cqi@redhat.com>
This will result in the error actually being in the state reason
of the failed module build instead of "An unknown error occurred
while validating the modulemd".
The original code before changing to is_unbuilt is to check if any
component builds in current batch has state None. This patch fixes that.
Signed-off-by: Chenxiong Qi <cqi@redhat.com>
A new method _map_message is added for converting a message object to
corresponding event handler and module build. This is used to shorten the
process_message method.
process_message is refactored so that:
* when handler is NO_OP, just return as earlier as possible because setting it
make no sense to set MBSConsumer.current_module_build_id back and forth.
* Re-raise error thrown from handler execution so that the caller is able to
get a chance to collect monitoring metric inside except clause handling
Exception. Otherwise, no failed metric is collected. This is the major
problem this patch is to fix.
* Ensure MBSConsumer.current_module_build_id is set back to None.
In consumer method, no traceback is logged inside except clause catching
Exception. process_method does that.
Signed-off-by: Chenxiong Qi <cqi@redhat.com>
