Some of the config vars have default values, it is unncessary to keep
them in settings.
Note: Some vars are still in conf/config.py even they have the default
values, like the '*_url' vars, they're kept there just as placeholder
and reminder for changing while deploying the service.
Revert the changes in a207d97 and 9a2efb3, at this moment, some code
relies on both settings from frontend and backend, and it's inconvenient
to run unittests with config split up to frontend only and backend
only.
Poller methods within original class MBSProducer become module level
functions and are registered as Celery periodic tasks.
Code logging the size of fedmsg-hub queue are removed from log_summary.
process_open_component_builds is still kept there and not converted to a
periodic task.
There are some small refactor:
* do not format string in logging method call.
* reformat some lines of code doing SQLAlchemy database query to make
them more readable.
Signed-off-by: Chenxiong Qi <cqi@redhat.com>
We started using `events.scheduler` to plan fake events instead of internal
queue implemented by fedmsg-hub. Thanks to that, we can stop using fedmsg-hub
completely for local builds.
In this commit, the `scheduler.local.main` is rewritten to not use fedmsg-hub.
The original `scheduler.local.main` is still needed by test_build tests and
therefore it is moved there.
It turns out we still have some backend code relies on the flask app,
so revert the change to init_config, and make minor change to create
config object for frontend and backend with the same function.
1. init_web_config: create Config object for frontend, load
configuration from `web_config.py`.
2. init_backend_config: create Config for backend, load configuration
from `backend_config.py`.
And two new classes inherit from `Config` in config.py:
1. WebConfig: representing the orchestrator frontend web configuration
2. BackendConfig: representing the orchestrator backend workers
configuration
Before calling init_{web,backend}_config, check sys.argv, if
"fedmsg-hub*", "celery" or "build_module_locally" is present, it's
running as backend.
To support multiple backend, we need to get rid of `further_work` concept
which is used in multiple places in the MBS code. Before this commit, if
one handler wanted to execute another handler, it planned this work by
constructing fake message and returning it. MBSConsumer then planned
its execution by adding it into the event loop.
In this commit, the new `events.scheduler` instance of new Scheduler
class is used to acomplish this. If handler wants to execute another
handler, it simply schedules it using `events.scheduler.add` method.
In the end of each handler, the `events.scheduler.run` method is
executed which calls all the scheduled handlers.
The idea is that when Celery is enabled, we can change the
`Scheduler.run` method to execute the handlers using the Celery, while
during the local builds, we could execute them directly without Celery.
Use of Scheduler also fixes the issue with ordering of such calls. If
we would call the handlers directly, they could have been executed
in the middle of another handler leading to behavior incompatible
with the current `further_work` concept. Using the Scheduler, these
calls are executed always in the end of the handler no matter when
they have been scheduled.
This patch allows to schedule tasks and launch workers with basic
config. To launch a worker:
celery -A module_build_service.celery_app -l info
Signed-off-by: Chenxiong Qi <cqi@redhat.com>
This patch drops message objects, defined by class BaseMessage and its
subclasses, and pass event info arguments to event handler directly.
Different event handler requires different arguments to handle a kind of
specific event. The event info is parsed from the raw message received
from message bus.
Signed-off-by: Chenxiong Qi <cqi@redhat.com>
For the purpose of migrating to Celery to run event handler inside a
worker, Config object is not serializable. And from the usage of config
argument, every event handler can just access module_build_service.conf
directly. This removal would make the migration easier.
Signed-off-by: Chenxiong Qi <cqi@redhat.com>
The accessible configured messaging backend is sigleton. This patch make
it possible to access the configured backend and avoid accessing a
"private" variable from module messaging.
Signed-off-by: Chenxiong Qi <cqi@redhat.com>
Message classes and FedmsgMessageParser are moved into dedicated Python module
under scheduler/ directory.
FedmsgMessageParser is decoupled from messaging.py by initializing a parser
object with known fedmsg services. This decouple avoids cycle import between
parser.py and messaging.py.
Signed-off-by: Chenxiong Qi <cqi@redhat.com>
Update the integration test README to explain the design of the test
case and test environment code. This should help future contributors.
Signed-off-by: Hunor Csomortáni <csomh@redhat.com>
Until now, it was assumed that the module-build command returned only
one build, so it was only one build_id. However, it is possible that
the module-build command will build more than one builds and therefore
a list of build_ids is needed. Also is needed to watch and cancel more
than one build.
For this reason run, watch, and cancel methods are methods of the
PackagingUtility class instead of Build class. Run method returns list
of Build objects instead of build_id. And it's also possible to cancel
and to watch on all generated module builds.
'rpm' is an optional dependency for kobo, which is required by the
integration tests.
Previously this dependency was implicitly satisfied by 'koji' pulling
in 'rpm-py-installer' which made sure 'rpm' is available. But
koji==1.20.0 made this dependency optional, so importing kobo.rpmlib
started failing.
Fix this by making 'rpm-py-installer' and explicit dependency for the
integration test environment.
Signed-off-by: Hunor Csomortáni <csomh@redhat.com>
Until now, to access the configuration of a scenario, the full path in
test_env had to be specified.
This might be cumbersome and error prone.
Create a scenario fixture, to make it easier to access the same
configuration.
Instead of
test_env["testdata"]["my_scenario"]["my_config"]
one can use the following:
scenario["my_config"]
Signed-off-by: Hunor Csomortáni <csomh@redhat.com>
