Verbose output from mock is not useful for someone trying to figure out
why their module build failed, and in fact makes it harder by adding
quite a bit of extraneous noise.
Mock doesn't normally log anything to stdout - so it's confusing to mention
separate logs in the messages. Combine the two output streams together.
(This is what koji does as well.)
The different build threads all are using the same basic build root
contents, so there's no reason to use separate caches - point the
root cache plugin for mock to a single location. (There's locking
inside Mock for updating the root cache.)
The mod time for the mock configuration file is used to determine
whether the root cache is out-of-date or not, so we want to avoid
changing the configuration timestamps when we don't change content
when we're just writing a per-thread mock configuration file again
with no substantive changes.
We do this by only updating the master mock.cfg file when we're
actually adding content, and propagating its mod time to the
per-thread configuration files.
When we want to pass in SCM options particular to a specific module build,
do that on the mock command line rather than by modifying mock.cfg - this
avoids invalidating the root cache.
For local builds, required modules are not necessarily in the local
database, so the method of looking up the build to find the koji tag
doesn't work reliably. However, the caller has the koji_tag - so just
pass it in.
When downloading files from Koji to make a local repository, display
a temporary status of which files are being displayed to the console
appended after any log messages. Updates are done by erasing the status
that was written, adding a log message, then writing the status again.
In MBS, there are two cases to send a message when a module build moves
to a new state. One is to create a new module build, with
ModuleBuild.create particularly, when user submit a module build.
Another one is to transition a module build to a new state with
ModuleBuild.transition. This commit handles these two cases in a little
different ways.
For the former, existing code is refactored by moving the publish call
outside ModuleBuild.create.
For the latter, message is sent in a hook of SQLAlchemy ORM event
after_commit rather than immediately inside the ModuleBuild.transition.
Both of these changes ensure the message is sent after the changes are
committed into database successfully. Then, the backend can have
confidence that the database has the module build data when receive a
message.
Signed-off-by: Chenxiong Qi <cqi@redhat.com>
This also includes `from __future__ import absolute_import`
in every file so that the imports are consistent in Python 2 and 3.
The Python 2 tests fail without this.
This moves the code used by the backend and API to common/submit.py,
the code used just by the API to web/submit.py, and the code used
just by the backend to scheduler/submit.py.
This puts backend specific code in either the builder or scheduler
subpackage. This puts API specific code in the new web subpackage.
Lastly, any code shared between the API and backend is placed in the
common subpackage.
The following handler arguments are not used at all:
1. `build_id` in handlers/components.py:build_task_finalize
2. `build_name` in handlers/tags.py:tagged
It turns out we still have some backend code relies on the flask app,
so revert the change to init_config, and make minor change to create
config object for frontend and backend with the same function.
1. init_web_config: create Config object for frontend, load
configuration from `web_config.py`.
2. init_backend_config: create Config for backend, load configuration
from `backend_config.py`.
And two new classes inherit from `Config` in config.py:
1. WebConfig: representing the orchestrator frontend web configuration
2. BackendConfig: representing the orchestrator backend workers
configuration
Before calling init_{web,backend}_config, check sys.argv, if
"fedmsg-hub*", "celery" or "build_module_locally" is present, it's
running as backend.
To support multiple backend, we need to get rid of `further_work` concept
which is used in multiple places in the MBS code. Before this commit, if
one handler wanted to execute another handler, it planned this work by
constructing fake message and returning it. MBSConsumer then planned
its execution by adding it into the event loop.
In this commit, the new `events.scheduler` instance of new Scheduler
class is used to acomplish this. If handler wants to execute another
handler, it simply schedules it using `events.scheduler.add` method.
In the end of each handler, the `events.scheduler.run` method is
executed which calls all the scheduled handlers.
The idea is that when Celery is enabled, we can change the
`Scheduler.run` method to execute the handlers using the Celery, while
during the local builds, we could execute them directly without Celery.
Use of Scheduler also fixes the issue with ordering of such calls. If
we would call the handlers directly, they could have been executed
in the middle of another handler leading to behavior incompatible
with the current `further_work` concept. Using the Scheduler, these
calls are executed always in the end of the handler no matter when
they have been scheduled.
This patch drops message objects, defined by class BaseMessage and its
subclasses, and pass event info arguments to event handler directly.
Different event handler requires different arguments to handle a kind of
specific event. The event info is parsed from the raw message received
from message bus.
Signed-off-by: Chenxiong Qi <cqi@redhat.com>
Message classes and FedmsgMessageParser are moved into dedicated Python module
under scheduler/ directory.
FedmsgMessageParser is decoupled from messaging.py by initializing a parser
object with known fedmsg services. This decouple avoids cycle import between
parser.py and messaging.py.
Signed-off-by: Chenxiong Qi <cqi@redhat.com>
