modules/fm-orchestrator
1
0
Fork 0
mirror of https://pagure.io/fm-orchestrator.git synced 2026-11-05 06:39:49 +08:00
Code Issues Actions Packages Projects Releases Wiki Activity

Rename module from rida to module_build_service

Browse Source 
Rename routes from /rida/1/module-builds/ to /module-build-service/1/module-builds/
This commit is contained in:
Matt Prahl
2016-10-21 10:40:44 -04:00
parent a551ad9214
commit b4082dc551
37 changed files with 276 additions and 275 deletions
Download Patch File Download Diff File Expand all files Collapse all files

2
.gitignore vendored
View File

@@ -1,5 +1,5 @@
*.pyc
rida.db
module_build_service.db
server.crt
server.key
.vagrant

10
Dockerfile
View File

@@ -3,14 +3,14 @@ FROM fedora:24
# so we don't have to compile those when fetched from PyPI
RUN dnf install -y python-pip python2-setuptools python2-cffi python2-zmq python2-cryptography koji python2-pdc-client python-m2ext && \
dnf autoremove -y && dnf clean all && \
mkdir /opt/fm-orchestrator/
WORKDIR /opt/fm-orchestrator/
COPY ./requirements.txt /opt/fm-orchestrator/
mkdir /opt/module_build_service/
WORKDIR /opt/module_build_service/
COPY ./requirements.txt /opt/module_build_service/
RUN pip install --user -r ./requirements.txt
COPY koji.conf /etc/rida/
COPY koji.conf /etc/module_build_service/
COPY . /opt/fm-orchestrator/
COPY . /opt/module_build_service/
RUN python2 ./manage.py upgradedb && python2 manage.py generatelocalhostcert
CMD ["python2", "manage.py", "runssl"]

56
README.rst
View File

@@ -1,5 +1,5 @@
The module build orchestrator for Modularity
============================================
The module build service for Modularity
=======================================
The orchestrator coordinates module builds and is responsible for a number of
tasks:
@@ -27,7 +27,7 @@ Module submission is done via posting the modulemd SCM URL.
::
POST /rida/module-builds/
POST /module-build-service/1/module-builds/
::
@@ -55,7 +55,7 @@ build task's URL. Querying the BPO service might be preferred, however.
::
GET /rida/module-builds/42
GET /module-build-service/1/module-builds/42
The response, if the task exists, would include various pieces of information
about the referenced build task.
@@ -93,7 +93,7 @@ There are a number of configurable GET parameters to change how the module build
An example of querying the "module-builds" resource without any additional parameters::
GET /rida/module-builds/
GET /module-build-service/1/module-builds/
::
@@ -145,9 +145,9 @@ An example of querying the "module-builds" resource without any additional param
}
],
"meta": {
"first": "https://rida.fedora.local:5000/rida/module-builds/?per_page=10&page=1",
"last": "https://rida.fedora.local:5000/rida/module-builds/?per_page=10&page=3",
"next": "https://rida.fedora.local:5000/rida/module-builds/?per_page=10&page=2",
"first": "https://127.0.0.1:5000/module-build-service/1/module-builds/?per_page=10&page=1",
"last": "https://127.0.0.1:5000/module-build-service/1/module-builds/?per_page=10&page=3",
"next": "https://127.0.0.1:5000/module-build-service/1/module-builds/?per_page=10&page=2",
"page": 1,
"pages": 3,
"per_page": 10,
@@ -158,7 +158,7 @@ An example of querying the "module-builds" resource without any additional param
An example of querying the "module-builds" resource with the "verbose", "per_page", and the "page" parameters::
GET /rida/module-builds/?verbose=true&per_page=3&page=1
GET /module-build-service/1/module-builds/?verbose=true&per_page=3&page=1
::
@@ -209,9 +209,9 @@ An example of querying the "module-builds" resource with the "verbose", "per_pag
}
],
"meta": {
"first": "https://127.0.0.1:5000/rida/module-builds/?per_page=3&page=1",
"last": "https://127.0.0.1:5000/rida/module-builds/?per_page=3&page=10",
"next": "https://127.0.0.1:5000/rida/module-builds/?per_page=3&page=2",
"first": "https://127.0.0.1:5000/module-build-service/1/module-builds/?per_page=3&page=1",
"last": "https://127.0.0.1:5000/module-build-service/1/module-builds/?per_page=3&page=10",
"next": "https://127.0.0.1:5000/module-build-service/1/module-builds/?per_page=3&page=2",
"page": 1,
"pages": 10,
"per_page": 3,
@@ -236,7 +236,7 @@ The module-builds can be filtered by a variety of GET parameters. These paramter
An example of querying the "module-builds" resource with the "state", and the "submitted_before" parameters::
GET /rida/module-builds/?state=done&submitted_before=2016-08-23T08:10:07Z
GET /module-build-service/1/module-builds/?state=done&submitted_before=2016-08-23T08:10:07Z
::
@@ -260,8 +260,8 @@ An example of querying the "module-builds" resource with the "state", and the "s
}
],
"meta": {
"first": "https://127.0.0.1:5000/rida/module-builds/?per_page=10&page=1",
"last": "https://127.0.0.1:5000/rida/module-builds/?per_page=10&page=1",
"first": "https://127.0.0.1:5000/module-build-service/1/module-builds/?per_page=10&page=1",
"last": "https://127.0.0.1:5000/module-build-service/1/module-builds/?per_page=10&page=1",
"page": 1,
"pages": 1,
"per_page": 3,
@@ -292,7 +292,7 @@ _`Module Build States`
You can see the list of possible states with::
from rida.models import BUILD_STATES
from module_build_service.models import BUILD_STATES
print(BUILD_STATES)
Here's a description of what each of them means:
@@ -346,8 +346,8 @@ Message Topic
The suffix for message topics concerning changes in module state is
``module.state.change``. Currently, it is expected that these messages are sent
from koji or ridad, i.e. the topic is prefixed with ``*.buildsys.`` or
``*.rida.``, respectively.
from koji or module_build_service_daemon, i.e. the topic is prefixed with ``*.buildsys.`` or
``*.module_build_service.``, respectively.
Message Body
------------
@@ -385,8 +385,8 @@ Development
``Logging``
----------
If you're running rida from scm then the DevConfiguration from config.py which contains LOG_LEVEL=debug should get applied. If you're having trouble just change LOG_LEVEL in BaseConfiguration.
See more about it in rida/__init__.py config.from_object()
If you're running module_build_service from scm then the DevConfiguration from config.py which contains LOG_LEVEL=debug should get applied. If you're having trouble just change LOG_LEVEL in BaseConfiguration.
See more about it in module_build_service/__init__.py config.from_object()
``Docker``
@@ -416,9 +416,9 @@ Once your environment is setup, run (depending on your OS, you may need to run i
$ vagrant up
This will start fm-orchestrator's frontend (API) and scheduler. To access the frontend, visit the following URL::
This will start module_build_service's frontend (API) and scheduler. To access the frontend, visit the following URL::
https://127.0.0.1:5000/rida/1/module-builds/
https://127.0.0.1:5000/module-build-service/1/module-builds/
At any point you may enter the guest VM with::
@@ -426,13 +426,13 @@ At any point you may enter the guest VM with::
To start the frontend manually, run the following inside the guest::
$ cd /opt/fm-orchestrator/src
$ cd /opt/module_build_service/src
$ python manage.py runssl --debug
To start the scheduler manually, run the following inside the guest::
$ cd /opt/fm-orchestrator/src
$ python ridad.py
$ cd /opt/module_build_service/src
$ python module_build_service_daemon.py
Alternatively, you can restart the Vagrant guest, which inherently starts/restarts the frontend and the scheduler with::
@@ -453,11 +453,11 @@ Setup Apache to host the CRL::
$ dnf install httpd && systemctl enable httpd && systemctl start httpd
$ mkdir -p /var/www/html/crl
$ ln -s /opt/fm-orchestrator/pki/ca.crl /var/www/html/crl/ca.crl
$ ln -s /opt/fm-orchestrator/pki/ca.crt /var/www/html/crl/ca.crt
$ ln -s /opt/module_build_service/pki/ca.crl /var/www/html/crl/ca.crl
$ ln -s /opt/module_build_service/pki/ca.crt /var/www/html/crl/ca.crt
Create a directory to house the fedmsg cache::
$ mkdir -p /etc/pki/fedmsg
Then uncomment the fedmsg signing configuration in fedmsg.d/rida.py.
Then uncomment the fedmsg signing configuration in fedmsg.d/module_build_service.py.

16
Vagrantfile vendored
View File

@@ -3,11 +3,11 @@
$script = <<SCRIPT
dnf install -y python python-virtualenv python-devel libffi-devel redhat-rpm-config openssl-devel gcc gcc-c++ koji git swig
pip install -r /opt/fm-orchestrator/src/requirements.txt
pip install -r /opt/fm-orchestrator/src/test-requirements.txt
cd /opt/fm-orchestrator/src
mkdir -p /etc/rida
cp -av koji.conf /etc/rida/
pip install -r /opt/module_build_service/src/requirements.txt
pip install -r /opt/module_build_service/src/test-requirements.txt
cd /opt/module_build_service/src
mkdir -p /etc/module_build_service
cp -av koji.conf /etc/module_build_service/
python manage.py upgradedb
python manage.py generatelocalhostcert
cp /home/vagrant/.fedora-server-ca.cert /root/.fedora-server-ca.cert
@@ -17,12 +17,12 @@ SCRIPT
Vagrant.configure("2") do |config|
config.vm.box = "boxcutter/fedora24"
config.vm.synced_folder "./", "/opt/fm-orchestrator/src"
config.vm.synced_folder "./", "/opt/module_build_service/src"
config.vm.provision "file", source: "~/.fedora-server-ca.cert", destination: "~/.fedora-server-ca.cert"
config.vm.provision "file", source: "~/.fedora-upload-ca.cert", destination: "~/.fedora-upload-ca.cert"
config.vm.provision "file", source: "~/.fedora.cert", destination: "~/.fedora.cert"
config.vm.network "forwarded_port", guest: 5000, host: 5000
config.vm.provision "shell", inline: $script
config.vm.provision :shell, inline: "cd /opt/fm-orchestrator/src && python manage.py runssl --debug &", run: "always"
config.vm.provision :shell, inline: "cd /opt/fm-orchestrator/src && python ridad.py &", run: "always"
config.vm.provision :shell, inline: "cd /opt/module_build_service/src && python manage.py runssl --debug &", run: "always"
config.vm.provision :shell, inline: "cd /opt/module_build_service/src && python module_build_service_daemon.py &", run: "always"
end

6
cleanup-koji-stg.sh
View File

@@ -3,7 +3,7 @@
# default's *42* lkocman's *43 as well
for mvr in testmodule-4.3.43-1 testmodule-4.3.42-1; do
koji --config /etc/rida/koji.conf remove-target $mvr
koji --config /etc/rida/koji.conf remove-tag $mvr
koji --config /etc/rida/koji.conf remove-tag $mvr-build
koji --config /etc/module_build_service/koji.conf remove-target $mvr
koji --config /etc/module_build_service/koji.conf remove-tag $mvr
koji --config /etc/module_build_service/koji.conf remove-tag $mvr-build
done

17
config.py
View File

@@ -5,7 +5,8 @@ class BaseConfiguration(object):
# Make this random (used to generate session keys)
SECRET_KEY = '74d9e9f9cd40e66fc6c4c2e9987dce48df3ce98542529fd0'
basedir = path.abspath(path.dirname(__file__))
SQLALCHEMY_DATABASE_URI = 'sqlite:///{0}'.format(path.join(basedir, 'rida.db'))
SQLALCHEMY_DATABASE_URI = 'sqlite:///{0}'.format(path.join(
basedir, 'module_build_service.db'))
SQLALCHEMY_TRACK_MODIFICATIONS = True
# Where we should run when running "manage.py runssl" directly.
HOST = '127.0.0.1'
@@ -13,7 +14,7 @@ class BaseConfiguration(object):
SYSTEM = 'koji'
MESSAGING = 'fedmsg' # or amq
KOJI_CONFIG = '/etc/rida/koji.conf'
KOJI_CONFIG = '/etc/module_build_service/koji.conf'
KOJI_PROFILE = 'koji'
KOJI_ARCHES = ['i686', 'armv7hl', 'x86_64']
PDC_URL = 'http://modularity.fedorainfracloud.org:8080/rest_api/v1'
@@ -43,7 +44,7 @@ class BaseConfiguration(object):
LOG_BACKEND = 'journal'
# Path to log file when LOG_BACKEND is set to "file".
LOG_FILE = 'rida.log'
LOG_FILE = 'module_build_service.log'
# Available log levels are: debug, info, warn, error.
LOG_LEVEL = 'info'
@@ -56,12 +57,12 @@ class BaseConfiguration(object):
# AMQ prefixed variables are required only while using 'amq' as messaging backend
# Addresses to listen to
AMQ_RECV_ADDRESSES = ['amqps://messaging.mydomain.com/Consumer.m8y.VirtualTopic.eng.koji',
'amqps://messaging.mydomain.com/Consumer.m8y.VirtualTopic.eng.rida',]
'amqps://messaging.mydomain.com/Consumer.m8y.VirtualTopic.eng.module_build_service',]
# Address for sending messages
AMQ_DEST_ADDRESS = 'amqps://messaging.mydomain.com/Consumer.m8y.VirtualTopic.eng.rida'
AMQ_CERT_FILE = '/etc/rida/msg-m8y-client.crt'
AMQ_PRIVATE_KEY_FILE = '/etc/rida/msg-m8y-client.key'
AMQ_TRUSTED_CERT_FILE = '/etc/rida/Root-CA.crt'
AMQ_DEST_ADDRESS = 'amqps://messaging.mydomain.com/Consumer.m8y.VirtualTopic.eng.module_build_service'
AMQ_CERT_FILE = '/etc/module_build_service/msg-m8y-client.crt'
AMQ_PRIVATE_KEY_FILE = '/etc/module_build_service/msg-m8y-client.key'
AMQ_TRUSTED_CERT_FILE = '/etc/module_build_service/Root-CA.crt'
class DevConfiguration(BaseConfiguration):
LOG_BACKEND = 'console'

12
docker-compose.yml
View File

@@ -1,20 +1,20 @@
version: "2"
services:
ridad:
module_build_service_daemon:
build:
context: .
command: python2 /opt/fm-orchestrator/ridad.py
command: python2 /opt/module_build_service/module_build_service_daemon.py
volumes:
- ./rida/:/opt/fm-orchestrator/rida/:Z # rida sources are bind-mounted, so you don't have to rebuild
- ./module_build_service/:/opt/module_build_service/module_build_service/:Z # module_build_service sources are bind-mounted, so you don't have to rebuild
- ~/.fedora.cert:/root/.fedora.cert:Z # koji credentials are mounted inside
- ~/.fedora-server-ca.cert:/root/.fedora-server-ca.cert:Z
- ~/.fedora-upload-ca.cert:/root/.fedora-upload-ca.cert:Z
rida:
module_build_service:
build:
context: .
ports:
- "5000:5000"
links:
- ridad
- module_build_service_daemon
volumes:
- ./rida/:/opt/fm-orchestrator/rida/:Z
- ./module_build_service/:/opt/module_build_service/module_build_service/:Z

4
fedmsg.d/rida.py → fedmsg.d/module_build_service.py
View File

@@ -13,7 +13,7 @@ config = {
# 'validate_signatures': True,
# 'crypto_backend': 'x509',
# 'crypto_validate_backends': ['x509'],
# 'ssldir': '/opt/fm-orchestrator/pki',
# 'ssldir': '/opt/module_build_service/pki',
# 'crl_location': 'http://localhost/crl/ca.crl',
# 'crl_cache': '/etc/pki/fedmsg/crl.pem',
# 'crl_cache_expiry': 10,
@@ -21,7 +21,7 @@ config = {
# 'ca_cert_cache': '/etc/pki/fedmsg/ca.crt',
# 'ca_cert_cache_expiry': 0, # Never expires
# 'certnames': {
# 'rida.localhost': 'localhost'
# 'module_build_service.localhost': 'localhost'
# }
# End of code signing configuration
}

8
jenkins-check-Dockerfile
View File

@@ -5,9 +5,9 @@ FROM fedora:24
# so we don't have to compile those when fetched from PyPI
RUN dnf install -y python python-virtualenv python-devel libffi-devel redhat-rpm-config openssl-devel gcc gcc-c++ koji git swig && \
dnf autoremove -y && dnf clean all && \
mkdir /opt/fm-orchestrator/
WORKDIR /opt/fm-orchestrator/
COPY ./requirements.txt /opt/fm-orchestrator/
mkdir /opt/module_build_service/
WORKDIR /opt/module_build_service/
COPY ./requirements.txt /opt/module_build_service/
RUN pip install --user -r ./requirements.txt
# This isn't what we do in jenkins, blah ...
@@ -18,6 +18,6 @@ RUN pip install --upgrade mock
# Need to force reinstall so nose gets the virtualenv modules.
RUN pip install nose --ignore-installed
COPY . /opt/fm-orchestrator/
COPY . /opt/module_build_service/
CMD ["nosetests", "."]

4
jenkins-check.sh
View File

@@ -1,4 +1,4 @@
#! /bin/sh -e
sudo docker build -f jenkins-check-Dockerfile -t fm-orchestrator/jenkins-check-24 .
sudo docker run fm-orchestrator/jenkins-check-24
sudo docker build -f jenkins-check-Dockerfile -t module_build_service/jenkins-check-24 .
sudo docker run module_build_service/jenkins-check-24

12
manage.py
View File

@@ -28,12 +28,12 @@ import os
import ssl
from shutil import rmtree
from rida import app, conf, db
from rida.config import Config
from rida.pdc import (
from module_build_service import app, conf, db
from module_build_service.config import Config
from module_build_service.pdc import (
get_pdc_client_session, get_module, get_module_runtime_dependencies,
get_module_tag, get_module_build_dependencies)
import rida.auth
import module_build_service.auth
manager = Manager(app)
@@ -98,7 +98,7 @@ def upgradedb():
@manager.command
def gendevfedmsgcert(pki_dir='/opt/fm-orchestrator/pki', force=False):
def gendevfedmsgcert(pki_dir='/opt/module_build_service/pki', force=False):
"""
Creates a CA, a certificate signed by that CA, and generates a CRL.
"""
@@ -242,7 +242,7 @@ def runssl(host=conf.host, port=conf.port, debug=False):
app.run(
host=host,
port=port,
request_handler=rida.auth.ClientCertRequestHandler,
request_handler=module_build_service.auth.ClientCertRequestHandler,
ssl_context=ssl_ctx,
debug=debug
)

15
rida/__init__.py → module_build_service/__init__.py
View File

@@ -44,10 +44,11 @@ for a number of tasks:
from flask import Flask
from flask_sqlalchemy import SQLAlchemy
from os import sys
import rida.logger
import module_build_service.logger
from logging import getLogger
from rida.errors import (ValidationError, Unauthorized, UnprocessableEntity,
Conflict, NotFound, Forbidden, json_error)
from module_build_service.errors import (
ValidationError, Unauthorized, UnprocessableEntity, Conflict, NotFound,
Forbidden, json_error)
app = Flask(__name__)
app.config.from_envvar("RIDA_SETTINGS", silent=True)
@@ -102,9 +103,9 @@ def notfound_error(e):
"""Flask error handler for Conflict exceptions"""
return json_error(404, 'Not Found', e.args[0])
import rida.config
conf = rida.config.from_app_config()
rida.logger.init_logging(conf)
import module_build_service.config
conf = module_build_service.config.from_app_config()
module_build_service.logger.init_logging(conf)
log = getLogger(__name__)
from rida import views
from module_build_service import views

2
rida/auth.py → module_build_service/auth.py
View File

@@ -25,7 +25,7 @@
from werkzeug.serving import WSGIRequestHandler
from rida.errors import Unauthorized
from module_build_service.errors import Unauthorized
import fedora.client

20
rida/builder.py → module_build_service/builder.py
View File

@@ -46,12 +46,12 @@ import kobo.rpmlib
import munch
from OpenSSL.SSL import SysCallError
from rida import log
import rida.utils
from module_build_service import log
import module_build_service.utils
logging.basicConfig(level=logging.DEBUG)
# TODO: read defaults from rida's config
# TODO: read defaults from module_build_service's config
KOJI_DEFAULT_GROUPS = {
'build': [
'bash',
@@ -118,7 +118,7 @@ class GenericBuilder(six.with_metaclass(ABCMeta)):
External Api for builders
Example usage:
config = rida.config.Config()
config = module_build_service.config.Config()
builder = Builder(module="testmodule-1.2-3", backend="koji", config)
builder.buildroot_connect()
builder.build(artifact_name="bash",
@@ -229,7 +229,7 @@ class Builder(object):
:param owner: a string representing who kicked off the builds
:param module: a module string e.g. 'testmodule-1.0'
:param backend: a string representing backend e.g. 'koji'
:param config: instance of rida.config.Config
:param config: instance of module_build_service.config.Config
Any additional arguments are optional extras which can be passed along
and are implementation-dependent.
@@ -251,7 +251,7 @@ class KojiModuleBuilder(GenericBuilder):
"""
:param owner: a string representing who kicked off the builds
:param module: string representing module
:param config: rida.config.Config instance
:param config: module_build_service.config.Config instance
:param tag_name: name of tag for given module
"""
self.owner = owner
@@ -275,7 +275,7 @@ class KojiModuleBuilder(GenericBuilder):
return "<KojiModuleBuilder module: %s, tag: %s>" % (
self.module_str, self.tag_name)
@rida.utils.retry(wait_on=koji.GenericError)
@module_build_service.utils.retry(wait_on=koji.GenericError)
def buildroot_ready(self, artifacts=None):
"""
:param artifacts=None - list of nvrs
@@ -350,7 +350,7 @@ chmod 644 %buildroot/%_rpmconfigdir/macros.d/macros.modules
* {today} Fedora-Modularity - {version}-{release}{disttag}
- autogenerated macro by Rida "The Orchestrator"
""".format(disttag=disttag, today=today, name=name, version=version, release=release)
td = tempfile.mkdtemp(prefix="rida-build-macros")
td = tempfile.mkdtemp(prefix="module_build_service-build-macros")
fd = open(os.path.join(td, "%s.spec" % name), "w")
fd.write(spec_content)
fd.close()
@@ -413,7 +413,7 @@ chmod 644 %buildroot/%_rpmconfigdir/macros.d/macros.modules
# TODO: handle in buildroot_add_artifact(install=true) and track groups as module buildrequires
groups = KOJI_DEFAULT_GROUPS
if groups:
@rida.utils.retry(wait_on=SysCallError, interval=5)
@module_build_service.utils.retry(wait_on=SysCallError, interval=5)
def add_groups():
return self._koji_add_groups_to_tag(
dest_tag=self.module_build_tag,
@@ -461,7 +461,7 @@ chmod 644 %buildroot/%_rpmconfigdir/macros.d/macros.modules
log.info("Waiting for task_id=%s to finish" % task_id)
timeout = 60 * 60 # 60 minutes
@rida.utils.retry(timeout=timeout, wait_on=koji.GenericError)
@module_build_service.utils.retry(timeout=timeout, wait_on=koji.GenericError)
def get_result():
log.debug("Waiting for task_id=%s to finish" % task_id)
task = self.koji_session.getTaskResult(task_id)

6
rida/config.py → module_build_service/config.py
View File

@@ -27,8 +27,8 @@
import six
from rida import app
from rida import logger
from module_build_service import app
from module_build_service import logger
def asbool(value):
""" Cast config values to boolean. """
@@ -76,7 +76,7 @@ class Config(object):
self._log_level = 0
self._krb_keytab = None
self._krb_principal = None
self._krb_ccache = "/tmp/krb5cc_rida"
self._krb_ccache = "/tmp/krb5cc_module_build_service"
self._messaging = ""
self._amq_recv_addresses = []
self._amq_dest_address = ""

0
rida/errors.py → module_build_service/errors.py
View File

0
rida/logger.py → module_build_service/logger.py
View File

16
rida/messaging.py → module_build_service/messaging.py
View File

@@ -32,7 +32,7 @@ try:
except ImportError:
from funcsigs import signature
from rida import log
from module_build_service import log
class BaseMessage(object):
@@ -63,7 +63,7 @@ class BaseMessage(object):
properties = json.loads(msg.properties, encoding='utf8')
service = properties.get('service')
if service not in ('koji', 'rida'):
if service not in ('koji', 'module_build_service'):
log.debug('Skipping msg due service=%s which is not related (msg=%r): ' % (service, msg))
return None
@@ -102,8 +102,8 @@ class BaseMessage(object):
msg.id, build_id, build_state, build_name,
build_version, build_release)
elif service == 'rida':
log.debug("Found rida related msg: %s" % msg)
elif service == 'module_build_service':
log.debug("Found module_build_service related msg: %s" % msg)
body = json.loads(msg.body, encoding='utf8')
if topic == 'module.state.change':
msg_obj = RidaModule(
@@ -125,7 +125,7 @@ class BaseMessage(object):
that the app looks for, otherwise None is returned
"""
regex_pattern = re.compile(
(r'(?P<category>buildsys|rida)(?:\.)'
(r'(?P<category>buildsys|module_build_service)(?:\.)'
r'(?P<object>build|repo|module)(?:(?:\.)'
r'(?P<subobject>state))?(?:\.)(?P<event>change|done)$'))
regex_results = re.search(regex_pattern, topic)
@@ -164,7 +164,7 @@ class BaseMessage(object):
repo_tag = msg_inner_msg.get('tag')
msg_obj = KojiRepoChange(msg_id, repo_tag)
elif category == 'rida' and object == 'module' and \
elif category == 'module_build_service' and object == 'module' and \
subobject == 'state' and event == 'change':
msg_obj = RidaModule(
msg_id, msg_inner_msg.get('id'), msg_inner_msg.get('state'))
@@ -212,7 +212,7 @@ class KojiRepoChange(BaseMessage):
class RidaModule(BaseMessage):
""" A class that inherits from BaseMessage to provide a message
object for a module event generated by rida
object for a module event generated by module_build_service
:param msg_id: the id of the msg (e.g. 2016-SomeGUID)
:param module_build_id: the id of the module build
:param module_build_state: the state of the module build
@@ -229,7 +229,7 @@ def publish(topic, msg, conf, service):
:param topic: the topic of the message (e.g. module.state.change)
:param msg: the message contents of the message (typically JSON)
:param conf: a Config object from the class in config.py
:param service: the system that is publishing the message (e.g. rida)
:param service: the system that is publishing the message (e.g. module_build_service)
:return:
"""
try:

18
rida/models.py → module_build_service/models.py
View File

@@ -33,8 +33,8 @@ from sqlalchemy import engine_from_config
from sqlalchemy.orm import validates, scoped_session, sessionmaker
import modulemd as _modulemd
from rida import db, log
import rida.messaging
from module_build_service import db, log
import module_build_service.messaging
# Just like koji.BUILD_STATES, except our own codes for modules.
@@ -44,7 +44,7 @@ BUILD_STATES = {
# publish the message
# validate that components are available
# and that you can fetch them.
# if all is good, go to wait: telling ridad to take over.
# if all is good, go to wait: telling module_build_service_daemon to take over.
# if something is bad, go straight to failed.
"init": 0,
# Here, the scheduler picks up tasks in wait.
@@ -147,7 +147,7 @@ class ModuleBuild(RidaBase):
@classmethod
def from_module_event(cls, session, event):
if type(event) == rida.messaging.RidaModule:
if type(event) == module_build_service.messaging.RidaModule:
return session.query(cls).filter(
cls.id == event.module_build_id).first()
else:
@@ -169,8 +169,8 @@ class ModuleBuild(RidaBase):
)
session.add(module)
session.commit()
rida.messaging.publish(
service='rida',
module_build_service.messaging.publish(
service='module_build_service',
topic='module.state.change',
msg=module.json(), # Note the state is "init" here...
conf=conf,
@@ -191,8 +191,8 @@ class ModuleBuild(RidaBase):
self.state_reason = state_reason
log.debug("%r, state %r->%r" % (self, old_state, self.state))
rida.messaging.publish(
service='rida',
module_build_service.messaging.publish(
service='module_build_service',
topic='module.state.change',
msg=self.json(), # Note the state is "init" here...
conf=conf,
@@ -310,7 +310,7 @@ class ComponentBuild(RidaBase):
@classmethod
def from_component_event(cls, session, event):
if type(event) == rida.messaging.KojiBuildChange:
if type(event) == module_build_service.messaging.KojiBuildChange:
return session.query(cls).filter(
cls.task_id == event.build_id).first()
else:

2
rida/pdc.py → module_build_service/pdc.py
View File

@@ -33,7 +33,7 @@ import six
def get_pdc_client_session(config):
"""
:param config: instance of rida.config.Config
:param config: instance of module_build_service.config.Config
:return pdc_client.PDCClient instance
"""
return PDCClient(config.pdc_url, config.pdc_develop, config.pdc_insecure) # hardcoded devel env

0
rida/scheduler/__init__.py → module_build_service/scheduler/__init__.py
View File

0
rida/scheduler/handlers/__init__.py → module_build_service/scheduler/handlers/__init__.py
View File

10
rida/scheduler/handlers/components.py → module_build_service/scheduler/handlers/components.py
View File

@@ -25,12 +25,12 @@
import logging
import rida.builder
import rida.pdc
import module_build_service.builder
import module_build_service.pdc
import koji
from rida import models, log
from module_build_service import models, log
logging.basicConfig(level=logging.DEBUG)
@@ -77,8 +77,8 @@ def _finalize(config, session, msg, state):
# And install the macros.
module_name = parent.name
tag = parent.koji_tag
builder = rida.builder.Builder(parent.owner, module_name, 'koji',
config, tag_name=tag)
builder = module_build_service.builder.Builder(parent.owner, module_name, 'koji',
config, tag_name=tag)
builder.buildroot_connect()
# tag && add to srpm-build group
nvr = "{}-{}-{}".format(msg.build_name, msg.build_version,

22
rida/scheduler/handlers/modules.py → module_build_service/scheduler/handlers/modules.py
View File

@@ -23,10 +23,10 @@
""" Handlers for module change events on the message bus. """
from rida import models, db, log
import rida.builder
import rida.pdc
import rida.utils
from module_build_service import models, db, log
import module_build_service.builder
import module_build_service.pdc
import module_build_service.utils
import koji
@@ -69,7 +69,7 @@ def wait(config, session, msg):
All we do here is request preparation of the buildroot.
The kicking off of individual component builds is handled elsewhere,
in rida.schedulers.handlers.repos.
in module_build_service.schedulers.handlers.repos.
"""
build = models.ModuleBuild.from_module_event(db.session, msg)
log.info("Found build=%r from message" % build)
@@ -84,7 +84,7 @@ def wait(config, session, msg):
tag = None
dependencies = None
pdc_session = rida.pdc.get_pdc_client_session(config)
pdc_session = module_build_service.pdc.get_pdc_client_session(config)
pdc_query = {
'name': module_info['name'],
@@ -92,13 +92,13 @@ def wait(config, session, msg):
'release': module_info['release'],
}
@rida.utils.retry(interval=10, timeout=30, wait_on=ValueError)
@module_build_service.utils.retry(interval=10, timeout=30, wait_on=ValueError)
def _get_deps_and_tag():
log.info("Getting %s deps from pdc" % module_info['name'])
dependencies = rida.pdc.get_module_build_dependencies(
dependencies = module_build_service.pdc.get_module_build_dependencies(
pdc_session, pdc_query, strict=True)
log.info("Getting %s tag from pdc" % module_info['name'])
tag = rida.pdc.get_module_tag(
tag = module_build_service.pdc.get_module_tag(
pdc_session, pdc_query, strict=True)
return dependencies, tag
@@ -118,8 +118,8 @@ def wait(config, session, msg):
log.debug("Assigning koji tag=%s to module build" % tag)
build.koji_tag = tag
builder = rida.builder.Builder(build.owner, build.name, 'koji', config,
tag_name=tag)
builder = module_build_service.builder.Builder(build.owner, build.name, 'koji', config,
tag_name=tag)
build.buildroot_task_id = builder.buildroot_connect()
log.debug("Adding dependencies %s into buildroot for module %s" % (dependencies, module_info))
builder.buildroot_add_repos(dependencies)

14
rida/scheduler/handlers/repos.py → module_build_service/scheduler/handlers/repos.py
View File

@@ -23,11 +23,11 @@
""" Handlers for repo change events on the message bus. """
import rida.builder
import rida.pdc
import module_build_service.builder
import module_build_service.pdc
import logging
import koji
from rida import models, log
from module_build_service import models, log
logging.basicConfig(level=logging.DEBUG)
@@ -43,7 +43,7 @@ def done(config, session, msg):
return
# It is possible that we have already failed.. but our repo is just being
# routinely regenerated. Just ignore that. If rida says the module is
# routinely regenerated. Just ignore that. If module_build_service says the module is
# dead, then the module is dead.
if module_build.state == models.BUILD_STATES['failed']:
log.info("Ignoring repo regen for already failed %r" % module_build)
@@ -75,7 +75,7 @@ def done(config, session, msg):
log.warn("Odd! All components in batch failed for %r." % module_build)
return
builder = rida.builder.Builder(module_build.owner, module_build.name,
builder = module_build_service.builder.Builder(module_build.owner, module_build.name,
'koji', config, tag_name=tag)
builder.buildroot_connect()
@@ -100,8 +100,8 @@ def done(config, session, msg):
if c.state != koji.BUILD_STATES['COMPLETE']
]
if leftover_components:
rida.utils.start_next_build_batch(config,
module_build, session, builder, components=leftover_components)
module_build_service.utils.start_next_build_batch(config,
module_build, session, builder, components=leftover_components)
else:
module_build.transition(config, state=models.BUILD_STATES['done'])
session.commit()

40
rida/scheduler/main.py → module_build_service/scheduler/main.py
View File

@@ -36,15 +36,15 @@ import threading
import time
import six.moves.queue as queue
import rida.config
import rida.messaging
import rida.scheduler.handlers.components
import rida.scheduler.handlers.modules
import rida.scheduler.handlers.repos
import module_build_service.config
import module_build_service.messaging
import module_build_service.scheduler.handlers.components
import module_build_service.scheduler.handlers.modules
import module_build_service.scheduler.handlers.repos
import koji
from rida import conf, models, log
from module_build_service import conf, models, log
class STOP_WORK(object):
@@ -68,7 +68,7 @@ class MessageIngest(threading.Thread):
def run(self):
for msg in rida.messaging.listen(conf):
for msg in module_build_service.messaging.listen(conf):
self.outgoing_work_queue.put(msg)
@@ -83,21 +83,21 @@ class MessageWorker(threading.Thread):
self.NO_OP = NO_OP = lambda config, session, msg: True
self.on_build_change = {
koji.BUILD_STATES["BUILDING"]: NO_OP,
koji.BUILD_STATES["COMPLETE"]: rida.scheduler.handlers.components.complete,
koji.BUILD_STATES["FAILED"]: rida.scheduler.handlers.components.failed,
koji.BUILD_STATES["CANCELED"]: rida.scheduler.handlers.components.canceled,
koji.BUILD_STATES["COMPLETE"]: module_build_service.scheduler.handlers.components.complete,
koji.BUILD_STATES["FAILED"]: module_build_service.scheduler.handlers.components.failed,
koji.BUILD_STATES["CANCELED"]: module_build_service.scheduler.handlers.components.canceled,
koji.BUILD_STATES["DELETED"]: NO_OP,
}
self.on_module_change = {
models.BUILD_STATES["init"]: NO_OP,
models.BUILD_STATES["wait"]: rida.scheduler.handlers.modules.wait,
models.BUILD_STATES["wait"]: module_build_service.scheduler.handlers.modules.wait,
models.BUILD_STATES["build"]: NO_OP,
models.BUILD_STATES["failed"]: NO_OP,
models.BUILD_STATES["done"]: rida.scheduler.handlers.modules.done, # XXX: DIRECT TRANSITION TO READY
models.BUILD_STATES["done"]: module_build_service.scheduler.handlers.modules.done, # XXX: DIRECT TRANSITION TO READY
models.BUILD_STATES["ready"]: NO_OP,
}
# Only one kind of repo change event, though...
self.on_repo_change = rida.scheduler.handlers.repos.done
self.on_repo_change = module_build_service.scheduler.handlers.repos.done
def sanity_check(self):
""" On startup, make sure our implementation is sane. """
@@ -140,11 +140,11 @@ class MessageWorker(threading.Thread):
.format(msg.msg_id, type(msg).__name__))
# Choose a handler for this message
if type(msg) == rida.messaging.KojiBuildChange:
if type(msg) == module_build_service.messaging.KojiBuildChange:
handler = self.on_build_change[msg.build_new_state]
elif type(msg) == rida.messaging.KojiRepoChange:
elif type(msg) == module_build_service.messaging.KojiRepoChange:
handler = self.on_repo_change
elif type(msg) == rida.messaging.RidaModule:
elif type(msg) == module_build_service.messaging.RidaModule:
handler = self.on_module_change[module_build_state_from_msg(msg)]
else:
log.debug("Unhandled message...")
@@ -189,7 +189,7 @@ class Poller(threading.Thread):
if conf.system == "koji":
# we don't do this on behalf of users
koji_session = (
rida.builder.KojiModuleBuilder.get_session(conf, None))
module_build_service.builder.KojiModuleBuilder.get_session(conf, None))
log.info("Querying tasks for statuses:")
res = models.ComponentBuild.query.filter_by(state=koji.BUILD_STATES['BUILDING']).all()
@@ -209,7 +209,7 @@ class Poller(threading.Thread):
log.info(" task %r is in state %r" % (component_build.task_id, task_info['state']))
if task_info['state'] in dead_states:
# Fake a fedmsg message on our internal queue
msg = rida.messaging.KojiBuildChange(
msg = module_build_service.messaging.KojiBuildChange(
msg_id='a faked internal message',
build_id=component_build.task_id,
build_name=component_build.package,
@@ -251,7 +251,7 @@ class Poller(threading.Thread):
'topic': '.module.build.state.change',
'msg': build.json(),
}
rida.scheduler.handlers.modules.wait(conf, session, msg)
module_build_service.scheduler.handlers.modules.wait(conf, session, msg)
def process_open_component_builds(self, session):
log.warning("process_open_component_builds is not yet implemented...")
@@ -261,7 +261,7 @@ class Poller(threading.Thread):
def main():
log.info("Starting ridad.")
log.info("Starting module_build_service_daemon.")
try:
work_queue = queue.Queue()

8
rida/scm.py → module_build_service/scm.py
View File

@@ -35,9 +35,9 @@ import re
import tempfile
import shutil
from rida import log
from rida.errors import Unauthorized, ValidationError
import rida.utils
from module_build_service import log
from module_build_service.errors import Unauthorized, ValidationError
import module_build_service.utils
class SCM(object):
@@ -94,7 +94,7 @@ class SCM(object):
raise ValidationError("Unhandled SCM scheme: %s" % self.scheme)
@staticmethod
@rida.utils.retry(wait_on=RuntimeError)
@module_build_service.utils.retry(wait_on=RuntimeError)
def _run(cmd, chdir=None):
proc = sp.Popen(cmd, stdout=sp.PIPE, stderr=sp.PIPE, cwd=chdir)
stdout, stderr = proc.communicate()

6
rida/utils.py → module_build_service/utils.py
View File

@@ -20,14 +20,14 @@
#
# Written by Ralph Bean <rbean@redhat.com>
# Matt Prahl <mprahl@redhat.com>
""" Utility functions for rida. """
""" Utility functions for module_build_service. """
from flask import request, url_for
from datetime import datetime
import re
import functools
import time
from rida import log, models
from rida.errors import ValidationError
from module_build_service import log, models
from module_build_service.errors import ValidationError
def retry(timeout=120, interval=30, wait_on=Exception):

28
rida/views.py → module_build_service/views.py
View File

@@ -32,15 +32,15 @@ from flask.views import MethodView
import json
import modulemd
import os
import rida.auth
import rida.scm
import module_build_service.auth
import module_build_service.scm
import shutil
import tempfile
import re
from rida import app, conf, db, log
from rida import models
from rida.utils import pagination_metadata, filter_module_builds
from rida.errors import (
from module_build_service import app, conf, db, log
from module_build_service import models
from module_build_service.utils import pagination_metadata, filter_module_builds
from module_build_service.errors import (
ValidationError, Unauthorized, UnprocessableEntity, Conflict, NotFound)
from multiprocessing.dummy import Pool as ThreadPool
@@ -75,8 +75,8 @@ class ModuleBuildAPI(MethodView):
raise NotFound('No such module found.')
def post(self):
username = rida.auth.get_username(request.environ)
rida.auth.assert_is_packager(username, fas_kwargs=dict(
username = module_build_service.auth.get_username(request.environ)
module_build_service.auth.assert_is_packager(username, fas_kwargs=dict(
base_url=conf.fas_url,
username=conf.fas_username,
password=conf.fas_password))
@@ -108,7 +108,7 @@ class ModuleBuildAPI(MethodView):
try:
log.debug('Verifying modulemd')
td = tempfile.mkdtemp()
scm = rida.scm.SCM(url, conf.scmurls)
scm = module_build_service.scm.SCM(url, conf.scmurls)
cod = scm.checkout(td)
cofn = os.path.join(cod, (scm.name + ".yaml"))
@@ -183,7 +183,7 @@ class ModuleBuildAPI(MethodView):
pkg["cache"] = conf.rpms_default_cache + pkgname
if not pkg.get("commit"):
try:
pkg["commit"] = rida.scm.SCM(
pkg["commit"] = module_build_service.scm.SCM(
pkg["repository"]).get_latest()
except Exception as e:
raise UnprocessableEntity(
@@ -201,7 +201,7 @@ class ModuleBuildAPI(MethodView):
# Checks the availability of SCM urls.
pool = ThreadPool(10)
err_msgs = pool.map(lambda data: "Cannot checkout {}".format(data[0])
if not rida.scm.SCM(data[1]).is_available()
if not module_build_service.scm.SCM(data[1]).is_available()
else None, full_urls)
for err_msg in err_msgs:
if err_msg:
@@ -239,11 +239,11 @@ class ModuleBuildAPI(MethodView):
def register_v1_api():
""" Registers version 1 of Rida API. """
module_view = ModuleBuildAPI.as_view('module_builds')
app.add_url_rule('/rida/1/module-builds/', defaults={'id': None},
app.add_url_rule('/module-build-service/1/module-builds/', defaults={'id': None},
view_func=module_view, methods=['GET'])
app.add_url_rule('/rida/1/module-builds/', view_func=module_view,
app.add_url_rule('/module-build-service/1/module-builds/', view_func=module_view,
methods=['POST'])
app.add_url_rule('/rida/1/module-builds/<int:id>', view_func=module_view,
app.add_url_rule('/module-build-service/1/module-builds/<int:id>', view_func=module_view,
methods=['GET'])
register_v1_api()

4
ridad.py → module_build_service_daemon.py
View File

@@ -24,7 +24,7 @@
# Ralph Bean <rbean@redhat.com>
"""The module build orchestrator for Modularity, the builder. """
import rida.scheduler.main
import module_build_service.scheduler.main
if __name__ == '__main__':
rida.scheduler.main.main()
module_build_service.scheduler.main.main()

4
submit-build.sh
View File

@@ -1,7 +1,7 @@
#!/bin/sh
echo "Submmiting a build of modules/testmodule, #020ea37251df5019fde9e7899d2f7d7a987dfbf5"
echo "Using https://localhost:5000/rida/module-builds/"
echo "Using https://localhost:5000/module_build_service/module-builds/"
echo "NOTE: You need to be a Fedora packager for this to work"
echo
curl --cert ~/.fedora.cert -k -H "Content-Type: text/json" --data @submit-build.json https://localhost:5000/rida/1/module-builds/
curl --cert ~/.fedora.cert -k -H "Content-Type: text/json" --data @submit-build.json https://localhost:5000/module-build-service/1/module-builds/
echo

10
tests/__init__.py
View File

@@ -20,8 +20,8 @@
#
# Written by Matt Prahl <mprahl@redhat.com
from datetime import datetime, timedelta
from rida import app, db
from rida.models import ModuleBuild, ComponentBuild
from module_build_service import app, db
from module_build_service.models import ModuleBuild, ComponentBuild
app.config.from_object('config.TestConfiguration')
@@ -65,7 +65,7 @@ def init_data():
component_two_build_one = ComponentBuild()
component_two_build_one.package = 'module-build-macros'
component_two_build_one.scmurl = \
('/tmp/rida-build-macrosWZUPeK/SRPMS/'
('/tmp/module_build_service-build-macrosWZUPeK/SRPMS/'
'module-build-macros-0.1-1.module_nginx_1_2.src.rpm')
component_two_build_one.format = 'rpms'
component_two_build_one.task_id = 12312321 + index
@@ -108,7 +108,7 @@ def init_data():
component_two_build_two = ComponentBuild()
component_two_build_two.package = 'module-build-macros'
component_two_build_two.scmurl = \
('/tmp/rida-build-macrosWZUPeK/SRPMS/'
('/tmp/module_build_service-build-macrosWZUPeK/SRPMS/'
'module-build-macros-0.1-1.module_postgresql_1_2.src.rpm')
component_two_build_two.format = 'rpms'
component_two_build_two.task_id = 47383993 + index
@@ -150,7 +150,7 @@ def init_data():
component_two_build_three = ComponentBuild()
component_two_build_three.package = 'module-build-macros'
component_two_build_three.scmurl = \
('/tmp/rida-build-macrosWZUPeK/SRPMS/'
('/tmp/module_build_service-build-macrosWZUPeK/SRPMS/'
'module-build-macros-0.1-1.module_testmodule_1_2.src.rpm')
component_two_build_three.format = 'rpms'
component_two_build_three.task_id = 47383993 + index

16
tests/test_auth.py
View File

@@ -26,20 +26,20 @@ from nose.tools import raises, eq_
import unittest
import mock
import rida.auth
import rida.errors
import module_build_service.auth
import module_build_service.errors
class TestAuthModule(unittest.TestCase):
@raises(rida.errors.Unauthorized)
@raises(module_build_service.errors.Unauthorized)
def test_get_username_failure(self):
rida.auth.get_username({})
module_build_service.auth.get_username({})
def test_get_username_good(self):
# https://www.youtube.com/watch?v=G-LtddOgUCE
name = "Joey Jo Jo Junior Shabadoo"
environ = {'SSL_CLIENT_CERT_commonName': name}
result = rida.auth.get_username(environ)
result = module_build_service.auth.get_username(environ)
eq_(result, name)
@mock.patch('fedora.client.AccountSystem')
@@ -54,9 +54,9 @@ class TestAuthModule(unittest.TestCase):
}
AccountSystem.return_value = FAS
# This should not raise an exception
rida.auth.assert_is_packager('ralph', dict())
module_build_service.auth.assert_is_packager('ralph', dict())
@raises(rida.errors.Unauthorized)
@raises(module_build_service.errors.Unauthorized)
@mock.patch('fedora.client.AccountSystem')
def test_assert_is_packager_failure(self, AccountSystem):
FAS = mock.MagicMock()
@@ -69,4 +69,4 @@ class TestAuthModule(unittest.TestCase):
}
AccountSystem.return_value = FAS
# This should not raise an exception
rida.auth.assert_is_packager('ralph', dict())
module_build_service.auth.assert_is_packager('ralph', dict())

22
tests/test_messaging/test_messaging_functions.py
View File

@@ -23,12 +23,12 @@ from __future__ import unicode_literals
import unittest
import mock
from datetime import datetime
import rida.messaging
import module_build_service.messaging
class TestUtilFunctions(unittest.TestCase):
rida_msg = msg = {
module_build_service_msg = msg = {
'component_builds': [1, 2],
'id': 1,
'name': 'testmodule',
@@ -69,8 +69,8 @@ class TestUtilFunctions(unittest.TestCase):
}
mock_tail_messages.side_effect = \
lambda: [('fedora-infrastructure', endpoint, topic, msg)]
msg_obj = next(rida.messaging._fedmsg_listen(None))
self.assertEquals(type(msg_obj), rida.messaging.KojiBuildChange)
msg_obj = next(module_build_service.messaging._fedmsg_listen(None))
self.assertEquals(type(msg_obj), module_build_service.messaging.KojiBuildChange)
self.assertEquals(msg_obj.build_id, 2345678)
self.assertEquals(msg_obj.build_new_state, 0)
self.assertEquals(msg_obj.build_name, 'some-package')
@@ -99,24 +99,24 @@ class TestUtilFunctions(unittest.TestCase):
}
mock_tail_messages.side_effect = \
lambda: [('fedora-infrastructure', endpoint, topic, msg)]
msg_obj = next(rida.messaging._fedmsg_listen(None))
self.assertEquals(type(msg_obj), rida.messaging.KojiRepoChange)
msg_obj = next(module_build_service.messaging._fedmsg_listen(None))
self.assertEquals(type(msg_obj), module_build_service.messaging.KojiRepoChange)
self.assertEquals(msg_obj.repo_tag, 'f23-build')
self.assertEquals(msg_obj.msg_id,
'2016-e05415d9-9b35-4f13-8b25-0daddeabfb8c')
@mock.patch('fedmsg.tail_messages')
def test_fedmsg_listen_rida_msg(self, mock_tail_messages):
def test_fedmsg_listen_module_build_service_msg(self, mock_tail_messages):
endpoint = 'tcp://hub.fedoraproject.org:9940'
topic = 'org.fedoraproject.prod.rida.module.state.change'
topic = 'org.fedoraproject.prod.module_build_service.module.state.change'
msg = {
'msg_id': '2016-e05415d9-9b35-4f13-8b25-0daddeabfb8c',
'topic': 'org.fedoraproject.prod.rida.module.state.change',
'msg': self.rida_msg
'topic': 'org.fedoraproject.prod.module_build_service.module.state.change',
'msg': self.module_build_service_msg
}
mock_tail_messages.side_effect = \
lambda: [('fedora-infrastructure', endpoint, topic, msg)]
msg_obj = next(rida.messaging._fedmsg_listen(None))
msg_obj = next(module_build_service.messaging._fedmsg_listen(None))
self.assertEquals(msg_obj.module_build_id, msg['msg']['id'])
self.assertEquals(msg_obj.module_build_state, msg['msg']['state'])
self.assertEquals(msg_obj.msg_id,

16
tests/test_scheduler/test_module_wait.py
View File

@@ -22,8 +22,8 @@
import unittest
import mock
import rida.messaging
import rida.scheduler.handlers.modules
import module_build_service.messaging
import module_build_service.scheduler.handlers.modules
class TestModuleWait(unittest.TestCase):
@@ -31,11 +31,11 @@ class TestModuleWait(unittest.TestCase):
def setUp(self):
self.config = mock.Mock()
self.session = mock.Mock()
self.fn = rida.scheduler.handlers.modules.wait
self.fn = module_build_service.scheduler.handlers.modules.wait
@mock.patch('rida.builder.KojiModuleBuilder')
@mock.patch('rida.models.ModuleBuild.from_module_event')
@mock.patch('rida.pdc')
@mock.patch('module_build_service.builder.KojiModuleBuilder')
@mock.patch('module_build_service.models.ModuleBuild.from_module_event')
@mock.patch('module_build_service.pdc')
def test_init_basic(self, pdc, from_module_event, KojiModuleBuilder):
builder = mock.Mock()
builder.get_disttag_srpm.return_value = 'some srpm disttag'
@@ -50,6 +50,6 @@ class TestModuleWait(unittest.TestCase):
from_module_event.return_value = mocked_module_build
msg = rida.messaging.RidaModule(msg_id=None, module_build_id=1,
module_build_state='some state')
msg = module_build_service.messaging.RidaModule(msg_id=None, module_build_id=1,
module_build_state='some state')
self.fn(config=self.config, session=self.session, msg=msg)

39
tests/test_scheduler/test_repo_done.py
View File

@@ -23,9 +23,9 @@
import unittest
import mock
import rida.messaging
import rida.scheduler.handlers.repos
import rida.models
import module_build_service.messaging
import module_build_service.scheduler.handlers.repos
import module_build_service.models
class TestRepoDone(unittest.TestCase):
@@ -37,23 +37,23 @@ class TestRepoDone(unittest.TestCase):
self.session = mock.Mock()
self.fn = rida.scheduler.handlers.repos.done
self.fn = module_build_service.scheduler.handlers.repos.done
@mock.patch('rida.models.ModuleBuild.from_repo_done_event')
@mock.patch('module_build_service.models.ModuleBuild.from_repo_done_event')
def test_no_match(self, from_repo_done_event):
""" Test that when a repo msg hits us and we have no match,
that we do nothing gracefully.
"""
from_repo_done_event.return_value = None
msg = rida.messaging.KojiRepoChange(
msg = module_build_service.messaging.KojiRepoChange(
'no matches for this...', '2016-some-guid')
self.fn(config=self.config, session=self.session, msg=msg)
@mock.patch('rida.builder.KojiModuleBuilder.buildroot_ready')
@mock.patch('rida.builder.KojiModuleBuilder.get_session')
@mock.patch('rida.builder.KojiModuleBuilder.build')
@mock.patch('rida.builder.KojiModuleBuilder.buildroot_connect')
@mock.patch('rida.models.ModuleBuild.from_repo_done_event')
@mock.patch('module_build_service.builder.KojiModuleBuilder.buildroot_ready')
@mock.patch('module_build_service.builder.KojiModuleBuilder.get_session')
@mock.patch('module_build_service.builder.KojiModuleBuilder.build')
@mock.patch('module_build_service.builder.KojiModuleBuilder.buildroot_connect')
@mock.patch('module_build_service.models.ModuleBuild.from_repo_done_event')
def test_a_single_match(self, from_repo_done_event, connect, build_fn, config, ready):
""" Test that when a repo msg hits us and we have a single match.
"""
@@ -75,17 +75,16 @@ class TestRepoDone(unittest.TestCase):
ready.return_value = True
msg = rida.messaging.KojiRepoChange(
msg = module_build_service.messaging.KojiRepoChange(
'no matches for this...', '2016-some-guid')
self.fn(config=self.config, session=self.session, msg=msg)
build_fn.assert_called_once_with(artifact_name='foo', source='full_scm_url')
@mock.patch('rida.builder.KojiModuleBuilder.buildroot_ready')
@mock.patch('rida.builder.KojiModuleBuilder.get_session')
@mock.patch('rida.builder.KojiModuleBuilder.build')
@mock.patch('rida.builder.KojiModuleBuilder.buildroot_connect')
@mock.patch('rida.models.ModuleBuild.from_repo_done_event')
@mock.patch('module_build_service.builder.KojiModuleBuilder.buildroot_ready')
@mock.patch('module_build_service.builder.KojiModuleBuilder.get_session')
@mock.patch('module_build_service.builder.KojiModuleBuilder.build')
@mock.patch('module_build_service.builder.KojiModuleBuilder.buildroot_connect')
@mock.patch('module_build_service.models.ModuleBuild.from_repo_done_event')
def test_a_single_match_build_fail(self, from_repo_done_event, connect, build_fn, config, ready):
""" Test that when a KojiModuleBuilder.build fails, the build is
marked as failed with proper state_reason.
@@ -109,11 +108,11 @@ class TestRepoDone(unittest.TestCase):
ready.return_value = True
msg = rida.messaging.KojiRepoChange(
msg = module_build_service.messaging.KojiRepoChange(
'no matches for this...', '2016-some-guid')
self.fn(config=self.config, session=self.session, msg=msg)
build_fn.assert_called_once_with(artifact_name='foo', source='full_scm_url')
module_build.transition.assert_called_once_with(self.config,
rida.models.BUILD_STATES["failed"],
module_build_service.models.BUILD_STATES["failed"],
'Failed to submit artifact foo to Koji')
self.assertEquals(unbuilt_component_build.state_reason, "Failed to submit to Koji")

86
tests/test_views/test_views.py
View File

@@ -37,7 +37,7 @@ class TestViews(unittest.TestCase):
init_data()
def test_query_build(self):
rv = self.client.get('/rida/1/module-builds/1')
rv = self.client.get('/module-build-service/1/module-builds/1')
data = json.loads(rv.data)
self.assertEquals(data['id'], 1)
self.assertEquals(data['name'], 'nginx')
@@ -53,29 +53,29 @@ class TestViews(unittest.TestCase):
self.assertEquals(data['time_submitted'], '2016-09-03T11:23:20Z')
def test_pagination_metadata(self):
rv = self.client.get('/rida/1/module-builds/?per_page=8&page=2')
rv = self.client.get('/module-build-service/1/module-builds/?per_page=8&page=2')
meta_data = json.loads(rv.data)['meta']
self.assertTrue(
'rida/1/module-builds/?per_page=8&page=1' in meta_data['prev'])
'module-build-service/1/module-builds/?per_page=8&page=1' in meta_data['prev'])
self.assertTrue(
'rida/1/module-builds/?per_page=8&page=3' in meta_data['next'])
'module-build-service/1/module-builds/?per_page=8&page=3' in meta_data['next'])
self.assertTrue(
'rida/1/module-builds/?per_page=8&page=4' in meta_data['last'])
'module-build-service/1/module-builds/?per_page=8&page=4' in meta_data['last'])
self.assertTrue(
'rida/1/module-builds/?per_page=8&page=1' in meta_data['first'])
'module-build-service/1/module-builds/?per_page=8&page=1' in meta_data['first'])
self.assertEquals(meta_data['total'], 30)
self.assertEquals(meta_data['per_page'], 8)
self.assertEquals(meta_data['pages'], 4)
self.assertEquals(meta_data['page'], 2)
def test_query_builds(self):
rv = self.client.get('/rida/1/module-builds/?per_page=2')
rv = self.client.get('/module-build-service/1/module-builds/?per_page=2')
items = json.loads(rv.data)['items']
self.assertEquals(items,
[{u'state': 3, u'id': 1}, {u'state': 3, u'id': 2}])
def test_query_builds_verbose(self):
rv = self.client.get('/rida/1/module-builds/?per_page=2&verbose=True')
rv = self.client.get('/module-build-service/1/module-builds/?per_page=2&verbose=True')
item = json.loads(rv.data)['items'][1]
self.assertEquals(item['id'], 2)
self.assertEquals(item['name'], 'postgressql')
@@ -91,76 +91,76 @@ class TestViews(unittest.TestCase):
self.assertEquals(item['time_submitted'], '2016-09-03T12:25:33Z')
def test_query_builds_filter_name(self):
rv = self.client.get('/rida/1/module-builds/?name=nginx')
rv = self.client.get('/module-build-service/1/module-builds/?name=nginx')
data = json.loads(rv.data)
self.assertEquals(data['meta']['total'], 10)
def test_query_builds_filter_completed_before(self):
rv = self.client.get(
'/rida/1/module-builds/?completed_before=2016-09-03T11:30:00Z')
'/module-build-service/1/module-builds/?completed_before=2016-09-03T11:30:00Z')
data = json.loads(rv.data)
self.assertEquals(data['meta']['total'], 2)
def test_query_builds_filter_completed_after(self):
rv = self.client.get(
'/rida/1/module-builds/?completed_after=2016-09-03T12:25:00Z')
'/module-build-service/1/module-builds/?completed_after=2016-09-03T12:25:00Z')
data = json.loads(rv.data)
self.assertEquals(data['meta']['total'], 8)
def test_query_builds_filter_submitted_before(self):
rv = self.client.get(
'/rida/1/module-builds/?submitted_before=2016-09-03T12:25:00Z')
'/module-build-service/1/module-builds/?submitted_before=2016-09-03T12:25:00Z')
data = json.loads(rv.data)
self.assertEquals(data['meta']['total'], 7)
def test_query_builds_filter_submitted_after(self):
rv = self.client.get(
'/rida/1/module-builds/?submitted_after=2016-09-03T12:25:00Z')
'/module-build-service/1/module-builds/?submitted_after=2016-09-03T12:25:00Z')
data = json.loads(rv.data)
self.assertEquals(data['meta']['total'], 23)
def test_query_builds_filter_modified_before(self):
rv = self.client.get(
'/rida/1/module-builds/?modified_before=2016-09-03T12:25:00Z')
'/module-build-service/1/module-builds/?modified_before=2016-09-03T12:25:00Z')
data = json.loads(rv.data)
self.assertEquals(data['meta']['total'], 6)
def test_query_builds_filter_modified_after(self):
rv = self.client.get(
'/rida/1/module-builds/?modified_after=2016-09-03T12:25:00Z')
'/module-build-service/1/module-builds/?modified_after=2016-09-03T12:25:00Z')
data = json.loads(rv.data)
self.assertEquals(data['meta']['total'], 24)
def test_query_builds_filter_owner(self):
rv = self.client.get(
'/rida/1/module-builds/?owner=Moe%20Szyslak')
'/module-build-service/1/module-builds/?owner=Moe%20Szyslak')
data = json.loads(rv.data)
self.assertEquals(data['meta']['total'], 10)
def test_query_builds_filter_state(self):
rv = self.client.get(
'/rida/1/module-builds/?state=3')
'/module-build-service/1/module-builds/?state=3')
data = json.loads(rv.data)
self.assertEquals(data['meta']['total'], 20)
def test_query_builds_two_filters(self):
rv = self.client.get('/rida/1/module-builds/?owner=Moe%20Szyslak'
rv = self.client.get('/module-build-service/1/module-builds/?owner=Moe%20Szyslak'
'&modified_after=2016-09-03T12:25:00Z')
data = json.loads(rv.data)
self.assertEquals(data['meta']['total'], 4)
def test_query_builds_filter_invalid_date(self):
rv = self.client.get(
'/rida/1/module-builds/?modified_after=2016-09-03T12:25:00-05:00')
'/module-build-service/1/module-builds/?modified_after=2016-09-03T12:25:00-05:00')
data = json.loads(rv.data)
self.assertEquals(data['error'], 'Bad Request')
self.assertEquals(data['message'], 'An invalid Zulu ISO 8601 timestamp'
' was provided for the \"modified_after\" parameter')
self.assertEquals(data['status'], 400)
@patch('rida.auth.get_username', return_value='Homer J. Simpson')
@patch('rida.auth.assert_is_packager')
@patch('rida.scm.SCM')
@patch('module_build_service.auth.get_username', return_value='Homer J. Simpson')
@patch('module_build_service.auth.assert_is_packager')
@patch('module_build_service.scm.SCM')
def test_submit_build(self, mocked_scm, mocked_assert_is_packager,
mocked_get_username):
def mocked_scm_checkout(temp_dir):
@@ -174,7 +174,7 @@ class TestViews(unittest.TestCase):
mocked_scm.return_value.checkout = mocked_scm_checkout
mocked_scm.return_value.name = 'fakemodule'
rv = self.client.post('/rida/1/module-builds/', data=json.dumps(
rv = self.client.post('/module-build-service/1/module-builds/', data=json.dumps(
{'scmurl': 'git://pkgs.stg.fedoraproject.org/modules/'
'testmodule.git?#68932c90de214d9d13feefbd35246a81b6cb8d49'}))
data = json.loads(rv.data)
@@ -195,7 +195,7 @@ class TestViews(unittest.TestCase):
self.assertEquals(data['state_name'], 'wait')
def test_submit_build_cert_error(self):
rv = self.client.post('/rida/1/module-builds/', data=json.dumps(
rv = self.client.post('/module-build-service/1/module-builds/', data=json.dumps(
{'scmurl': 'git://pkgs.stg.fedoraproject.org/modules/'
'testmodule.git?#48932b90de214d9d13feefbd35246a81b6cb8d49'}))
data = json.loads(rv.data)
@@ -206,11 +206,11 @@ class TestViews(unittest.TestCase):
self.assertEquals(data['status'], 401)
self.assertEquals(data['error'], 'Unauthorized')
@patch('rida.auth.get_username', return_value='Homer J. Simpson')
@patch('rida.auth.assert_is_packager')
@patch('module_build_service.auth.get_username', return_value='Homer J. Simpson')
@patch('module_build_service.auth.assert_is_packager')
def test_submit_build_scm_url_error(self, mocked_assert_is_packager,
mocked_get_username):
rv = self.client.post('/rida/1/module-builds/', data=json.dumps(
rv = self.client.post('/module-build-service/1/module-builds/', data=json.dumps(
{'scmurl': 'git://badurl.com'}))
data = json.loads(rv.data)
self.assertEquals(data['message'], 'The submitted scmurl '
@@ -218,12 +218,12 @@ class TestViews(unittest.TestCase):
self.assertEquals(data['status'], 401)
self.assertEquals(data['error'], 'Unauthorized')
@patch('rida.auth.get_username', return_value='Homer J. Simpson')
@patch('rida.auth.assert_is_packager')
@patch('module_build_service.auth.get_username', return_value='Homer J. Simpson')
@patch('module_build_service.auth.assert_is_packager')
def test_submit_build_scm_url_without_hash(self,
mocked_assert_is_packager,
mocked_get_username):
rv = self.client.post('/rida/1/module-builds/', data=json.dumps(
rv = self.client.post('/module-build-service/1/module-builds/', data=json.dumps(
{'scmurl': 'git://pkgs.stg.fedoraproject.org/modules/'
'testmodule.git'}))
data = json.loads(rv.data)
@@ -233,13 +233,13 @@ class TestViews(unittest.TestCase):
self.assertEquals(data['status'], 401)
self.assertEquals(data['error'], 'Unauthorized')
@patch('rida.auth.get_username', return_value='Homer J. Simpson')
@patch('rida.auth.assert_is_packager')
@patch('rida.scm.SCM')
@patch('module_build_service.auth.get_username', return_value='Homer J. Simpson')
@patch('module_build_service.auth.assert_is_packager')
@patch('module_build_service.scm.SCM')
def test_submit_build_bad_modulemd(self, mocked_scm,
mocked_assert_is_packager,
mocked_get_username):
rv = self.client.post('/rida/1/module-builds/', data=json.dumps(
rv = self.client.post('/module-build-service/1/module-builds/', data=json.dumps(
{'scmurl': 'git://badurl.com'}))
def mocked_scm_checkout(temp_dir):
scm_dir = path.join(temp_dir, 'fakemodule')
@@ -253,7 +253,7 @@ class TestViews(unittest.TestCase):
mocked_scm.return_value.checkout = mocked_scm_checkout
mocked_scm.return_value.name = 'fakemodule'
rv = self.client.post('/rida/1/module-builds/', data=json.dumps(
rv = self.client.post('/module-build-service/1/module-builds/', data=json.dumps(
{'scmurl': 'git://pkgs.stg.fedoraproject.org/modules/'
'testmodule.git?#68932c90de214d9d13feefbd35246a81b6cb8d49'}))
data = json.loads(rv.data)
@@ -262,9 +262,9 @@ class TestViews(unittest.TestCase):
self.assertEquals(data['status'], 422)
self.assertEquals(data['error'], 'Unprocessable Entity')
@patch('rida.auth.get_username', return_value='Homer J. Simpson')
@patch('rida.auth.assert_is_packager')
@patch('rida.scm.SCM')
@patch('module_build_service.auth.get_username', return_value='Homer J. Simpson')
@patch('module_build_service.auth.assert_is_packager')
@patch('module_build_service.scm.SCM')
def test_submit_build_scm_parallalization(self, mocked_scm,
mocked_assert_is_packager, mocked_get_username):
def mocked_scm_checkout(temp_dir):
@@ -284,7 +284,7 @@ class TestViews(unittest.TestCase):
mocked_scm.return_value.checkout = mocked_scm_checkout
mocked_scm.return_value.name = 'base-runtime'
mocked_scm.return_value.is_available = mocked_scm_is_available
rv = self.client.post('/rida/1/module-builds/', data=json.dumps(
rv = self.client.post('/module-build-service/1/module-builds/', data=json.dumps(
{'scmurl': 'git://pkgs.stg.fedoraproject.org/modules/'
'testmodule.git?#68932c90de214d9d13feefbd35246a81b6cb8d49'}))
data = json.loads(rv.data)
@@ -307,9 +307,9 @@ class TestViews(unittest.TestCase):
# max to complete.
self.assertTrue(time.time() - start < 3)
@patch('rida.auth.get_username', return_value='Homer J. Simpson')
@patch('rida.auth.assert_is_packager')
@patch('rida.scm.SCM')
@patch('module_build_service.auth.get_username', return_value='Homer J. Simpson')
@patch('module_build_service.auth.assert_is_packager')
@patch('module_build_service.scm.SCM')
def test_submit_build_scm_non_available(self, mocked_scm,
mocked_assert_is_packager, mocked_get_username):
def mocked_scm_checkout(temp_dir):
@@ -327,7 +327,7 @@ class TestViews(unittest.TestCase):
mocked_scm.return_value.checkout = mocked_scm_checkout
mocked_scm.return_value.name = 'base-runtime'
mocked_scm.return_value.is_available = mocked_scm_is_available
rv = self.client.post('/rida/1/module-builds/', data=json.dumps(
rv = self.client.post('/module-build-service/1/module-builds/', data=json.dumps(
{'scmurl': 'git://pkgs.stg.fedoraproject.org/modules/'
'testmodule.git?#68932c90de214d9d13feefbd35246a81b6cb8d49'}))
data = json.loads(rv.data)