Go back to using a file-backed SQLite database for tests

Using a memory database causes tests/test_build to intermittently
fail, because using the same pysqlite3 connection object from multiple
threads - as was done so that the threads shared the same memory database
- is not, in the end, thread safe. One thread will stomp on the transaction
state of other threads, resulting in errors from starting a new transaction
when another is already in progress, or trying to commit a transaction
that is not in progress.

To avoid a significant speed penalty, the session-scope fixture sets up
a database in the pytest temporary directory, which will typically be on
tmpfs. Time to complete all tests:

 memory backend:               38 seconds
 file on tmpfs:                40 seconds
 file on nvme ssd with btrfs: 137 seconds

MBSSQLAlchemy, which attempted to make the memory backend work, is removed.

Session hooks are installed on the Session class rather than on the
scoped_session instance - this works better when we're changing from
one database to another at test setup time.
This commit is contained in:
Owen W. Taylor
2021-01-22 14:41:05 -05:00
parent 3a633967ec
commit a96774a1fd
5 changed files with 34 additions and 28 deletions

View File

@@ -24,7 +24,6 @@ import pkg_resources
from flask import Flask, has_app_context, url_for
from flask_sqlalchemy import SQLAlchemy
from sqlalchemy.pool import StaticPool
from module_build_service.common.config import config_section
from module_build_service.web.proxy import ReverseProxy
@@ -40,27 +39,7 @@ app.wsgi_app = ReverseProxy(app.wsgi_app)
app.config.from_object(config_section)
class MBSSQLAlchemy(SQLAlchemy):
"""
Inherits from SQLAlchemy and if SQLite in-memory database is used,
sets the driver options so multiple threads can share the same database.
This is used *only* during tests to make them faster.
"""
def apply_driver_hacks(self, app, info, options):
if info.drivername == "sqlite" and info.database in (None, "", ":memory:"):
options["poolclass"] = StaticPool
options["connect_args"] = {"check_same_thread": False}
try:
del options["pool_size"]
except KeyError:
pass
super(MBSSQLAlchemy, self).apply_driver_hacks(app, info, options)
db = MBSSQLAlchemy(app)
db = SQLAlchemy(app)
def get_url_for(*args, **kwargs):