WarlockFish/fedora-infra_ansible
1
0
Fork 0
mirror of https://pagure.io/fedora-infra/ansible.git synced 2026-04-25 02:50:15 +08:00
Code Issues Packages Projects Releases Wiki Activity
31,813 Commits 9 Branches 0 Tags
b100615e6a1bc66767f40bcdac499fc71fc6bdde
Commit Graph

98 Commits

Author SHA1 Message Date
Kevin Fenzi
ab7d661781 koji hub: re-enable gc jobs, disable session cleanup on db-koji01 
Signed-off-by: Kevin Fenzi <kevin@scrye.com>
 2020-06-11 12:08:59 -07:00
Kevin Fenzi
2fe67cd8c1 postgresql-server: fix indent 
Signed-off-by: Kevin Fenzi <kevin@scrye.com>
 2020-06-10 11:55:55 -07:00
Stephen Smoogen
7ed926aafc backup-database needs to be installed on all postgres boxes 2020-06-09 15:04:43 -04:00
Stephen Smoogen
b0c389b2dc make backups on all systems 2020-06-09 15:01:30 -04:00
Stephen Smoogen
e76b85ec2d and this will unfix my fix by fixing the other fix for postgres 2020-06-09 14:51:04 -04:00
Stephen Smoogen
422265dc0b RHEL-8 postgres boxes needed pxz and other items spelled out 2020-06-09 14:47:38 -04:00
Stephen Smoogen
3015b75976 remove db-koji02 as it is now db-koji01 2020-06-09 14:29:28 -04:00
Pierre-Yves Chibon
6710db8d0d Make some more of the postgresql configuration key adjustable 
Basically, if the variables are defined in the host, use them, otherwise
use the current values.

Signed-off-by: Pierre-Yves Chibon <pingou@pingoured.fr>
 2020-05-22 09:08:00 +00:00
Kevin Fenzi
b9eed95b04 db-fas01.iad2: move back to rhel7, fas is not happy with new postgres 
Signed-off-by: Kevin Fenzi <kevin@scrye.com>
 2020-05-20 14:40:29 -07:00
Kevin Fenzi
c034afc53c postgresql server: fix indent 
Signed-off-by: Kevin Fenzi <kevin@scrye.com>
 2020-05-19 12:20:58 -07:00
Kevin Fenzi
e918a3b08f postgresql / server: enable the postgresql 12 module on rhel8 hosts 
Signed-off-by: Kevin Fenzi <kevin@scrye.com>
 2020-05-19 12:16:45 -07:00
Pierre-Yves Chibon
13ad0cea66 postgresql_server: do not install our current postgresql.conf on RHEL8 
Our postgresql.conf is from postgresql 9.2 while RHEL8 ships 10.x which
leads to postgresql no longer wanting to start (as seen on pagure-stg01).

Signed-off-by: Pierre-Yves Chibon <pingou@pingoured.fr>
 2020-04-24 21:34:29 +02:00
Pierre-Yves Chibon
e7673a2538 postgresql_server: adjust the name of the packages for RHEL8 
Signed-off-by: Pierre-Yves Chibon <pingou@pingoured.fr>
 2020-04-24 21:34:29 +02:00
Pierre-Yves Chibon
56cf212d1f postgresql_server: there is no python2-psycopg on RHEL8, so use python3 
Signed-off-by: Pierre-Yves Chibon <pingou@pingoured.fr>
 2020-04-24 21:34:29 +02:00
Kevin Fenzi
7242a1a583 db-koji02: actually deploy the right backup script 
Signed-off-by: Kevin Fenzi <kevin@scrye.com>
 2020-04-24 21:34:28 +02:00
Kevin Fenzi
bd50c6e4bc db-koji02: make sure backup uses all cpus 
Signed-off-by: Kevin Fenzi <kevin@scrye.com>
 2020-04-24 21:34:28 +02:00
Kevin Fenzi
fd6a78e876 postgresql_server: increase autovacuum_freeze_max_age to avoid autovaccum locking the buildroot_listing table (for a while more) 
Signed-off-by: Kevin Fenzi <kevin@scrye.com>
 2020-04-24 21:34:21 +02:00
Kevin Fenzi
e93d539e29 postgresql_server: also set this on db-koji02 ad they have to match 
Signed-off-by: Kevin Fenzi <kevin@scrye.com>
 2020-04-24 21:34:20 +02:00
Kevin Fenzi
b4f82f8e38 postgresql_server: adjust connections only for db-koji01, leave pagure01 and others back at 750 
Signed-off-by: Kevin Fenzi <kevin@scrye.com>
 2020-04-24 21:34:20 +02:00
Kevin Fenzi
fa46e8bc0e koji_hub: switch to event workers in apache and bump limits way up. 
By default apache uses prefork and a limit of 250. It's possible that this limit was
the thing causing us issues over the last week. This moves to the event mpm and ups limits
a lot. It also needs to up limits on db connections or the increased workers will just
cause the db server to overload.

With this setup, builders are no longer dropping out, but it's not clear if it's solved
all the issues we have been seeing.

Signed-off-by: Kevin Fenzi <kevin@scrye.com>
 2020-04-24 21:34:20 +02:00
Kevin Fenzi
3641c508fd postgresql defaults: move default to defaults. 
Signed-off-by: Kevin Fenzi <kevin@scrye.com>
 2020-04-24 21:34:16 +02:00
Tim Flink
48ef2735d3 postgresql_server: add conditional for small postgresql instances 
The recent changes to postgresql_server increased the required memory past
what one of my VMs had. I've added a conditional in postgresql.conf to put
some memory settings back where they used to be (controlled by
small_postgres_instance, default is false) and created a default to not use
the small_postgres_instance settings unless specified
 2020-04-24 21:34:16 +02:00
Kevin Fenzi
993fd9f64e postgresql_server / db-koji02: set max_standby_streaming_delay to -1 ( ∞ ) 
Turns out I had set this on the master (db-koji01) which is ignored.
We need to set it on the standby. With a value of -1, the standby will wait
for conflicting transactions/locks to complete however long it takes.
If this doesn't work to get us a good backup on db-koji02, no harm and we
can try something else.

Signed-off-by: Kevin Fenzi <kevin@scrye.com>
 2020-04-24 21:34:14 +02:00
Kevin Fenzi
05e5b4fe79 postgresql: increase the max_standby_streaming_delay to 90s 
It's hoped that this would allow us to get more reliable backups on the hot spare.

Signed-off-by: Kevin Fenzi <kevin@scrye.com>
 2020-04-24 21:34:13 +02:00
Kevin Fenzi
955db508b1 postgresql_server: also set hot_standby_feedback or the master might delete rows the standby needs during a pg_dump 
Signed-off-by: Kevin Fenzi <kevin@scrye.com>
 2020-04-24 21:34:13 +02:00
Kevin Fenzi
d2fa094677 postgresql_server: add missing ' to end of connection statement 
Signed-off-by: Kevin Fenzi <kevin@scrye.com>
 2020-04-24 21:34:13 +02:00
Kevin Fenzi
0c895651fa postgresql / kojidb standby: try connecting 02 to 01, not 02 to itself. 
Signed-off-by: Kevin Fenzi <kevin@scrye.com>
 2020-04-24 21:34:13 +02:00
Kevin Fenzi
0a5cb2c51c postgresql_server: Fix spelling mistake 
Signed-off-by: Kevin Fenzi <kevin@scrye.com>
 2020-04-24 21:34:13 +02:00
Kevin Fenzi
1613c724ca postgresql-server: sadly, "all" doesn't match replication, that has to be seperate. 
Signed-off-by: Kevin Fenzi <kevin@scrye.com>
 2020-04-24 21:34:13 +02:00
Kevin Fenzi
9747e55c72 db-koji02: create vm and config files for hot standby kojidb. 
Signed-off-by: Kevin Fenzi <kevin@scrye.com>
 2020-04-24 21:34:12 +02:00
Kevin Fenzi
603fe5e8e9 postgresql / db-koji: Move instead to logging just the slow stuff. 
Signed-off-by: Kevin Fenzi <kevin@scrye.com>
 2020-04-24 21:34:12 +02:00
Kevin Fenzi
b5f8425273 postgresql / db-koji01: log queries 
I'd like to log queries on db-koji01 for a short time to try and see whats causing us such pain.
After we have collected a bunch of queries we can revert this until we sort out what needs
to be changed. We may also change this log all to log just slow queries (per smooge's suggestion).
Hopefully this will get us the info we need to track this down.

Signed-off-by: Kevin Fenzi <kevin@scrye.com>
 2020-04-24 21:34:12 +02:00
Kevin Fenzi
66c7df9a53 postgresql / db-koji01: more tweaks 
Signed-off-by: Kevin Fenzi <kevin@scrye.com>
 2020-04-24 21:34:12 +02:00
Kevin Fenzi
184f4c93bc postgresql / db_koji01: adjust stuff to hopefully give us good backups. 
Signed-off-by: Kevin Fenzi <kevin@scrye.com>
 2020-04-24 21:34:12 +02:00
Kevin Fenzi
6052756805 postgresql_server / db-koji01: Adjust a bunch more for performance. 
The current settings cause database dumps to drive the load way up
and make the entire application slow, so we need to adjust.
Using pgtune, these values might well be better.

shared_buffers + effective_cache_size should = total memory.
random_page_cost should be lowered a bunch since we are on ssds there.
1.1 is only slightly more than 1.0 for sequential.
effective_io_concurrency should also be raised a bunch for ssds.
a few other values should be higher based on memory.

Signed-off-by: Kevin Fenzi <kevin@scrye.com>
 2020-04-24 21:34:12 +02:00
Kevin Fenzi
81fb4582e7 ansible: change when conditions to use == instead of is when checking strings. 
Signed-off-by: Kevin Fenzi <kevin@scrye.com>
 2020-04-24 21:34:10 +02:00
Karsten Hopp
c9ed62ac32 update ansible_distribution_major_version conditionals 
Signed-off-by: Karsten Hopp <karsten@redhat.com>
 2020-04-24 21:34:10 +02:00
Kevin Fenzi
8f025109bf Revert "postgresql_server: drop this cleanup script to see if the problem still even occurs." 
This reverts commit f68a9eb6dbc2003ed6136ba71e30fcb1c3180608.
 2020-04-24 21:34:07 +02:00
Kevin Fenzi
7ef0ae55e9 postgresql_server: drop this cleanup script to see if the problem still even occurs. 
Signed-off-by: Kevin Fenzi <kevin@scrye.com>
 2020-04-24 21:34:07 +02:00
Karsten Hopp
65fc4a4ecc postgresql_server: dnf -> package, with_item changes 2019-06-11 17:15:27 +02:00
Kevin Fenzi
c96327ccfe anitya/release-monitoring: db backup cron has to be mode 644, not 755. 
https://pagure.io/fedora-infrastructure/issue/7479

Signed-off-by: Kevin Fenzi <kevin@scrye.com>
 2019-01-07 18:27:03 +00:00
Kevin Fenzi
8dcc3712af Switch anitya backups to use --exclude-table-data rather than excluding entire tables 
This allows people to use the db dump without having to manually create the missing tables.
 2018-10-10 16:19:19 +00:00
Kevin Fenzi
f0c596c427 Try and make a public anitya dump with the right tables. 2018-10-08 17:41:07 +00:00
Kevin Fenzi
84bd8c1c4a Also install pxz on fedora 2018-06-30 04:04:17 +00:00
Stephen Smoogen
1719435ec7 try to make pxz a little less likely to starve itselv 2018-06-25 16:26:59 +00:00
Kevin Fenzi
069bfc9a41 only keep the last backup on hosts, it iss copied to backup server anyhow 2018-06-07 23:38:46 +00:00
Pierre-Yves Chibon
4de44ac0fc Fix query in the datanommer SAR script 
Signed-off-by: Pierre-Yves Chibon <pingou@pingoured.fr>
 2018-05-22 10:21:42 +02:00
Patrick Uiterwijk
8797997ab2 Get rid of temporary files alltogether 
Signed-off-by: Patrick Uiterwijk <puiterwijk@redhat.com>
 2018-05-18 13:13:47 +02:00
Patrick Uiterwijk
2fb0b14230 Do not keep full file in memory 
Signed-off-by: Patrick Uiterwijk <puiterwijk@redhat.com>
 2018-05-18 13:06:04 +02:00
Patrick Uiterwijk
b6f06b4a27 Get rid of postgres warning 
Signed-off-by: Patrick Uiterwijk <puiterwijk@redhat.com>
 2018-05-18 13:03:34 +02:00