mirror of
https://pagure.io/fedora-infra/ansible.git
synced 2026-04-24 10:31:56 +08:00
83948e50e3
RCA of issue sent on IRC: It's a very interresting edge case and related to my previous diag. In short: both the pagure main app and pagure docs app were using the same process pool (WSGIProcessDaemon). As soon as they would both be loaded in the same thread, they would both load the FFI (C wrapper) code, and only the latest process to load it would still have valid type references, the other would start sending wrong references, which causes it to error out (correctly), because it doesn't know the types it got. So basically, the fix I just applied is put pagure docs into its own WSGI daemon process, that keeps them nicely separated. the reason that this didn't hit in staging and why it also worked *sometimes* in production is that it would only crash if: 1. both pagure main app and docs app were loaded in the thread that's used for the current request 2. pagure docs app was loaded last in the current thread, overriding the types for pagure main app, and 3. we have 4 processes with 4 threads each, so each request gets into one of 16 threads, making the staging not likely to hit the previous two conditions, but prod has so many requests it's likely to hit 1 and 2
== ansible repository/structure ==
files - files and templates for use in playbooks/tasks
- subdirs for specific tasks/dirs highly recommended
inventory - where the inventory and additional vars is stored
- All files in this directory in ini format
- added together for total inventory
group_vars:
- per group variables set here in a file per group
host_vars:
- per host variables set here in a file per host
library - library of custom local ansible modules
playbooks - collections of plays we want to run on systems
groups: groups of hosts configured from one playbook.
hosts: playbooks for single hosts.
manual: playbooks that are only run manually by an admin as needed.
tasks - snippets of tasks that should be included in plays
roles - specific roles to be use in playbooks.
Each role has it's own files/templates/vars
filter_plugins - Jinja filters
master.yml - This is the master playbook, consisting of all
current group and host playbooks. Note that the
daily cron doesn't run this, it runs even over
playbooks that are not yet included in master.
This playbook is usefull for making changes over
multiple groups/hosts usually with -t (tag).
== Paths ==
public path for everything is:
/srv/web/infra/ansible
private path - which is sysadmin-main accessible only is:
/srv/private/ansible
In general to run any ansible playbook you will want to run:
sudo -i ansible-playbook /path/to/playbook.yml
== Scheduled check-diff ==
Every night a cron job runs over all playbooks under playbooks/{groups}{hosts}
with the ansible --check --diff options. A report from this is sent to
sysadmin-logs. In the ideal state this report would be empty.
== Idempotency ==
All playbooks should be idempotent. Ie, if run once they should bring the
machine(s) to the desired state, and if run again N times after that they should
make 0 changes (because the machine(s) are in the desired state).
Please make sure your playbooks are idempotent.
== Can be run anytime ==
When a playbook or change is checked into ansible you should assume
that it could be run at ANY TIME. Always make sure the checked in state
is the desired state. Always test changes when they land so they don't
surprise you later.