Commit Graph

844 Commits

Author SHA1 Message Date
Kevin Fenzi
0d08f15f41 ip6tables: allow dhcp6d from aws
While we actually use SLAAC in aws, there's a dhcp6d sending out the
router advertisements, so without that the instance doesn't get an ipv6
ip and just doesn't work. With this it does.

Signed-off-by: Kevin Fenzi <kevin@scrye.com>
2023-01-09 19:06:27 -08:00
Kevin Fenzi
f46beef7ac base: update list of build machines
Signed-off-by: Kevin Fenzi <kevin@scrye.com>
2022-12-05 16:35:48 -08:00
Kevin Fenzi
8cfa0a2a0b base: you cannot redirect in a command
Signed-off-by: Kevin Fenzi <kevin@scrye.com>
2022-08-01 12:34:38 -07:00
Kevin Fenzi
08ccdd41fc base: make sure we install and setup ipset for builders
Signed-off-by: Kevin Fenzi <kevin@scrye.com>
2022-08-01 12:31:20 -07:00
Kevin Fenzi
6898b5b35a koji_builders: enable ipset for osbuild access in prod
Signed-off-by: Kevin Fenzi <kevin@scrye.com>
2022-07-28 16:47:55 -07:00
Kevin Fenzi
fbc18add50 koji_builder: make iptables rule staging only until we are sure it is working there
Signed-off-by: Kevin Fenzi <kevin@scrye.com>
2022-07-28 12:54:03 -07:00
Kevin Fenzi
1951c7fb1b koji_builder: first cut at a script to update osbuild api ip for builder firewalls
This is a quick, hacked up script that just runs once per minute and
updates the ip addresses for the osbuild koji plugin. The script calls
systemd's resolvectl without cache and puts the ips in a ipset. The
koji_builder firewall has a added rule to check that ipset for outgoing
connections that are allowed.

TODO: add some kind of error checking
TODO: probibly won't work on s390x builders as they can't reach the host
even with open firewalls, but should work for others.

Signed-off-by: Kevin Fenzi <kevin@scrye.com>
2022-07-28 11:29:07 -07:00
Kevin Fenzi
9ddd3e2e6d postfix / gateway: add bysender map
This is a hack to work around SPF screwing us for @fedoraproject.org
aliases. It only fixes email from @redhat.com, but due to bugzilla thats
a lot of email.

Without this:
bugzilla@redhat.com -> user@fedoraproject.org (expands) ->
user@gmail.com sent out directly to gmail and gets rejected because
we aren't in the redhat.com SPF record.

With this:

bugzilla@redhat.com -> user@fedoraproject.org (expands) ->
user@gmail.com but sent to mx2.redhat.com to deliver. Since
mx2.redhat.com definitely is in the redhat.com SPF record the email is
delivered fine and SPF checks pass.

This won't help for other domains with -all SPF records, but at least it
helps for all the redhat.com emails, of which there are a lot going to
fedoraproject.org aliases. :)

Signed-off-by: Kevin Fenzi <kevin@scrye.com>
2022-07-26 15:58:43 -07:00
Kevin Fenzi
7cf2d7b56a iptables / builders / staging: adjust to allow proxy01/10/101/110 in staging
Signed-off-by: Kevin Fenzi <kevin@scrye.com>
2022-07-26 14:22:55 -07:00
Kevin Fenzi
24088dfe9b postfix: put CA file back, needed to verify other certs
Signed-off-by: Kevin Fenzi <kevin@scrye.com>
2022-07-21 16:42:22 -07:00
Kevin Fenzi
a3c96c4032 postfix: gateway: consolidate tls settings and use chain-file.
Signed-off-by: Kevin Fenzi <kevin@scrye.com>
2022-07-21 16:33:25 -07:00
Kevin Fenzi
a336ba2a9e postfix: use a chain file for ssl certs
Signed-off-by: Kevin Fenzi <kevin@scrye.com>
2022-07-21 16:25:10 -07:00
Kevin Fenzi
b54b8a0bb7 bastion / postfix: adjust gateway ssl cert to base name
Signed-off-by: Kevin Fenzi <kevin@scrye.com>
2022-07-21 13:39:43 -07:00
Kevin Fenzi
a9c5fa224f kojibuilder / staging: add in rules for staging builders to talk to ipa and nfs on koji01.stg
Signed-off-by: Kevin Fenzi <kevin@scrye.com>
2022-06-28 16:44:49 -07:00
Kevin Fenzi
c31c99a252 pagure: enable tls for smtp in prod too
Signed-off-by: Kevin Fenzi <kevin@scrye.com>
2022-06-13 18:07:22 -07:00
Kevin Fenzi
9953c1346f postfix / also install tls_policy on pagure*
Signed-off-by: Kevin Fenzi <kevin@scrye.com>
2022-06-13 18:03:59 -07:00
Kevin Fenzi
352ac4d6fd pagure / staging: correct path to certs, install tls_policy map
Signed-off-by: Kevin Fenzi <kevin@scrye.com>
2022-06-13 18:01:00 -07:00
Kevin Fenzi
0df1496ae6 pagure / staging: enable tls for postfix.
Signed-off-by: Kevin Fenzi <kevin@scrye.com>
2022-06-13 17:54:07 -07:00
Kevin Fenzi
92867b50a2 base / crypto-policies: move fedora instances back to DEFAULT
We changed this to DEFAULT:FEDORA32 a while back because the certs for
the old totpcgi sudo needed it to work. Now thats all gone and we are
100% on ipa and sssd, this should no longer be needed.

Signed-off-by: Kevin Fenzi <kevin@scrye.com>
2022-06-06 14:10:20 -07:00
Kevin Fenzi
17959a301c base: no more docker anywhere, drop this
We aren't using docker anywhere anymore, so we don't need to uselessly
try and restart it all the time.

Signed-off-by: Kevin Fenzi <kevin@scrye.com>
2022-06-02 15:14:54 -07:00
Kevin Fenzi
33403e750e bastion: readd may tls for all non rh sites
Signed-off-by: Kevin Fenzi <kevin@scrye.com>
2022-05-24 17:46:04 -07:00
Kevin Fenzi
caf5f130c8 bastion / postfix: clean up tls_policy duplicates
Signed-off-by: Kevin Fenzi <kevin@scrye.com>
2022-05-24 12:33:26 -07:00
Kevin Fenzi
a1af68173d bastion / postfix: disable tls_resue for now
Something is broken with smtp_tls_connection_reuse = yes, so disable it
for now. Also, setup a tls_policy map file and tell it to not use tls
for mx2.redhat.com. The normal smtp connection reuse works just fine, so
this will keep mail flowing until we can one day figure out why tls
connection reuse is busted.

Signed-off-by: Kevin Fenzi <kevin@scrye.com>
2022-05-24 12:24:30 -07:00
Kevin Fenzi
81771937c2 bastion / postfix: need to set session cache for tls connections and timeout
Signed-off-by: Kevin Fenzi <kevin@scrye.com>
2022-05-24 09:13:08 -07:00
Kevin Fenzi
105cad26dc postfix / main.cf: fix releng and vpn config and drop phx2
We fixed the config in a PR from aheath1992 for most of the machines,
but we need to fix vpn (proxies in particular) and releng boxes now.
Also, while we are here, lets drop the phx2 file since it's not used
anymore.

Signed-off-by: Kevin Fenzi <kevin@scrye.com>
2022-05-23 16:43:01 -07:00
Kevin Fenzi
b1395e9088 bastion / postfix: re-add missing smtp_connection_cache_destinations
Without this it wasn't caching tls connections and was going over the
small limit redhat.com mx had. Hopefully this gets mail flowing again.

Signed-off-by: Kevin Fenzi <kevin@scrye.com>
2022-05-23 08:44:36 -07:00
Andrew Heath
32764a7f3a Testing https://access.redhat.com/solutions/5420401 on postfix systems 2022-05-17 00:29:25 +00:00
Kevin Fenzi
ba294ed5c6 postfix / gateway: fix tlsproxy to actually work
So, without this tlsproxy wasn't working and no connection reuse was
happening. With it, it seems to be processing away nicely and reusing
connections.

Signed-off-by: Kevin Fenzi <kevin@scrye.com>
2022-04-26 12:31:48 -07:00
Kevin Fenzi
6d3114e3bd base / postfix / bastion/gateway: fix up email flow to @redhat.com
Recently, redhat.com changed internal MX servers. The new servers are
have rate limits on incoming emails from one ip and admins there don't
want to add a bunch of exceptions, so we need to adjust our end to not
flood connections to them. Currently, connections burst up to 100 (the
smtp postfix default) which goes over their limits and causes the
internal MX to reject emails from us for a while.

So, this change:

* Adds some domains to fast_flush. This allows us to use postqueue -s
  domain to flush emails to a particular domain.

* Changes the smtp limit to 40. This is under the redhat.com limit.

* Has ansible actually install the master.cf.gateway on bastion servers.
  Currently they were using the stock/default one.

* Enables the tlsproxy service, which is actually needed to get that tls
  reuse working.

After these changes, we keep few connections to the redhat.com mx open,
but we reuse them and send more emails over existing connections. No
'too many connection emails' have happened since the changes.
The queue slowly seems to be processing down.

Since this was causing an outage of email, I have already applied these
things to bastion01, but I'd like to make sure we match up to whats in
ansible.

Signed-off-by: Kevin Fenzi <kevin@scrye.com>
2022-04-24 16:59:35 +00:00
Kevin Fenzi
affac369ab postfix / main.cf / gateway: enable smtp_tls_connection_reuse
Right now we are hitting redhat.com mx server connection limits.
This might be because we are starting too many new connections at once.
Enabling this should reduce the new connections by reusing existing
ones.

Signed-off-by: Kevin Fenzi <kevin@scrye.com>
2022-04-20 16:07:40 -07:00
Kevin Fenzi
44caedc87d bastion: switch to mx2 for redhat.com emails
It seems this is a load balancer that we should be using according to
the internal ticket.

Signed-off-by: Kevin Fenzi <kevin@scrye.com>
2022-04-05 07:58:45 -07:00
Kevin Fenzi
20b3db2561 bastion: switch email gateway for redhat.com emails
The old ext-mx seems no more, switch to new gateway.

Signed-off-by: Kevin Fenzi <kevin@scrye.com>
2022-04-04 16:20:27 -07:00
Kevin Fenzi
fbb4e4691f Adjust the list of hosts that do not use linux-system-roles/network
In addition to the cloud_aws group, we want to exclude ibiblio virthosts
as they use bonding and thats a more complex setup. Someday we should
get it working here.

Signed-off-by: Kevin Fenzi <kevin@scrye.com>
2022-01-26 14:09:43 -08:00
Kevin Fenzi
eb991fa9c1 base / big network cleanup
Everything should now be using linux-system-roles/network, so we drop
our hacky nmcli calls and everything that referred to them, including
exclude variables. Also, lets just let NM handle resolv.conf so it's not
wrong all the time on reboots.

Signed-off-by: Kevin Fenzi <kevin@scrye.com>
2022-01-26 12:14:23 -08:00
Kevin Fenzi
f824885753 base / linux-system-roles: the group is cloud_aws
Signed-off-by: Kevin Fenzi <kevin@scrye.com>
2022-01-25 15:16:05 -08:00
Kevin Fenzi
f6a8b121f7 base / linux-system-roles: double quote the when line
Signed-off-by: Kevin Fenzi <kevin@scrye.com>
2022-01-25 15:10:19 -08:00
Kevin Fenzi
796f59fa1b base / linux-system-roles: try and reword when
Signed-off-by: Kevin Fenzi <kevin@scrye.com>
2022-01-25 15:07:45 -08:00
Kevin Fenzi
ec42acdcab base / linux-system-roles: Clean up conditional
Signed-off-by: Kevin Fenzi <kevin@scrye.com>
2022-01-25 15:02:04 -08:00
Kevin Fenzi
6ebe03a470 base: linux-system-roles: cloud datacenter to be wider
Signed-off-by: Kevin Fenzi <kevin@scrye.com>
2022-01-25 14:52:13 -08:00
Kevin Fenzi
d1b419b257 base: linux-system-roles are not used in cloud/aws
Signed-off-by: Kevin Fenzi <kevin@scrye.com>
2022-01-25 14:45:57 -08:00
Pavel Raiskup
408a990a24 base: try s/dnf/package/ module
TASK [base : Ensure iptables is installed] ************************************
Wednesday 05 January 2022  10:49:01 +0000 (0:00:00.251)       0:01:17.816 *****
Wednesday 05 January 2022  10:49:01 +0000 (0:00:00.251)       0:01:17.815 *****
fatal: [noc02.fedoraproject.org]: FAILED! => {"changed": false, "cmd":
"dnf install -y python2-dnf", "msg": "[Errno 2] No such file or directory", "rc": 2}
2022-01-05 13:03:37 +01:00
Mark O Brien
148f8307d7 base: ensure iptables is installed
Signed-off-by: Mark O Brien <markobri@redhat.com>
2021-12-21 17:06:32 +00:00
T.C. Williams
995f499a6d Adjust rsyslog imjournal rate-limiting settings to prevent dropped messages under load. fedora-infrastructure-9512 2021-11-19 00:30:32 +00:00
Eddie Jennings, Jr
6ef496d56a Reconfigure IPv6
Reconfigure IPv6

Reconfigure IPv6

Reconfigure IPv6

Reconfigure IPv6

Reconfigure IPv6

Reconfigure IPv6

Reconfigure IPv6

Configure IPv6

Reconfigure IPv6

Reconfigure IPv6

Reconfigure IPv6

Reconfigure IPv6

Reconfigure IPv6

Reconfigure IPv6

Reconfigure IPv6

Reconfigure IPv6

Configure IPv6

Update IPv6 address for noc02 rule

Update IPv6 address in confg for noc02 address change

Update IPv6 address for proxy04

Update IPv6 address for torrent02
2021-11-08 22:56:05 +00:00
Nils Philippsen
c782eceae1 Move syncHttpLogs.sh into web-data-analysis role
Signed-off-by: Nils Philippsen <nils@redhat.com>
2021-07-23 13:06:23 +02:00
Adam Saleh
eb42244c28 Remove the comment, timeouts should be sufficient. 2021-07-22 21:34:00 +02:00
Adam Saleh
be40017d00 Change the syncHttp timeout to 2h max, comment out the misbehaving proxy09 2021-07-22 21:34:00 +02:00
Adam Saleh
721e3e9864 Add 3 times retry and 30m+30m rsync timeout to syncHttpLogs 2021-07-22 21:34:00 +02:00
Kevin Fenzi
5d6f460b42 buildvm_s390x: the z/vm s390x instances use enc900 as network interface
Signed-off-by: Kevin Fenzi <kevin@scrye.com>
2021-07-07 13:43:22 -07:00
Kevin Fenzi
ad91eae586 base: adjust grouping to give the desired result in when for NM tweaking
Signed-off-by: Kevin Fenzi <kevin@scrye.com>
2021-07-07 12:56:46 -07:00