[MDBF-312] s390x-sles broken Created: 2022-01-11  Updated: 2023-03-16  Resolved: 2022-02-14

Status: Closed
Project: MariaDB Foundation Development
Component/s: None
Affects Version/s: None
Fix Version/s: N/A

Type: Task Priority: Major
Reporter: Vlad Bogolin Assignee: Vlad Bogolin
Resolution: Fixed Votes: 0
Labels: buildbot, s390x, sles
Remaining Estimate: 0d
Time Spent: 3h
Original Estimate: Not Specified

Issue Links:
PartOf
is part of MDBF-41 Milestone 5: Desirable fixes Open
Problem/Incident
is caused by MDBF-344 Check container internet access after... Open

 Description   

The s390x SLES builder is not starting. Probably related to the container not having access to the master, but needs to be investigated furher.

 Comments   
Comment by Vlad Bogolin [ 2022-01-12 ]

The containers have no access. Used sudo sysctl -w net.ipv4.ip_forward=1 to resolve the issue. Now the containers connect to the master.

Comment by Faustin Lammler [ 2022-01-17 ] 

sudo sysctl -w net.ipv4.ip_forward=1

Comment by Vlad Bogolin [ 2022-01-17 ]

Command after the reboot to deal with firewall issues

| sudo -s
 
| iptables -F
 
| iptables -X
 
| iptables -t nat -F
 
| iptables -t nat -X
 
| iptables -t mangle -F
 
| iptables -t mangle -X
 
| iptables -P INPUT ACCEPT
 
| iptables -P OUTPUT ACCEPT
 
| iptables -P FORWARD ACCEPT
 
| ufw enable

Comment by Vlad Bogolin [ 2022-01-17 ]

Here is how to reinstall the wireguard module if kernel is upgraded on
the ibm-s390x-sles15 machine (based on
https://www.wireguard.com/compilation/) :

|faust@mariadb03:~> cd /home/faust/
 
|faust@mariadb03:~> make -C wireguard-linux-compat/src -j$(nproc)
 
|make: Entering directory '/home/faust/wireguard-linux-compat/src'
 
|  CC [M]  /home/faust/wireguard-linux-compat/src/main.o
 
|  CC [M]  /home/faust/wireguard-linux-compat/src/noise.o
 
|  CC [M]  /home/faust/wireguard-linux-compat/src/device.o
 
|  CC [M]  /home/faust/wireguard-linux-compat/src/peer.o
 
|  CC [M]  /home/faust/wireguard-linux-compat/src/timers.o
 
|  CC [M]  /home/faust/wireguard-linux-compat/src/queueing.o
 
|  CC [M]  /home/faust/wireguard-linux-compat/src/send.o
 
|  CC [M]  /home/faust/wireguard-linux-compat/src/receive.o
 
|  CC [M]  /home/faust/wireguard-linux-compat/src/socket.o
 
|  CC [M]  /home/faust/wireguard-linux-compat/src/peerlookup.o
 
|  CC [M]  /home/faust/wireguard-linux-compat/src/allowedips.o
 
|  CC [M]  /home/faust/wireguard-linux-compat/src/ratelimiter.o
 
|  CC [M]  /home/faust/wireguard-linux-compat/src/cookie.o
 
|  CC [M]  /home/faust/wireguard-linux-compat/src/netlink.o
 
|  CC [M]  /home/faust/wireguard-linux-compat/src/crypto/zinc/chacha20/chacha20.o
 
|  CC [M]  /home/faust/wireguard-linux-compat/src/crypto/zinc/poly1305/poly1305.o
 
|  CC [M]  /home/faust/wireguard-linux-compat/src/crypto/zinc/chacha20poly1305.o
 
|  CC [M]  /home/faust/wireguard-linux-compat/src/crypto/zinc/blake2s/blake2s.o
 
|  CC [M]  /home/faust/wireguard-linux-compat/src/crypto/zinc/curve25519/curve25519.o
 
|  LD [M]  /home/faust/wireguard-linux-compat/src/wireguard.o
 
|  Building modules, stage 2.
 
|  MODPOST 1 modules
 
|  CC      /home/faust/wireguard-linux-compat/src/wireguard.mod.o
 
|  LD [M]  /home/faust/wireguard-linux-compat/src/wireguard.ko
 
|make: Leaving directory '/home/faust/wireguard-linux-compat/src'
 
|faust@mariadb03:~> sudo make -C wireguard-linux-compat/src install
 
|make: Entering directory '/home/faust/wireguard-linux-compat/src'
 
|  INSTALL /home/faust/wireguard-linux-compat/src/wireguard.ko
 
|  DEPMOD  5.3.18-24.96-default
 
|Warning: modules_install: missing 'System.map' file. Skipping depmod.
 
|depmod -b "/" -a 5.3.18-24.96-default
 
|make: Leaving directory '/home/faust/wireguard-linux-compat/src'
 
|faust@mariadb03:~> sudo modprobe wireguard
 
|faust@mariadb03:~> sudo systemctl start wg-quick@wg0
 
|faust@mariadb03:~> sudo wg
 
|interface: wg0
 
|  public key: r2vvba7y3BZjZBD6Z9zy/ZODfYVgmhWSC6XSxNtQ6CA=
 
|  private key: (hidden)
 
|  listening port: 52791
 
|
 
|peer: aUgFc8jTyCp9hdoeAgxxPi0xMg6Sa2wxjrMUlRF6H1g=
 
|  endpoint: 135.181.143.118:51820
 
|  allowed ips: 100.64.100.1/32
 
|  latest handshake: 3 seconds ago
 
|  transfer: 13.65 KiB received, 6.36 KiB sent
 
|  persistent keepalive: every 25 seconds

Comment by Faustin Lammler [ 2022-03-22 ]

After reboot we had still a firewall problem.

The following line was commented in `/etc/init.d/boot.local`:

#iptables-restore /etc/linuxone/iptables.save

Reboot tested twice, should be ok now.

Comment by Faustin Lammler [ 2022-03-22 ]

Same was done for the rhel machine:

The following line was commented in `/etc/rc.d/rc.local`:

#iptables-restore < /etc/sysconfig/iptables.save

Reboot tested, should be ok now.

Comment by Faustin Lammler [ 2022-09-23 ]

For ubuntu, remove the netfilter-persistent package:

sudo apt purge netfilter-persistent

Comment by Faustin Lammler [ 2022-10-13 ]

On s390x rhel8:

The following patch needs to be applied https://github.com/WireGuard/wireguard-linux-compat/commit/8cfcb57cdcc020deb7727e73c231f6ea08e692fd

wget https://github.com/WireGuard/wireguard-linux-compat/commit/8cfcb57cdcc020deb7727e73c231f6ea08e692fd.patch
 
cd wireguard-linux-compat
 
git apply ../8cfcb57cdcc020deb7727e73c231f6ea08e692fd.patch
 
make -C src -j$(nproc)

Then the following error can appear:

[linux1@mariadbrhrel8 ~]$ sudo make -C wireguard-linux-compat/src install
 
make: Entering directory '/home/linux1/wireguard-linux-compat/src'
 
  INSTALL /home/linux1/wireguard-linux-compat/src/wireguard.ko
 
At main.c:160:
 
- SSL error:02001002:system library:fopen:No such file or directory: crypto/bio/bss_file.c:69
 
- SSL error:2006D080:BIO routines:BIO_new_file:no such file: crypto/bio/bss_file.c:76
 
sign-file: certs/signing_key.pem: No such file or directory
 
  DEPMOD  4.18.0-372.26.1.el8_6.s390x
 
depmod -b "/" -a 4.18.0-372.26.1.el8_6.s390x
 
make: Leaving directory '/home/linux1/wireguard-linux-compat/src'

This can be resolved with https://github.com/andikleen/simple-pt/issues/8#issuecomment-813438385

  • ufw deployment is also problematic, here is the workaround

[linux1@mariadbrhrel8 yum.repos.d]$ sudo ufw status
 
Status: inactive
 
[linux1@mariadbrhrel8 yum.repos.d]$ sudo ufw allow ssh
 
ERROR: problem running
 
[linux1@mariadbrhrel8 yum.repos.d]$ sudo ufw disable
 
Firewall stopped and disabled on system startup
 
[linux1@mariadbrhrel8 yum.repos.d]$ sudo ufw allow ssh
 
Skipping adding existing rule
 
Rules updated (v6)
 
[linux1@mariadbrhrel8 yum.repos.d]$ sudo ufw enable
 
Command may disrupt existing ssh connections. Proceed with operation (y|n)? y
 
Firewall is active and enabled on system startup

Comment by Faustin Lammler [ 2022-10-13 ]

For wireguard sles15 some step were missing:

1/ apply the following patch

diff --git a/src/compat/compat.h b/src/compat/compat.h
 
index 91d4388..e15b76b 100644
 
--- a/src/compat/compat.h
 
+++ b/src/compat/compat.h
 
@@ -849,17 +849,6 @@ static inline void skb_mark_not_on_list(struct sk_buff *skb)
 
 #endif
 
 #endif
 
 
 
-#if LINUX_VERSION_CODE < KERNEL_VERSION(5, 5, 0)
 
-#define genl_dumpit_info(cb) ({ \
 
-       struct { struct nlattr **attrs; } *a = (void *)((u8 *)cb->args + offsetofend(struct dump_ctx, next_allowedip)); \
 
-       BUILD_BUG_ON(sizeof(cb->args) < offsetofend(struct dump_ctx, next_allowedip) + sizeof(*a)); \
 
-       a->attrs = genl_family_attrbuf(&genl_family); \
 
-       if (nlmsg_parse(cb->nlh, GENL_HDRLEN + genl_family.hdrsize, a->attrs, genl_family.maxattr, device_policy, NULL) < 0) \
 
-               memset(a->attrs, 0, (genl_family.maxattr + 1) * sizeof(struct nlattr *)); \
 
-       a; \
 
-})
 
-#endif
 
-
 
 #if LINUX_VERSION_CODE < KERNEL_VERSION(5, 6, 0)
 
 #include <linux/skbuff.h>
 
 #ifndef skb_list_walk_safe

2/ add the following in /etc/modprobe.d/10-unsupported-modules.conf

allow_unsupported_modules 1

Comment by Faustin Lammler [ 2022-10-13 ]

For sles15, you need to install the following packages:

sudo zypper install zabbix-agent cron python3-rpm

here is the list of role that don't work for automatic deployment:

  • ufw
  • zabbix-agent (needs to be installed manually)
  • docker (needs to be installed manually)

For the firewall, it needs to be reseted:

sudo rm /etc/iptables.save
 
sudo ufw disable
 
sudo ufw allow ssh
 
sudo ufw enable
 
sudo reboot

Generated at Thu Feb 08 03:36:58 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.