Testing:
From docker compose written to help this:
Two test environments: - top level Almalinux. , deb directory, there's small bit of debian based tested.
build
|
~/repos/mariadb.org-tools/daniel/MDEV-19210
|
$ docker-compose build
|
Building node1
|
STEP 1/4: FROM quay.io/almalinuxorg/8-init
|
Trying to pull quay.io/almalinuxorg/8-init:latest...
|
....
|
General principle is that a shutdown of any galera node will be able to recover an IST, provided the data cache of the donor node contains all the changes, without a full SST.
start a nodes - just systemd in a container at this point
|
$ docker-compose up
|
Creating network "mdev-19210_default" with the default driver
|
Creating volume "mdev-19210_v1" with default driver
|
Creating volume "mdev-19210_v2" with default driver
|
Creating volume "mdev-19210_v3" with default driver
|
Creating mdev-19210_node1_1 ... done
|
Attaching to mdev-19210_node1_1
|
Leave this as a running terminal
Per docs this is first step in a cluster:
start new cluster - galera_new_cluster
|
$ docker-compose exec node1 galera_new_cluster
|
$ docker-compose exec node1 journalctl -u mariadb.service
|
This shows it up as a primary.
$ docker-compose exec node2 systemctl start mariadb.service
|
$ docker-compose exec node2 journalctl -u mariadb.service -f
|
-- Logs begin at Wed 2023-12-20 06:32:51 UTC. --
|
Dec 20 06:35:01 f1bb63bfc6aa mariadbd[61]: 2023-12-20 6:35:01 0 [Note] WSREP: Shifting JOINER -> JOINED (TO: 2)
|
Dec 20 06:35:01 f1bb63bfc6aa mariadbd[61]: 2023-12-20 6:35:01 0 [Note] WSREP: Processing event queue:... -nan% (0/0 events) complete.
|
Dec 20 06:35:01 f1bb63bfc6aa mariadbd[61]: 2023-12-20 6:35:01 0 [Note] WSREP: Member 1.0 (f1bb63bfc6aa) synced with group.
|
Dec 20 06:35:01 f1bb63bfc6aa mariadbd[61]: 2023-12-20 6:35:01 0 [Note] WSREP: Processing event queue:...100.0% (1/1 events) complete.
|
Dec 20 06:35:01 f1bb63bfc6aa mariadbd[61]: 2023-12-20 6:35:01 0 [Note] WSREP: Shifting JOINED -> SYNCED (TO: 2)
|
Dec 20 06:35:01 f1bb63bfc6aa mariadbd[61]: 2023-12-20 6:35:01 2 [Note] WSREP: Server f1bb63bfc6aa synced with group
|
Dec 20 06:35:01 f1bb63bfc6aa mariadbd[61]: 2023-12-20 6:35:01 2 [Note] WSREP: Server status change joined -> synced
|
Dec 20 06:35:01 f1bb63bfc6aa mariadbd[61]: 2023-12-20 6:35:01 2 [Note] WSREP: Synchronized with group, ready for connections
|
Dec 20 06:35:01 f1bb63bfc6aa mariadbd[61]: 2023-12-20 6:35:01 2 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
|
Dec 20 06:35:01 f1bb63bfc6aa systemd[1]: Started MariaDB 11.4.0 database server.
|
^C
|
Same with node3.
start node3
|
$ docker-compose exec node3 systemctl start mariadb.service
|
|
$ docker-compose exec node3 mariadb -e 'show global status like "wsrep%"'
|
Test 1: shutdown a node. Add data to cluster, see if, how it recovers
$ docker-compose exec node2 systemctl stop mariadb.service
|
$ docker-compose exec node3 mariadb test -e 'create table t (i int not null primary key); insert into t select seq from seq_1_to_10000'
|
$ docker-compose exec node2 journalctl -f -u mariadb.service
|
-- Logs begin at Wed 2023-12-20 06:32:51 UTC. --
|
Dec 20 06:38:33 f1bb63bfc6aa mariadbd[61]: 2023-12-20 6:38:33 0 [Note] WSREP: Deinitializing allowlist service v1
|
Dec 20 06:38:33 f1bb63bfc6aa mariadbd[61]: 2023-12-20 6:38:33 0 [Note] InnoDB: FTS optimize thread exiting.
|
Dec 20 06:38:33 f1bb63bfc6aa mariadbd[61]: 2023-12-20 6:38:33 0 [Note] InnoDB: Starting shutdown...
|
Dec 20 06:38:33 f1bb63bfc6aa mariadbd[61]: 2023-12-20 6:38:33 0 [Note] InnoDB: Dumping buffer pool(s) to /var/lib/mysql/ib_buffer_pool
|
Dec 20 06:38:33 f1bb63bfc6aa mariadbd[61]: 2023-12-20 6:38:33 0 [Note] InnoDB: Buffer pool(s) dump completed at 231220 6:38:33
|
Dec 20 06:38:33 f1bb63bfc6aa mariadbd[61]: 2023-12-20 6:38:33 0 [Note] InnoDB: Removed temporary tablespace data file: "./ibtmp1"
|
Dec 20 06:38:33 f1bb63bfc6aa mariadbd[61]: 2023-12-20 6:38:33 0 [Note] InnoDB: Shutdown completed; log sequence number 59218; transaction id 39
|
Dec 20 06:38:33 f1bb63bfc6aa mariadbd[61]: 2023-12-20 6:38:33 0 [Note] /usr/sbin/mariadbd: Shutdown complete
|
Dec 20 06:38:33 f1bb63bfc6aa systemd[1]: mariadb.service: Succeeded.
|
Dec 20 06:38:33 f1bb63bfc6aa systemd[1]: Stopped MariaDB 11.4.0 database server.
|
Trigger start:
$ docker-compose exec node2 systemctl start mariadb.service
|
Observer form logs:
1. a quick start was made to find the position
2. the second start resumed from the log position
3. I saw a full SST here, thought it would be cached.
$ docker-compose exec node2 mariadb test -e 'select count(*) from t'
|
+----------+
|
| count(*) |
|
+----------+
|
| 10000 |
|
+----------+
|
$ docker-compose exec node3 systemctl stop mariadb.service
|
$ docker-compose exec node3 bash -c 'rm -rf /var/lib/mysql/*'
|
$ docker-compose exec node3 systemctl start mariadb.service
|
Observer the full rejoin of node3.
Try same on node1 to ensure it wasn't special because it was the bootstrap node.
reset evironment
|
docker-compose down -v
|
According to a conversation between marko and danblack on Zulip, we may need to merge this pull request to improve our Galera testing with systemd.