[#MDEV-28256] Galera SST fails with 'InnoDB: Upgrade after a crash is not supported'

Hi, I'm trying to join a new 10.5 node to a 10.4 Galera cluster as part of a migration to new AWS instance types.

The symptom is that the initial SST completes, but on the restart the joining node then fails with this error:

Apr  7 10:39:52 ip-172-31-39-121 mariadbd[129747]: 2022-04-07 10:39:52 0 [ERROR] InnoDB: Upgrade after a crash is not supported. The redo log was created with MariaDB 10.4.22.

After various Googling this seems to be related to crash recovery files ib_logfile0 and ib_logfile1, which don't exist on the empty JOINING node until the SST completes. The DONOR node was cleanly restarted prior to the sync. I'm not sure if this is a known issue?

Things I have already tried:

Setting innodb_fast_shutdown=0 on DONOR and JOINING nodes
Deleting ib_logfile0 and ib_logfile1 on the JOINER and restarting

Is there a known work-around for this? I'm cautious to try upgrading an existing node to 10.5 in case it has the same error.

Galera Config

[mysqld]

binlog_format=ROW

default-storage-engine=innodb

innodb_autoinc_lock_mode=2

query_cache_type=0

query_cache_size=0

bind-address=0.0.0.0

[galera]

wsrep_on=ON

# Galera Provider Configuration

wsrep_provider=/usr/lib/galera/libgalera_smm.so

wsrep_provider_options="gcache.size=5G"

# Galera Cluster Configuration

wsrep_cluster_name="mariadb-cluster"

wsrep_cluster_address="gcomm://172.31.34.27,172.31.35.130,172.31.37.249,172.31.40.66,172.31.37.12"

# Galera Synchronization Congifuration

wsrep_sst_method = rsync

wsrep_sst_donor = ES5_AWS

# Galera Node Configuration

wsrep_node_name="ES6_AWS"

wsrep_node_address="172.31.39.121"

JOINER Logs

Apr  7 10:35:46 ip-172-31-39-121 systemd[1]: Starting MariaDB 10.5.15 database server...

Apr  7 10:35:47 ip-172-31-39-121 sh[129656]: WSREP: Recovered position 00000000-0000-0000-0000-000000000000:-1

Apr  7 10:35:47 ip-172-31-39-121 mariadbd[129747]: 2022-04-07 10:35:47 0 [Note] /usr/sbin/mariadbd (mysqld 10.5.15-MariaDB-1:10.5.15+maria~focal) starting as process 129747 ...

Apr  7 10:35:47 ip-172-31-39-121 mariadbd[129747]: 2022-04-07 10:35:47 0 [Note] WSREP: Loading provider /usr/lib/galera/libgalera_smm.so initial position: 00000000-0000-0000-0000-000000000000:-1

Apr  7 10:35:47 ip-172-31-39-121 mariadbd[129747]: 2022-04-07 10:35:47 0 [Note] WSREP: wsrep_load(): loading provider library '/usr/lib/galera/libgalera_smm.so'

Apr  7 10:35:47 ip-172-31-39-121 mariadbd[129747]: 2022-04-07 10:35:47 0 [Note] WSREP: wsrep_load(): Galera 26.4.11(r67341d07) by Codership Oy <info@codership.com> loaded successfully.

Apr  7 10:35:47 ip-172-31-39-121 mariadbd[129747]: 2022-04-07 10:35:47 0 [Note] WSREP: CRC-32C: using hardware acceleration.

Apr  7 10:35:47 ip-172-31-39-121 mariadbd[129747]: 2022-04-07 10:35:47 0 [Warning] WSREP: Could not open state file for reading: '/var/lib/mysql//grastate.dat'

Apr  7 10:35:47 ip-172-31-39-121 mariadbd[129747]: 2022-04-07 10:35:47 0 [Note] WSREP: Found saved state: 00000000-0000-0000-0000-000000000000:-1, safe_to_bootstrap: 1

Apr  7 10:35:47 ip-172-31-39-121 mariadbd[129747]: 2022-04-07 10:35:47 0 [Note] WSREP: GCache DEBUG: opened preamble:

Apr  7 10:35:47 ip-172-31-39-121 mariadbd[129747]: Version: 0

Apr  7 10:35:47 ip-172-31-39-121 mariadbd[129747]: UUID: 00000000-0000-0000-0000-000000000000

Apr  7 10:35:47 ip-172-31-39-121 mariadbd[129747]: Seqno: -1 - -1

Apr  7 10:35:47 ip-172-31-39-121 mariadbd[129747]: Offset: -1

Apr  7 10:35:47 ip-172-31-39-121 mariadbd[129747]: Synced: 0

Apr  7 10:35:47 ip-172-31-39-121 mariadbd[129747]: 2022-04-07 10:35:47 0 [Note] WSREP: Skipped GCache ring buffer recovery: could not determine history UUID.

Apr  7 10:35:47 ip-172-31-39-121 mariadbd[129747]: 2022-04-07 10:35:47 0 [Warning] WSREP: Option 'gcs.fc_master_slave' is deprecated and will be removed in the future versions, please use 'gcs.fc_single_primary' instead.

Apr  7 10:35:47 ip-172-31-39-121 mariadbd[129747]: 2022-04-07 10:35:47 0 [Note] WSREP: Passing config to GCS: base_dir = /var/lib/mysql/; base_host = 172.31.39.121; base_port = 4567; cert.log_conflicts = no; cert.optimistic_pa = yes; debug = no; evs.auto_evict = 0; evs.delay_margin = PT1S; evs.delayed_keep_period = PT30S; evs.inactive_check_period = PT0.5S; evs.inactive_timeout = PT15S; evs.join_retrans_period = PT1S; evs.max_install_timeouts = 3; evs.send_window = 4; evs.stats_report_period = PT1M; evs.suspect_timeout = PT5S; evs.user_send_window = 2; evs.view_forget_timeout = PT24H; gcache.dir = /var/lib/mysql/; gcache.keep_pages_size = 0; gcache.keep_plaintext_size = 128M; gcache.mem_size = 0; gcache.name = galera.cache; gcache.page_size = 128M; gcache.recover = yes; gcache.size = 5G; gcomm.thread_prio = ; gcs.fc_debug = 0; gcs.fc_factor = 1.0; gcs.fc_limit = 16; gcs.fc_master_slave = no; gcs.fc_single_primary = no; gcs.max_packet_size = 64500; gcs.max_throttle = 0.25; gcs.recv_q_hard_limit = 9223372036854775807; gcs.recv_q_soft_limit = 0.25; gcs.sync_donor = no; gmcast.segment = 0;

Apr  7 10:35:47 ip-172-31-39-121 mariadbd[129747]: 2022-04-07 10:35:47 0 [Note] WSREP: Start replication

Apr  7 10:35:47 ip-172-31-39-121 mariadbd[129747]: 2022-04-07 10:35:47 0 [Note] WSREP: Connecting with bootstrap option: 0

Apr  7 10:35:47 ip-172-31-39-121 mariadbd[129747]: 2022-04-07 10:35:47 0 [Note] WSREP: Setting GCS initial position to 00000000-0000-0000-0000-000000000000:-1

Apr  7 10:35:47 ip-172-31-39-121 mariadbd[129747]: 2022-04-07 10:35:47 0 [Note] WSREP: protonet asio version 0

Apr  7 10:35:47 ip-172-31-39-121 mariadbd[129747]: 2022-04-07 10:35:47 0 [Note] WSREP: Using CRC-32C for message checksums.

Apr  7 10:35:47 ip-172-31-39-121 mariadbd[129747]: 2022-04-07 10:35:47 0 [Note] WSREP: backend: asio

Apr  7 10:35:47 ip-172-31-39-121 mariadbd[129747]: 2022-04-07 10:35:47 0 [Note] WSREP: gcomm thread scheduling priority set to other:0

Apr  7 10:35:47 ip-172-31-39-121 mariadbd[129747]: 2022-04-07 10:35:47 0 [Warning] WSREP: access file(/var/lib/mysql//gvwstate.dat) failed(No such file or directory)

Apr  7 10:35:47 ip-172-31-39-121 mariadbd[129747]: 2022-04-07 10:35:47 0 [Note] WSREP: restore pc from disk failed

Apr  7 10:35:47 ip-172-31-39-121 mariadbd[129747]: 2022-04-07 10:35:47 0 [Note] WSREP: GMCast version 0

Apr  7 10:35:47 ip-172-31-39-121 mariadbd[129747]: 2022-04-07 10:35:47 0 [Note] WSREP: (7c67946e-b045, 'tcp://0.0.0.0:4567') listening at tcp://0.0.0.0:4567

Apr  7 10:35:47 ip-172-31-39-121 mariadbd[129747]: 2022-04-07 10:35:47 0 [Note] WSREP: (7c67946e-b045, 'tcp://0.0.0.0:4567') multicast: , ttl: 1

Apr  7 10:35:47 ip-172-31-39-121 mariadbd[129747]: 2022-04-07 10:35:47 0 [Note] WSREP: EVS version 1

Apr  7 10:35:47 ip-172-31-39-121 mariadbd[129747]: 2022-04-07 10:35:47 0 [Note] WSREP: gcomm: connecting to group 'mariadb-cluster', peer '172.31.34.27:,172.31.35.130:,172.31.37.249:,172.31.40.66:,172.31.37.12:'

Apr  7 10:35:47 ip-172-31-39-121 mariadbd[129747]: 2022-04-07 10:35:47 0 [Note] WSREP: (7c67946e-b045, 'tcp://0.0.0.0:4567') connection established to 95267912-98ce tcp://172.31.34.27:4567

Apr  7 10:35:47 ip-172-31-39-121 mariadbd[129747]: 2022-04-07 10:35:47 0 [Note] WSREP: (7c67946e-b045, 'tcp://0.0.0.0:4567') turning message relay requesting on, nonlive peers: tcp://172.31.37.75:4567 tcp://172.31.9.176:4567

Apr  7 10:35:47 ip-172-31-39-121 mariadbd[129747]: 2022-04-07 10:35:47 0 [Note] WSREP: (7c67946e-b045, 'tcp://0.0.0.0:4567') connection established to 98319f4a-8b85 tcp://172.31.35.130:4567

Apr  7 10:35:47 ip-172-31-39-121 mariadbd[129747]: 2022-04-07 10:35:47 0 [Note] WSREP: (7c67946e-b045, 'tcp://0.0.0.0:4567') connection established to da9e7474-8746 tcp://172.31.40.66:4567

Apr  7 10:35:47 ip-172-31-39-121 mariadbd[129747]: 2022-04-07 10:35:47 0 [Note] WSREP: (7c67946e-b045, 'tcp://0.0.0.0:4567') connection established to e4e23eea-bbd0 tcp://172.31.37.249:4567

Apr  7 10:35:47 ip-172-31-39-121 mariadbd[129747]: 2022-04-07 10:35:47 0 [Note] WSREP: (7c67946e-b045, 'tcp://0.0.0.0:4567') connection established to 44508de1-af53 tcp://172.31.37.75:4567

Apr  7 10:35:47 ip-172-31-39-121 mariadbd[129747]: 2022-04-07 10:35:47 0 [Note] WSREP: (7c67946e-b045, 'tcp://0.0.0.0:4567') connection established to eb5f53de-8588 tcp://172.31.9.176:4567

Apr  7 10:35:47 ip-172-31-39-121 mariadbd[129747]: 2022-04-07 10:35:47 0 [Note] WSREP: (7c67946e-b045, 'tcp://0.0.0.0:4567') connection established to eb5f53de-8588 tcp://172.31.9.176:4567

Apr  7 10:35:47 ip-172-31-39-121 mariadbd[129747]: 2022-04-07 10:35:47 0 [Note] WSREP: Deferred close timer started for socket with remote endpoint: tcp://172.31.9.176:39010

Apr  7 10:35:47 ip-172-31-39-121 mariadbd[129747]: 2022-04-07 10:35:47 0 [Note] WSREP: Deferred close timer handle_wait Operation aborted. for 0xaaab09f1f940

Apr  7 10:35:47 ip-172-31-39-121 mariadbd[129747]: 2022-04-07 10:35:47 0 [Note] WSREP: Deferred close timer destruct

Apr  7 10:35:48 ip-172-31-39-121 mariadbd[129747]: 2022-04-07 10:35:48 0 [Note] WSREP: EVS version upgrade 0 -> 1

Apr  7 10:35:48 ip-172-31-39-121 mariadbd[129747]: 2022-04-07 10:35:48 0 [Note] WSREP: declaring 44508de1-af53 at tcp://172.31.37.75:4567 stable

Apr  7 10:35:48 ip-172-31-39-121 mariadbd[129747]: 2022-04-07 10:35:48 0 [Note] WSREP: declaring 95267912-98ce at tcp://172.31.34.27:4567 stable

Apr  7 10:35:48 ip-172-31-39-121 mariadbd[129747]: 2022-04-07 10:35:48 0 [Note] WSREP: declaring 98319f4a-8b85 at tcp://172.31.35.130:4567 stable

Apr  7 10:35:48 ip-172-31-39-121 mariadbd[129747]: 2022-04-07 10:35:48 0 [Note] WSREP: declaring da9e7474-8746 at tcp://172.31.40.66:4567 stable

Apr  7 10:35:48 ip-172-31-39-121 mariadbd[129747]: 2022-04-07 10:35:48 0 [Note] WSREP: declaring e4e23eea-bbd0 at tcp://172.31.37.249:4567 stable

Apr  7 10:35:48 ip-172-31-39-121 mariadbd[129747]: 2022-04-07 10:35:48 0 [Note] WSREP: declaring eb5f53de-8588 at tcp://172.31.9.176:4567 stable

Apr  7 10:35:48 ip-172-31-39-121 mariadbd[129747]: 2022-04-07 10:35:48 0 [Note] WSREP: PC protocol upgrade 0 -> 1

Apr  7 10:35:49 ip-172-31-39-121 mariadbd[129747]: 2022-04-07 10:35:49 0 [Note] WSREP: Node 44508de1-af53 state prim

Apr  7 10:35:49 ip-172-31-39-121 mariadbd[129747]: 2022-04-07 10:35:49 0 [Note] WSREP: view(view_id(PRIM,44508de1-af53,378) memb {

Apr  7 10:35:49 ip-172-31-39-121 mariadbd[129747]: #01144508de1-af53,0

Apr  7 10:35:49 ip-172-31-39-121 mariadbd[129747]: #0117c67946e-b045,0

Apr  7 10:35:49 ip-172-31-39-121 mariadbd[129747]: #01195267912-98ce,0

Apr  7 10:35:49 ip-172-31-39-121 mariadbd[129747]: #01198319f4a-8b85,0

Apr  7 10:35:49 ip-172-31-39-121 mariadbd[129747]: #011da9e7474-8746,0

Apr  7 10:35:49 ip-172-31-39-121 mariadbd[129747]: #011e4e23eea-bbd0,0

Apr  7 10:35:49 ip-172-31-39-121 mariadbd[129747]: #011eb5f53de-8588,0

Apr  7 10:35:49 ip-172-31-39-121 mariadbd[129747]: } joined {

Apr  7 10:35:49 ip-172-31-39-121 mariadbd[129747]: } left {

Apr  7 10:35:49 ip-172-31-39-121 mariadbd[129747]: } partitioned {

Apr  7 10:35:49 ip-172-31-39-121 mariadbd[129747]: })

Apr  7 10:35:49 ip-172-31-39-121 mariadbd[129747]: 2022-04-07 10:35:49 0 [Note] WSREP: save pc into disk

Apr  7 10:35:49 ip-172-31-39-121 mariadbd[129747]: 2022-04-07 10:35:49 0 [Note] WSREP: discarding pending addr without UUID: tcp://172.31.37.12:4567

Apr  7 10:35:49 ip-172-31-39-121 mariadbd[129747]: 2022-04-07 10:35:49 0 [Note] WSREP: discarding pending addr proto entry 0xaaab09eff5a0

Apr  7 10:35:49 ip-172-31-39-121 mariadbd[129747]: 2022-04-07 10:35:49 0 [Note] WSREP: gcomm: connected

Apr  7 10:35:49 ip-172-31-39-121 mariadbd[129747]: 2022-04-07 10:35:49 0 [Note] WSREP: Changing maximum packet size to 64500, resulting msg size: 32636

Apr  7 10:35:49 ip-172-31-39-121 mariadbd[129747]: 2022-04-07 10:35:49 0 [Note] WSREP: Shifting CLOSED -> OPEN (TO: 0)

Apr  7 10:35:49 ip-172-31-39-121 mariadbd[129747]: 2022-04-07 10:35:49 0 [Note] WSREP: Opened channel 'mariadb-cluster'

Apr  7 10:35:49 ip-172-31-39-121 mariadbd[129747]: 2022-04-07 10:35:49 0 [Note] WSREP: New COMPONENT: primary = yes, bootstrap = no, my_idx = 1, memb_num = 7

Apr  7 10:35:49 ip-172-31-39-121 mariadbd[129747]: 2022-04-07 10:35:49 0 [Note] WSREP: STATE EXCHANGE: Waiting for state UUID.

Apr  7 10:35:49 ip-172-31-39-121 mariadbd[129747]: 2022-04-07 10:35:49 0 [Note] WSREP: STATE EXCHANGE: sent state msg: 7dcd956b-b65e-11ec-8734-924b8afeca4d

Apr  7 10:35:49 ip-172-31-39-121 mariadbd[129747]: 2022-04-07 10:35:49 0 [Note] WSREP: STATE EXCHANGE: got state msg: 7dcd956b-b65e-11ec-8734-924b8afeca4d from 0 (ES5_AWS)

Apr  7 10:35:49 ip-172-31-39-121 mariadbd[129747]: 2022-04-07 10:35:49 0 [Note] WSREP: STATE EXCHANGE: got state msg: 7dcd956b-b65e-11ec-8734-924b8afeca4d from 2 (ES1_AWS)

Apr  7 10:35:49 ip-172-31-39-121 mariadbd[129747]: 2022-04-07 10:35:49 0 [Note] WSREP: STATE EXCHANGE: got state msg: 7dcd956b-b65e-11ec-8734-924b8afeca4d from 3 (ES2_AWS)

Apr  7 10:35:49 ip-172-31-39-121 mariadbd[129747]: 2022-04-07 10:35:49 0 [Note] WSREP: STATE EXCHANGE: got state msg: 7dcd956b-b65e-11ec-8734-924b8afeca4d from 4 (ES4_AWS)

Apr  7 10:35:49 ip-172-31-39-121 mariadbd[129747]: 2022-04-07 10:35:49 0 [Note] WSREP: STATE EXCHANGE: got state msg: 7dcd956b-b65e-11ec-8734-924b8afeca4d from 5 (ES3_AWS)

Apr  7 10:35:49 ip-172-31-39-121 mariadbd[129747]: 2022-04-07 10:35:49 1 [Note] WSREP: Starting rollbacker thread 1

Apr  7 10:35:49 ip-172-31-39-121 mariadbd[129747]: 2022-04-07 10:35:49 2 [Note] WSREP: Starting applier thread 2

Apr  7 10:35:49 ip-172-31-39-121 mariadbd[129747]: 2022-04-07 10:35:49 0 [Note] WSREP: STATE EXCHANGE: got state msg: 7dcd956b-b65e-11ec-8734-924b8afeca4d from 6 (BACKUP_AWS)

Apr  7 10:35:49 ip-172-31-39-121 mariadbd[129747]: 2022-04-07 10:35:49 0 [Note] WSREP: STATE EXCHANGE: got state msg: 7dcd956b-b65e-11ec-8734-924b8afeca4d from 1 (ES6_AWS)

Apr  7 10:35:49 ip-172-31-39-121 mariadbd[129747]: 2022-04-07 10:35:49 0 [Note] WSREP: Quorum results:

Apr  7 10:35:49 ip-172-31-39-121 mariadbd[129747]: #011version    = 6,

Apr  7 10:35:49 ip-172-31-39-121 mariadbd[129747]: #011component  = PRIMARY,

Apr  7 10:35:49 ip-172-31-39-121 mariadbd[129747]: #011conf_id    = 363,

Apr  7 10:35:49 ip-172-31-39-121 mariadbd[129747]: #011members    = 6/7 (joined/total),

Apr  7 10:35:49 ip-172-31-39-121 mariadbd[129747]: #011act_id     = 5064032831,

Apr  7 10:35:49 ip-172-31-39-121 mariadbd[129747]: #011last_appl. = 5064032726,

Apr  7 10:35:49 ip-172-31-39-121 mariadbd[129747]: #011protocols  = 2/10/4 (gcs/repl/appl),

Apr  7 10:35:49 ip-172-31-39-121 mariadbd[129747]: #011vote policy= 0,

Apr  7 10:35:49 ip-172-31-39-121 mariadbd[129747]: #011group UUID = 1d4206f9-647d-11ea-a963-0ed98924f90b

Apr  7 10:35:49 ip-172-31-39-121 mariadbd[129747]: 2022-04-07 10:35:49 0 [Note] WSREP: Flow-control interval: [42, 42]

Apr  7 10:35:49 ip-172-31-39-121 mariadbd[129747]: 2022-04-07 10:35:49 0 [Note] WSREP: Shifting OPEN -> PRIMARY (TO: 5064032832)

Apr  7 10:35:49 ip-172-31-39-121 mariadbd[129747]: 2022-04-07 10:35:49 2 [Note] WSREP: ####### processing CC 5064032832, local, ordered

Apr  7 10:35:49 ip-172-31-39-121 mariadbd[129747]: 2022-04-07 10:35:49 2 [Note] WSREP: Process first view: 1d4206f9-647d-11ea-a963-0ed98924f90b my uuid: 7c67946e-b65e-11ec-b045-eb2b92474255

Apr  7 10:35:49 ip-172-31-39-121 mariadbd[129747]: 2022-04-07 10:35:49 2 [Note] WSREP: Server ES6_AWS connected to cluster at position 1d4206f9-647d-11ea-a963-0ed98924f90b:5064032832 with ID 7c67946e-b65e-11ec-b045-eb2b92474255

Apr  7 10:35:49 ip-172-31-39-121 mariadbd[129747]: 2022-04-07 10:35:49 2 [Note] WSREP: Server status change disconnected -> connected

Apr  7 10:35:49 ip-172-31-39-121 mariadbd[129747]: 2022-04-07 10:35:49 2 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.

Apr  7 10:35:49 ip-172-31-39-121 mariadbd[129747]: 2022-04-07 10:35:49 2 [Note] WSREP: ####### My UUID: 7c67946e-b65e-11ec-b045-eb2b92474255

Apr  7 10:35:49 ip-172-31-39-121 mariadbd[129747]: 2022-04-07 10:35:49 2 [Note] WSREP: Cert index reset to 00000000-0000-0000-0000-000000000000:-1 (proto: 10), state transfer needed: yes

Apr  7 10:35:49 ip-172-31-39-121 mariadbd[129747]: 2022-04-07 10:35:49 0 [Note] WSREP: Service thread queue flushed.

Apr  7 10:35:49 ip-172-31-39-121 mariadbd[129747]: 2022-04-07 10:35:49 2 [Note] WSREP: ####### Assign initial position for certification: 00000000-0000-0000-0000-000000000000:-1, protocol version: -1

Apr  7 10:35:49 ip-172-31-39-121 mariadbd[129747]: 2022-04-07 10:35:49 2 [Note] WSREP: State transfer required:

Apr  7 10:35:49 ip-172-31-39-121 mariadbd[129747]: #011Group state: 1d4206f9-647d-11ea-a963-0ed98924f90b:5064032832

Apr  7 10:35:49 ip-172-31-39-121 mariadbd[129747]: #011Local state: 00000000-0000-0000-0000-000000000000:-1

Apr  7 10:35:49 ip-172-31-39-121 mariadbd[129747]: 2022-04-07 10:35:49 2 [Note] WSREP: Server status change connected -> joiner

Apr  7 10:35:49 ip-172-31-39-121 mariadbd[129747]: 2022-04-07 10:35:49 2 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.

Apr  7 10:35:49 ip-172-31-39-121 mariadbd[129747]: 2022-04-07 10:35:49 0 [Note] WSREP: Joiner monitor thread started to monitor

Apr  7 10:35:49 ip-172-31-39-121 mariadbd[129747]: 2022-04-07 10:35:49 0 [Note] WSREP: Running: 'wsrep_sst_rsync --role 'joiner' --address '172.31.39.121' --datadir '/var/lib/mysql/' --parent '129747' --mysqld-args --wsrep_start_position=00000000-0000-0000-0000-000000000000:-1'

Apr  7 10:35:49 ip-172-31-39-121 rsyncd[129866]: rsyncd version 3.1.3 starting, listening on port 4444

Apr  7 10:35:49 ip-172-31-39-121 mariadbd[129747]: 2022-04-07 10:35:49 2 [Note] WSREP: ####### IST uuid:00000000-0000-0000-0000-000000000000 f: 0, l: 5064032832, STRv: 3

Apr  7 10:35:49 ip-172-31-39-121 mariadbd[129747]: 2022-04-07 10:35:49 2 [Note] WSREP: IST receiver addr using tcp://172.31.39.121:4568

Apr  7 10:35:49 ip-172-31-39-121 mariadbd[129747]: 2022-04-07 10:35:49 2 [Note] WSREP: Prepared IST receiver for 0-5064032832, listening at: tcp://172.31.39.121:4568

Apr  7 10:35:49 ip-172-31-39-121 mariadbd[129747]: 2022-04-07 10:35:49 0 [Note] WSREP: Member 1.0 (ES6_AWS) requested state transfer from 'ES5_AWS'. Selected 0.0 (ES5_AWS)(SYNCED) as donor.

Apr  7 10:35:49 ip-172-31-39-121 mariadbd[129747]: 2022-04-07 10:35:49 0 [Note] WSREP: Shifting PRIMARY -> JOINER (TO: 5064032832)

Apr  7 10:35:49 ip-172-31-39-121 mariadbd[129747]: 2022-04-07 10:35:49 2 [Note] WSREP: Requesting state transfer: success, donor: 0

Apr  7 10:35:49 ip-172-31-39-121 mariadbd[129747]: 2022-04-07 10:35:49 2 [Note] WSREP: Resetting GCache seqno map due to different histories.

Apr  7 10:35:49 ip-172-31-39-121 mariadbd[129747]: 2022-04-07 10:35:49 2 [Note] WSREP: GCache history reset: 00000000-0000-0000-0000-000000000000:0 -> 1d4206f9-647d-11ea-a963-0ed98924f90b:5064032832

Apr  7 10:35:50 ip-172-31-39-121 rsyncd[129878]: connect from ip-172-31-37-75.eu-west-1.compute.internal (172.31.37.75)

Apr  7 10:35:50 ip-172-31-39-121 rsyncd[129878]: rsync to rsync_sst/ from ip-172-31-37-75.eu-west-1.compute.internal (172.31.37.75)

...

Apr  7 10:39:52 ip-172-31-39-121 mariadbd[129747]: 2022-04-07 10:39:52 0 [Note] WSREP: 0.0 (ES5_AWS): State transfer to 1.0 (ES6_AWS) complete.

Apr  7 10:39:52 ip-172-31-39-121 mariadbd[129757]: WSREP_SST: [INFO] Joiner cleanup: rsync PID=129866, stunnel PID=0 (20220407 10:39:52.306)

Apr  7 10:39:52 ip-172-31-39-121 mariadbd[129747]: 2022-04-07 10:39:52 0 [Note] WSREP: Member 0.0 (ES5_AWS) synced with group.

Apr  7 10:39:52 ip-172-31-39-121 rsyncd[129866]: sent 0 bytes  received 0 bytes  total size 0

Apr  7 10:39:52 ip-172-31-39-121 mariadbd[129757]: WSREP_SST: [INFO] Joiner cleanup done. (20220407 10:39:52.825)

Apr  7 10:39:52 ip-172-31-39-121 mariadbd[129747]: 2022-04-07 10:39:52 3 [Note] WSREP: SST received

Apr  7 10:39:52 ip-172-31-39-121 mariadbd[129747]: 2022-04-07 10:39:52 3 [Note] WSREP: Server status change joiner -> initializing

Apr  7 10:39:52 ip-172-31-39-121 mariadbd[129747]: 2022-04-07 10:39:52 3 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.

Apr  7 10:39:52 ip-172-31-39-121 mariadbd[129747]: 2022-04-07 10:39:52 0 [Note] InnoDB: Uses event mutexes

Apr  7 10:39:52 ip-172-31-39-121 mariadbd[129747]: 2022-04-07 10:39:52 0 [Note] InnoDB: Compressed tables use zlib 1.2.11

Apr  7 10:39:52 ip-172-31-39-121 mariadbd[129747]: 2022-04-07 10:39:52 0 [Note] InnoDB: Number of pools: 1

Apr  7 10:39:52 ip-172-31-39-121 mariadbd[129747]: 2022-04-07 10:39:52 0 [Note] InnoDB: Using ARMv8 crc32 + pmull instructions

Apr  7 10:39:52 ip-172-31-39-121 mariadbd[129747]: 2022-04-07 10:39:52 0 [Note] InnoDB: Using Linux native AIO

Apr  7 10:39:52 ip-172-31-39-121 mariadbd[129747]: 2022-04-07 10:39:52 0 [Note] InnoDB: Initializing buffer pool, total size = 134217728, chunk size = 134217728

Apr  7 10:39:52 ip-172-31-39-121 mariadbd[129747]: 2022-04-07 10:39:52 0 [Note] InnoDB: Completed initialization of buffer pool

Apr  7 10:39:52 ip-172-31-39-121 mariadbd[129747]: 2022-04-07 10:39:52 0 [ERROR] InnoDB: Upgrade after a crash is not supported. The redo log was created with MariaDB 10.4.22.

Apr  7 10:39:52 ip-172-31-39-121 mariadbd[129747]: 2022-04-07 10:39:52 0 [ERROR] InnoDB: Plugin initialization aborted with error Generic error

Apr  7 10:39:52 ip-172-31-39-121 mariadbd[129747]: 2022-04-07 10:39:52 0 [Note] InnoDB: Starting shutdown...

Apr  7 10:39:53 ip-172-31-39-121 mariadbd[129747]: 2022-04-07 10:39:53 0 [ERROR] Plugin 'InnoDB' init function returned error.

Apr  7 10:39:53 ip-172-31-39-121 mariadbd[129747]: 2022-04-07 10:39:53 0 [ERROR] Plugin 'InnoDB' registration as a STORAGE ENGINE failed.

Apr  7 10:39:53 ip-172-31-39-121 mariadbd[129747]: 2022-04-07 10:39:53 0 [Note] Plugin 'FEEDBACK' is disabled.

Apr  7 10:39:53 ip-172-31-39-121 mariadbd[129747]: 2022-04-07 10:39:53 0 [ERROR] Unknown/unsupported storage engine: innodb

Apr  7 10:39:53 ip-172-31-39-121 mariadbd[129747]: 2022-04-07 10:39:53 0 [ERROR] Aborting

Apr  7 10:39:53 ip-172-31-39-121 mariadbd[129747]: terminate called after throwing an instance of 'wsrep::runtime_error'

Apr  7 10:39:53 ip-172-31-39-121 mariadbd[129747]:   what():  State wait was interrupted

Apr  7 10:39:53 ip-172-31-39-121 mariadbd[129747]: 220407 10:39:53 [ERROR] mysqld got signal 6 ;

Apr  7 10:39:53 ip-172-31-39-121 mariadbd[129747]: This could be because you hit a bug. It is also possible that this binary

Apr  7 10:39:53 ip-172-31-39-121 mariadbd[129747]: or one of the libraries it was linked against is corrupt, improperly built,

Apr  7 10:39:53 ip-172-31-39-121 mariadbd[129747]: or misconfigured. This error can also be caused by malfunctioning hardware.

Apr  7 10:39:53 ip-172-31-39-121 mariadbd[129747]: To report this bug, see https://mariadb.com/kb/en/reporting-bugs

Apr  7 10:39:53 ip-172-31-39-121 mariadbd[129747]: We will try our best to scrape up some info that will hopefully help

Apr  7 10:39:53 ip-172-31-39-121 mariadbd[129747]: diagnose the problem, but since we have already crashed,

Apr  7 10:39:53 ip-172-31-39-121 mariadbd[129747]: something is definitely wrong and this may fail.

Apr  7 10:39:53 ip-172-31-39-121 mariadbd[129747]: Server version: 10.5.15-MariaDB-1:10.5.15+maria~focal

Apr  7 10:39:53 ip-172-31-39-121 mariadbd[129747]: key_buffer_size=134217728

Apr  7 10:39:53 ip-172-31-39-121 mariadbd[129747]: read_buffer_size=131072

Apr  7 10:39:53 ip-172-31-39-121 mariadbd[129747]: max_used_connections=0

Apr  7 10:39:53 ip-172-31-39-121 mariadbd[129747]: max_threads=153

Apr  7 10:39:53 ip-172-31-39-121 mariadbd[129747]: thread_count=4

Apr  7 10:39:53 ip-172-31-39-121 mariadbd[129747]: It is possible that mysqld could use up to

Apr  7 10:39:53 ip-172-31-39-121 mariadbd[129747]: key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 467879 K  bytes of memory

Apr  7 10:39:53 ip-172-31-39-121 mariadbd[129747]: Hope that's ok; if not, decrease some variables in the equation.

Apr  7 10:39:53 ip-172-31-39-121 mariadbd[129747]: Thread pointer: 0xfffe1c002068

Apr  7 10:39:53 ip-172-31-39-121 mariadbd[129747]: Attempting backtrace. You can use the following information to find out

Apr  7 10:39:53 ip-172-31-39-121 mariadbd[129747]: where mysqld died. If you see no messages after this, something went

Apr  7 10:39:53 ip-172-31-39-121 mariadbd[129747]: terribly wrong...

Apr  7 10:39:53 ip-172-31-39-121 mariadbd[129747]: stack_bottom = 0xfffe42579630 thread_stack 0x49000

Apr  7 10:39:53 ip-172-31-39-121 mariadbd[129747]: Printing to addr2line failed

Apr  7 10:39:53 ip-172-31-39-121 mariadbd[129747]: /usr/sbin/mariadbd(my_print_stacktrace+0x30)[0xaaaacc7023f0]

Apr  7 10:39:53 ip-172-31-39-121 mariadbd[129747]: /usr/sbin/mariadbd(handle_fatal_signal+0x45c)[0xaaaacc1b137c]

Apr  7 10:39:53 ip-172-31-39-121 mariadbd[129747]: linux-vdso.so.1(__kernel_rt_sigreturn+0x0)[0xffff89d95794]

Apr  7 10:39:53 ip-172-31-39-121 mariadbd[129747]: /lib/aarch64-linux-gnu/atomics/libc.so.6(gsignal+0xe0)[0xffff8942dc20]

Apr  7 10:39:53 ip-172-31-39-121 mariadbd[129747]: /lib/aarch64-linux-gnu/atomics/libc.so.6(abort+0xfc)[0xffff8941aa14]

Apr  7 10:39:53 ip-172-31-39-121 mariadbd[129747]: /lib/aarch64-linux-gnu/libstdc++.so.6(_ZN9__gnu_cxx27__verbose_terminate_handlerEv+0x19c)[0xffff896b88cc]

Apr  7 10:39:53 ip-172-31-39-121 mariadbd[129747]: /lib/aarch64-linux-gnu/libstdc++.so.6(+0x9f21c)[0xffff896b621c]

Apr  7 10:39:53 ip-172-31-39-121 mariadbd[129747]: /lib/aarch64-linux-gnu/libstdc++.so.6(+0x9f280)[0xffff896b6280]

Apr  7 10:39:53 ip-172-31-39-121 mariadbd[129747]: /lib/aarch64-linux-gnu/libstdc++.so.6(+0x9f574)[0xffff896b6574]

Apr  7 10:39:53 ip-172-31-39-121 mariadbd[129747]: /usr/sbin/mariadbd(_ZNK5wsrep12server_state16wait_until_stateERSt11unique_lockINS_5mutexEENS0_5stateE+0x1dc)[0xaaaacc7808ec]

Apr  7 10:39:53 ip-172-31-39-121 mariadbd[129747]: /usr/sbin/mariadbd(_ZN5wsrep12server_state12sst_receivedERNS_14client_serviceEi+0xb04)[0xaaaacc78a5f4]

Apr  7 10:39:53 ip-172-31-39-121 mariadbd[129747]: /usr/sbin/mariadbd(+0xb7b6bc)[0xaaaacc43b6bc]

Apr  7 10:39:53 ip-172-31-39-121 mariadbd[129747]: /usr/sbin/mariadbd(+0xb7c324)[0xaaaacc43c324]

Apr  7 10:39:53 ip-172-31-39-121 mariadbd[129747]: /usr/sbin/mariadbd(+0xb0bca0)[0xaaaacc3cbca0]

Apr  7 10:39:53 ip-172-31-39-121 mariadbd[129747]: /lib/aarch64-linux-gnu/atomics/libpthread.so.0(+0x73f0)[0xffff898173f0]

Apr  7 10:39:53 ip-172-31-39-121 mariadbd[129747]: /lib/aarch64-linux-gnu/atomics/libc.so.6(+0xd00dc)[0xffff894ca0dc]

Apr  7 10:39:53 ip-172-31-39-121 mariadbd[129747]: Trying to get some variables.

Apr  7 10:39:53 ip-172-31-39-121 mariadbd[129747]: Some pointers may be invalid and cause the dump to abort.

Apr  7 10:39:53 ip-172-31-39-121 mariadbd[129747]: Query (0x0): (null)

Apr  7 10:39:53 ip-172-31-39-121 mariadbd[129747]: Connection ID (thread ID): 3

Apr  7 10:39:53 ip-172-31-39-121 mariadbd[129747]: Status: NOT_KILLED

Apr  7 10:39:53 ip-172-31-39-121 mariadbd[129747]: Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=on,condition_pushdown_from_having=on,not_null_range_scan=off

Apr  7 10:39:53 ip-172-31-39-121 mariadbd[129747]: The manual page at https://mariadb.com/kb/en/how-to-produce-a-full-stack-trace-for-mysqld/ contains

Apr  7 10:39:53 ip-172-31-39-121 mariadbd[129747]: information that should help you find out what is causing the crash.

Apr  7 10:39:53 ip-172-31-39-121 mariadbd[129747]: We think the query pointer is invalid, but we will try to print it anyway.

Apr  7 10:39:53 ip-172-31-39-121 mariadbd[129747]: Query:

Apr  7 10:39:53 ip-172-31-39-121 mariadbd[129747]: Writing a core file...

Apr  7 10:39:53 ip-172-31-39-121 mariadbd[129747]: Working directory at /media/storage/mysql

Apr  7 10:39:53 ip-172-31-39-121 mariadbd[129747]: Resource Limits:

Apr  7 10:39:53 ip-172-31-39-121 mariadbd[129747]: Limit                     Soft Limit           Hard Limit           Units

Apr  7 10:39:53 ip-172-31-39-121 mariadbd[129747]: Max cpu time              unlimited            unlimited            seconds

Apr  7 10:39:53 ip-172-31-39-121 mariadbd[129747]: Max file size             unlimited            unlimited            bytes

Apr  7 10:39:53 ip-172-31-39-121 mariadbd[129747]: Max data size             unlimited            unlimited            bytes

Apr  7 10:39:53 ip-172-31-39-121 mariadbd[129747]: Max stack size            8388608              unlimited            bytes

Apr  7 10:39:53 ip-172-31-39-121 mariadbd[129747]: Max core file size        0                    unlimited            bytes

Apr  7 10:39:53 ip-172-31-39-121 mariadbd[129747]: Max resident set          unlimited            unlimited            bytes

Apr  7 10:39:53 ip-172-31-39-121 mariadbd[129747]: Max processes             125387               125387               processes

Apr  7 10:39:53 ip-172-31-39-121 mariadbd[129747]: Max open files            32768                32768                files

Apr  7 10:39:53 ip-172-31-39-121 mariadbd[129747]: Max locked memory         65536                65536                bytes

Apr  7 10:39:53 ip-172-31-39-121 mariadbd[129747]: Max address space         unlimited            unlimited            bytes

Apr  7 10:39:53 ip-172-31-39-121 mariadbd[129747]: Max file locks            unlimited            unlimited            locks

Apr  7 10:39:53 ip-172-31-39-121 mariadbd[129747]: Max pending signals       125387               125387               signals

Apr  7 10:39:53 ip-172-31-39-121 mariadbd[129747]: Max msgqueue size         819200               819200               bytes

Apr  7 10:39:53 ip-172-31-39-121 mariadbd[129747]: Max nice priority         0                    0

Apr  7 10:39:53 ip-172-31-39-121 mariadbd[129747]: Max realtime priority     0                    0

Apr  7 10:39:53 ip-172-31-39-121 mariadbd[129747]: Max realtime timeout      unlimited            unlimited            us

Apr  7 10:39:53 ip-172-31-39-121 mariadbd[129747]: Core pattern: |/usr/share/apport/apport %p %s %c %d %P %E

Apr  7 10:39:53 ip-172-31-39-121 systemd[1]: mariadb.service: Main process exited, code=dumped, status=6/ABRT

Apr  7 10:39:53 ip-172-31-39-121 systemd[1]: mariadb.service: Failed with result 'core-dump'.

Apr  7 10:39:53 ip-172-31-39-121 systemd[1]: Failed to start MariaDB 10.5.15 database server.

DONOR Logs

Apr  7 11:35:47 es5 mysqld[29230]: 2022-04-07 11:35:47 0 [Note] WSREP: (44508de1-af53, 'tcp://0.0.0.0:4567') turning message relay requesting on, nonlive peers: tcp://172.31.39.121:4567

Apr  7 11:35:47 es5 mysqld[29230]: 2022-04-07 11:35:47 0 [Note] WSREP: (44508de1-af53, 'tcp://0.0.0.0:4567') connection established to 7c67946e-b045 tcp://172.31.39.121:4567

Apr  7 11:35:48 es5 mysqld[29230]: 2022-04-07 11:35:48 0 [Note] WSREP: declaring 7c67946e-b045 at tcp://172.31.39.121:4567 stable

Apr  7 11:35:48 es5 mysqld[29230]: 2022-04-07 11:35:48 0 [Note] WSREP: declaring 95267912-98ce at tcp://172.31.34.27:4567 stable

Apr  7 11:35:48 es5 mysqld[29230]: 2022-04-07 11:35:48 0 [Note] WSREP: declaring 98319f4a-8b85 at tcp://172.31.35.130:4567 stable

Apr  7 11:35:48 es5 mysqld[29230]: 2022-04-07 11:35:48 0 [Note] WSREP: declaring da9e7474-8746 at tcp://172.31.40.66:4567 stable

Apr  7 11:35:48 es5 mysqld[29230]: 2022-04-07 11:35:48 0 [Note] WSREP: declaring e4e23eea-bbd0 at tcp://172.31.37.249:4567 stable

Apr  7 11:35:48 es5 mysqld[29230]: 2022-04-07 11:35:48 0 [Note] WSREP: declaring eb5f53de-8588 at tcp://172.31.9.176:4567 stable

Apr  7 11:35:49 es5 mysqld[29230]: 2022-04-07 11:35:49 0 [Note] WSREP: Node 44508de1-af53 state prim

Apr  7 11:35:49 es5 mysqld[29230]: 2022-04-07 11:35:49 0 [Note] WSREP: view(view_id(PRIM,44508de1-af53,378) memb {

Apr  7 11:35:49 es5 mysqld[29230]: #01144508de1-af53,0

Apr  7 11:35:49 es5 mysqld[29230]: #0117c67946e-b045,0

Apr  7 11:35:49 es5 mysqld[29230]: #01195267912-98ce,0

Apr  7 11:35:49 es5 mysqld[29230]: #01198319f4a-8b85,0

Apr  7 11:35:49 es5 mysqld[29230]: #011da9e7474-8746,0

Apr  7 11:35:49 es5 mysqld[29230]: #011e4e23eea-bbd0,0

Apr  7 11:35:49 es5 mysqld[29230]: #011eb5f53de-8588,0

Apr  7 11:35:49 es5 mysqld[29230]: } joined {

Apr  7 11:35:49 es5 mysqld[29230]: } left {

Apr  7 11:35:49 es5 mysqld[29230]: } partitioned {

Apr  7 11:35:49 es5 mysqld[29230]: })

Apr  7 11:35:49 es5 mysqld[29230]: 2022-04-07 11:35:49 0 [Note] WSREP: save pc into disk

Apr  7 11:35:49 es5 mysqld[29230]: 2022-04-07 11:35:49 0 [Note] WSREP: New COMPONENT: primary = yes, bootstrap = no, my_idx = 0, memb_num = 7

Apr  7 11:35:49 es5 mysqld[29230]: 2022-04-07 11:35:49 0 [Note] WSREP: STATE_EXCHANGE: sent state UUID: 7dcd956b-b65e-11ec-8734-924b8afeca4d

Apr  7 11:35:49 es5 mysqld[29230]: 2022-04-07 11:35:49 0 [Note] WSREP: STATE EXCHANGE: sent state msg: 7dcd956b-b65e-11ec-8734-924b8afeca4d

Apr  7 11:35:49 es5 mysqld[29230]: 2022-04-07 11:35:49 0 [Note] WSREP: STATE EXCHANGE: got state msg: 7dcd956b-b65e-11ec-8734-924b8afeca4d from 0 (ES5_AWS)

Apr  7 11:35:49 es5 mysqld[29230]: 2022-04-07 11:35:49 0 [Note] WSREP: STATE EXCHANGE: got state msg: 7dcd956b-b65e-11ec-8734-924b8afeca4d from 2 (ES1_AWS)

Apr  7 11:35:49 es5 mysqld[29230]: 2022-04-07 11:35:49 0 [Note] WSREP: STATE EXCHANGE: got state msg: 7dcd956b-b65e-11ec-8734-924b8afeca4d from 3 (ES2_AWS)

Apr  7 11:35:49 es5 mysqld[29230]: 2022-04-07 11:35:49 0 [Note] WSREP: STATE EXCHANGE: got state msg: 7dcd956b-b65e-11ec-8734-924b8afeca4d from 4 (ES4_AWS)

Apr  7 11:35:49 es5 mysqld[29230]: 2022-04-07 11:35:49 0 [Note] WSREP: STATE EXCHANGE: got state msg: 7dcd956b-b65e-11ec-8734-924b8afeca4d from 5 (ES3_AWS)

Apr  7 11:35:49 es5 mysqld[29230]: 2022-04-07 11:35:49 0 [Note] WSREP: STATE EXCHANGE: got state msg: 7dcd956b-b65e-11ec-8734-924b8afeca4d from 6 (BACKUP_AWS)

Apr  7 11:35:49 es5 mysqld[29230]: 2022-04-07 11:35:49 0 [Note] WSREP: STATE EXCHANGE: got state msg: 7dcd956b-b65e-11ec-8734-924b8afeca4d from 1 (ES6_AWS)

Apr  7 11:35:49 es5 mysqld[29230]: 2022-04-07 11:35:49 0 [Note] WSREP: Quorum results:

Apr  7 11:35:49 es5 mysqld[29230]: #011version    = 6,

Apr  7 11:35:49 es5 mysqld[29230]: #011component  = PRIMARY,

Apr  7 11:35:49 es5 mysqld[29230]: #011conf_id    = 363,

Apr  7 11:35:49 es5 mysqld[29230]: #011members    = 6/7 (joined/total),

Apr  7 11:35:49 es5 mysqld[29230]: #011act_id     = 5064032831,

Apr  7 11:35:49 es5 mysqld[29230]: #011last_appl. = 5064032726,

Apr  7 11:35:49 es5 mysqld[29230]: #011protocols  = 2/10/4 (gcs/repl/appl),

Apr  7 11:35:49 es5 mysqld[29230]: #011vote policy= 0,

Apr  7 11:35:49 es5 mysqld[29230]: #011group UUID = 1d4206f9-647d-11ea-a963-0ed98924f90b

Apr  7 11:35:49 es5 mysqld[29230]: 2022-04-07 11:35:49 0 [Note] WSREP: Flow-control interval: [42, 42]

Apr  7 11:35:49 es5 mysqld[29230]: 2022-04-07 11:35:49 2 [Note] WSREP: ####### processing CC 5064032832, local, ordered

Apr  7 11:35:49 es5 mysqld[29230]: 2022-04-07 11:35:49 2 [Note] WSREP: ####### My UUID: 44508de1-b657-11ec-af53-dbbee42dcffe

Apr  7 11:35:49 es5 mysqld[29230]: 2022-04-07 11:35:49 2 [Note] WSREP: Skipping cert index reset

Apr  7 11:35:49 es5 mysqld[29230]: 2022-04-07 11:35:49 2 [Note] WSREP: REPL Protocols: 10 (5)

Apr  7 11:35:49 es5 mysqld[29230]: 2022-04-07 11:35:49 2 [Note] WSREP: ####### Adjusting cert position: 5064032831 -> 5064032832

Apr  7 11:35:49 es5 mysqld[29230]: 2022-04-07 11:35:49 0 [Note] WSREP: Service thread queue flushed.

Apr  7 11:35:49 es5 mysqld[29230]: 2022-04-07 11:35:49 2 [Note] WSREP: ================================================

Apr  7 11:35:49 es5 mysqld[29230]: View:

Apr  7 11:35:49 es5 mysqld[29230]:   id: 1d4206f9-647d-11ea-a963-0ed98924f90b:5064032832

Apr  7 11:35:49 es5 mysqld[29230]:   status: primary

Apr  7 11:35:49 es5 mysqld[29230]:   protocol_version: 4

Apr  7 11:35:49 es5 mysqld[29230]:   capabilities: MULTI-MASTER, CERTIFICATION, PARALLEL_APPLYING, REPLAY, ISOLATION, PAUSE, CAUSAL_READ, INCREMENTAL_WS, UNORDERED, PREORDERED, STREAMING, NBO

Apr  7 11:35:49 es5 mysqld[29230]:   final: no

Apr  7 11:35:49 es5 mysqld[29230]:   own_index: 0

Apr  7 11:35:49 es5 mysqld[29230]:   members(7):

Apr  7 11:35:49 es5 mysqld[29230]: #0110: 44508de1-b657-11ec-af53-dbbee42dcffe, ES5_AWS

Apr  7 11:35:49 es5 mysqld[29230]: #0111: 7c67946e-b65e-11ec-b045-eb2b92474255, ES6_AWS

Apr  7 11:35:49 es5 mysqld[29230]: #0112: 95267912-5df1-11ec-98ce-a66d469d045c, ES1_AWS

Apr  7 11:35:49 es5 mysqld[29230]: #0113: 98319f4a-b5f9-11ec-8b85-e69cdab5035e, ES2_AWS

Apr  7 11:35:49 es5 mysqld[29230]: #0114: da9e7474-5df2-11ec-8746-dfd73835a1c5, ES4_AWS

Apr  7 11:35:49 es5 mysqld[29230]: #0115: e4e23eea-5df0-11ec-bbd0-8b132f61eb3d, ES3_AWS

Apr  7 11:35:49 es5 mysqld[29230]: #0116: eb5f53de-882b-11ec-8588-1a528d6537ef, BACKUP_AWS

Apr  7 11:35:49 es5 mysqld[29230]: =================================================

Apr  7 11:35:49 es5 mysqld[29230]: 2022-04-07 11:35:49 2 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.

Apr  7 11:35:49 es5 mysqld[29230]: 2022-04-07 11:35:49 2 [Note] WSREP: Lowest cert index boundary for CC from group: 5064032727

Apr  7 11:35:49 es5 mysqld[29230]: 2022-04-07 11:35:49 2 [Note] WSREP: Min available from gcache for CC from group: 5063284333

Apr  7 11:35:49 es5 mysqld[29230]: 2022-04-07 11:35:49 0 [Note] WSREP: Member 1.0 (ES6_AWS) requested state transfer from 'ES5_AWS'. Selected 0.0 (ES5_AWS)(SYNCED) as donor.

Apr  7 11:35:49 es5 mysqld[29230]: 2022-04-07 11:35:49 0 [Note] WSREP: Shifting SYNCED -> DONOR/DESYNCED (TO: 5064032832)

Apr  7 11:35:49 es5 mysqld[29230]: 2022-04-07 11:35:49 2 [Note] WSREP: Detected STR version: 1, req_len: 125, req: STRv1

Apr  7 11:35:49 es5 mysqld[29230]: 2022-04-07 11:35:49 2 [Note] WSREP: Cert index preload: 5064032727 -> 5064032832

Apr  7 11:35:49 es5 mysqld[29230]: 2022-04-07 11:35:49 2 [Note] WSREP: Server status change synced -> donor

Apr  7 11:35:49 es5 mysqld[29230]: 2022-04-07 11:35:49 0 [Note] WSREP: async IST sender starting to serve tcp://172.31.39.121:4568 sending 5064032727-5064032832, preload starts from 5064032727

Apr  7 11:35:49 es5 mysqld[29230]: 2022-04-07 11:35:49 2 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.

Apr  7 11:35:49 es5 mysqld[29230]: 2022-04-07 11:35:49 2 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.

Apr  7 11:35:49 es5 mysqld[29230]: 2022-04-07 11:35:49 0 [Note] WSREP: Donor monitor thread started to monitor

Apr  7 11:35:49 es5 mysqld[29230]: 2022-04-07 11:35:49 0 [Note] WSREP: Running: 'wsrep_sst_rsync --role 'donor' --address '172.31.39.121:4444/rsync_sst' --local-port '3306' --socket '/var/run/mysqld/mysqld.sock' --datadir '/media/storage/mysql/' --gtid '1d4206f9-647d-11ea-a963-0ed98924f90b:5064032832' --gtid-domain-id '0' --mysqld-args --wsrep_start_position=1d4206f9-647d-11ea-a963-0ed98924f90b:5063284262'

Apr  7 11:35:49 es5 mysqld[29230]: 2022-04-07 11:35:49 0 [Note] WSREP: IST sender 5064032727 -> 5064032832

Apr  7 11:35:49 es5 mysqld[29230]: 2022-04-07 11:35:49 2 [Note] WSREP: sst_donor_thread signaled with 0

Apr  7 11:35:49 es5 mysqld[29230]: 2022-04-07 11:35:49 0 [Note] WSREP: Flushing tables for SST...

Apr  7 11:35:49 es5 mysqld[29230]: 2022-04-07 11:35:49 0 [Note] WSREP: pause

Apr  7 11:35:49 es5 mysqld[29230]: 2022-04-07 11:35:49 0 [Note] WSREP: Provider paused at 1d4206f9-647d-11ea-a963-0ed98924f90b:5064032832 (778520)

Apr  7 11:35:49 es5 mysqld[29230]: 2022-04-07 11:35:49 0 [Note] WSREP: Server paused at: 5064032832

Apr  7 11:35:49 es5 mysqld[29230]: 2022-04-07 11:35:49 0 [Note] WSREP: Tables flushed.

Apr  7 11:35:50 es5 mysqld[29230]: 2022-04-07 11:35:50 0 [Note] WSREP: (44508de1-af53, 'tcp://0.0.0.0:4567') turning message relay requesting off

Apr  7 11:39:01 es5 CRON[10599]: (root) CMD (  [ -x /usr/lib/php/sessionclean ] && if [ ! -d /run/systemd/system ]; then /usr/lib/php/sessionclean; fi)

Apr  7 11:39:05 es5 systemd[1]: Starting Clean php session files...

Apr  7 11:39:06 es5 systemd[1]: Started Clean php session files.

Apr  7 11:39:51 es5 mysqld[29230]: 2022-04-07 11:39:51 0 [Warning] InnoDB: A long semaphore wait:

Apr  7 11:39:51 es5 mysqld[29230]: --Thread 139702254601984 has waited at log0log.cc line 882 for 241.00 seconds the semaphore:

Apr  7 11:39:51 es5 mysqld[29230]: Mutex at 0x55fd5cf32080, Mutex LOG_WRITE created log0log.cc:537, lock var 2

Apr  7 11:39:51 es5 mysqld[29230]: 2022-04-07 11:39:51 0 [Note] InnoDB: A semaphore wait:

Apr  7 11:39:51 es5 mysqld[29230]: --Thread 139702254601984 has waited at log0log.cc line 882 for 241.00 seconds the semaphore:

Apr  7 11:39:51 es5 mysqld[29230]: Mutex at 0x55fd5cf32080, Mutex LOG_WRITE created log0log.cc:537, lock var 2

Apr  7 11:39:51 es5 mysqld[29230]: InnoDB: ###### Starts InnoDB Monitor for 30 secs to print diagnostic info:

Apr  7 11:39:51 es5 mysqld[29230]: InnoDB: Pending reads 0, writes 0

Apr  7 11:39:52 es5 mysqld[29230]: 2022-04-07 11:39:52 0 [Note] WSREP: resume

Apr  7 11:39:52 es5 mysqld[29230]: 2022-04-07 11:39:52 0 [Note] WSREP: resuming provider at 778520

Apr  7 11:39:52 es5 mysqld[29230]: 2022-04-07 11:39:52 0 [Note] WSREP: Provider resumed.

Apr  7 11:39:52 es5 mysqld[29230]: 2022-04-07 11:39:52 0 [Note] WSREP: SST sent: 1d4206f9-647d-11ea-a963-0ed98924f90b:5064032832

Apr  7 11:39:52 es5 mysqld[29230]: 2022-04-07 11:39:52 0 [Note] WSREP: Server status change donor -> joined

Apr  7 11:39:52 es5 mysqld[29230]: 2022-04-07 11:39:52 0 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.

Apr  7 11:39:52 es5 mysqld[29230]: 2022-04-07 11:39:52 0 [Note] WSREP: Donor monitor thread ended with total time 243 sec

Apr  7 11:39:52 es5 mysqld[29230]: 2022-04-07 11:39:52 0 [Note] WSREP: 0.0 (ES5_AWS): State transfer to 1.0 (ES6_AWS) complete.

Apr  7 11:39:52 es5 mysqld[29230]: 2022-04-07 11:39:52 0 [Note] WSREP: Shifting DONOR/DESYNCED -> JOINED (TO: 5064034794)

Apr  7 11:39:52 es5 mysqld[29230]: 2022-04-07 11:39:52 0 [Note] WSREP: Member 0.0 (ES5_AWS) synced with group.

Apr  7 11:39:52 es5 mysqld[29230]: 2022-04-07 11:39:52 0 [Note] WSREP: Shifting JOINED -> SYNCED (TO: 5064034794)

Apr  7 11:39:52 es5 mysqld[29230]: 2022-04-07 11:39:52 2 [Note] WSREP: Server ES5_AWS synced with group

Apr  7 11:39:52 es5 mysqld[29230]: 2022-04-07 11:39:52 2 [Note] WSREP: Server status change joined -> synced

Apr  7 11:39:52 es5 mysqld[29230]: 2022-04-07 11:39:52 2 [Note] WSREP: Synchronized with group, ready for connections

Apr  7 11:39:52 es5 mysqld[29230]: 2022-04-07 11:39:52 2 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.

Apr  7 11:39:53 es5 mysqld[29230]: 2022-04-07 11:39:53 0 [Note] WSREP: (44508de1-af53, 'tcp://0.0.0.0:4567') turning message relay requesting on, nonlive peers: tcp://172.31.39.121:4567

Apr  7 11:39:53 es5 mysqld[29230]: 2022-04-07 11:39:53 0 [Note] WSREP: declaring 95267912-98ce at tcp://172.31.34.27:4567 stable

Apr  7 11:39:53 es5 mysqld[29230]: 2022-04-07 11:39:53 0 [Note] WSREP: declaring 98319f4a-8b85 at tcp://172.31.35.130:4567 stable

Apr  7 11:39:53 es5 mysqld[29230]: 2022-04-07 11:39:53 0 [Note] WSREP: declaring da9e7474-8746 at tcp://172.31.40.66:4567 stable

Apr  7 11:39:53 es5 mysqld[29230]: 2022-04-07 11:39:53 0 [Note] WSREP: declaring e4e23eea-bbd0 at tcp://172.31.37.249:4567 stable

Apr  7 11:39:53 es5 mysqld[29230]: 2022-04-07 11:39:53 0 [Note] WSREP: declaring eb5f53de-8588 at tcp://172.31.9.176:4567 stable

Apr  7 11:39:53 es5 mysqld[29230]: 2022-04-07 11:39:53 0 [Note] WSREP: forgetting 7c67946e-b045 (tcp://172.31.39.121:4567)

Apr  7 11:39:53 es5 mysqld[29230]: 2022-04-07 11:39:53 0 [Note] WSREP: (44508de1-af53, 'tcp://0.0.0.0:4567') turning message relay requesting off

Apr  7 11:39:53 es5 mysqld[29230]: 2022-04-07 11:39:53 0 [Note] WSREP: Node 44508de1-af53 state prim

Apr  7 11:39:53 es5 mysqld[29230]: 2022-04-07 11:39:53 0 [Note] WSREP: view(view_id(PRIM,44508de1-af53,379) memb {

Apr  7 11:39:53 es5 mysqld[29230]: #01144508de1-af53,0

Apr  7 11:39:53 es5 mysqld[29230]: #01195267912-98ce,0

Apr  7 11:39:53 es5 mysqld[29230]: #01198319f4a-8b85,0

Apr  7 11:39:53 es5 mysqld[29230]: #011da9e7474-8746,0

Apr  7 11:39:53 es5 mysqld[29230]: #011e4e23eea-bbd0,0

Apr  7 11:39:53 es5 mysqld[29230]: #011eb5f53de-8588,0

Apr  7 11:39:53 es5 mysqld[29230]: } joined {

Apr  7 11:39:53 es5 mysqld[29230]: } left {

Apr  7 11:39:53 es5 mysqld[29230]: } partitioned {

Apr  7 11:39:53 es5 mysqld[29230]: #0117c67946e-b045,0

Apr  7 11:39:53 es5 mysqld[29230]: })

Apr  7 11:39:53 es5 mysqld[29230]: 2022-04-07 11:39:53 0 [Note] WSREP: save pc into disk

Apr  7 11:39:53 es5 mysqld[29230]: 2022-04-07 11:39:53 0 [Note] WSREP: forgetting 7c67946e-b045 (tcp://172.31.39.121:4567)

Apr  7 11:39:53 es5 mysqld[29230]: 2022-04-07 11:39:53 0 [Note] WSREP: New COMPONENT: primary = yes, bootstrap = no, my_idx = 0, memb_num = 6

Apr  7 11:39:53 es5 mysqld[29230]: 2022-04-07 11:39:53 0 [Note] WSREP: STATE_EXCHANGE: sent state UUID: 0f078c2c-b65f-11ec-b06d-de758fe27da8

Apr  7 11:39:53 es5 mysqld[29230]: 2022-04-07 11:39:53 0 [Note] WSREP: STATE EXCHANGE: sent state msg: 0f078c2c-b65f-11ec-b06d-de758fe27da8

Apr  7 11:39:53 es5 mysqld[29230]: 2022-04-07 11:39:53 0 [Note] WSREP: STATE EXCHANGE: got state msg: 0f078c2c-b65f-11ec-b06d-de758fe27da8 from 0 (ES5_AWS)

Apr  7 11:39:53 es5 mysqld[29230]: 2022-04-07 11:39:53 0 [Note] WSREP: STATE EXCHANGE: got state msg: 0f078c2c-b65f-11ec-b06d-de758fe27da8 from 1 (ES1_AWS)

Apr  7 11:39:53 es5 mysqld[29230]: 2022-04-07 11:39:53 0 [Note] WSREP: STATE EXCHANGE: got state msg: 0f078c2c-b65f-11ec-b06d-de758fe27da8 from 2 (ES2_AWS)

Apr  7 11:39:53 es5 mysqld[29230]: 2022-04-07 11:39:53 0 [Note] WSREP: STATE EXCHANGE: got state msg: 0f078c2c-b65f-11ec-b06d-de758fe27da8 from 3 (ES4_AWS)

Apr  7 11:39:53 es5 mysqld[29230]: 2022-04-07 11:39:53 0 [Note] WSREP: STATE EXCHANGE: got state msg: 0f078c2c-b65f-11ec-b06d-de758fe27da8 from 4 (ES3_AWS)

Apr  7 11:39:53 es5 mysqld[29230]: 2022-04-07 11:39:53 0 [Note] WSREP: STATE EXCHANGE: got state msg: 0f078c2c-b65f-11ec-b06d-de758fe27da8 from 5 (BACKUP_AWS)

Apr  7 11:39:53 es5 mysqld[29230]: 2022-04-07 11:39:53 0 [Note] WSREP: Quorum results:

Apr  7 11:39:53 es5 mysqld[29230]: #011version    = 6,

Apr  7 11:39:53 es5 mysqld[29230]: #011component  = PRIMARY,

Apr  7 11:39:53 es5 mysqld[29230]: #011conf_id    = 364,

Apr  7 11:39:53 es5 mysqld[29230]: #011members    = 6/6 (joined/total),

Apr  7 11:39:53 es5 mysqld[29230]: #011act_id     = 5064034794,

Apr  7 11:39:53 es5 mysqld[29230]: #011last_appl. = 5064034689,

Apr  7 11:39:53 es5 mysqld[29230]: #011protocols  = 2/10/4 (gcs/repl/appl),

Apr  7 11:39:53 es5 mysqld[29230]: #011vote policy= 0,

Apr  7 11:39:53 es5 mysqld[29230]: #011group UUID = 1d4206f9-647d-11ea-a963-0ed98924f90b

Apr  7 11:39:53 es5 mysqld[29230]: 2022-04-07 11:39:53 0 [Note] WSREP: Flow-control interval: [39, 39]

Apr  7 11:39:53 es5 mysqld[29230]: 2022-04-07 11:39:53 2 [Note] WSREP: ####### processing CC 5064034795, local, ordered

Apr  7 11:39:53 es5 mysqld[29230]: 2022-04-07 11:39:53 2 [Note] WSREP: ####### My UUID: 44508de1-b657-11ec-af53-dbbee42dcffe

Apr  7 11:39:53 es5 mysqld[29230]: 2022-04-07 11:39:53 2 [Note] WSREP: Skipping cert index reset

Apr  7 11:39:53 es5 mysqld[29230]: 2022-04-07 11:39:53 2 [Note] WSREP: REPL Protocols: 10 (5)

Apr  7 11:39:53 es5 mysqld[29230]: 2022-04-07 11:39:53 2 [Note] WSREP: ####### Adjusting cert position: 5064034794 -> 5064034795

Apr  7 11:39:53 es5 mysqld[29230]: 2022-04-07 11:39:53 0 [Note] WSREP: Service thread queue flushed.

Apr  7 11:39:53 es5 mysqld[29230]: 2022-04-07 11:39:53 2 [Note] WSREP: ================================================

Apr  7 11:39:53 es5 mysqld[29230]: View:

Apr  7 11:39:53 es5 mysqld[29230]:   id: 1d4206f9-647d-11ea-a963-0ed98924f90b:5064034795

Apr  7 11:39:53 es5 mysqld[29230]:   status: primary

Apr  7 11:39:53 es5 mysqld[29230]:   protocol_version: 4

Apr  7 11:39:53 es5 mysqld[29230]:   capabilities: MULTI-MASTER, CERTIFICATION, PARALLEL_APPLYING, REPLAY, ISOLATION, PAUSE, CAUSAL_READ, INCREMENTAL_WS, UNORDERED, PREORDERED, STREAMING, NBO

Apr  7 11:39:53 es5 mysqld[29230]:   final: no

Apr  7 11:39:53 es5 mysqld[29230]:   own_index: 0

Apr  7 11:39:53 es5 mysqld[29230]:   members(6):

Apr  7 11:39:53 es5 mysqld[29230]: #0110: 44508de1-b657-11ec-af53-dbbee42dcffe, ES5_AWS

Apr  7 11:39:53 es5 mysqld[29230]: #0111: 95267912-5df1-11ec-98ce-a66d469d045c, ES1_AWS

Apr  7 11:39:53 es5 mysqld[29230]: #0112: 98319f4a-b5f9-11ec-8b85-e69cdab5035e, ES2_AWS

Apr  7 11:39:53 es5 mysqld[29230]: #0113: da9e7474-5df2-11ec-8746-dfd73835a1c5, ES4_AWS

Apr  7 11:39:53 es5 mysqld[29230]: #0114: e4e23eea-5df0-11ec-bbd0-8b132f61eb3d, ES3_AWS

Apr  7 11:39:53 es5 mysqld[29230]: #0115: eb5f53de-882b-11ec-8588-1a528d6537ef, BACKUP_AWS

Apr  7 11:39:53 es5 mysqld[29230]: =================================================

Apr  7 11:39:53 es5 mysqld[29230]: 2022-04-07 11:39:53 2 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.

Apr  7 11:39:53 es5 mysqld[29230]: 2022-04-07 11:39:53 2 [Note] WSREP: Lowest cert index boundary for CC from group: 5064032727

Apr  7 11:39:53 es5 mysqld[29230]: 2022-04-07 11:39:53 2 [Note] WSREP: Min available from gcache for CC from group: 5063284333

Apr  7 11:39:53 es5 mysqld[29230]: 2022-04-07 11:39:53 0 [Note] WSREP: async IST sender served

Apr  7 11:39:56 es5 mysqld[29230]: 2022-04-07 11:39:56 0 [Note] WSREP: (44508de1-af53, 'tcp://0.0.0.0:4567') connection to peer eb5f53de-8588 with addr tcp://172.31.9.176:4567 timed out, no messages seen in PT3S, socket stats: rtt: 872 rttvar: 864 rto: 204000 lost: 0 last_data_recv: 3104 cwnd: 10 last_queued_since: 33142 last_delivered_since: 3103372544 send_queue_length: 1 send_queue_bytes: 804 segment: 0 messages: 1

Apr  7 11:39:56 es5 mysqld[29230]: 2022-04-07 11:39:56 0 [Note] WSREP: Deferred close timer started for socket with remote endpoint: tcp://172.31.9.176:60056

Apr  7 11:39:56 es5 mysqld[29230]: 2022-04-07 11:39:56 0 [Note] WSREP: (44508de1-af53, 'tcp://0.0.0.0:4567') turning message relay requesting on, nonlive peers: tcp://172.31.9.176:4567

Apr  7 11:39:56 es5 mysqld[29230]: 2022-04-07 11:39:56 0 [Note] WSREP: Deferred close timer handle_wait Operation aborted. for 0x55fd61a9f5b0

Apr  7 11:39:56 es5 mysqld[29230]: 2022-04-07 11:39:56 0 [Note] WSREP: Deferred close timer destruct

[MDEV-28256] Galera SST fails with 'InnoDB: Upgrade after a crash is not supported' Created: 2022-04-07 Updated: 2022-06-26 Resolved: 2022-06-26
Status:	Closed
Project:	MariaDB Server
Component/s:	Galera SST
Affects Version/s:	10.5.15
Fix Version/s:	N/A