Details
-
Bug
-
Status: Closed (View Workflow)
-
Major
-
Resolution: Incomplete
-
10.5.15
-
Ubuntu 20.04 LTS
Description
Hi, I'm trying to join a new 10.5 node to a 10.4 Galera cluster as part of a migration to new AWS instance types.
The symptom is that the initial SST completes, but on the restart the joining node then fails with this error:
Apr 7 10:39:52 ip-172-31-39-121 mariadbd[129747]: 2022-04-07 10:39:52 0 [ERROR] InnoDB: Upgrade after a crash is not supported. The redo log was created with MariaDB 10.4.22.
|
After various Googling this seems to be related to crash recovery files ib_logfile0 and ib_logfile1, which don't exist on the empty JOINING node until the SST completes. The DONOR node was cleanly restarted prior to the sync. I'm not sure if this is a known issue?
Things I have already tried:
- Setting innodb_fast_shutdown=0 on DONOR and JOINING nodes
- Deleting ib_logfile0 and ib_logfile1 on the JOINER and restarting
Is there a known work-around for this? I'm cautious to try upgrading an existing node to 10.5 in case it has the same error.
Galera Config
[mysqld]
|
binlog_format=ROW
|
default-storage-engine=innodb
|
innodb_autoinc_lock_mode=2
|
query_cache_type=0
|
query_cache_size=0
|
bind-address=0.0.0.0
|
|
|
[galera]
|
wsrep_on=ON
|
|
|
# Galera Provider Configuration
|
wsrep_provider=/usr/lib/galera/libgalera_smm.so
|
wsrep_provider_options="gcache.size=5G"
|
|
|
# Galera Cluster Configuration
|
wsrep_cluster_name="mariadb-cluster"
|
wsrep_cluster_address="gcomm://172.31.34.27,172.31.35.130,172.31.37.249,172.31.40.66,172.31.37.12"
|
|
|
# Galera Synchronization Congifuration
|
wsrep_sst_method = rsync
|
wsrep_sst_donor = ES5_AWS
|
|
|
# Galera Node Configuration
|
wsrep_node_name="ES6_AWS"
|
wsrep_node_address="172.31.39.121"
|
JOINER Logs
Apr 7 10:35:46 ip-172-31-39-121 systemd[1]: Starting MariaDB 10.5.15 database server...
|
Apr 7 10:35:47 ip-172-31-39-121 sh[129656]: WSREP: Recovered position 00000000-0000-0000-0000-000000000000:-1
|
Apr 7 10:35:47 ip-172-31-39-121 mariadbd[129747]: 2022-04-07 10:35:47 0 [Note] /usr/sbin/mariadbd (mysqld 10.5.15-MariaDB-1:10.5.15+maria~focal) starting as process 129747 ...
|
Apr 7 10:35:47 ip-172-31-39-121 mariadbd[129747]: 2022-04-07 10:35:47 0 [Note] WSREP: Loading provider /usr/lib/galera/libgalera_smm.so initial position: 00000000-0000-0000-0000-000000000000:-1
|
Apr 7 10:35:47 ip-172-31-39-121 mariadbd[129747]: 2022-04-07 10:35:47 0 [Note] WSREP: wsrep_load(): loading provider library '/usr/lib/galera/libgalera_smm.so'
|
Apr 7 10:35:47 ip-172-31-39-121 mariadbd[129747]: 2022-04-07 10:35:47 0 [Note] WSREP: wsrep_load(): Galera 26.4.11(r67341d07) by Codership Oy <info@codership.com> loaded successfully.
|
Apr 7 10:35:47 ip-172-31-39-121 mariadbd[129747]: 2022-04-07 10:35:47 0 [Note] WSREP: CRC-32C: using hardware acceleration.
|
Apr 7 10:35:47 ip-172-31-39-121 mariadbd[129747]: 2022-04-07 10:35:47 0 [Warning] WSREP: Could not open state file for reading: '/var/lib/mysql//grastate.dat'
|
Apr 7 10:35:47 ip-172-31-39-121 mariadbd[129747]: 2022-04-07 10:35:47 0 [Note] WSREP: Found saved state: 00000000-0000-0000-0000-000000000000:-1, safe_to_bootstrap: 1
|
Apr 7 10:35:47 ip-172-31-39-121 mariadbd[129747]: 2022-04-07 10:35:47 0 [Note] WSREP: GCache DEBUG: opened preamble:
|
Apr 7 10:35:47 ip-172-31-39-121 mariadbd[129747]: Version: 0
|
Apr 7 10:35:47 ip-172-31-39-121 mariadbd[129747]: UUID: 00000000-0000-0000-0000-000000000000
|
Apr 7 10:35:47 ip-172-31-39-121 mariadbd[129747]: Seqno: -1 - -1
|
Apr 7 10:35:47 ip-172-31-39-121 mariadbd[129747]: Offset: -1
|
Apr 7 10:35:47 ip-172-31-39-121 mariadbd[129747]: Synced: 0
|
Apr 7 10:35:47 ip-172-31-39-121 mariadbd[129747]: 2022-04-07 10:35:47 0 [Note] WSREP: Skipped GCache ring buffer recovery: could not determine history UUID.
|
Apr 7 10:35:47 ip-172-31-39-121 mariadbd[129747]: 2022-04-07 10:35:47 0 [Warning] WSREP: Option 'gcs.fc_master_slave' is deprecated and will be removed in the future versions, please use 'gcs.fc_single_primary' instead.
|
Apr 7 10:35:47 ip-172-31-39-121 mariadbd[129747]: 2022-04-07 10:35:47 0 [Note] WSREP: Passing config to GCS: base_dir = /var/lib/mysql/; base_host = 172.31.39.121; base_port = 4567; cert.log_conflicts = no; cert.optimistic_pa = yes; debug = no; evs.auto_evict = 0; evs.delay_margin = PT1S; evs.delayed_keep_period = PT30S; evs.inactive_check_period = PT0.5S; evs.inactive_timeout = PT15S; evs.join_retrans_period = PT1S; evs.max_install_timeouts = 3; evs.send_window = 4; evs.stats_report_period = PT1M; evs.suspect_timeout = PT5S; evs.user_send_window = 2; evs.view_forget_timeout = PT24H; gcache.dir = /var/lib/mysql/; gcache.keep_pages_size = 0; gcache.keep_plaintext_size = 128M; gcache.mem_size = 0; gcache.name = galera.cache; gcache.page_size = 128M; gcache.recover = yes; gcache.size = 5G; gcomm.thread_prio = ; gcs.fc_debug = 0; gcs.fc_factor = 1.0; gcs.fc_limit = 16; gcs.fc_master_slave = no; gcs.fc_single_primary = no; gcs.max_packet_size = 64500; gcs.max_throttle = 0.25; gcs.recv_q_hard_limit = 9223372036854775807; gcs.recv_q_soft_limit = 0.25; gcs.sync_donor = no; gmcast.segment = 0;
|
Apr 7 10:35:47 ip-172-31-39-121 mariadbd[129747]: 2022-04-07 10:35:47 0 [Note] WSREP: Start replication
|
Apr 7 10:35:47 ip-172-31-39-121 mariadbd[129747]: 2022-04-07 10:35:47 0 [Note] WSREP: Connecting with bootstrap option: 0
|
Apr 7 10:35:47 ip-172-31-39-121 mariadbd[129747]: 2022-04-07 10:35:47 0 [Note] WSREP: Setting GCS initial position to 00000000-0000-0000-0000-000000000000:-1
|
Apr 7 10:35:47 ip-172-31-39-121 mariadbd[129747]: 2022-04-07 10:35:47 0 [Note] WSREP: protonet asio version 0
|
Apr 7 10:35:47 ip-172-31-39-121 mariadbd[129747]: 2022-04-07 10:35:47 0 [Note] WSREP: Using CRC-32C for message checksums.
|
Apr 7 10:35:47 ip-172-31-39-121 mariadbd[129747]: 2022-04-07 10:35:47 0 [Note] WSREP: backend: asio
|
Apr 7 10:35:47 ip-172-31-39-121 mariadbd[129747]: 2022-04-07 10:35:47 0 [Note] WSREP: gcomm thread scheduling priority set to other:0
|
Apr 7 10:35:47 ip-172-31-39-121 mariadbd[129747]: 2022-04-07 10:35:47 0 [Warning] WSREP: access file(/var/lib/mysql//gvwstate.dat) failed(No such file or directory)
|
Apr 7 10:35:47 ip-172-31-39-121 mariadbd[129747]: 2022-04-07 10:35:47 0 [Note] WSREP: restore pc from disk failed
|
Apr 7 10:35:47 ip-172-31-39-121 mariadbd[129747]: 2022-04-07 10:35:47 0 [Note] WSREP: GMCast version 0
|
Apr 7 10:35:47 ip-172-31-39-121 mariadbd[129747]: 2022-04-07 10:35:47 0 [Note] WSREP: (7c67946e-b045, 'tcp://0.0.0.0:4567') listening at tcp://0.0.0.0:4567
|
Apr 7 10:35:47 ip-172-31-39-121 mariadbd[129747]: 2022-04-07 10:35:47 0 [Note] WSREP: (7c67946e-b045, 'tcp://0.0.0.0:4567') multicast: , ttl: 1
|
Apr 7 10:35:47 ip-172-31-39-121 mariadbd[129747]: 2022-04-07 10:35:47 0 [Note] WSREP: EVS version 1
|
Apr 7 10:35:47 ip-172-31-39-121 mariadbd[129747]: 2022-04-07 10:35:47 0 [Note] WSREP: gcomm: connecting to group 'mariadb-cluster', peer '172.31.34.27:,172.31.35.130:,172.31.37.249:,172.31.40.66:,172.31.37.12:'
|
Apr 7 10:35:47 ip-172-31-39-121 mariadbd[129747]: 2022-04-07 10:35:47 0 [Note] WSREP: (7c67946e-b045, 'tcp://0.0.0.0:4567') connection established to 95267912-98ce tcp://172.31.34.27:4567
|
Apr 7 10:35:47 ip-172-31-39-121 mariadbd[129747]: 2022-04-07 10:35:47 0 [Note] WSREP: (7c67946e-b045, 'tcp://0.0.0.0:4567') turning message relay requesting on, nonlive peers: tcp://172.31.37.75:4567 tcp://172.31.9.176:4567
|
Apr 7 10:35:47 ip-172-31-39-121 mariadbd[129747]: 2022-04-07 10:35:47 0 [Note] WSREP: (7c67946e-b045, 'tcp://0.0.0.0:4567') connection established to 98319f4a-8b85 tcp://172.31.35.130:4567
|
Apr 7 10:35:47 ip-172-31-39-121 mariadbd[129747]: 2022-04-07 10:35:47 0 [Note] WSREP: (7c67946e-b045, 'tcp://0.0.0.0:4567') connection established to da9e7474-8746 tcp://172.31.40.66:4567
|
Apr 7 10:35:47 ip-172-31-39-121 mariadbd[129747]: 2022-04-07 10:35:47 0 [Note] WSREP: (7c67946e-b045, 'tcp://0.0.0.0:4567') connection established to e4e23eea-bbd0 tcp://172.31.37.249:4567
|
Apr 7 10:35:47 ip-172-31-39-121 mariadbd[129747]: 2022-04-07 10:35:47 0 [Note] WSREP: (7c67946e-b045, 'tcp://0.0.0.0:4567') connection established to 44508de1-af53 tcp://172.31.37.75:4567
|
Apr 7 10:35:47 ip-172-31-39-121 mariadbd[129747]: 2022-04-07 10:35:47 0 [Note] WSREP: (7c67946e-b045, 'tcp://0.0.0.0:4567') connection established to eb5f53de-8588 tcp://172.31.9.176:4567
|
Apr 7 10:35:47 ip-172-31-39-121 mariadbd[129747]: 2022-04-07 10:35:47 0 [Note] WSREP: (7c67946e-b045, 'tcp://0.0.0.0:4567') connection established to eb5f53de-8588 tcp://172.31.9.176:4567
|
Apr 7 10:35:47 ip-172-31-39-121 mariadbd[129747]: 2022-04-07 10:35:47 0 [Note] WSREP: Deferred close timer started for socket with remote endpoint: tcp://172.31.9.176:39010
|
Apr 7 10:35:47 ip-172-31-39-121 mariadbd[129747]: 2022-04-07 10:35:47 0 [Note] WSREP: Deferred close timer handle_wait Operation aborted. for 0xaaab09f1f940
|
Apr 7 10:35:47 ip-172-31-39-121 mariadbd[129747]: 2022-04-07 10:35:47 0 [Note] WSREP: Deferred close timer destruct
|
Apr 7 10:35:48 ip-172-31-39-121 mariadbd[129747]: 2022-04-07 10:35:48 0 [Note] WSREP: EVS version upgrade 0 -> 1
|
Apr 7 10:35:48 ip-172-31-39-121 mariadbd[129747]: 2022-04-07 10:35:48 0 [Note] WSREP: declaring 44508de1-af53 at tcp://172.31.37.75:4567 stable
|
Apr 7 10:35:48 ip-172-31-39-121 mariadbd[129747]: 2022-04-07 10:35:48 0 [Note] WSREP: declaring 95267912-98ce at tcp://172.31.34.27:4567 stable
|
Apr 7 10:35:48 ip-172-31-39-121 mariadbd[129747]: 2022-04-07 10:35:48 0 [Note] WSREP: declaring 98319f4a-8b85 at tcp://172.31.35.130:4567 stable
|
Apr 7 10:35:48 ip-172-31-39-121 mariadbd[129747]: 2022-04-07 10:35:48 0 [Note] WSREP: declaring da9e7474-8746 at tcp://172.31.40.66:4567 stable
|
Apr 7 10:35:48 ip-172-31-39-121 mariadbd[129747]: 2022-04-07 10:35:48 0 [Note] WSREP: declaring e4e23eea-bbd0 at tcp://172.31.37.249:4567 stable
|
Apr 7 10:35:48 ip-172-31-39-121 mariadbd[129747]: 2022-04-07 10:35:48 0 [Note] WSREP: declaring eb5f53de-8588 at tcp://172.31.9.176:4567 stable
|
Apr 7 10:35:48 ip-172-31-39-121 mariadbd[129747]: 2022-04-07 10:35:48 0 [Note] WSREP: PC protocol upgrade 0 -> 1
|
Apr 7 10:35:49 ip-172-31-39-121 mariadbd[129747]: 2022-04-07 10:35:49 0 [Note] WSREP: Node 44508de1-af53 state prim
|
Apr 7 10:35:49 ip-172-31-39-121 mariadbd[129747]: 2022-04-07 10:35:49 0 [Note] WSREP: view(view_id(PRIM,44508de1-af53,378) memb {
|
Apr 7 10:35:49 ip-172-31-39-121 mariadbd[129747]: #01144508de1-af53,0
|
Apr 7 10:35:49 ip-172-31-39-121 mariadbd[129747]: #0117c67946e-b045,0
|
Apr 7 10:35:49 ip-172-31-39-121 mariadbd[129747]: #01195267912-98ce,0
|
Apr 7 10:35:49 ip-172-31-39-121 mariadbd[129747]: #01198319f4a-8b85,0
|
Apr 7 10:35:49 ip-172-31-39-121 mariadbd[129747]: #011da9e7474-8746,0
|
Apr 7 10:35:49 ip-172-31-39-121 mariadbd[129747]: #011e4e23eea-bbd0,0
|
Apr 7 10:35:49 ip-172-31-39-121 mariadbd[129747]: #011eb5f53de-8588,0
|
Apr 7 10:35:49 ip-172-31-39-121 mariadbd[129747]: } joined {
|
Apr 7 10:35:49 ip-172-31-39-121 mariadbd[129747]: } left {
|
Apr 7 10:35:49 ip-172-31-39-121 mariadbd[129747]: } partitioned {
|
Apr 7 10:35:49 ip-172-31-39-121 mariadbd[129747]: })
|
Apr 7 10:35:49 ip-172-31-39-121 mariadbd[129747]: 2022-04-07 10:35:49 0 [Note] WSREP: save pc into disk
|
Apr 7 10:35:49 ip-172-31-39-121 mariadbd[129747]: 2022-04-07 10:35:49 0 [Note] WSREP: discarding pending addr without UUID: tcp://172.31.37.12:4567
|
Apr 7 10:35:49 ip-172-31-39-121 mariadbd[129747]: 2022-04-07 10:35:49 0 [Note] WSREP: discarding pending addr proto entry 0xaaab09eff5a0
|
Apr 7 10:35:49 ip-172-31-39-121 mariadbd[129747]: 2022-04-07 10:35:49 0 [Note] WSREP: gcomm: connected
|
Apr 7 10:35:49 ip-172-31-39-121 mariadbd[129747]: 2022-04-07 10:35:49 0 [Note] WSREP: Changing maximum packet size to 64500, resulting msg size: 32636
|
Apr 7 10:35:49 ip-172-31-39-121 mariadbd[129747]: 2022-04-07 10:35:49 0 [Note] WSREP: Shifting CLOSED -> OPEN (TO: 0)
|
Apr 7 10:35:49 ip-172-31-39-121 mariadbd[129747]: 2022-04-07 10:35:49 0 [Note] WSREP: Opened channel 'mariadb-cluster'
|
Apr 7 10:35:49 ip-172-31-39-121 mariadbd[129747]: 2022-04-07 10:35:49 0 [Note] WSREP: New COMPONENT: primary = yes, bootstrap = no, my_idx = 1, memb_num = 7
|
Apr 7 10:35:49 ip-172-31-39-121 mariadbd[129747]: 2022-04-07 10:35:49 0 [Note] WSREP: STATE EXCHANGE: Waiting for state UUID.
|
Apr 7 10:35:49 ip-172-31-39-121 mariadbd[129747]: 2022-04-07 10:35:49 0 [Note] WSREP: STATE EXCHANGE: sent state msg: 7dcd956b-b65e-11ec-8734-924b8afeca4d
|
Apr 7 10:35:49 ip-172-31-39-121 mariadbd[129747]: 2022-04-07 10:35:49 0 [Note] WSREP: STATE EXCHANGE: got state msg: 7dcd956b-b65e-11ec-8734-924b8afeca4d from 0 (ES5_AWS)
|
Apr 7 10:35:49 ip-172-31-39-121 mariadbd[129747]: 2022-04-07 10:35:49 0 [Note] WSREP: STATE EXCHANGE: got state msg: 7dcd956b-b65e-11ec-8734-924b8afeca4d from 2 (ES1_AWS)
|
Apr 7 10:35:49 ip-172-31-39-121 mariadbd[129747]: 2022-04-07 10:35:49 0 [Note] WSREP: STATE EXCHANGE: got state msg: 7dcd956b-b65e-11ec-8734-924b8afeca4d from 3 (ES2_AWS)
|
Apr 7 10:35:49 ip-172-31-39-121 mariadbd[129747]: 2022-04-07 10:35:49 0 [Note] WSREP: STATE EXCHANGE: got state msg: 7dcd956b-b65e-11ec-8734-924b8afeca4d from 4 (ES4_AWS)
|
Apr 7 10:35:49 ip-172-31-39-121 mariadbd[129747]: 2022-04-07 10:35:49 0 [Note] WSREP: STATE EXCHANGE: got state msg: 7dcd956b-b65e-11ec-8734-924b8afeca4d from 5 (ES3_AWS)
|
Apr 7 10:35:49 ip-172-31-39-121 mariadbd[129747]: 2022-04-07 10:35:49 1 [Note] WSREP: Starting rollbacker thread 1
|
Apr 7 10:35:49 ip-172-31-39-121 mariadbd[129747]: 2022-04-07 10:35:49 2 [Note] WSREP: Starting applier thread 2
|
Apr 7 10:35:49 ip-172-31-39-121 mariadbd[129747]: 2022-04-07 10:35:49 0 [Note] WSREP: STATE EXCHANGE: got state msg: 7dcd956b-b65e-11ec-8734-924b8afeca4d from 6 (BACKUP_AWS)
|
Apr 7 10:35:49 ip-172-31-39-121 mariadbd[129747]: 2022-04-07 10:35:49 0 [Note] WSREP: STATE EXCHANGE: got state msg: 7dcd956b-b65e-11ec-8734-924b8afeca4d from 1 (ES6_AWS)
|
Apr 7 10:35:49 ip-172-31-39-121 mariadbd[129747]: 2022-04-07 10:35:49 0 [Note] WSREP: Quorum results:
|
Apr 7 10:35:49 ip-172-31-39-121 mariadbd[129747]: #011version = 6,
|
Apr 7 10:35:49 ip-172-31-39-121 mariadbd[129747]: #011component = PRIMARY,
|
Apr 7 10:35:49 ip-172-31-39-121 mariadbd[129747]: #011conf_id = 363,
|
Apr 7 10:35:49 ip-172-31-39-121 mariadbd[129747]: #011members = 6/7 (joined/total),
|
Apr 7 10:35:49 ip-172-31-39-121 mariadbd[129747]: #011act_id = 5064032831,
|
Apr 7 10:35:49 ip-172-31-39-121 mariadbd[129747]: #011last_appl. = 5064032726,
|
Apr 7 10:35:49 ip-172-31-39-121 mariadbd[129747]: #011protocols = 2/10/4 (gcs/repl/appl),
|
Apr 7 10:35:49 ip-172-31-39-121 mariadbd[129747]: #011vote policy= 0,
|
Apr 7 10:35:49 ip-172-31-39-121 mariadbd[129747]: #011group UUID = 1d4206f9-647d-11ea-a963-0ed98924f90b
|
Apr 7 10:35:49 ip-172-31-39-121 mariadbd[129747]: 2022-04-07 10:35:49 0 [Note] WSREP: Flow-control interval: [42, 42]
|
Apr 7 10:35:49 ip-172-31-39-121 mariadbd[129747]: 2022-04-07 10:35:49 0 [Note] WSREP: Shifting OPEN -> PRIMARY (TO: 5064032832)
|
Apr 7 10:35:49 ip-172-31-39-121 mariadbd[129747]: 2022-04-07 10:35:49 2 [Note] WSREP: ####### processing CC 5064032832, local, ordered
|
Apr 7 10:35:49 ip-172-31-39-121 mariadbd[129747]: 2022-04-07 10:35:49 2 [Note] WSREP: Process first view: 1d4206f9-647d-11ea-a963-0ed98924f90b my uuid: 7c67946e-b65e-11ec-b045-eb2b92474255
|
Apr 7 10:35:49 ip-172-31-39-121 mariadbd[129747]: 2022-04-07 10:35:49 2 [Note] WSREP: Server ES6_AWS connected to cluster at position 1d4206f9-647d-11ea-a963-0ed98924f90b:5064032832 with ID 7c67946e-b65e-11ec-b045-eb2b92474255
|
Apr 7 10:35:49 ip-172-31-39-121 mariadbd[129747]: 2022-04-07 10:35:49 2 [Note] WSREP: Server status change disconnected -> connected
|
Apr 7 10:35:49 ip-172-31-39-121 mariadbd[129747]: 2022-04-07 10:35:49 2 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
|
Apr 7 10:35:49 ip-172-31-39-121 mariadbd[129747]: 2022-04-07 10:35:49 2 [Note] WSREP: ####### My UUID: 7c67946e-b65e-11ec-b045-eb2b92474255
|
Apr 7 10:35:49 ip-172-31-39-121 mariadbd[129747]: 2022-04-07 10:35:49 2 [Note] WSREP: Cert index reset to 00000000-0000-0000-0000-000000000000:-1 (proto: 10), state transfer needed: yes
|
Apr 7 10:35:49 ip-172-31-39-121 mariadbd[129747]: 2022-04-07 10:35:49 0 [Note] WSREP: Service thread queue flushed.
|
Apr 7 10:35:49 ip-172-31-39-121 mariadbd[129747]: 2022-04-07 10:35:49 2 [Note] WSREP: ####### Assign initial position for certification: 00000000-0000-0000-0000-000000000000:-1, protocol version: -1
|
Apr 7 10:35:49 ip-172-31-39-121 mariadbd[129747]: 2022-04-07 10:35:49 2 [Note] WSREP: State transfer required:
|
Apr 7 10:35:49 ip-172-31-39-121 mariadbd[129747]: #011Group state: 1d4206f9-647d-11ea-a963-0ed98924f90b:5064032832
|
Apr 7 10:35:49 ip-172-31-39-121 mariadbd[129747]: #011Local state: 00000000-0000-0000-0000-000000000000:-1
|
Apr 7 10:35:49 ip-172-31-39-121 mariadbd[129747]: 2022-04-07 10:35:49 2 [Note] WSREP: Server status change connected -> joiner
|
Apr 7 10:35:49 ip-172-31-39-121 mariadbd[129747]: 2022-04-07 10:35:49 2 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
|
Apr 7 10:35:49 ip-172-31-39-121 mariadbd[129747]: 2022-04-07 10:35:49 0 [Note] WSREP: Joiner monitor thread started to monitor
|
Apr 7 10:35:49 ip-172-31-39-121 mariadbd[129747]: 2022-04-07 10:35:49 0 [Note] WSREP: Running: 'wsrep_sst_rsync --role 'joiner' --address '172.31.39.121' --datadir '/var/lib/mysql/' --parent '129747' --mysqld-args --wsrep_start_position=00000000-0000-0000-0000-000000000000:-1'
|
Apr 7 10:35:49 ip-172-31-39-121 rsyncd[129866]: rsyncd version 3.1.3 starting, listening on port 4444
|
Apr 7 10:35:49 ip-172-31-39-121 mariadbd[129747]: 2022-04-07 10:35:49 2 [Note] WSREP: ####### IST uuid:00000000-0000-0000-0000-000000000000 f: 0, l: 5064032832, STRv: 3
|
Apr 7 10:35:49 ip-172-31-39-121 mariadbd[129747]: 2022-04-07 10:35:49 2 [Note] WSREP: IST receiver addr using tcp://172.31.39.121:4568
|
Apr 7 10:35:49 ip-172-31-39-121 mariadbd[129747]: 2022-04-07 10:35:49 2 [Note] WSREP: Prepared IST receiver for 0-5064032832, listening at: tcp://172.31.39.121:4568
|
Apr 7 10:35:49 ip-172-31-39-121 mariadbd[129747]: 2022-04-07 10:35:49 0 [Note] WSREP: Member 1.0 (ES6_AWS) requested state transfer from 'ES5_AWS'. Selected 0.0 (ES5_AWS)(SYNCED) as donor.
|
Apr 7 10:35:49 ip-172-31-39-121 mariadbd[129747]: 2022-04-07 10:35:49 0 [Note] WSREP: Shifting PRIMARY -> JOINER (TO: 5064032832)
|
Apr 7 10:35:49 ip-172-31-39-121 mariadbd[129747]: 2022-04-07 10:35:49 2 [Note] WSREP: Requesting state transfer: success, donor: 0
|
Apr 7 10:35:49 ip-172-31-39-121 mariadbd[129747]: 2022-04-07 10:35:49 2 [Note] WSREP: Resetting GCache seqno map due to different histories.
|
Apr 7 10:35:49 ip-172-31-39-121 mariadbd[129747]: 2022-04-07 10:35:49 2 [Note] WSREP: GCache history reset: 00000000-0000-0000-0000-000000000000:0 -> 1d4206f9-647d-11ea-a963-0ed98924f90b:5064032832
|
Apr 7 10:35:50 ip-172-31-39-121 rsyncd[129878]: connect from ip-172-31-37-75.eu-west-1.compute.internal (172.31.37.75)
|
Apr 7 10:35:50 ip-172-31-39-121 rsyncd[129878]: rsync to rsync_sst/ from ip-172-31-37-75.eu-west-1.compute.internal (172.31.37.75)
|
|
|
...
|
|
|
Apr 7 10:39:52 ip-172-31-39-121 mariadbd[129747]: 2022-04-07 10:39:52 0 [Note] WSREP: 0.0 (ES5_AWS): State transfer to 1.0 (ES6_AWS) complete.
|
Apr 7 10:39:52 ip-172-31-39-121 mariadbd[129757]: WSREP_SST: [INFO] Joiner cleanup: rsync PID=129866, stunnel PID=0 (20220407 10:39:52.306)
|
Apr 7 10:39:52 ip-172-31-39-121 mariadbd[129747]: 2022-04-07 10:39:52 0 [Note] WSREP: Member 0.0 (ES5_AWS) synced with group.
|
Apr 7 10:39:52 ip-172-31-39-121 rsyncd[129866]: sent 0 bytes received 0 bytes total size 0
|
Apr 7 10:39:52 ip-172-31-39-121 mariadbd[129757]: WSREP_SST: [INFO] Joiner cleanup done. (20220407 10:39:52.825)
|
Apr 7 10:39:52 ip-172-31-39-121 mariadbd[129747]: 2022-04-07 10:39:52 3 [Note] WSREP: SST received
|
Apr 7 10:39:52 ip-172-31-39-121 mariadbd[129747]: 2022-04-07 10:39:52 3 [Note] WSREP: Server status change joiner -> initializing
|
Apr 7 10:39:52 ip-172-31-39-121 mariadbd[129747]: 2022-04-07 10:39:52 3 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
|
Apr 7 10:39:52 ip-172-31-39-121 mariadbd[129747]: 2022-04-07 10:39:52 0 [Note] InnoDB: Uses event mutexes
|
Apr 7 10:39:52 ip-172-31-39-121 mariadbd[129747]: 2022-04-07 10:39:52 0 [Note] InnoDB: Compressed tables use zlib 1.2.11
|
Apr 7 10:39:52 ip-172-31-39-121 mariadbd[129747]: 2022-04-07 10:39:52 0 [Note] InnoDB: Number of pools: 1
|
Apr 7 10:39:52 ip-172-31-39-121 mariadbd[129747]: 2022-04-07 10:39:52 0 [Note] InnoDB: Using ARMv8 crc32 + pmull instructions
|
Apr 7 10:39:52 ip-172-31-39-121 mariadbd[129747]: 2022-04-07 10:39:52 0 [Note] InnoDB: Using Linux native AIO
|
Apr 7 10:39:52 ip-172-31-39-121 mariadbd[129747]: 2022-04-07 10:39:52 0 [Note] InnoDB: Initializing buffer pool, total size = 134217728, chunk size = 134217728
|
Apr 7 10:39:52 ip-172-31-39-121 mariadbd[129747]: 2022-04-07 10:39:52 0 [Note] InnoDB: Completed initialization of buffer pool
|
Apr 7 10:39:52 ip-172-31-39-121 mariadbd[129747]: 2022-04-07 10:39:52 0 [ERROR] InnoDB: Upgrade after a crash is not supported. The redo log was created with MariaDB 10.4.22.
|
Apr 7 10:39:52 ip-172-31-39-121 mariadbd[129747]: 2022-04-07 10:39:52 0 [ERROR] InnoDB: Plugin initialization aborted with error Generic error
|
Apr 7 10:39:52 ip-172-31-39-121 mariadbd[129747]: 2022-04-07 10:39:52 0 [Note] InnoDB: Starting shutdown...
|
Apr 7 10:39:53 ip-172-31-39-121 mariadbd[129747]: 2022-04-07 10:39:53 0 [ERROR] Plugin 'InnoDB' init function returned error.
|
Apr 7 10:39:53 ip-172-31-39-121 mariadbd[129747]: 2022-04-07 10:39:53 0 [ERROR] Plugin 'InnoDB' registration as a STORAGE ENGINE failed.
|
Apr 7 10:39:53 ip-172-31-39-121 mariadbd[129747]: 2022-04-07 10:39:53 0 [Note] Plugin 'FEEDBACK' is disabled.
|
Apr 7 10:39:53 ip-172-31-39-121 mariadbd[129747]: 2022-04-07 10:39:53 0 [ERROR] Unknown/unsupported storage engine: innodb
|
Apr 7 10:39:53 ip-172-31-39-121 mariadbd[129747]: 2022-04-07 10:39:53 0 [ERROR] Aborting
|
Apr 7 10:39:53 ip-172-31-39-121 mariadbd[129747]: terminate called after throwing an instance of 'wsrep::runtime_error'
|
Apr 7 10:39:53 ip-172-31-39-121 mariadbd[129747]: what(): State wait was interrupted
|
Apr 7 10:39:53 ip-172-31-39-121 mariadbd[129747]: 220407 10:39:53 [ERROR] mysqld got signal 6 ;
|
Apr 7 10:39:53 ip-172-31-39-121 mariadbd[129747]: This could be because you hit a bug. It is also possible that this binary
|
Apr 7 10:39:53 ip-172-31-39-121 mariadbd[129747]: or one of the libraries it was linked against is corrupt, improperly built,
|
Apr 7 10:39:53 ip-172-31-39-121 mariadbd[129747]: or misconfigured. This error can also be caused by malfunctioning hardware.
|
Apr 7 10:39:53 ip-172-31-39-121 mariadbd[129747]: To report this bug, see https://mariadb.com/kb/en/reporting-bugs
|
Apr 7 10:39:53 ip-172-31-39-121 mariadbd[129747]: We will try our best to scrape up some info that will hopefully help
|
Apr 7 10:39:53 ip-172-31-39-121 mariadbd[129747]: diagnose the problem, but since we have already crashed,
|
Apr 7 10:39:53 ip-172-31-39-121 mariadbd[129747]: something is definitely wrong and this may fail.
|
Apr 7 10:39:53 ip-172-31-39-121 mariadbd[129747]: Server version: 10.5.15-MariaDB-1:10.5.15+maria~focal
|
Apr 7 10:39:53 ip-172-31-39-121 mariadbd[129747]: key_buffer_size=134217728
|
Apr 7 10:39:53 ip-172-31-39-121 mariadbd[129747]: read_buffer_size=131072
|
Apr 7 10:39:53 ip-172-31-39-121 mariadbd[129747]: max_used_connections=0
|
Apr 7 10:39:53 ip-172-31-39-121 mariadbd[129747]: max_threads=153
|
Apr 7 10:39:53 ip-172-31-39-121 mariadbd[129747]: thread_count=4
|
Apr 7 10:39:53 ip-172-31-39-121 mariadbd[129747]: It is possible that mysqld could use up to
|
Apr 7 10:39:53 ip-172-31-39-121 mariadbd[129747]: key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 467879 K bytes of memory
|
Apr 7 10:39:53 ip-172-31-39-121 mariadbd[129747]: Hope that's ok; if not, decrease some variables in the equation.
|
Apr 7 10:39:53 ip-172-31-39-121 mariadbd[129747]: Thread pointer: 0xfffe1c002068
|
Apr 7 10:39:53 ip-172-31-39-121 mariadbd[129747]: Attempting backtrace. You can use the following information to find out
|
Apr 7 10:39:53 ip-172-31-39-121 mariadbd[129747]: where mysqld died. If you see no messages after this, something went
|
Apr 7 10:39:53 ip-172-31-39-121 mariadbd[129747]: terribly wrong...
|
Apr 7 10:39:53 ip-172-31-39-121 mariadbd[129747]: stack_bottom = 0xfffe42579630 thread_stack 0x49000
|
Apr 7 10:39:53 ip-172-31-39-121 mariadbd[129747]: Printing to addr2line failed
|
Apr 7 10:39:53 ip-172-31-39-121 mariadbd[129747]: /usr/sbin/mariadbd(my_print_stacktrace+0x30)[0xaaaacc7023f0]
|
Apr 7 10:39:53 ip-172-31-39-121 mariadbd[129747]: /usr/sbin/mariadbd(handle_fatal_signal+0x45c)[0xaaaacc1b137c]
|
Apr 7 10:39:53 ip-172-31-39-121 mariadbd[129747]: linux-vdso.so.1(__kernel_rt_sigreturn+0x0)[0xffff89d95794]
|
Apr 7 10:39:53 ip-172-31-39-121 mariadbd[129747]: /lib/aarch64-linux-gnu/atomics/libc.so.6(gsignal+0xe0)[0xffff8942dc20]
|
Apr 7 10:39:53 ip-172-31-39-121 mariadbd[129747]: /lib/aarch64-linux-gnu/atomics/libc.so.6(abort+0xfc)[0xffff8941aa14]
|
Apr 7 10:39:53 ip-172-31-39-121 mariadbd[129747]: /lib/aarch64-linux-gnu/libstdc++.so.6(_ZN9__gnu_cxx27__verbose_terminate_handlerEv+0x19c)[0xffff896b88cc]
|
Apr 7 10:39:53 ip-172-31-39-121 mariadbd[129747]: /lib/aarch64-linux-gnu/libstdc++.so.6(+0x9f21c)[0xffff896b621c]
|
Apr 7 10:39:53 ip-172-31-39-121 mariadbd[129747]: /lib/aarch64-linux-gnu/libstdc++.so.6(+0x9f280)[0xffff896b6280]
|
Apr 7 10:39:53 ip-172-31-39-121 mariadbd[129747]: /lib/aarch64-linux-gnu/libstdc++.so.6(+0x9f574)[0xffff896b6574]
|
Apr 7 10:39:53 ip-172-31-39-121 mariadbd[129747]: /usr/sbin/mariadbd(_ZNK5wsrep12server_state16wait_until_stateERSt11unique_lockINS_5mutexEENS0_5stateE+0x1dc)[0xaaaacc7808ec]
|
Apr 7 10:39:53 ip-172-31-39-121 mariadbd[129747]: /usr/sbin/mariadbd(_ZN5wsrep12server_state12sst_receivedERNS_14client_serviceEi+0xb04)[0xaaaacc78a5f4]
|
Apr 7 10:39:53 ip-172-31-39-121 mariadbd[129747]: /usr/sbin/mariadbd(+0xb7b6bc)[0xaaaacc43b6bc]
|
Apr 7 10:39:53 ip-172-31-39-121 mariadbd[129747]: /usr/sbin/mariadbd(+0xb7c324)[0xaaaacc43c324]
|
Apr 7 10:39:53 ip-172-31-39-121 mariadbd[129747]: /usr/sbin/mariadbd(+0xb0bca0)[0xaaaacc3cbca0]
|
Apr 7 10:39:53 ip-172-31-39-121 mariadbd[129747]: /lib/aarch64-linux-gnu/atomics/libpthread.so.0(+0x73f0)[0xffff898173f0]
|
Apr 7 10:39:53 ip-172-31-39-121 mariadbd[129747]: /lib/aarch64-linux-gnu/atomics/libc.so.6(+0xd00dc)[0xffff894ca0dc]
|
Apr 7 10:39:53 ip-172-31-39-121 mariadbd[129747]: Trying to get some variables.
|
Apr 7 10:39:53 ip-172-31-39-121 mariadbd[129747]: Some pointers may be invalid and cause the dump to abort.
|
Apr 7 10:39:53 ip-172-31-39-121 mariadbd[129747]: Query (0x0): (null)
|
Apr 7 10:39:53 ip-172-31-39-121 mariadbd[129747]: Connection ID (thread ID): 3
|
Apr 7 10:39:53 ip-172-31-39-121 mariadbd[129747]: Status: NOT_KILLED
|
Apr 7 10:39:53 ip-172-31-39-121 mariadbd[129747]: Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=on,condition_pushdown_from_having=on,not_null_range_scan=off
|
Apr 7 10:39:53 ip-172-31-39-121 mariadbd[129747]: The manual page at https://mariadb.com/kb/en/how-to-produce-a-full-stack-trace-for-mysqld/ contains
|
Apr 7 10:39:53 ip-172-31-39-121 mariadbd[129747]: information that should help you find out what is causing the crash.
|
Apr 7 10:39:53 ip-172-31-39-121 mariadbd[129747]: We think the query pointer is invalid, but we will try to print it anyway.
|
Apr 7 10:39:53 ip-172-31-39-121 mariadbd[129747]: Query:
|
Apr 7 10:39:53 ip-172-31-39-121 mariadbd[129747]: Writing a core file...
|
Apr 7 10:39:53 ip-172-31-39-121 mariadbd[129747]: Working directory at /media/storage/mysql
|
Apr 7 10:39:53 ip-172-31-39-121 mariadbd[129747]: Resource Limits:
|
Apr 7 10:39:53 ip-172-31-39-121 mariadbd[129747]: Limit Soft Limit Hard Limit Units
|
Apr 7 10:39:53 ip-172-31-39-121 mariadbd[129747]: Max cpu time unlimited unlimited seconds
|
Apr 7 10:39:53 ip-172-31-39-121 mariadbd[129747]: Max file size unlimited unlimited bytes
|
Apr 7 10:39:53 ip-172-31-39-121 mariadbd[129747]: Max data size unlimited unlimited bytes
|
Apr 7 10:39:53 ip-172-31-39-121 mariadbd[129747]: Max stack size 8388608 unlimited bytes
|
Apr 7 10:39:53 ip-172-31-39-121 mariadbd[129747]: Max core file size 0 unlimited bytes
|
Apr 7 10:39:53 ip-172-31-39-121 mariadbd[129747]: Max resident set unlimited unlimited bytes
|
Apr 7 10:39:53 ip-172-31-39-121 mariadbd[129747]: Max processes 125387 125387 processes
|
Apr 7 10:39:53 ip-172-31-39-121 mariadbd[129747]: Max open files 32768 32768 files
|
Apr 7 10:39:53 ip-172-31-39-121 mariadbd[129747]: Max locked memory 65536 65536 bytes
|
Apr 7 10:39:53 ip-172-31-39-121 mariadbd[129747]: Max address space unlimited unlimited bytes
|
Apr 7 10:39:53 ip-172-31-39-121 mariadbd[129747]: Max file locks unlimited unlimited locks
|
Apr 7 10:39:53 ip-172-31-39-121 mariadbd[129747]: Max pending signals 125387 125387 signals
|
Apr 7 10:39:53 ip-172-31-39-121 mariadbd[129747]: Max msgqueue size 819200 819200 bytes
|
Apr 7 10:39:53 ip-172-31-39-121 mariadbd[129747]: Max nice priority 0 0
|
Apr 7 10:39:53 ip-172-31-39-121 mariadbd[129747]: Max realtime priority 0 0
|
Apr 7 10:39:53 ip-172-31-39-121 mariadbd[129747]: Max realtime timeout unlimited unlimited us
|
Apr 7 10:39:53 ip-172-31-39-121 mariadbd[129747]: Core pattern: |/usr/share/apport/apport %p %s %c %d %P %E
|
Apr 7 10:39:53 ip-172-31-39-121 systemd[1]: mariadb.service: Main process exited, code=dumped, status=6/ABRT
|
Apr 7 10:39:53 ip-172-31-39-121 systemd[1]: mariadb.service: Failed with result 'core-dump'.
|
Apr 7 10:39:53 ip-172-31-39-121 systemd[1]: Failed to start MariaDB 10.5.15 database server.
|
DONOR Logs
Apr 7 11:35:47 es5 mysqld[29230]: 2022-04-07 11:35:47 0 [Note] WSREP: (44508de1-af53, 'tcp://0.0.0.0:4567') turning message relay requesting on, nonlive peers: tcp://172.31.39.121:4567
|
Apr 7 11:35:47 es5 mysqld[29230]: 2022-04-07 11:35:47 0 [Note] WSREP: (44508de1-af53, 'tcp://0.0.0.0:4567') connection established to 7c67946e-b045 tcp://172.31.39.121:4567
|
Apr 7 11:35:48 es5 mysqld[29230]: 2022-04-07 11:35:48 0 [Note] WSREP: declaring 7c67946e-b045 at tcp://172.31.39.121:4567 stable
|
Apr 7 11:35:48 es5 mysqld[29230]: 2022-04-07 11:35:48 0 [Note] WSREP: declaring 95267912-98ce at tcp://172.31.34.27:4567 stable
|
Apr 7 11:35:48 es5 mysqld[29230]: 2022-04-07 11:35:48 0 [Note] WSREP: declaring 98319f4a-8b85 at tcp://172.31.35.130:4567 stable
|
Apr 7 11:35:48 es5 mysqld[29230]: 2022-04-07 11:35:48 0 [Note] WSREP: declaring da9e7474-8746 at tcp://172.31.40.66:4567 stable
|
Apr 7 11:35:48 es5 mysqld[29230]: 2022-04-07 11:35:48 0 [Note] WSREP: declaring e4e23eea-bbd0 at tcp://172.31.37.249:4567 stable
|
Apr 7 11:35:48 es5 mysqld[29230]: 2022-04-07 11:35:48 0 [Note] WSREP: declaring eb5f53de-8588 at tcp://172.31.9.176:4567 stable
|
Apr 7 11:35:49 es5 mysqld[29230]: 2022-04-07 11:35:49 0 [Note] WSREP: Node 44508de1-af53 state prim
|
Apr 7 11:35:49 es5 mysqld[29230]: 2022-04-07 11:35:49 0 [Note] WSREP: view(view_id(PRIM,44508de1-af53,378) memb {
|
Apr 7 11:35:49 es5 mysqld[29230]: #01144508de1-af53,0
|
Apr 7 11:35:49 es5 mysqld[29230]: #0117c67946e-b045,0
|
Apr 7 11:35:49 es5 mysqld[29230]: #01195267912-98ce,0
|
Apr 7 11:35:49 es5 mysqld[29230]: #01198319f4a-8b85,0
|
Apr 7 11:35:49 es5 mysqld[29230]: #011da9e7474-8746,0
|
Apr 7 11:35:49 es5 mysqld[29230]: #011e4e23eea-bbd0,0
|
Apr 7 11:35:49 es5 mysqld[29230]: #011eb5f53de-8588,0
|
Apr 7 11:35:49 es5 mysqld[29230]: } joined {
|
Apr 7 11:35:49 es5 mysqld[29230]: } left {
|
Apr 7 11:35:49 es5 mysqld[29230]: } partitioned {
|
Apr 7 11:35:49 es5 mysqld[29230]: })
|
Apr 7 11:35:49 es5 mysqld[29230]: 2022-04-07 11:35:49 0 [Note] WSREP: save pc into disk
|
Apr 7 11:35:49 es5 mysqld[29230]: 2022-04-07 11:35:49 0 [Note] WSREP: New COMPONENT: primary = yes, bootstrap = no, my_idx = 0, memb_num = 7
|
Apr 7 11:35:49 es5 mysqld[29230]: 2022-04-07 11:35:49 0 [Note] WSREP: STATE_EXCHANGE: sent state UUID: 7dcd956b-b65e-11ec-8734-924b8afeca4d
|
Apr 7 11:35:49 es5 mysqld[29230]: 2022-04-07 11:35:49 0 [Note] WSREP: STATE EXCHANGE: sent state msg: 7dcd956b-b65e-11ec-8734-924b8afeca4d
|
Apr 7 11:35:49 es5 mysqld[29230]: 2022-04-07 11:35:49 0 [Note] WSREP: STATE EXCHANGE: got state msg: 7dcd956b-b65e-11ec-8734-924b8afeca4d from 0 (ES5_AWS)
|
Apr 7 11:35:49 es5 mysqld[29230]: 2022-04-07 11:35:49 0 [Note] WSREP: STATE EXCHANGE: got state msg: 7dcd956b-b65e-11ec-8734-924b8afeca4d from 2 (ES1_AWS)
|
Apr 7 11:35:49 es5 mysqld[29230]: 2022-04-07 11:35:49 0 [Note] WSREP: STATE EXCHANGE: got state msg: 7dcd956b-b65e-11ec-8734-924b8afeca4d from 3 (ES2_AWS)
|
Apr 7 11:35:49 es5 mysqld[29230]: 2022-04-07 11:35:49 0 [Note] WSREP: STATE EXCHANGE: got state msg: 7dcd956b-b65e-11ec-8734-924b8afeca4d from 4 (ES4_AWS)
|
Apr 7 11:35:49 es5 mysqld[29230]: 2022-04-07 11:35:49 0 [Note] WSREP: STATE EXCHANGE: got state msg: 7dcd956b-b65e-11ec-8734-924b8afeca4d from 5 (ES3_AWS)
|
Apr 7 11:35:49 es5 mysqld[29230]: 2022-04-07 11:35:49 0 [Note] WSREP: STATE EXCHANGE: got state msg: 7dcd956b-b65e-11ec-8734-924b8afeca4d from 6 (BACKUP_AWS)
|
Apr 7 11:35:49 es5 mysqld[29230]: 2022-04-07 11:35:49 0 [Note] WSREP: STATE EXCHANGE: got state msg: 7dcd956b-b65e-11ec-8734-924b8afeca4d from 1 (ES6_AWS)
|
Apr 7 11:35:49 es5 mysqld[29230]: 2022-04-07 11:35:49 0 [Note] WSREP: Quorum results:
|
Apr 7 11:35:49 es5 mysqld[29230]: #011version = 6,
|
Apr 7 11:35:49 es5 mysqld[29230]: #011component = PRIMARY,
|
Apr 7 11:35:49 es5 mysqld[29230]: #011conf_id = 363,
|
Apr 7 11:35:49 es5 mysqld[29230]: #011members = 6/7 (joined/total),
|
Apr 7 11:35:49 es5 mysqld[29230]: #011act_id = 5064032831,
|
Apr 7 11:35:49 es5 mysqld[29230]: #011last_appl. = 5064032726,
|
Apr 7 11:35:49 es5 mysqld[29230]: #011protocols = 2/10/4 (gcs/repl/appl),
|
Apr 7 11:35:49 es5 mysqld[29230]: #011vote policy= 0,
|
Apr 7 11:35:49 es5 mysqld[29230]: #011group UUID = 1d4206f9-647d-11ea-a963-0ed98924f90b
|
Apr 7 11:35:49 es5 mysqld[29230]: 2022-04-07 11:35:49 0 [Note] WSREP: Flow-control interval: [42, 42]
|
Apr 7 11:35:49 es5 mysqld[29230]: 2022-04-07 11:35:49 2 [Note] WSREP: ####### processing CC 5064032832, local, ordered
|
Apr 7 11:35:49 es5 mysqld[29230]: 2022-04-07 11:35:49 2 [Note] WSREP: ####### My UUID: 44508de1-b657-11ec-af53-dbbee42dcffe
|
Apr 7 11:35:49 es5 mysqld[29230]: 2022-04-07 11:35:49 2 [Note] WSREP: Skipping cert index reset
|
Apr 7 11:35:49 es5 mysqld[29230]: 2022-04-07 11:35:49 2 [Note] WSREP: REPL Protocols: 10 (5)
|
Apr 7 11:35:49 es5 mysqld[29230]: 2022-04-07 11:35:49 2 [Note] WSREP: ####### Adjusting cert position: 5064032831 -> 5064032832
|
Apr 7 11:35:49 es5 mysqld[29230]: 2022-04-07 11:35:49 0 [Note] WSREP: Service thread queue flushed.
|
Apr 7 11:35:49 es5 mysqld[29230]: 2022-04-07 11:35:49 2 [Note] WSREP: ================================================
|
Apr 7 11:35:49 es5 mysqld[29230]: View:
|
Apr 7 11:35:49 es5 mysqld[29230]: id: 1d4206f9-647d-11ea-a963-0ed98924f90b:5064032832
|
Apr 7 11:35:49 es5 mysqld[29230]: status: primary
|
Apr 7 11:35:49 es5 mysqld[29230]: protocol_version: 4
|
Apr 7 11:35:49 es5 mysqld[29230]: capabilities: MULTI-MASTER, CERTIFICATION, PARALLEL_APPLYING, REPLAY, ISOLATION, PAUSE, CAUSAL_READ, INCREMENTAL_WS, UNORDERED, PREORDERED, STREAMING, NBO
|
Apr 7 11:35:49 es5 mysqld[29230]: final: no
|
Apr 7 11:35:49 es5 mysqld[29230]: own_index: 0
|
Apr 7 11:35:49 es5 mysqld[29230]: members(7):
|
Apr 7 11:35:49 es5 mysqld[29230]: #0110: 44508de1-b657-11ec-af53-dbbee42dcffe, ES5_AWS
|
Apr 7 11:35:49 es5 mysqld[29230]: #0111: 7c67946e-b65e-11ec-b045-eb2b92474255, ES6_AWS
|
Apr 7 11:35:49 es5 mysqld[29230]: #0112: 95267912-5df1-11ec-98ce-a66d469d045c, ES1_AWS
|
Apr 7 11:35:49 es5 mysqld[29230]: #0113: 98319f4a-b5f9-11ec-8b85-e69cdab5035e, ES2_AWS
|
Apr 7 11:35:49 es5 mysqld[29230]: #0114: da9e7474-5df2-11ec-8746-dfd73835a1c5, ES4_AWS
|
Apr 7 11:35:49 es5 mysqld[29230]: #0115: e4e23eea-5df0-11ec-bbd0-8b132f61eb3d, ES3_AWS
|
Apr 7 11:35:49 es5 mysqld[29230]: #0116: eb5f53de-882b-11ec-8588-1a528d6537ef, BACKUP_AWS
|
Apr 7 11:35:49 es5 mysqld[29230]: =================================================
|
Apr 7 11:35:49 es5 mysqld[29230]: 2022-04-07 11:35:49 2 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
|
Apr 7 11:35:49 es5 mysqld[29230]: 2022-04-07 11:35:49 2 [Note] WSREP: Lowest cert index boundary for CC from group: 5064032727
|
Apr 7 11:35:49 es5 mysqld[29230]: 2022-04-07 11:35:49 2 [Note] WSREP: Min available from gcache for CC from group: 5063284333
|
Apr 7 11:35:49 es5 mysqld[29230]: 2022-04-07 11:35:49 0 [Note] WSREP: Member 1.0 (ES6_AWS) requested state transfer from 'ES5_AWS'. Selected 0.0 (ES5_AWS)(SYNCED) as donor.
|
Apr 7 11:35:49 es5 mysqld[29230]: 2022-04-07 11:35:49 0 [Note] WSREP: Shifting SYNCED -> DONOR/DESYNCED (TO: 5064032832)
|
Apr 7 11:35:49 es5 mysqld[29230]: 2022-04-07 11:35:49 2 [Note] WSREP: Detected STR version: 1, req_len: 125, req: STRv1
|
Apr 7 11:35:49 es5 mysqld[29230]: 2022-04-07 11:35:49 2 [Note] WSREP: Cert index preload: 5064032727 -> 5064032832
|
Apr 7 11:35:49 es5 mysqld[29230]: 2022-04-07 11:35:49 2 [Note] WSREP: Server status change synced -> donor
|
Apr 7 11:35:49 es5 mysqld[29230]: 2022-04-07 11:35:49 0 [Note] WSREP: async IST sender starting to serve tcp://172.31.39.121:4568 sending 5064032727-5064032832, preload starts from 5064032727
|
Apr 7 11:35:49 es5 mysqld[29230]: 2022-04-07 11:35:49 2 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
|
Apr 7 11:35:49 es5 mysqld[29230]: 2022-04-07 11:35:49 2 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
|
Apr 7 11:35:49 es5 mysqld[29230]: 2022-04-07 11:35:49 0 [Note] WSREP: Donor monitor thread started to monitor
|
Apr 7 11:35:49 es5 mysqld[29230]: 2022-04-07 11:35:49 0 [Note] WSREP: Running: 'wsrep_sst_rsync --role 'donor' --address '172.31.39.121:4444/rsync_sst' --local-port '3306' --socket '/var/run/mysqld/mysqld.sock' --datadir '/media/storage/mysql/' --gtid '1d4206f9-647d-11ea-a963-0ed98924f90b:5064032832' --gtid-domain-id '0' --mysqld-args --wsrep_start_position=1d4206f9-647d-11ea-a963-0ed98924f90b:5063284262'
|
Apr 7 11:35:49 es5 mysqld[29230]: 2022-04-07 11:35:49 0 [Note] WSREP: IST sender 5064032727 -> 5064032832
|
Apr 7 11:35:49 es5 mysqld[29230]: 2022-04-07 11:35:49 2 [Note] WSREP: sst_donor_thread signaled with 0
|
Apr 7 11:35:49 es5 mysqld[29230]: 2022-04-07 11:35:49 0 [Note] WSREP: Flushing tables for SST...
|
Apr 7 11:35:49 es5 mysqld[29230]: 2022-04-07 11:35:49 0 [Note] WSREP: pause
|
Apr 7 11:35:49 es5 mysqld[29230]: 2022-04-07 11:35:49 0 [Note] WSREP: Provider paused at 1d4206f9-647d-11ea-a963-0ed98924f90b:5064032832 (778520)
|
Apr 7 11:35:49 es5 mysqld[29230]: 2022-04-07 11:35:49 0 [Note] WSREP: Server paused at: 5064032832
|
Apr 7 11:35:49 es5 mysqld[29230]: 2022-04-07 11:35:49 0 [Note] WSREP: Tables flushed.
|
Apr 7 11:35:50 es5 mysqld[29230]: 2022-04-07 11:35:50 0 [Note] WSREP: (44508de1-af53, 'tcp://0.0.0.0:4567') turning message relay requesting off
|
Apr 7 11:39:01 es5 CRON[10599]: (root) CMD ( [ -x /usr/lib/php/sessionclean ] && if [ ! -d /run/systemd/system ]; then /usr/lib/php/sessionclean; fi)
|
Apr 7 11:39:05 es5 systemd[1]: Starting Clean php session files...
|
Apr 7 11:39:06 es5 systemd[1]: Started Clean php session files.
|
Apr 7 11:39:51 es5 mysqld[29230]: 2022-04-07 11:39:51 0 [Warning] InnoDB: A long semaphore wait:
|
Apr 7 11:39:51 es5 mysqld[29230]: --Thread 139702254601984 has waited at log0log.cc line 882 for 241.00 seconds the semaphore:
|
Apr 7 11:39:51 es5 mysqld[29230]: Mutex at 0x55fd5cf32080, Mutex LOG_WRITE created log0log.cc:537, lock var 2
|
Apr 7 11:39:51 es5 mysqld[29230]: 2022-04-07 11:39:51 0 [Note] InnoDB: A semaphore wait:
|
Apr 7 11:39:51 es5 mysqld[29230]: --Thread 139702254601984 has waited at log0log.cc line 882 for 241.00 seconds the semaphore:
|
Apr 7 11:39:51 es5 mysqld[29230]: Mutex at 0x55fd5cf32080, Mutex LOG_WRITE created log0log.cc:537, lock var 2
|
Apr 7 11:39:51 es5 mysqld[29230]: InnoDB: ###### Starts InnoDB Monitor for 30 secs to print diagnostic info:
|
Apr 7 11:39:51 es5 mysqld[29230]: InnoDB: Pending reads 0, writes 0
|
Apr 7 11:39:52 es5 mysqld[29230]: 2022-04-07 11:39:52 0 [Note] WSREP: resume
|
Apr 7 11:39:52 es5 mysqld[29230]: 2022-04-07 11:39:52 0 [Note] WSREP: resuming provider at 778520
|
Apr 7 11:39:52 es5 mysqld[29230]: 2022-04-07 11:39:52 0 [Note] WSREP: Provider resumed.
|
Apr 7 11:39:52 es5 mysqld[29230]: 2022-04-07 11:39:52 0 [Note] WSREP: SST sent: 1d4206f9-647d-11ea-a963-0ed98924f90b:5064032832
|
Apr 7 11:39:52 es5 mysqld[29230]: 2022-04-07 11:39:52 0 [Note] WSREP: Server status change donor -> joined
|
Apr 7 11:39:52 es5 mysqld[29230]: 2022-04-07 11:39:52 0 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
|
Apr 7 11:39:52 es5 mysqld[29230]: 2022-04-07 11:39:52 0 [Note] WSREP: Donor monitor thread ended with total time 243 sec
|
Apr 7 11:39:52 es5 mysqld[29230]: 2022-04-07 11:39:52 0 [Note] WSREP: 0.0 (ES5_AWS): State transfer to 1.0 (ES6_AWS) complete.
|
Apr 7 11:39:52 es5 mysqld[29230]: 2022-04-07 11:39:52 0 [Note] WSREP: Shifting DONOR/DESYNCED -> JOINED (TO: 5064034794)
|
Apr 7 11:39:52 es5 mysqld[29230]: 2022-04-07 11:39:52 0 [Note] WSREP: Member 0.0 (ES5_AWS) synced with group.
|
Apr 7 11:39:52 es5 mysqld[29230]: 2022-04-07 11:39:52 0 [Note] WSREP: Shifting JOINED -> SYNCED (TO: 5064034794)
|
Apr 7 11:39:52 es5 mysqld[29230]: 2022-04-07 11:39:52 2 [Note] WSREP: Server ES5_AWS synced with group
|
Apr 7 11:39:52 es5 mysqld[29230]: 2022-04-07 11:39:52 2 [Note] WSREP: Server status change joined -> synced
|
Apr 7 11:39:52 es5 mysqld[29230]: 2022-04-07 11:39:52 2 [Note] WSREP: Synchronized with group, ready for connections
|
Apr 7 11:39:52 es5 mysqld[29230]: 2022-04-07 11:39:52 2 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
|
Apr 7 11:39:53 es5 mysqld[29230]: 2022-04-07 11:39:53 0 [Note] WSREP: (44508de1-af53, 'tcp://0.0.0.0:4567') turning message relay requesting on, nonlive peers: tcp://172.31.39.121:4567
|
Apr 7 11:39:53 es5 mysqld[29230]: 2022-04-07 11:39:53 0 [Note] WSREP: declaring 95267912-98ce at tcp://172.31.34.27:4567 stable
|
Apr 7 11:39:53 es5 mysqld[29230]: 2022-04-07 11:39:53 0 [Note] WSREP: declaring 98319f4a-8b85 at tcp://172.31.35.130:4567 stable
|
Apr 7 11:39:53 es5 mysqld[29230]: 2022-04-07 11:39:53 0 [Note] WSREP: declaring da9e7474-8746 at tcp://172.31.40.66:4567 stable
|
Apr 7 11:39:53 es5 mysqld[29230]: 2022-04-07 11:39:53 0 [Note] WSREP: declaring e4e23eea-bbd0 at tcp://172.31.37.249:4567 stable
|
Apr 7 11:39:53 es5 mysqld[29230]: 2022-04-07 11:39:53 0 [Note] WSREP: declaring eb5f53de-8588 at tcp://172.31.9.176:4567 stable
|
Apr 7 11:39:53 es5 mysqld[29230]: 2022-04-07 11:39:53 0 [Note] WSREP: forgetting 7c67946e-b045 (tcp://172.31.39.121:4567)
|
Apr 7 11:39:53 es5 mysqld[29230]: 2022-04-07 11:39:53 0 [Note] WSREP: (44508de1-af53, 'tcp://0.0.0.0:4567') turning message relay requesting off
|
Apr 7 11:39:53 es5 mysqld[29230]: 2022-04-07 11:39:53 0 [Note] WSREP: Node 44508de1-af53 state prim
|
Apr 7 11:39:53 es5 mysqld[29230]: 2022-04-07 11:39:53 0 [Note] WSREP: view(view_id(PRIM,44508de1-af53,379) memb {
|
Apr 7 11:39:53 es5 mysqld[29230]: #01144508de1-af53,0
|
Apr 7 11:39:53 es5 mysqld[29230]: #01195267912-98ce,0
|
Apr 7 11:39:53 es5 mysqld[29230]: #01198319f4a-8b85,0
|
Apr 7 11:39:53 es5 mysqld[29230]: #011da9e7474-8746,0
|
Apr 7 11:39:53 es5 mysqld[29230]: #011e4e23eea-bbd0,0
|
Apr 7 11:39:53 es5 mysqld[29230]: #011eb5f53de-8588,0
|
Apr 7 11:39:53 es5 mysqld[29230]: } joined {
|
Apr 7 11:39:53 es5 mysqld[29230]: } left {
|
Apr 7 11:39:53 es5 mysqld[29230]: } partitioned {
|
Apr 7 11:39:53 es5 mysqld[29230]: #0117c67946e-b045,0
|
Apr 7 11:39:53 es5 mysqld[29230]: })
|
Apr 7 11:39:53 es5 mysqld[29230]: 2022-04-07 11:39:53 0 [Note] WSREP: save pc into disk
|
Apr 7 11:39:53 es5 mysqld[29230]: 2022-04-07 11:39:53 0 [Note] WSREP: forgetting 7c67946e-b045 (tcp://172.31.39.121:4567)
|
Apr 7 11:39:53 es5 mysqld[29230]: 2022-04-07 11:39:53 0 [Note] WSREP: New COMPONENT: primary = yes, bootstrap = no, my_idx = 0, memb_num = 6
|
Apr 7 11:39:53 es5 mysqld[29230]: 2022-04-07 11:39:53 0 [Note] WSREP: STATE_EXCHANGE: sent state UUID: 0f078c2c-b65f-11ec-b06d-de758fe27da8
|
Apr 7 11:39:53 es5 mysqld[29230]: 2022-04-07 11:39:53 0 [Note] WSREP: STATE EXCHANGE: sent state msg: 0f078c2c-b65f-11ec-b06d-de758fe27da8
|
Apr 7 11:39:53 es5 mysqld[29230]: 2022-04-07 11:39:53 0 [Note] WSREP: STATE EXCHANGE: got state msg: 0f078c2c-b65f-11ec-b06d-de758fe27da8 from 0 (ES5_AWS)
|
Apr 7 11:39:53 es5 mysqld[29230]: 2022-04-07 11:39:53 0 [Note] WSREP: STATE EXCHANGE: got state msg: 0f078c2c-b65f-11ec-b06d-de758fe27da8 from 1 (ES1_AWS)
|
Apr 7 11:39:53 es5 mysqld[29230]: 2022-04-07 11:39:53 0 [Note] WSREP: STATE EXCHANGE: got state msg: 0f078c2c-b65f-11ec-b06d-de758fe27da8 from 2 (ES2_AWS)
|
Apr 7 11:39:53 es5 mysqld[29230]: 2022-04-07 11:39:53 0 [Note] WSREP: STATE EXCHANGE: got state msg: 0f078c2c-b65f-11ec-b06d-de758fe27da8 from 3 (ES4_AWS)
|
Apr 7 11:39:53 es5 mysqld[29230]: 2022-04-07 11:39:53 0 [Note] WSREP: STATE EXCHANGE: got state msg: 0f078c2c-b65f-11ec-b06d-de758fe27da8 from 4 (ES3_AWS)
|
Apr 7 11:39:53 es5 mysqld[29230]: 2022-04-07 11:39:53 0 [Note] WSREP: STATE EXCHANGE: got state msg: 0f078c2c-b65f-11ec-b06d-de758fe27da8 from 5 (BACKUP_AWS)
|
Apr 7 11:39:53 es5 mysqld[29230]: 2022-04-07 11:39:53 0 [Note] WSREP: Quorum results:
|
Apr 7 11:39:53 es5 mysqld[29230]: #011version = 6,
|
Apr 7 11:39:53 es5 mysqld[29230]: #011component = PRIMARY,
|
Apr 7 11:39:53 es5 mysqld[29230]: #011conf_id = 364,
|
Apr 7 11:39:53 es5 mysqld[29230]: #011members = 6/6 (joined/total),
|
Apr 7 11:39:53 es5 mysqld[29230]: #011act_id = 5064034794,
|
Apr 7 11:39:53 es5 mysqld[29230]: #011last_appl. = 5064034689,
|
Apr 7 11:39:53 es5 mysqld[29230]: #011protocols = 2/10/4 (gcs/repl/appl),
|
Apr 7 11:39:53 es5 mysqld[29230]: #011vote policy= 0,
|
Apr 7 11:39:53 es5 mysqld[29230]: #011group UUID = 1d4206f9-647d-11ea-a963-0ed98924f90b
|
Apr 7 11:39:53 es5 mysqld[29230]: 2022-04-07 11:39:53 0 [Note] WSREP: Flow-control interval: [39, 39]
|
Apr 7 11:39:53 es5 mysqld[29230]: 2022-04-07 11:39:53 2 [Note] WSREP: ####### processing CC 5064034795, local, ordered
|
Apr 7 11:39:53 es5 mysqld[29230]: 2022-04-07 11:39:53 2 [Note] WSREP: ####### My UUID: 44508de1-b657-11ec-af53-dbbee42dcffe
|
Apr 7 11:39:53 es5 mysqld[29230]: 2022-04-07 11:39:53 2 [Note] WSREP: Skipping cert index reset
|
Apr 7 11:39:53 es5 mysqld[29230]: 2022-04-07 11:39:53 2 [Note] WSREP: REPL Protocols: 10 (5)
|
Apr 7 11:39:53 es5 mysqld[29230]: 2022-04-07 11:39:53 2 [Note] WSREP: ####### Adjusting cert position: 5064034794 -> 5064034795
|
Apr 7 11:39:53 es5 mysqld[29230]: 2022-04-07 11:39:53 0 [Note] WSREP: Service thread queue flushed.
|
Apr 7 11:39:53 es5 mysqld[29230]: 2022-04-07 11:39:53 2 [Note] WSREP: ================================================
|
Apr 7 11:39:53 es5 mysqld[29230]: View:
|
Apr 7 11:39:53 es5 mysqld[29230]: id: 1d4206f9-647d-11ea-a963-0ed98924f90b:5064034795
|
Apr 7 11:39:53 es5 mysqld[29230]: status: primary
|
Apr 7 11:39:53 es5 mysqld[29230]: protocol_version: 4
|
Apr 7 11:39:53 es5 mysqld[29230]: capabilities: MULTI-MASTER, CERTIFICATION, PARALLEL_APPLYING, REPLAY, ISOLATION, PAUSE, CAUSAL_READ, INCREMENTAL_WS, UNORDERED, PREORDERED, STREAMING, NBO
|
Apr 7 11:39:53 es5 mysqld[29230]: final: no
|
Apr 7 11:39:53 es5 mysqld[29230]: own_index: 0
|
Apr 7 11:39:53 es5 mysqld[29230]: members(6):
|
Apr 7 11:39:53 es5 mysqld[29230]: #0110: 44508de1-b657-11ec-af53-dbbee42dcffe, ES5_AWS
|
Apr 7 11:39:53 es5 mysqld[29230]: #0111: 95267912-5df1-11ec-98ce-a66d469d045c, ES1_AWS
|
Apr 7 11:39:53 es5 mysqld[29230]: #0112: 98319f4a-b5f9-11ec-8b85-e69cdab5035e, ES2_AWS
|
Apr 7 11:39:53 es5 mysqld[29230]: #0113: da9e7474-5df2-11ec-8746-dfd73835a1c5, ES4_AWS
|
Apr 7 11:39:53 es5 mysqld[29230]: #0114: e4e23eea-5df0-11ec-bbd0-8b132f61eb3d, ES3_AWS
|
Apr 7 11:39:53 es5 mysqld[29230]: #0115: eb5f53de-882b-11ec-8588-1a528d6537ef, BACKUP_AWS
|
Apr 7 11:39:53 es5 mysqld[29230]: =================================================
|
Apr 7 11:39:53 es5 mysqld[29230]: 2022-04-07 11:39:53 2 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
|
Apr 7 11:39:53 es5 mysqld[29230]: 2022-04-07 11:39:53 2 [Note] WSREP: Lowest cert index boundary for CC from group: 5064032727
|
Apr 7 11:39:53 es5 mysqld[29230]: 2022-04-07 11:39:53 2 [Note] WSREP: Min available from gcache for CC from group: 5063284333
|
Apr 7 11:39:53 es5 mysqld[29230]: 2022-04-07 11:39:53 0 [Note] WSREP: async IST sender served
|
Apr 7 11:39:56 es5 mysqld[29230]: 2022-04-07 11:39:56 0 [Note] WSREP: (44508de1-af53, 'tcp://0.0.0.0:4567') connection to peer eb5f53de-8588 with addr tcp://172.31.9.176:4567 timed out, no messages seen in PT3S, socket stats: rtt: 872 rttvar: 864 rto: 204000 lost: 0 last_data_recv: 3104 cwnd: 10 last_queued_since: 33142 last_delivered_since: 3103372544 send_queue_length: 1 send_queue_bytes: 804 segment: 0 messages: 1
|
Apr 7 11:39:56 es5 mysqld[29230]: 2022-04-07 11:39:56 0 [Note] WSREP: Deferred close timer started for socket with remote endpoint: tcp://172.31.9.176:60056
|
Apr 7 11:39:56 es5 mysqld[29230]: 2022-04-07 11:39:56 0 [Note] WSREP: (44508de1-af53, 'tcp://0.0.0.0:4567') turning message relay requesting on, nonlive peers: tcp://172.31.9.176:4567
|
Apr 7 11:39:56 es5 mysqld[29230]: 2022-04-07 11:39:56 0 [Note] WSREP: Deferred close timer handle_wait Operation aborted. for 0x55fd61a9f5b0
|
Apr 7 11:39:56 es5 mysqld[29230]: 2022-04-07 11:39:56 0 [Note] WSREP: Deferred close timer destruct
|
Attachments
Issue Links
- relates to
-
MDEV-12353 Efficient InnoDB redo log record format
-
- Closed
-
-
-
- Closed
-
-
-
- Closed
-
Activity
Hi Marko,
Thanks for the quick response. I hadn't actually realised our 10.4 nodes were not on the latest 10.4.x, I think our apt mirror was maybe out of date.
I can test with 10.4.24 as that seems to be available in the apt repo now I've changed the mirror, but looking at MDEV-27437 I think I'd need to wait for 10.4.25?
mattjones, the fix of MDEV-25975 is not present in any release yet, so unless you are willing to upgrade the donor to a development snapshot of 10.4, you would have to wait for the 10.4.25 release.
According to a comment in MDEV-27437, the 10.4 to 10.5 upgrade has been tested to work now.
I’m setting this to "needs feedback", waiting for your feedback from a test with 10.4.25 or a recent 10.4 development snapshot.
I've managed to get a new 10.4.x instance working after realising my problem was the separate bug MDEV-28260, so I don't need to test urgently enough to warrant trying to get this working right this second.
That said, as soon as 10.4.25 is released I'll give the upgrade another test.
mattjones, as noted in MDEV-27437, writes on the donor will not be disabled for wsrep_sst_method=mariabackup. That is, the fix of MDEV-25975 only helps wsrep_sst_method=rsync, which is what you are using. I will keep this ticket open until you have had a chance to test it.
mattjones, the 10.3.35, 10.4.25 and other release is finally out. Please give an update once you are able to test it.
Note: The donor node needs to be a new version; the joiner may be older. So, if you are upgrading from 10.3 or 10.4 to a later major version, you must first upgrade to 10.3.35 to 10.4.25 and use wsrep_sst_method=rsync.
The upgrade check was added in
MDEV-12353, and it should be correct. Deleting the log files is never a good idea. Starting withMDEV-27199in MariaDB 10.8, a log file is required to exist on startup.I think that
MDEV-25975could fix this. Are you able to test this when the donor is the latest development snapshot of 10.4?