Details
-
Bug
-
Status: Closed (View Workflow)
-
Major
-
Resolution: Incomplete
-
10.5.15
-
Ubuntu 20.04 LTS
Description
Hi, I'm trying to join a new 10.5 node to a 10.4 Galera cluster as part of a migration to new AWS instance types.
The symptom is that the initial SST completes, but on the restart the joining node then fails with this error:
Apr 7 10:39:52 ip-172-31-39-121 mariadbd[129747]: 2022-04-07 10:39:52 0 [ERROR] InnoDB: Upgrade after a crash is not supported. The redo log was created with MariaDB 10.4.22.
|
After various Googling this seems to be related to crash recovery files ib_logfile0 and ib_logfile1, which don't exist on the empty JOINING node until the SST completes. The DONOR node was cleanly restarted prior to the sync. I'm not sure if this is a known issue?
Things I have already tried:
- Setting innodb_fast_shutdown=0 on DONOR and JOINING nodes
- Deleting ib_logfile0 and ib_logfile1 on the JOINER and restarting
Is there a known work-around for this? I'm cautious to try upgrading an existing node to 10.5 in case it has the same error.
Galera Config
[mysqld]
|
binlog_format=ROW
|
default-storage-engine=innodb
|
innodb_autoinc_lock_mode=2
|
query_cache_type=0
|
query_cache_size=0
|
bind-address=0.0.0.0
|
|
|
[galera]
|
wsrep_on=ON
|
|
|
# Galera Provider Configuration
|
wsrep_provider=/usr/lib/galera/libgalera_smm.so
|
wsrep_provider_options="gcache.size=5G"
|
|
|
# Galera Cluster Configuration
|
wsrep_cluster_name="mariadb-cluster"
|
wsrep_cluster_address="gcomm://172.31.34.27,172.31.35.130,172.31.37.249,172.31.40.66,172.31.37.12"
|
|
|
# Galera Synchronization Congifuration
|
wsrep_sst_method = rsync
|
wsrep_sst_donor = ES5_AWS
|
|
|
# Galera Node Configuration
|
wsrep_node_name="ES6_AWS"
|
wsrep_node_address="172.31.39.121"
|
JOINER Logs
Apr 7 10:35:46 ip-172-31-39-121 systemd[1]: Starting MariaDB 10.5.15 database server...
|
Apr 7 10:35:47 ip-172-31-39-121 sh[129656]: WSREP: Recovered position 00000000-0000-0000-0000-000000000000:-1
|
Apr 7 10:35:47 ip-172-31-39-121 mariadbd[129747]: 2022-04-07 10:35:47 0 [Note] /usr/sbin/mariadbd (mysqld 10.5.15-MariaDB-1:10.5.15+maria~focal) starting as process 129747 ...
|
Apr 7 10:35:47 ip-172-31-39-121 mariadbd[129747]: 2022-04-07 10:35:47 0 [Note] WSREP: Loading provider /usr/lib/galera/libgalera_smm.so initial position: 00000000-0000-0000-0000-000000000000:-1
|
Apr 7 10:35:47 ip-172-31-39-121 mariadbd[129747]: 2022-04-07 10:35:47 0 [Note] WSREP: wsrep_load(): loading provider library '/usr/lib/galera/libgalera_smm.so'
|
Apr 7 10:35:47 ip-172-31-39-121 mariadbd[129747]: 2022-04-07 10:35:47 0 [Note] WSREP: wsrep_load(): Galera 26.4.11(r67341d07) by Codership Oy <info@codership.com> loaded successfully.
|
Apr 7 10:35:47 ip-172-31-39-121 mariadbd[129747]: 2022-04-07 10:35:47 0 [Note] WSREP: CRC-32C: using hardware acceleration.
|
Apr 7 10:35:47 ip-172-31-39-121 mariadbd[129747]: 2022-04-07 10:35:47 0 [Warning] WSREP: Could not open state file for reading: '/var/lib/mysql//grastate.dat'
|
Apr 7 10:35:47 ip-172-31-39-121 mariadbd[129747]: 2022-04-07 10:35:47 0 [Note] WSREP: Found saved state: 00000000-0000-0000-0000-000000000000:-1, safe_to_bootstrap: 1
|
Apr 7 10:35:47 ip-172-31-39-121 mariadbd[129747]: 2022-04-07 10:35:47 0 [Note] WSREP: GCache DEBUG: opened preamble:
|
Apr 7 10:35:47 ip-172-31-39-121 mariadbd[129747]: Version: 0
|
Apr 7 10:35:47 ip-172-31-39-121 mariadbd[129747]: UUID: 00000000-0000-0000-0000-000000000000
|
Apr 7 10:35:47 ip-172-31-39-121 mariadbd[129747]: Seqno: -1 - -1
|
Apr 7 10:35:47 ip-172-31-39-121 mariadbd[129747]: Offset: -1
|
Apr 7 10:35:47 ip-172-31-39-121 mariadbd[129747]: Synced: 0
|
Apr 7 10:35:47 ip-172-31-39-121 mariadbd[129747]: 2022-04-07 10:35:47 0 [Note] WSREP: Skipped GCache ring buffer recovery: could not determine history UUID.
|
Apr 7 10:35:47 ip-172-31-39-121 mariadbd[129747]: 2022-04-07 10:35:47 0 [Warning] WSREP: Option 'gcs.fc_master_slave' is deprecated and will be removed in the future versions, please use 'gcs.fc_single_primary' instead.
|
Apr 7 10:35:47 ip-172-31-39-121 mariadbd[129747]: 2022-04-07 10:35:47 0 [Note] WSREP: Passing config to GCS: base_dir = /var/lib/mysql/; base_host = 172.31.39.121; base_port = 4567; cert.log_conflicts = no; cert.optimistic_pa = yes; debug = no; evs.auto_evict = 0; evs.delay_margin = PT1S; evs.delayed_keep_period = PT30S; evs.inactive_check_period = PT0.5S; evs.inactive_timeout = PT15S; evs.join_retrans_period = PT1S; evs.max_install_timeouts = 3; evs.send_window = 4; evs.stats_report_period = PT1M; evs.suspect_timeout = PT5S; evs.user_send_window = 2; evs.view_forget_timeout = PT24H; gcache.dir = /var/lib/mysql/; gcache.keep_pages_size = 0; gcache.keep_plaintext_size = 128M; gcache.mem_size = 0; gcache.name = galera.cache; gcache.page_size = 128M; gcache.recover = yes; gcache.size = 5G; gcomm.thread_prio = ; gcs.fc_debug = 0; gcs.fc_factor = 1.0; gcs.fc_limit = 16; gcs.fc_master_slave = no; gcs.fc_single_primary = no; gcs.max_packet_size = 64500; gcs.max_throttle = 0.25; gcs.recv_q_hard_limit = 9223372036854775807; gcs.recv_q_soft_limit = 0.25; gcs.sync_donor = no; gmcast.segment = 0;
|
Apr 7 10:35:47 ip-172-31-39-121 mariadbd[129747]: 2022-04-07 10:35:47 0 [Note] WSREP: Start replication
|
Apr 7 10:35:47 ip-172-31-39-121 mariadbd[129747]: 2022-04-07 10:35:47 0 [Note] WSREP: Connecting with bootstrap option: 0
|
Apr 7 10:35:47 ip-172-31-39-121 mariadbd[129747]: 2022-04-07 10:35:47 0 [Note] WSREP: Setting GCS initial position to 00000000-0000-0000-0000-000000000000:-1
|
Apr 7 10:35:47 ip-172-31-39-121 mariadbd[129747]: 2022-04-07 10:35:47 0 [Note] WSREP: protonet asio version 0
|
Apr 7 10:35:47 ip-172-31-39-121 mariadbd[129747]: 2022-04-07 10:35:47 0 [Note] WSREP: Using CRC-32C for message checksums.
|
Apr 7 10:35:47 ip-172-31-39-121 mariadbd[129747]: 2022-04-07 10:35:47 0 [Note] WSREP: backend: asio
|
Apr 7 10:35:47 ip-172-31-39-121 mariadbd[129747]: 2022-04-07 10:35:47 0 [Note] WSREP: gcomm thread scheduling priority set to other:0
|
Apr 7 10:35:47 ip-172-31-39-121 mariadbd[129747]: 2022-04-07 10:35:47 0 [Warning] WSREP: access file(/var/lib/mysql//gvwstate.dat) failed(No such file or directory)
|
Apr 7 10:35:47 ip-172-31-39-121 mariadbd[129747]: 2022-04-07 10:35:47 0 [Note] WSREP: restore pc from disk failed
|
Apr 7 10:35:47 ip-172-31-39-121 mariadbd[129747]: 2022-04-07 10:35:47 0 [Note] WSREP: GMCast version 0
|
Apr 7 10:35:47 ip-172-31-39-121 mariadbd[129747]: 2022-04-07 10:35:47 0 [Note] WSREP: (7c67946e-b045, 'tcp://0.0.0.0:4567') listening at tcp://0.0.0.0:4567
|
Apr 7 10:35:47 ip-172-31-39-121 mariadbd[129747]: 2022-04-07 10:35:47 0 [Note] WSREP: (7c67946e-b045, 'tcp://0.0.0.0:4567') multicast: , ttl: 1
|
Apr 7 10:35:47 ip-172-31-39-121 mariadbd[129747]: 2022-04-07 10:35:47 0 [Note] WSREP: EVS version 1
|
Apr 7 10:35:47 ip-172-31-39-121 mariadbd[129747]: 2022-04-07 10:35:47 0 [Note] WSREP: gcomm: connecting to group 'mariadb-cluster', peer '172.31.34.27:,172.31.35.130:,172.31.37.249:,172.31.40.66:,172.31.37.12:'
|
Apr 7 10:35:47 ip-172-31-39-121 mariadbd[129747]: 2022-04-07 10:35:47 0 [Note] WSREP: (7c67946e-b045, 'tcp://0.0.0.0:4567') connection established to 95267912-98ce tcp://172.31.34.27:4567
|
Apr 7 10:35:47 ip-172-31-39-121 mariadbd[129747]: 2022-04-07 10:35:47 0 [Note] WSREP: (7c67946e-b045, 'tcp://0.0.0.0:4567') turning message relay requesting on, nonlive peers: tcp://172.31.37.75:4567 tcp://172.31.9.176:4567
|
Apr 7 10:35:47 ip-172-31-39-121 mariadbd[129747]: 2022-04-07 10:35:47 0 [Note] WSREP: (7c67946e-b045, 'tcp://0.0.0.0:4567') connection established to 98319f4a-8b85 tcp://172.31.35.130:4567
|
Apr 7 10:35:47 ip-172-31-39-121 mariadbd[129747]: 2022-04-07 10:35:47 0 [Note] WSREP: (7c67946e-b045, 'tcp://0.0.0.0:4567') connection established to da9e7474-8746 tcp://172.31.40.66:4567
|
Apr 7 10:35:47 ip-172-31-39-121 mariadbd[129747]: 2022-04-07 10:35:47 0 [Note] WSREP: (7c67946e-b045, 'tcp://0.0.0.0:4567') connection established to e4e23eea-bbd0 tcp://172.31.37.249:4567
|
Apr 7 10:35:47 ip-172-31-39-121 mariadbd[129747]: 2022-04-07 10:35:47 0 [Note] WSREP: (7c67946e-b045, 'tcp://0.0.0.0:4567') connection established to 44508de1-af53 tcp://172.31.37.75:4567
|
Apr 7 10:35:47 ip-172-31-39-121 mariadbd[129747]: 2022-04-07 10:35:47 0 [Note] WSREP: (7c67946e-b045, 'tcp://0.0.0.0:4567') connection established to eb5f53de-8588 tcp://172.31.9.176:4567
|
Apr 7 10:35:47 ip-172-31-39-121 mariadbd[129747]: 2022-04-07 10:35:47 0 [Note] WSREP: (7c67946e-b045, 'tcp://0.0.0.0:4567') connection established to eb5f53de-8588 tcp://172.31.9.176:4567
|
Apr 7 10:35:47 ip-172-31-39-121 mariadbd[129747]: 2022-04-07 10:35:47 0 [Note] WSREP: Deferred close timer started for socket with remote endpoint: tcp://172.31.9.176:39010
|
Apr 7 10:35:47 ip-172-31-39-121 mariadbd[129747]: 2022-04-07 10:35:47 0 [Note] WSREP: Deferred close timer handle_wait Operation aborted. for 0xaaab09f1f940
|
Apr 7 10:35:47 ip-172-31-39-121 mariadbd[129747]: 2022-04-07 10:35:47 0 [Note] WSREP: Deferred close timer destruct
|
Apr 7 10:35:48 ip-172-31-39-121 mariadbd[129747]: 2022-04-07 10:35:48 0 [Note] WSREP: EVS version upgrade 0 -> 1
|
Apr 7 10:35:48 ip-172-31-39-121 mariadbd[129747]: 2022-04-07 10:35:48 0 [Note] WSREP: declaring 44508de1-af53 at tcp://172.31.37.75:4567 stable
|
Apr 7 10:35:48 ip-172-31-39-121 mariadbd[129747]: 2022-04-07 10:35:48 0 [Note] WSREP: declaring 95267912-98ce at tcp://172.31.34.27:4567 stable
|
Apr 7 10:35:48 ip-172-31-39-121 mariadbd[129747]: 2022-04-07 10:35:48 0 [Note] WSREP: declaring 98319f4a-8b85 at tcp://172.31.35.130:4567 stable
|
Apr 7 10:35:48 ip-172-31-39-121 mariadbd[129747]: 2022-04-07 10:35:48 0 [Note] WSREP: declaring da9e7474-8746 at tcp://172.31.40.66:4567 stable
|
Apr 7 10:35:48 ip-172-31-39-121 mariadbd[129747]: 2022-04-07 10:35:48 0 [Note] WSREP: declaring e4e23eea-bbd0 at tcp://172.31.37.249:4567 stable
|
Apr 7 10:35:48 ip-172-31-39-121 mariadbd[129747]: 2022-04-07 10:35:48 0 [Note] WSREP: declaring eb5f53de-8588 at tcp://172.31.9.176:4567 stable
|
Apr 7 10:35:48 ip-172-31-39-121 mariadbd[129747]: 2022-04-07 10:35:48 0 [Note] WSREP: PC protocol upgrade 0 -> 1
|
Apr 7 10:35:49 ip-172-31-39-121 mariadbd[129747]: 2022-04-07 10:35:49 0 [Note] WSREP: Node 44508de1-af53 state prim
|
Apr 7 10:35:49 ip-172-31-39-121 mariadbd[129747]: 2022-04-07 10:35:49 0 [Note] WSREP: view(view_id(PRIM,44508de1-af53,378) memb {
|
Apr 7 10:35:49 ip-172-31-39-121 mariadbd[129747]: #01144508de1-af53,0
|
Apr 7 10:35:49 ip-172-31-39-121 mariadbd[129747]: #0117c67946e-b045,0
|
Apr 7 10:35:49 ip-172-31-39-121 mariadbd[129747]: #01195267912-98ce,0
|
Apr 7 10:35:49 ip-172-31-39-121 mariadbd[129747]: #01198319f4a-8b85,0
|
Apr 7 10:35:49 ip-172-31-39-121 mariadbd[129747]: #011da9e7474-8746,0
|
Apr 7 10:35:49 ip-172-31-39-121 mariadbd[129747]: #011e4e23eea-bbd0,0
|
Apr 7 10:35:49 ip-172-31-39-121 mariadbd[129747]: #011eb5f53de-8588,0
|
Apr 7 10:35:49 ip-172-31-39-121 mariadbd[129747]: } joined {
|
Apr 7 10:35:49 ip-172-31-39-121 mariadbd[129747]: } left {
|
Apr 7 10:35:49 ip-172-31-39-121 mariadbd[129747]: } partitioned {
|
Apr 7 10:35:49 ip-172-31-39-121 mariadbd[129747]: })
|
Apr 7 10:35:49 ip-172-31-39-121 mariadbd[129747]: 2022-04-07 10:35:49 0 [Note] WSREP: save pc into disk
|
Apr 7 10:35:49 ip-172-31-39-121 mariadbd[129747]: 2022-04-07 10:35:49 0 [Note] WSREP: discarding pending addr without UUID: tcp://172.31.37.12:4567
|
Apr 7 10:35:49 ip-172-31-39-121 mariadbd[129747]: 2022-04-07 10:35:49 0 [Note] WSREP: discarding pending addr proto entry 0xaaab09eff5a0
|
Apr 7 10:35:49 ip-172-31-39-121 mariadbd[129747]: 2022-04-07 10:35:49 0 [Note] WSREP: gcomm: connected
|
Apr 7 10:35:49 ip-172-31-39-121 mariadbd[129747]: 2022-04-07 10:35:49 0 [Note] WSREP: Changing maximum packet size to 64500, resulting msg size: 32636
|
Apr 7 10:35:49 ip-172-31-39-121 mariadbd[129747]: 2022-04-07 10:35:49 0 [Note] WSREP: Shifting CLOSED -> OPEN (TO: 0)
|
Apr 7 10:35:49 ip-172-31-39-121 mariadbd[129747]: 2022-04-07 10:35:49 0 [Note] WSREP: Opened channel 'mariadb-cluster'
|
Apr 7 10:35:49 ip-172-31-39-121 mariadbd[129747]: 2022-04-07 10:35:49 0 [Note] WSREP: New COMPONENT: primary = yes, bootstrap = no, my_idx = 1, memb_num = 7
|
Apr 7 10:35:49 ip-172-31-39-121 mariadbd[129747]: 2022-04-07 10:35:49 0 [Note] WSREP: STATE EXCHANGE: Waiting for state UUID.
|
Apr 7 10:35:49 ip-172-31-39-121 mariadbd[129747]: 2022-04-07 10:35:49 0 [Note] WSREP: STATE EXCHANGE: sent state msg: 7dcd956b-b65e-11ec-8734-924b8afeca4d
|
Apr 7 10:35:49 ip-172-31-39-121 mariadbd[129747]: 2022-04-07 10:35:49 0 [Note] WSREP: STATE EXCHANGE: got state msg: 7dcd956b-b65e-11ec-8734-924b8afeca4d from 0 (ES5_AWS)
|
Apr 7 10:35:49 ip-172-31-39-121 mariadbd[129747]: 2022-04-07 10:35:49 0 [Note] WSREP: STATE EXCHANGE: got state msg: 7dcd956b-b65e-11ec-8734-924b8afeca4d from 2 (ES1_AWS)
|
Apr 7 10:35:49 ip-172-31-39-121 mariadbd[129747]: 2022-04-07 10:35:49 0 [Note] WSREP: STATE EXCHANGE: got state msg: 7dcd956b-b65e-11ec-8734-924b8afeca4d from 3 (ES2_AWS)
|
Apr 7 10:35:49 ip-172-31-39-121 mariadbd[129747]: 2022-04-07 10:35:49 0 [Note] WSREP: STATE EXCHANGE: got state msg: 7dcd956b-b65e-11ec-8734-924b8afeca4d from 4 (ES4_AWS)
|
Apr 7 10:35:49 ip-172-31-39-121 mariadbd[129747]: 2022-04-07 10:35:49 0 [Note] WSREP: STATE EXCHANGE: got state msg: 7dcd956b-b65e-11ec-8734-924b8afeca4d from 5 (ES3_AWS)
|
Apr 7 10:35:49 ip-172-31-39-121 mariadbd[129747]: 2022-04-07 10:35:49 1 [Note] WSREP: Starting rollbacker thread 1
|
Apr 7 10:35:49 ip-172-31-39-121 mariadbd[129747]: 2022-04-07 10:35:49 2 [Note] WSREP: Starting applier thread 2
|
Apr 7 10:35:49 ip-172-31-39-121 mariadbd[129747]: 2022-04-07 10:35:49 0 [Note] WSREP: STATE EXCHANGE: got state msg: 7dcd956b-b65e-11ec-8734-924b8afeca4d from 6 (BACKUP_AWS)
|
Apr 7 10:35:49 ip-172-31-39-121 mariadbd[129747]: 2022-04-07 10:35:49 0 [Note] WSREP: STATE EXCHANGE: got state msg: 7dcd956b-b65e-11ec-8734-924b8afeca4d from 1 (ES6_AWS)
|
Apr 7 10:35:49 ip-172-31-39-121 mariadbd[129747]: 2022-04-07 10:35:49 0 [Note] WSREP: Quorum results:
|
Apr 7 10:35:49 ip-172-31-39-121 mariadbd[129747]: #011version = 6,
|
Apr 7 10:35:49 ip-172-31-39-121 mariadbd[129747]: #011component = PRIMARY,
|
Apr 7 10:35:49 ip-172-31-39-121 mariadbd[129747]: #011conf_id = 363,
|
Apr 7 10:35:49 ip-172-31-39-121 mariadbd[129747]: #011members = 6/7 (joined/total),
|
Apr 7 10:35:49 ip-172-31-39-121 mariadbd[129747]: #011act_id = 5064032831,
|
Apr 7 10:35:49 ip-172-31-39-121 mariadbd[129747]: #011last_appl. = 5064032726,
|
Apr 7 10:35:49 ip-172-31-39-121 mariadbd[129747]: #011protocols = 2/10/4 (gcs/repl/appl),
|
Apr 7 10:35:49 ip-172-31-39-121 mariadbd[129747]: #011vote policy= 0,
|
Apr 7 10:35:49 ip-172-31-39-121 mariadbd[129747]: #011group UUID = 1d4206f9-647d-11ea-a963-0ed98924f90b
|
Apr 7 10:35:49 ip-172-31-39-121 mariadbd[129747]: 2022-04-07 10:35:49 0 [Note] WSREP: Flow-control interval: [42, 42]
|
Apr 7 10:35:49 ip-172-31-39-121 mariadbd[129747]: 2022-04-07 10:35:49 0 [Note] WSREP: Shifting OPEN -> PRIMARY (TO: 5064032832)
|
Apr 7 10:35:49 ip-172-31-39-121 mariadbd[129747]: 2022-04-07 10:35:49 2 [Note] WSREP: ####### processing CC 5064032832, local, ordered
|
Apr 7 10:35:49 ip-172-31-39-121 mariadbd[129747]: 2022-04-07 10:35:49 2 [Note] WSREP: Process first view: 1d4206f9-647d-11ea-a963-0ed98924f90b my uuid: 7c67946e-b65e-11ec-b045-eb2b92474255
|
Apr 7 10:35:49 ip-172-31-39-121 mariadbd[129747]: 2022-04-07 10:35:49 2 [Note] WSREP: Server ES6_AWS connected to cluster at position 1d4206f9-647d-11ea-a963-0ed98924f90b:5064032832 with ID 7c67946e-b65e-11ec-b045-eb2b92474255
|
Apr 7 10:35:49 ip-172-31-39-121 mariadbd[129747]: 2022-04-07 10:35:49 2 [Note] WSREP: Server status change disconnected -> connected
|
Apr 7 10:35:49 ip-172-31-39-121 mariadbd[129747]: 2022-04-07 10:35:49 2 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
|
Apr 7 10:35:49 ip-172-31-39-121 mariadbd[129747]: 2022-04-07 10:35:49 2 [Note] WSREP: ####### My UUID: 7c67946e-b65e-11ec-b045-eb2b92474255
|
Apr 7 10:35:49 ip-172-31-39-121 mariadbd[129747]: 2022-04-07 10:35:49 2 [Note] WSREP: Cert index reset to 00000000-0000-0000-0000-000000000000:-1 (proto: 10), state transfer needed: yes
|
Apr 7 10:35:49 ip-172-31-39-121 mariadbd[129747]: 2022-04-07 10:35:49 0 [Note] WSREP: Service thread queue flushed.
|
Apr 7 10:35:49 ip-172-31-39-121 mariadbd[129747]: 2022-04-07 10:35:49 2 [Note] WSREP: ####### Assign initial position for certification: 00000000-0000-0000-0000-000000000000:-1, protocol version: -1
|
Apr 7 10:35:49 ip-172-31-39-121 mariadbd[129747]: 2022-04-07 10:35:49 2 [Note] WSREP: State transfer required:
|
Apr 7 10:35:49 ip-172-31-39-121 mariadbd[129747]: #011Group state: 1d4206f9-647d-11ea-a963-0ed98924f90b:5064032832
|
Apr 7 10:35:49 ip-172-31-39-121 mariadbd[129747]: #011Local state: 00000000-0000-0000-0000-000000000000:-1
|
Apr 7 10:35:49 ip-172-31-39-121 mariadbd[129747]: 2022-04-07 10:35:49 2 [Note] WSREP: Server status change connected -> joiner
|
Apr 7 10:35:49 ip-172-31-39-121 mariadbd[129747]: 2022-04-07 10:35:49 2 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
|
Apr 7 10:35:49 ip-172-31-39-121 mariadbd[129747]: 2022-04-07 10:35:49 0 [Note] WSREP: Joiner monitor thread started to monitor
|
Apr 7 10:35:49 ip-172-31-39-121 mariadbd[129747]: 2022-04-07 10:35:49 0 [Note] WSREP: Running: 'wsrep_sst_rsync --role 'joiner' --address '172.31.39.121' --datadir '/var/lib/mysql/' --parent '129747' --mysqld-args --wsrep_start_position=00000000-0000-0000-0000-000000000000:-1'
|
Apr 7 10:35:49 ip-172-31-39-121 rsyncd[129866]: rsyncd version 3.1.3 starting, listening on port 4444
|
Apr 7 10:35:49 ip-172-31-39-121 mariadbd[129747]: 2022-04-07 10:35:49 2 [Note] WSREP: ####### IST uuid:00000000-0000-0000-0000-000000000000 f: 0, l: 5064032832, STRv: 3
|
Apr 7 10:35:49 ip-172-31-39-121 mariadbd[129747]: 2022-04-07 10:35:49 2 [Note] WSREP: IST receiver addr using tcp://172.31.39.121:4568
|
Apr 7 10:35:49 ip-172-31-39-121 mariadbd[129747]: 2022-04-07 10:35:49 2 [Note] WSREP: Prepared IST receiver for 0-5064032832, listening at: tcp://172.31.39.121:4568
|
Apr 7 10:35:49 ip-172-31-39-121 mariadbd[129747]: 2022-04-07 10:35:49 0 [Note] WSREP: Member 1.0 (ES6_AWS) requested state transfer from 'ES5_AWS'. Selected 0.0 (ES5_AWS)(SYNCED) as donor.
|
Apr 7 10:35:49 ip-172-31-39-121 mariadbd[129747]: 2022-04-07 10:35:49 0 [Note] WSREP: Shifting PRIMARY -> JOINER (TO: 5064032832)
|
Apr 7 10:35:49 ip-172-31-39-121 mariadbd[129747]: 2022-04-07 10:35:49 2 [Note] WSREP: Requesting state transfer: success, donor: 0
|
Apr 7 10:35:49 ip-172-31-39-121 mariadbd[129747]: 2022-04-07 10:35:49 2 [Note] WSREP: Resetting GCache seqno map due to different histories.
|
Apr 7 10:35:49 ip-172-31-39-121 mariadbd[129747]: 2022-04-07 10:35:49 2 [Note] WSREP: GCache history reset: 00000000-0000-0000-0000-000000000000:0 -> 1d4206f9-647d-11ea-a963-0ed98924f90b:5064032832
|
Apr 7 10:35:50 ip-172-31-39-121 rsyncd[129878]: connect from ip-172-31-37-75.eu-west-1.compute.internal (172.31.37.75)
|
Apr 7 10:35:50 ip-172-31-39-121 rsyncd[129878]: rsync to rsync_sst/ from ip-172-31-37-75.eu-west-1.compute.internal (172.31.37.75)
|
|
|
...
|
|
|
Apr 7 10:39:52 ip-172-31-39-121 mariadbd[129747]: 2022-04-07 10:39:52 0 [Note] WSREP: 0.0 (ES5_AWS): State transfer to 1.0 (ES6_AWS) complete.
|
Apr 7 10:39:52 ip-172-31-39-121 mariadbd[129757]: WSREP_SST: [INFO] Joiner cleanup: rsync PID=129866, stunnel PID=0 (20220407 10:39:52.306)
|
Apr 7 10:39:52 ip-172-31-39-121 mariadbd[129747]: 2022-04-07 10:39:52 0 [Note] WSREP: Member 0.0 (ES5_AWS) synced with group.
|
Apr 7 10:39:52 ip-172-31-39-121 rsyncd[129866]: sent 0 bytes received 0 bytes total size 0
|
Apr 7 10:39:52 ip-172-31-39-121 mariadbd[129757]: WSREP_SST: [INFO] Joiner cleanup done. (20220407 10:39:52.825)
|
Apr 7 10:39:52 ip-172-31-39-121 mariadbd[129747]: 2022-04-07 10:39:52 3 [Note] WSREP: SST received
|
Apr 7 10:39:52 ip-172-31-39-121 mariadbd[129747]: 2022-04-07 10:39:52 3 [Note] WSREP: Server status change joiner -> initializing
|
Apr 7 10:39:52 ip-172-31-39-121 mariadbd[129747]: 2022-04-07 10:39:52 3 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
|
Apr 7 10:39:52 ip-172-31-39-121 mariadbd[129747]: 2022-04-07 10:39:52 0 [Note] InnoDB: Uses event mutexes
|
Apr 7 10:39:52 ip-172-31-39-121 mariadbd[129747]: 2022-04-07 10:39:52 0 [Note] InnoDB: Compressed tables use zlib 1.2.11
|
Apr 7 10:39:52 ip-172-31-39-121 mariadbd[129747]: 2022-04-07 10:39:52 0 [Note] InnoDB: Number of pools: 1
|
Apr 7 10:39:52 ip-172-31-39-121 mariadbd[129747]: 2022-04-07 10:39:52 0 [Note] InnoDB: Using ARMv8 crc32 + pmull instructions
|
Apr 7 10:39:52 ip-172-31-39-121 mariadbd[129747]: 2022-04-07 10:39:52 0 [Note] InnoDB: Using Linux native AIO
|
Apr 7 10:39:52 ip-172-31-39-121 mariadbd[129747]: 2022-04-07 10:39:52 0 [Note] InnoDB: Initializing buffer pool, total size = 134217728, chunk size = 134217728
|
Apr 7 10:39:52 ip-172-31-39-121 mariadbd[129747]: 2022-04-07 10:39:52 0 [Note] InnoDB: Completed initialization of buffer pool
|
Apr 7 10:39:52 ip-172-31-39-121 mariadbd[129747]: 2022-04-07 10:39:52 0 [ERROR] InnoDB: Upgrade after a crash is not supported. The redo log was created with MariaDB 10.4.22.
|
Apr 7 10:39:52 ip-172-31-39-121 mariadbd[129747]: 2022-04-07 10:39:52 0 [ERROR] InnoDB: Plugin initialization aborted with error Generic error
|
Apr 7 10:39:52 ip-172-31-39-121 mariadbd[129747]: 2022-04-07 10:39:52 0 [Note] InnoDB: Starting shutdown...
|
Apr 7 10:39:53 ip-172-31-39-121 mariadbd[129747]: 2022-04-07 10:39:53 0 [ERROR] Plugin 'InnoDB' init function returned error.
|
Apr 7 10:39:53 ip-172-31-39-121 mariadbd[129747]: 2022-04-07 10:39:53 0 [ERROR] Plugin 'InnoDB' registration as a STORAGE ENGINE failed.
|
Apr 7 10:39:53 ip-172-31-39-121 mariadbd[129747]: 2022-04-07 10:39:53 0 [Note] Plugin 'FEEDBACK' is disabled.
|
Apr 7 10:39:53 ip-172-31-39-121 mariadbd[129747]: 2022-04-07 10:39:53 0 [ERROR] Unknown/unsupported storage engine: innodb
|
Apr 7 10:39:53 ip-172-31-39-121 mariadbd[129747]: 2022-04-07 10:39:53 0 [ERROR] Aborting
|
Apr 7 10:39:53 ip-172-31-39-121 mariadbd[129747]: terminate called after throwing an instance of 'wsrep::runtime_error'
|
Apr 7 10:39:53 ip-172-31-39-121 mariadbd[129747]: what(): State wait was interrupted
|
Apr 7 10:39:53 ip-172-31-39-121 mariadbd[129747]: 220407 10:39:53 [ERROR] mysqld got signal 6 ;
|
Apr 7 10:39:53 ip-172-31-39-121 mariadbd[129747]: This could be because you hit a bug. It is also possible that this binary
|
Apr 7 10:39:53 ip-172-31-39-121 mariadbd[129747]: or one of the libraries it was linked against is corrupt, improperly built,
|
Apr 7 10:39:53 ip-172-31-39-121 mariadbd[129747]: or misconfigured. This error can also be caused by malfunctioning hardware.
|
Apr 7 10:39:53 ip-172-31-39-121 mariadbd[129747]: To report this bug, see https://mariadb.com/kb/en/reporting-bugs
|
Apr 7 10:39:53 ip-172-31-39-121 mariadbd[129747]: We will try our best to scrape up some info that will hopefully help
|
Apr 7 10:39:53 ip-172-31-39-121 mariadbd[129747]: diagnose the problem, but since we have already crashed,
|
Apr 7 10:39:53 ip-172-31-39-121 mariadbd[129747]: something is definitely wrong and this may fail.
|
Apr 7 10:39:53 ip-172-31-39-121 mariadbd[129747]: Server version: 10.5.15-MariaDB-1:10.5.15+maria~focal
|
Apr 7 10:39:53 ip-172-31-39-121 mariadbd[129747]: key_buffer_size=134217728
|
Apr 7 10:39:53 ip-172-31-39-121 mariadbd[129747]: read_buffer_size=131072
|
Apr 7 10:39:53 ip-172-31-39-121 mariadbd[129747]: max_used_connections=0
|
Apr 7 10:39:53 ip-172-31-39-121 mariadbd[129747]: max_threads=153
|
Apr 7 10:39:53 ip-172-31-39-121 mariadbd[129747]: thread_count=4
|
Apr 7 10:39:53 ip-172-31-39-121 mariadbd[129747]: It is possible that mysqld could use up to
|
Apr 7 10:39:53 ip-172-31-39-121 mariadbd[129747]: key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 467879 K bytes of memory
|
Apr 7 10:39:53 ip-172-31-39-121 mariadbd[129747]: Hope that's ok; if not, decrease some variables in the equation.
|
Apr 7 10:39:53 ip-172-31-39-121 mariadbd[129747]: Thread pointer: 0xfffe1c002068
|
Apr 7 10:39:53 ip-172-31-39-121 mariadbd[129747]: Attempting backtrace. You can use the following information to find out
|
Apr 7 10:39:53 ip-172-31-39-121 mariadbd[129747]: where mysqld died. If you see no messages after this, something went
|
Apr 7 10:39:53 ip-172-31-39-121 mariadbd[129747]: terribly wrong...
|
Apr 7 10:39:53 ip-172-31-39-121 mariadbd[129747]: stack_bottom = 0xfffe42579630 thread_stack 0x49000
|
Apr 7 10:39:53 ip-172-31-39-121 mariadbd[129747]: Printing to addr2line failed
|
Apr 7 10:39:53 ip-172-31-39-121 mariadbd[129747]: /usr/sbin/mariadbd(my_print_stacktrace+0x30)[0xaaaacc7023f0]
|
Apr 7 10:39:53 ip-172-31-39-121 mariadbd[129747]: /usr/sbin/mariadbd(handle_fatal_signal+0x45c)[0xaaaacc1b137c]
|
Apr 7 10:39:53 ip-172-31-39-121 mariadbd[129747]: linux-vdso.so.1(__kernel_rt_sigreturn+0x0)[0xffff89d95794]
|
Apr 7 10:39:53 ip-172-31-39-121 mariadbd[129747]: /lib/aarch64-linux-gnu/atomics/libc.so.6(gsignal+0xe0)[0xffff8942dc20]
|
Apr 7 10:39:53 ip-172-31-39-121 mariadbd[129747]: /lib/aarch64-linux-gnu/atomics/libc.so.6(abort+0xfc)[0xffff8941aa14]
|
Apr 7 10:39:53 ip-172-31-39-121 mariadbd[129747]: /lib/aarch64-linux-gnu/libstdc++.so.6(_ZN9__gnu_cxx27__verbose_terminate_handlerEv+0x19c)[0xffff896b88cc]
|
Apr 7 10:39:53 ip-172-31-39-121 mariadbd[129747]: /lib/aarch64-linux-gnu/libstdc++.so.6(+0x9f21c)[0xffff896b621c]
|
Apr 7 10:39:53 ip-172-31-39-121 mariadbd[129747]: /lib/aarch64-linux-gnu/libstdc++.so.6(+0x9f280)[0xffff896b6280]
|
Apr 7 10:39:53 ip-172-31-39-121 mariadbd[129747]: /lib/aarch64-linux-gnu/libstdc++.so.6(+0x9f574)[0xffff896b6574]
|
Apr 7 10:39:53 ip-172-31-39-121 mariadbd[129747]: /usr/sbin/mariadbd(_ZNK5wsrep12server_state16wait_until_stateERSt11unique_lockINS_5mutexEENS0_5stateE+0x1dc)[0xaaaacc7808ec]
|
Apr 7 10:39:53 ip-172-31-39-121 mariadbd[129747]: /usr/sbin/mariadbd(_ZN5wsrep12server_state12sst_receivedERNS_14client_serviceEi+0xb04)[0xaaaacc78a5f4]
|
Apr 7 10:39:53 ip-172-31-39-121 mariadbd[129747]: /usr/sbin/mariadbd(+0xb7b6bc)[0xaaaacc43b6bc]
|
Apr 7 10:39:53 ip-172-31-39-121 mariadbd[129747]: /usr/sbin/mariadbd(+0xb7c324)[0xaaaacc43c324]
|
Apr 7 10:39:53 ip-172-31-39-121 mariadbd[129747]: /usr/sbin/mariadbd(+0xb0bca0)[0xaaaacc3cbca0]
|
Apr 7 10:39:53 ip-172-31-39-121 mariadbd[129747]: /lib/aarch64-linux-gnu/atomics/libpthread.so.0(+0x73f0)[0xffff898173f0]
|
Apr 7 10:39:53 ip-172-31-39-121 mariadbd[129747]: /lib/aarch64-linux-gnu/atomics/libc.so.6(+0xd00dc)[0xffff894ca0dc]
|
Apr 7 10:39:53 ip-172-31-39-121 mariadbd[129747]: Trying to get some variables.
|
Apr 7 10:39:53 ip-172-31-39-121 mariadbd[129747]: Some pointers may be invalid and cause the dump to abort.
|
Apr 7 10:39:53 ip-172-31-39-121 mariadbd[129747]: Query (0x0): (null)
|
Apr 7 10:39:53 ip-172-31-39-121 mariadbd[129747]: Connection ID (thread ID): 3
|
Apr 7 10:39:53 ip-172-31-39-121 mariadbd[129747]: Status: NOT_KILLED
|
Apr 7 10:39:53 ip-172-31-39-121 mariadbd[129747]: Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=on,condition_pushdown_from_having=on,not_null_range_scan=off
|
Apr 7 10:39:53 ip-172-31-39-121 mariadbd[129747]: The manual page at https://mariadb.com/kb/en/how-to-produce-a-full-stack-trace-for-mysqld/ contains
|
Apr 7 10:39:53 ip-172-31-39-121 mariadbd[129747]: information that should help you find out what is causing the crash.
|
Apr 7 10:39:53 ip-172-31-39-121 mariadbd[129747]: We think the query pointer is invalid, but we will try to print it anyway.
|
Apr 7 10:39:53 ip-172-31-39-121 mariadbd[129747]: Query:
|
Apr 7 10:39:53 ip-172-31-39-121 mariadbd[129747]: Writing a core file...
|
Apr 7 10:39:53 ip-172-31-39-121 mariadbd[129747]: Working directory at /media/storage/mysql
|
Apr 7 10:39:53 ip-172-31-39-121 mariadbd[129747]: Resource Limits:
|
Apr 7 10:39:53 ip-172-31-39-121 mariadbd[129747]: Limit Soft Limit Hard Limit Units
|
Apr 7 10:39:53 ip-172-31-39-121 mariadbd[129747]: Max cpu time unlimited unlimited seconds
|
Apr 7 10:39:53 ip-172-31-39-121 mariadbd[129747]: Max file size unlimited unlimited bytes
|
Apr 7 10:39:53 ip-172-31-39-121 mariadbd[129747]: Max data size unlimited unlimited bytes
|
Apr 7 10:39:53 ip-172-31-39-121 mariadbd[129747]: Max stack size 8388608 unlimited bytes
|
Apr 7 10:39:53 ip-172-31-39-121 mariadbd[129747]: Max core file size 0 unlimited bytes
|
Apr 7 10:39:53 ip-172-31-39-121 mariadbd[129747]: Max resident set unlimited unlimited bytes
|
Apr 7 10:39:53 ip-172-31-39-121 mariadbd[129747]: Max processes 125387 125387 processes
|
Apr 7 10:39:53 ip-172-31-39-121 mariadbd[129747]: Max open files 32768 32768 files
|
Apr 7 10:39:53 ip-172-31-39-121 mariadbd[129747]: Max locked memory 65536 65536 bytes
|
Apr 7 10:39:53 ip-172-31-39-121 mariadbd[129747]: Max address space unlimited unlimited bytes
|
Apr 7 10:39:53 ip-172-31-39-121 mariadbd[129747]: Max file locks unlimited unlimited locks
|
Apr 7 10:39:53 ip-172-31-39-121 mariadbd[129747]: Max pending signals 125387 125387 signals
|
Apr 7 10:39:53 ip-172-31-39-121 mariadbd[129747]: Max msgqueue size 819200 819200 bytes
|
Apr 7 10:39:53 ip-172-31-39-121 mariadbd[129747]: Max nice priority 0 0
|
Apr 7 10:39:53 ip-172-31-39-121 mariadbd[129747]: Max realtime priority 0 0
|
Apr 7 10:39:53 ip-172-31-39-121 mariadbd[129747]: Max realtime timeout unlimited unlimited us
|
Apr 7 10:39:53 ip-172-31-39-121 mariadbd[129747]: Core pattern: |/usr/share/apport/apport %p %s %c %d %P %E
|
Apr 7 10:39:53 ip-172-31-39-121 systemd[1]: mariadb.service: Main process exited, code=dumped, status=6/ABRT
|
Apr 7 10:39:53 ip-172-31-39-121 systemd[1]: mariadb.service: Failed with result 'core-dump'.
|
Apr 7 10:39:53 ip-172-31-39-121 systemd[1]: Failed to start MariaDB 10.5.15 database server.
|
DONOR Logs
Apr 7 11:35:47 es5 mysqld[29230]: 2022-04-07 11:35:47 0 [Note] WSREP: (44508de1-af53, 'tcp://0.0.0.0:4567') turning message relay requesting on, nonlive peers: tcp://172.31.39.121:4567
|
Apr 7 11:35:47 es5 mysqld[29230]: 2022-04-07 11:35:47 0 [Note] WSREP: (44508de1-af53, 'tcp://0.0.0.0:4567') connection established to 7c67946e-b045 tcp://172.31.39.121:4567
|
Apr 7 11:35:48 es5 mysqld[29230]: 2022-04-07 11:35:48 0 [Note] WSREP: declaring 7c67946e-b045 at tcp://172.31.39.121:4567 stable
|
Apr 7 11:35:48 es5 mysqld[29230]: 2022-04-07 11:35:48 0 [Note] WSREP: declaring 95267912-98ce at tcp://172.31.34.27:4567 stable
|
Apr 7 11:35:48 es5 mysqld[29230]: 2022-04-07 11:35:48 0 [Note] WSREP: declaring 98319f4a-8b85 at tcp://172.31.35.130:4567 stable
|
Apr 7 11:35:48 es5 mysqld[29230]: 2022-04-07 11:35:48 0 [Note] WSREP: declaring da9e7474-8746 at tcp://172.31.40.66:4567 stable
|
Apr 7 11:35:48 es5 mysqld[29230]: 2022-04-07 11:35:48 0 [Note] WSREP: declaring e4e23eea-bbd0 at tcp://172.31.37.249:4567 stable
|
Apr 7 11:35:48 es5 mysqld[29230]: 2022-04-07 11:35:48 0 [Note] WSREP: declaring eb5f53de-8588 at tcp://172.31.9.176:4567 stable
|
Apr 7 11:35:49 es5 mysqld[29230]: 2022-04-07 11:35:49 0 [Note] WSREP: Node 44508de1-af53 state prim
|
Apr 7 11:35:49 es5 mysqld[29230]: 2022-04-07 11:35:49 0 [Note] WSREP: view(view_id(PRIM,44508de1-af53,378) memb {
|
Apr 7 11:35:49 es5 mysqld[29230]: #01144508de1-af53,0
|
Apr 7 11:35:49 es5 mysqld[29230]: #0117c67946e-b045,0
|
Apr 7 11:35:49 es5 mysqld[29230]: #01195267912-98ce,0
|
Apr 7 11:35:49 es5 mysqld[29230]: #01198319f4a-8b85,0
|
Apr 7 11:35:49 es5 mysqld[29230]: #011da9e7474-8746,0
|
Apr 7 11:35:49 es5 mysqld[29230]: #011e4e23eea-bbd0,0
|
Apr 7 11:35:49 es5 mysqld[29230]: #011eb5f53de-8588,0
|
Apr 7 11:35:49 es5 mysqld[29230]: } joined {
|
Apr 7 11:35:49 es5 mysqld[29230]: } left {
|
Apr 7 11:35:49 es5 mysqld[29230]: } partitioned {
|
Apr 7 11:35:49 es5 mysqld[29230]: })
|
Apr 7 11:35:49 es5 mysqld[29230]: 2022-04-07 11:35:49 0 [Note] WSREP: save pc into disk
|
Apr 7 11:35:49 es5 mysqld[29230]: 2022-04-07 11:35:49 0 [Note] WSREP: New COMPONENT: primary = yes, bootstrap = no, my_idx = 0, memb_num = 7
|
Apr 7 11:35:49 es5 mysqld[29230]: 2022-04-07 11:35:49 0 [Note] WSREP: STATE_EXCHANGE: sent state UUID: 7dcd956b-b65e-11ec-8734-924b8afeca4d
|
Apr 7 11:35:49 es5 mysqld[29230]: 2022-04-07 11:35:49 0 [Note] WSREP: STATE EXCHANGE: sent state msg: 7dcd956b-b65e-11ec-8734-924b8afeca4d
|
Apr 7 11:35:49 es5 mysqld[29230]: 2022-04-07 11:35:49 0 [Note] WSREP: STATE EXCHANGE: got state msg: 7dcd956b-b65e-11ec-8734-924b8afeca4d from 0 (ES5_AWS)
|
Apr 7 11:35:49 es5 mysqld[29230]: 2022-04-07 11:35:49 0 [Note] WSREP: STATE EXCHANGE: got state msg: 7dcd956b-b65e-11ec-8734-924b8afeca4d from 2 (ES1_AWS)
|
Apr 7 11:35:49 es5 mysqld[29230]: 2022-04-07 11:35:49 0 [Note] WSREP: STATE EXCHANGE: got state msg: 7dcd956b-b65e-11ec-8734-924b8afeca4d from 3 (ES2_AWS)
|
Apr 7 11:35:49 es5 mysqld[29230]: 2022-04-07 11:35:49 0 [Note] WSREP: STATE EXCHANGE: got state msg: 7dcd956b-b65e-11ec-8734-924b8afeca4d from 4 (ES4_AWS)
|
Apr 7 11:35:49 es5 mysqld[29230]: 2022-04-07 11:35:49 0 [Note] WSREP: STATE EXCHANGE: got state msg: 7dcd956b-b65e-11ec-8734-924b8afeca4d from 5 (ES3_AWS)
|
Apr 7 11:35:49 es5 mysqld[29230]: 2022-04-07 11:35:49 0 [Note] WSREP: STATE EXCHANGE: got state msg: 7dcd956b-b65e-11ec-8734-924b8afeca4d from 6 (BACKUP_AWS)
|
Apr 7 11:35:49 es5 mysqld[29230]: 2022-04-07 11:35:49 0 [Note] WSREP: STATE EXCHANGE: got state msg: 7dcd956b-b65e-11ec-8734-924b8afeca4d from 1 (ES6_AWS)
|
Apr 7 11:35:49 es5 mysqld[29230]: 2022-04-07 11:35:49 0 [Note] WSREP: Quorum results:
|
Apr 7 11:35:49 es5 mysqld[29230]: #011version = 6,
|
Apr 7 11:35:49 es5 mysqld[29230]: #011component = PRIMARY,
|
Apr 7 11:35:49 es5 mysqld[29230]: #011conf_id = 363,
|
Apr 7 11:35:49 es5 mysqld[29230]: #011members = 6/7 (joined/total),
|
Apr 7 11:35:49 es5 mysqld[29230]: #011act_id = 5064032831,
|
Apr 7 11:35:49 es5 mysqld[29230]: #011last_appl. = 5064032726,
|
Apr 7 11:35:49 es5 mysqld[29230]: #011protocols = 2/10/4 (gcs/repl/appl),
|
Apr 7 11:35:49 es5 mysqld[29230]: #011vote policy= 0,
|
Apr 7 11:35:49 es5 mysqld[29230]: #011group UUID = 1d4206f9-647d-11ea-a963-0ed98924f90b
|
Apr 7 11:35:49 es5 mysqld[29230]: 2022-04-07 11:35:49 0 [Note] WSREP: Flow-control interval: [42, 42]
|
Apr 7 11:35:49 es5 mysqld[29230]: 2022-04-07 11:35:49 2 [Note] WSREP: ####### processing CC 5064032832, local, ordered
|
Apr 7 11:35:49 es5 mysqld[29230]: 2022-04-07 11:35:49 2 [Note] WSREP: ####### My UUID: 44508de1-b657-11ec-af53-dbbee42dcffe
|
Apr 7 11:35:49 es5 mysqld[29230]: 2022-04-07 11:35:49 2 [Note] WSREP: Skipping cert index reset
|
Apr 7 11:35:49 es5 mysqld[29230]: 2022-04-07 11:35:49 2 [Note] WSREP: REPL Protocols: 10 (5)
|
Apr 7 11:35:49 es5 mysqld[29230]: 2022-04-07 11:35:49 2 [Note] WSREP: ####### Adjusting cert position: 5064032831 -> 5064032832
|
Apr 7 11:35:49 es5 mysqld[29230]: 2022-04-07 11:35:49 0 [Note] WSREP: Service thread queue flushed.
|
Apr 7 11:35:49 es5 mysqld[29230]: 2022-04-07 11:35:49 2 [Note] WSREP: ================================================
|
Apr 7 11:35:49 es5 mysqld[29230]: View:
|
Apr 7 11:35:49 es5 mysqld[29230]: id: 1d4206f9-647d-11ea-a963-0ed98924f90b:5064032832
|
Apr 7 11:35:49 es5 mysqld[29230]: status: primary
|
Apr 7 11:35:49 es5 mysqld[29230]: protocol_version: 4
|
Apr 7 11:35:49 es5 mysqld[29230]: capabilities: MULTI-MASTER, CERTIFICATION, PARALLEL_APPLYING, REPLAY, ISOLATION, PAUSE, CAUSAL_READ, INCREMENTAL_WS, UNORDERED, PREORDERED, STREAMING, NBO
|
Apr 7 11:35:49 es5 mysqld[29230]: final: no
|
Apr 7 11:35:49 es5 mysqld[29230]: own_index: 0
|
Apr 7 11:35:49 es5 mysqld[29230]: members(7):
|
Apr 7 11:35:49 es5 mysqld[29230]: #0110: 44508de1-b657-11ec-af53-dbbee42dcffe, ES5_AWS
|
Apr 7 11:35:49 es5 mysqld[29230]: #0111: 7c67946e-b65e-11ec-b045-eb2b92474255, ES6_AWS
|
Apr 7 11:35:49 es5 mysqld[29230]: #0112: 95267912-5df1-11ec-98ce-a66d469d045c, ES1_AWS
|
Apr 7 11:35:49 es5 mysqld[29230]: #0113: 98319f4a-b5f9-11ec-8b85-e69cdab5035e, ES2_AWS
|
Apr 7 11:35:49 es5 mysqld[29230]: #0114: da9e7474-5df2-11ec-8746-dfd73835a1c5, ES4_AWS
|
Apr 7 11:35:49 es5 mysqld[29230]: #0115: e4e23eea-5df0-11ec-bbd0-8b132f61eb3d, ES3_AWS
|
Apr 7 11:35:49 es5 mysqld[29230]: #0116: eb5f53de-882b-11ec-8588-1a528d6537ef, BACKUP_AWS
|
Apr 7 11:35:49 es5 mysqld[29230]: =================================================
|
Apr 7 11:35:49 es5 mysqld[29230]: 2022-04-07 11:35:49 2 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
|
Apr 7 11:35:49 es5 mysqld[29230]: 2022-04-07 11:35:49 2 [Note] WSREP: Lowest cert index boundary for CC from group: 5064032727
|
Apr 7 11:35:49 es5 mysqld[29230]: 2022-04-07 11:35:49 2 [Note] WSREP: Min available from gcache for CC from group: 5063284333
|
Apr 7 11:35:49 es5 mysqld[29230]: 2022-04-07 11:35:49 0 [Note] WSREP: Member 1.0 (ES6_AWS) requested state transfer from 'ES5_AWS'. Selected 0.0 (ES5_AWS)(SYNCED) as donor.
|
Apr 7 11:35:49 es5 mysqld[29230]: 2022-04-07 11:35:49 0 [Note] WSREP: Shifting SYNCED -> DONOR/DESYNCED (TO: 5064032832)
|
Apr 7 11:35:49 es5 mysqld[29230]: 2022-04-07 11:35:49 2 [Note] WSREP: Detected STR version: 1, req_len: 125, req: STRv1
|
Apr 7 11:35:49 es5 mysqld[29230]: 2022-04-07 11:35:49 2 [Note] WSREP: Cert index preload: 5064032727 -> 5064032832
|
Apr 7 11:35:49 es5 mysqld[29230]: 2022-04-07 11:35:49 2 [Note] WSREP: Server status change synced -> donor
|
Apr 7 11:35:49 es5 mysqld[29230]: 2022-04-07 11:35:49 0 [Note] WSREP: async IST sender starting to serve tcp://172.31.39.121:4568 sending 5064032727-5064032832, preload starts from 5064032727
|
Apr 7 11:35:49 es5 mysqld[29230]: 2022-04-07 11:35:49 2 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
|
Apr 7 11:35:49 es5 mysqld[29230]: 2022-04-07 11:35:49 2 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
|
Apr 7 11:35:49 es5 mysqld[29230]: 2022-04-07 11:35:49 0 [Note] WSREP: Donor monitor thread started to monitor
|
Apr 7 11:35:49 es5 mysqld[29230]: 2022-04-07 11:35:49 0 [Note] WSREP: Running: 'wsrep_sst_rsync --role 'donor' --address '172.31.39.121:4444/rsync_sst' --local-port '3306' --socket '/var/run/mysqld/mysqld.sock' --datadir '/media/storage/mysql/' --gtid '1d4206f9-647d-11ea-a963-0ed98924f90b:5064032832' --gtid-domain-id '0' --mysqld-args --wsrep_start_position=1d4206f9-647d-11ea-a963-0ed98924f90b:5063284262'
|
Apr 7 11:35:49 es5 mysqld[29230]: 2022-04-07 11:35:49 0 [Note] WSREP: IST sender 5064032727 -> 5064032832
|
Apr 7 11:35:49 es5 mysqld[29230]: 2022-04-07 11:35:49 2 [Note] WSREP: sst_donor_thread signaled with 0
|
Apr 7 11:35:49 es5 mysqld[29230]: 2022-04-07 11:35:49 0 [Note] WSREP: Flushing tables for SST...
|
Apr 7 11:35:49 es5 mysqld[29230]: 2022-04-07 11:35:49 0 [Note] WSREP: pause
|
Apr 7 11:35:49 es5 mysqld[29230]: 2022-04-07 11:35:49 0 [Note] WSREP: Provider paused at 1d4206f9-647d-11ea-a963-0ed98924f90b:5064032832 (778520)
|
Apr 7 11:35:49 es5 mysqld[29230]: 2022-04-07 11:35:49 0 [Note] WSREP: Server paused at: 5064032832
|
Apr 7 11:35:49 es5 mysqld[29230]: 2022-04-07 11:35:49 0 [Note] WSREP: Tables flushed.
|
Apr 7 11:35:50 es5 mysqld[29230]: 2022-04-07 11:35:50 0 [Note] WSREP: (44508de1-af53, 'tcp://0.0.0.0:4567') turning message relay requesting off
|
Apr 7 11:39:01 es5 CRON[10599]: (root) CMD ( [ -x /usr/lib/php/sessionclean ] && if [ ! -d /run/systemd/system ]; then /usr/lib/php/sessionclean; fi)
|
Apr 7 11:39:05 es5 systemd[1]: Starting Clean php session files...
|
Apr 7 11:39:06 es5 systemd[1]: Started Clean php session files.
|
Apr 7 11:39:51 es5 mysqld[29230]: 2022-04-07 11:39:51 0 [Warning] InnoDB: A long semaphore wait:
|
Apr 7 11:39:51 es5 mysqld[29230]: --Thread 139702254601984 has waited at log0log.cc line 882 for 241.00 seconds the semaphore:
|
Apr 7 11:39:51 es5 mysqld[29230]: Mutex at 0x55fd5cf32080, Mutex LOG_WRITE created log0log.cc:537, lock var 2
|
Apr 7 11:39:51 es5 mysqld[29230]: 2022-04-07 11:39:51 0 [Note] InnoDB: A semaphore wait:
|
Apr 7 11:39:51 es5 mysqld[29230]: --Thread 139702254601984 has waited at log0log.cc line 882 for 241.00 seconds the semaphore:
|
Apr 7 11:39:51 es5 mysqld[29230]: Mutex at 0x55fd5cf32080, Mutex LOG_WRITE created log0log.cc:537, lock var 2
|
Apr 7 11:39:51 es5 mysqld[29230]: InnoDB: ###### Starts InnoDB Monitor for 30 secs to print diagnostic info:
|
Apr 7 11:39:51 es5 mysqld[29230]: InnoDB: Pending reads 0, writes 0
|
Apr 7 11:39:52 es5 mysqld[29230]: 2022-04-07 11:39:52 0 [Note] WSREP: resume
|
Apr 7 11:39:52 es5 mysqld[29230]: 2022-04-07 11:39:52 0 [Note] WSREP: resuming provider at 778520
|
Apr 7 11:39:52 es5 mysqld[29230]: 2022-04-07 11:39:52 0 [Note] WSREP: Provider resumed.
|
Apr 7 11:39:52 es5 mysqld[29230]: 2022-04-07 11:39:52 0 [Note] WSREP: SST sent: 1d4206f9-647d-11ea-a963-0ed98924f90b:5064032832
|
Apr 7 11:39:52 es5 mysqld[29230]: 2022-04-07 11:39:52 0 [Note] WSREP: Server status change donor -> joined
|
Apr 7 11:39:52 es5 mysqld[29230]: 2022-04-07 11:39:52 0 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
|
Apr 7 11:39:52 es5 mysqld[29230]: 2022-04-07 11:39:52 0 [Note] WSREP: Donor monitor thread ended with total time 243 sec
|
Apr 7 11:39:52 es5 mysqld[29230]: 2022-04-07 11:39:52 0 [Note] WSREP: 0.0 (ES5_AWS): State transfer to 1.0 (ES6_AWS) complete.
|
Apr 7 11:39:52 es5 mysqld[29230]: 2022-04-07 11:39:52 0 [Note] WSREP: Shifting DONOR/DESYNCED -> JOINED (TO: 5064034794)
|
Apr 7 11:39:52 es5 mysqld[29230]: 2022-04-07 11:39:52 0 [Note] WSREP: Member 0.0 (ES5_AWS) synced with group.
|
Apr 7 11:39:52 es5 mysqld[29230]: 2022-04-07 11:39:52 0 [Note] WSREP: Shifting JOINED -> SYNCED (TO: 5064034794)
|
Apr 7 11:39:52 es5 mysqld[29230]: 2022-04-07 11:39:52 2 [Note] WSREP: Server ES5_AWS synced with group
|
Apr 7 11:39:52 es5 mysqld[29230]: 2022-04-07 11:39:52 2 [Note] WSREP: Server status change joined -> synced
|
Apr 7 11:39:52 es5 mysqld[29230]: 2022-04-07 11:39:52 2 [Note] WSREP: Synchronized with group, ready for connections
|
Apr 7 11:39:52 es5 mysqld[29230]: 2022-04-07 11:39:52 2 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
|
Apr 7 11:39:53 es5 mysqld[29230]: 2022-04-07 11:39:53 0 [Note] WSREP: (44508de1-af53, 'tcp://0.0.0.0:4567') turning message relay requesting on, nonlive peers: tcp://172.31.39.121:4567
|
Apr 7 11:39:53 es5 mysqld[29230]: 2022-04-07 11:39:53 0 [Note] WSREP: declaring 95267912-98ce at tcp://172.31.34.27:4567 stable
|
Apr 7 11:39:53 es5 mysqld[29230]: 2022-04-07 11:39:53 0 [Note] WSREP: declaring 98319f4a-8b85 at tcp://172.31.35.130:4567 stable
|
Apr 7 11:39:53 es5 mysqld[29230]: 2022-04-07 11:39:53 0 [Note] WSREP: declaring da9e7474-8746 at tcp://172.31.40.66:4567 stable
|
Apr 7 11:39:53 es5 mysqld[29230]: 2022-04-07 11:39:53 0 [Note] WSREP: declaring e4e23eea-bbd0 at tcp://172.31.37.249:4567 stable
|
Apr 7 11:39:53 es5 mysqld[29230]: 2022-04-07 11:39:53 0 [Note] WSREP: declaring eb5f53de-8588 at tcp://172.31.9.176:4567 stable
|
Apr 7 11:39:53 es5 mysqld[29230]: 2022-04-07 11:39:53 0 [Note] WSREP: forgetting 7c67946e-b045 (tcp://172.31.39.121:4567)
|
Apr 7 11:39:53 es5 mysqld[29230]: 2022-04-07 11:39:53 0 [Note] WSREP: (44508de1-af53, 'tcp://0.0.0.0:4567') turning message relay requesting off
|
Apr 7 11:39:53 es5 mysqld[29230]: 2022-04-07 11:39:53 0 [Note] WSREP: Node 44508de1-af53 state prim
|
Apr 7 11:39:53 es5 mysqld[29230]: 2022-04-07 11:39:53 0 [Note] WSREP: view(view_id(PRIM,44508de1-af53,379) memb {
|
Apr 7 11:39:53 es5 mysqld[29230]: #01144508de1-af53,0
|
Apr 7 11:39:53 es5 mysqld[29230]: #01195267912-98ce,0
|
Apr 7 11:39:53 es5 mysqld[29230]: #01198319f4a-8b85,0
|
Apr 7 11:39:53 es5 mysqld[29230]: #011da9e7474-8746,0
|
Apr 7 11:39:53 es5 mysqld[29230]: #011e4e23eea-bbd0,0
|
Apr 7 11:39:53 es5 mysqld[29230]: #011eb5f53de-8588,0
|
Apr 7 11:39:53 es5 mysqld[29230]: } joined {
|
Apr 7 11:39:53 es5 mysqld[29230]: } left {
|
Apr 7 11:39:53 es5 mysqld[29230]: } partitioned {
|
Apr 7 11:39:53 es5 mysqld[29230]: #0117c67946e-b045,0
|
Apr 7 11:39:53 es5 mysqld[29230]: })
|
Apr 7 11:39:53 es5 mysqld[29230]: 2022-04-07 11:39:53 0 [Note] WSREP: save pc into disk
|
Apr 7 11:39:53 es5 mysqld[29230]: 2022-04-07 11:39:53 0 [Note] WSREP: forgetting 7c67946e-b045 (tcp://172.31.39.121:4567)
|
Apr 7 11:39:53 es5 mysqld[29230]: 2022-04-07 11:39:53 0 [Note] WSREP: New COMPONENT: primary = yes, bootstrap = no, my_idx = 0, memb_num = 6
|
Apr 7 11:39:53 es5 mysqld[29230]: 2022-04-07 11:39:53 0 [Note] WSREP: STATE_EXCHANGE: sent state UUID: 0f078c2c-b65f-11ec-b06d-de758fe27da8
|
Apr 7 11:39:53 es5 mysqld[29230]: 2022-04-07 11:39:53 0 [Note] WSREP: STATE EXCHANGE: sent state msg: 0f078c2c-b65f-11ec-b06d-de758fe27da8
|
Apr 7 11:39:53 es5 mysqld[29230]: 2022-04-07 11:39:53 0 [Note] WSREP: STATE EXCHANGE: got state msg: 0f078c2c-b65f-11ec-b06d-de758fe27da8 from 0 (ES5_AWS)
|
Apr 7 11:39:53 es5 mysqld[29230]: 2022-04-07 11:39:53 0 [Note] WSREP: STATE EXCHANGE: got state msg: 0f078c2c-b65f-11ec-b06d-de758fe27da8 from 1 (ES1_AWS)
|
Apr 7 11:39:53 es5 mysqld[29230]: 2022-04-07 11:39:53 0 [Note] WSREP: STATE EXCHANGE: got state msg: 0f078c2c-b65f-11ec-b06d-de758fe27da8 from 2 (ES2_AWS)
|
Apr 7 11:39:53 es5 mysqld[29230]: 2022-04-07 11:39:53 0 [Note] WSREP: STATE EXCHANGE: got state msg: 0f078c2c-b65f-11ec-b06d-de758fe27da8 from 3 (ES4_AWS)
|
Apr 7 11:39:53 es5 mysqld[29230]: 2022-04-07 11:39:53 0 [Note] WSREP: STATE EXCHANGE: got state msg: 0f078c2c-b65f-11ec-b06d-de758fe27da8 from 4 (ES3_AWS)
|
Apr 7 11:39:53 es5 mysqld[29230]: 2022-04-07 11:39:53 0 [Note] WSREP: STATE EXCHANGE: got state msg: 0f078c2c-b65f-11ec-b06d-de758fe27da8 from 5 (BACKUP_AWS)
|
Apr 7 11:39:53 es5 mysqld[29230]: 2022-04-07 11:39:53 0 [Note] WSREP: Quorum results:
|
Apr 7 11:39:53 es5 mysqld[29230]: #011version = 6,
|
Apr 7 11:39:53 es5 mysqld[29230]: #011component = PRIMARY,
|
Apr 7 11:39:53 es5 mysqld[29230]: #011conf_id = 364,
|
Apr 7 11:39:53 es5 mysqld[29230]: #011members = 6/6 (joined/total),
|
Apr 7 11:39:53 es5 mysqld[29230]: #011act_id = 5064034794,
|
Apr 7 11:39:53 es5 mysqld[29230]: #011last_appl. = 5064034689,
|
Apr 7 11:39:53 es5 mysqld[29230]: #011protocols = 2/10/4 (gcs/repl/appl),
|
Apr 7 11:39:53 es5 mysqld[29230]: #011vote policy= 0,
|
Apr 7 11:39:53 es5 mysqld[29230]: #011group UUID = 1d4206f9-647d-11ea-a963-0ed98924f90b
|
Apr 7 11:39:53 es5 mysqld[29230]: 2022-04-07 11:39:53 0 [Note] WSREP: Flow-control interval: [39, 39]
|
Apr 7 11:39:53 es5 mysqld[29230]: 2022-04-07 11:39:53 2 [Note] WSREP: ####### processing CC 5064034795, local, ordered
|
Apr 7 11:39:53 es5 mysqld[29230]: 2022-04-07 11:39:53 2 [Note] WSREP: ####### My UUID: 44508de1-b657-11ec-af53-dbbee42dcffe
|
Apr 7 11:39:53 es5 mysqld[29230]: 2022-04-07 11:39:53 2 [Note] WSREP: Skipping cert index reset
|
Apr 7 11:39:53 es5 mysqld[29230]: 2022-04-07 11:39:53 2 [Note] WSREP: REPL Protocols: 10 (5)
|
Apr 7 11:39:53 es5 mysqld[29230]: 2022-04-07 11:39:53 2 [Note] WSREP: ####### Adjusting cert position: 5064034794 -> 5064034795
|
Apr 7 11:39:53 es5 mysqld[29230]: 2022-04-07 11:39:53 0 [Note] WSREP: Service thread queue flushed.
|
Apr 7 11:39:53 es5 mysqld[29230]: 2022-04-07 11:39:53 2 [Note] WSREP: ================================================
|
Apr 7 11:39:53 es5 mysqld[29230]: View:
|
Apr 7 11:39:53 es5 mysqld[29230]: id: 1d4206f9-647d-11ea-a963-0ed98924f90b:5064034795
|
Apr 7 11:39:53 es5 mysqld[29230]: status: primary
|
Apr 7 11:39:53 es5 mysqld[29230]: protocol_version: 4
|
Apr 7 11:39:53 es5 mysqld[29230]: capabilities: MULTI-MASTER, CERTIFICATION, PARALLEL_APPLYING, REPLAY, ISOLATION, PAUSE, CAUSAL_READ, INCREMENTAL_WS, UNORDERED, PREORDERED, STREAMING, NBO
|
Apr 7 11:39:53 es5 mysqld[29230]: final: no
|
Apr 7 11:39:53 es5 mysqld[29230]: own_index: 0
|
Apr 7 11:39:53 es5 mysqld[29230]: members(6):
|
Apr 7 11:39:53 es5 mysqld[29230]: #0110: 44508de1-b657-11ec-af53-dbbee42dcffe, ES5_AWS
|
Apr 7 11:39:53 es5 mysqld[29230]: #0111: 95267912-5df1-11ec-98ce-a66d469d045c, ES1_AWS
|
Apr 7 11:39:53 es5 mysqld[29230]: #0112: 98319f4a-b5f9-11ec-8b85-e69cdab5035e, ES2_AWS
|
Apr 7 11:39:53 es5 mysqld[29230]: #0113: da9e7474-5df2-11ec-8746-dfd73835a1c5, ES4_AWS
|
Apr 7 11:39:53 es5 mysqld[29230]: #0114: e4e23eea-5df0-11ec-bbd0-8b132f61eb3d, ES3_AWS
|
Apr 7 11:39:53 es5 mysqld[29230]: #0115: eb5f53de-882b-11ec-8588-1a528d6537ef, BACKUP_AWS
|
Apr 7 11:39:53 es5 mysqld[29230]: =================================================
|
Apr 7 11:39:53 es5 mysqld[29230]: 2022-04-07 11:39:53 2 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
|
Apr 7 11:39:53 es5 mysqld[29230]: 2022-04-07 11:39:53 2 [Note] WSREP: Lowest cert index boundary for CC from group: 5064032727
|
Apr 7 11:39:53 es5 mysqld[29230]: 2022-04-07 11:39:53 2 [Note] WSREP: Min available from gcache for CC from group: 5063284333
|
Apr 7 11:39:53 es5 mysqld[29230]: 2022-04-07 11:39:53 0 [Note] WSREP: async IST sender served
|
Apr 7 11:39:56 es5 mysqld[29230]: 2022-04-07 11:39:56 0 [Note] WSREP: (44508de1-af53, 'tcp://0.0.0.0:4567') connection to peer eb5f53de-8588 with addr tcp://172.31.9.176:4567 timed out, no messages seen in PT3S, socket stats: rtt: 872 rttvar: 864 rto: 204000 lost: 0 last_data_recv: 3104 cwnd: 10 last_queued_since: 33142 last_delivered_since: 3103372544 send_queue_length: 1 send_queue_bytes: 804 segment: 0 messages: 1
|
Apr 7 11:39:56 es5 mysqld[29230]: 2022-04-07 11:39:56 0 [Note] WSREP: Deferred close timer started for socket with remote endpoint: tcp://172.31.9.176:60056
|
Apr 7 11:39:56 es5 mysqld[29230]: 2022-04-07 11:39:56 0 [Note] WSREP: (44508de1-af53, 'tcp://0.0.0.0:4567') turning message relay requesting on, nonlive peers: tcp://172.31.9.176:4567
|
Apr 7 11:39:56 es5 mysqld[29230]: 2022-04-07 11:39:56 0 [Note] WSREP: Deferred close timer handle_wait Operation aborted. for 0x55fd61a9f5b0
|
Apr 7 11:39:56 es5 mysqld[29230]: 2022-04-07 11:39:56 0 [Note] WSREP: Deferred close timer destruct
|
Attachments
Issue Links
- relates to
-
MDEV-12353 Efficient InnoDB redo log record format
-
- Closed
-
-
-
- Closed
-
-
-
- Closed
-