[MDEV-32158] wsrep_sst_mariabackup use /tmp dir during SST rather then user defined tmpdir - Jira

Details

Type: Bug
Status: Closed (View Workflow)
Priority: Major
Resolution: Fixed
Affects Version/s: 10.5, 10.6
Fix Version/s: 10.5.26, 10.6.19, 10.11.9, 11.1.6, 11.2.5, 11.4.3
Component/s: Galera SST
Labels:
None

Description

During SST for joiner node , wsrep_sst_mariabackup consider /tmp as a working directory rather then the user defined.

My understanding during SST wsrep_sst_mariabackup should be using tmpdir defined under [mysqld] section not the* /tmp* dir.

Log from Donor while Joiner to trying to get sync using SST

2023-09-13  0:55:36 0 [Note] WSREP: (af83c624-a657, 'tcp://0.0.0.0:4567') connection established to c9e7c8ae-92d0 tcp://192.168.47.42:4567

2023-09-13  0:55:36 0 [Note] WSREP: (af83c624-a657, 'tcp://0.0.0.0:4567') turning message relay requesting on, nonlive peers:

2023-09-13  0:55:37 0 [Note] WSREP: declaring c9e7c8ae-92d0 at tcp://192.168.47.42:4567 stable

2023-09-13  0:55:37 0 [Note] WSREP: Node af83c624-a657 state prim

2023-09-13  0:55:37 0 [Note] WSREP: view(view_id(PRIM,af83c624-a657,76) memb {

        af83c624-a657,0

        c9e7c8ae-92d0,0

} joined {

} left {

} partitioned {

})

2023-09-13  0:55:37 0 [Note] WSREP: save pc into disk

2023-09-13  0:55:37 0 [Note] WSREP: New COMPONENT: primary = yes, bootstrap = no, my_idx = 0, memb_num = 2

2023-09-13  0:55:37 0 [Note] WSREP: STATE_EXCHANGE: sent state UUID: c7a87140-51f1-11ee-87cf-87f3f3a5fe55

2023-09-13  0:55:37 0 [Note] WSREP: STATE EXCHANGE: sent state msg: c7a87140-51f1-11ee-87cf-87f3f3a5fe55

2023-09-13  0:55:37 0 [Note] WSREP: STATE EXCHANGE: got state msg: c7a87140-51f1-11ee-87cf-87f3f3a5fe55 from 0 (db001)

2023-09-13  0:55:37 0 [Note] WSREP: STATE EXCHANGE: got state msg: c7a87140-51f1-11ee-87cf-87f3f3a5fe55 from 1 (db002)

2023-09-13  0:55:37 0 [Note] WSREP: Quorum results:

        version    = 6,

        component  = PRIMARY,

        conf_id    = 1,

        members    = 1/2 (joined/total),

        act_id     = 90,

        last_appl. = 89,

        protocols  = 2/10/4 (gcs/repl/appl),

        vote policy= 0,

        group UUID = 8f95c1cc-430f-11ee-8404-8f9a8663c992

2023-09-13  0:55:37 0 [Note] WSREP: Flow-control interval: [23, 23]

2023-09-13  0:55:37 2 [Note] WSREP: ####### processing CC 91, local, ordered

2023-09-13  0:55:37 2 [Note] WSREP: ####### My UUID: af83c624-4477-11ee-a657-0627ac9b29e9

2023-09-13  0:55:37 2 [Note] WSREP: Skipping cert index reset

2023-09-13  0:55:37 2 [Note] WSREP: REPL Protocols: 10 (5)

2023-09-13  0:55:37 2 [Note] WSREP: ####### Adjusting cert position: 90 -> 91

2023-09-13  0:55:37 0 [Note] WSREP: Service thread queue flushed.

2023-09-13  0:55:37 2 [Note] WSREP: ================================================

View:

  id: 8f95c1cc-430f-11ee-8404-8f9a8663c992:91

  status: primary

  protocol_version: 4

  capabilities: MULTI-MASTER, CERTIFICATION, PARALLEL_APPLYING, REPLAY, ISOLATION, PAUSE, CAUSAL_READ, INCREMENTAL_WS, UNORDERED, PREORDERED, STREAMING, NBO

  final: no

  own_index: 0

  members(2):

        0: af83c624-4477-11ee-a657-0627ac9b29e9, db001

        1: c9e7c8ae-4476-11ee-92d0-f20d5682fc83, db002

=================================================

2023-09-13  0:55:37 2 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.

2023-09-13  0:55:37 2 [Note] WSREP: Lowest cert index boundary for CC from group: 91

2023-09-13  0:55:37 2 [Note] WSREP: Min available from gcache for CC from group: 1

2023-09-13  0:55:37 0 [Note] WSREP: Member 1.0 (db002) requested state transfer from '*any*'. Selected 0.0 (db001)(SYNCED) as donor.

2023-09-13  0:55:37 0 [Note] WSREP: Shifting SYNCED -> DONOR/DESYNCED (TO: 91)

2023-09-13  0:55:37 2 [Note] WSREP: Detected STR version: 1, req_len: 131, req: STRv1

2023-09-13  0:55:37 2 [Note] WSREP: Cert index preload: 91 -> 91

2023-09-13  0:55:37 2 [Note] WSREP: Server status change synced -> donor

2023-09-13  0:55:37 2 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.

2023-09-13  0:55:37 0 [Note] WSREP: async IST sender starting to serve tcp://192.168.47.42:4568 sending 91-91, preload starts from 91

2023-09-13  0:55:37 0 [Note] WSREP: IST sender 91 -> 91

2023-09-13  0:55:37 0 [Note] WSREP: Donor monitor thread started to monitor

2023-09-13  0:55:37 0 [Note] WSREP: Running: 'wsrep_sst_mariabackup --role 'donor' --address '192.168.47.42:4444/xtrabackup_sst//1' --local-port 3306 --socket '/var/lib/mysql/mysql.sock' --progress 0 --datadir '/var/lib/mysql/' --gtid '8f95c1cc-430f-11ee-8404-8f9a8663c992:91' --gtid-domain-id 0 --binlog '/var/lib/mysql/bin' --mysqld-args --wsrep-new-cluster --wsrep_start_position=8f95c1cc-430f-11ee-8404-8f9a8663c992:89,0-1-33'

2023-09-13  0:55:37 2 [Note] WSREP: sst_donor_thread signaled with 0

WSREP_SST: [INFO] mariabackup SST started on donor (20230913 00:55:37.688)

WSREP_SST: [INFO] SSL configuration: CA='', CAPATH='', CERT='', KEY='', MODE='DISABLED', encrypt='0' (20230913 00:55:37.727)

WSREP_SST: [INFO] Progress reporting tool pv not found in path: /usr//bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/usr/sbin:/usr/bin:/sbin:/bin (20230913 00:55:37.824)

WSREP_SST: [INFO] Disabling all progress/rate-limiting (20230913 00:55:37.826)

WSREP_SST: [INFO] Streaming with mbstream (20230913 00:55:37.997)

WSREP_SST: [INFO] Using socat as streamer (20230913 00:55:37.999)

WSREP_SST: [INFO] Using '/var/lib/mysql/tmp.23l7f5jblf' as mariadb-backup temporary directory (20230913 00:55:38.011)

WSREP_SST: [INFO] Using '/tmp/tmp.t42TMS6WSR' as mariadb-abackup working directory (20230913 00:55:38.015)

..

.....

2023-09-13  0:55:50 0 [Note] WSREP: Member 0.0 (db001) synced with group.

2023-09-13  0:55:50 0 [Note] WSREP: Processing event queue:...100.0% (1/1 events) complete.

2023-09-13  0:55:50 0 [Note] WSREP: Shifting JOINED -> SYNCED (TO: 91)

2023-09-13  0:55:50 2 [Note] WSREP: Server db001 synced with group

2023-09-13  0:55:50 2 [Note] WSREP: Server status change joined -> synced

2023-09-13  0:55:50 2 [Note] WSREP: Synchronized with group, ready for connections

2023-09-13  0:55:50 2 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.

WSREP_SST: [INFO] Total time on donor: 0 seconds (20230913 00:55:50.414)

WSREP_SST: [INFO] mariabackup SST completed on donor (20230913 00:55:50.417)

WSREP_SST: [INFO] Cleaning up temporary directories (20230913 00:55:50.420)

2023-09-13  0:55:50 0 [Note] WSREP: Donor monitor thread ended with total time 13 sec

2023-09-13  0:55:51 0 [Note] WSREP: async IST sender served

2023-09-13  0:55:51 0 [Note] WSREP: 1.0 (db002): State transfer from 0.0 (db001) complete.

2023-09-13  0:55:51 0 [Note] WSREP: Member 1.0 (db002) synced with group.

https://github.com/MariaDB/server/blob/e987b9350cb83038c73a165922986441f345a3a9/scripts/wsrep_sst_mariabackup.sh#L1030

Donor node

[root@gcn-01 ~]# cat /etc/my.cnf|grep -i tmpdir

tmpdir                          = /var/lib/mysql/

Joiner node

[root@gcn-02 mysql]#  cat /etc/my.cnf|grep -i tmpdir

tmpdir                          = /var/lib/mysql/

Attachments

Activity

Ascending order - Click to sort in descending order

Pramod Mahto created issue - 2023-09-13 05:14

Pramod Mahto made changes - 2023-09-13 05:16

Field	Original Value	New Value
Description	During SST for joiner node , wsrep_sst_mariabackup consider /tmp as a working directory rather then the user defined. My understanding during SST wsrep_sst_mariabackup should be using tmpdir defined under [mysqld] section not the* /tmp* dir. Log from Donor while Joiner to trying to get sync using SST {code} 2023-09-13 0:55:36 0 [Note] WSREP: (af83c624-a657, 'tcp://0.0.0.0:4567') connection established to c9e7c8ae-92d0 tcp://192.168.47.42:4567 2023-09-13 0:55:36 0 [Note] WSREP: (af83c624-a657, 'tcp://0.0.0.0:4567') turning message relay requesting on, nonlive peers: 2023-09-13 0:55:37 0 [Note] WSREP: declaring c9e7c8ae-92d0 at tcp://192.168.47.42:4567 stable 2023-09-13 0:55:37 0 [Note] WSREP: Node af83c624-a657 state prim 2023-09-13 0:55:37 0 [Note] WSREP: view(view_id(PRIM,af83c624-a657,76) memb { af83c624-a657,0 c9e7c8ae-92d0,0 } joined { } left { } partitioned { }) 2023-09-13 0:55:37 0 [Note] WSREP: save pc into disk 2023-09-13 0:55:37 0 [Note] WSREP: New COMPONENT: primary = yes, bootstrap = no, my_idx = 0, memb_num = 2 2023-09-13 0:55:37 0 [Note] WSREP: STATE_EXCHANGE: sent state UUID: c7a87140-51f1-11ee-87cf-87f3f3a5fe55 2023-09-13 0:55:37 0 [Note] WSREP: STATE EXCHANGE: sent state msg: c7a87140-51f1-11ee-87cf-87f3f3a5fe55 2023-09-13 0:55:37 0 [Note] WSREP: STATE EXCHANGE: got state msg: c7a87140-51f1-11ee-87cf-87f3f3a5fe55 from 0 (db001) 2023-09-13 0:55:37 0 [Note] WSREP: STATE EXCHANGE: got state msg: c7a87140-51f1-11ee-87cf-87f3f3a5fe55 from 1 (db002) 2023-09-13 0:55:37 0 [Note] WSREP: Quorum results: version = 6, component = PRIMARY, conf_id = 1, members = 1/2 (joined/total), act_id = 90, last_appl. = 89, protocols = 2/10/4 (gcs/repl/appl), vote policy= 0, group UUID = 8f95c1cc-430f-11ee-8404-8f9a8663c992 2023-09-13 0:55:37 0 [Note] WSREP: Flow-control interval: [23, 23] 2023-09-13 0:55:37 2 [Note] WSREP: ####### processing CC 91, local, ordered 2023-09-13 0:55:37 2 [Note] WSREP: ####### My UUID: af83c624-4477-11ee-a657-0627ac9b29e9 2023-09-13 0:55:37 2 [Note] WSREP: Skipping cert index reset 2023-09-13 0:55:37 2 [Note] WSREP: REPL Protocols: 10 (5) 2023-09-13 0:55:37 2 [Note] WSREP: ####### Adjusting cert position: 90 -> 91 2023-09-13 0:55:37 0 [Note] WSREP: Service thread queue flushed. 2023-09-13 0:55:37 2 [Note] WSREP: ================================================ View: id: 8f95c1cc-430f-11ee-8404-8f9a8663c992:91 status: primary protocol_version: 4 capabilities: MULTI-MASTER, CERTIFICATION, PARALLEL_APPLYING, REPLAY, ISOLATION, PAUSE, CAUSAL_READ, INCREMENTAL_WS, UNORDERED, PREORDERED, STREAMING, NBO final: no own_index: 0 members(2): 0: af83c624-4477-11ee-a657-0627ac9b29e9, db001 1: c9e7c8ae-4476-11ee-92d0-f20d5682fc83, db002 ================================================= 2023-09-13 0:55:37 2 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification. 2023-09-13 0:55:37 2 [Note] WSREP: Lowest cert index boundary for CC from group: 91 2023-09-13 0:55:37 2 [Note] WSREP: Min available from gcache for CC from group: 1 2023-09-13 0:55:37 0 [Note] WSREP: Member 1.0 (db002) requested state transfer from 'any'. Selected 0.0 (db001)(SYNCED) as donor. 2023-09-13 0:55:37 0 [Note] WSREP: Shifting SYNCED -> DONOR/DESYNCED (TO: 91) 2023-09-13 0:55:37 2 [Note] WSREP: Detected STR version: 1, req_len: 131, req: STRv1 2023-09-13 0:55:37 2 [Note] WSREP: Cert index preload: 91 -> 91 2023-09-13 0:55:37 2 [Note] WSREP: Server status change synced -> donor 2023-09-13 0:55:37 2 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification. 2023-09-13 0:55:37 0 [Note] WSREP: async IST sender starting to serve tcp://192.168.47.42:4568 sending 91-91, preload starts from 91 2023-09-13 0:55:37 0 [Note] WSREP: IST sender 91 -> 91 2023-09-13 0:55:37 0 [Note] WSREP: Donor monitor thread started to monitor 2023-09-13 0:55:37 0 [Note] WSREP: Running: 'wsrep_sst_mariabackup --role 'donor' --address '192.168.47.42:4444/xtrabackup_sst//1' --local-port 3306 --socket '/var/lib/mysql/mysql.sock' --progress 0 --datadir '/var/lib/mysql/' --gtid '8f95c1cc-430f-11ee-8404-8f9a8663c992:91' --gtid-domain-id 0 --binlog '/var/lib/mysql/bin' --mysqld-args --wsrep-new-cluster --wsrep_start_position=8f95c1cc-430f-11ee-8404-8f9a8663c992:89,0-1-33' 2023-09-13 0:55:37 2 [Note] WSREP: sst_donor_thread signaled with 0 WSREP_SST: [INFO] mariabackup SST started on donor (20230913 00:55:37.688) WSREP_SST: [INFO] SSL configuration: CA='', CAPATH='', CERT='', KEY='', MODE='DISABLED', encrypt='0' (20230913 00:55:37.727) WSREP_SST: [INFO] Progress reporting tool pv not found in path: /usr//bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/usr/sbin:/usr/bin:/sbin:/bin (20230913 00:55:37.824) WSREP_SST: [INFO] Disabling all progress/rate-limiting (20230913 00:55:37.826) WSREP_SST: [INFO] Streaming with mbstream (20230913 00:55:37.997) WSREP_SST: [INFO] Using socat as streamer (20230913 00:55:37.999) WSREP_SST: [INFO] Using '/var/lib/mysql/tmp.23l7f5jblf' as mariadb-backup temporary directory (20230913 00:55:38.011) WSREP_SST: [INFO] Using '/tmp/tmp.t42TMS6WSR' as mariadb-abackup working directory (20230913 00:55:38.015) .. ..... 2023-09-13 0:55:50 0 [Note] WSREP: Member 0.0 (db001) synced with group. 2023-09-13 0:55:50 0 [Note] WSREP: Processing event queue:...100.0% (1/1 events) complete. 2023-09-13 0:55:50 0 [Note] WSREP: Shifting JOINED -> SYNCED (TO: 91) 2023-09-13 0:55:50 2 [Note] WSREP: Server db001 synced with group 2023-09-13 0:55:50 2 [Note] WSREP: Server status change joined -> synced 2023-09-13 0:55:50 2 [Note] WSREP: Synchronized with group, ready for connections 2023-09-13 0:55:50 2 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification. WSREP_SST: [INFO] Total time on donor: 0 seconds (20230913 00:55:50.414) WSREP_SST: [INFO] mariabackup SST completed on donor (20230913 00:55:50.417) WSREP_SST: [INFO] Cleaning up temporary directories (20230913 00:55:50.420) 2023-09-13 0:55:50 0 [Note] WSREP: Donor monitor thread ended with total time 13 sec 2023-09-13 0:55:51 0 [Note] WSREP: async IST sender served 2023-09-13 0:55:51 0 [Note] WSREP: 1.0 (db002): State transfer from 0.0 (db001) complete. 2023-09-13 0:55:51 0 [Note] WSREP: Member 1.0 (db002) synced with group. {code} https://github.com/MariaDB/server/blob/e987b9350cb83038c73a165922986441f345a3a9/scripts/wsrep_sst_mariabackup.sh#L1030 Donor node {code} [root@gcn-01 ~]# cat /etc/my.cnf\|grep -i tmpdir tmpdir = /var/lib/mysql/ {code} Joiner node {code} [root@gcn-02 mysql]# cat /etc/my.cnf\|grep -i tmpdir tmpdir = /var/lib/mysql/ {code}	During SST for joiner node , wsrep_sst_mariabackup consider /tmp as a working directory rather then the user defined. My understanding during SST wsrep_sst_mariabackup should be using tmpdir defined under [mysqld] section not the* /tmp* dir. Log from Donor while Joiner to trying to get sync using SST {code} 2023-09-13 0:55:36 0 [Note] WSREP: (af83c624-a657, 'tcp://0.0.0.0:4567') connection established to c9e7c8ae-92d0 tcp://192.168.47.42:4567 2023-09-13 0:55:36 0 [Note] WSREP: (af83c624-a657, 'tcp://0.0.0.0:4567') turning message relay requesting on, nonlive peers: 2023-09-13 0:55:37 0 [Note] WSREP: declaring c9e7c8ae-92d0 at tcp://192.168.47.42:4567 stable 2023-09-13 0:55:37 0 [Note] WSREP: Node af83c624-a657 state prim 2023-09-13 0:55:37 0 [Note] WSREP: view(view_id(PRIM,af83c624-a657,76) memb { af83c624-a657,0 c9e7c8ae-92d0,0 } joined { } left { } partitioned { }) 2023-09-13 0:55:37 0 [Note] WSREP: save pc into disk 2023-09-13 0:55:37 0 [Note] WSREP: New COMPONENT: primary = yes, bootstrap = no, my_idx = 0, memb_num = 2 2023-09-13 0:55:37 0 [Note] WSREP: STATE_EXCHANGE: sent state UUID: c7a87140-51f1-11ee-87cf-87f3f3a5fe55 2023-09-13 0:55:37 0 [Note] WSREP: STATE EXCHANGE: sent state msg: c7a87140-51f1-11ee-87cf-87f3f3a5fe55 2023-09-13 0:55:37 0 [Note] WSREP: STATE EXCHANGE: got state msg: c7a87140-51f1-11ee-87cf-87f3f3a5fe55 from 0 (db001) 2023-09-13 0:55:37 0 [Note] WSREP: STATE EXCHANGE: got state msg: c7a87140-51f1-11ee-87cf-87f3f3a5fe55 from 1 (db002) 2023-09-13 0:55:37 0 [Note] WSREP: Quorum results: version = 6, component = PRIMARY, conf_id = 1, members = 1/2 (joined/total), act_id = 90, last_appl. = 89, protocols = 2/10/4 (gcs/repl/appl), vote policy= 0, group UUID = 8f95c1cc-430f-11ee-8404-8f9a8663c992 2023-09-13 0:55:37 0 [Note] WSREP: Flow-control interval: [23, 23] 2023-09-13 0:55:37 2 [Note] WSREP: ####### processing CC 91, local, ordered 2023-09-13 0:55:37 2 [Note] WSREP: ####### My UUID: af83c624-4477-11ee-a657-0627ac9b29e9 2023-09-13 0:55:37 2 [Note] WSREP: Skipping cert index reset 2023-09-13 0:55:37 2 [Note] WSREP: REPL Protocols: 10 (5) 2023-09-13 0:55:37 2 [Note] WSREP: ####### Adjusting cert position: 90 -> 91 2023-09-13 0:55:37 0 [Note] WSREP: Service thread queue flushed. 2023-09-13 0:55:37 2 [Note] WSREP: ================================================ View: id: 8f95c1cc-430f-11ee-8404-8f9a8663c992:91 status: primary protocol_version: 4 capabilities: MULTI-MASTER, CERTIFICATION, PARALLEL_APPLYING, REPLAY, ISOLATION, PAUSE, CAUSAL_READ, INCREMENTAL_WS, UNORDERED, PREORDERED, STREAMING, NBO final: no own_index: 0 members(2): 0: af83c624-4477-11ee-a657-0627ac9b29e9, db001 1: c9e7c8ae-4476-11ee-92d0-f20d5682fc83, db002 ================================================= 2023-09-13 0:55:37 2 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification. 2023-09-13 0:55:37 2 [Note] WSREP: Lowest cert index boundary for CC from group: 91 2023-09-13 0:55:37 2 [Note] WSREP: Min available from gcache for CC from group: 1 2023-09-13 0:55:37 0 [Note] WSREP: Member 1.0 (db002) requested state transfer from 'any'. Selected 0.0 (db001)(SYNCED) as donor. 2023-09-13 0:55:37 0 [Note] WSREP: Shifting SYNCED -> DONOR/DESYNCED (TO: 91) 2023-09-13 0:55:37 2 [Note] WSREP: Detected STR version: 1, req_len: 131, req: STRv1 2023-09-13 0:55:37 2 [Note] WSREP: Cert index preload: 91 -> 91 2023-09-13 0:55:37 2 [Note] WSREP: Server status change synced -> donor 2023-09-13 0:55:37 2 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification. 2023-09-13 0:55:37 0 [Note] WSREP: async IST sender starting to serve tcp://192.168.47.42:4568 sending 91-91, preload starts from 91 2023-09-13 0:55:37 0 [Note] WSREP: IST sender 91 -> 91 2023-09-13 0:55:37 0 [Note] WSREP: Donor monitor thread started to monitor 2023-09-13 0:55:37 0 [Note] WSREP: Running: 'wsrep_sst_mariabackup --role 'donor' --address '192.168.47.42:4444/xtrabackup_sst//1' --local-port 3306 --socket '/var/lib/mysql/mysql.sock' --progress 0 --datadir '/var/lib/mysql/' --gtid '8f95c1cc-430f-11ee-8404-8f9a8663c992:91' --gtid-domain-id 0 --binlog '/var/lib/mysql/bin' --mysqld-args --wsrep-new-cluster --wsrep_start_position=8f95c1cc-430f-11ee-8404-8f9a8663c992:89,0-1-33' 2023-09-13 0:55:37 2 [Note] WSREP: sst_donor_thread signaled with 0 WSREP_SST: [INFO] mariabackup SST started on donor (20230913 00:55:37.688) WSREP_SST: [INFO] SSL configuration: CA='', CAPATH='', CERT='', KEY='', MODE='DISABLED', encrypt='0' (20230913 00:55:37.727) WSREP_SST: [INFO] Progress reporting tool pv not found in path: /usr//bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/usr/sbin:/usr/bin:/sbin:/bin (20230913 00:55:37.824) WSREP_SST: [INFO] Disabling all progress/rate-limiting (20230913 00:55:37.826) WSREP_SST: [INFO] Streaming with mbstream (20230913 00:55:37.997) WSREP_SST: [INFO] Using socat as streamer (20230913 00:55:37.999) {color:#DE350B}WSREP_SST: [INFO] Using '/var/lib/mysql/tmp.23l7f5jblf' as mariadb-backup temporary directory (20230913 00:55:38.011) WSREP_SST: [INFO] Using '/tmp/tmp.t42TMS6WSR' as mariadb-abackup working directory (20230913 00:55:38.015){color} .. ..... 2023-09-13 0:55:50 0 [Note] WSREP: Member 0.0 (db001) synced with group. 2023-09-13 0:55:50 0 [Note] WSREP: Processing event queue:...100.0% (1/1 events) complete. 2023-09-13 0:55:50 0 [Note] WSREP: Shifting JOINED -> SYNCED (TO: 91) 2023-09-13 0:55:50 2 [Note] WSREP: Server db001 synced with group 2023-09-13 0:55:50 2 [Note] WSREP: Server status change joined -> synced 2023-09-13 0:55:50 2 [Note] WSREP: Synchronized with group, ready for connections 2023-09-13 0:55:50 2 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification. WSREP_SST: [INFO] Total time on donor: 0 seconds (20230913 00:55:50.414) WSREP_SST: [INFO] mariabackup SST completed on donor (20230913 00:55:50.417) WSREP_SST: [INFO] Cleaning up temporary directories (20230913 00:55:50.420) 2023-09-13 0:55:50 0 [Note] WSREP: Donor monitor thread ended with total time 13 sec 2023-09-13 0:55:51 0 [Note] WSREP: async IST sender served 2023-09-13 0:55:51 0 [Note] WSREP: 1.0 (db002): State transfer from 0.0 (db001) complete. 2023-09-13 0:55:51 0 [Note] WSREP: Member 1.0 (db002) synced with group. {code} https://github.com/MariaDB/server/blob/e987b9350cb83038c73a165922986441f345a3a9/scripts/wsrep_sst_mariabackup.sh#L1030 Donor node {code} [root@gcn-01 ~]# cat /etc/my.cnf\|grep -i tmpdir tmpdir = /var/lib/mysql/ {code} Joiner node {code} [root@gcn-02 mysql]# cat /etc/my.cnf\|grep -i tmpdir tmpdir = /var/lib/mysql/ {code}

Pramod Mahto made changes - 2023-09-13 05:17

Description

During SST for joiner node , *wsrep_sst_mariabackup* consider */tmp* as a working directory rather then the user defined.

My understanding during SST wsrep_sst_mariabackup should be using *tmpdir* defined under *[mysqld]* section not the* /tmp* dir.

Log from Donor while Joiner to trying to get sync using SST

{code}

2023-09-13 0:55:36 0 [Note] WSREP: (af83c624-a657, 'tcp://0.0.0.0:4567') connection established to c9e7c8ae-92d0 tcp://192.168.47.42:4567
2023-09-13 0:55:36 0 [Note] WSREP: (af83c624-a657, 'tcp://0.0.0.0:4567') turning message relay requesting on, nonlive peers:
2023-09-13 0:55:37 0 [Note] WSREP: declaring c9e7c8ae-92d0 at tcp://192.168.47.42:4567 stable
2023-09-13 0:55:37 0 [Note] WSREP: Node af83c624-a657 state prim
2023-09-13 0:55:37 0 [Note] WSREP: view(view_id(PRIM,af83c624-a657,76) memb {
        af83c624-a657,0
        c9e7c8ae-92d0,0
} joined {
} left {
} partitioned {
})
2023-09-13 0:55:37 0 [Note] WSREP: save pc into disk
2023-09-13 0:55:37 0 [Note] WSREP: New COMPONENT: primary = yes, bootstrap = no, my_idx = 0, memb_num = 2
2023-09-13 0:55:37 0 [Note] WSREP: STATE_EXCHANGE: sent state UUID: c7a87140-51f1-11ee-87cf-87f3f3a5fe55
2023-09-13 0:55:37 0 [Note] WSREP: STATE EXCHANGE: sent state msg: c7a87140-51f1-11ee-87cf-87f3f3a5fe55
2023-09-13 0:55:37 0 [Note] WSREP: STATE EXCHANGE: got state msg: c7a87140-51f1-11ee-87cf-87f3f3a5fe55 from 0 (db001)
2023-09-13 0:55:37 0 [Note] WSREP: STATE EXCHANGE: got state msg: c7a87140-51f1-11ee-87cf-87f3f3a5fe55 from 1 (db002)
2023-09-13 0:55:37 0 [Note] WSREP: Quorum results:
        version = 6,
        component = PRIMARY,
        conf_id = 1,
        members = 1/2 (joined/total),
        act_id = 90,
        last_appl. = 89,
        protocols = 2/10/4 (gcs/repl/appl),
        vote policy= 0,
        group UUID = 8f95c1cc-430f-11ee-8404-8f9a8663c992
2023-09-13 0:55:37 0 [Note] WSREP: Flow-control interval: [23, 23]
2023-09-13 0:55:37 2 [Note] WSREP: ####### processing CC 91, local, ordered
2023-09-13 0:55:37 2 [Note] WSREP: ####### My UUID: af83c624-4477-11ee-a657-0627ac9b29e9
2023-09-13 0:55:37 2 [Note] WSREP: Skipping cert index reset
2023-09-13 0:55:37 2 [Note] WSREP: REPL Protocols: 10 (5)
2023-09-13 0:55:37 2 [Note] WSREP: ####### Adjusting cert position: 90 -> 91
2023-09-13 0:55:37 0 [Note] WSREP: Service thread queue flushed.
2023-09-13 0:55:37 2 [Note] WSREP: ================================================
View:
  id: 8f95c1cc-430f-11ee-8404-8f9a8663c992:91
  status: primary
  protocol_version: 4
  capabilities: MULTI-MASTER, CERTIFICATION, PARALLEL_APPLYING, REPLAY, ISOLATION, PAUSE, CAUSAL_READ, INCREMENTAL_WS, UNORDERED, PREORDERED, STREAMING, NBO
  final: no
  own_index: 0
  members(2):
        0: af83c624-4477-11ee-a657-0627ac9b29e9, db001
        1: c9e7c8ae-4476-11ee-92d0-f20d5682fc83, db002
=================================================
2023-09-13 0:55:37 2 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
2023-09-13 0:55:37 2 [Note] WSREP: Lowest cert index boundary for CC from group: 91
2023-09-13 0:55:37 2 [Note] WSREP: Min available from gcache for CC from group: 1
2023-09-13 0:55:37 0 [Note] WSREP: Member 1.0 (db002) requested state transfer from '*any*'. Selected 0.0 (db001)(SYNCED) as donor.
2023-09-13 0:55:37 0 [Note] WSREP: Shifting SYNCED -> DONOR/DESYNCED (TO: 91)
2023-09-13 0:55:37 2 [Note] WSREP: Detected STR version: 1, req_len: 131, req: STRv1
2023-09-13 0:55:37 2 [Note] WSREP: Cert index preload: 91 -> 91
2023-09-13 0:55:37 2 [Note] WSREP: Server status change synced -> donor
2023-09-13 0:55:37 2 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
2023-09-13 0:55:37 0 [Note] WSREP: async IST sender starting to serve tcp://192.168.47.42:4568 sending 91-91, preload starts from 91
2023-09-13 0:55:37 0 [Note] WSREP: IST sender 91 -> 91
2023-09-13 0:55:37 0 [Note] WSREP: Donor monitor thread started to monitor
2023-09-13 0:55:37 0 [Note] WSREP: Running: 'wsrep_sst_mariabackup --role 'donor' --address '192.168.47.42:4444/xtrabackup_sst//1' --local-port 3306 --socket '/var/lib/mysql/mysql.sock' --progress 0 --datadir '/var/lib/mysql/' --gtid '8f95c1cc-430f-11ee-8404-8f9a8663c992:91' --gtid-domain-id 0 --binlog '/var/lib/mysql/bin' --mysqld-args --wsrep-new-cluster --wsrep_start_position=8f95c1cc-430f-11ee-8404-8f9a8663c992:89,0-1-33'
2023-09-13 0:55:37 2 [Note] WSREP: sst_donor_thread signaled with 0
WSREP_SST: [INFO] mariabackup SST started on donor (20230913 00:55:37.688)
WSREP_SST: [INFO] SSL configuration: CA='', CAPATH='', CERT='', KEY='', MODE='DISABLED', encrypt='0' (20230913 00:55:37.727)
WSREP_SST: [INFO] Progress reporting tool pv not found in path: /usr//bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/usr/sbin:/usr/bin:/sbin:/bin (20230913 00:55:37.824)
WSREP_SST: [INFO] Disabling all progress/rate-limiting (20230913 00:55:37.826)
WSREP_SST: [INFO] Streaming with mbstream (20230913 00:55:37.997)
WSREP_SST: [INFO] Using socat as streamer (20230913 00:55:37.999)

{color:#DE350B}WSREP_SST: [INFO] Using '/var/lib/mysql/tmp.23l7f5jblf' as mariadb-backup temporary directory (20230913 00:55:38.011)
WSREP_SST: [INFO] Using '/tmp/tmp.t42TMS6WSR' as mariadb-abackup working directory (20230913 00:55:38.015){color}

..
.....
2023-09-13 0:55:50 0 [Note] WSREP: Member 0.0 (db001) synced with group.
2023-09-13 0:55:50 0 [Note] WSREP: Processing event queue:...100.0% (1/1 events) complete.
2023-09-13 0:55:50 0 [Note] WSREP: Shifting JOINED -> SYNCED (TO: 91)
2023-09-13 0:55:50 2 [Note] WSREP: Server db001 synced with group
2023-09-13 0:55:50 2 [Note] WSREP: Server status change joined -> synced
2023-09-13 0:55:50 2 [Note] WSREP: Synchronized with group, ready for connections
2023-09-13 0:55:50 2 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
WSREP_SST: [INFO] Total time on donor: 0 seconds (20230913 00:55:50.414)
WSREP_SST: [INFO] mariabackup SST completed on donor (20230913 00:55:50.417)
WSREP_SST: [INFO] Cleaning up temporary directories (20230913 00:55:50.420)
2023-09-13 0:55:50 0 [Note] WSREP: Donor monitor thread ended with total time 13 sec
2023-09-13 0:55:51 0 [Note] WSREP: async IST sender served
2023-09-13 0:55:51 0 [Note] WSREP: 1.0 (db002): State transfer from 0.0 (db001) complete.
2023-09-13 0:55:51 0 [Note] WSREP: Member 1.0 (db002) synced with group.

{code}

https://github.com/MariaDB/server/blob/e987b9350cb83038c73a165922986441f345a3a9/scripts/wsrep_sst_mariabackup.sh#L1030

Donor node
{code}

[root@gcn-01 ~]# cat /etc/my.cnf|grep -i tmpdir
tmpdir = /var/lib/mysql/

{code}

Joiner node

{code}

[root@gcn-02 mysql]# cat /etc/my.cnf|grep -i tmpdir
tmpdir = /var/lib/mysql/

{code}

During SST for joiner node , *wsrep_sst_mariabackup* consider */tmp* as a working directory rather then the user defined.

My understanding during SST wsrep_sst_mariabackup should be using *tmpdir* defined under *[mysqld]* section not the* /tmp* dir.

Log from Donor while Joiner to trying to get sync using SST

{code}

2023-09-13 0:55:36 0 [Note] WSREP: (af83c624-a657, 'tcp://0.0.0.0:4567') connection established to c9e7c8ae-92d0 tcp://192.168.47.42:4567
2023-09-13 0:55:36 0 [Note] WSREP: (af83c624-a657, 'tcp://0.0.0.0:4567') turning message relay requesting on, nonlive peers:
2023-09-13 0:55:37 0 [Note] WSREP: declaring c9e7c8ae-92d0 at tcp://192.168.47.42:4567 stable
2023-09-13 0:55:37 0 [Note] WSREP: Node af83c624-a657 state prim
2023-09-13 0:55:37 0 [Note] WSREP: view(view_id(PRIM,af83c624-a657,76) memb {
        af83c624-a657,0
        c9e7c8ae-92d0,0
} joined {
} left {
} partitioned {
})
2023-09-13 0:55:37 0 [Note] WSREP: save pc into disk
2023-09-13 0:55:37 0 [Note] WSREP: New COMPONENT: primary = yes, bootstrap = no, my_idx = 0, memb_num = 2
2023-09-13 0:55:37 0 [Note] WSREP: STATE_EXCHANGE: sent state UUID: c7a87140-51f1-11ee-87cf-87f3f3a5fe55
2023-09-13 0:55:37 0 [Note] WSREP: STATE EXCHANGE: sent state msg: c7a87140-51f1-11ee-87cf-87f3f3a5fe55
2023-09-13 0:55:37 0 [Note] WSREP: STATE EXCHANGE: got state msg: c7a87140-51f1-11ee-87cf-87f3f3a5fe55 from 0 (db001)
2023-09-13 0:55:37 0 [Note] WSREP: STATE EXCHANGE: got state msg: c7a87140-51f1-11ee-87cf-87f3f3a5fe55 from 1 (db002)
2023-09-13 0:55:37 0 [Note] WSREP: Quorum results:
        version = 6,
        component = PRIMARY,
        conf_id = 1,
        members = 1/2 (joined/total),
        act_id = 90,
        last_appl. = 89,
        protocols = 2/10/4 (gcs/repl/appl),
        vote policy= 0,
        group UUID = 8f95c1cc-430f-11ee-8404-8f9a8663c992
2023-09-13 0:55:37 0 [Note] WSREP: Flow-control interval: [23, 23]
2023-09-13 0:55:37 2 [Note] WSREP: ####### processing CC 91, local, ordered
2023-09-13 0:55:37 2 [Note] WSREP: ####### My UUID: af83c624-4477-11ee-a657-0627ac9b29e9
2023-09-13 0:55:37 2 [Note] WSREP: Skipping cert index reset
2023-09-13 0:55:37 2 [Note] WSREP: REPL Protocols: 10 (5)
2023-09-13 0:55:37 2 [Note] WSREP: ####### Adjusting cert position: 90 -> 91
2023-09-13 0:55:37 0 [Note] WSREP: Service thread queue flushed.
2023-09-13 0:55:37 2 [Note] WSREP: ================================================
View:
  id: 8f95c1cc-430f-11ee-8404-8f9a8663c992:91
  status: primary
  protocol_version: 4
  capabilities: MULTI-MASTER, CERTIFICATION, PARALLEL_APPLYING, REPLAY, ISOLATION, PAUSE, CAUSAL_READ, INCREMENTAL_WS, UNORDERED, PREORDERED, STREAMING, NBO
  final: no
  own_index: 0
  members(2):
        0: af83c624-4477-11ee-a657-0627ac9b29e9, db001
        1: c9e7c8ae-4476-11ee-92d0-f20d5682fc83, db002
=================================================
2023-09-13 0:55:37 2 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
2023-09-13 0:55:37 2 [Note] WSREP: Lowest cert index boundary for CC from group: 91
2023-09-13 0:55:37 2 [Note] WSREP: Min available from gcache for CC from group: 1
2023-09-13 0:55:37 0 [Note] WSREP: Member 1.0 (db002) requested state transfer from '*any*'. Selected 0.0 (db001)(SYNCED) as donor.
2023-09-13 0:55:37 0 [Note] WSREP: Shifting SYNCED -> DONOR/DESYNCED (TO: 91)
2023-09-13 0:55:37 2 [Note] WSREP: Detected STR version: 1, req_len: 131, req: STRv1
2023-09-13 0:55:37 2 [Note] WSREP: Cert index preload: 91 -> 91
2023-09-13 0:55:37 2 [Note] WSREP: Server status change synced -> donor
2023-09-13 0:55:37 2 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
2023-09-13 0:55:37 0 [Note] WSREP: async IST sender starting to serve tcp://192.168.47.42:4568 sending 91-91, preload starts from 91
2023-09-13 0:55:37 0 [Note] WSREP: IST sender 91 -> 91
2023-09-13 0:55:37 0 [Note] WSREP: Donor monitor thread started to monitor
2023-09-13 0:55:37 0 [Note] WSREP: Running: 'wsrep_sst_mariabackup --role 'donor' --address '192.168.47.42:4444/xtrabackup_sst//1' --local-port 3306 --socket '/var/lib/mysql/mysql.sock' --progress 0 --datadir '/var/lib/mysql/' --gtid '8f95c1cc-430f-11ee-8404-8f9a8663c992:91' --gtid-domain-id 0 --binlog '/var/lib/mysql/bin' --mysqld-args --wsrep-new-cluster --wsrep_start_position=8f95c1cc-430f-11ee-8404-8f9a8663c992:89,0-1-33'
2023-09-13 0:55:37 2 [Note] WSREP: sst_donor_thread signaled with 0
WSREP_SST: [INFO] mariabackup SST started on donor (20230913 00:55:37.688)
WSREP_SST: [INFO] SSL configuration: CA='', CAPATH='', CERT='', KEY='', MODE='DISABLED', encrypt='0' (20230913 00:55:37.727)
WSREP_SST: [INFO] Progress reporting tool pv not found in path: /usr//bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/usr/sbin:/usr/bin:/sbin:/bin (20230913 00:55:37.824)
WSREP_SST: [INFO] Disabling all progress/rate-limiting (20230913 00:55:37.826)
WSREP_SST: [INFO] Streaming with mbstream (20230913 00:55:37.997)
WSREP_SST: [INFO] Using socat as streamer (20230913 00:55:37.999)

{color:#DE350B}WSREP_SST: [INFO] Using '/var/lib/mysql/tmp.23l7f5jblf' as mariadb-backup temporary directory (20230913 00:55:38.011){color}
{color:#DE350B}WSREP_SST: [INFO] Using '/tmp/tmp.t42TMS6WSR' as mariadb-abackup working directory (20230913 00:55:38.015){color}

..
.....
2023-09-13 0:55:50 0 [Note] WSREP: Member 0.0 (db001) synced with group.
2023-09-13 0:55:50 0 [Note] WSREP: Processing event queue:...100.0% (1/1 events) complete.
2023-09-13 0:55:50 0 [Note] WSREP: Shifting JOINED -> SYNCED (TO: 91)
2023-09-13 0:55:50 2 [Note] WSREP: Server db001 synced with group
2023-09-13 0:55:50 2 [Note] WSREP: Server status change joined -> synced
2023-09-13 0:55:50 2 [Note] WSREP: Synchronized with group, ready for connections
2023-09-13 0:55:50 2 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
WSREP_SST: [INFO] Total time on donor: 0 seconds (20230913 00:55:50.414)
WSREP_SST: [INFO] mariabackup SST completed on donor (20230913 00:55:50.417)
WSREP_SST: [INFO] Cleaning up temporary directories (20230913 00:55:50.420)
2023-09-13 0:55:50 0 [Note] WSREP: Donor monitor thread ended with total time 13 sec
2023-09-13 0:55:51 0 [Note] WSREP: async IST sender served
2023-09-13 0:55:51 0 [Note] WSREP: 1.0 (db002): State transfer from 0.0 (db001) complete.
2023-09-13 0:55:51 0 [Note] WSREP: Member 1.0 (db002) synced with group.

{code}

https://github.com/MariaDB/server/blob/e987b9350cb83038c73a165922986441f345a3a9/scripts/wsrep_sst_mariabackup.sh#L1030

Donor node
{code}

[root@gcn-01 ~]# cat /etc/my.cnf|grep -i tmpdir
tmpdir = /var/lib/mysql/

{code}

Joiner node

{code}

[root@gcn-02 mysql]# cat /etc/my.cnf|grep -i tmpdir
tmpdir = /var/lib/mysql/

{code}

Pramod Mahto made changes - 2023-09-13 05:17

Description

During SST for joiner node , *wsrep_sst_mariabackup* consider */tmp* as a working directory rather then the user defined.

My understanding during SST wsrep_sst_mariabackup should be using *tmpdir* defined under *[mysqld]* section not the* /tmp* dir.

Log from Donor while Joiner to trying to get sync using SST

{code}

2023-09-13 0:55:36 0 [Note] WSREP: (af83c624-a657, 'tcp://0.0.0.0:4567') connection established to c9e7c8ae-92d0 tcp://192.168.47.42:4567
2023-09-13 0:55:36 0 [Note] WSREP: (af83c624-a657, 'tcp://0.0.0.0:4567') turning message relay requesting on, nonlive peers:
2023-09-13 0:55:37 0 [Note] WSREP: declaring c9e7c8ae-92d0 at tcp://192.168.47.42:4567 stable
2023-09-13 0:55:37 0 [Note] WSREP: Node af83c624-a657 state prim
2023-09-13 0:55:37 0 [Note] WSREP: view(view_id(PRIM,af83c624-a657,76) memb {
        af83c624-a657,0
        c9e7c8ae-92d0,0
} joined {
} left {
} partitioned {
})
2023-09-13 0:55:37 0 [Note] WSREP: save pc into disk
2023-09-13 0:55:37 0 [Note] WSREP: New COMPONENT: primary = yes, bootstrap = no, my_idx = 0, memb_num = 2
2023-09-13 0:55:37 0 [Note] WSREP: STATE_EXCHANGE: sent state UUID: c7a87140-51f1-11ee-87cf-87f3f3a5fe55
2023-09-13 0:55:37 0 [Note] WSREP: STATE EXCHANGE: sent state msg: c7a87140-51f1-11ee-87cf-87f3f3a5fe55
2023-09-13 0:55:37 0 [Note] WSREP: STATE EXCHANGE: got state msg: c7a87140-51f1-11ee-87cf-87f3f3a5fe55 from 0 (db001)
2023-09-13 0:55:37 0 [Note] WSREP: STATE EXCHANGE: got state msg: c7a87140-51f1-11ee-87cf-87f3f3a5fe55 from 1 (db002)
2023-09-13 0:55:37 0 [Note] WSREP: Quorum results:
        version = 6,
        component = PRIMARY,
        conf_id = 1,
        members = 1/2 (joined/total),
        act_id = 90,
        last_appl. = 89,
        protocols = 2/10/4 (gcs/repl/appl),
        vote policy= 0,
        group UUID = 8f95c1cc-430f-11ee-8404-8f9a8663c992
2023-09-13 0:55:37 0 [Note] WSREP: Flow-control interval: [23, 23]
2023-09-13 0:55:37 2 [Note] WSREP: ####### processing CC 91, local, ordered
2023-09-13 0:55:37 2 [Note] WSREP: ####### My UUID: af83c624-4477-11ee-a657-0627ac9b29e9
2023-09-13 0:55:37 2 [Note] WSREP: Skipping cert index reset
2023-09-13 0:55:37 2 [Note] WSREP: REPL Protocols: 10 (5)
2023-09-13 0:55:37 2 [Note] WSREP: ####### Adjusting cert position: 90 -> 91
2023-09-13 0:55:37 0 [Note] WSREP: Service thread queue flushed.
2023-09-13 0:55:37 2 [Note] WSREP: ================================================
View:
  id: 8f95c1cc-430f-11ee-8404-8f9a8663c992:91
  status: primary
  protocol_version: 4
  capabilities: MULTI-MASTER, CERTIFICATION, PARALLEL_APPLYING, REPLAY, ISOLATION, PAUSE, CAUSAL_READ, INCREMENTAL_WS, UNORDERED, PREORDERED, STREAMING, NBO
  final: no
  own_index: 0
  members(2):
        0: af83c624-4477-11ee-a657-0627ac9b29e9, db001
        1: c9e7c8ae-4476-11ee-92d0-f20d5682fc83, db002
=================================================
2023-09-13 0:55:37 2 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
2023-09-13 0:55:37 2 [Note] WSREP: Lowest cert index boundary for CC from group: 91
2023-09-13 0:55:37 2 [Note] WSREP: Min available from gcache for CC from group: 1
2023-09-13 0:55:37 0 [Note] WSREP: Member 1.0 (db002) requested state transfer from '*any*'. Selected 0.0 (db001)(SYNCED) as donor.
2023-09-13 0:55:37 0 [Note] WSREP: Shifting SYNCED -> DONOR/DESYNCED (TO: 91)
2023-09-13 0:55:37 2 [Note] WSREP: Detected STR version: 1, req_len: 131, req: STRv1
2023-09-13 0:55:37 2 [Note] WSREP: Cert index preload: 91 -> 91
2023-09-13 0:55:37 2 [Note] WSREP: Server status change synced -> donor
2023-09-13 0:55:37 2 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
2023-09-13 0:55:37 0 [Note] WSREP: async IST sender starting to serve tcp://192.168.47.42:4568 sending 91-91, preload starts from 91
2023-09-13 0:55:37 0 [Note] WSREP: IST sender 91 -> 91
2023-09-13 0:55:37 0 [Note] WSREP: Donor monitor thread started to monitor
2023-09-13 0:55:37 0 [Note] WSREP: Running: 'wsrep_sst_mariabackup --role 'donor' --address '192.168.47.42:4444/xtrabackup_sst//1' --local-port 3306 --socket '/var/lib/mysql/mysql.sock' --progress 0 --datadir '/var/lib/mysql/' --gtid '8f95c1cc-430f-11ee-8404-8f9a8663c992:91' --gtid-domain-id 0 --binlog '/var/lib/mysql/bin' --mysqld-args --wsrep-new-cluster --wsrep_start_position=8f95c1cc-430f-11ee-8404-8f9a8663c992:89,0-1-33'
2023-09-13 0:55:37 2 [Note] WSREP: sst_donor_thread signaled with 0
WSREP_SST: [INFO] mariabackup SST started on donor (20230913 00:55:37.688)
WSREP_SST: [INFO] SSL configuration: CA='', CAPATH='', CERT='', KEY='', MODE='DISABLED', encrypt='0' (20230913 00:55:37.727)
WSREP_SST: [INFO] Progress reporting tool pv not found in path: /usr//bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/usr/sbin:/usr/bin:/sbin:/bin (20230913 00:55:37.824)
WSREP_SST: [INFO] Disabling all progress/rate-limiting (20230913 00:55:37.826)
WSREP_SST: [INFO] Streaming with mbstream (20230913 00:55:37.997)
WSREP_SST: [INFO] Using socat as streamer (20230913 00:55:37.999)

{color:#DE350B}WSREP_SST: [INFO] Using '/var/lib/mysql/tmp.23l7f5jblf' as mariadb-backup temporary directory (20230913 00:55:38.011){color}
{color:#DE350B}WSREP_SST: [INFO] Using '/tmp/tmp.t42TMS6WSR' as mariadb-abackup working directory (20230913 00:55:38.015){color}

..
.....
2023-09-13 0:55:50 0 [Note] WSREP: Member 0.0 (db001) synced with group.
2023-09-13 0:55:50 0 [Note] WSREP: Processing event queue:...100.0% (1/1 events) complete.
2023-09-13 0:55:50 0 [Note] WSREP: Shifting JOINED -> SYNCED (TO: 91)
2023-09-13 0:55:50 2 [Note] WSREP: Server db001 synced with group
2023-09-13 0:55:50 2 [Note] WSREP: Server status change joined -> synced
2023-09-13 0:55:50 2 [Note] WSREP: Synchronized with group, ready for connections
2023-09-13 0:55:50 2 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
WSREP_SST: [INFO] Total time on donor: 0 seconds (20230913 00:55:50.414)
WSREP_SST: [INFO] mariabackup SST completed on donor (20230913 00:55:50.417)
WSREP_SST: [INFO] Cleaning up temporary directories (20230913 00:55:50.420)
2023-09-13 0:55:50 0 [Note] WSREP: Donor monitor thread ended with total time 13 sec
2023-09-13 0:55:51 0 [Note] WSREP: async IST sender served
2023-09-13 0:55:51 0 [Note] WSREP: 1.0 (db002): State transfer from 0.0 (db001) complete.
2023-09-13 0:55:51 0 [Note] WSREP: Member 1.0 (db002) synced with group.

{code}

https://github.com/MariaDB/server/blob/e987b9350cb83038c73a165922986441f345a3a9/scripts/wsrep_sst_mariabackup.sh#L1030

Donor node
{code}

[root@gcn-01 ~]# cat /etc/my.cnf|grep -i tmpdir
tmpdir = /var/lib/mysql/

{code}

Joiner node

{code}

[root@gcn-02 mysql]# cat /etc/my.cnf|grep -i tmpdir
tmpdir = /var/lib/mysql/

{code}

During SST for joiner node , *wsrep_sst_mariabackup* consider */tmp* as a working directory rather then the user defined.

My understanding during SST wsrep_sst_mariabackup should be using *tmpdir* defined under *[mysqld]* section not the* /tmp* dir.

Log from Donor while Joiner to trying to get sync using SST

{code}

2023-09-13 0:55:36 0 [Note] WSREP: (af83c624-a657, 'tcp://0.0.0.0:4567') connection established to c9e7c8ae-92d0 tcp://192.168.47.42:4567
2023-09-13 0:55:36 0 [Note] WSREP: (af83c624-a657, 'tcp://0.0.0.0:4567') turning message relay requesting on, nonlive peers:
2023-09-13 0:55:37 0 [Note] WSREP: declaring c9e7c8ae-92d0 at tcp://192.168.47.42:4567 stable
2023-09-13 0:55:37 0 [Note] WSREP: Node af83c624-a657 state prim
2023-09-13 0:55:37 0 [Note] WSREP: view(view_id(PRIM,af83c624-a657,76) memb {
        af83c624-a657,0
        c9e7c8ae-92d0,0
} joined {
} left {
} partitioned {
})
2023-09-13 0:55:37 0 [Note] WSREP: save pc into disk
2023-09-13 0:55:37 0 [Note] WSREP: New COMPONENT: primary = yes, bootstrap = no, my_idx = 0, memb_num = 2
2023-09-13 0:55:37 0 [Note] WSREP: STATE_EXCHANGE: sent state UUID: c7a87140-51f1-11ee-87cf-87f3f3a5fe55
2023-09-13 0:55:37 0 [Note] WSREP: STATE EXCHANGE: sent state msg: c7a87140-51f1-11ee-87cf-87f3f3a5fe55
2023-09-13 0:55:37 0 [Note] WSREP: STATE EXCHANGE: got state msg: c7a87140-51f1-11ee-87cf-87f3f3a5fe55 from 0 (db001)
2023-09-13 0:55:37 0 [Note] WSREP: STATE EXCHANGE: got state msg: c7a87140-51f1-11ee-87cf-87f3f3a5fe55 from 1 (db002)
2023-09-13 0:55:37 0 [Note] WSREP: Quorum results:
        version = 6,
        component = PRIMARY,
        conf_id = 1,
        members = 1/2 (joined/total),
        act_id = 90,
        last_appl. = 89,
        protocols = 2/10/4 (gcs/repl/appl),
        vote policy= 0,
        group UUID = 8f95c1cc-430f-11ee-8404-8f9a8663c992
2023-09-13 0:55:37 0 [Note] WSREP: Flow-control interval: [23, 23]
2023-09-13 0:55:37 2 [Note] WSREP: ####### processing CC 91, local, ordered
2023-09-13 0:55:37 2 [Note] WSREP: ####### My UUID: af83c624-4477-11ee-a657-0627ac9b29e9
2023-09-13 0:55:37 2 [Note] WSREP: Skipping cert index reset
2023-09-13 0:55:37 2 [Note] WSREP: REPL Protocols: 10 (5)
2023-09-13 0:55:37 2 [Note] WSREP: ####### Adjusting cert position: 90 -> 91
2023-09-13 0:55:37 0 [Note] WSREP: Service thread queue flushed.
2023-09-13 0:55:37 2 [Note] WSREP: ================================================
View:
  id: 8f95c1cc-430f-11ee-8404-8f9a8663c992:91
  status: primary
  protocol_version: 4
  capabilities: MULTI-MASTER, CERTIFICATION, PARALLEL_APPLYING, REPLAY, ISOLATION, PAUSE, CAUSAL_READ, INCREMENTAL_WS, UNORDERED, PREORDERED, STREAMING, NBO
  final: no
  own_index: 0
  members(2):
        0: af83c624-4477-11ee-a657-0627ac9b29e9, db001
        1: c9e7c8ae-4476-11ee-92d0-f20d5682fc83, db002
=================================================
2023-09-13 0:55:37 2 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
2023-09-13 0:55:37 2 [Note] WSREP: Lowest cert index boundary for CC from group: 91
2023-09-13 0:55:37 2 [Note] WSREP: Min available from gcache for CC from group: 1
2023-09-13 0:55:37 0 [Note] WSREP: Member 1.0 (db002) requested state transfer from '*any*'. Selected 0.0 (db001)(SYNCED) as donor.
2023-09-13 0:55:37 0 [Note] WSREP: Shifting SYNCED -> DONOR/DESYNCED (TO: 91)
2023-09-13 0:55:37 2 [Note] WSREP: Detected STR version: 1, req_len: 131, req: STRv1
2023-09-13 0:55:37 2 [Note] WSREP: Cert index preload: 91 -> 91
2023-09-13 0:55:37 2 [Note] WSREP: Server status change synced -> donor
2023-09-13 0:55:37 2 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
2023-09-13 0:55:37 0 [Note] WSREP: async IST sender starting to serve tcp://192.168.47.42:4568 sending 91-91, preload starts from 91
2023-09-13 0:55:37 0 [Note] WSREP: IST sender 91 -> 91
2023-09-13 0:55:37 0 [Note] WSREP: Donor monitor thread started to monitor
2023-09-13 0:55:37 0 [Note] WSREP: Running: 'wsrep_sst_mariabackup --role 'donor' --address '192.168.47.42:4444/xtrabackup_sst//1' --local-port 3306 --socket '/var/lib/mysql/mysql.sock' --progress 0 --datadir '/var/lib/mysql/' --gtid '8f95c1cc-430f-11ee-8404-8f9a8663c992:91' --gtid-domain-id 0 --binlog '/var/lib/mysql/bin' --mysqld-args --wsrep-new-cluster --wsrep_start_position=8f95c1cc-430f-11ee-8404-8f9a8663c992:89,0-1-33'
2023-09-13 0:55:37 2 [Note] WSREP: sst_donor_thread signaled with 0
WSREP_SST: [INFO] mariabackup SST started on donor (20230913 00:55:37.688)
WSREP_SST: [INFO] SSL configuration: CA='', CAPATH='', CERT='', KEY='', MODE='DISABLED', encrypt='0' (20230913 00:55:37.727)
WSREP_SST: [INFO] Progress reporting tool pv not found in path: /usr//bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/usr/sbin:/usr/bin:/sbin:/bin (20230913 00:55:37.824)
WSREP_SST: [INFO] Disabling all progress/rate-limiting (20230913 00:55:37.826)
WSREP_SST: [INFO] Streaming with mbstream (20230913 00:55:37.997)
WSREP_SST: [INFO] Using socat as streamer (20230913 00:55:37.999)

*_WSREP_SST: [INFO] Using '/var/lib/mysql/tmp.23l7f5jblf' as mariadb-backup temporary directory (20230913 00:55:38.011)
WSREP_SST: [INFO] Using '/tmp/tmp.t42TMS6WSR' as mariadb-abackup working directory (20230913 00:55:38.015)_*

..
.....
2023-09-13 0:55:50 0 [Note] WSREP: Member 0.0 (db001) synced with group.
2023-09-13 0:55:50 0 [Note] WSREP: Processing event queue:...100.0% (1/1 events) complete.
2023-09-13 0:55:50 0 [Note] WSREP: Shifting JOINED -> SYNCED (TO: 91)
2023-09-13 0:55:50 2 [Note] WSREP: Server db001 synced with group
2023-09-13 0:55:50 2 [Note] WSREP: Server status change joined -> synced
2023-09-13 0:55:50 2 [Note] WSREP: Synchronized with group, ready for connections
2023-09-13 0:55:50 2 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
WSREP_SST: [INFO] Total time on donor: 0 seconds (20230913 00:55:50.414)
WSREP_SST: [INFO] mariabackup SST completed on donor (20230913 00:55:50.417)
WSREP_SST: [INFO] Cleaning up temporary directories (20230913 00:55:50.420)
2023-09-13 0:55:50 0 [Note] WSREP: Donor monitor thread ended with total time 13 sec
2023-09-13 0:55:51 0 [Note] WSREP: async IST sender served
2023-09-13 0:55:51 0 [Note] WSREP: 1.0 (db002): State transfer from 0.0 (db001) complete.
2023-09-13 0:55:51 0 [Note] WSREP: Member 1.0 (db002) synced with group.

{code}

https://github.com/MariaDB/server/blob/e987b9350cb83038c73a165922986441f345a3a9/scripts/wsrep_sst_mariabackup.sh#L1030

Donor node
{code}

[root@gcn-01 ~]# cat /etc/my.cnf|grep -i tmpdir
tmpdir = /var/lib/mysql/

{code}

Joiner node

{code}

[root@gcn-02 mysql]# cat /etc/my.cnf|grep -i tmpdir
tmpdir = /var/lib/mysql/

{code}

Pramod Mahto made changes - 2023-09-13 05:18

Description

During SST for joiner node , *wsrep_sst_mariabackup* consider */tmp* as a working directory rather then the user defined.

My understanding during SST wsrep_sst_mariabackup should be using *tmpdir* defined under *[mysqld]* section not the* /tmp* dir.

Log from Donor while Joiner to trying to get sync using SST

{code}

2023-09-13 0:55:36 0 [Note] WSREP: (af83c624-a657, 'tcp://0.0.0.0:4567') connection established to c9e7c8ae-92d0 tcp://192.168.47.42:4567
2023-09-13 0:55:36 0 [Note] WSREP: (af83c624-a657, 'tcp://0.0.0.0:4567') turning message relay requesting on, nonlive peers:
2023-09-13 0:55:37 0 [Note] WSREP: declaring c9e7c8ae-92d0 at tcp://192.168.47.42:4567 stable
2023-09-13 0:55:37 0 [Note] WSREP: Node af83c624-a657 state prim
2023-09-13 0:55:37 0 [Note] WSREP: view(view_id(PRIM,af83c624-a657,76) memb {
        af83c624-a657,0
        c9e7c8ae-92d0,0
} joined {
} left {
} partitioned {
})
2023-09-13 0:55:37 0 [Note] WSREP: save pc into disk
2023-09-13 0:55:37 0 [Note] WSREP: New COMPONENT: primary = yes, bootstrap = no, my_idx = 0, memb_num = 2
2023-09-13 0:55:37 0 [Note] WSREP: STATE_EXCHANGE: sent state UUID: c7a87140-51f1-11ee-87cf-87f3f3a5fe55
2023-09-13 0:55:37 0 [Note] WSREP: STATE EXCHANGE: sent state msg: c7a87140-51f1-11ee-87cf-87f3f3a5fe55
2023-09-13 0:55:37 0 [Note] WSREP: STATE EXCHANGE: got state msg: c7a87140-51f1-11ee-87cf-87f3f3a5fe55 from 0 (db001)
2023-09-13 0:55:37 0 [Note] WSREP: STATE EXCHANGE: got state msg: c7a87140-51f1-11ee-87cf-87f3f3a5fe55 from 1 (db002)
2023-09-13 0:55:37 0 [Note] WSREP: Quorum results:
        version = 6,
        component = PRIMARY,
        conf_id = 1,
        members = 1/2 (joined/total),
        act_id = 90,
        last_appl. = 89,
        protocols = 2/10/4 (gcs/repl/appl),
        vote policy= 0,
        group UUID = 8f95c1cc-430f-11ee-8404-8f9a8663c992
2023-09-13 0:55:37 0 [Note] WSREP: Flow-control interval: [23, 23]
2023-09-13 0:55:37 2 [Note] WSREP: ####### processing CC 91, local, ordered
2023-09-13 0:55:37 2 [Note] WSREP: ####### My UUID: af83c624-4477-11ee-a657-0627ac9b29e9
2023-09-13 0:55:37 2 [Note] WSREP: Skipping cert index reset
2023-09-13 0:55:37 2 [Note] WSREP: REPL Protocols: 10 (5)
2023-09-13 0:55:37 2 [Note] WSREP: ####### Adjusting cert position: 90 -> 91
2023-09-13 0:55:37 0 [Note] WSREP: Service thread queue flushed.
2023-09-13 0:55:37 2 [Note] WSREP: ================================================
View:
  id: 8f95c1cc-430f-11ee-8404-8f9a8663c992:91
  status: primary
  protocol_version: 4
  capabilities: MULTI-MASTER, CERTIFICATION, PARALLEL_APPLYING, REPLAY, ISOLATION, PAUSE, CAUSAL_READ, INCREMENTAL_WS, UNORDERED, PREORDERED, STREAMING, NBO
  final: no
  own_index: 0
  members(2):
        0: af83c624-4477-11ee-a657-0627ac9b29e9, db001
        1: c9e7c8ae-4476-11ee-92d0-f20d5682fc83, db002
=================================================
2023-09-13 0:55:37 2 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
2023-09-13 0:55:37 2 [Note] WSREP: Lowest cert index boundary for CC from group: 91
2023-09-13 0:55:37 2 [Note] WSREP: Min available from gcache for CC from group: 1
2023-09-13 0:55:37 0 [Note] WSREP: Member 1.0 (db002) requested state transfer from '*any*'. Selected 0.0 (db001)(SYNCED) as donor.
2023-09-13 0:55:37 0 [Note] WSREP: Shifting SYNCED -> DONOR/DESYNCED (TO: 91)
2023-09-13 0:55:37 2 [Note] WSREP: Detected STR version: 1, req_len: 131, req: STRv1
2023-09-13 0:55:37 2 [Note] WSREP: Cert index preload: 91 -> 91
2023-09-13 0:55:37 2 [Note] WSREP: Server status change synced -> donor
2023-09-13 0:55:37 2 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
2023-09-13 0:55:37 0 [Note] WSREP: async IST sender starting to serve tcp://192.168.47.42:4568 sending 91-91, preload starts from 91
2023-09-13 0:55:37 0 [Note] WSREP: IST sender 91 -> 91
2023-09-13 0:55:37 0 [Note] WSREP: Donor monitor thread started to monitor
2023-09-13 0:55:37 0 [Note] WSREP: Running: 'wsrep_sst_mariabackup --role 'donor' --address '192.168.47.42:4444/xtrabackup_sst//1' --local-port 3306 --socket '/var/lib/mysql/mysql.sock' --progress 0 --datadir '/var/lib/mysql/' --gtid '8f95c1cc-430f-11ee-8404-8f9a8663c992:91' --gtid-domain-id 0 --binlog '/var/lib/mysql/bin' --mysqld-args --wsrep-new-cluster --wsrep_start_position=8f95c1cc-430f-11ee-8404-8f9a8663c992:89,0-1-33'
2023-09-13 0:55:37 2 [Note] WSREP: sst_donor_thread signaled with 0
WSREP_SST: [INFO] mariabackup SST started on donor (20230913 00:55:37.688)
WSREP_SST: [INFO] SSL configuration: CA='', CAPATH='', CERT='', KEY='', MODE='DISABLED', encrypt='0' (20230913 00:55:37.727)
WSREP_SST: [INFO] Progress reporting tool pv not found in path: /usr//bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/usr/sbin:/usr/bin:/sbin:/bin (20230913 00:55:37.824)
WSREP_SST: [INFO] Disabling all progress/rate-limiting (20230913 00:55:37.826)
WSREP_SST: [INFO] Streaming with mbstream (20230913 00:55:37.997)
WSREP_SST: [INFO] Using socat as streamer (20230913 00:55:37.999)

*_WSREP_SST: [INFO] Using '/var/lib/mysql/tmp.23l7f5jblf' as mariadb-backup temporary directory (20230913 00:55:38.011)
WSREP_SST: [INFO] Using '/tmp/tmp.t42TMS6WSR' as mariadb-abackup working directory (20230913 00:55:38.015)_*

..
.....
2023-09-13 0:55:50 0 [Note] WSREP: Member 0.0 (db001) synced with group.
2023-09-13 0:55:50 0 [Note] WSREP: Processing event queue:...100.0% (1/1 events) complete.
2023-09-13 0:55:50 0 [Note] WSREP: Shifting JOINED -> SYNCED (TO: 91)
2023-09-13 0:55:50 2 [Note] WSREP: Server db001 synced with group
2023-09-13 0:55:50 2 [Note] WSREP: Server status change joined -> synced
2023-09-13 0:55:50 2 [Note] WSREP: Synchronized with group, ready for connections
2023-09-13 0:55:50 2 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
WSREP_SST: [INFO] Total time on donor: 0 seconds (20230913 00:55:50.414)
WSREP_SST: [INFO] mariabackup SST completed on donor (20230913 00:55:50.417)
WSREP_SST: [INFO] Cleaning up temporary directories (20230913 00:55:50.420)
2023-09-13 0:55:50 0 [Note] WSREP: Donor monitor thread ended with total time 13 sec
2023-09-13 0:55:51 0 [Note] WSREP: async IST sender served
2023-09-13 0:55:51 0 [Note] WSREP: 1.0 (db002): State transfer from 0.0 (db001) complete.
2023-09-13 0:55:51 0 [Note] WSREP: Member 1.0 (db002) synced with group.

{code}

https://github.com/MariaDB/server/blob/e987b9350cb83038c73a165922986441f345a3a9/scripts/wsrep_sst_mariabackup.sh#L1030

Donor node
{code}

[root@gcn-01 ~]# cat /etc/my.cnf|grep -i tmpdir
tmpdir = /var/lib/mysql/

{code}

Joiner node

{code}

[root@gcn-02 mysql]# cat /etc/my.cnf|grep -i tmpdir
tmpdir = /var/lib/mysql/

{code}

During SST for joiner node , *wsrep_sst_mariabackup* consider */tmp* as a working directory rather then the user defined.

My understanding during SST wsrep_sst_mariabackup should be using *tmpdir* defined under *[mysqld]* section not the* /tmp* dir.

Log from Donor while Joiner to trying to get sync using SST

{code}

2023-09-13 0:55:36 0 [Note] WSREP: (af83c624-a657, 'tcp://0.0.0.0:4567') connection established to c9e7c8ae-92d0 tcp://192.168.47.42:4567
2023-09-13 0:55:36 0 [Note] WSREP: (af83c624-a657, 'tcp://0.0.0.0:4567') turning message relay requesting on, nonlive peers:
2023-09-13 0:55:37 0 [Note] WSREP: declaring c9e7c8ae-92d0 at tcp://192.168.47.42:4567 stable
2023-09-13 0:55:37 0 [Note] WSREP: Node af83c624-a657 state prim
2023-09-13 0:55:37 0 [Note] WSREP: view(view_id(PRIM,af83c624-a657,76) memb {
        af83c624-a657,0
        c9e7c8ae-92d0,0
} joined {
} left {
} partitioned {
})
2023-09-13 0:55:37 0 [Note] WSREP: save pc into disk
2023-09-13 0:55:37 0 [Note] WSREP: New COMPONENT: primary = yes, bootstrap = no, my_idx = 0, memb_num = 2
2023-09-13 0:55:37 0 [Note] WSREP: STATE_EXCHANGE: sent state UUID: c7a87140-51f1-11ee-87cf-87f3f3a5fe55
2023-09-13 0:55:37 0 [Note] WSREP: STATE EXCHANGE: sent state msg: c7a87140-51f1-11ee-87cf-87f3f3a5fe55
2023-09-13 0:55:37 0 [Note] WSREP: STATE EXCHANGE: got state msg: c7a87140-51f1-11ee-87cf-87f3f3a5fe55 from 0 (db001)
2023-09-13 0:55:37 0 [Note] WSREP: STATE EXCHANGE: got state msg: c7a87140-51f1-11ee-87cf-87f3f3a5fe55 from 1 (db002)
2023-09-13 0:55:37 0 [Note] WSREP: Quorum results:
        version = 6,
        component = PRIMARY,
        conf_id = 1,
        members = 1/2 (joined/total),
        act_id = 90,
        last_appl. = 89,
        protocols = 2/10/4 (gcs/repl/appl),
        vote policy= 0,
        group UUID = 8f95c1cc-430f-11ee-8404-8f9a8663c992
2023-09-13 0:55:37 0 [Note] WSREP: Flow-control interval: [23, 23]
2023-09-13 0:55:37 2 [Note] WSREP: ####### processing CC 91, local, ordered
2023-09-13 0:55:37 2 [Note] WSREP: ####### My UUID: af83c624-4477-11ee-a657-0627ac9b29e9
2023-09-13 0:55:37 2 [Note] WSREP: Skipping cert index reset
2023-09-13 0:55:37 2 [Note] WSREP: REPL Protocols: 10 (5)
2023-09-13 0:55:37 2 [Note] WSREP: ####### Adjusting cert position: 90 -> 91
2023-09-13 0:55:37 0 [Note] WSREP: Service thread queue flushed.
2023-09-13 0:55:37 2 [Note] WSREP: ================================================
View:
  id: 8f95c1cc-430f-11ee-8404-8f9a8663c992:91
  status: primary
  protocol_version: 4
  capabilities: MULTI-MASTER, CERTIFICATION, PARALLEL_APPLYING, REPLAY, ISOLATION, PAUSE, CAUSAL_READ, INCREMENTAL_WS, UNORDERED, PREORDERED, STREAMING, NBO
  final: no
  own_index: 0
  members(2):
        0: af83c624-4477-11ee-a657-0627ac9b29e9, db001
        1: c9e7c8ae-4476-11ee-92d0-f20d5682fc83, db002
=================================================
2023-09-13 0:55:37 2 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
2023-09-13 0:55:37 2 [Note] WSREP: Lowest cert index boundary for CC from group: 91
2023-09-13 0:55:37 2 [Note] WSREP: Min available from gcache for CC from group: 1
2023-09-13 0:55:37 0 [Note] WSREP: Member 1.0 (db002) requested state transfer from '*any*'. Selected 0.0 (db001)(SYNCED) as donor.
2023-09-13 0:55:37 0 [Note] WSREP: Shifting SYNCED -> DONOR/DESYNCED (TO: 91)
2023-09-13 0:55:37 2 [Note] WSREP: Detected STR version: 1, req_len: 131, req: STRv1
2023-09-13 0:55:37 2 [Note] WSREP: Cert index preload: 91 -> 91
2023-09-13 0:55:37 2 [Note] WSREP: Server status change synced -> donor
2023-09-13 0:55:37 2 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
2023-09-13 0:55:37 0 [Note] WSREP: async IST sender starting to serve tcp://192.168.47.42:4568 sending 91-91, preload starts from 91
2023-09-13 0:55:37 0 [Note] WSREP: IST sender 91 -> 91
2023-09-13 0:55:37 0 [Note] WSREP: Donor monitor thread started to monitor
2023-09-13 0:55:37 0 [Note] WSREP: Running: 'wsrep_sst_mariabackup --role 'donor' --address '192.168.47.42:4444/xtrabackup_sst//1' --local-port 3306 --socket '/var/lib/mysql/mysql.sock' --progress 0 --datadir '/var/lib/mysql/' --gtid '8f95c1cc-430f-11ee-8404-8f9a8663c992:91' --gtid-domain-id 0 --binlog '/var/lib/mysql/bin' --mysqld-args --wsrep-new-cluster --wsrep_start_position=8f95c1cc-430f-11ee-8404-8f9a8663c992:89,0-1-33'
2023-09-13 0:55:37 2 [Note] WSREP: sst_donor_thread signaled with 0
WSREP_SST: [INFO] mariabackup SST started on donor (20230913 00:55:37.688)
WSREP_SST: [INFO] SSL configuration: CA='', CAPATH='', CERT='', KEY='', MODE='DISABLED', encrypt='0' (20230913 00:55:37.727)
WSREP_SST: [INFO] Progress reporting tool pv not found in path: /usr//bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/usr/sbin:/usr/bin:/sbin:/bin (20230913 00:55:37.824)
WSREP_SST: [INFO] Disabling all progress/rate-limiting (20230913 00:55:37.826)
WSREP_SST: [INFO] Streaming with mbstream (20230913 00:55:37.997)
WSREP_SST: [INFO] Using socat as streamer (20230913 00:55:37.999)

WSREP_SST: [INFO] Using '/var/lib/mysql/tmp.23l7f5jblf' as mariadb-backup temporary directory (20230913 00:55:38.011)
WSREP_SST: [INFO] Using '/tmp/tmp.t42TMS6WSR' as mariadb-abackup working directory (20230913 00:55:38.015)

..
.....
2023-09-13 0:55:50 0 [Note] WSREP: Member 0.0 (db001) synced with group.
2023-09-13 0:55:50 0 [Note] WSREP: Processing event queue:...100.0% (1/1 events) complete.
2023-09-13 0:55:50 0 [Note] WSREP: Shifting JOINED -> SYNCED (TO: 91)
2023-09-13 0:55:50 2 [Note] WSREP: Server db001 synced with group
2023-09-13 0:55:50 2 [Note] WSREP: Server status change joined -> synced
2023-09-13 0:55:50 2 [Note] WSREP: Synchronized with group, ready for connections
2023-09-13 0:55:50 2 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
WSREP_SST: [INFO] Total time on donor: 0 seconds (20230913 00:55:50.414)
WSREP_SST: [INFO] mariabackup SST completed on donor (20230913 00:55:50.417)
WSREP_SST: [INFO] Cleaning up temporary directories (20230913 00:55:50.420)
2023-09-13 0:55:50 0 [Note] WSREP: Donor monitor thread ended with total time 13 sec
2023-09-13 0:55:51 0 [Note] WSREP: async IST sender served
2023-09-13 0:55:51 0 [Note] WSREP: 1.0 (db002): State transfer from 0.0 (db001) complete.
2023-09-13 0:55:51 0 [Note] WSREP: Member 1.0 (db002) synced with group.

{code}

https://github.com/MariaDB/server/blob/e987b9350cb83038c73a165922986441f345a3a9/scripts/wsrep_sst_mariabackup.sh#L1030

Donor node
{code}

[root@gcn-01 ~]# cat /etc/my.cnf|grep -i tmpdir
tmpdir = /var/lib/mysql/

{code}

Joiner node

{code}

[root@gcn-02 mysql]# cat /etc/my.cnf|grep -i tmpdir
tmpdir = /var/lib/mysql/

{code}

Pramod Mahto made changes - 2023-09-13 05:19

Component/s		Galera SST [ 10121 ]
Component/s	Galera SST [ 14920 ]
Key	~~MENT-1943~~	~~MDEV-32158~~
Affects Version/s		10.5 [ 23123 ]
Affects Version/s		10.6 [ 24028 ]
Affects Version/s	10.5 [ 23608 ]
Affects Version/s	10.6 [ 24027 ]
Project	MariaDB Enterprise [ 11500 ]	MariaDB Server [ 10000 ]

Julien Fritsch made changes - 2023-09-13 06:29

Fix Version/s		10.5 [ 23123 ]
Fix Version/s		10.6 [ 24028 ]

Julien Fritsch made changes - 2023-09-13 06:29

Assignee

Julius Goryavsky [ sysprg ]

Julius Goryavsky made changes - 2024-06-11 12:20

Status

Open [ 1 ]

In Progress [ 3 ]

Julius Goryavsky made changes - 2024-06-15 22:11

Fix Version/s		10.5.26 [ 29832 ]
Fix Version/s	10.5 [ 23123 ]
Fix Version/s	10.6 [ 24028 ]
Resolution		Fixed [ 1 ]
Status	In Progress [ 3 ]	Closed [ 6 ]

JiraAutomate made changes - 2024-06-15 22:11

Fix Version/s		10.6.19 [ 29833 ]
Fix Version/s		10.11.9 [ 29834 ]
Fix Version/s		11.1.6 [ 29835 ]
Fix Version/s		11.2.5 [ 29836 ]
Fix Version/s		11.4.3 [ 29837 ]

Yakov Kushnirsky made changes - 2024-06-17 16:49

Link

This issue is blocked by TODO-4779 [ TODO-4779 ]

Jira Automation (IT) made changes - 2024-07-04 00:57

Zendesk Related Tickets		201660 183479
Zendesk active tickets		201660

People

Assignee:: Julius Goryavsky

Reporter:: Pramod Mahto

Votes:: 1 Vote for this issue

Watchers:: 6 Start watching this issue

Dates

Created:: 2023-09-13 05:14

Updated:: 2025-02-27 22:42

Resolved:: 2024-06-15 22:11

Git Integration

Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.

MariaDB Server

Details

Description

Attachments

Activity

People

Dates

Git Integration