[MDEV-31467] wsrep_sst_mariabackup not working on FreeBSD - Jira

Details

Type: Bug
Status: Closed (View Workflow)
Priority: Critical
Resolution: Fixed
Affects Version/s: 10.6.13
Fix Version/s: 10.4.32, 10.5.23, 10.6.16, 10.10.7, 10.11.6, 11.0.4, 11.1.3, 11.2.2, 11.3.1
Component/s: Galera SST
Labels:
- crash
- galera
Environment:
FreeBSD, all current versions

Description

With wsrep_sst_method=mariabackup the SST/IST process fails at this stage:

WSREP_SST: [INFO] mariabackup SST started on joiner (20230613 09:04:17.N)

WSREP_SST: [INFO] SSL configuration: CA='', CAPATH='', CERT='', KEY='', MODE='DISABLED', encrypt='0' (20230613 09:04:17.N)

WSREP_SST: [INFO] Streaming with mbstream (20230613 09:04:18.N)

WSREP_SST: [INFO] Using socat as streamer (20230613 09:04:18.N)

Usage: timeout [--signal sig | -s sig] [--preserve-status] [--kill-after time | -k time] [--foreground] <duration> <command> <arg ...>

WSREP_SST: [INFO] Evaluating timeout -s9 300 socat -u TCP-LISTEN:4444,reuseaddr stdio | '/usr/local//bin/mbstream' -x; RC=( ${PIPESTATUS[@]} )

(20230613 09:04:18.N)

2023-06-13  9:04:18 2 [Note] WSREP: ####### IST uuid:7f4bd4bc-73ee-11ed-8cc1-77200a87198f f: 203400, l: 203407, STRv: 3

2023-06-13  9:04:18 2 [Note] WSREP: IST receiver addr using tcp://192.168.1.1:4568

2023-06-13  9:04:18 2 [Note] WSREP: Prepared IST receiver for 203400-203407, listening at: tcp://192.168.1.1:4568

2023-06-13  9:04:18 0 [Note] WSREP: Member 1.0 (galera1) requested state transfer from '*any*'. Selected 0.0 (galera3)(SYNCED) as don

or.

2023-06-13  9:04:18 0 [Note] WSREP: Shifting PRIMARY -> JOINER (TO: 203407)

2023-06-13  9:04:18 2 [Note] WSREP: Requesting state transfer: success, donor: 0

2023-06-13  9:04:18 0 [Warning] WSREP: 0.0 (galera3): State transfer to 1.0 (galera1) failed: -2 (No such file or directory)

2023-06-13  9:04:18 0 [ERROR] WSREP: /wrkdirs/usr/ports/databases/galera26/work/galera-release_26.4.14/gcs/src/gcs_group.cpp:gcs_group_handle_j

oin_msg():1172: Will never receive state. Need to abort.

2023-06-13  9:04:18 0 [Note] WSREP: gcomm: terminating thread

2023-06-13  9:04:18 0 [Note] WSREP: gcomm: joining thread

2023-06-13  9:04:18 0 [Note] WSREP: gcomm: closing backend

2023-06-13  9:04:18 0 [Note] WSREP: view(view_id(NON_PRIM,76dfdac7-93fe,53) memb {

        82d2bc20-a998,0

} joined {

} left {

} partitioned {

        76dfdac7-93fe,0

        8f298c44-9cca,0

})

2023-06-13  9:04:18 0 [Note] WSREP: PC protocol downgrade 1 -> 0

2023-06-13  9:04:18 0 [Note] WSREP: view((empty))

2023-06-13  9:04:18 0 [Note] WSREP: gcomm: closed

2023-06-13  9:04:18 0 [Note] WSREP: mariadbd: Terminated.

The timeout command seems to expect a different call on FreeBSD systems than on Linux.

Attachments

Activity

Ascending order - Click to sort in descending order

Thomas Babut created issue - 2023-06-13 07:39

Thomas Babut made changes - 2023-06-13 07:40

Field	Original Value	New Value
Description	With {{wsrep_sst_method=mariabackup}} the SST/IST process fails in this stage: {code:java} WSREP_SST: [INFO] mariabackup SST started on joiner (20230613 09:04:17.N) WSREP_SST: [INFO] SSL configuration: CA='', CAPATH='', CERT='', KEY='', MODE='DISABLED', encrypt='0' (20230613 09:04:17.N) WSREP_SST: [INFO] Streaming with mbstream (20230613 09:04:18.N) WSREP_SST: [INFO] Using socat as streamer (20230613 09:04:18.N) Usage: timeout [--signal sig \| -s sig] [--preserve-status] [--kill-after time \| -k time] [--foreground] <duration> <command> <arg ...> WSREP_SST: [INFO] Evaluating timeout -s9 300 socat -u TCP-LISTEN:4444,reuseaddr stdio \| '/usr/local//bin/mbstream' -x; RC=( ${PIPESTATUS[@]} ) (20230613 09:04:18.N) 2023-06-13 9:04:18 2 [Note] WSREP: ####### IST uuid:7f4bd4bc-73ee-11ed-8cc1-77200a87198f f: 203400, l: 203407, STRv: 3 2023-06-13 9:04:18 2 [Note] WSREP: IST receiver addr using tcp://10.40.160.171:4568 2023-06-13 9:04:18 2 [Note] WSREP: Prepared IST receiver for 203400-203407, listening at: tcp://10.40.160.171:4568 2023-06-13 9:04:18 0 [Note] WSREP: Member 1.0 (vega-galera1) requested state transfer from 'any'. Selected 0.0 (vega-galera3)(SYNCED) as don or. 2023-06-13 9:04:18 0 [Note] WSREP: Shifting PRIMARY -> JOINER (TO: 203407) 2023-06-13 9:04:18 2 [Note] WSREP: Requesting state transfer: success, donor: 0 2023-06-13 9:04:18 0 [Warning] WSREP: 0.0 (vega-galera3): State transfer to 1.0 (vega-galera1) failed: -2 (No such file or directory) 2023-06-13 9:04:18 0 [ERROR] WSREP: /wrkdirs/usr/ports/databases/galera26/work/galera-release_26.4.14/gcs/src/gcs_group.cpp:gcs_group_handle_j oin_msg():1172: Will never receive state. Need to abort. 2023-06-13 9:04:18 0 [Note] WSREP: gcomm: terminating thread 2023-06-13 9:04:18 0 [Note] WSREP: gcomm: joining thread 2023-06-13 9:04:18 0 [Note] WSREP: gcomm: closing backend 2023-06-13 9:04:18 0 [Note] WSREP: view(view_id(NON_PRIM,76dfdac7-93fe,53) memb { 82d2bc20-a998,0 } joined { } left { } partitioned { 76dfdac7-93fe,0 8f298c44-9cca,0 }) 2023-06-13 9:04:18 0 [Note] WSREP: PC protocol downgrade 1 -> 0 2023-06-13 9:04:18 0 [Note] WSREP: view((empty)) 2023-06-13 9:04:18 0 [Note] WSREP: gcomm: closed 2023-06-13 9:04:18 0 [Note] WSREP: mariadbd: Terminated. {code} The {{timeout}} command seems to expect a different call on FreeBSD systems than on Linux.	With {{wsrep_sst_method=mariabackup}} the SST/IST process fails in this stage: {code:java} WSREP_SST: [INFO] mariabackup SST started on joiner (20230613 09:04:17.N) WSREP_SST: [INFO] SSL configuration: CA='', CAPATH='', CERT='', KEY='', MODE='DISABLED', encrypt='0' (20230613 09:04:17.N) WSREP_SST: [INFO] Streaming with mbstream (20230613 09:04:18.N) WSREP_SST: [INFO] Using socat as streamer (20230613 09:04:18.N) Usage: timeout [--signal sig \| -s sig] [--preserve-status] [--kill-after time \| -k time] [--foreground] <duration> <command> <arg ...> WSREP_SST: [INFO] Evaluating timeout -s9 300 socat -u TCP-LISTEN:4444,reuseaddr stdio \| '/usr/local//bin/mbstream' -x; RC=( ${PIPESTATUS[@]} ) (20230613 09:04:18.N) 2023-06-13 9:04:18 2 [Note] WSREP: ####### IST uuid:7f4bd4bc-73ee-11ed-8cc1-77200a87198f f: 203400, l: 203407, STRv: 3 2023-06-13 9:04:18 2 [Note] WSREP: IST receiver addr using tcp://192.168.1.1:4568 2023-06-13 9:04:18 2 [Note] WSREP: Prepared IST receiver for 203400-203407, listening at: tcp://192.168.1.1:4568 2023-06-13 9:04:18 0 [Note] WSREP: Member 1.0 (galera1) requested state transfer from 'any'. Selected 0.0 (galera3)(SYNCED) as don or. 2023-06-13 9:04:18 0 [Note] WSREP: Shifting PRIMARY -> JOINER (TO: 203407) 2023-06-13 9:04:18 2 [Note] WSREP: Requesting state transfer: success, donor: 0 2023-06-13 9:04:18 0 [Warning] WSREP: 0.0 (galera3): State transfer to 1.0 (galera1) failed: -2 (No such file or directory) 2023-06-13 9:04:18 0 [ERROR] WSREP: /wrkdirs/usr/ports/databases/galera26/work/galera-release_26.4.14/gcs/src/gcs_group.cpp:gcs_group_handle_j oin_msg():1172: Will never receive state. Need to abort. 2023-06-13 9:04:18 0 [Note] WSREP: gcomm: terminating thread 2023-06-13 9:04:18 0 [Note] WSREP: gcomm: joining thread 2023-06-13 9:04:18 0 [Note] WSREP: gcomm: closing backend 2023-06-13 9:04:18 0 [Note] WSREP: view(view_id(NON_PRIM,76dfdac7-93fe,53) memb { 82d2bc20-a998,0 } joined { } left { } partitioned { 76dfdac7-93fe,0 8f298c44-9cca,0 }) 2023-06-13 9:04:18 0 [Note] WSREP: PC protocol downgrade 1 -> 0 2023-06-13 9:04:18 0 [Note] WSREP: view((empty)) 2023-06-13 9:04:18 0 [Note] WSREP: gcomm: closed 2023-06-13 9:04:18 0 [Note] WSREP: mariadbd: Terminated. {code} The {{timeout}} command seems to expect a different call on FreeBSD systems than on Linux.

Thomas Babut made changes - 2023-06-13 07:41

Description

With {{wsrep_sst_method=mariabackup}} the SST/IST process fails in this stage:

{code:java}
WSREP_SST: [INFO] mariabackup SST started on joiner (20230613 09:04:17.N)
WSREP_SST: [INFO] SSL configuration: CA='', CAPATH='', CERT='', KEY='', MODE='DISABLED', encrypt='0' (20230613 09:04:17.N)
WSREP_SST: [INFO] Streaming with mbstream (20230613 09:04:18.N)
WSREP_SST: [INFO] Using socat as streamer (20230613 09:04:18.N)
*Usage: timeout [--signal sig | -s sig] [--preserve-status] [--kill-after time | -k time] [--foreground] <duration> <command> <arg ...>*
WSREP_SST: [INFO] Evaluating timeout -s9 300 socat -u TCP-LISTEN:4444,reuseaddr stdio | '/usr/local//bin/mbstream' -x; RC=( ${PIPESTATUS[@]} )
(20230613 09:04:18.N)
2023-06-13 9:04:18 2 [Note] WSREP: ####### IST uuid:7f4bd4bc-73ee-11ed-8cc1-77200a87198f f: 203400, l: 203407, STRv: 3
2023-06-13 9:04:18 2 [Note] WSREP: IST receiver addr using tcp://192.168.1.1:4568
2023-06-13 9:04:18 2 [Note] WSREP: Prepared IST receiver for 203400-203407, listening at: tcp://192.168.1.1:4568
2023-06-13 9:04:18 0 [Note] WSREP: Member 1.0 (galera1) requested state transfer from '*any*'. Selected 0.0 (galera3)(SYNCED) as don
or.
2023-06-13 9:04:18 0 [Note] WSREP: Shifting PRIMARY -> JOINER (TO: 203407)
2023-06-13 9:04:18 2 [Note] WSREP: Requesting state transfer: success, donor: 0
2023-06-13 9:04:18 0 [Warning] WSREP: 0.0 (galera3): State transfer to 1.0 (galera1) failed: -2 (No such file or directory)
2023-06-13 9:04:18 0 [ERROR] WSREP: /wrkdirs/usr/ports/databases/galera26/work/galera-release_26.4.14/gcs/src/gcs_group.cpp:gcs_group_handle_j
oin_msg():1172: Will never receive state. Need to abort.
2023-06-13 9:04:18 0 [Note] WSREP: gcomm: terminating thread
2023-06-13 9:04:18 0 [Note] WSREP: gcomm: joining thread
2023-06-13 9:04:18 0 [Note] WSREP: gcomm: closing backend
2023-06-13 9:04:18 0 [Note] WSREP: view(view_id(NON_PRIM,76dfdac7-93fe,53) memb {
        82d2bc20-a998,0
} joined {
} left {
} partitioned {
        76dfdac7-93fe,0
        8f298c44-9cca,0
})
2023-06-13 9:04:18 0 [Note] WSREP: PC protocol downgrade 1 -> 0
2023-06-13 9:04:18 0 [Note] WSREP: view((empty))
2023-06-13 9:04:18 0 [Note] WSREP: gcomm: closed
2023-06-13 9:04:18 0 [Note] WSREP: mariadbd: Terminated.
{code}
The {{timeout}} command seems to expect a different call on FreeBSD systems than on Linux.

With {{wsrep_sst_method=mariabackup}} the SST/IST process fails in this stage:

{code:java}
WSREP_SST: [INFO] mariabackup SST started on joiner (20230613 09:04:17.N)
WSREP_SST: [INFO] SSL configuration: CA='', CAPATH='', CERT='', KEY='', MODE='DISABLED', encrypt='0' (20230613 09:04:17.N)
WSREP_SST: [INFO] Streaming with mbstream (20230613 09:04:18.N)
WSREP_SST: [INFO] Using socat as streamer (20230613 09:04:18.N)

Usage: timeout [--signal sig | -s sig] [--preserve-status] [--kill-after time | -k time] [--foreground] <duration> <command> <arg ...>
WSREP_SST: [INFO] Evaluating timeout -s9 300 socat -u TCP-LISTEN:4444,reuseaddr stdio | '/usr/local//bin/mbstream' -x; RC=( ${PIPESTATUS[@]} )
(20230613 09:04:18.N)

2023-06-13 9:04:18 2 [Note] WSREP: ####### IST uuid:7f4bd4bc-73ee-11ed-8cc1-77200a87198f f: 203400, l: 203407, STRv: 3
2023-06-13 9:04:18 2 [Note] WSREP: IST receiver addr using tcp://192.168.1.1:4568
2023-06-13 9:04:18 2 [Note] WSREP: Prepared IST receiver for 203400-203407, listening at: tcp://192.168.1.1:4568
2023-06-13 9:04:18 0 [Note] WSREP: Member 1.0 (galera1) requested state transfer from '*any*'. Selected 0.0 (galera3)(SYNCED) as don
or.
2023-06-13 9:04:18 0 [Note] WSREP: Shifting PRIMARY -> JOINER (TO: 203407)
2023-06-13 9:04:18 2 [Note] WSREP: Requesting state transfer: success, donor: 0
2023-06-13 9:04:18 0 [Warning] WSREP: 0.0 (galera3): State transfer to 1.0 (galera1) failed: -2 (No such file or directory)
2023-06-13 9:04:18 0 [ERROR] WSREP: /wrkdirs/usr/ports/databases/galera26/work/galera-release_26.4.14/gcs/src/gcs_group.cpp:gcs_group_handle_j
oin_msg():1172: Will never receive state. Need to abort.
2023-06-13 9:04:18 0 [Note] WSREP: gcomm: terminating thread
2023-06-13 9:04:18 0 [Note] WSREP: gcomm: joining thread
2023-06-13 9:04:18 0 [Note] WSREP: gcomm: closing backend
2023-06-13 9:04:18 0 [Note] WSREP: view(view_id(NON_PRIM,76dfdac7-93fe,53) memb {
        82d2bc20-a998,0
} joined {
} left {
} partitioned {
        76dfdac7-93fe,0
        8f298c44-9cca,0
})
2023-06-13 9:04:18 0 [Note] WSREP: PC protocol downgrade 1 -> 0
2023-06-13 9:04:18 0 [Note] WSREP: view((empty))
2023-06-13 9:04:18 0 [Note] WSREP: gcomm: closed
2023-06-13 9:04:18 0 [Note] WSREP: mariadbd: Terminated.
{code}
The {{timeout}} command seems to expect a different call on FreeBSD systems than on Linux.

Thomas Babut made changes - 2023-06-13 08:39

Description

With {{wsrep_sst_method=mariabackup}} the SST/IST process fails in this stage:

{code:java}
WSREP_SST: [INFO] mariabackup SST started on joiner (20230613 09:04:17.N)
WSREP_SST: [INFO] SSL configuration: CA='', CAPATH='', CERT='', KEY='', MODE='DISABLED', encrypt='0' (20230613 09:04:17.N)
WSREP_SST: [INFO] Streaming with mbstream (20230613 09:04:18.N)
WSREP_SST: [INFO] Using socat as streamer (20230613 09:04:18.N)

Usage: timeout [--signal sig | -s sig] [--preserve-status] [--kill-after time | -k time] [--foreground] <duration> <command> <arg ...>
WSREP_SST: [INFO] Evaluating timeout -s9 300 socat -u TCP-LISTEN:4444,reuseaddr stdio | '/usr/local//bin/mbstream' -x; RC=( ${PIPESTATUS[@]} )
(20230613 09:04:18.N)

2023-06-13 9:04:18 2 [Note] WSREP: ####### IST uuid:7f4bd4bc-73ee-11ed-8cc1-77200a87198f f: 203400, l: 203407, STRv: 3
2023-06-13 9:04:18 2 [Note] WSREP: IST receiver addr using tcp://192.168.1.1:4568
2023-06-13 9:04:18 2 [Note] WSREP: Prepared IST receiver for 203400-203407, listening at: tcp://192.168.1.1:4568
2023-06-13 9:04:18 0 [Note] WSREP: Member 1.0 (galera1) requested state transfer from '*any*'. Selected 0.0 (galera3)(SYNCED) as don
or.
2023-06-13 9:04:18 0 [Note] WSREP: Shifting PRIMARY -> JOINER (TO: 203407)
2023-06-13 9:04:18 2 [Note] WSREP: Requesting state transfer: success, donor: 0
2023-06-13 9:04:18 0 [Warning] WSREP: 0.0 (galera3): State transfer to 1.0 (galera1) failed: -2 (No such file or directory)
2023-06-13 9:04:18 0 [ERROR] WSREP: /wrkdirs/usr/ports/databases/galera26/work/galera-release_26.4.14/gcs/src/gcs_group.cpp:gcs_group_handle_j
oin_msg():1172: Will never receive state. Need to abort.
2023-06-13 9:04:18 0 [Note] WSREP: gcomm: terminating thread
2023-06-13 9:04:18 0 [Note] WSREP: gcomm: joining thread
2023-06-13 9:04:18 0 [Note] WSREP: gcomm: closing backend
2023-06-13 9:04:18 0 [Note] WSREP: view(view_id(NON_PRIM,76dfdac7-93fe,53) memb {
        82d2bc20-a998,0
} joined {
} left {
} partitioned {
        76dfdac7-93fe,0
        8f298c44-9cca,0
})
2023-06-13 9:04:18 0 [Note] WSREP: PC protocol downgrade 1 -> 0
2023-06-13 9:04:18 0 [Note] WSREP: view((empty))
2023-06-13 9:04:18 0 [Note] WSREP: gcomm: closed
2023-06-13 9:04:18 0 [Note] WSREP: mariadbd: Terminated.
{code}
The {{timeout}} command seems to expect a different call on FreeBSD systems than on Linux.

With {{wsrep_sst_method=mariabackup}} the SST/IST process fails at this stage:

{code:java}
WSREP_SST: [INFO] mariabackup SST started on joiner (20230613 09:04:17.N)
WSREP_SST: [INFO] SSL configuration: CA='', CAPATH='', CERT='', KEY='', MODE='DISABLED', encrypt='0' (20230613 09:04:17.N)
WSREP_SST: [INFO] Streaming with mbstream (20230613 09:04:18.N)
WSREP_SST: [INFO] Using socat as streamer (20230613 09:04:18.N)

Usage: timeout [--signal sig | -s sig] [--preserve-status] [--kill-after time | -k time] [--foreground] <duration> <command> <arg ...>
WSREP_SST: [INFO] Evaluating timeout -s9 300 socat -u TCP-LISTEN:4444,reuseaddr stdio | '/usr/local//bin/mbstream' -x; RC=( ${PIPESTATUS[@]} )
(20230613 09:04:18.N)

2023-06-13 9:04:18 2 [Note] WSREP: ####### IST uuid:7f4bd4bc-73ee-11ed-8cc1-77200a87198f f: 203400, l: 203407, STRv: 3
2023-06-13 9:04:18 2 [Note] WSREP: IST receiver addr using tcp://192.168.1.1:4568
2023-06-13 9:04:18 2 [Note] WSREP: Prepared IST receiver for 203400-203407, listening at: tcp://192.168.1.1:4568
2023-06-13 9:04:18 0 [Note] WSREP: Member 1.0 (galera1) requested state transfer from '*any*'. Selected 0.0 (galera3)(SYNCED) as don
or.
2023-06-13 9:04:18 0 [Note] WSREP: Shifting PRIMARY -> JOINER (TO: 203407)
2023-06-13 9:04:18 2 [Note] WSREP: Requesting state transfer: success, donor: 0
2023-06-13 9:04:18 0 [Warning] WSREP: 0.0 (galera3): State transfer to 1.0 (galera1) failed: -2 (No such file or directory)
2023-06-13 9:04:18 0 [ERROR] WSREP: /wrkdirs/usr/ports/databases/galera26/work/galera-release_26.4.14/gcs/src/gcs_group.cpp:gcs_group_handle_j
oin_msg():1172: Will never receive state. Need to abort.
2023-06-13 9:04:18 0 [Note] WSREP: gcomm: terminating thread
2023-06-13 9:04:18 0 [Note] WSREP: gcomm: joining thread
2023-06-13 9:04:18 0 [Note] WSREP: gcomm: closing backend
2023-06-13 9:04:18 0 [Note] WSREP: view(view_id(NON_PRIM,76dfdac7-93fe,53) memb {
        82d2bc20-a998,0
} joined {
} left {
} partitioned {
        76dfdac7-93fe,0
        8f298c44-9cca,0
})
2023-06-13 9:04:18 0 [Note] WSREP: PC protocol downgrade 1 -> 0
2023-06-13 9:04:18 0 [Note] WSREP: view((empty))
2023-06-13 9:04:18 0 [Note] WSREP: gcomm: closed
2023-06-13 9:04:18 0 [Note] WSREP: mariadbd: Terminated.
{code}
The {{timeout}} command seems to expect a different call on FreeBSD systems than on Linux.

Daniel Black made changes - 2023-07-06 04:51

Fix Version/s

10.6 [ 24028 ]

Daniel Black made changes - 2023-07-06 04:51

Assignee

Julius Goryavsky [ sysprg ]

Julius Goryavsky made changes - 2023-10-17 11:50

Status

Open [ 1 ]

In Progress [ 3 ]

Julius Goryavsky made changes - 2023-10-17 11:52

Fix Version/s		10.4.32 [ 29300 ]
Fix Version/s		10.5.23 [ 29012 ]
Fix Version/s		10.6.16 [ 29014 ]
Fix Version/s		10.10.7 [ 29018 ]
Fix Version/s		10.11.6 [ 29020 ]
Fix Version/s		11.0.4 [ 29021 ]
Fix Version/s		11.1.3 [ 29023 ]
Fix Version/s		11.2.2 [ 29035 ]
Fix Version/s		11.3.1 [ 29416 ]
Fix Version/s	10.6 [ 24028 ]
Resolution		Fixed [ 1 ]
Status	In Progress [ 3 ]	Closed [ 6 ]

People

Assignee:: Julius Goryavsky

Reporter:: Thomas Babut

Votes:: 1 Vote for this issue

Watchers:: 4 Start watching this issue

Dates

Created:: 2023-06-13 07:39

Updated:: 2023-12-07 18:30

Resolved:: 2023-10-17 11:52

Git Integration

Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.

MariaDB Server

Details

Description

Attachments

Activity

People

Dates

Git Integration