[MDEV-27459] SST works as expected on joiner node but donor node never leaves donor state. - Jira

Details

Type: Bug
Status: Closed (View Workflow)
Priority: Critical
Resolution: Fixed
Affects Version/s: 10.4.21, 10.5.12
Fix Version/s: 10.4.23, 10.5.14, 10.6.6, 10.7.2
Component/s: Galera, Galera SST
Labels:
None
Environment:
Ubuntu Linux

Description

Hi Team,

Galera donor node left in "donor/desynced" after SST with mariabackup. Here I have attached the logs. Please check it.

After SST the donor node is left in:

MariaDB [(none)]> show status like 'wsrep_local_state%';

+---------------------------+--------------------------------------+

| Variable_name | Value |

+---------------------------+--------------------------------------+

| wsrep_local_state_uuid | abacdfcc-5c70-11ea-b34c-c2bcad908195 |

| wsrep_local_state | 2 |

| wsrep_local_state_comment | Donor/Desynced |

+---------------------------+--------------------------------------+

3 rows in set (0.001 sec)

Attachments

- Sort By Name
- Sort By Date
- Ascending
- Descending
- Thumbnails
- List
- Download All

db-prod02.donor.txt
20 kB
2022-01-10 13:36
db-prod04.joiner.txt
87 kB
2022-01-10 13:36
node1.err
19 kB
2022-01-13 06:33
node2.err
38 kB
2022-01-13 06:33

Issue Links

blocks

MDEV-27789 mysql_upgrade / mariadb-upgrade in 10.6.6 is putting password in host argument

Closed

relates to

MDEV-26918 mariabackup SST donor stuck in DONOR/DESYNCED state

Closed

MDEV-26969 mariabackup sst donor stuck in Donor/Desynced even after the SST succeeds

Closed

Activity

Ascending order - Click to sort in descending order

Pon Suresh Pandian (Inactive) created issue - 2022-01-10 13:39

Ralf Becker added a comment - 2022-01-11 13:24

Happens for me (5 node geo-distributed Galera cluster running 10.5.13) too in roughly 50% of my SSTs.
Killing/restarting the stuck donor brings it back to nor with an IST.

Ralf

Ralf Becker added a comment - 2022-01-11 13:24 Happens for me (5 node geo-distributed Galera cluster running 10.5.13) too in roughly 50% of my SSTs. Killing/restarting the stuck donor brings it back to nor with an IST. Ralf

Jean-Louis Dupond added a comment - 2022-01-11 14:56

Same happens here on 10.5.12.
Also using mariabackup.

Could this get some priority? As the only fix is restarting which is annoying.

Jean-Louis Dupond added a comment - 2022-01-11 14:56 Same happens here on 10.5.12. Also using mariabackup. Could this get some priority? As the only fix is restarting which is annoying.

Julien Fritsch made changes - 2022-01-11 16:59

Field	Original Value	New Value
Assignee		Ramesh Sivaraman [ JIRAUSER48189 ]

Ramesh Sivaraman made changes - 2022-01-13 06:33

Attachment

node1.err [ 61668 ]

Ramesh Sivaraman made changes - 2022-01-13 06:33

Attachment

node2.err [ 61669 ]

Ramesh Sivaraman made changes - 2022-01-13 06:43

Assignee

Ramesh Sivaraman [ JIRAUSER48189 ]

Jan Lindström [ jplindst ]

Ramesh Sivaraman made changes - 2022-01-13 06:46

Assignee

Jan Lindström [ jplindst ]

Seppo Jaakola [ seppo ]

Ramesh Sivaraman made changes - 2022-01-13 06:46

Priority

Major [ 3 ]

Critical [ 2 ]

Ramesh Sivaraman made changes - 2022-01-13 06:47

Link

This issue relates to ~~MDEV-26969~~ [ ~~MDEV-26969~~ ]

Ramesh Sivaraman made changes - 2022-01-13 06:47

Link

This issue relates to ~~MDEV-26918~~ [ ~~MDEV-26918~~ ]

Ramesh Sivaraman made changes - 2022-01-13 06:49

Fix Version/s

10.4 [ 22408 ]

Ramesh Sivaraman made changes - 2022-01-13 06:49

Status

Open [ 1 ]

Confirmed [ 10101 ]

Julien Fritsch made changes - 2022-01-14 06:49

Status

Confirmed [ 10101 ]

Open [ 1 ]

Julien Fritsch made changes - 2022-01-14 06:50

Status

Open [ 1 ]

Needs Feedback [ 10501 ]

Jan Lindström (Inactive) added a comment - 2022-01-14 10:17

julien.fritsch Based on Seppo's tests this does not look like regression so not a blocker for next release.

Jan Lindström (Inactive) added a comment - 2022-01-14 10:17 julien.fritsch Based on Seppo's tests this does not look like regression so not a blocker for next release.

Julien Fritsch made changes - 2022-01-14 13:37

Status

Needs Feedback [ 10501 ]

Open [ 1 ]

Julien Fritsch made changes - 2022-01-14 13:38

Status

Open [ 1 ]

Confirmed [ 10101 ]

Seppo Jaakola made changes - 2022-01-17 09:13

Status

Confirmed [ 10101 ]

In Progress [ 3 ]

Seppo Jaakola made changes - 2022-01-17 09:16

Status

In Progress [ 3 ]

Stalled [ 10000 ]

Seppo Jaakola added a comment - 2022-01-17 09:28

This was analyzed to be a problem in galera side. A fix has been prepared to both galera 3 and 4 versions, and I cannot reproduce the issue with this test scenario anymore.

Seppo Jaakola added a comment - 2022-01-17 09:28 This was analyzed to be a problem in galera side. A fix has been prepared to both galera 3 and 4 versions, and I cannot reproduce the issue with this test scenario anymore.

Seppo Jaakola added a comment - 2022-01-17 09:33

fixes are merged in galera-bugs 3.x, 4.x and 4.ee HEAD, please confirm if customer's use case is fixed by this

Seppo Jaakola added a comment - 2022-01-17 09:33 fixes are merged in galera-bugs 3.x, 4.x and 4.ee HEAD, please confirm if customer's use case is fixed by this

Seppo Jaakola made changes - 2022-01-17 09:33

Assignee	Seppo Jaakola [ seppo ]	Jan Lindström [ jplindst ]
Status	Stalled [ 10000 ]	In Review [ 10002 ]

Jan Lindström (Inactive) made changes - 2022-01-20 06:21

Status

In Review [ 10002 ]

In Testing [ 10301 ]

Jan Lindström (Inactive) added a comment - 2022-01-20 06:22

Should be fixed with Galera library 26.4.11 with commit 67341d07

Jan Lindström (Inactive) added a comment - 2022-01-20 06:22 Should be fixed with Galera library 26.4.11 with commit 67341d07

Jan Lindström (Inactive) made changes - 2022-01-20 06:22

Assignee

Jan Lindström [ jplindst ]

Ramesh Sivaraman [ JIRAUSER48189 ]

Ramesh Sivaraman added a comment - 2022-01-20 08:38

jplindst bug fix looks good. Donor wsrep state changes correctly in the given test case and in normal SST.

Ramesh Sivaraman added a comment - 2022-01-20 08:38 jplindst bug fix looks good. Donor wsrep state changes correctly in the given test case and in normal SST.

Ramesh Sivaraman made changes - 2022-01-20 08:38

Assignee

Ramesh Sivaraman [ JIRAUSER48189 ]

Jan Lindström [ jplindst ]

Ramesh Sivaraman made changes - 2022-01-20 08:39

Status

In Testing [ 10301 ]

Stalled [ 10000 ]

Jan Lindström (Inactive) added a comment - 2022-01-20 09:50

ponsuresh.pandians Can customer test with 26.4.11 Galera library ?

Jan Lindström (Inactive) added a comment - 2022-01-20 09:50 ponsuresh.pandians Can customer test with 26.4.11 Galera library ?

Jan Lindström (Inactive) made changes - 2022-01-20 09:51

Labels

need_feedback

Julien Fritsch made changes - 2022-01-20 10:19

Status

Stalled [ 10000 ]

Open [ 1 ]

Julien Fritsch made changes - 2022-01-20 10:19

Status

Open [ 1 ]

Needs Feedback [ 10501 ]

Julien Fritsch made changes - 2022-01-20 10:19

Labels

need_feedback

Lars Mikkelsen added a comment - 2022-01-20 10:58

Please provide download link for galera-26.4.11 so I can test this

Newest version I can find on archive.mariadb.org is 26.4.6 as it is packaged even in the newer tar files.

➜  Downloads mdmd5 galera-26.4.9-systemd-x86_64/usr/lib/galera/libgalera_smm.so mariadb-10.4.21-linux-systemd-x86_64/lib/galera/libgalera_smm.so galera-26.4.6-systemd-x86_64/usr/lib/galera/libgalera_smm.so

MD5 (galera-26.4.9-systemd-x86_64/usr/lib/galera/libgalera_smm.so) = a04f49ca276e50bb0a48e7decfeebcff

MD5 (mariadb-10.4.21-linux-systemd-x86_64/lib/galera/libgalera_smm.so) = a04f49ca276e50bb0a48e7decfeebcff

MD5 (galera-26.4.6-systemd-x86_64/usr/lib/galera/libgalera_smm.so) = a04f49ca276e50bb0a48e7decfeebcff

Some of your build rutines may need a checkup.

Lars Mikkelsen added a comment - 2022-01-20 10:58 Please provide download link for galera-26.4.11 so I can test this Newest version I can find on archive.mariadb.org is 26.4.6 as it is packaged even in the newer tar files. ➜ Downloads mdmd5 galera-26.4.9-systemd-x86_64/usr/lib/galera/libgalera_smm.so mariadb-10.4.21-linux-systemd-x86_64/lib/galera/libgalera_smm.so galera-26.4.6-systemd-x86_64/usr/lib/galera/libgalera_smm.so MD5 (galera-26.4.9-systemd-x86_64/usr/lib/galera/libgalera_smm.so) = a04f49ca276e50bb0a48e7decfeebcff MD5 (mariadb-10.4.21-linux-systemd-x86_64/lib/galera/libgalera_smm.so) = a04f49ca276e50bb0a48e7decfeebcff MD5 (galera-26.4.6-systemd-x86_64/usr/lib/galera/libgalera_smm.so) = a04f49ca276e50bb0a48e7decfeebcff Some of your build rutines may need a checkup.

Lars Mikkelsen added a comment - 2022-01-20 11:17

And in status on a 10.4.21 server its also the reported version:

MariaDB [(none)]> show status like 'wsrep_provider_ver%';

+------------------------+------------------+

| Variable_name          | Value            |

+------------------------+------------------+

| wsrep_provider_version | 26.4.6(r1d8d67c) |

+------------------------+------------------+

1 row in set (0.000 sec)

Lars Mikkelsen added a comment - 2022-01-20 11:17 And in status on a 10.4.21 server its also the reported version: MariaDB [(none)]> show status like 'wsrep_provider_ver%'; +------------------------+------------------+ | Variable_name | Value | +------------------------+------------------+ | wsrep_provider_version | 26.4.6(r1d8d67c) | +------------------------+------------------+ 1 row in set (0.000 sec)

Valerii Kravchuk made changes - 2022-01-24 10:25

Affects Version/s

10.5.12 [ 26025 ]

Julien Fritsch made changes - 2022-01-25 17:03

Status

Needs Feedback [ 10501 ]

Open [ 1 ]

Jan Lindström (Inactive) added a comment - 2022-01-26 06:32

ponsuresh.pandians You can compile newest Galera library from sources if you need it now. If not we will release MariaDB Community Server soon with new Galera library. This link https://dlm.mariadb.com/browse/mariadb_server/76/1194/ seems to contain 26.4.9. I do not know why your link shows so old library.

Jan Lindström (Inactive) added a comment - 2022-01-26 06:32 ponsuresh.pandians You can compile newest Galera library from sources if you need it now. If not we will release MariaDB Community Server soon with new Galera library. This link https://dlm.mariadb.com/browse/mariadb_server/76/1194/ seems to contain 26.4.9. I do not know why your link shows so old library.

Lars Mikkelsen added a comment - 2022-01-26 06:48

It actually dont. Package names are correct and says 26.4.9 but once you unpack you will see the version is only 26.4.6 and from october 2020.

This is all ready reported to and confirmed by your support.

Lars Mikkelsen added a comment - 2022-01-26 06:48 It actually dont. Package names are correct and says 26.4.9 but once you unpack you will see the version is only 26.4.6 and from october 2020. This is all ready reported to and confirmed by your support.

Jan Lindström (Inactive) added a comment - 2022-01-26 07:02

lmk@netic.dk Thank you for your report. I opened https://jira.mariadb.org/browse/TODO-3320 to fix this.

Jan Lindström (Inactive) added a comment - 2022-01-26 07:02 lmk@netic.dk Thank you for your report. I opened https://jira.mariadb.org/browse/TODO-3320 to fix this.

Jan Lindström (Inactive) added a comment - 2022-01-26 07:09

Please try following: https://archive.mariadb.org/mariadb-10.4.22/galera-26.4.9/

Jan Lindström (Inactive) added a comment - 2022-01-26 07:09 Please try following: https://archive.mariadb.org/mariadb-10.4.22/galera-26.4.9/

Lars Mikkelsen added a comment - 2022-01-26 07:19

Thats also an old version from october 2020

➜  galera wget https://archive.mariadb.org/mariadb-10.4.22/galera-26.4.9/bintar/galera-26.4.9-systemd-x86_64.tar.gz

--2022-01-26 08:16:35--  https://archive.mariadb.org/mariadb-10.4.22/galera-26.4.9/bintar/galera-26.4.9-systemd-x86_64.tar.gz

Resolving archive.mariadb.org (archive.mariadb.org)... 2a01:4f8:c17:cad6::1, 138.201.152.105

Connecting to archive.mariadb.org (archive.mariadb.org)|2a01:4f8:c17:cad6::1|:443... connected.

HTTP request sent, awaiting response... 200 OK

Length: 20394498 (19M) [application/octet-stream]

Saving to: ‘galera-26.4.9-systemd-x86_64.tar.gz’

galera-26.4.9-systemd-x86_64.tar.gz                        100%[=======================================================================================================================================>]  19,45M  4,71MB/s    in 5,5s

2022-01-26 08:16:41 (3,52 MB/s) - ‘galera-26.4.9-systemd-x86_64.tar.gz’ saved [20394498/20394498]

➜  galera tar zxvf galera-26.4.9-systemd-x86_64.tar.gz

x galera-26.4.9-systemd-x86_64/

x galera-26.4.9-systemd-x86_64/usr/

x galera-26.4.9-systemd-x86_64/usr/lib/

x galera-26.4.9-systemd-x86_64/usr/lib/galera/

x galera-26.4.9-systemd-x86_64/usr/lib/galera/libgalera_smm.so

x galera-26.4.9-systemd-x86_64/usr/lib/libgalera_smm.so

...

x galera-26.4.9-systemd-x86_64/etc/

x galera-26.4.9-systemd-x86_64/etc/default/

x galera-26.4.9-systemd-x86_64/etc/default/garb

x galera-26.4.9-systemd-x86_64/etc/init.d/

x galera-26.4.9-systemd-x86_64/etc/init.d/garb

➜  galera ls -l galera-26.4.9-systemd-x86_64/usr/lib/galera/libgalera_smm.so

-rw-r--r--  1 lmk  staff  40221093 Oct 22  2020 galera-26.4.9-systemd-x86_64/usr/lib/galera/libgalera_smm.so

Lars Mikkelsen added a comment - 2022-01-26 07:19 Thats also an old version from october 2020 ➜ galera wget https://archive.mariadb.org/mariadb-10.4.22/galera-26.4.9/bintar/galera-26.4.9-systemd-x86_64.tar.gz --2022-01-26 08:16:35-- https://archive.mariadb.org/mariadb-10.4.22/galera-26.4.9/bintar/galera-26.4.9-systemd-x86_64.tar.gz Resolving archive.mariadb.org (archive.mariadb.org)... 2a01:4f8:c17:cad6::1, 138.201.152.105 Connecting to archive.mariadb.org (archive.mariadb.org)|2a01:4f8:c17:cad6::1|:443... connected. HTTP request sent, awaiting response... 200 OK Length: 20394498 (19M) [application/octet-stream] Saving to: ‘galera-26.4.9-systemd-x86_64.tar.gz’ galera-26.4.9-systemd-x86_64.tar.gz 100%[=======================================================================================================================================>] 19,45M 4,71MB/s in 5,5s 2022-01-26 08:16:41 (3,52 MB/s) - ‘galera-26.4.9-systemd-x86_64.tar.gz’ saved [20394498/20394498] ➜ galera tar zxvf galera-26.4.9-systemd-x86_64.tar.gz x galera-26.4.9-systemd-x86_64/ x galera-26.4.9-systemd-x86_64/usr/ x galera-26.4.9-systemd-x86_64/usr/lib/ x galera-26.4.9-systemd-x86_64/usr/lib/galera/ x galera-26.4.9-systemd-x86_64/usr/lib/galera/libgalera_smm.so x galera-26.4.9-systemd-x86_64/usr/lib/libgalera_smm.so ... x galera-26.4.9-systemd-x86_64/etc/ x galera-26.4.9-systemd-x86_64/etc/default/ x galera-26.4.9-systemd-x86_64/etc/default/garb x galera-26.4.9-systemd-x86_64/etc/init.d/ x galera-26.4.9-systemd-x86_64/etc/init.d/garb ➜ galera ls -l galera-26.4.9-systemd-x86_64/usr/lib/galera/libgalera_smm.so -rw-r--r-- 1 lmk staff 40221093 Oct 22 2020 galera-26.4.9-systemd-x86_64/usr/lib/galera/libgalera_smm.so

Jan Lindström (Inactive) added a comment - 2022-01-26 07:29

Debian seems to be fine but bintar not.

Jan Lindström (Inactive) added a comment - 2022-01-26 07:29 Debian seems to be fine but bintar not.

Jan Lindström (Inactive) made changes - 2022-02-08 08:14

Fix Version/s		10.4.23 [ 26807 ]
Fix Version/s		10.5.14 [ 26809 ]
Fix Version/s		10.6.6 [ 26811 ]
Fix Version/s		10.7.2 [ 26813 ]
Fix Version/s	10.4 [ 22408 ]
Resolution		Fixed [ 1 ]
Status	Open [ 1 ]	Closed [ 6 ]

Jan Lindström (Inactive) added a comment - 2022-02-08 14:11

Fixed on Galera library 26.4.11 commit 9561a159c

Jan Lindström (Inactive) added a comment - 2022-02-08 14:11 Fixed on Galera library 26.4.11 commit 9561a159c

Pramod Mahto made changes - 2022-06-27 03:37

Link

This issue blocks ~~MDEV-27789~~ [ ~~MDEV-27789~~ ]

Jira Automation (IT) made changes - 2024-07-04 01:55

Zendesk Related Tickets

108294 158663 191328

People

Assignee:: Jan Lindström (Inactive)

Reporter:: Pon Suresh Pandian (Inactive)

Votes:: 5 Vote for this issue

Watchers:: 17 Start watching this issue

Dates

Created:: 2022-01-10 13:39

Updated:: 2024-07-07 20:41

Resolved:: 2022-02-08 08:14

Git Integration

Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.

MariaDB Server

Details

Description

Attachments

Attachments

Issue Links

Activity

People

Dates

Git Integration