[MDEV-15933] Cannot resume Node SYNCED state when wsrep_desync is done after FTWRL - Jira

Details

Type: Bug
Status: Closed (View Workflow)
Priority: Major
Resolution: Fixed
Affects Version/s: 10.3.6, 10.1(EOL), 10.2(EOL)
Fix Version/s: 10.1.36, 10.2.18, 10.3.10
Component/s: Galera
Labels:
None
Environment:
CentOS 7.4; Galera 2 Nodes

Description

Cannot resume Node SYNCED state when wsrep_desync is done after FTWRL

1. Issue ' flush tables with read lock;' on Galera Node
MariaDB [(none)]> flush tables with read lock;
Query OK, 0 rows affected (0.000 sec)

2. Set wsrep_desync – Command hangs and wsrep_desync=1 is not applied ,
but afterwards cannot resume SYNCED state

MariaDB [(none)]> set global wsrep_desync=1;
^CCtrl-C – query killed. Continuing normally.
^CCtrl-C – query killed. Continuing normally.
ERROR 2013 (HY000): Lost connection to MySQL server during query
MariaDB [(none)]> show processlist ;
ERROR 2006 (HY000): MySQL server has gone away
No connection. Trying to reconnect...
Connection id: 81
Current database: *** NONE ***

------------------------------------------------------------------------------------------------

User

Host

Command

Time

State

Info

Progress

------------------------------------------------------------------------------------------------

1	system user		NULL	Sleep	3331	WSREP aborter idle	NULL
2	system user		NULL	Sleep	3331	NULL	NULL
4	system user		NULL	Daemon	NULL	InnoDB purge coordinator	NULL
5	system user		NULL	Daemon	NULL	InnoDB purge worker	NULL
3	system user		NULL	Daemon	NULL	InnoDB purge worker	NULL
6	system user		NULL	Daemon	NULL	InnoDB purge worker	NULL
7	system user		NULL	Daemon	NULL	InnoDB shutdown handler	NULL
9	system user		NULL	Sleep	3331	NULL	NULL
8	system user		NULL	Sleep	3331	NULL	NULL
10	system user		NULL	Sleep	3331	NULL	NULL
26	system user		NULL	Sleep	3331	NULL	NULL
17	system user		NULL	Sleep	3331	NULL	NULL
16	system user		NULL	Sleep	3331	NULL	NULL
19	system user		NULL	Sleep	3331	NULL	NULL
20	system user		NULL	Sleep	3331	NULL	NULL
22	system user		NULL	Sleep	3331	NULL	NULL
23	system user		NULL	Sleep	3331	NULL	NULL
18	system user		NULL	Sleep	3331	NULL	NULL
24	system user		NULL	Sleep	3331	NULL	NULL
21	system user		NULL	Sleep	3331	NULL	NULL
14	system user		NULL	Sleep	3331	NULL	NULL
25	system user		NULL	Sleep	3331	NULL	NULL
35	system user		NULL	Sleep	3331	NULL	NULL
27	system user		NULL	Sleep	3331	NULL	NULL
13	system user		NULL	Sleep	3331	NULL	NULL
34	system user		NULL	Sleep	3331	NULL	NULL
11	system user		NULL	Sleep	3331	NULL	NULL
30	system user		NULL	Sleep	3331	NULL	NULL
37	system user		NULL	Sleep	3331	NULL	NULL
15	system user		NULL	Sleep	3331	NULL	NULL
33	system user		NULL	Sleep	3331	NULL	NULL
32	system user		NULL	Sleep	3331	NULL	NULL
31	system user		NULL	Sleep	3331	NULL	NULL
12	system user		NULL	Sleep	3331	NULL	NULL
55	system user		NULL	Sleep	3331	NULL	NULL
38	system user		NULL	Sleep	3331	NULL	NULL
51	system user		NULL	Sleep	3331	NULL	NULL
42	system user		NULL	Sleep	3331	NULL	NULL
43	system user		NULL	Sleep	3331	NULL	NULL
36	system user		NULL	Sleep	3331	NULL	NULL
41	system user		NULL	Sleep	3331	NULL	NULL
45	system user		NULL	Sleep	3331	NULL	NULL
44	system user		NULL	Sleep	3331	NULL	NULL
39	system user		NULL	Sleep	3331	NULL	NULL
48	system user		NULL	Sleep	3331	NULL	NULL
46	system user		NULL	Sleep	3331	NULL	NULL
47	system user		NULL	Sleep	3331	NULL	NULL
50	system user		NULL	Sleep	3331	NULL	NULL
52	system user		NULL	Sleep	3331	NULL	NULL
53	system user		NULL	Sleep	3331	NULL	NULL
40	system user		NULL	Sleep	3331	NULL	NULL
28	system user		NULL	Sleep	3331	NULL	NULL
49	system user		NULL	Sleep	3331	NULL	NULL
29	system user		NULL	Sleep	3331	NULL	NULL
58	system user		NULL	Sleep	3331	NULL	NULL
59	system user		NULL	Sleep	3331	NULL	NULL
61	system user		NULL	Sleep	3331	NULL	NULL
56	system user		NULL	Sleep	3331	NULL	NULL
60	system user		NULL	Sleep	3331	NULL	NULL
62	system user		NULL	Sleep	3331	NULL	NULL
57	system user		NULL	Sleep	3331	NULL	NULL
63	system user		NULL	Sleep	3331	NULL	NULL
64	system user		NULL	Sleep	3331	NULL	NULL
54	system user		NULL	Sleep	3331	NULL	NULL
65	system user		NULL	Sleep	3331	NULL	NULL
69	system user		NULL	Sleep	3331	NULL	NULL
66	system user		NULL	Sleep	3331	NULL	NULL
67	system user		NULL	Sleep	3331	NULL	NULL
70	system user		NULL	Sleep	3331	NULL	NULL
68	system user		NULL	Sleep	3331	NULL	NULL
78	root	localhost	NULL	Killed	30	NULL	set global wsrep_desync=1
81	root	localhost	NULL	Query	0	Init	show processlist

------------------------------------------------------------------------------------------------
72 rows in set (0.001 sec)

Node moved in DESYNCED state and cannot be recovered
2018-04-19 17:34:17 0 [Note] WSREP: Member 0.0 (t4w6) desyncs itself from group
2018-04-19 17:34:17 0 [Note] WSREP: Shifting SYNCED -> DONOR/DESYNCED (TO: 2)
2018-04-19 17:34:17 78 [Note] WSREP: Provider paused at 9f23373a-3e6d-11e8-b625-0646494b946c:2 (31)

MariaDB [(none)]> unlock tables ;
Query OK, 0 rows affected (0.000 sec)

MariaDB [(none)]> show variables like '%wsrep_desync%' ;
--------------------+

Variable_name

Value

--------------------+

wsrep_desync

OFF

--------------------+
1 row in set (0.002 sec)

MariaDB [(none)]> show status like 'wsrep_local_state_comment' ;
-----------------------------------------+

Variable_name

Value

-----------------------------------------+

wsrep_local_state_comment

Donor/Desynced

-----------------------------------------+
1 row in set (0.001 sec)

Attachments

Issue Links

relates to

MDEV-15067 Node fails to rejoin mdb cluster after recovering with mariabackup with option --galera-info

Closed

Activity

Ascending order - Click to sort in descending order

Zdravelina Sokolovska (Inactive) created issue - 2018-04-19 15:20

Zdravelina Sokolovska (Inactive) added a comment - 2018-04-19 15:39

In addition fail restart mariadb.service

# systemctl restart  mariadb.service

Job for mariadb.service failed because a timeout was exceeded. See "systemctl status mariadb.service" and "journalctl -xe" for details.

[root@t4w6 ~]# mysql -u root -p1

^C

[root@t4w6 ~]# mysql -u root -p1

^C

[root@t4w6 ~]# systemctl status mariadb.service

● mariadb.service - MariaDB 10.3.5 database server

   Loaded: loaded (/usr/lib/systemd/system/mariadb.service; enabled; vendor preset: disabled)

  Drop-In: /etc/systemd/system/mariadb.service.d

           └─migrated-from-my.cnf-settings.conf

   Active: failed (Result: timeout) since Thu 2018-04-19 18:30:51 EEST; 48s ago

     Docs: man:mysqld(8)

           https://mariadb.com/kb/en/library/systemd/

  Process: 1142 ExecStartPost=/bin/sh -c systemctl unset-environment _WSREP_START_POSITION (code=exited, status=0/SUCCESS)

  Process: 1389 ExecStartPre=/bin/sh -c [ ! -e /usr/bin/galera_recovery ] && VAR= ||   VAR=`/usr/bin/galera_recovery`; [ $? -eq 0 ]   && systemctl set-environment _WSREP_START_POSITION=$VAR || exit 1 (code=killed, signal=TERM)

  Process: 1387 ExecStartPre=/bin/sh -c systemctl unset-environment _WSREP_START_POSITION (code=exited, status=0/SUCCESS)

 Main PID: 1043

   CGroup: /system.slice/mariadb.service

           └─1043 /usr/sbin/mysqld --wsrep_start_position=9f23373a-3e6d-11e8-b625-0646494b946c:2

Apr 19 18:29:21 t4w6.xentio.lan systemd[1]: Starting MariaDB 10.3.5 database server...

Apr 19 18:30:51 t4w6.xentio.lan systemd[1]: mariadb.service start-pre operation timed out. Terminating.

Apr 19 18:30:51 t4w6.xentio.lan systemd[1]: Failed to start MariaDB 10.3.5 database server.

Apr 19 18:30:51 t4w6.xentio.lan systemd[1]: Unit mariadb.service entered failed state.

Apr 19 18:30:51 t4w6.xentio.lan systemd[1]: mariadb.service failed.

2018-04-19 18:26:20 0 [Note] /usr/sbin/mysqld (unknown): Normal shutdown

2018-04-19 18:26:20 0 [Note] WSREP: Stop replication

2018-04-19 18:26:20 0 [Note] WSREP: Closing send monitor...

2018-04-19 18:26:20 0 [Note] WSREP: Closed send monitor.

2018-04-19 18:26:20 0 [Note] WSREP: gcomm: terminating thread

2018-04-19 18:26:20 0 [Note] WSREP: gcomm: joining thread

2018-04-19 18:26:20 0 [Note] WSREP: gcomm: closing backend

2018-04-19 18:26:24 0 [Note] WSREP: (3c3e4c49, 'tcp://0.0.0.0:4567') connection to peer 6a5bf34e with addr tcp://192.168.104.195:4567 timed out, no messages seen in PT3S

2018-04-19 18:26:24 0 [Note] WSREP: (3c3e4c49, 'tcp://0.0.0.0:4567') turning message relay requesting on, nonlive peers: tcp://192.168.104.195:4567

2018-04-19 18:26:25 0 [Note] WSREP: (3c3e4c49, 'tcp://0.0.0.0:4567') reconnecting to 6a5bf34e (tcp://192.168.104.195:4567), attempt 0

2018-04-19 18:26:26 0 [Note] WSREP: evs::proto(3c3e4c49, LEAVING, view_id(REG,3c3e4c49,154)) suspecting node: 6a5bf34e

2018-04-19 18:26:26 0 [Note] WSREP: evs::proto(3c3e4c49, LEAVING, view_id(REG,3c3e4c49,154)) suspected node without join message, declaring inactive

2018-04-19 18:26:26 0 [Note] WSREP: view(view_id(NON_PRIM,3c3e4c49,154) memb {

        3c3e4c49,0

} joined {

} left {

} partitioned {

        6a5bf34e,0

})

2018-04-19 18:26:26 0 [Note] WSREP: view((empty))

2018-04-19 18:26:26 0 [Note] WSREP: gcomm: closed

2018-04-19 18:26:26 0 [Note] WSREP: New COMPONENT: primary = no, bootstrap = no, my_idx = 0, memb_num = 1

2018-04-19 18:26:26 0 [Note] WSREP: Flow-control interval: [16, 16]

2018-04-19 18:26:26 0 [Note] WSREP: Trying to continue unpaused monitor

2018-04-19 18:26:26 0 [Note] WSREP: Received NON-PRIMARY.

2018-04-19 18:26:26 0 [Note] WSREP: Shifting DONOR/DESYNCED -> OPEN (TO: 2)

2018-04-19 18:26:26 0 [Note] WSREP: Received self-leave message.

2018-04-19 18:26:26 0 [Note] WSREP: Flow-control interval: [0, 0]

2018-04-19 18:26:26 0 [Note] WSREP: Trying to continue unpaused monitor

2018-04-19 18:26:26 0 [Note] WSREP: Received SELF-LEAVE. Closing connection.

2018-04-19 18:26:26 0 [Note] WSREP: Shifting OPEN -> CLOSED (TO: 2)

2018-04-19 18:26:26 0 [Note] WSREP: RECV thread exiting 0: Success

2018-04-19 18:26:26 0 [Note] WSREP: recv_thread() joined.

2018-04-19 18:26:26 0 [Note] WSREP: Closing replication queue.

2018-04-19 18:26:26 0 [Note] WSREP: Closing slave action queue.

2018-04-19 18:26:28 0 [Note] WSREP: killing local connection: 78

Zdravelina Sokolovska (Inactive) added a comment - 2018-04-19 15:39 In addition fail restart mariadb.service # systemctl restart mariadb.service Job for mariadb.service failed because a timeout was exceeded. See "systemctl status mariadb.service" and "journalctl -xe" for details. [root@t4w6 ~]# mysql -u root -p1 ^C [root@t4w6 ~]# mysql -u root -p1 ^C [root@t4w6 ~]# systemctl status mariadb.service ● mariadb.service - MariaDB 10.3.5 database server Loaded: loaded (/usr/lib/systemd/system/mariadb.service; enabled; vendor preset: disabled) Drop-In: /etc/systemd/system/mariadb.service.d └─migrated-from-my.cnf-settings.conf Active: failed (Result: timeout) since Thu 2018-04-19 18:30:51 EEST; 48s ago Docs: man:mysqld(8) https://mariadb.com/kb/en/library/systemd/ Process: 1142 ExecStartPost=/bin/sh -c systemctl unset-environment _WSREP_START_POSITION (code=exited, status=0/SUCCESS) Process: 1389 ExecStartPre=/bin/sh -c [ ! -e /usr/bin/galera_recovery ] && VAR= || VAR=`/usr/bin/galera_recovery`; [ $? -eq 0 ] && systemctl set-environment _WSREP_START_POSITION=$VAR || exit 1 (code=killed, signal=TERM) Process: 1387 ExecStartPre=/bin/sh -c systemctl unset-environment _WSREP_START_POSITION (code=exited, status=0/SUCCESS) Main PID: 1043 CGroup: /system.slice/mariadb.service └─1043 /usr/sbin/mysqld --wsrep_start_position=9f23373a-3e6d-11e8-b625-0646494b946c:2 Apr 19 18:29:21 t4w6.xentio.lan systemd[1]: Starting MariaDB 10.3.5 database server... Apr 19 18:30:51 t4w6.xentio.lan systemd[1]: mariadb.service start-pre operation timed out. Terminating. Apr 19 18:30:51 t4w6.xentio.lan systemd[1]: Failed to start MariaDB 10.3.5 database server. Apr 19 18:30:51 t4w6.xentio.lan systemd[1]: Unit mariadb.service entered failed state. Apr 19 18:30:51 t4w6.xentio.lan systemd[1]: mariadb.service failed. 2018-04-19 18:26:20 0 [Note] /usr/sbin/mysqld (unknown): Normal shutdown 2018-04-19 18:26:20 0 [Note] WSREP: Stop replication 2018-04-19 18:26:20 0 [Note] WSREP: Closing send monitor... 2018-04-19 18:26:20 0 [Note] WSREP: Closed send monitor. 2018-04-19 18:26:20 0 [Note] WSREP: gcomm: terminating thread 2018-04-19 18:26:20 0 [Note] WSREP: gcomm: joining thread 2018-04-19 18:26:20 0 [Note] WSREP: gcomm: closing backend 2018-04-19 18:26:24 0 [Note] WSREP: (3c3e4c49, 'tcp://0.0.0.0:4567') connection to peer 6a5bf34e with addr tcp://192.168.104.195:4567 timed out, no messages seen in PT3S 2018-04-19 18:26:24 0 [Note] WSREP: (3c3e4c49, 'tcp://0.0.0.0:4567') turning message relay requesting on, nonlive peers: tcp://192.168.104.195:4567 2018-04-19 18:26:25 0 [Note] WSREP: (3c3e4c49, 'tcp://0.0.0.0:4567') reconnecting to 6a5bf34e (tcp://192.168.104.195:4567), attempt 0 2018-04-19 18:26:26 0 [Note] WSREP: evs::proto(3c3e4c49, LEAVING, view_id(REG,3c3e4c49,154)) suspecting node: 6a5bf34e 2018-04-19 18:26:26 0 [Note] WSREP: evs::proto(3c3e4c49, LEAVING, view_id(REG,3c3e4c49,154)) suspected node without join message, declaring inactive 2018-04-19 18:26:26 0 [Note] WSREP: view(view_id(NON_PRIM,3c3e4c49,154) memb { 3c3e4c49,0 } joined { } left { } partitioned { 6a5bf34e,0 }) 2018-04-19 18:26:26 0 [Note] WSREP: view((empty)) 2018-04-19 18:26:26 0 [Note] WSREP: gcomm: closed 2018-04-19 18:26:26 0 [Note] WSREP: New COMPONENT: primary = no, bootstrap = no, my_idx = 0, memb_num = 1 2018-04-19 18:26:26 0 [Note] WSREP: Flow-control interval: [16, 16] 2018-04-19 18:26:26 0 [Note] WSREP: Trying to continue unpaused monitor 2018-04-19 18:26:26 0 [Note] WSREP: Received NON-PRIMARY. 2018-04-19 18:26:26 0 [Note] WSREP: Shifting DONOR/DESYNCED -> OPEN (TO: 2) 2018-04-19 18:26:26 0 [Note] WSREP: Received self-leave message. 2018-04-19 18:26:26 0 [Note] WSREP: Flow-control interval: [0, 0] 2018-04-19 18:26:26 0 [Note] WSREP: Trying to continue unpaused monitor 2018-04-19 18:26:26 0 [Note] WSREP: Received SELF-LEAVE. Closing connection. 2018-04-19 18:26:26 0 [Note] WSREP: Shifting OPEN -> CLOSED (TO: 2) 2018-04-19 18:26:26 0 [Note] WSREP: RECV thread exiting 0: Success 2018-04-19 18:26:26 0 [Note] WSREP: recv_thread() joined. 2018-04-19 18:26:26 0 [Note] WSREP: Closing replication queue. 2018-04-19 18:26:26 0 [Note] WSREP: Closing slave action queue. 2018-04-19 18:26:28 0 [Note] WSREP: killing local connection: 78

Zdravelina Sokolovska (Inactive) made changes - 2018-04-19 15:41

Field	Original Value	New Value
Affects Version/s		10.3.6 [ 23003 ]

Elena Stepanova made changes - 2018-05-01 23:57

Fix Version/s		10.3 [ 22126 ]
Assignee		Sachin Setiya [ sachin.setiya.007 ]

Sergei Golubchik made changes - 2018-05-02 10:58

Assignee

Sachin Setiya [ sachin.setiya.007 ]

Jan Lindström [ jplindst ]

Zdravelina Sokolovska (Inactive) made changes - 2018-05-04 13:30

Link

This issue relates to ~~MDEV-15067~~ [ ~~MDEV-15067~~ ]

Ralf Gebhardt made changes - 2018-05-17 15:48

Assignee

Jan Lindström [ jplindst ]

Seppo Jaakola [ seppo ]

Mario Karuza (Inactive) made changes - 2018-08-10 12:41

Assignee

Seppo Jaakola [ seppo ]

Mario Karuza [ mkaruza ]

Mario Karuza (Inactive) made changes - 2018-08-10 12:41

Status

Open [ 1 ]

In Progress [ 3 ]

Mario Karuza (Inactive) added a comment - 2018-08-14 08:40

This is fixed by not allowing to set wsrep_desync after global read lock is acquired.
PR is done to 10.1.

Mario Karuza (Inactive) added a comment - 2018-08-14 08:40 This is fixed by not allowing to set wsrep_desync after global read lock is acquired. PR is done to 10.1.

Mario Karuza (Inactive) made changes - 2018-08-14 08:41

Assignee	Mario Karuza [ mkaruza ]	winstone [ winstone ]
Status	In Progress [ 3 ]	In Review [ 10002 ]

Jan Lindström (Inactive) made changes - 2018-08-15 07:18

Assignee

winstone [ winstone ]

Jan Lindström [ jplindst ]

Jan Lindström (Inactive) made changes - 2018-08-15 07:46

Status

In Review [ 10002 ]

Stalled [ 10000 ]

Jan Lindström (Inactive) made changes - 2018-08-15 07:46

Status

Stalled [ 10000 ]

In Progress [ 3 ]

Jan Lindström (Inactive) made changes - 2018-08-15 07:46

Affects Version/s

10.1 [ 16100 ]

Jan Lindström (Inactive) made changes - 2018-08-15 07:47

Affects Version/s

10.2 [ 14601 ]

Jan Lindström (Inactive) made changes - 2018-08-15 07:47

Fix Version/s		10.1 [ 16100 ]
Fix Version/s		10.2 [ 14601 ]

Jan Lindström (Inactive) made changes - 2018-08-15 07:47

issue.field.resolutiondate

2018-08-15 07:47:51.0

2018-08-15 07:47:51.63

Jan Lindström (Inactive) made changes - 2018-08-15 07:47

Fix Version/s		10.1.36 [ 23117 ]
Fix Version/s		10.2.18 [ 23112 ]
Fix Version/s		10.3.10 [ 23140 ]
Fix Version/s	10.2 [ 14601 ]
Fix Version/s	10.1 [ 16100 ]
Fix Version/s	10.3 [ 22126 ]
Resolution		Fixed [ 1 ]
Status	In Progress [ 3 ]	Closed [ 6 ]

Sergei Golubchik made changes - 2021-12-06 21:47

Workflow

MariaDB v3 [ 86650 ]

MariaDB v4 [ 154191 ]

People

Assignee:: Jan Lindström (Inactive)

Reporter:: Zdravelina Sokolovska (Inactive)

Votes:: 0 Vote for this issue

Watchers:: 4 Start watching this issue

Dates

Created:: 2018-04-19 15:20

Updated:: 2018-08-15 07:47

Resolved:: 2018-08-15 07:47

Git Integration

Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.

MariaDB Server

Details

Description

Attachments

Issue Links

Activity

People

Dates

Git Integration