[MDEV-16204] Data inconsistency is observed on the cluster after waiting for table metadata lock occurred on Galera Node Created: 2018-05-17  Updated: 2023-06-06  Resolved: 2023-06-06

Status: Closed
Project: MariaDB Server
Component/s: Galera
Affects Version/s: 10.3.6
Fix Version/s: N/A

Type: Bug Priority: Major
Reporter: Zdravelina Sokolovska (Inactive) Assignee: Seppo Jaakola
Resolution: Won't Fix Votes: 1
Labels: None
Environment:

Galera Master-Master 2 Nodes ;CentOS 7.4.

Issue Links:
Relates
relates to MDEV-16206 create or replace table as select in... Closed

 Description   

Data inconsistency is observed on the cluster after waiting for table metadata lock occurred on Galera Node

the wsrep status does not show any indication about the occurred inconsistency ,
cluster state is given operational and Synced;

On Node 1 occurred waiting for table metadata lock
After that next received DDL operations on Node1 stuck .
On Node 2 however are observed completed operations , Node1 and Node2
return different results .

Below on Node1 table CUSTOMER returns results while on Node2
table CUSTOMER does not exists.

Node2

 
 
MariaDB [(none)]>  select count(*) from tpcc.CUSTOMER ;
 
ERROR 1146 (42S02): Table 'tpcc.CUSTOMER' doesn't exist

Node1

MariaDB [(none)]> select count(*) from tpcc.CUSTOMER ;
 
+----------+
 
| count(*) |
 
+----------+
 
|    30000 |
 
+----------+
 
1 row in set (0.030 sec)
 
 
 



MariaDB [(none)]> show processlist ;
 
+----+-------------+-----------------+------+------------+-------+-------------------------------------------+-------------------------------------------+----------+
 
| Id | User        | Host            | db   | Command    | Time  | State                                     | Info                                      | Progress |
 
+----+-------------+-----------------+------+------------+-------+-------------------------------------------+-------------------------------------------+----------+
 
|  1 | system user |                 | NULL | Sleep      | 70918 | Committed 26860                           | NULL                                      |    0.000 |
 
|  2 | system user |                 | NULL | Sleep      | 86903 | WSREP aborter idle                        | NULL                                      |    0.000 |
 
|  3 | system user |                 | NULL | Daemon     |  NULL | InnoDB purge worker                       | NULL                                      |    0.000 |
 
|  4 | system user |                 | NULL | Daemon     |  NULL | InnoDB purge worker                       | NULL                                      |    0.000 |
 
|  5 | system user |                 | NULL | Daemon     |  NULL | InnoDB purge coordinator                  | NULL                                      |    0.000 |
 
|  6 | system user |                 | NULL | Daemon     |  NULL | InnoDB purge worker                       | NULL                                      |    0.000 |
 
|  7 | system user |                 | NULL | Daemon     |  NULL | InnoDB shutdown handler                   | NULL                                      |    0.000 |
 
| 13 | root        | localhost       | cctt | Killed     | 62724 | Init                                      | CREATE OR REPLACE TABLE t1 AS SELECT f1() |    0.000 |
 
| 14 | root        | localhost       | cctt | Field List |  5423 | Waiting for table metadata lock           |                                           |    0.000 |
 
| 25 | root        | localhost       | NULL | Query      |   690 | Waiting for stored function metadata lock | drop function cctt.f1                     |    0.000 |
 
| 27 | root        | localhost:40292 | tpcc | Query      |   209 | Checking permissions                      | DROP TABLE IF EXISTS CUSTOMER             |    0.000 |
 
| 28 | root        | localhost       | NULL | Query      |     0 | Init                                      | show processlist                          |    0.000 |
 
+----+-------------+-----------------+------+------------+-------+-------------------------------------------+-------------------------------------------+----------+
 
12 rows in set (0.000 sec)
 
 

\

MariaDB [(none)]> show status like '%wsrep%' ;
 
+------------------------------+-------------------------------------------+
 
| Variable_name                | Value                                     |
 
+------------------------------+-------------------------------------------+
 
| wsrep_apply_oooe             | 0.016129                                  |
 
| wsrep_apply_oool             | 0.000000                                  |
 
| wsrep_apply_window           | 1.024194                                  |
 
| wsrep_causal_reads           | 0                                         |
 
| wsrep_cert_deps_distance     | 1.661290                                  |
 
| wsrep_cert_index_size        | 34                                        |
 
| wsrep_cert_interval          | 0.024194                                  |
 
| wsrep_cluster_conf_id        | 88                                        |
 
| wsrep_cluster_size           | 2                                         |
 
| wsrep_cluster_state_uuid     | 3c15149f-5766-11e8-9a99-22bc53d40581      |
 
| wsrep_cluster_status         | Primary                                   |
 
| wsrep_commit_oooe            | 0.000000                                  |
 
| wsrep_commit_oool            | 0.000000                                  |
 
| wsrep_commit_window          | 1.000000                                  |
 
| wsrep_connected              | ON                                        |
 
| wsrep_desync_count           | 0                                         |
 
| wsrep_evs_delayed            |                                           |
 
| wsrep_evs_evict_list         |                                           |
 
| wsrep_evs_repl_latency       | 0/0/0/0/0                                 |
 
| wsrep_evs_state              | OPERATIONAL                               |
 
| wsrep_flow_control_paused    | 0.000000                                  |
 
| wsrep_flow_control_paused_ns | 0                                         |
 
| wsrep_flow_control_recv      | 0                                         |
 
| wsrep_flow_control_sent      | 0                                         |
 
| wsrep_gcomm_uuid             | d09be1b9-58f0-11e8-a6c7-de268e185d14      |
 
| wsrep_incoming_addresses     | 192.168.104.195:3306,192.168.104.196:3306 |
 
| wsrep_last_committed         | 26892                                     |
 
| wsrep_local_bf_aborts        | 0                                         |
 
| wsrep_local_cached_downto    | 26772                                     |
 
| wsrep_local_cert_failures    | 0                                         |
 
| wsrep_local_commits          | 55                                        |
 
| wsrep_local_index            | 1                                         |
 
| wsrep_local_recv_queue       | 0                                         |
 
| wsrep_local_recv_queue_avg   | 0.004484                                  |
 
| wsrep_local_recv_queue_max   | 2                                         |
 
| wsrep_local_recv_queue_min   | 0                                         |
 
| wsrep_local_replays          | 0                                         |
 
| wsrep_local_send_queue       | 0                                         |
 
| wsrep_local_send_queue_avg   | 0.000000                                  |
 
| wsrep_local_send_queue_max   | 1                                         |
 
| wsrep_local_send_queue_min   | 0                                         |
 
| wsrep_local_state            | 4                                         |
 
| wsrep_local_state_comment    | Synced                                    |
 
| wsrep_local_state_uuid       | 3c15149f-5766-11e8-9a99-22bc53d40581      |
 
| wsrep_protocol_version       | 7                                         |
 
| wsrep_provider_name          | Galera                                    |
 
| wsrep_provider_vendor        | Codership Oy <info@codership.com>         |
 
| wsrep_provider_version       | 25.3.23(r3789)                            |
 
| wsrep_ready                  | ON                                        |
 
| wsrep_received               | 223                                       |
 
| wsrep_received_bytes         | 20873                                     |
 
| wsrep_repl_data_bytes        | 36695                                     |
 
| wsrep_repl_keys              | 280                                       |
 
| wsrep_repl_keys_bytes        | 4633                                      |
 
| wsrep_repl_other_bytes       | 0                                         |
 
| wsrep_replicated             | 119                                       |
 
| wsrep_replicated_bytes       | 48944                                     |
 
| wsrep_thread_count           | 2                                         |
 
+------------------------------+-------------------------------------------+

It appears consecutively that Node1 hangs on 'drop' and 'create' statements
at 'Checking permissions' state while Node2 is processing all statements .

Node 1
 
MariaDB [(none)]> create database eee
 
    -> ;
 
use eee;
 
Command hangs


Node 1 processlist
 
28      root    localhost       NULL    Query   21258   Checking permissions    create database eee    


Node 1
 
MariaDB [(none)]> show databases like '%eee%' ;
 
Empty set (0.001 sec)
 
 
 
 
 
 
 
Node 2
 
MariaDB [(none)]>  show databases like '%eee%' ;
 
+------------------+
 
| Database (%eee%) |
 
+------------------+
 
| eee              |
 
+------------------+
 
1 row in set (0.000 sec)
 
 
 
MariaDB [(none)]>



 Comments   
Comment by Zdravelina Sokolovska (Inactive) [ 2018-05-22 ]

At least wsrep status should notify about the occurred inconsistency in the cluster .

Comment by Zdravelina Sokolovska (Inactive) [ 2018-05-22 ]

Galera Cluster v4 may avoid such data inconsistency via the intended inconsistency voting protocol in galera4 ?

Comment by Jan Lindström [ 2023-06-06 ]

10.3 is EOL soon.

Generated at Thu Feb 08 08:27:09 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.