[MDEV-19691] galera node crash after create table x, create table y(foreign key references x) Created: 2019-06-05  Updated: 2022-05-24  Resolved: 2022-05-16

Status: Closed
Project: MariaDB Server
Component/s: Galera
Affects Version/s: 10.1.40
Fix Version/s: N/A

Type: Bug Priority: Major
Reporter: Adam Bambuch Assignee: Jan Lindström (Inactive)
Resolution: Won't Fix Votes: 0
Labels: crash, galera
Environment:

server1 (master-slave replication) => server2 (galera) <=> server3 and server4

MariaDB [(none)]> show variables like "%wsrep%";
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

Variable_name Value

-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

wsrep_osu_method TOI
wsrep_auto_increment_control ON
wsrep_causal_reads OFF
wsrep_certification_rules strict
wsrep_certify_nonpk ON
wsrep_cluster_address -----------
wsrep_cluster_name +_lab_cluster
wsrep_convert_lock_to_trx OFF
wsrep_data_home_dir /var/lib/mysql/
wsrep_dbug_option  
wsrep_debug OFF
wsrep_desync OFF
wsrep_dirty_reads OFF
wsrep_drupal_282555_workaround OFF
wsrep_forced_binlog_format NONE
wsrep_gtid_domain_id 0
wsrep_gtid_mode OFF
wsrep_load_data_splitting ON
wsrep_log_conflicts OFF
wsrep_max_ws_rows 0
wsrep_max_ws_size 2147483647
wsrep_mysql_replication_bundle 0
wsrep_node_address  
wsrep_node_incoming_address AUTO
wsrep_node_name -------lab-dbsb1
wsrep_notify_cmd /opt/galeranotify/galeranotify.py
wsrep_on ON
wsrep_patch_version wsrep_25.24
wsrep_provider /usr/lib/galera/libgalera_smm.so
wsrep_provider_options base_dir = /var/lib/mysql/; base_host = 10.50.0.31; base_port = 4567; cert.log_conflicts = no; debug = no; evs.auto_evict = 0; evs.causal_keepalive_period = PT1S; evs.debug_log_mask = 0x1; evs.delay_margin = PT1S; evs.delayed_keep_period = PT30S; evs.inactive_check_period = PT0.5S; evs.inactive_timeout = PT15S; evs.info_log_mask = 0; evs.install_timeout = PT7.5S; evs.join_retrans_period = PT1S; evs.keepalive_period = PT1S; evs.max_install_timeouts = 3; evs.send_window = 4; evs.stats_report_period = PT1M; evs.suspect_timeout = PT5S; evs.use_aggregate = true; evs.user_send_window = 2; evs.version = 0; evs.view_forget_timeout = P1D; gcache.dir = /var/lib/mysql/; gcache.keep_pages_size = 0; gcache.mem_size = 0; gcache.name = /var/lib/mysql//galera.cache; gcache.page_size = 128M; gcache.recover = no; gcache.size = 2G; gcomm.thread_prio = ; gcs.fc_debug = 0; gcs.fc_factor = 1.0; gcs.fc_limit = 16; gcs.fc_master_slave = no; gcs.max_packet_size = 64500; gcs.max_throttle = 0.25; gcs.recv_q_hard_limit = 9223372036854775807; gcs.recv_q_soft_limit = 0.25; gcs.sync_donor = no; gmcast.listen_addr = tcp://0.0.0.0:4567; gmcast.mcast_addr = ; gmcast.mcast_ttl = 1; gmcast.peer_timeout = PT3S; gmcast.segment = 0; gmcast.time_wait = PT5S; gmcast.version = 0; ist.recv_addr = 10.50.0.31; pc.announce_timeout = PT3S; pc.checksum = false; pc.ignore_quorum = false; pc.ignore_sb = false; pc.linger = PT20S; pc.npvo = false; pc.recovery = true; pc.version = 0; pc.wait_prim = true; pc.wait_prim_timeout = PT30S; pc.weight = 1; protonet.backend = asio; protonet.version = 0; repl.causal_read_timeout = PT30S; repl.commit_order = 3; repl.key_format = FLAT8; repl.max_ws_size = 2147483647; repl.proto_max = 7; socket.checksum = 2; socket.recv_buf_size = 212992;
wsrep_recover OFF
wsrep_reject_queries NONE
wsrep_replicate_myisam ON
wsrep_restart_slave OFF
wsrep_retry_autocommit 1
wsrep_slave_fk_checks ON
wsrep_slave_uk_checks OFF
wsrep_slave_threads 20
wsrep_sst_auth
wsrep_sst_donor  
wsrep_sst_donor_rejects_queries OFF
wsrep_sst_method rsync
wsrep_sst_receive_address 10.50.0.31
wsrep_start_position 722b863e-ed2f-11e5-9393-3349539a4c2c:1845044
wsrep_sync_wait 0

-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
45 rows in set (0.00 sec)


 Description   

We have topology like this:
server1 (master-slave replication) => server2 (galera) <=> server3 and server4

When following sql is executed on server1, nodes server3 and server4 kills itself because of "Node consistency compromized".

DROP TABLE IF EXISTS `a`;
 
/*!40101 SET @saved_cs_client = @@character_set_client */;
 
/*!40101 SET character_set_client = utf8 */;
 
CREATE TABLE `a` (
 
`id` bigint(20) unsigned NOT NULL AUTO_INCREMENT,
 
`ct` mediumint(8) unsigned NOT NULL,
 
`dt` tinyint(4) NOT NULL DEFAULT '0',
 
`et` datetime NOT NULL,
 
`ft` datetime DEFAULT NULL,
 
`gt` varchar(255) COLLATE utf8_czech_ci NOT NULL,
 
`ht` varchar(255) COLLATE utf8_czech_ci NOT NULL,
 
`it` varchar(255) COLLATE utf8_czech_ci NOT NULL,
 
`jt` mediumtext COLLATE utf8_czech_ci NOT NULL,
 
`tt` text COLLATE utf8_czech_ci NOT NULL,
 
`xt` text COLLATE utf8_czech_ci NOT NULL,
 
PRIMARY KEY (`id`),
 
KEY `FK_a_projects` (`ct`),
 
KEY `dt` (`dt`),
 
CONSTRAINT `FK_a_projects` FOREIGN KEY (`ct`) REFERENCES `projects` (`id`) ON DELETE CASCADE ON UPDATE CASCADE
 
) ENGINE=InnoDB AUTO_INCREMENT=53577 DEFAULT CHARSET=utf8 COLLATE=utf8_czech_ci ROW_FORMAT=DYNAMIC;
 
 
 
DROP TABLE IF EXISTS `a_bts`;
 
/*!40101 SET @saved_cs_client = @@character_set_client */;
 
/*!40101 SET character_set_client = utf8 */;
 
CREATE TABLE `a_bts` (
 
`at` bigint(20) unsigned NOT NULL,
 
`bt` varchar(255) COLLATE utf8_czech_ci NOT NULL,
 
PRIMARY KEY (`at`,`bt`),
 
CONSTRAINT `FK_a_bts_a` FOREIGN KEY (`at`) REFERENCES `a` (`id`) ON DELETE CASCADE ON UPDATE CASCADE
 
) ENGINE=InnoDB DEFAULT CHARSET=utf8 COLLATE=utf8_czech_ci;


Jun  3 09:07:12 +-lab-dbsb2 mysqld: 2019-06-03  9:07:12 140065444357888 [Note] WSREP: REPL Protocols: 7 (3, 2)
 
Jun  3 09:07:12 +-lab-dbsb2 mysqld: 2019-06-03  9:07:12 140065444357888 [Note] WSREP: Assign initial position for certification: 1843806, protocol version: 3
 
Jun  3 09:07:12 +-lab-dbsb2 mysqld: 2019-06-03  9:07:12 140063115572992 [Note] WSREP: Service thread queue flushed.
 
Jun  3 09:07:15 +-lab-dbsb2 mysqld: 2019-06-03  9:07:15 140063065241344 [Note] WSREP: (eaf937c8, 'tcp://0.0.0.0:4567') turning message relay requesting off
 
Jun  3 09:13:18 +-lab-dbsb2 mysqld: 2019-06-03  9:13:18 140065428539136 [Warning] WSREP: BF applier failed to open_and_lock_tables: 1146, fatal: 0 wsrep = (exec_mode: 1 conflict_state: 0 seqno: 1845039)
 
Jun  3 09:13:18 +-lab-dbsb2 mysqld: 2019-06-03  9:13:18 140065428539136 [ERROR] Slave SQL: Error executing row event: 'Table 'st_cms.a' doesn't exist', Internal MariaDB error code: 1146
 
Jun  3 09:13:18 +-lab-dbsb2 mysqld: 2019-06-03  9:13:18 140065428539136 [Warning] WSREP: RBR event 14 Update_rows_v1 apply warning: 1146, 1845039
 
Jun  3 09:13:18 +-lab-dbsb2 mysqld: 2019-06-03  9:13:18 140065428539136 [Warning] WSREP: Failed to apply app buffer: seqno: 1845039, status: 1
 
Jun  3 09:13:18 +-lab-dbsb2 mysqld: #011 at galera/src/trx_handle.cpp:apply():351
 
Jun  3 09:13:18 +-lab-dbsb2 mysqld: Retrying 2th time
 
Jun  3 09:13:18 +-lab-dbsb2 mysqld: 2019-06-03  9:13:18 140065428539136 [Warning] WSREP: BF applier failed to open_and_lock_tables: 1146, fatal: 0 wsrep = (exec_mode: 1 conflict_state: 0 seqno: 1845039)
 
Jun  3 09:13:18 +-lab-dbsb2 mysqld: 2019-06-03  9:13:18 140065428539136 [ERROR] Slave SQL: Error executing row event: 'Table 'st_cms.a' doesn't exist', Internal MariaDB error code: 1146
 
Jun  3 09:13:18 +-lab-dbsb2 mysqld: 2019-06-03  9:13:18 140065428539136 [Warning] WSREP: RBR event 14 Update_rows_v1 apply warning: 1146, 1845039
 
Jun  3 09:13:18 +-lab-dbsb2 mysqld: 2019-06-03  9:13:18 140065428539136 [Warning] WSREP: Failed to apply app buffer: seqno: 1845039, status: 1
 
Jun  3 09:13:18 +-lab-dbsb2 mysqld: #011 at galera/src/trx_handle.cpp:apply():351
 
Jun  3 09:13:18 +-lab-dbsb2 mysqld: Retrying 3th time
 
Jun  3 09:13:18 +-lab-dbsb2 mysqld: 2019-06-03  9:13:18 140065428539136 [Warning] WSREP: BF applier failed to open_and_lock_tables: 1146, fatal: 0 wsrep = (exec_mode: 1 conflict_state: 0 seqno: 1845039)
 
Jun  3 09:13:18 +-lab-dbsb2 mysqld: 2019-06-03  9:13:18 140065428539136 [ERROR] Slave SQL: Error executing row event: 'Table 'st_cms.a' doesn't exist', Internal MariaDB error code: 1146
 
Jun  3 09:13:18 +-lab-dbsb2 mysqld: 2019-06-03  9:13:18 140065428539136 [Warning] WSREP: RBR event 14 Update_rows_v1 apply warning: 1146, 1845039
 
Jun  3 09:13:18 +-lab-dbsb2 mysqld: 2019-06-03  9:13:18 140065428539136 [Warning] WSREP: Failed to apply app buffer: seqno: 1845039, status: 1
 
Jun  3 09:13:18 +-lab-dbsb2 mysqld: #011 at galera/src/trx_handle.cpp:apply():351
 
Jun  3 09:13:18 +-lab-dbsb2 mysqld: Retrying 4th time
 
Jun  3 09:13:18 +-lab-dbsb2 mysqld: 2019-06-03  9:13:18 140065428539136 [Warning] WSREP: BF applier failed to open_and_lock_tables: 1146, fatal: 0 wsrep = (exec_mode: 1 conflict_state: 0 seqno: 1845039)
 
Jun  3 09:13:18 +-lab-dbsb2 mysqld: 2019-06-03  9:13:18 140065428539136 [ERROR] Slave SQL: Error executing row event: 'Table 'st_cms.a' doesn't exist', Internal MariaDB error code: 1146
 
Jun  3 09:13:18 +-lab-dbsb2 mysqld: 2019-06-03  9:13:18 140065428539136 [Warning] WSREP: RBR event 14 Update_rows_v1 apply warning: 1146, 1845039
 
Jun  3 09:13:18 +-lab-dbsb2 mysqld: 2019-06-03  9:13:18 140065428539136 [ERROR] WSREP: Failed to apply trx: source: d21e550c-85cd-11e9-bcb1-3f613e076d5d version: 3 local: 0 state: APPLYING flags: 1 conn_id: 25 trx_id: 166358317 seqnos (l: 1558, g: 1845039, s: 1845038, d: 1845012, ts: 1533043168716954)
 
Jun  3 09:13:18 +-lab-dbsb2 mysqld: 2019-06-03  9:13:18 140065428539136 [ERROR] WSREP: Failed to apply trx 1845039 4 times
 
Jun  3 09:13:18 +-lab-dbsb2 mysqld: 2019-06-03  9:13:18 140065428539136 [ERROR] WSREP: Node consistency compromized, aborting...



 Comments   
Comment by Jan Lindström (Inactive) [ 2022-05-16 ]

10.1 has been EOL.

Generated at Thu Feb 08 08:53:37 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.