[MDEV-9515] Galera to Galera Async Master / Master Replication Crash Cluster Created: 2016-02-03  Updated: 2016-03-30  Resolved: 2016-03-22

Status: Closed
Project: MariaDB Server
Component/s: Galera, Replication
Affects Version/s: 10.1.10, 10.1.11, 10.1.12
Fix Version/s: 10.1.13

Type: Bug Priority: Blocker
Reporter: Bernd Buffen Assignee: Nirbhay Choubey (Inactive)
Resolution: Fixed Votes: 1
Labels: None
Environment:
  1. cat /etc/debian_version
    8.3
  2. uname -a
    Linux j285499.servers.jiffybox.net 4.1.16-x86_64-jb1 #1 SMP Wed Jan 27 07:37:00 CET 2016 x86_64 GNU/Linux
    –

MariaDB [(none)]> select version();
------------------------------

version()

------------------------------

10.1.11-MariaDB-1~jessie-log

------------------------------
1 row in set (0.00 sec)

MariaDB [(none)]>

Attachments: Zip Archive Archiv.zip     File g1n2.err     File syslog.cpy.gz    
Issue Links:
Relates
relates to MDEV-9498 MariaDB server with Galera replicatio... Closed

 Description   

Two Galera Cluster with 3 nodes. Node 1 of each cluster is connected as master / master Async Replication.

The cluster crashes every time direct after a table has been created and a few insert and updates Statements.

Usually the table is lost on a cluster.

If a start only start Node 1 from each Cluster without WSREP_PROVIDER it works fine.

 Comments   
Comment by Bernd Buffen [ 2016-02-04 ]

More log ..... 

160204 15:44:43 [ERROR] mysqld got signal 11 ;
 
 
 
2016-02-04 15:43:36 140127502333696 [Note] WSREP: forgetting 17292a96 (tcp://134.119.46.42:4567)
 
2016-02-04 15:43:36 140127502333696 [Note] WSREP: Node 1803db2b state prim
 
2016-02-04 15:43:36 140127502333696 [Note] WSREP: view(view_id(PRIM,1803db2b,5) memb {
 
	1803db2b,0
 
	9a684190,0
 
} joined {
 
} left {
 
} partitioned {
 
	17292a96,0
 
})
 
2016-02-04 15:43:36 140127502333696 [Note] WSREP: save pc into disk
 
2016-02-04 15:43:36 140127502333696 [Note] WSREP: forgetting 17292a96 (tcp://134.119.46.42:4567)
 
2016-02-04 15:43:36 140127493940992 [Note] WSREP: New COMPONENT: primary = yes, bootstrap = no, my_idx = 1, memb_num = 2
 
2016-02-04 15:43:36 140127493940992 [Note] WSREP: STATE EXCHANGE: Waiting for state UUID.
 
2016-02-04 15:43:36 140127493940992 [Note] WSREP: STATE EXCHANGE: sent state msg: abc81e79-cb4d-11e5-bf6b-d3af400d96d2
 
2016-02-04 15:43:36 140127493940992 [Note] WSREP: STATE EXCHANGE: got state msg: abc81e79-cb4d-11e5-bf6b-d3af400d96d2 from 0 (G2N3)
 
2016-02-04 15:43:36 140127493940992 [Note] WSREP: STATE EXCHANGE: got state msg: abc81e79-cb4d-11e5-bf6b-d3af400d96d2 from 1 (G2N1)
 
2016-02-04 15:43:36 140127493940992 [Note] WSREP: Quorum results:
 
	version    = 3,
 
	component  = PRIMARY,
 
	conf_id    = 4,
 
	members    = 2/2 (joined/total),
 
	act_id     = 12,
 
	last_appl. = 0,
 
	protocols  = 0/7/3 (gcs/repl/appl),
 
	group UUID = 870209fd-cb49-11e5-a506-6e706a3c4a43
 
2016-02-04 15:43:36 140127493940992 [Note] WSREP: Flow-control interval: [23, 23]
 
2016-02-04 15:43:36 140127855196928 [Note] WSREP: New cluster view: global state: 870209fd-cb49-11e5-a506-6e706a3c4a43:12, view# 5: Primary, number of nodes: 2, my index: 1, protocol version 3
 
2016-02-04 15:43:36 140127855196928 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
 
2016-02-04 15:43:36 140127855196928 [Note] WSREP: REPL Protocols: 7 (3, 2)
 
2016-02-04 15:43:36 140127560443648 [Note] WSREP: Service thread queue flushed.
 
2016-02-04 15:43:36 140127855196928 [Note] WSREP: Assign initial position for certification: 12, protocol version: 3
 
2016-02-04 15:43:36 140127560443648 [Note] WSREP: Service thread queue flushed.
 
2016-02-04 15:43:37 140127502333696 [Note] WSREP: forgetting 1803db2b (tcp://10.1.38.126:4567)
 
2016-02-04 15:43:37 140127502333696 [Note] WSREP: forgetting 1803db2b (tcp://134.119.46.59:4567)
 
2016-02-04 15:43:37 140127502333696 [Note] WSREP: Node 9a684190 state prim
 
2016-02-04 15:43:37 140127502333696 [Note] WSREP: view(view_id(PRIM,9a684190,6) memb {
 
	9a684190,0
 
} joined {
 
} left {
 
} partitioned {
 
	1803db2b,0
 
})
 
2016-02-04 15:43:37 140127502333696 [Note] WSREP: save pc into disk
 
2016-02-04 15:43:37 140127502333696 [Note] WSREP: forgetting 17292a96 (tcp://134.119.46.42:4567)
 
2016-02-04 15:43:37 140127502333696 [Note] WSREP: forgetting 1803db2b (tcp://10.1.38.126:4567)
 
2016-02-04 15:43:37 140127502333696 [Note] WSREP: forgetting 1803db2b (tcp://134.119.46.59:4567)
 
2016-02-04 15:43:37 140127493940992 [Note] WSREP: New COMPONENT: primary = yes, bootstrap = no, my_idx = 0, memb_num = 1
 
2016-02-04 15:43:37 140127493940992 [Note] WSREP: STATE_EXCHANGE: sent state UUID: acb06481-cb4d-11e5-8d9f-8763f227c10f
 
2016-02-04 15:43:37 140127493940992 [Note] WSREP: STATE EXCHANGE: sent state msg: acb06481-cb4d-11e5-8d9f-8763f227c10f
 
2016-02-04 15:43:37 140127493940992 [Note] WSREP: STATE EXCHANGE: got state msg: acb06481-cb4d-11e5-8d9f-8763f227c10f from 0 (G2N1)
 
2016-02-04 15:43:37 140127493940992 [Note] WSREP: Quorum results:
 
	version    = 3,
 
	component  = PRIMARY,
 
	conf_id    = 5,
 
	members    = 1/1 (joined/total),
 
	act_id     = 12,
 
	last_appl. = 0,
 
	protocols  = 0/7/3 (gcs/repl/appl),
 
	group UUID = 870209fd-cb49-11e5-a506-6e706a3c4a43
 
2016-02-04 15:43:37 140127493940992 [Note] WSREP: Flow-control interval: [16, 16]
 
2016-02-04 15:43:37 140127855196928 [Note] WSREP: New cluster view: global state: 870209fd-cb49-11e5-a506-6e706a3c4a43:12, view# 6: Primary, number of nodes: 1, my index: 0, protocol version 3
 
2016-02-04 15:43:37 140127855196928 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
 
2016-02-04 15:43:37 140127855196928 [Note] WSREP: REPL Protocols: 7 (3, 2)
 
2016-02-04 15:43:37 140127560443648 [Note] WSREP: Service thread queue flushed.
 
2016-02-04 15:43:37 140127855196928 [Note] WSREP: Assign initial position for certification: 12, protocol version: 3
 
2016-02-04 15:43:37 140127560443648 [Note] WSREP: Service thread queue flushed.
 
2016-02-04 15:43:41 140127502333696 [Note] WSREP:  cleaning up 17292a96 (tcp://134.119.46.42:4567)
 
2016-02-04 15:43:41 140127502333696 [Note] WSREP: (9a684190, 'tcp://0.0.0.0:4567') turning message relay requesting on, nonlive peers:
 
2016-02-04 15:43:42 140127502333696 [Note] WSREP: declaring c95ec118 at tcp://134.119.46.42:4567 stable
 
2016-02-04 15:43:42 140127502333696 [Note] WSREP: Node 9a684190 state prim
 
2016-02-04 15:43:42 140127502333696 [Note] WSREP: view(view_id(PRIM,9a684190,7) memb {
 
	9a684190,0
 
	c95ec118,0
 
} joined {
 
} left {
 
} partitioned {
 
})
 
2016-02-04 15:43:42 140127502333696 [Note] WSREP: save pc into disk
 
2016-02-04 15:43:42 140127502333696 [Note] WSREP: forgetting 1803db2b (tcp://10.1.38.126:4567)
 
2016-02-04 15:43:42 140127493940992 [Note] WSREP: New COMPONENT: primary = yes, bootstrap = no, my_idx = 0, memb_num = 2
 
2016-02-04 15:43:42 140127502333696 [Note] WSREP: forgetting 1803db2b (tcp://134.119.46.59:4567)
 
2016-02-04 15:43:42 140127493940992 [Note] WSREP: STATE_EXCHANGE: sent state UUID: af7092d1-cb4d-11e5-a71e-7fa5748e9870
 
2016-02-04 15:43:42 140127493940992 [Note] WSREP: STATE EXCHANGE: sent state msg: af7092d1-cb4d-11e5-a71e-7fa5748e9870
 
2016-02-04 15:43:42 140127493940992 [Note] WSREP: STATE EXCHANGE: got state msg: af7092d1-cb4d-11e5-a71e-7fa5748e9870 from 0 (G2N1)
 
2016-02-04 15:43:42 140127493940992 [Note] WSREP: STATE EXCHANGE: got state msg: af7092d1-cb4d-11e5-a71e-7fa5748e9870 from 1 (G2N2)
 
2016-02-04 15:43:42 140127493940992 [Note] WSREP: Quorum results:
 
	version    = 3,
 
	component  = PRIMARY,
 
	conf_id    = 6,
 
	members    = 1/2 (joined/total),
 
	act_id     = 12,
 
	last_appl. = 0,
 
	protocols  = 0/7/3 (gcs/repl/appl),
 
	group UUID = 870209fd-cb49-11e5-a506-6e706a3c4a43
 
2016-02-04 15:43:42 140127493940992 [Note] WSREP: Flow-control interval: [23, 23]
 
2016-02-04 15:43:42 140127855196928 [Note] WSREP: New cluster view: global state: 870209fd-cb49-11e5-a506-6e706a3c4a43:12, view# 7: Primary, number of nodes: 2, my index: 0, protocol version 3
 
2016-02-04 15:43:42 140127855196928 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
 
2016-02-04 15:43:42 140127855196928 [Note] WSREP: REPL Protocols: 7 (3, 2)
 
2016-02-04 15:43:42 140127560443648 [Note] WSREP: Service thread queue flushed.
 
2016-02-04 15:43:42 140127855196928 [Note] WSREP: Assign initial position for certification: 12, protocol version: 3
 
2016-02-04 15:43:42 140127560443648 [Note] WSREP: Service thread queue flushed.
 
2016-02-04 15:43:42 140127493940992 [Note] WSREP: Member 1.0 (G2N2) requested state transfer from '*any*'. Selected 0.0 (G2N1)(SYNCED) as donor.
 
2016-02-04 15:43:42 140127493940992 [Note] WSREP: Shifting SYNCED -> DONOR/DESYNCED (TO: 12)
 
2016-02-04 15:43:42 140127855196928 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
 
2016-02-04 15:43:42 140126617331456 [Note] WSREP: Running: 'wsrep_sst_rsync --role 'donor' --address '10.1.38.109:4444/rsync_sst' --socket '/var/run/mysqld/mysqld.sock' --datadir '/var/lib/mysql/'    --binlog 'binlog2' --gtid '870209fd-cb49-11e5-a506-6e706a3c4a43:12' --gtid-domain-id '0''
 
2016-02-04 15:43:42 140127855196928 [Note] WSREP: sst_donor_thread signaled with 0
 
2016-02-04 15:43:42 140126617331456 [Note] WSREP: Flushing tables for SST...
 
2016-02-04 15:43:42 140126617331456 [Note] WSREP: Provider paused at 870209fd-cb49-11e5-a506-6e706a3c4a43:12 (26)
 
2016-02-04 15:43:42 140126617331456 [Note] WSREP: Tables flushed.
 
WSREP_SST: [INFO] Preparing binlog files for transfer: (20160204 15:43:42.813)
 
binlog2.000015
 
2016-02-04 15:43:42 140127502333696 [Note] WSREP: remote endpoint tcp://134.119.46.59:4567 changed identity 1803db2b -> afbfafb7
 
2016-02-04 15:43:43 140127502333696 [Note] WSREP: declaring afbfafb7 at tcp://134.119.46.59:4567 stable
 
2016-02-04 15:43:43 140127502333696 [Note] WSREP: declaring c95ec118 at tcp://10.1.38.109:4567 stable
 
2016-02-04 15:43:43 140127502333696 [Note] WSREP: Node 9a684190 state prim
 
2016-02-04 15:43:43 140127502333696 [Note] WSREP: view(view_id(PRIM,9a684190,8) memb {
 
	9a684190,0
 
	afbfafb7,0
 
	c95ec118,0
 
} joined {
 
} left {
 
} partitioned {
 
})
 
2016-02-04 15:43:43 140127502333696 [Note] WSREP: save pc into disk
 
2016-02-04 15:43:43 140127502333696 [Note] WSREP: forgetting 1803db2b (tcp://10.1.38.126:4567)
 
2016-02-04 15:43:43 140127493940992 [Note] WSREP: New COMPONENT: primary = yes, bootstrap = no, my_idx = 0, memb_num = 3
 
2016-02-04 15:43:43 140127493940992 [Note] WSREP: STATE_EXCHANGE: sent state UUID: b04a2e2f-cb4d-11e5-a7dc-33111947724d
 
2016-02-04 15:43:43 140127493940992 [Note] WSREP: STATE EXCHANGE: sent state msg: b04a2e2f-cb4d-11e5-a7dc-33111947724d
 
2016-02-04 15:43:43 140127493940992 [Note] WSREP: STATE EXCHANGE: got state msg: b04a2e2f-cb4d-11e5-a7dc-33111947724d from 0 (G2N1)
 
2016-02-04 15:43:43 140127493940992 [Note] WSREP: STATE EXCHANGE: got state msg: b04a2e2f-cb4d-11e5-a7dc-33111947724d from 2 (G2N2)
 
2016-02-04 15:43:43 140127493940992 [Note] WSREP: STATE EXCHANGE: got state msg: b04a2e2f-cb4d-11e5-a7dc-33111947724d from 1 (G2N3)
 
2016-02-04 15:43:43 140127493940992 [Note] WSREP: Quorum results:
 
	version    = 3,
 
	component  = PRIMARY,
 
	conf_id    = 7,
 
	members    = 1/3 (joined/total),
 
	act_id     = 12,
 
	last_appl. = 0,
 
	protocols  = 0/7/3 (gcs/repl/appl),
 
	group UUID = 870209fd-cb49-11e5-a506-6e706a3c4a43
 
2016-02-04 15:43:43 140127493940992 [Note] WSREP: Flow-control interval: [28, 28]
 
2016-02-04 15:43:44 140127493940992 [Warning] WSREP: Member 1.0 (G2N3) requested state transfer from '*any*', but it is impossible to select State Transfer donor: Resource temporarily unavailable
 
2016-02-04 15:43:45 140127493940992 [Warning] WSREP: Member 1.0 (G2N3) requested state transfer from '*any*', but it is impossible to select State Transfer donor: Resource temporarily unavailable
 
2016-02-04 15:43:46 140127502333696 [Note] WSREP: (9a684190, 'tcp://0.0.0.0:4567') turning message relay requesting off
 
2016-02-04 15:43:46 140127493940992 [Warning] WSREP: Member 1.0 (G2N3) requested state transfer from '*any*', but it is impossible to select State Transfer donor: Resource temporarily unavailable
 
2016-02-04 15:43:46 140127502333696 [Note] WSREP:  cleaning up 1803db2b (tcp://10.1.38.126:4567)
 
2016-02-04 15:43:47 140127493940992 [Warning] WSREP: Member 1.0 (G2N3) requested state transfer from '*any*', but it is impossible to select State Transfer donor: Resource temporarily unavailable
 
2016-02-04 15:43:48 140127493940992 [Warning] WSREP: Member 1.0 (G2N3) requested state transfer from '*any*', but it is impossible to select State Transfer donor: Resource temporarily unavailable
 
2016-02-04 15:43:49 140127493940992 [Warning] WSREP: Member 1.0 (G2N3) requested state transfer from '*any*', but it is impossible to select State Transfer donor: Resource temporarily unavailable
 
2016-02-04 15:43:50 140127493940992 [Warning] WSREP: Member 1.0 (G2N3) requested state transfer from '*any*', but it is impossible to select State Transfer donor: Resource temporarily unavailable
 
2016-02-04 15:43:51 140127493940992 [Warning] WSREP: Member 1.0 (G2N3) requested state transfer from '*any*', but it is impossible to select State Transfer donor: Resource temporarily unavailable
 
2016-02-04 15:43:52 140127493940992 [Warning] WSREP: Member 1.0 (G2N3) requested state transfer from '*any*', but it is impossible to select State Transfer donor: Resource temporarily unavailable
 
2016-02-04 15:43:53 140127493940992 [Warning] WSREP: Member 1.0 (G2N3) requested state transfer from '*any*', but it is impossible to select State Transfer donor: Resource temporarily unavailable
 
2016-02-04 15:43:54 140127493940992 [Warning] WSREP: Member 1.0 (G2N3) requested state transfer from '*any*', but it is impossible to select State Transfer donor: Resource temporarily unavailable
 
2016-02-04 15:43:55 140127493940992 [Warning] WSREP: Member 1.0 (G2N3) requested state transfer from '*any*', but it is impossible to select State Transfer donor: Resource temporarily unavailable
 
2016-02-04 15:43:56 140127493940992 [Warning] WSREP: Member 1.0 (G2N3) requested state transfer from '*any*', but it is impossible to select State Transfer donor: Resource temporarily unavailable
 
2016-02-04 15:43:57 140127493940992 [Warning] WSREP: Member 1.0 (G2N3) requested state transfer from '*any*', but it is impossible to select State Transfer donor: Resource temporarily unavailable
 
2016-02-04 15:43:58 140127493940992 [Warning] WSREP: Member 1.0 (G2N3) requested state transfer from '*any*', but it is impossible to select State Transfer donor: Resource temporarily unavailable
 
2016-02-04 15:43:59 140127493940992 [Warning] WSREP: Member 1.0 (G2N3) requested state transfer from '*any*', but it is impossible to select State Transfer donor: Resource temporarily unavailable
 
2016-02-04 15:44:00 140127493940992 [Warning] WSREP: Member 1.0 (G2N3) requested state transfer from '*any*', but it is impossible to select State Transfer donor: Resource temporarily unavailable
 
2016-02-04 15:44:01 140127493940992 [Warning] WSREP: Member 1.0 (G2N3) requested state transfer from '*any*', but it is impossible to select State Transfer donor: Resource temporarily unavailable
 
2016-02-04 15:44:02 140127493940992 [Warning] WSREP: Member 1.0 (G2N3) requested state transfer from '*any*', but it is impossible to select State Transfer donor: Resource temporarily unavailable
 
2016-02-04 15:44:03 140127493940992 [Warning] WSREP: Member 1.0 (G2N3) requested state transfer from '*any*', but it is impossible to select State Transfer donor: Resource temporarily unavailable
 
2016-02-04 15:44:04 140127493940992 [Warning] WSREP: Member 1.0 (G2N3) requested state transfer from '*any*', but it is impossible to select State Transfer donor: Resource temporarily unavailable
 
2016-02-04 15:44:05 140127493940992 [Warning] WSREP: Member 1.0 (G2N3) requested state transfer from '*any*', but it is impossible to select State Transfer donor: Resource temporarily unavailable
 
2016-02-04 15:44:06 140127493940992 [Warning] WSREP: Member 1.0 (G2N3) requested state transfer from '*any*', but it is impossible to select State Transfer donor: Resource temporarily unavailable
 
2016-02-04 15:44:07 140127493940992 [Warning] WSREP: Member 1.0 (G2N3) requested state transfer from '*any*', but it is impossible to select State Transfer donor: Resource temporarily unavailable
 
2016-02-04 15:44:09 140127493940992 [Warning] WSREP: Member 1.0 (G2N3) requested state transfer from '*any*', but it is impossible to select State Transfer donor: Resource temporarily unavailable
 
2016-02-04 15:44:10 140127493940992 [Warning] WSREP: Member 1.0 (G2N3) requested state transfer from '*any*', but it is impossible to select State Transfer donor: Resource temporarily unavailable
 
2016-02-04 15:44:11 140127493940992 [Warning] WSREP: Member 1.0 (G2N3) requested state transfer from '*any*', but it is impossible to select State Transfer donor: Resource temporarily unavailable
 
2016-02-04 15:44:11 140126617331456 [Note] WSREP: resuming provider at 26
 
2016-02-04 15:44:11 140126617331456 [Note] WSREP: Provider resumed.
 
2016-02-04 15:44:11 140127855196928 [Note] WSREP: New cluster view: global state: 870209fd-cb49-11e5-a506-6e706a3c4a43:12, view# 8: Primary, number of nodes: 3, my index: 0, protocol version 3
 
2016-02-04 15:44:11 140127855196928 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
 
2016-02-04 15:44:11 140127855196928 [Note] WSREP: REPL Protocols: 7 (3, 2)
 
2016-02-04 15:44:11 140127560443648 [Note] WSREP: Service thread queue flushed.
 
2016-02-04 15:44:11 140127855196928 [Note] WSREP: Assign initial position for certification: 12, protocol version: 3
 
2016-02-04 15:44:11 140127560443648 [Note] WSREP: Service thread queue flushed.
 
2016-02-04 15:44:12 140127493940992 [Warning] WSREP: Member 1.0 (G2N3) requested state transfer from '*any*', but it is impossible to select State Transfer donor: Resource temporarily unavailable
 
2016-02-04 15:44:12 140127493940992 [Note] WSREP: 0.0 (G2N1): State transfer to 2.0 (G2N2) complete.
 
2016-02-04 15:44:12 140127493940992 [Note] WSREP: Shifting DONOR/DESYNCED -> JOINED (TO: 13)
 
2016-02-04 15:44:12 140127493940992 [Note] WSREP: Member 0.0 (G2N1) synced with group.
 
2016-02-04 15:44:12 140127493940992 [Note] WSREP: Shifting JOINED -> SYNCED (TO: 13)
 
2016-02-04 15:44:12 140127855196928 [Note] WSREP: Synchronized with group, ready for connections
 
2016-02-04 15:44:12 140127855196928 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
 
2016-02-04 15:44:13 140127493940992 [Note] WSREP: Member 1.0 (G2N3) requested state transfer from '*any*'. Selected 0.0 (G2N1)(SYNCED) as donor.
 
2016-02-04 15:44:13 140127493940992 [Note] WSREP: Shifting SYNCED -> DONOR/DESYNCED (TO: 13)
 
2016-02-04 15:44:13 140127855196928 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
 
2016-02-04 15:44:13 140126608938752 [Note] WSREP: Running: 'wsrep_sst_rsync --role 'donor' --address '10.1.38.126:4444/rsync_sst' --socket '/var/run/mysqld/mysqld.sock' --datadir '/var/lib/mysql/'    --binlog 'binlog2' --gtid '870209fd-cb49-11e5-a506-6e706a3c4a43:13' --gtid-domain-id '0''
 
2016-02-04 15:44:13 140127855196928 [Note] WSREP: sst_donor_thread signaled with 0
 
2016-02-04 15:44:13 140126608938752 [Note] WSREP: Flushing tables for SST...
 
2016-02-04 15:44:13 140126608938752 [Note] WSREP: Provider paused at 870209fd-cb49-11e5-a506-6e706a3c4a43:13 (32)
 
2016-02-04 15:44:13 140126608938752 [Note] WSREP: Tables flushed.
 
WSREP_SST: [INFO] Preparing binlog files for transfer: (20160204 15:44:13.421)
 
binlog2.000016
 
2016-02-04 15:44:14 140127493940992 [Note] WSREP: 2.0 (G2N2): State transfer from 0.0 (G2N1) complete.
 
2016-02-04 15:44:14 140127493940992 [Note] WSREP: Member 2.0 (G2N2) synced with group.
 
2016-02-04 15:44:43 140126608938752 [Note] WSREP: resuming provider at 32
 
2016-02-04 15:44:43 140126608938752 [Note] WSREP: Provider resumed.
 
160204 15:44:43 [ERROR] mysqld got signal 11 ;
 
This could be because you hit a bug. It is also possible that this binary
 
or one of the libraries it was linked against is corrupt, improperly built,
 
or misconfigured. This error can also be caused by malfunctioning hardware.
 
 
 
To report this bug, see http://kb.askmonty.org/en/reporting-bugs
 
 
 
We will try our best to scrape up some info that will hopefully help
 
diagnose the problem, but since we have already crashed,
 
something is definitely wrong and this may fail.
 
 
 
Server version: 10.1.11-MariaDB-1~jessie-log
 
key_buffer_size=134217728
 
read_buffer_size=2097152
 
max_used_connections=6
 
max_threads=102
 
thread_count=6
 
It is possible that mysqld could use up to
 
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 759826 K  bytes of memory
 
Hope that's ok; if not, decrease some variables in the equation.
 
 
 
Thread pointer: 0x0x7f7205c54008
 
Attempting backtrace. You can use the following information to find out
 
where mysqld died. If you see no messages after this, something went
 
terribly wrong...
 
stack_bottom = 0x7f72061f69a8 thread_stack 0x48400
 
mysqld(my_print_stacktrace+0x2e)[0x56114572032e]
 
mysqld(handle_fatal_signal+0x34d)[0x5611452628fd]
 
/lib/x86_64-linux-gnu/libpthread.so.0(+0xf8d0)[0x7f720ecb48d0]
 
mysqld(_ZN13MYSQL_BIN_LOG13mark_xid_doneEmb+0x7b)[0x561145317bcb]
 
mysqld(_ZN13MYSQL_BIN_LOG5unlogEmy+0x2a)[0x561145317e4a]
 
mysqld(_Z15ha_commit_transP3THDb+0x758)[0x561145265e98]
 
mysqld(_ZN15rpl_slave_state11record_gtidEP3THDPK8rpl_gtidybb+0x802)[0x5611451efb92]
 
mysqld(_ZN19Gtid_list_log_event14do_apply_eventEP14rpl_group_info+0xa6)[0x561145324026]
 
mysqld(_Z26apply_event_and_update_posP9Log_eventP3THDP14rpl_group_infoP19rpl_parallel_thread+0x1e1)[0x561145065291]
 
mysqld(handle_slave_sql+0x2762)[0x561145068e22]
 
/lib/x86_64-linux-gnu/libpthread.so.0(+0x80a4)[0x7f720ecad0a4]
 
/lib/x86_64-linux-gnu/libc.so.6(clone+0x6d)[0x7f720ce599cd]
 
 
 
Trying to get some variables.
 
Some pointers may be invalid and cause the dump to abort.
 
Query (0x0): is an invalid pointer
 
Connection ID (thread ID): 17
 
Status: NOT_KILLED
 
 
 
Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=off,table_elimination=on,extended_keys=on,exists_to_in=on
 
 
 
The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
 
information that should help you find out what is causing the crash.

Comment by Nirbhay Choubey (Inactive) [ 2016-02-16 ]

Bernd Buffen Hi!, I could not reproduce this problem (C1 <--- async repl ---> C2). Could you share the server configuration used for the cluster nodes, especially for ones acting as async master?

Comment by Bernd Buffen [ 2016-02-16 ]

@Nirbhay Choubey Hello, it so you write it. the replication is everytime from Node1 C1N1 <--- async --> C2N1. If have minimize the Cluster to see whats wrong. So when i have only 1 Node in each cluster ( i know thats no Cluster ) it works fine. When i put a second NODE in cluster 1 it crashes direcly when i write in each node from the Cluster. The Async replication is only for one Schema and the table is a minimum (2 fields). The configuration is in the archiv.zip. Iv you want to have it separate mc.cnf files write me a mail.

Comment by Nirbhay Choubey (Inactive) [ 2016-02-19 ] 

Server version: 10.1.11-MariaDB-1~jessie-log
 
key_buffer_size=134217728
 
read_buffer_size=2097152
 
max_used_connections=4
 
max_threads=102
 
thread_count=6
 
It is possible that mysqld could use up to
 
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 759826 K  bytes of memory
 
Hope that's ok; if not, decrease some variables in the equation.
 
 
 
Thread pointer: 0x0x7f8107ff1008
 
Attempting backtrace. You can use the following information to find out
 
where mysqld died. If you see no messages after this, something went
 
terribly wrong...
 
stack_bottom = 0x7f82b45bd9a8 thread_stack 0x48400
 
(my_addr_resolve failure: fork)
 
mysqld(my_print_stacktrace+0x2e) [0x564cfe8b432e]
 
mysqld(handle_fatal_signal+0x34d) [0x564cfe3f68fd]
 
/lib/x86_64-linux-gnu/libpthread.so.0(+0xf8d0) [0x7f82b724f8d0]
 
mysqld(THD::binlog_set_stmt_begin()+0x28) [0x564cfe4a1348]
 
mysqld(THD::binlog_start_trans_and_stmt()+0x38) [0x564cfe4a13b8]
 
mysqld(THD::binlog_write_table_map(TABLE*, bool, char*)+0x160) [0x564cfe4b2020]
 
mysqld(handler::ha_write_row(unsigned char*)+0x323) [0x564cfe400993]
 
mysqld(rpl_slave_state::record_gtid(THD*, rpl_gtid const*, unsigned long long, bool, bool)+0x3fe) [0x564cfe38378e]
 
mysqld(Xid_log_event::do_apply_event(rpl_group_info*)+0x74) [0x564cfe4b81d4]
 
mysqld(apply_event_and_update_pos(Log_event*, THD*, rpl_group_info*, rpl_parallel_thread*)+0x1e1) [0x564cfe1f9291]
 
mysqld(handle_slave_sql+0x2762) [0x564cfe1fce22]
 
/lib/x86_64-linux-gnu/libpthread.so.0(+0x80a4) [0x7f82b72480a4]
 
/lib/x86_64-linux-gnu/libc.so.6(clone+0x6d) [0x7f82b53f49cd]
 
 
 
Trying to get some variables.
 
Some pointers may be invalid and cause the dump to abort.
 
Query (0x0): is an invalid pointer
 
Connection ID (thread ID): 35850
 
Status: NOT_KILLED

Comment by Bernd Buffen [ 2016-02-19 ]

Nirbhay Choubey - sorry my Question: if have seen your last comment. can you reproduce the crash ?

Comment by Nirbhay Choubey (Inactive) [ 2016-02-19 ]

Bernd Buffen No, I just pulled it from the attached logs.

Comment by Bernd Buffen [ 2016-02-20 ]

-[Nirbhay Choubey] - I you want to have access the real server you can give me your ssh key. You can do / test anything with the Sever. only let me know it you use them that i not test on them.

Comment by Bernd Buffen [ 2016-02-23 ]

-[Nirbhay Choubey]
I have compile it, but without ssl.

i start cluster 1 node 1 : ./bin/mysqld --wsrep-new-cluster --log-error=/tmp/mysqld.err&
i start cluster 2 node 1 : ./bin/mysqld --wsrep-new-cluster --log-error=/tmp/mysqld.err&
i start cluster 1 node 2 : ./bin/mysqld --log-error=/tmp/mysqld.err&

log in
> MariaDB [(none)]> select version();
> ---------------------------
> | version() |
> ---------------------------
> | 10.1.12-MariaDB-debug-log |
> ---------------------------
> 1 row in set (0.00 sec)
>
> MariaDB [(none)]>

> use bernd;
> truncate customer;
> select * from customer;

and start this one time on cluster 2 Node 1
insert into customer select NULL,now() -interval seq SECOND from seq_1_to_120000;

after this i start it on all 3 Nodes and node 2 cluster 1 crashes.

the log is in g1n2.err

Hope it helps

Bernd

Comment by Bernd Buffen [ 2016-02-23 ]

here also a backtrace when i start mysql in gdb (cluster 1 Node 1)

 
 
Program received signal SIGABRT, Aborted.
 
[Switching to Thread 0x7ffff7f9db00 (LWP 2509)]
 
0x00007ffff6799067 in __GI_raise (sig=sig@entry=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:56
 
56	../nptl/sysdeps/unix/sysv/linux/raise.c: No such file or directory.
 
(gdb)
 
(gdb)
 
(gdb)
 
(gdb) bt
 
#0  0x00007ffff6799067 in __GI_raise (sig=sig@entry=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:56
 
#1  0x00007ffff679a448 in __GI_abort () at abort.c:89
 
#2  0x0000555555fa3686 in trx_sys_update_wsrep_checkpoint (xid=0x7fff85354810, sys_header=0x7fffd3fdc026 "", mtr=0x7ffff7f9ad60)
 
    at /root/server/storage/xtradb/trx/trx0sys.cc:358
 
#3  0x0000555555faa492 in trx_write_serialisation_history (trx=0x7fff85354678, mtr=0x7ffff7f9ad60) at /root/server/storage/xtradb/trx/trx0trx.cc:1243
 
#4  0x0000555555fab4a8 in trx_commit_low (trx=0x7fff85354678, mtr=0x7ffff7f9ad60) at /root/server/storage/xtradb/trx/trx0trx.cc:1626
 
#5  0x0000555555fab56c in trx_commit (trx=0x7fff85354678) at /root/server/storage/xtradb/trx/trx0trx.cc:1673
 
#6  0x0000555555f9eb61 in trx_rollback_finish (trx=0x7fff85354678) at /root/server/storage/xtradb/trx/trx0roll.cc:1338
 
#7  0x0000555555f9c370 in trx_rollback_to_savepoint_low (trx=0x7fff85354678, savept=0x0) at /root/server/storage/xtradb/trx/trx0roll.cc:125
 
#8  0x0000555555f9c6a9 in trx_rollback_for_mysql_low (trx=0x7fff85354678) at /root/server/storage/xtradb/trx/trx0roll.cc:180
 
#9  0x0000555555f9c9d3 in trx_rollback_for_mysql (trx=0x7fff85354678) at /root/server/storage/xtradb/trx/trx0roll.cc:211
 
#10 0x0000555555e11216 in innobase_rollback (hton=0x7ffff5c256f0, thd=0x7fffe5c16070, rollback_trx=true)
 
    at /root/server/storage/xtradb/handler/ha_innodb.cc:4716
 
#11 0x0000555555c5c384 in ha_rollback_trans (thd=0x7fffe5c16070, all=true) at /root/server/sql/handler.cc:1658
 
#12 0x0000555555b68ca5 in trans_rollback (thd=0x7fffe5c16070) at /root/server/sql/transaction.cc:343
 
#13 0x0000555555bdda0a in wsrep_rollback (thd=0x7fffe5c16070, global_seqno=8) at /root/server/sql/wsrep_applier.cc:327
 
#14 0x0000555555bddb19 in wsrep_commit_cb (ctx=0x7fffe5c16070, flags=1, meta=0x7ffff7f9bd20, exit=0x7ffff7f9b830, commit=false)
 
    at /root/server/sql/wsrep_applier.cc:356
 
---Type <return> to continue, or q <return> to quit---
 
#15 0x00007ffff4578537 in apply_trx_ws (recv_ctx=recv_ctx@entry=0x7fffe5c16070,
 
    apply_cb=0x555555bdd473 <wsrep_apply_cb(void*, void const*, unsigned long, unsigned int, wsrep_trx_meta const*)>,
 
    commit_cb=0x555555bdda82 <wsrep_commit_cb(void*, unsigned int, wsrep_trx_meta const*, bool*, bool)>, trx=..., meta=...)
 
    at galera/src/replicator_smm.cpp:68
 
#16 0x00007ffff457b18b in galera::ReplicatorSMM::apply_trx (this=this@entry=0x7ffff5f70e00, recv_ctx=recv_ctx@entry=0x7fffe5c16070,
 
    trx=trx@entry=0x7fffe5ca0800) at galera/src/replicator_smm.cpp:433
 
#17 0x00007ffff457e6ee in galera::ReplicatorSMM::process_trx (this=0x7ffff5f70e00, recv_ctx=0x7fffe5c16070, trx=0x7fffe5ca0800)
 
    at galera/src/replicator_smm.cpp:1224
 
#18 0x00007ffff4557f88 in galera::GcsActionSource::dispatch (this=this@entry=0x7ffff5f71450, recv_ctx=recv_ctx@entry=0x7fffe5c16070, act=...,
 
    exit_loop=@0x7ffff7f9c68c: false) at galera/src/gcs_action_source.cpp:116
 
#19 0x00007ffff4559d62 in galera::GcsActionSource::process (this=0x7ffff5f71450, recv_ctx=0x7fffe5c16070, exit_loop=@0x7ffff7f9c68c: false)
 
    at galera/src/gcs_action_source.cpp:181
 
#20 0x00007ffff457ec13 in galera::ReplicatorSMM::async_recv (this=0x7ffff5f70e00, recv_ctx=0x7fffe5c16070) at galera/src/replicator_smm.cpp:355
 
#21 0x00007ffff4592108 in galera_recv (gh=<optimized out>, recv_ctx=<optimized out>) at galera/src/wsrep_provider.cpp:239
 
#22 0x0000555555bdf249 in wsrep_replication_process (thd=0x7fffe5c16070) at /root/server/sql/wsrep_thd.cc:315
 
#23 0x0000555555bcefdc in start_wsrep_THD (arg=0x555555bdf182 <wsrep_replication_process(THD*)>) at /root/server/sql/wsrep_mysqld.cc:1823
 
#24 0x00007ffff7bc70a4 in start_thread (arg=0x7ffff7f9db00) at pthread_create.c:309
 
#25 0x00007ffff684c87d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

Generated at Thu Feb 08 07:35:15 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.