[MDEV-18317] Mariadb Crashed due to BF-BF X lock conflict Created: 2019-01-21  Updated: 2020-01-13  Resolved: 2019-04-15

Status: Closed
Project: MariaDB Server
Component/s: Galera
Affects Version/s: 10.2.19
Fix Version/s: N/A

Type: Bug Priority: Critical
Reporter: TAO ZHOU Assignee: Jan Lindström (Inactive)
Resolution: Incomplete Votes: 0
Labels: need_feedback
Environment:

FreeBSD11.2

Issue Links:
Relates
relates to MDEV-21466 BF-BF X lock conflict, replication fa... Closed

 Description   

Configuration of 3 nodes(two nodes + one arbitrator).

Error logs as below.

2019-01-19 20:11:51 50738248448 [Note] WSREP: EX-EX trx conflict for key (2,FLAT8)8632c156 4e7d3bb4: source: 915c701c-13cd-11e9-9ddd-eef50d21aff0 version: 4 local: 1 state: MUST_ABORT flags: 1 conn_id: 388433 trx_id: 107678074869 seqnos (l: 6356516, g: 1404623292, s: 1404623290, d: 1404606848, ts: 889004531810575) <---> source: bd601f60-123e-11e9-b2df-cb043767660f version: 4 local: 0 state: APPLYING flags: 1 conn_id: 377280 trx_id: 107678071877 seqnos (l: 6356515, g: 1404623291, s: 1404623290, d: 1404622944, ts: 14095281738886391)
 
2019-01-19 20:14:26 50740154368 [Note] InnoDB: BF-BF X lock conflict,mode: 1027 supremum: 0
 
2019-01-19 20:14:26 50740154368 [Note] InnoDB: conflicts states: my 0 locked 0
 
RECORD LOCKS space id 324921 page no 1570 n bits 240 index PRIMARY of table `w2live_college`.`WebURLAlias` trx id 107678078543 lock_mode X locks rec but not gap
 
Record lock, heap no 100 PHYSICAL RECORD: n_fields 12; compact format; info bits 32
 
 0: len 8; hex 8000000000f28c5e; asc        ^;;
 
 1: len 6; hex 0019121d164f; asc      O;;
 
 2: len 7; hex 030003001b2eca; asc      . ;;
 
 3: len 8; hex 8000000000134d64; asc       Md;;
 
 4: len 5; hex 999f92c24a; asc     J;;
 
 5: len 5; hex 999f92c24a; asc     J;;
 
 6: len 8; hex 2f70726976616379; asc /privacy;;
 
 7: len 1; hex 01; asc  ;;
 
 8: len 8; hex 800000000000b668; asc        h;;
 
 9: SQL NULL;
 
 10: SQL NULL;
 
 11: SQL NULL;
 
 
 
Record lock, heap no 112 PHYSICAL RECORD: n_fields 12; compact format; info bits 32
 
 0: len 8; hex 8000000000f28c76; asc        v;;
 
 1: len 6; hex 0019121d164f; asc      O;;
 
 2: len 7; hex 030003001b2d8b; asc      - ;;
 
 3: len 8; hex 8000000000134d60; asc       M`;;
 
 4: len 5; hex 999ede8bb0; asc      ;;
 
 5: len 5; hex 999ede8bb0; asc      ;;
 
 6: len 30; hex 2f61727469636c65732f6c6f73652d796f75722d666561722d6f662d6772; asc /articles/lose-your-fear-of-gr; (total 35 bytes);
 
 7: len 1; hex 01; asc  ;;
 
 8: len 8; hex 800000000000b668; asc        h;;
 
 9: SQL NULL;
 
 10: SQL NULL;
 
 11: SQL NULL;
 
 
 
Record lock, heap no 113 PHYSICAL RECORD: n_fields 12; compact format; info bits 32
 
 0: len 8; hex 8000000000f28c78; asc        x;;
 
 1: len 6; hex 0019121d164f; asc      O;;
 
 2: len 7; hex 030003001b2b16; asc      + ;;
 
 3: len 8; hex 8000000000134d56; asc       MV;;
 
 4: len 5; hex 99a08d016d; asc     m;;
 
 5: len 5; hex 99a08d016d; asc     m;;
 
 6: len 8; hex 2f73756767657374; asc /suggest;;
 
 7: len 1; hex 00; asc  ;;
 
 8: len 8; hex 800000000000b668; asc        h;;
 
 9: SQL NULL;
 
 10: SQL NULL;
 
 11: SQL NULL;
 
 
 
Record lock, heap no 121 PHYSICAL RECORD: n_fields 12; compact format; info bits 32
 
 0: len 8; hex 8000000000f28c88; asc         ;;
 
 1: len 6; hex 0019121d164f; asc      O;;
 
 2: len 7; hex 030003001b297e; asc      )~;;
 
 3: len 8; hex 8000000000134d50; asc       MP;;
 
 4: len 5; hex 999f11212e; asc    !.;;
 
 5: len 5; hex 999f11212e; asc    !.;;
 
 6: len 30; hex 2f61727469636c65732f646f6e616c642d7472756d702d616e642d746865; asc /articles/donald-trump-and-the; (total 49 bytes);
 
 7: len 1; hex 01; asc  ;;
 
 8: len 8; hex 800000000000b668; asc        h;;
 
 9: SQL NULL;
 
 10: SQL NULL;
 
 11: SQL NULL;
 
 
 
Record lock, heap no 133 PHYSICAL RECORD: n_fields 12; compact format; info bits 32
 
 0: len 8; hex 8000000000f28ca0; asc         ;;
 
 1: len 6; hex 0019121d164f; asc      O;;
 
 2: len 7; hex 030003001b2e39; asc      .9;;
 
 3: len 8; hex 8000000000134d62; asc       Mb;;
 
 4: len 5; hex 999d51178a; asc   Q  ;;
 
 5: len 5; hex 999d51178a; asc   Q  ;;
 
 6: len 6; hex 2f6c6567616c; asc /legal;;
 
 7: len 1; hex 01; asc  ;;
 
 8: len 8; hex 800000000000b668; asc        h;;
 
 9: SQL NULL;
 
 10: SQL NULL;
 
 11: SQL NULL;
 
 
 
Record lock, heap no 151 PHYSICAL RECORD: n_fields 12; compact format; info bits 32
 
 0: len 8; hex 8000000000f28cc4; asc         ;;
 
 1: len 6; hex 0019121d164f; asc      O;;
 
 2: len 7; hex 030003001b2a3a; asc      *:;;
 
 3: len 8; hex 8000000000134d52; asc       MR;;
 
 4: len 5; hex 999dcccd78; asc     x;;
 
 5: len 5; hex 999dcccd78; asc     x;;
 
 6: len 8; hex 2f73656e696f7273; asc /seniors;;
 
 7: len 1; hex 01; asc  ;;
 
 8: len 8; hex 800000000000b668; asc        h;;
 
 9: SQL NULL;
 
 10: SQL NULL;
 
 11: SQL NULL;
 
 
 
Record lock, heap no 165 PHYSICAL RECORD: n_fields 12; compact format; info bits 32
 
 0: len 8; hex 8000000000f28ce0; asc         ;;
 
 1: len 6; hex 0019121d164f; asc      O;;
 
 2: len 7; hex 030003001b2c3b; asc      ,;;;
 
 3: len 8; hex 8000000000134d5c; asc       M\;;
 
 4: len 5; hex 999f6ea3eb; asc   n  ;;
 
 5: len 5; hex 999f6ea3eb; asc   n  ;;
 
 6: len 30; hex 2f61727469636c65732f69662d77652d63616e742d77696e2d7468656e2d; asc /articles/if-we-cant-win-then-; (total 50 bytes);
 
 7: len 1; hex 01; asc  ;;
 
 8: len 8; hex 800000000000b668; asc        h;;
 
 9: SQL NULL;
 
 10: SQL NULL;
 
 11: SQL NULL;
 
 
 
2019-01-19 20:14:26 50740154368 [Note] InnoDB:  SQL1: DELETE FROM WebURLAlias where webSiteVersionId = 46696r�B\
 
2019-01-19 20:14:26 50740154368 [Note] InnoDB:  SQL2: DELETE FROM WebNodeType where webSiteVersionId = 46696r�B\
 
2019-01-19 20:14:26 50740154368 [ERROR] mysqld: Can't find record in 'WebURLAlias'
 
2019-01-19 20:14:26 50740154368 [ERROR] Slave SQL: Could not execute Delete_rows_v1 event on table w2live_college.WebURLAlias; Can't find record in 'WebURLAlias', Error_code: 1032; handler error HA_ERR_KEY_NOT_FOUND; the event's master log FIRST, end_log_pos 24701, Internal MariaDB error code: 1032
 
2019-01-19 20:14:26 50740154368 [Warning] WSREP: RBR event 5 Delete_rows_v1 apply warning: 120, 1404625113
 
2019-01-19 20:14:26 50740154368 [Warning] WSREP: Failed to apply app buffer: seqno: 1404625113, status: 1
 
	 at galera/src/trx_handle.cpp:apply():353
 
Retrying 2th time
 
2019-01-19 20:14:26 50740154368 [Note] InnoDB: BF-BF X lock conflict,mode: 1027 supremum: 0
 
2019-01-19 20:14:26 50740154368 [Note] InnoDB: conflicts states: my 0 locked 0
 
RECORD LOCKS space id 324921 page no 1570 n bits 240 index PRIMARY of table `w2live_college`.`WebURLAlias` trx id 107678078543 lock_mode X locks rec but not gap
 
Record lock, heap no 86 PHYSICAL RECORD: n_fields 12; compact format; info bits 32
 
 0: len 8; hex 8000000000f28c42; asc        B;;
 
 1: len 6; hex 0019121d164f; asc      O;;
 
 2: len 7; hex 030003001b3c48; asc      <H;;
 
 3: len 8; hex 8000000000134dac; asc       M ;;
 
 4: len 5; hex 999f7a7768; asc   zwh;;
 
 5: len 5; hex 999f7a7768; asc   zwh;;
 
 6: len 11; hex 2f66726565726570656174; asc /freerepeat;;
 
 7: len 1; hex 01; asc  ;;
 
 8: len 8; hex 800000000000b668; asc        h;;
 
 9: SQL NULL;
 
 10: SQL NULL;
 
 11: SQL NULL;
 
 
 
Record lock, heap no 100 PHYSICAL RECORD: n_fields 12; compact format; info bits 32
 
 0: len 8; hex 8000000000f28c5e; asc        ^;;
 
 1: len 6; hex 0019121d164f; asc      O;;
 
 2: len 7; hex 030003001b2eca; asc      . ;;
 
 3: len 8; hex 8000000000134d64; asc       Md;;
 
 4: len 5; hex 999f92c24a; asc     J;;
 
 5: len 5; hex 999f92c24a; asc     J;;
 
 6: len 8; hex 2f70726976616379; asc /privacy;;
 
 7: len 1; hex 01; asc  ;;
 
 8: len 8; hex 800000000000b668; asc        h;;
 
 9: SQL NULL;
 
 10: SQL NULL;
 
 11: SQL NULL;
 
 
 
Record lock, heap no 112 PHYSICAL RECORD: n_fields 12; compact format; info bits 32
 
 0: len 8; hex 8000000000f28c76; asc        v;;
 
 1: len 6; hex 0019121d164f; asc      O;;
 
 2: len 7; hex 030003001b2d8b; asc      - ;;
 
 3: len 8; hex 8000000000134d60; asc       M`;;
 
 4: len 5; hex 999ede8bb0; asc      ;;
 
 5: len 5; hex 999ede8bb0; asc      ;;
 
 6: len 30; hex 2f61727469636c65732f6c6f73652d796f75722d666561722d6f662d6772; asc /articles/lose-your-fear-of-gr; (total 35 bytes);
 
 7: len 1; hex 01; asc  ;;
 
 8: len 8; hex 800000000000b668; asc        h;;
 
 9: SQL NULL;
 
 10: SQL NULL;
 
 11: SQL NULL;
 
 
 
Record lock, heap no 113 PHYSICAL RECORD: n_fields 12; compact format; info bits 32
 
 0: len 8; hex 8000000000f28c78; asc        x;;
 
 1: len 6; hex 0019121d164f; asc      O;;
 
 2: len 7; hex 030003001b2b16; asc      + ;;
 
 3: len 8; hex 8000000000134d56; asc       MV;;
 
 4: len 5; hex 99a08d016d; asc     m;;
 
 5: len 5; hex 99a08d016d; asc     m;;
 
 6: len 8; hex 2f73756767657374; asc /suggest;;
 
 7: len 1; hex 00; asc  ;;
 
 8: len 8; hex 800000000000b668; asc        h;;
 
 9: SQL NULL;
 
 10: SQL NULL;
 
 11: SQL NULL;
 
 
 
Record lock, heap no 121 PHYSICAL RECORD: n_fields 12; compact format; info bits 32
 
 0: len 8; hex 8000000000f28c88; asc         ;;
 
 1: len 6; hex 0019121d164f; asc      O;;
 
 2: len 7; hex 030003001b297e; asc      )~;;
 
 3: len 8; hex 8000000000134d50; asc       MP;;
 
 4: len 5; hex 999f11212e; asc    !.;;
 
 5: len 5; hex 999f11212e; asc    !.;;
 
 6: len 30; hex 2f61727469636c65732f646f6e616c642d7472756d702d616e642d746865; asc /articles/donald-trump-and-the; (total 49 bytes);
 
 7: len 1; hex 01; asc  ;;
 
 8: len 8; hex 800000000000b668; asc        h;;
 
 9: SQL NULL;
 
 10: SQL NULL;
 
 11: SQL NULL;
 
 
 
Record lock, heap no 126 PHYSICAL RECORD: n_fields 12; compact format; info bits 32
 
 0: len 8; hex 8000000000f28c92; asc         ;;
 
 1: len 6; hex 0019121d164f; asc      O;;
 
 2: len 7; hex 030003001b3a4f; asc      :O;;
 
 3: len 8; hex 8000000000134da0; asc       M ;;
 
 4: len 5; hex 99a0a0f499; asc      ;;
 
 5: len 5; hex 99a0a0f499; asc      ;;
 
 6: len 4; hex 2f6e6670; asc /nfp;;
 
 7: len 1; hex 01; asc  ;;
 
 8: len 8; hex 800000000000b668; asc        h;;
 
 9: SQL NULL;
 
 10: SQL NULL;
 
 11: SQL NULL;
 
 
 
Record lock, heap no 133 PHYSICAL RECORD: n_fields 12; compact format; info bits 32
 
 0: len 8; hex 8000000000f28ca0; asc         ;;
 
 1: len 6; hex 0019121d164f; asc      O;;
 
 2: len 7; hex 030003001b2e39; asc      .9;;
 
 3: len 8; hex 8000000000134d62; asc       Mb;;
 
 4: len 5; hex 999d51178a; asc   Q  ;;
 
 5: len 5; hex 999d51178a; asc   Q  ;;
 
 6: len 6; hex 2f6c6567616c; asc /legal;;
 
 7: len 1; hex 01; asc  ;;
 
 8: len 8; hex 800000000000b668; asc        h;;
 
 9: SQL NULL;
 
 10: SQL NULL;
 
 11: SQL NULL;
 
 
 
Record lock, heap no 138 PHYSICAL RECORD: n_fields 12; compact format; info bits 32
 
 0: len 8; hex 8000000000f28caa; asc         ;;
 
 1: len 6; hex 0019121d164f; asc      O;;
 
 2: len 7; hex 030003001b34ac; asc      4 ;;
 
 3: len 8; hex 8000000000134d84; asc       M ;;
 
 4: len 5; hex 99a0671664; asc   g d;;
 
 5: len 5; hex 99a0671664; asc   g d;;
 
 6: len 30; hex 2f61727469636c65732f7468652d62696c6c696f6e2d646f6c6c61722d62; asc /articles/the-billion-dollar-b; (total 64 bytes);
 
 7: len 1; hex 01; asc  ;;
 
 8: len 8; hex 800000000000b668; asc        h;;
 
 9: SQL NULL;
 
 10: SQL NULL;
 
 11: SQL NULL;
 
 
 
Record lock, heap no 141 PHYSICAL RECORD: n_fields 12; compact format; info bits 32
 
 0: len 8; hex 8000000000f28cb0; asc         ;;
 
 1: len 6; hex 0019121d164f; asc      O;;
 
 2: len 7; hex 030003001b3739; asc      79;;
 
 3: len 8; hex 8000000000134d90; asc       M ;;
 
 4: len 5; hex 99a0ac7745; asc    wE;;
 
 5: len 5; hex 99a0ac7745; asc    wE;;
 
 6: len 14; hex 2f636f72706f7261746570617373; asc /corporatepass;;
 
 7: len 1; hex 01; asc  ;;
 
 8: len 8; hex 800000000000b668; asc        h;;
 
 9: SQL NULL;
 
 10: SQL NULL;
 
 11: SQL NULL;
 
 
 
Record lock, heap no 147 PHYSICAL RECORD: n_fields 12; compact format; info bits 32
 
 0: len 8; hex 8000000000f28cbc; asc         ;;
 
 1: len 6; hex 0019121d164f; asc      O;;
 
 2: len 7; hex 030003001b33d3; asc      3 ;;
 
 3: len 8; hex 8000000000134d80; asc       M ;;
 
 4: len 5; hex 999f39311a; asc   91 ;;
 
 5: len 5; hex 999f39311a; asc   91 ;;
 
 6: len 6; hex 2f6461746573; asc /dates;;
 
 7: len 1; hex 01; asc  ;;
 
 8: len 8; hex 800000000000b668; asc        h;;
 
 9: SQL NULL;
 
 10: SQL NULL;
 
 11: SQL NULL;
 
 
 
Record lock, heap no 151 PHYSICAL RECORD: n_fields 12; compact format; info bits 32
 
 0: len 8; hex 8000000000f28cc4; asc         ;;
 
 1: len 6; hex 0019121d164f; asc      O;;
 
 2: len 7; hex 030003001b2a3a; asc      *:;;
 
 3: len 8; hex 8000000000134d52; asc       MR;;
 
 4: len 5; hex 999dcccd78; asc     x;;
 
 5: len 5; hex 999dcccd78; asc     x;;
 
 6: len 8; hex 2f73656e696f7273; asc /seniors;;
 
 7: len 1; hex 01; asc  ;;
 
 8: len 8; hex 800000000000b668; asc        h;;
 
 9: SQL NULL;
 
 10: SQL NULL;
 
 11: SQL NULL;
 
 
 
Record lock, heap no 159 PHYSICAL RECORD: n_fields 12; compact format; info bits 32
 
 0: len 8; hex 8000000000f28cd4; asc         ;;
 
 1: len 6; hex 0019121d164f; asc      O;;
 
 2: len 7; hex 030003001b3bb8; asc      ; ;;
 
 3: len 8; hex 8000000000134daa; asc       M ;;
 
 4: len 5; hex 99a10aee14; asc      ;;
 
 5: len 5; hex 99a10aee14; asc      ;;
 
 6: len 5; hex 2f6c696e6b; asc /link;;
 
 7: len 1; hex 01; asc  ;;
 
 8: len 8; hex 800000000000b668; asc        h;;
 
 9: SQL NULL;
 
 10: SQL NULL;
 
 11: SQL NULL;
 
 
 
Record lock, heap no 161 PHYSICAL RECORD: n_fields 12; compact format; info bits 32
 
 0: len 8; hex 8000000000f28cd8; asc         ;;
 
 1: len 6; hex 0019121d164f; asc      O;;
 
 2: len 7; hex 030003001b3608; asc      6 ;;
 
 3: len 8; hex 8000000000134d8a; asc       M ;;
 
 4: len 5; hex 99a086b575; asc     u;;
 
 5: len 5; hex 99a086b575; asc     u;;
 
 6: len 22; hex 2f6469726563746f72792f696e7374727563746f7273; asc /directory/instructors;;
 
 7: len 1; hex 01; asc  ;;
 
 8: len 8; hex 800000000000b668; asc        h;;
 
 9: SQL NULL;
 
 10: SQL NULL;
 
 11: SQL NULL;
 
 
 
Record lock, heap no 162 PHYSICAL RECORD: n_fields 12; compact format; info bits 32
 
 0: len 8; hex 8000000000f28cda; asc         ;;
 
 1: len 6; hex 0019121d164f; asc      O;;
 
 2: len 7; hex 030003001b3282; asc      2 ;;
 
 3: len 8; hex 8000000000134d7a; asc       Mz;;
 
 4: len 5; hex 999f6e6bbb; asc   nk ;;
 
 5: len 5; hex 999f6e6bbb; asc   nk ;;
 
 6: len 30; hex 2f61727469636c65732f7468652d6e65776c792d72656c65617365642d6a; asc /articles/the-newly-released-j; (total 52 bytes);
 
 7: len 1; hex 01; asc  ;;
 
 8: len 8; hex 800000000000b668; asc        h;;
 
 9: SQL NULL;
 
 10: SQL NULL;
 
 11: SQL NULL;
 
 
 
Record lock, heap no 165 PHYSICAL RECORD: n_fields 12; compact format; info bits 32
 
 0: len 8; hex 8000000000f28ce0; asc         ;;
 
 1: len 6; hex 0019121d164f; asc      O;;
 
 2: len 7; hex 030003001b2c3b; asc      ,;;;
 
 3: len 8; hex 8000000000134d5c; asc       M\;;
 
 4: len 5; hex 999f6ea3eb; asc   n  ;;
 
 5: len 5; hex 999f6ea3eb; asc   n  ;;
 
 6: len 30; hex 2f61727469636c65732f69662d77652d63616e742d77696e2d7468656e2d; asc /articles/if-we-cant-win-then-; (total 50 bytes);
 
 7: len 1; hex 01; asc  ;;
 
 8: len 8; hex 800000000000b668; asc        h;;
 
 9: SQL NULL;
 
 10: SQL NULL;
 
 11: SQL NULL;
 
 
 
2019-01-19 20:14:27 50740154368 [Note] InnoDB:  SQL1: DELETE FROM WebURLAlias where webSiteVersionId = 46696r�B\
 
190119 20:14:27 [ERROR] mysqld got signal 11 ;
 
This could be because you hit a bug. It is also possible that this binary
 
or one of the libraries it was linked against is corrupt, improperly built,
 
or misconfigured. This error can also be caused by malfunctioning hardware.
 
 
 
To report this bug, see https://mariadb.com/kb/en/reporting-bugs
 
 
 
We will try our best to scrape up some info that will hopefully help
 
diagnose the problem, but since we have already crashed,
 
something is definitely wrong and this may fail.
 
 
 
Server version: 10.2.19-MariaDB-log
 
key_buffer_size=67108864
 
read_buffer_size=131072
 
max_used_connections=118
 
max_threads=2002
 
thread_count=161
 
It is possible that mysqld could use up to
 
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 33164362 K  bytes of memory
 
Hope that's ok; if not, decrease some variables in the equation.
 
 
 
Thread pointer: 0x83a61f448
 
Attempting backtrace. You can use the following information to find out
 
where mysqld died. If you see no messages after this, something went
 
terribly wrong...
 
stack_bottom = 0x7fffdb1e1b88 thread_stack 0x49000
 
0xc6287e <my_print_stacktrace+0x2e> at /usr/local/libexec/mysqld
 
0x7926a4 <handle_fatal_signal+0x294> at /usr/local/libexec/mysqld
 
0x8032f4946 <pthread_sigmask+0x536> at /lib/libthr.so.3
 
0x8032f3eb2 <pthread_getspecific+0xe12> at /lib/libthr.so.3
 
 
 
Trying to get some variables.
 
Some pointers may be invalid and cause the dump to abort.
 
Query (0x852edd803): DELETE FROM WebURLAlias where webSiteVersionId = 46696
 
 
 
Connection ID (thread ID): 39
 
Status: NOT_KILLED
 
 
 
Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=off,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on
 
 
 
The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
 
information that should help you find out what is causing the crash.



 Comments   
Comment by TAO ZHOU [ 2019-01-21 ]

Can someone please investigate this.
This problem is critical. It makes galera not usable at all.

Comment by TAO ZHOU [ 2019-01-22 ]

It just happened again. And this time, I cannot start the second node. It crashes immediately after started with exactly the same error log.

error logs:

Version: '10.2.19-MariaDB-log'  socket: '/tmp/mysql.sock'  port: 3306  FreeBSD Ports
 
2019-01-22 11:11:59 50726265856 [Warning] WSREP: no THD for trx: 107680793867
 
2019-01-22 11:11:59 50731757824 [Warning] WSREP: no THD for trx: 107680793866
 
2019-01-22 11:11:59 50726264576 [Note] InnoDB: BF-BF X lock conflict,mode: 1027 supremum: 0
 
2019-01-22 11:11:59 50726264576 [Note] InnoDB: conflicts states: my 0 locked 0
 
RECORD LOCKS space id 324921 page no 2092 n bits 240 index PRIMARY of table `w2live_college`.`WebURLAlias` trx id 107680795987 lock_mode X locks rec but not gap
 
Record lock, heap no 5 PHYSICAL RECORD: n_fields 12; compact format; info bits 32
 
 0: len 8; hex 8000000000f2e5b4; asc         ;;
 
 1: len 6; hex 001912468d53; asc    F S;;
 
 2: len 7; hex 3d00000038325e; asc =   82^;;
 
 3: len 8; hex 80000000001354ac; asc       T ;;
 
 4: len 5; hex 999f38d042; asc   8 B;;
 
 5: len 5; hex 999f38d042; asc   8 B;;
 
 6: len 30; hex 2f61727469636c65732f737461747565732d696e2d616d65726963612d77; asc /articles/statues-in-america-w; (total 52 bytes);
 
 7: len 1; hex 01; asc  ;;
 
 8: len 8; hex 800000000000b694; asc         ;;
 
 9: SQL NULL;
 
 10: SQL NULL;
 
 11: SQL NULL;
 
 
 
Record lock, heap no 7 PHYSICAL RECORD: n_fields 12; compact format; info bits 32
 
 0: len 8; hex 8000000000f2e5b8; asc         ;;
 
 1: len 6; hex 001912468d53; asc    F S;;
 
 2: len 7; hex 3d0000003836d2; asc =   86 ;;
 
 3: len 8; hex 80000000001354ba; asc       T ;;
 
 4: len 5; hex 99a0a8cb00; asc      ;;
 
 5: len 5; hex 99a0a8cb00; asc      ;;
 
 6: len 21; hex 2f696e7374727563746f72732f68616e64626f6f6b; asc /instructors/handbook;;
 
 7: len 1; hex 01; asc  ;;
 
 8: len 8; hex 800000000000b694; asc         ;;
 
 9: SQL NULL;
 
 10: SQL NULL;
 
 11: SQL NULL;
 
 
 
Record lock, heap no 17 PHYSICAL RECORD: n_fields 12; compact format; info bits 32
 
 0: len 8; hex 8000000000f2e5cc; asc         ;;
 
 1: len 6; hex 001912468d53; asc    F S;;
 
 2: len 7; hex 3d000000382f45; asc =   8/E;;
 
 3: len 8; hex 80000000001354a2; asc       T ;;
 
 4: len 5; hex 999d51180a; asc   Q  ;;
 
 5: len 5; hex 999d51180a; asc   Q  ;;
 
 6: len 8; hex 2f636f6e74616374; asc /contact;;
 
 7: len 1; hex 01; asc  ;;
 
 8: len 8; hex 800000000000b694; asc         ;;
 
 9: SQL NULL;
 
 10: SQL NULL;
 
 11: SQL NULL;
 
 
 
Record lock, heap no 19 PHYSICAL RECORD: n_fields 12; compact format; info bits 32
 
 0: len 8; hex 8000000000f2e5d0; asc         ;;
 
 1: len 6; hex 001912468d53; asc    F S;;
 
 2: len 7; hex 3d000000382ea5; asc =   8. ;;
 
 3: len 8; hex 80000000001354a0; asc       T ;;
 
 4: len 5; hex 99a06f0408; asc   o  ;;
 
 5: len 5; hex 99a06f0408; asc   o  ;;
 
 6: len 21; hex 2f637573746f6d697365642d736f6c7574696f6e73; asc /customised-solutions;;
 
 7: len 1; hex 01; asc  ;;
 
 8: len 8; hex 800000000000b694; asc         ;;
 
 9: SQL NULL;
 
 10: SQL NULL;
 
 11: SQL NULL;
 
 
 
Record lock, heap no 33 PHYSICAL RECORD: n_fields 12; compact format; info bits 32
 
 0: len 8; hex 8000000000f2e5ec; asc         ;;
 
 1: len 6; hex 001912468d53; asc    F S;;
 
 2: len 7; hex 3d000000383a1b; asc =   8: ;;
 
 3: len 8; hex 80000000001354c6; asc       T ;;
 
 4: len 5; hex 999dccd4da; asc      ;;
 
 5: len 5; hex 999dccd4da; asc      ;;
 
 6: len 6; hex 2f7374616666; asc /staff;;
 
 7: len 1; hex 01; asc  ;;
 
 8: len 8; hex 800000000000b694; asc         ;;
 
 9: SQL NULL;
 
 10: SQL NULL;
 
 11: SQL NULL;
 
 
 
Record lock, heap no 54 PHYSICAL RECORD: n_fields 12; compact format; info bits 32
 
 0: len 8; hex 8000000000f2e616; asc         ;;
 
 1: len 6; hex 001912468d53; asc    F S;;
 
 2: len 7; hex 3d000000383446; asc =   84F;;
 
 3: len 8; hex 80000000001354b2; asc       T ;;
 
 4: len 5; hex 999e508d88; asc   P  ;;
 
 5: len 5; hex 999e508d88; asc   P  ;;
 
 6: len 8; hex 2f7061796d656e74; asc /payment;;
 
 7: len 1; hex 01; asc  ;;
 
 8: len 8; hex 800000000000b694; asc         ;;
 
 9: SQL NULL;
 
 10: SQL NULL;
 
 11: SQL NULL;
 
 
 
Record lock, heap no 55 PHYSICAL RECORD: n_fields 12; compact format; info bits 32
 
 0: len 8; hex 8000000000f2e618; asc         ;;
 
 1: len 6; hex 001912468d53; asc    F S;;
 
 2: len 7; hex 3d00000038331d; asc =   83 ;;
 
 3: len 8; hex 80000000001354ae; asc       T ;;
 
 4: len 5; hex 999f92be70; asc     p;;
 
 5: len 5; hex 999f92be70; asc     p;;
 
 6: len 10; hex 2f636f70797269676874; asc /copyright;;
 
 7: len 1; hex 01; asc  ;;
 
 8: len 8; hex 800000000000b694; asc         ;;
 
 9: SQL NULL;
 
 10: SQL NULL;
 
 11: SQL NULL;
 
 
 
Record lock, heap no 65 PHYSICAL RECORD: n_fields 12; compact format; info bits 32
 
 0: len 8; hex 8000000000f2e62c; asc        ,;;
 
 1: len 6; hex 001912468d53; asc    F S;;
 
 2: len 7; hex 3d000000382d4b; asc =   8-K;;
 
 3: len 8; hex 800000000013549c; asc       T ;;
 
 4: len 5; hex 999ede8b65; asc     e;;
 
 5: len 5; hex 999ede8b65; asc     e;;
 
 6: len 30; hex 2f61727469636c65732f7768792d65646974696e672d69732d657373656e; asc /articles/why-editing-is-essen; (total 60 bytes);
 
 7: len 1; hex 01; asc  ;;
 
 8: len 8; hex 800000000000b694; asc         ;;
 
 9: SQL NULL;
 
 10: SQL NULL;
 
 11: SQL NULL;
 
 
 
Record lock, heap no 101 PHYSICAL RECORD: n_fields 12; compact format; info bits 32
 
 0: len 8; hex 8000000000f2e674; asc        t;;
 
 1: len 6; hex 001912468d53; asc    F S;;
 
 2: len 7; hex 3d000000383803; asc =   88 ;;
 
 3: len 8; hex 80000000001354c0; asc       T ;;
 
 4: len 5; hex 999d5117cb; asc   Q  ;;
 
 5: len 5; hex 999d5117cb; asc   Q  ;;
 
 6: len 13; hex 2f636f6e6669726d6174696f6e; asc /confirmation;;
 
 7: len 1; hex 01; asc  ;;
 
 8: len 8; hex 800000000000b694; asc         ;;
 
 9: SQL NULL;
 
 10: SQL NULL;
 
 11: SQL NULL;
 
 
 
Record lock, heap no 102 PHYSICAL RECORD: n_fields 12; compact format; info bits 32
 
 0: len 8; hex 8000000000f2e676; asc        v;;
 
 1: len 6; hex 001912468d53; asc    F S;;
 
 2: len 7; hex 3d000000383610; asc =   86 ;;
 
 3: len 8; hex 80000000001354b8; asc       T ;;
 
 4: len 5; hex 99a10abe58; asc     X;;
 
 5: len 5; hex 99a10abe58; asc     X;;
 
 6: len 30; hex 2f61727469636c65732f76616c75652d6372656174696f6e2d696e2d676c; asc /articles/value-creation-in-gl; (total 55 bytes);
 
 7: len 1; hex 01; asc  ;;
 
 8: len 8; hex 800000000000b694; asc         ;;
 
 9: SQL NULL;
 
 10: SQL NULL;
 
 11: SQL NULL;
 
 
 
Record lock, heap no 104 PHYSICAL RECORD: n_fields 12; compact format; info bits 32
 
 0: len 8; hex 8000000000f2e67a; asc        z;;
 
 1: len 6; hex 001912468d53; asc    F S;;
 
 2: len 7; hex 3d000000383569; asc =   85i;;
 
 3: len 8; hex 80000000001354b6; asc       T ;;
 
 4: len 5; hex 99a086b9ce; asc      ;;
 
 5: len 5; hex 99a086b9ce; asc      ;;
 
 6: len 28; hex 2f696e7374727563746f72732f636f757273652d70726f706f73616c; asc /instructors/course-proposal;;
 
 7: len 1; hex 01; asc  ;;
 
 8: len 8; hex 800000000000b694; asc         ;;
 
 9: SQL NULL;
 
 10: SQL NULL;
 
 11: SQL NULL;
 
 
 
Record lock, heap no 118 PHYSICAL RECORD: n_fields 12; compact format; info bits 32
 
 0: len 8; hex 8000000000f2e696; asc         ;;
 
 1: len 6; hex 001912468d53; asc    F S;;
 
 2: len 7; hex 3d000000382e11; asc =   8. ;;
 
 3: len 8; hex 800000000013549e; asc       T ;;
 
 4: len 5; hex 999f393132; asc   912;;
 
 5: len 5; hex 999f393132; asc   912;;
 
 6: len 9; hex 2f7375626a65637473; asc /subjects;;
 
 7: len 1; hex 01; asc  ;;
 
 8: len 8; hex 800000000000b694; asc         ;;
 
 9: SQL NULL;
 
 10: SQL NULL;
 
 11: SQL NULL;
 
 
 
Record lock, heap no 153 PHYSICAL RECORD: n_fields 12; compact format; info bits 32
 
 0: len 8; hex 8000000000f2e6dc; asc         ;;
 
 1: len 6; hex 001912468d53; asc    F S;;
 
 2: len 7; hex 3d00000038296d; asc =   8)m;;
 
 3: len 8; hex 80000000001354bc; asc       T ;;
 
 4: len 5; hex 999ed077f7; asc    w ;;
 
 5: len 5; hex 999ed077f7; asc    w ;;
 
 6: len 10; hex 2f6c6f636174696f6e73; asc /locations;;
 
 7: len 1; hex 01; asc  ;;
 
 8: len 8; hex 800000000000b694; asc         ;;
 
 9: SQL NULL;
 
 10: SQL NULL;
 
 11: SQL NULL;
 
 
 
Record lock, heap no 165 PHYSICAL RECORD: n_fields 12; compact format; info bits 32
 
 0: len 8; hex 8000000000f2e6f4; asc         ;;
 
 1: len 6; hex 001912468d53; asc    F S;;
 
 2: len 7; hex 3d000000382bf9; asc =   8+ ;;
 
 3: len 8; hex 8000000000135498; asc       T ;;
 
 4: len 5; hex 999fc32385; asc    # ;;
 
 5: len 5; hex 999fc32385; asc    # ;;
 
 6: len 30; hex 2f61727469636c65732f776861742d61626f75742d6c6561646572736869; asc /articles/what-about-leadershi; (total 46 bytes);
 
 7: len 1; hex 01; asc  ;;
 
 8: len 8; hex 800000000000b694; asc         ;;
 
 9: SQL NULL;
 
 10: SQL NULL;
 
 11: SQL NULL;
 
 
 
2019-01-22 11:11:59 50726264576 [Note] InnoDB:  SQL1: DELETE FROM WebURLAlias where webSiteVersionId = 46740��E\
 
2019-01-22 11:11:59 50726264576 [Note] InnoDB:  SQL2: DELETE FROM WebNodeType where webSiteVersionId = 46740��E\
 
2019-01-22 11:11:59 50726264576 [ERROR] mysqld: Can't find record in 'WebURLAlias'
 
2019-01-22 11:11:59 50726264576 [ERROR] Slave SQL: Could not execute Delete_rows_v1 event on table w2live_college.WebURLAlias; Can't find record in 'WebURLAlias', Error_code: 1032; handler error HA_ERR_KEY_NOT_FOUND; the event's master log FIRST, end_log_pos 8317, Internal MariaDB error code: 1032
 
2019-01-22 11:11:59 50726264576 [Warning] WSREP: RBR event 3 Delete_rows_v1 apply warning: 120, 1405961887
 
2019-01-22 11:11:59 50726264576 [Warning] WSREP: Failed to apply app buffer: seqno: 1405961887, status: 1
 
	 at galera/src/trx_handle.cpp:apply():353
 
Retrying 2th time
 
2019-01-22 11:11:59 50726264576 [Note] InnoDB: BF-BF X lock conflict,mode: 1027 supremum: 0
 
2019-01-22 11:11:59 50726264576 [Note] InnoDB: conflicts states: my 0 locked 0
 
RECORD LOCKS space id 324921 page no 2092 n bits 240 index PRIMARY of table `w2live_college`.`WebURLAlias` trx id 107680795987 lock_mode X locks rec but not gap
 
Record lock, heap no 5 PHYSICAL RECORD: n_fields 12; compact format; info bits 32
 
 0: len 8; hex 8000000000f2e5b4; asc         ;;
 
 1: len 6; hex 001912468d53; asc    F S;;
 
 2: len 7; hex 3d00000038325e; asc =   82^;;
 
 3: len 8; hex 80000000001354ac; asc       T ;;
 
 4: len 5; hex 999f38d042; asc   8 B;;
 
 5: len 5; hex 999f38d042; asc   8 B;;
 
 6: len 30; hex 2f61727469636c65732f737461747565732d696e2d616d65726963612d77; asc /articles/statues-in-america-w; (total 52 bytes);
 
 7: len 1; hex 01; asc  ;;
 
 8: len 8; hex 800000000000b694; asc         ;;
 
 9: SQL NULL;
 
 10: SQL NULL;
 
 11: SQL NULL;
 
 
 
Record lock, heap no 7 PHYSICAL RECORD: n_fields 12; compact format; info bits 32
 
 0: len 8; hex 8000000000f2e5b8; asc         ;;
 
 1: len 6; hex 001912468d53; asc    F S;;
 
 2: len 7; hex 3d0000003836d2; asc =   86 ;;
 
 3: len 8; hex 80000000001354ba; asc       T ;;
 
 4: len 5; hex 99a0a8cb00; asc      ;;
 
 5: len 5; hex 99a0a8cb00; asc      ;;
 
 6: len 21; hex 2f696e7374727563746f72732f68616e64626f6f6b; asc /instructors/handbook;;
 
 7: len 1; hex 01; asc  ;;
 
 8: len 8; hex 800000000000b694; asc         ;;
 
 9: SQL NULL;
 
 10: SQL NULL;
 
 11: SQL NULL;
 
 
 
Record lock, heap no 17 PHYSICAL RECORD: n_fields 12; compact format; info bits 32
 
 0: len 8; hex 8000000000f2e5cc; asc         ;;
 
 1: len 6; hex 001912468d53; asc    F S;;
 
 2: len 7; hex 3d000000382f45; asc =   8/E;;
 
 3: len 8; hex 80000000001354a2; asc       T ;;
 
 4: len 5; hex 999d51180a; asc   Q  ;;
 
 5: len 5; hex 999d51180a; asc   Q  ;;
 
 6: len 8; hex 2f636f6e74616374; asc /contact;;
 
 7: len 1; hex 01; asc  ;;
 
 8: len 8; hex 800000000000b694; asc         ;;
 
 9: SQL NULL;
 
 10: SQL NULL;
 
 11: SQL NULL;
 
 
 
Record lock, heap no 18 PHYSICAL RECORD: n_fields 12; compact format; info bits 32
 
 0: len 8; hex 8000000000f2e5ce; asc         ;;
 
 1: len 6; hex 001912468d53; asc    F S;;
 
 2: len 7; hex 3d000000383be5; asc =   8; ;;
 
 3: len 8; hex 80000000001354cc; asc       T ;;
 
 4: len 5; hex 99a0866b97; asc    k ;;
 
 5: len 5; hex 99a0866b97; asc    k ;;
 
 6: len 6; hex 2f72656d6974; asc /remit;;
 
 7: len 1; hex 01; asc  ;;
 
 8: len 8; hex 800000000000b694; asc         ;;
 
 9: SQL NULL;
 
 10: SQL NULL;
 
 11: SQL NULL;
 
 
 
Record lock, heap no 19 PHYSICAL RECORD: n_fields 12; compact format; info bits 32
 
 0: len 8; hex 8000000000f2e5d0; asc         ;;
 
 1: len 6; hex 001912468d53; asc    F S;;
 
 2: len 7; hex 3d000000382ea5; asc =   8. ;;
 
 3: len 8; hex 80000000001354a0; asc       T ;;
 
 4: len 5; hex 99a06f0408; asc   o  ;;
 
 5: len 5; hex 99a06f0408; asc   o  ;;
 
 6: len 21; hex 2f637573746f6d697365642d736f6c7574696f6e73; asc /customised-solutions;;
 
 7: len 1; hex 01; asc  ;;
 
 8: len 8; hex 800000000000b694; asc         ;;
 
 9: SQL NULL;
 
 10: SQL NULL;
 
 11: SQL NULL;
 
 
 
Record lock, heap no 26 PHYSICAL RECORD: n_fields 12; compact format; info bits 32
 
 0: len 8; hex 8000000000f2e5de; asc         ;;
 
 1: len 6; hex 001912468d53; asc    F S;;
 
 2: len 7; hex 3d000412f504be; asc =      ;;
 
 3: len 8; hex 80000000001354ee; asc       T ;;
 
 4: len 5; hex 99a0ae6e2f; asc    n/;;
 
 5: len 5; hex 99a0ae6e2f; asc    n/;;
 
 6: len 30; hex 2f63616d7075732f666163696c69746965732f726f6f6d626f6f6b696e67; asc /campus/facilities/roombooking;;
 
 7: len 1; hex 01; asc  ;;
 
 8: len 8; hex 800000000000b694; asc         ;;
 
 9: SQL NULL;
 
 10: SQL NULL;
 
 11: SQL NULL;
 
 
 
Record lock, heap no 32 PHYSICAL RECORD: n_fields 12; compact format; info bits 32
 
 0: len 8; hex 8000000000f2e5ea; asc         ;;
 
 1: len 6; hex 001912468d53; asc    F S;;
 
 2: len 7; hex 3d000412f50038; asc =     8;;
 
 3: len 8; hex 80000000001354da; asc       T ;;
 
 4: len 5; hex 999f6cc4ce; asc   l  ;;
 
 5: len 5; hex 999f6cc4ce; asc   l  ;;
 
 6: len 30; hex 2f61727469636c65732f7468652d6665642d69732d69742d6261636b2d74; asc /articles/the-fed-is-it-back-t; (total 42 bytes);
 
 7: len 1; hex 01; asc  ;;
 
 8: len 8; hex 800000000000b694; asc         ;;
 
 9: SQL NULL;
 
 10: SQL NULL;
 
 11: SQL NULL;
 
 
 
Record lock, heap no 33 PHYSICAL RECORD: n_fields 12; compact format; info bits 32
 
 0: len 8; hex 8000000000f2e5ec; asc         ;;
 
 1: len 6; hex 001912468d53; asc    F S;;
 
 2: len 7; hex 3d000000383a1b; asc =   8: ;;
 
 3: len 8; hex 80000000001354c6; asc       T ;;
 
 4: len 5; hex 999dccd4da; asc      ;;
 
 5: len 5; hex 999dccd4da; asc      ;;
 
 6: len 6; hex 2f7374616666; asc /staff;;
 
 7: len 1; hex 01; asc  ;;
 
 8: len 8; hex 800000000000b694; asc         ;;
 
 9: SQL NULL;
 
 10: SQL NULL;
 
 11: SQL NULL;
 
 
 
Record lock, heap no 48 PHYSICAL RECORD: n_fields 12; compact format; info bits 32
 
 0: len 8; hex 8000000000f2e60a; asc         ;;
 
 1: len 6; hex 001912468d53; asc    F S;;
 
 2: len 7; hex 3d000412f502bd; asc =      ;;
 
 3: len 8; hex 80000000001354e2; asc       T ;;
 
 4: len 5; hex 999f7a7768; asc   zwh;;
 
 5: len 5; hex 999f7a7768; asc   zwh;;
 
 6: len 11; hex 2f66726565726570656174; asc /freerepeat;;
 
 7: len 1; hex 01; asc  ;;
 
 8: len 8; hex 800000000000b694; asc         ;;
 
 9: SQL NULL;
 
 10: SQL NULL;
 
 11: SQL NULL;
 
 
 
Record lock, heap no 54 PHYSICAL RECORD: n_fields 12; compact format; info bits 32
 
 0: len 8; hex 8000000000f2e616; asc         ;;
 
 1: len 6; hex 001912468d53; asc    F S;;
 
 2: len 7; hex 3d000000383446; asc =   84F;;
 
 3: len 8; hex 80000000001354b2; asc       T ;;
 
 4: len 5; hex 999e508d88; asc   P  ;;
 
 5: len 5; hex 999e508d88; asc   P  ;;
 
 6: len 8; hex 2f7061796d656e74; asc /payment;;
 
 7: len 1; hex 01; asc  ;;
 
 8: len 8; hex 800000000000b694; asc         ;;
 
 9: SQL NULL;
 
 10: SQL NULL;
 
 11: SQL NULL;
 
 
 
Record lock, heap no 55 PHYSICAL RECORD: n_fields 12; compact format; info bits 32
 
 0: len 8; hex 8000000000f2e618; asc         ;;
 
 1: len 6; hex 001912468d53; asc    F S;;
 
 2: len 7; hex 3d00000038331d; asc =   83 ;;
 
 3: len 8; hex 80000000001354ae; asc       T ;;
 
 4: len 5; hex 999f92be70; asc     p;;
 
 5: len 5; hex 999f92be70; asc     p;;
 
 6: len 10; hex 2f636f70797269676874; asc /copyright;;
 
 7: len 1; hex 01; asc  ;;
 
 8: len 8; hex 800000000000b694; asc         ;;
 
 9: SQL NULL;
 
 10: SQL NULL;
 
 11: SQL NULL;
 
 
 
Record lock, heap no 65 PHYSICAL RECORD: n_fields 12; compact format; info bits 32
 
 0: len 8; hex 8000000000f2e62c; asc        ,;;
 
 1: len 6; hex 001912468d53; asc    F S;;
 
 2: len 7; hex 3d000000382d4b; asc =   8-K;;
 
 3: len 8; hex 800000000013549c; asc       T ;;
 
 4: len 5; hex 999ede8b65; asc     e;;
 
 5: len 5; hex 999ede8b65; asc     e;;
 
 6: len 30; hex 2f61727469636c65732f7768792d65646974696e672d69732d657373656e; asc /articles/why-editing-is-essen; (total 60 bytes);
 
 7: len 1; hex 01; asc  ;;
 
 8: len 8; hex 800000000000b694; asc         ;;
 
 9: SQL NULL;
 
 10: SQL NULL;
 
 11: SQL NULL;
 
 
 
Record lock, heap no 67 PHYSICAL RECORD: n_fields 12; compact format; info bits 32
 
 0: len 8; hex 8000000000f2e630; asc        0;;
 
 1: len 6; hex 001912468d53; asc    F S;;
 
 2: len 7; hex 3d000412f507e1; asc =      ;;
 
 3: len 8; hex 80000000001354f8; asc       T ;;
 
 4: len 5; hex 99a1127a9b; asc    z ;;
 
 5: len 5; hex 99a1127a9b; asc    z ;;
 
 6: len 20; hex 2f656e74657270726973652f747261696e696e67; asc /enterprise/training;;
 
 7: len 1; hex 01; asc  ;;
 
 8: len 8; hex 800000000000b694; asc         ;;
 
 9: SQL NULL;
 
 10: SQL NULL;
 
 11: SQL NULL;
 
 
 
Record lock, heap no 82 PHYSICAL RECORD: n_fields 12; compact format; info bits 32
 
 0: len 8; hex 8000000000f2e64e; asc        N;;
 
 1: len 6; hex 001912468d53; asc    F S;;
 
 2: len 7; hex 3d000412f503e3; asc =      ;;
 
 3: len 8; hex 80000000001354e8; asc       T ;;
 
 4: len 5; hex 99a0030a6f; asc     o;;
 
 5: len 5; hex 99a0030a6f; asc     o;;
 
 6: len 8; hex 2f72657669657773; asc /reviews;;
 
 7: len 1; hex 01; asc  ;;
 
 8: len 8; hex 800000000000b694; asc         ;;
 
 9: SQL NULL;
 
 10: SQL NULL;
 
 11: SQL NULL;
 
 
 
Record lock, heap no 87 PHYSICAL RECORD: n_fields 12; compact format; info bits 32
 
 0: len 8; hex 8000000000f2e658; asc        X;;
 
 1: len 6; hex 001912468d53; asc    F S;;
 
 2: len 7; hex 3d000412f5022e; asc =     .;;
 
 3: len 8; hex 80000000001354e0; asc       T ;;
 
 4: len 5; hex 99a0a0f499; asc      ;;
 
 5: len 5; hex 99a0a0f499; asc      ;;
 
 6: len 4; hex 2f6e6670; asc /nfp;;
 
 7: len 1; hex 01; asc  ;;
 
 8: len 8; hex 800000000000b694; asc         ;;
 
 9: SQL NULL;
 
 10: SQL NULL;
 
 11: SQL NULL;
 
 
 
Record lock, heap no 94 PHYSICAL RECORD: n_fields 12; compact format; info bits 32
 
 0: len 8; hex 8000000000f2e666; asc        f;;
 
 1: len 6; hex 001912468d53; asc    F S;;
 
 2: len 7; hex 3d000412f5068f; asc =      ;;
 
 3: len 8; hex 80000000001354f4; asc       T ;;
 
 4: len 5; hex 999f11212e; asc    !.;;
 
 5: len 5; hex 999f11212e; asc    !.;;
 
 6: len 30; hex 2f61727469636c65732f646f6e616c642d7472756d702d616e642d746865; asc /articles/donald-trump-and-the; (total 49 bytes);
 
 7: len 1; hex 01; asc  ;;
 
 8: len 8; hex 800000000000b694; asc         ;;
 
 9: SQL NULL;
 
 10: SQL NULL;
 
 11: SQL NULL;
 
 
 
Record lock, heap no 101 PHYSICAL RECORD: n_fields 12; compact format; info bits 32
 
 0: len 8; hex 8000000000f2e674; asc        t;;
 
 1: len 6; hex 001912468d53; asc    F S;;
 
 2: len 7; hex 3d000000383803; asc =   88 ;;
 
 3: len 8; hex 80000000001354c0; asc       T ;;
 
 4: len 5; hex 999d5117cb; asc   Q  ;;
 
 5: len 5; hex 999d5117cb; asc   Q  ;;
 
 6: len 13; hex 2f636f6e6669726d6174696f6e; asc /confirmation;;
 
 7: len 1; hex 01; asc  ;;
 
 8: len 8; hex 800000000000b694; asc         ;;
 
 9: SQL NULL;
 
 10: SQL NULL;
 
 11: SQL NULL;
 
 
 
Record lock, heap no 102 PHYSICAL RECORD: n_fields 12; compact format; info bits 32
 
 0: len 8; hex 8000000000f2e676; asc        v;;
 
 1: len 6; hex 001912468d53; asc    F S;;
 
 2: len 7; hex 3d000000383610; asc =   86 ;;
 
 3: len 8; hex 80000000001354b8; asc       T ;;
 
 4: len 5; hex 99a10abe58; asc     X;;
 
 5: len 5; hex 99a10abe58; asc     X;;
 
 6: len 30; hex 2f61727469636c65732f76616c75652d6372656174696f6e2d696e2d676c; asc /articles/value-creation-in-gl; (total 55 bytes);
 
 7: len 1; hex 01; asc  ;;
 
 8: len 8; hex 800000000000b694; asc         ;;
 
 9: SQL NULL;
 
 10: SQL NULL;
 
 11: SQL NULL;
 
 
 
Record lock, heap no 103 PHYSICAL RECORD: n_fields 12; compact format; info bits 32
 
 0: len 8; hex 8000000000f2e678; asc        x;;
 
 1: len 6; hex 001912468d53; asc    F S;;
 
 2: len 7; hex 3d000412f50352; asc =     R;;
 
 3: len 8; hex 80000000001354e4; asc       T ;;
 
 4: len 5; hex 99a08504ae; asc      ;;
 
 5: len 5; hex 99a08504ae; asc      ;;
 
 6: len 6; hex 2f6261646765; asc /badge;;
 
 7: len 1; hex 01; asc  ;;
 
 8: len 8; hex 800000000000b694; asc         ;;
 
 9: SQL NULL;
 
 10: SQL NULL;
 
 11: SQL NULL;
 
 
 
Record lock, heap no 104 PHYSICAL RECORD: n_fields 12; compact format; info bits 32
 
 0: len 8; hex 8000000000f2e67a; asc        z;;
 
 1: len 6; hex 001912468d53; asc    F S;;
 
 2: len 7; hex 3d000000383569; asc =   85i;;
 
 3: len 8; hex 80000000001354b6; asc       T ;;
 
 4: len 5; hex 99a086b9ce; asc      ;;
 
 5: len 5; hex 99a086b9ce; asc      ;;
 
 6: len 28; hex 2f696e7374727563746f72732f636f757273652d70726f706f73616c; asc /instructors/course-proposal;;
 
 7: len 1; hex 01; asc  ;;
 
 8: len 8; hex 800000000000b694; asc         ;;
 
 9: SQL NULL;
 
 10: SQL NULL;
 
 11: SQL NULL;
 
 
 
Record lock, heap no 109 PHYSICAL RECORD: n_fields 12; compact format; info bits 32
 
 0: len 8; hex 8000000000f2e684; asc         ;;
 
 1: len 6; hex 001912468d53; asc    F S;;
 
 2: len 7; hex 3d000412f5051e; asc =      ;;
 
 3: len 8; hex 80000000001354ee; asc       T ;;
 
 4: len 5; hex 99a0ae6613; asc    f ;;
 
 5: len 5; hex 99a0ae6613; asc    f ;;
 
 6: len 12; hex 2f726f6f6d626f6f6b696e67; asc /roombooking;;
 
 7: len 1; hex 00; asc  ;;
 
 8: len 8; hex 800000000000b694; asc         ;;
 
 9: SQL NULL;
 
 10: SQL NULL;
 
 11: SQL NULL;
 
 
 
Record lock, heap no 118 PHYSICAL RECORD: n_fields 12; compact format; info bits 32
 
 0: len 8; hex 8000000000f2e696; asc         ;;
 
 1: len 6; hex 001912468d53; asc    F S;;
 
 2: len 7; hex 3d000000382e11; asc =   8. ;;
 
 3: len 8; hex 800000000013549e; asc       T ;;
 
 4: len 5; hex 999f393132; asc   912;;
 
 5: len 5; hex 999f393132; asc   912;;
 
 6: len 9; hex 2f7375626a65637473; asc /subjects;;
 
 7: len 1; hex 01; asc  ;;
 
 8: len 8; hex 800000000000b694; asc         ;;
 
 9: SQL NULL;
 
 10: SQL NULL;
 
 11: SQL NULL;
 
 
 
Record lock, heap no 131 PHYSICAL RECORD: n_fields 12; compact format; info bits 32
 
 0: len 8; hex 8000000000f2e6b0; asc         ;;
 
 1: len 6; hex 001912468d53; asc    F S;;
 
 2: len 7; hex 3d000412f500ed; asc =      ;;
 
 3: len 8; hex 80000000001354dc; asc       T ;;
 
 4: len 5; hex 999dccce06; asc      ;;
 
 5: len 5; hex 999dccce06; asc      ;;
 
 6: len 9; hex 2f73747564656e7473; asc /students;;
 
 7: len 1; hex 01; asc  ;;
 
 8: len 8; hex 800000000000b694; asc         ;;
 
 9: SQL NULL;
 
 10: SQL NULL;
 
 11: SQL NULL;
 
 
 
Record lock, heap no 134 PHYSICAL RECORD: n_fields 12; compact format; info bits 32
 
 0: len 8; hex 8000000000f2e6b6; asc         ;;
 
 1: len 6; hex 001912468d53; asc    F S;;
 
 2: len 7; hex 3d000412f50ab7; asc =      ;;
 
 3: len 8; hex 8000000000135502; asc       U ;;
 
 4: len 5; hex 999f6ea3eb; asc   n  ;;
 
 5: len 5; hex 999f6ea3eb; asc   n  ;;
 
 6: len 30; hex 2f61727469636c65732f69662d77652d63616e742d77696e2d7468656e2d; asc /articles/if-we-cant-win-then-; (total 50 bytes);
 
 7: len 1; hex 01; asc  ;;
 
 8: len 8; hex 800000000000b694; asc         ;;
 
 9: SQL NULL;
 
 10: SQL NULL;
 
 11: SQL NULL;
 
 
 
Record lock, heap no 153 PHYSICAL RECORD: n_fields 12; compact format; info bits 32
 
 0: len 8; hex 8000000000f2e6dc; asc         ;;
 
 1: len 6; hex 001912468d53; asc    F S;;
 
 2: len 7; hex 3d00000038296d; asc =   8)m;;
 
 3: len 8; hex 80000000001354bc; asc       T ;;
 
 4: len 5; hex 999ed077f7; asc    w ;;
 
 5: len 5; hex 999ed077f7; asc    w ;;
 
 6: len 10; hex 2f6c6f636174696f6e73; asc /locations;;
 
 7: len 1; hex 01; asc  ;;
 
 8: len 8; hex 800000000000b694; asc         ;;
 
 9: SQL NULL;
 
 10: SQL NULL;
 
 11: SQL NULL;
 
 
 
Record lock, heap no 165 PHYSICAL RECORD: n_fields 12; compact format; info bits 32
 
 0: len 8; hex 8000000000f2e6f4; asc         ;;
 
 1: len 6; hex 001912468d53; asc    F S;;
 
 2: len 7; hex 3d000000382bf9; asc =   8+ ;;
 
 3: len 8; hex 8000000000135498; asc       T ;;
 
 4: len 5; hex 999fc32385; asc    # ;;
 
 5: len 5; hex 999fc32385; asc    # ;;
 
 6: len 30; hex 2f61727469636c65732f776861742d61626f75742d6c6561646572736869; asc /articles/what-about-leadershi; (total 46 bytes);
 
 7: len 1; hex 01; asc  ;;
 
 8: len 8; hex 800000000000b694; asc         ;;
 
 9: SQL NULL;
 
 10: SQL NULL;
 
 11: SQL NULL;
 
 
 
2019-01-22 11:12:00 50726264576 [Note] InnoDB:  SQL1: DELETE FROM WebURLAlias where webSiteVersionId = 46740��E\
 
190122 11:12:00 [ERROR] mysqld got signal 11 ;
 
This could be because you hit a bug. It is also possible that this binary
 
or one of the libraries it was linked against is corrupt, improperly built,
 
or misconfigured. This error can also be caused by malfunctioning hardware.
 
 
 
To report this bug, see https://mariadb.com/kb/en/reporting-bugs
 
 
 
We will try our best to scrape up some info that will hopefully help
 
diagnose the problem, but since we have already crashed,
 
something is definitely wrong and this may fail.
 
 
 
Server version: 10.2.19-MariaDB-log
 
key_buffer_size=67108864
 
read_buffer_size=131072
 
max_used_connections=0
 
max_threads=2002
 
thread_count=55
 
It is possible that mysqld could use up to
 
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 33164362 K  bytes of memory
 
Hope that's ok; if not, decrease some variables in the equation.
 
 
 
Thread pointer: 0xbdb6189c8
 
Attempting backtrace. You can use the following information to find out
 
where mysqld died. If you see no messages after this, something went
 
terribly wrong...
 
stack_bottom = 0x7fffdabc6b88 thread_stack 0x49000
 
0xc6287e <my_print_stacktrace+0x2e> at /usr/local/libexec/mysqld
 
0x7926a4 <handle_fatal_signal+0x294> at /usr/local/libexec/mysqld
 
0x8032f4946 <pthread_sigmask+0x536> at /lib/libthr.so.3
 
0x8032f3eb2 <pthread_getspecific+0xe12> at /lib/libthr.so.3
 
 
 
Trying to get some variables.
 
Some pointers may be invalid and cause the dump to abort.
 
Query (0x87ece0b93): DELETE FROM WebURLAlias where webSiteVersionId = 46740
 
 
 
Connection ID (thread ID): 53
 
Status: NOT_KILLED
 
 
 
Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=off,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on
 
 
 
The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
 
information that should help you find out what is causing the crash.

Comment by TAO ZHOU [ 2019-01-22 ]

Looks like it relates to this one MDEV-14877

Comment by Jan Lindström (Inactive) [ 2019-01-29 ]

Hi, could you please provide full unedited error log from crashing node and its configuration. Furthermore, can you provide output from following commands:

  • show create table WebURLAlias;
  • show create table WebNodeType;
  • show create table w2live_college;
Comment by TAO ZHOU [ 2019-01-29 ]

Here's the output of 'show create':

| WebURLAlias | CREATE TABLE `WebURLAlias` (
 
  `id` bigint(20) NOT NULL AUTO_INCREMENT,
 
  `webNodeId` bigint(20) DEFAULT NULL,
 
  `created` datetime NOT NULL,
 
  `modified` datetime NOT NULL,
 
  `urlPath` varchar(512) COLLATE utf8_unicode_ci DEFAULT NULL,
 
  `isDefault` bit(1) NOT NULL DEFAULT b'0',
 
  `webSiteVersionId` bigint(20) NOT NULL,
 
  `redirectTo` varchar(512) COLLATE utf8_unicode_ci DEFAULT NULL,
 
  `matchType` smallint(6) DEFAULT NULL,
 
  `specialPage` smallint(6) DEFAULT NULL,
 
  PRIMARY KEY (`id`),
 
  KEY `webNode` (`webNodeId`),
 
  KEY `urlPath` (`urlPath`(100)),
 
  KEY `WebUrlAlias_webSiteVersion` (`webSiteVersionId`),
 
  CONSTRAINT `WebURLAlias_webNode` FOREIGN KEY (`webNodeId`) REFERENCES `WebNode` (`id`) ,
 
  CONSTRAINT `WebUrlAlias_webSiteVersion` FOREIGN KEY (`webSiteVersionId`) REFERENCES `WebSiteVersion` (`id`) 
) ENGINE=InnoDB AUTO_INCREMENT=16832661 DEFAULT CHARSET=utf8 COLLATE=utf8_unicode_ci |


WebNodeType | CREATE TABLE `WebNodeType` (
 
  `id` bigint(20) NOT NULL AUTO_INCREMENT,
 
  `name` varchar(128) COLLATE utf8_unicode_ci NOT NULL,
 
  `created` datetime NOT NULL,
 
  `modified` datetime NOT NULL,
 
  `webSiteVersionId` bigint(20) NOT NULL,
 
  `webSiteLayoutId` bigint(20) NOT NULL,
 
  PRIMARY KEY (`id`),
 
  KEY `name_idx` (`name`),
 
  KEY `webSiteLayoutId` (`webSiteLayoutId`),
 
  KEY `WebNodeType_webSiteVersion` (`webSiteVersionId`),
 
  CONSTRAINT `WebNodeType_ibfk_1` FOREIGN KEY (`webSiteLayoutId`) REFERENCES `WebSiteLayout` (`id`),
 
  CONSTRAINT `WebNodeType_webSiteVersion` FOREIGN KEY (`webSiteVersionId`) REFERENCES `WebSiteVersion` (`id`)
 
) ENGINE=InnoDB AUTO_INCREMENT=123366 DEFAULT CHARSET=utf8 COLLATE=utf8_unicode_ci |

w2live_college is the database name, not a table.

Comment by TAO ZHOU [ 2019-01-29 ]

The full log is huge, so I truncated it. The log I posted here is exactly the original log, from the time mysqld was started to the time it completely crashed.

Comment by Jan Lindström (Inactive) [ 2019-03-18 ]

Can you easily repeat this problem ? Do you have wsrep_slave_threads > 1 ? We have some evidence that complex foreign keys with ON DELETE action cause some problems on DELETE, but exact problem has not being able to repeat (and that way identified and fixed).

Generated at Thu Feb 08 08:43:10 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.