[MDEV-18317] Mariadb Crashed due to BF-BF X lock conflict - Jira

XML

Word

Printable

Details

Type: Bug
Status: Closed (View Workflow)
Priority: Critical
Resolution: Incomplete
Affects Version/s: 10.2.19
Fix Version/s: N/A
Component/s: Galera
Labels:
- need_feedback
Environment:
FreeBSD11.2

Description

Configuration of 3 nodes(two nodes + one arbitrator).

Error logs as below.

2019-01-19 20:11:51 50738248448 [Note] WSREP: EX-EX trx conflict for key (2,FLAT8)8632c156 4e7d3bb4: source: 915c701c-13cd-11e9-9ddd-eef50d21aff0 version: 4 local: 1 state: MUST_ABORT flags: 1 conn_id: 388433 trx_id: 107678074869 seqnos (l: 6356516, g: 1404623292, s: 1404623290, d: 1404606848, ts: 889004531810575) <---> source: bd601f60-123e-11e9-b2df-cb043767660f version: 4 local: 0 state: APPLYING flags: 1 conn_id: 377280 trx_id: 107678071877 seqnos (l: 6356515, g: 1404623291, s: 1404623290, d: 1404622944, ts: 14095281738886391)

2019-01-19 20:14:26 50740154368 [Note] InnoDB: BF-BF X lock conflict,mode: 1027 supremum: 0

2019-01-19 20:14:26 50740154368 [Note] InnoDB: conflicts states: my 0 locked 0

RECORD LOCKS space id 324921 page no 1570 n bits 240 index PRIMARY of table `w2live_college`.`WebURLAlias` trx id 107678078543 lock_mode X locks rec but not gap

Record lock, heap no 100 PHYSICAL RECORD: n_fields 12; compact format; info bits 32

 0: len 8; hex 8000000000f28c5e; asc        ^;;

 1: len 6; hex 0019121d164f; asc      O;;

 2: len 7; hex 030003001b2eca; asc      . ;;

 3: len 8; hex 8000000000134d64; asc       Md;;

 4: len 5; hex 999f92c24a; asc     J;;

 5: len 5; hex 999f92c24a; asc     J;;

 6: len 8; hex 2f70726976616379; asc /privacy;;

 7: len 1; hex 01; asc  ;;

 8: len 8; hex 800000000000b668; asc        h;;

 9: SQL NULL;

 10: SQL NULL;

 11: SQL NULL;

Record lock, heap no 112 PHYSICAL RECORD: n_fields 12; compact format; info bits 32

 0: len 8; hex 8000000000f28c76; asc        v;;

 1: len 6; hex 0019121d164f; asc      O;;

 2: len 7; hex 030003001b2d8b; asc      - ;;

 3: len 8; hex 8000000000134d60; asc       M`;;

 4: len 5; hex 999ede8bb0; asc      ;;

 5: len 5; hex 999ede8bb0; asc      ;;

 6: len 30; hex 2f61727469636c65732f6c6f73652d796f75722d666561722d6f662d6772; asc /articles/lose-your-fear-of-gr; (total 35 bytes);

 7: len 1; hex 01; asc  ;;

 8: len 8; hex 800000000000b668; asc        h;;

 9: SQL NULL;

 10: SQL NULL;

 11: SQL NULL;

Record lock, heap no 113 PHYSICAL RECORD: n_fields 12; compact format; info bits 32

 0: len 8; hex 8000000000f28c78; asc        x;;

 1: len 6; hex 0019121d164f; asc      O;;

 2: len 7; hex 030003001b2b16; asc      + ;;

 3: len 8; hex 8000000000134d56; asc       MV;;

 4: len 5; hex 99a08d016d; asc     m;;

 5: len 5; hex 99a08d016d; asc     m;;

 6: len 8; hex 2f73756767657374; asc /suggest;;

 7: len 1; hex 00; asc  ;;

 8: len 8; hex 800000000000b668; asc        h;;

 9: SQL NULL;

 10: SQL NULL;

 11: SQL NULL;

Record lock, heap no 121 PHYSICAL RECORD: n_fields 12; compact format; info bits 32

 0: len 8; hex 8000000000f28c88; asc         ;;

 1: len 6; hex 0019121d164f; asc      O;;

 2: len 7; hex 030003001b297e; asc      )~;;

 3: len 8; hex 8000000000134d50; asc       MP;;

 4: len 5; hex 999f11212e; asc    !.;;

 5: len 5; hex 999f11212e; asc    !.;;

 6: len 30; hex 2f61727469636c65732f646f6e616c642d7472756d702d616e642d746865; asc /articles/donald-trump-and-the; (total 49 bytes);

 7: len 1; hex 01; asc  ;;

 8: len 8; hex 800000000000b668; asc        h;;

 9: SQL NULL;

 10: SQL NULL;

 11: SQL NULL;

Record lock, heap no 133 PHYSICAL RECORD: n_fields 12; compact format; info bits 32

 0: len 8; hex 8000000000f28ca0; asc         ;;

 1: len 6; hex 0019121d164f; asc      O;;

 2: len 7; hex 030003001b2e39; asc      .9;;

 3: len 8; hex 8000000000134d62; asc       Mb;;

 4: len 5; hex 999d51178a; asc   Q  ;;

 5: len 5; hex 999d51178a; asc   Q  ;;

 6: len 6; hex 2f6c6567616c; asc /legal;;

 7: len 1; hex 01; asc  ;;

 8: len 8; hex 800000000000b668; asc        h;;

 9: SQL NULL;

 10: SQL NULL;

 11: SQL NULL;

Record lock, heap no 151 PHYSICAL RECORD: n_fields 12; compact format; info bits 32

 0: len 8; hex 8000000000f28cc4; asc         ;;

 1: len 6; hex 0019121d164f; asc      O;;

 2: len 7; hex 030003001b2a3a; asc      *:;;

 3: len 8; hex 8000000000134d52; asc       MR;;

 4: len 5; hex 999dcccd78; asc     x;;

 5: len 5; hex 999dcccd78; asc     x;;

 6: len 8; hex 2f73656e696f7273; asc /seniors;;

 7: len 1; hex 01; asc  ;;

 8: len 8; hex 800000000000b668; asc        h;;

 9: SQL NULL;

 10: SQL NULL;

 11: SQL NULL;

Record lock, heap no 165 PHYSICAL RECORD: n_fields 12; compact format; info bits 32

 0: len 8; hex 8000000000f28ce0; asc         ;;

 1: len 6; hex 0019121d164f; asc      O;;

 2: len 7; hex 030003001b2c3b; asc      ,;;;

 3: len 8; hex 8000000000134d5c; asc       M\;;

 4: len 5; hex 999f6ea3eb; asc   n  ;;

 5: len 5; hex 999f6ea3eb; asc   n  ;;

 6: len 30; hex 2f61727469636c65732f69662d77652d63616e742d77696e2d7468656e2d; asc /articles/if-we-cant-win-then-; (total 50 bytes);

 7: len 1; hex 01; asc  ;;

 8: len 8; hex 800000000000b668; asc        h;;

 9: SQL NULL;

 10: SQL NULL;

 11: SQL NULL;

2019-01-19 20:14:26 50740154368 [Note] InnoDB:  SQL1: DELETE FROM WebURLAlias where webSiteVersionId = 46696r�B\

2019-01-19 20:14:26 50740154368 [Note] InnoDB:  SQL2: DELETE FROM WebNodeType where webSiteVersionId = 46696r�B\

2019-01-19 20:14:26 50740154368 [ERROR] mysqld: Can't find record in 'WebURLAlias'

2019-01-19 20:14:26 50740154368 [ERROR] Slave SQL: Could not execute Delete_rows_v1 event on table w2live_college.WebURLAlias; Can't find record in 'WebURLAlias', Error_code: 1032; handler error HA_ERR_KEY_NOT_FOUND; the event's master log FIRST, end_log_pos 24701, Internal MariaDB error code: 1032

2019-01-19 20:14:26 50740154368 [Warning] WSREP: RBR event 5 Delete_rows_v1 apply warning: 120, 1404625113

2019-01-19 20:14:26 50740154368 [Warning] WSREP: Failed to apply app buffer: seqno: 1404625113, status: 1

	 at galera/src/trx_handle.cpp:apply():353

Retrying 2th time

2019-01-19 20:14:26 50740154368 [Note] InnoDB: BF-BF X lock conflict,mode: 1027 supremum: 0

2019-01-19 20:14:26 50740154368 [Note] InnoDB: conflicts states: my 0 locked 0

RECORD LOCKS space id 324921 page no 1570 n bits 240 index PRIMARY of table `w2live_college`.`WebURLAlias` trx id 107678078543 lock_mode X locks rec but not gap

Record lock, heap no 86 PHYSICAL RECORD: n_fields 12; compact format; info bits 32

 0: len 8; hex 8000000000f28c42; asc        B;;

 1: len 6; hex 0019121d164f; asc      O;;

 2: len 7; hex 030003001b3c48; asc      <H;;

 3: len 8; hex 8000000000134dac; asc       M ;;

 4: len 5; hex 999f7a7768; asc   zwh;;

 5: len 5; hex 999f7a7768; asc   zwh;;

 6: len 11; hex 2f66726565726570656174; asc /freerepeat;;

 7: len 1; hex 01; asc  ;;

 8: len 8; hex 800000000000b668; asc        h;;

 9: SQL NULL;

 10: SQL NULL;

 11: SQL NULL;

Record lock, heap no 100 PHYSICAL RECORD: n_fields 12; compact format; info bits 32

 0: len 8; hex 8000000000f28c5e; asc        ^;;

 1: len 6; hex 0019121d164f; asc      O;;

 2: len 7; hex 030003001b2eca; asc      . ;;

 3: len 8; hex 8000000000134d64; asc       Md;;

 4: len 5; hex 999f92c24a; asc     J;;

 5: len 5; hex 999f92c24a; asc     J;;

 6: len 8; hex 2f70726976616379; asc /privacy;;

 7: len 1; hex 01; asc  ;;

 8: len 8; hex 800000000000b668; asc        h;;

 9: SQL NULL;

 10: SQL NULL;

 11: SQL NULL;

Record lock, heap no 112 PHYSICAL RECORD: n_fields 12; compact format; info bits 32

 0: len 8; hex 8000000000f28c76; asc        v;;

 1: len 6; hex 0019121d164f; asc      O;;

 2: len 7; hex 030003001b2d8b; asc      - ;;

 3: len 8; hex 8000000000134d60; asc       M`;;

 4: len 5; hex 999ede8bb0; asc      ;;

 5: len 5; hex 999ede8bb0; asc      ;;

 6: len 30; hex 2f61727469636c65732f6c6f73652d796f75722d666561722d6f662d6772; asc /articles/lose-your-fear-of-gr; (total 35 bytes);

 7: len 1; hex 01; asc  ;;

 8: len 8; hex 800000000000b668; asc        h;;

 9: SQL NULL;

 10: SQL NULL;

 11: SQL NULL;

Record lock, heap no 113 PHYSICAL RECORD: n_fields 12; compact format; info bits 32

 0: len 8; hex 8000000000f28c78; asc        x;;

 1: len 6; hex 0019121d164f; asc      O;;

 2: len 7; hex 030003001b2b16; asc      + ;;

 3: len 8; hex 8000000000134d56; asc       MV;;

 4: len 5; hex 99a08d016d; asc     m;;

 5: len 5; hex 99a08d016d; asc     m;;

 6: len 8; hex 2f73756767657374; asc /suggest;;

 7: len 1; hex 00; asc  ;;

 8: len 8; hex 800000000000b668; asc        h;;

 9: SQL NULL;

 10: SQL NULL;

 11: SQL NULL;

Record lock, heap no 121 PHYSICAL RECORD: n_fields 12; compact format; info bits 32

 0: len 8; hex 8000000000f28c88; asc         ;;

 1: len 6; hex 0019121d164f; asc      O;;

 2: len 7; hex 030003001b297e; asc      )~;;

 3: len 8; hex 8000000000134d50; asc       MP;;

 4: len 5; hex 999f11212e; asc    !.;;

 5: len 5; hex 999f11212e; asc    !.;;

 6: len 30; hex 2f61727469636c65732f646f6e616c642d7472756d702d616e642d746865; asc /articles/donald-trump-and-the; (total 49 bytes);

 7: len 1; hex 01; asc  ;;

 8: len 8; hex 800000000000b668; asc        h;;

 9: SQL NULL;

 10: SQL NULL;

 11: SQL NULL;

Record lock, heap no 126 PHYSICAL RECORD: n_fields 12; compact format; info bits 32

 0: len 8; hex 8000000000f28c92; asc         ;;

 1: len 6; hex 0019121d164f; asc      O;;

 2: len 7; hex 030003001b3a4f; asc      :O;;

 3: len 8; hex 8000000000134da0; asc       M ;;

 4: len 5; hex 99a0a0f499; asc      ;;

 5: len 5; hex 99a0a0f499; asc      ;;

 6: len 4; hex 2f6e6670; asc /nfp;;

 7: len 1; hex 01; asc  ;;

 8: len 8; hex 800000000000b668; asc        h;;

 9: SQL NULL;

 10: SQL NULL;

 11: SQL NULL;

Record lock, heap no 133 PHYSICAL RECORD: n_fields 12; compact format; info bits 32

 0: len 8; hex 8000000000f28ca0; asc         ;;

 1: len 6; hex 0019121d164f; asc      O;;

 2: len 7; hex 030003001b2e39; asc      .9;;

 3: len 8; hex 8000000000134d62; asc       Mb;;

 4: len 5; hex 999d51178a; asc   Q  ;;

 5: len 5; hex 999d51178a; asc   Q  ;;

 6: len 6; hex 2f6c6567616c; asc /legal;;

 7: len 1; hex 01; asc  ;;

 8: len 8; hex 800000000000b668; asc        h;;

 9: SQL NULL;

 10: SQL NULL;

 11: SQL NULL;

Record lock, heap no 138 PHYSICAL RECORD: n_fields 12; compact format; info bits 32

 0: len 8; hex 8000000000f28caa; asc         ;;

 1: len 6; hex 0019121d164f; asc      O;;

 2: len 7; hex 030003001b34ac; asc      4 ;;

 3: len 8; hex 8000000000134d84; asc       M ;;

 4: len 5; hex 99a0671664; asc   g d;;

 5: len 5; hex 99a0671664; asc   g d;;

 6: len 30; hex 2f61727469636c65732f7468652d62696c6c696f6e2d646f6c6c61722d62; asc /articles/the-billion-dollar-b; (total 64 bytes);

 7: len 1; hex 01; asc  ;;

 8: len 8; hex 800000000000b668; asc        h;;

 9: SQL NULL;

 10: SQL NULL;

 11: SQL NULL;

Record lock, heap no 141 PHYSICAL RECORD: n_fields 12; compact format; info bits 32

 0: len 8; hex 8000000000f28cb0; asc         ;;

 1: len 6; hex 0019121d164f; asc      O;;

 2: len 7; hex 030003001b3739; asc      79;;

 3: len 8; hex 8000000000134d90; asc       M ;;

 4: len 5; hex 99a0ac7745; asc    wE;;

 5: len 5; hex 99a0ac7745; asc    wE;;

 6: len 14; hex 2f636f72706f7261746570617373; asc /corporatepass;;

 7: len 1; hex 01; asc  ;;

 8: len 8; hex 800000000000b668; asc        h;;

 9: SQL NULL;

 10: SQL NULL;

 11: SQL NULL;

Record lock, heap no 147 PHYSICAL RECORD: n_fields 12; compact format; info bits 32

 0: len 8; hex 8000000000f28cbc; asc         ;;

 1: len 6; hex 0019121d164f; asc      O;;

 2: len 7; hex 030003001b33d3; asc      3 ;;

 3: len 8; hex 8000000000134d80; asc       M ;;

 4: len 5; hex 999f39311a; asc   91 ;;

 5: len 5; hex 999f39311a; asc   91 ;;

 6: len 6; hex 2f6461746573; asc /dates;;

 7: len 1; hex 01; asc  ;;

 8: len 8; hex 800000000000b668; asc        h;;

 9: SQL NULL;

 10: SQL NULL;

 11: SQL NULL;

Record lock, heap no 151 PHYSICAL RECORD: n_fields 12; compact format; info bits 32

 0: len 8; hex 8000000000f28cc4; asc         ;;

 1: len 6; hex 0019121d164f; asc      O;;

 2: len 7; hex 030003001b2a3a; asc      *:;;

 3: len 8; hex 8000000000134d52; asc       MR;;

 4: len 5; hex 999dcccd78; asc     x;;

 5: len 5; hex 999dcccd78; asc     x;;

 6: len 8; hex 2f73656e696f7273; asc /seniors;;

 7: len 1; hex 01; asc  ;;

 8: len 8; hex 800000000000b668; asc        h;;

 9: SQL NULL;

 10: SQL NULL;

 11: SQL NULL;

Record lock, heap no 159 PHYSICAL RECORD: n_fields 12; compact format; info bits 32

 0: len 8; hex 8000000000f28cd4; asc         ;;

 1: len 6; hex 0019121d164f; asc      O;;

 2: len 7; hex 030003001b3bb8; asc      ; ;;

 3: len 8; hex 8000000000134daa; asc       M ;;

 4: len 5; hex 99a10aee14; asc      ;;

 5: len 5; hex 99a10aee14; asc      ;;

 6: len 5; hex 2f6c696e6b; asc /link;;

 7: len 1; hex 01; asc  ;;

 8: len 8; hex 800000000000b668; asc        h;;

 9: SQL NULL;

 10: SQL NULL;

 11: SQL NULL;

Record lock, heap no 161 PHYSICAL RECORD: n_fields 12; compact format; info bits 32

 0: len 8; hex 8000000000f28cd8; asc         ;;

 1: len 6; hex 0019121d164f; asc      O;;

 2: len 7; hex 030003001b3608; asc      6 ;;

 3: len 8; hex 8000000000134d8a; asc       M ;;

 4: len 5; hex 99a086b575; asc     u;;

 5: len 5; hex 99a086b575; asc     u;;

 6: len 22; hex 2f6469726563746f72792f696e7374727563746f7273; asc /directory/instructors;;

 7: len 1; hex 01; asc  ;;

 8: len 8; hex 800000000000b668; asc        h;;

 9: SQL NULL;

 10: SQL NULL;

 11: SQL NULL;

Record lock, heap no 162 PHYSICAL RECORD: n_fields 12; compact format; info bits 32

 0: len 8; hex 8000000000f28cda; asc         ;;

 1: len 6; hex 0019121d164f; asc      O;;

 2: len 7; hex 030003001b3282; asc      2 ;;

 3: len 8; hex 8000000000134d7a; asc       Mz;;

 4: len 5; hex 999f6e6bbb; asc   nk ;;

 5: len 5; hex 999f6e6bbb; asc   nk ;;

 6: len 30; hex 2f61727469636c65732f7468652d6e65776c792d72656c65617365642d6a; asc /articles/the-newly-released-j; (total 52 bytes);

 7: len 1; hex 01; asc  ;;

 8: len 8; hex 800000000000b668; asc        h;;

 9: SQL NULL;

 10: SQL NULL;

 11: SQL NULL;

Record lock, heap no 165 PHYSICAL RECORD: n_fields 12; compact format; info bits 32

 0: len 8; hex 8000000000f28ce0; asc         ;;

 1: len 6; hex 0019121d164f; asc      O;;

 2: len 7; hex 030003001b2c3b; asc      ,;;;

 3: len 8; hex 8000000000134d5c; asc       M\;;

 4: len 5; hex 999f6ea3eb; asc   n  ;;

 5: len 5; hex 999f6ea3eb; asc   n  ;;

 6: len 30; hex 2f61727469636c65732f69662d77652d63616e742d77696e2d7468656e2d; asc /articles/if-we-cant-win-then-; (total 50 bytes);

 7: len 1; hex 01; asc  ;;

 8: len 8; hex 800000000000b668; asc        h;;

 9: SQL NULL;

 10: SQL NULL;

 11: SQL NULL;

2019-01-19 20:14:27 50740154368 [Note] InnoDB:  SQL1: DELETE FROM WebURLAlias where webSiteVersionId = 46696r�B\

190119 20:14:27 [ERROR] mysqld got signal 11 ;

This could be because you hit a bug. It is also possible that this binary

or one of the libraries it was linked against is corrupt, improperly built,

or misconfigured. This error can also be caused by malfunctioning hardware.

To report this bug, see https://mariadb.com/kb/en/reporting-bugs

We will try our best to scrape up some info that will hopefully help

diagnose the problem, but since we have already crashed,

something is definitely wrong and this may fail.

Server version: 10.2.19-MariaDB-log

key_buffer_size=67108864

read_buffer_size=131072

max_used_connections=118

max_threads=2002

thread_count=161

It is possible that mysqld could use up to

key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 33164362 K  bytes of memory

Hope that's ok; if not, decrease some variables in the equation.

Thread pointer: 0x83a61f448

Attempting backtrace. You can use the following information to find out

where mysqld died. If you see no messages after this, something went

terribly wrong...

stack_bottom = 0x7fffdb1e1b88 thread_stack 0x49000

0xc6287e <my_print_stacktrace+0x2e> at /usr/local/libexec/mysqld

0x7926a4 <handle_fatal_signal+0x294> at /usr/local/libexec/mysqld

0x8032f4946 <pthread_sigmask+0x536> at /lib/libthr.so.3

0x8032f3eb2 <pthread_getspecific+0xe12> at /lib/libthr.so.3

Trying to get some variables.

Some pointers may be invalid and cause the dump to abort.

Query (0x852edd803): DELETE FROM WebURLAlias where webSiteVersionId = 46696

Connection ID (thread ID): 39

Status: NOT_KILLED

Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=off,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on

The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains

information that should help you find out what is causing the crash.

Attachments

Issue Links

relates to

MDEV-21466 BF-BF X lock conflict, replication fails, cluster unusable

Closed

Activity

People

Assignee:: Jan Lindström (Inactive)

Reporter:: TAO ZHOU

Votes:: 0 Vote for this issue

Watchers:: 5 Start watching this issue

Dates

Created:: 2019-01-21 02:59

Updated:: 2020-01-13 18:07

Resolved:: 2019-04-15 11:14

Git Integration

Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.