Uploaded image for project: 'MariaDB Server'
  1. MariaDB Server
  2. MDEV-25551

Assertion `mode_ == m_local || transaction_.is_streaming()` failed in BF abort

    XMLWordPrintable

    Details

      Description

      We are experiencing frequent crashes on instances that are trying to replicate a particular sequence of statements:

      2021-04-26T15:11:01.993175+00:00 db2.mydomain.example mysqld[21482]: 2021-04-26 15:11:01 12 [Note] InnoDB: *** Priority TRANSACTION:
      2021-04-26T15:11:01.993335+00:00 db2.mydomain.example mysqld[21482]: TRANSACTION 6470784, ACTIVE 0 sec starting index read
      2021-04-26T15:11:01.993459+00:00 db2.mydomain.example mysqld[21482]: mysql tables in use 1, locked 1
      2021-04-26T15:11:01.993579+00:00 db2.mydomain.example mysqld[21482]: 1 lock struct(s), heap size 1128, 0 row lock(s)
      2021-04-26T15:11:01.993702+00:00 db2.mydomain.example mysqld[21482]: MySQL thread id 12, OS thread handle 140241487787776, query id 7 Update_rows_log_event::find_row(2294786)
      2021-04-26T15:11:01.993819+00:00 db2.mydomain.example mysqld[21482]: BINLOG '
      2021-04-26T15:11:01.993941+00:00 db2.mydomain.example mysqld[21482]: S8WCYBMBAAAAPwAAAClBBAAAADUAAAAAAAEAB25ldXRyb24AD2FsZW1iaWNfdmVyc2lvbgABDwJg
      2021-04-26T15:11:01.994057+00:00 db2.mydomain.example mysqld[21482]: AAC5oiS8
      2021-04-26T15:11:01.994171+00:00 db2.mydomain.example mysqld[21482]: S8WCYBgBAAAAPAAAAGVBBAAAADUAAAAAAAEAAf///gtjNmMxMTI5OTJjOf4LNWZmY2VlYmZhZGHQ
      2021-04-26T15:11:01.994285+00:00 db2.mydomain.example mysqld[21482]: CaZR
      2021-04-26T15:11:01.994404+00:00 db2.mydomain.example mysqld[21482]: '
      2021-04-26T15:11:01.994518+00:00 db2.mydomain.example mysqld[21482]: 2021-04-26 15:11:01 12 [Note] InnoDB: *** Priority TRANSACTION:
      2021-04-26T15:11:01.994631+00:00 db2.mydomain.example mysqld[21482]: TRANSACTION 6470782, ACTIVE 0 sec
      2021-04-26T15:11:01.994761+00:00 db2.mydomain.example mysqld[21482]: 2 lock struct(s), heap size 1128, 1 row lock(s), undo log entries 1
      2021-04-26T15:11:01.994891+00:00 db2.mydomain.example mysqld[21482]: MySQL thread id 2, OS thread handle 140241761765120, query id 5 committing
      2021-04-26T15:11:01.995017+00:00 db2.mydomain.example mysqld[21482]: 2021-04-26 15:11:01 12 [Note] InnoDB: *** WAITING FOR THIS LOCK TO BE GRANTED:
      2021-04-26T15:11:01.995132+00:00 db2.mydomain.example mysqld[21482]: RECORD LOCKS space id 13788 page no 3 n bits 72 index GEN_CLUST_INDEX of table `neutron`.`alembic_version` trx id 6470782 lock_mode X locks rec but not gap
      2021-04-26T15:11:01.995259+00:00 db2.mydomain.example mysqld[21482]: Record lock, heap no 2
      2021-04-26T15:11:01.995386+00:00 db2.mydomain.example mysqld[21482]: 2021-04-26 15:11:01 12 [Note] InnoDB:  SQL1: BINLOG '
      2021-04-26T15:11:01.995504+00:00 db2.mydomain.example mysqld[21482]: S8WCYBMBAAAAPwAAAClBBAAAADUAAAAAAAEAB25ldXRyb24AD2FsZW1iaWNfdmVyc2lvbgABDwJg
      2021-04-26T15:11:01.995623+00:00 db2.mydomain.example mysqld[21482]: AAC5oiS8
      2021-04-26T15:11:01.995744+00:00 db2.mydomain.example mysqld[21482]: S8WCYBgBAAAAPAAAAGVBBAAAADUAAAAAAAEAAf///gtjNmMxMTI5OTJjOf4LNWZmY2VlYmZhZGHQ
      2021-04-26T15:11:01.995857+00:00 db2.mydomain.example mysqld[21482]: CaZR
      2021-04-26T15:11:01.995977+00:00 db2.mydomain.example mysqld[21482]: 'K305202`^S^A
      2021-04-26T15:11:01.996115+00:00 db2.mydomain.example mysqld[21482]: 2021-04-26 15:11:01 12 [Note] InnoDB:  SQL2: NULL
      2021-04-26T15:11:01.996231+00:00 db2.mydomain.example mysqld[21482]: 2021-04-26 15:11:01 12 [Note] WSREP: cluster conflict due to high priority abort for threads:
      2021-04-26T15:11:01.996351+00:00 db2.mydomain.example mysqld[21482]: 2021-04-26 15:11:01 12 [Note] WSREP: Winning thread:
      2021-04-26T15:11:01.996471+00:00 db2.mydomain.example mysqld[21482]:    THD: 12, mode: high priority, state: exec, conflict: executing, seqno: 2294786
      2021-04-26T15:11:01.996584+00:00 db2.mydomain.example mysqld[21482]:    SQL: BINLOG '
      2021-04-26T15:11:01.996700+00:00 db2.mydomain.example mysqld[21482]: S8WCYBMBAAAAPwAAAClBBAAAADUAAAAAAAEAB25ldXRyb24AD2FsZW1iaWNfdmVyc2lvbgABDwJg
      2021-04-26T15:11:01.996813+00:00 db2.mydomain.example mysqld[21482]: AAC5oiS8
      2021-04-26T15:11:01.996932+00:00 db2.mydomain.example mysqld[21482]: S8WCYBgBAAAAPAAAAGVBBAAAADUAAAAAAAEAAf///gtjNmMxMTI5OTJjOf4LNWZmY2VlYmZhZGHQ
      2021-04-26T15:11:01.997046+00:00 db2.mydomain.example mysqld[21482]: CaZR
      2021-04-26T15:11:01.997166+00:00 db2.mydomain.example mysqld[21482]: 'K305202`^S^A
      2021-04-26T15:11:01.997284+00:00 db2.mydomain.example mysqld[21482]: 2021-04-26 15:11:01 12 [Note] WSREP: Victim thread:
      2021-04-26T15:11:01.997399+00:00 db2.mydomain.example mysqld[21482]:    THD: 2, mode: high priority, state: exec, conflict: committing, seqno: 2294788
      2021-04-26T15:11:01.997514+00:00 db2.mydomain.example mysqld[21482]:    SQL: NULL
      2021-04-26T15:11:01.997632+00:00 db2.mydomain.example mysqld[21482]: 2021-04-26 15:11:01 12 [Note] WSREP: context: /home/abuild/rpmbuild/BUILD/mariadb-10.4.17/storage/innobase/handler/ha_innodb.cc:18632
      2021-04-26T15:11:01.997749+00:00 db2.mydomain.example mysqld[21482]: mysqld: /home/abuild/rpmbuild/BUILD/mariadb-10.4.17/wsrep-lib/include/wsrep/client_state.hpp:603: int wsrep::client_state::bf_abort(wsrep::seqno): Assertion `mode_ == m_local || transaction_.is_streaming()' failed.
      2021-04-26T15:11:01.997864+00:00 db2.mydomain.example mysqld[21482]: 210426 15:11:01 [ERROR] mysqld got signal 6 ;
      2021-04-26T15:11:01.997982+00:00 db2.mydomain.example mysqld[21482]: This could be because you hit a bug. It is also possible that this binary
      2021-04-26T15:11:01.998098+00:00 db2.mydomain.example mysqld[21482]: or one of the libraries it was linked against is corrupt, improperly built,
      2021-04-26T15:11:01.998212+00:00 db2.mydomain.example mysqld[21482]: or misconfigured. This error can also be caused by malfunctioning hardware.
      2021-04-26T15:11:01.998327+00:00 db2.mydomain.example mysqld[21482]:
      2021-04-26T15:11:01.998441+00:00 db2.mydomain.example mysqld[21482]: To report this bug, see https://mariadb.com/kb/en/reporting-bugs
      2021-04-26T15:11:01.998555+00:00 db2.mydomain.example mysqld[21482]:
      2021-04-26T15:11:01.998669+00:00 db2.mydomain.example mysqld[21482]: We will try our best to scrape up some info that will hopefully help
      2021-04-26T15:11:01.998788+00:00 db2.mydomain.example mysqld[21482]: diagnose the problem, but since we have already crashed,
      2021-04-26T15:11:01.998903+00:00 db2.mydomain.example mysqld[21482]: something is definitely wrong and this may fail.
      2021-04-26T15:11:01.999016+00:00 db2.mydomain.example mysqld[21482]:
      2021-04-26T15:11:01.999130+00:00 db2.mydomain.example mysqld[21482]: Server version: 10.4.17-MariaDB
      2021-04-26T15:11:01.999243+00:00 db2.mydomain.example mysqld[21482]: key_buffer_size=67108864
      2021-04-26T15:11:01.999357+00:00 db2.mydomain.example mysqld[21482]: read_buffer_size=131072
      2021-04-26T15:11:01.999470+00:00 db2.mydomain.example mysqld[21482]: max_used_connections=0
      2021-04-26T15:11:01.999590+00:00 db2.mydomain.example mysqld[21482]: max_threads=10002
      2021-04-26T15:11:01.999705+00:00 db2.mydomain.example mysqld[21482]: thread_count=11
      2021-04-26T15:11:01.999819+00:00 db2.mydomain.example mysqld[21482]: It is possible that mysqld could use up to
      2021-04-26T15:11:01.999932+00:00 db2.mydomain.example mysqld[21482]: key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 22077178 K  bytes of memory
      2021-04-26T15:11:02.000064+00:00 db2.mydomain.example mysqld[21482]: Hope that's ok; if not, decrease some variables in the equation.
      2021-04-26T15:11:02.000181+00:00 db2.mydomain.example mysqld[21482]:
      2021-04-26T15:11:02.000294+00:00 db2.mydomain.example mysqld[21482]: Thread pointer: 0x7f89e4000bf8
      2021-04-26T15:11:02.000409+00:00 db2.mydomain.example mysqld[21482]: Attempting backtrace. You can use the following information to find out
      2021-04-26T15:11:02.000548+00:00 db2.mydomain.example mysqld[21482]: where mysqld died. If you see no messages after this, something went
      2021-04-26T15:11:02.000664+00:00 db2.mydomain.example mysqld[21482]: terribly wrong...
      2021-04-26T15:11:02.000779+00:00 db2.mydomain.example mysqld[21482]: stack_bottom = 0x7f8c840fed08 thread_stack 0x49000
      2021-04-26T15:11:02.014312+00:00 db2.mydomain.example mysqld[21482]: /usr/sbin/mysqld(my_print_stacktrace+0x3c)[0x55fcbd6027cc]
      2021-04-26T15:11:02.014890+00:00 db2.mydomain.example mysqld[21482]: /usr/sbin/mysqld(handle_fatal_signal+0x535)[0x55fcbd119f05]
      2021-04-26T15:11:02.028285+00:00 db2.mydomain.example mysqld[21482]: /lib64/libpthread.so.0(+0x132d0)[0x7f8c99aab2d0]
      2021-04-26T15:11:02.041285+00:00 db2.mydomain.example mysqld[21482]: /lib64/libc.so.6(gsignal+0x110)[0x7f8c981dc520]
      2021-04-26T15:11:02.041518+00:00 db2.mydomain.example mysqld[21482]: /lib64/libc.so.6(abort+0x151)[0x7f8c981ddb01]
      2021-04-26T15:11:02.041640+00:00 db2.mydomain.example mysqld[21482]: /lib64/libc.so.6(+0x31b1a)[0x7f8c981d4b1a]
      2021-04-26T15:11:02.041765+00:00 db2.mydomain.example mysqld[21482]: /lib64/libc.so.6(+0x31b92)[0x7f8c981d4b92]
      2021-04-26T15:11:02.055417+00:00 db2.mydomain.example mysqld[21482]: /usr/sbin/mysqld(_Z14wsrep_bf_abortPK3THDPS_+0x4a7)[0x55fcbd090a27]
      2021-04-26T15:11:02.055927+00:00 db2.mydomain.example mysqld[21482]: /usr/sbin/mysqld(wsrep_thd_bf_abort+0x2c)[0x55fcbd097efc]
      2021-04-26T15:11:02.056560+00:00 db2.mydomain.example mysqld[21482]: /usr/sbin/mysqld(+0xa1c4d8)[0x55fcbd2ad4d8]
      2021-04-26T15:11:02.057158+00:00 db2.mydomain.example mysqld[21482]: /usr/sbin/mysqld(+0xa5bf28)[0x55fcbd2ecf28]
      2021-04-26T15:11:02.057757+00:00 db2.mydomain.example mysqld[21482]: /usr/sbin/mysqld(+0xa5fe22)[0x55fcbd2f0e22]
      2021-04-26T15:11:02.058374+00:00 db2.mydomain.example mysqld[21482]: /usr/sbin/mysqld(+0xa659f1)[0x55fcbd2f69f1]
      2021-04-26T15:11:02.059071+00:00 db2.mydomain.example mysqld[21482]: /usr/sbin/mysqld(+0xaf071f)[0x55fcbd38171f]
      2021-04-26T15:11:02.059626+00:00 db2.mydomain.example mysqld[21482]: /usr/sbin/mysqld(+0xaf5e8d)[0x55fcbd386e8d]
      2021-04-26T15:11:02.060132+00:00 db2.mydomain.example mysqld[21482]: /usr/sbin/mysqld(+0xa25033)[0x55fcbd2b6033]
      2021-04-26T15:11:02.060786+00:00 db2.mydomain.example mysqld[21482]: /usr/sbin/mysqld(+0xa253cb)[0x55fcbd2b63cb]
      2021-04-26T15:11:02.061444+00:00 db2.mydomain.example mysqld[21482]: /usr/sbin/mysqld(_ZN7handler11ha_rnd_nextEPh+0x13f)[0x55fcbd11e9bf]
      2021-04-26T15:11:02.062070+00:00 db2.mydomain.example mysqld[21482]: /usr/sbin/mysqld(_ZN14Rows_log_event8find_rowEP14rpl_group_info+0x4be)[0x55fcbd21152e]
      2021-04-26T15:11:02.062778+00:00 db2.mydomain.example mysqld[21482]: /usr/sbin/mysqld(_ZN21Update_rows_log_event11do_exec_rowEP14rpl_group_info+0xd7)[0x55fcbd211ab7]
      2021-04-26T15:11:02.063406+00:00 db2.mydomain.example mysqld[21482]: /usr/sbin/mysqld(_ZN14Rows_log_event14do_apply_eventEP14rpl_group_info+0x23f)[0x55fcbd204f8f]
      2021-04-26T15:11:02.063950+00:00 db2.mydomain.example mysqld[21482]: /usr/sbin/mysqld(_Z18wsrep_apply_eventsP3THDP14Relay_log_infoPKvm+0x1d4)[0x55fcbd08f544]
      2021-04-26T15:11:02.064562+00:00 db2.mydomain.example mysqld[21482]: /usr/sbin/mysqld(_ZN21Wsrep_applier_service15apply_write_setERKN5wsrep7ws_metaERKNS0_12const_bufferERNS0_14mutable_bufferE+0x9a)[0x55fcbd07846a]
      2021-04-26T15:11:02.065161+00:00 db2.mydomain.example mysqld[21482]: /usr/sbin/mysqld(+0xdde375)[0x55fcbd66f375]
      2021-04-26T15:11:02.065811+00:00 db2.mydomain.example mysqld[21482]: /usr/sbin/mysqld(_ZN5wsrep12server_state8on_applyERNS_21high_priority_serviceERKNS_9ws_handleERKNS_7ws_metaERKNS_12const_bufferE+0x195)[0x55fcbd66fde5]
      2021-04-26T15:11:02.066320+00:00 db2.mydomain.example mysqld[21482]: /usr/sbin/mysqld(+0xdf4749)[0x55fcbd685749]
      2021-04-26T15:11:02.078445+00:00 db2.mydomain.example mysqld[21482]: /usr/lib64/galera-4/libgalera_smm.so(+0x19f93e)[0x7f8c9693893e]
      2021-04-26T15:11:02.078649+00:00 db2.mydomain.example mysqld[21482]: /usr/lib64/galera-4/libgalera_smm.so(+0x1f0f48)[0x7f8c96989f48]
      2021-04-26T15:11:02.078796+00:00 db2.mydomain.example mysqld[21482]: /usr/lib64/galera-4/libgalera_smm.so(+0x2048f2)[0x7f8c9699d8f2]
      2021-04-26T15:11:02.078909+00:00 db2.mydomain.example mysqld[21482]: /usr/lib64/galera-4/libgalera_smm.so(+0x207347)[0x7f8c969a0347]
      2021-04-26T15:11:02.079024+00:00 db2.mydomain.example mysqld[21482]: /usr/lib64/galera-4/libgalera_smm.so(+0x1f3a23)[0x7f8c9698ca23]
      2021-04-26T15:11:02.079150+00:00 db2.mydomain.example mysqld[21482]: /usr/lib64/galera-4/libgalera_smm.so(+0x2119bb)[0x7f8c969aa9bb]
      2021-04-26T15:11:02.091078+00:00 db2.mydomain.example mysqld[21482]: /usr/sbin/mysqld(_ZN5wsrep18wsrep_provider_v2611run_applierEPNS_21high_priority_serviceE+0xe)[0x55fcbd688b2e]
      2021-04-26T15:11:02.091596+00:00 db2.mydomain.example mysqld[21482]: /usr/sbin/mysqld(+0x8004db)[0x55fcbd0914db]
      2021-04-26T15:11:02.092123+00:00 db2.mydomain.example mysqld[21482]: /usr/sbin/mysqld(_Z15start_wsrep_THDPv+0x439)[0x55fcbd0821a9]
      2021-04-26T15:11:02.103934+00:00 db2.mydomain.example mysqld[21482]: /lib64/libpthread.so.0(+0x84f9)[0x7f8c99aa04f9]
      2021-04-26T15:11:02.115539+00:00 db2.mydomain.example mysqld[21482]: /lib64/libc.so.6(clone+0x3f)[0x7f8c9829efbf]
      2021-04-26T15:11:02.115757+00:00 db2.mydomain.example mysqld[21482]:
      2021-04-26T15:11:02.115906+00:00 db2.mydomain.example mysqld[21482]: Trying to get some variables.
      2021-04-26T15:11:02.116042+00:00 db2.mydomain.example mysqld[21482]: Some pointers may be invalid and cause the dump to abort.
      2021-04-26T15:11:02.116164+00:00 db2.mydomain.example mysqld[21482]: Query (0x7f8c8945f273): BINLOG '
      2021-04-26T15:11:02.116289+00:00 db2.mydomain.example mysqld[21482]: S8WCYBMBAAAAPwAAAClBBAAAADUAAAAAAAEAB25ldXRyb24AD2FsZW1iaWNfdmVyc2lvbgABDwJg
      2021-04-26T15:11:02.116440+00:00 db2.mydomain.example mysqld[21482]: AAC5oiS8
      2021-04-26T15:11:02.116549+00:00 db2.mydomain.example mysqld[21482]: S8WCYBgBAAAAPAAAAGVBBAAAADUAAAAAAAEAAf///gtjNmMxMTI5OTJjOf4LNWZmY2VlYmZhZGHQ
      2021-04-26T15:11:02.116666+00:00 db2.mydomain.example mysqld[21482]: CaZR
      2021-04-26T15:11:02.116773+00:00 db2.mydomain.example mysqld[21482]: '
      2021-04-26T15:11:02.116906+00:00 db2.mydomain.example mysqld[21482]:
      2021-04-26T15:11:02.117014+00:00 db2.mydomain.example mysqld[21482]: Connection ID (thread ID): 12
      2021-04-26T15:11:02.117121+00:00 db2.mydomain.example mysqld[21482]: Status: NOT_KILLED
      2021-04-26T15:11:02.117235+00:00 db2.mydomain.example mysqld[21482]:
      2021-04-26T15:11:02.117352+00:00 db2.mydomain.example mysqld[21482]: Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=o>
      2021-04-26T15:11:02.117466+00:00 db2.mydomain.example mysqld[21482]:
      2021-04-26T15:11:02.117586+00:00 db2.mydomain.example mysqld[21482]: The manual page at https://mariadb.com/kb/en/how-to-produce-a-full-stack-trace-for-mysqld/ contains
      2021-04-26T15:11:02.117706+00:00 db2.mydomain.example mysqld[21482]: information that should help you find out what is causing the crash.
      2021-04-26T15:11:02.117818+00:00 db2.mydomain.example mysqld[21482]: Writing a core file...
      2021-04-26T15:11:02.117926+00:00 db2.mydomain.example mysqld[21482]: Working directory at /var/lib/mysql
      2021-04-26T15:11:02.118041+00:00 db2.mydomain.example mysqld[21482]: Resource Limits:
      2021-04-26T15:11:02.118153+00:00 db2.mydomain.example mysqld[21482]: Limit                     Soft Limit           Hard Limit           Units
      2021-04-26T15:11:02.118262+00:00 db2.mydomain.example mysqld[21482]: Max cpu time              unlimited            unlimited            seconds
      2021-04-26T15:11:02.118370+00:00 db2.mydomain.example mysqld[21482]: Max file size             unlimited            unlimited            bytes
      2021-04-26T15:11:02.118491+00:00 db2.mydomain.example mysqld[21482]: Max data size             unlimited            unlimited            bytes
      2021-04-26T15:11:02.118619+00:00 db2.mydomain.example mysqld[21482]: Max stack size            8388608              unlimited            bytes
      2021-04-26T15:11:02.118754+00:00 db2.mydomain.example mysqld[21482]: Max core file size        unlimited            unlimited            bytes
      2021-04-26T15:11:02.118872+00:00 db2.mydomain.example mysqld[21482]: Max resident set          unlimited            unlimited            bytes
      2021-04-26T15:11:02.118987+00:00 db2.mydomain.example mysqld[21482]: Max processes             unlimited            unlimited            processes
      2021-04-26T15:11:02.119100+00:00 db2.mydomain.example mysqld[21482]: Max open files            1048576              1048576              files
      2021-04-26T15:11:02.119234+00:00 db2.mydomain.example mysqld[21482]: Max locked memory         65536                65536                bytes
      2021-04-26T15:11:02.119357+00:00 db2.mydomain.example mysqld[21482]: Max address space         unlimited            unlimited            bytes
      2021-04-26T15:11:02.119479+00:00 db2.mydomain.example mysqld[21482]: Max file locks            unlimited            unlimited            locks
      2021-04-26T15:11:02.119600+00:00 db2.mydomain.example mysqld[21482]: Max pending signals       125259               125259               signals
      2021-04-26T15:11:02.119794+00:00 db2.mydomain.example mysqld[21482]: Max msgqueue size         819200               819200               bytes
      2021-04-26T15:11:02.119917+00:00 db2.mydomain.example mysqld[21482]: Max nice priority         0                    0
      2021-04-26T15:11:02.120061+00:00 db2.mydomain.example mysqld[21482]: Max realtime priority     0                    0
      2021-04-26T15:11:02.120329+00:00 db2.mydomain.example mysqld[21482]: Max realtime timeout      unlimited            unlimited            us
      2021-04-26T15:11:02.120448+00:00 db2.mydomain.example mysqld[21482]: Core pattern: |/usr/sbin/process-dump coredump %h core.%h.%t.%e.%p.gz
      

      We are using 10.4.17 with MDEV-23851 MDEV-24229 BF-BF conflict issues, and MDEV-23851 BF-BF Conflict issue because of UK GAP locks patches applied.

      We are aware of MDEV-24923 and created a build from 10.4 (https://github.com/MariaDB/server/commit/ee455e6f2e6c9cda921c0801210786123d9f8b95) but this initial crash is still reproducible. However, the subsequent crashes during the recovery phase, after restarting the failed instance seems to be fixed.
      I managed to isolate and trigger the fault with the following simple SQL statements:

      CREATE TABLE t1 (f1 VARCHAR(32) NOT NULL) ENGINE=InnoDB;
      INSERT INTO t1 (f1) VALUES ('0e66c5227a8a');
      INSERT INTO t1 (f1) VALUES ('c6c112992c9');
       
      START TRANSACTION;
      UPDATE t1 SET f1='5ffceebfada' WHERE t1.f1 = 'c6c112992c9';
      COMMIT;
       
      START TRANSACTION;
      DELETE FROM t1 WHERE t1.f1 = '5ffceebfada';
      COMMIT;
       
      DROP TABLE t1;
      

      Sometimes it needs couple of thousand executions to observe the fault.

      My theory about the fault is that in some cases it is possible that an applier thread start processing the transaction with the DELETE statement before another applier managed to execute the UPDATE one.
      When this latter thread reaches the actual UPDATE statement, it tries to aquire a lock that is being held by the transaction executed in the former thread which is at COMMIT stage and that causes a lock conflict between these two priority transactions. The lock holder transaction got aborted (I guess due to the higher sequence number) but instead of doing an expected rollback, wait and replay attempt it initiates a controlled crash because the assertion criteria hasn't met.

      Right now, I'm trying to get familiar with mtr to implement this aforementioned scenario in a test case to have a quick and stable reproducer.

        Attachments

          Issue Links

            Activity

              People

              Assignee:
              jplindst Jan Lindström
              Reporter:
              TheGOro Gabor Orosz
              Votes:
              3 Vote for this issue
              Watchers:
              10 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved: