[MDEV-20640] [ERROR] InnoDB: tried to purge non-delete-marked record and Assertion `0' failed in row_purge_remove_sec_if_poss_leaf Created: 2019-09-20  Updated: 2024-01-02  Resolved: 2024-01-02

Status: Closed
Project: MariaDB Server
Component/s: Locking, Storage Engine - InnoDB, Virtual Columns
Affects Version/s: 10.3, 10.4, 10.5, 10.6, 10.7, 10.8, 10.9, 10.10
Fix Version/s: N/A

Type: Bug Priority: Critical
Reporter: Elena Stepanova Assignee: Nikita Malyavin
Resolution: Cannot Reproduce Votes: 2
Labels: corruption

Issue Links:
Relates
relates to MDEV-16222 Assertion `0' failed in row_purge_rem... Closed
relates to MDEV-26951 gcol.innodb_virtual_basic fails with ... Stalled
relates to MDEV-29181 Potential corruption on Foreign key u... In Review
relates to MDEV-29666 InnoDB fails to purge secondary index... Closed
relates to MDEV-29068 Cascade foreign key updates do not ap... Closed
relates to MDEV-29181 Potential corruption on Foreign key u... In Review

 Description   

Note: On some reason, it's not reproducible for me on ASAN builds of the same revision, only on non-ASAN debug.

--source include/have_innodb.inc
 
CREATE TABLE t1 ( 
    pk INT AUTO_INCREMENT,
    a BIT(10),
    b BIT(10) AS (a) VIRTUAL,
    PRIMARY KEY(pk),
    UNIQUE(b)
) ENGINE=InnoDB;
 
INSERT INTO t1 (a) VALUES (b'110'),(b'1'),(b'011');
 
SELECT pk, a INTO OUTFILE 'load.data' FROM t1;
LOAD DATA INFILE 'load.data' REPLACE INTO TABLE t1 (pk, a);
ALTER TABLE t1 ADD COLUMN c INT;
 
# Cleanup
DROP TABLE t1;
--let $datadir= `SELECT @@datadir`
--remove_file $datadir/test/load.data

10.3 debug 90a9c4ca

2019-09-20 21:04:21 3 [ERROR] InnoDB: tried to purge non-delete-marked record in index `b` of table `test`.`t1`: tuple: TUPLE (info_bits=0, 2 fields): {[2]  (0x0006),[4]    (0x80000001)}, record: COMPACT RECORD(info_bi
ts=0, 2 fields): {[2]  (0x0006),[4]    (0x80000001)}
mysqld: /data/src/10.3/storage/innobase/row/row0purge.cc:596: bool row_purge_remove_sec_if_poss_leaf(purge_node_t*, dict_index_t*, const dtuple_t*): Assertion `0' failed.
190920 21:04:21 [ERROR] mysqld got signal 6 ;
 
#7  0x00007f8bc61ccf12 in __GI___assert_fail (assertion=0x555c1df9c761 "0", file=0x555c1df9cf08 "/data/src/10.3/storage/innobase/row/row0purge.cc", line=596, function=0x555c1df9e4c0 <row_purge_remove_sec_if_poss_leaf(purge_node_t*, dict_index_t*, dtuple_t const*)::__PRETTY_FUNCTION__> "bool row_purge_remove_sec_if_poss_leaf(purge_node_t*, dict_index_t*, const dtuple_t*)") at assert.c:101
#8  0x0000555c1d8087e6 in row_purge_remove_sec_if_poss_leaf (node=0x555c201a8138, index=0x7f8b6c070a78, entry=0x7f8b90008d88) at /data/src/10.3/storage/innobase/row/row0purge.cc:596
#9  0x0000555c1d808b6c in row_purge_remove_sec_if_poss (node=0x555c201a8138, index=0x7f8b6c070a78, entry=0x7f8b90008d88) at /data/src/10.3/storage/innobase/row/row0purge.cc:692
#10 0x0000555c1d808d94 in row_purge_del_mark (node=0x555c201a8138) at /data/src/10.3/storage/innobase/row/row0purge.cc:766
#11 0x0000555c1d80a96f in row_purge_record_func (node=0x555c201a8138, undo_rec=0x555c201a86e8 "", thr=0x555c201a7f68, updated_extern=false) at /data/src/10.3/storage/innobase/row/row0purge.cc:1191
#12 0x0000555c1d80ac86 in row_purge (node=0x555c201a8138, undo_rec=0x555c201a86e8 "", thr=0x555c201a7f68) at /data/src/10.3/storage/innobase/row/row0purge.cc:1258
#13 0x0000555c1d80aede in row_purge_step (thr=0x555c201a7f68) at /data/src/10.3/storage/innobase/row/row0purge.cc:1317
#14 0x0000555c1d777d50 in que_thr_step (thr=0x555c201a7f68) at /data/src/10.3/storage/innobase/que/que0que.cc:1037
#15 0x0000555c1d777fe9 in que_run_threads_low (thr=0x555c201a7f68) at /data/src/10.3/storage/innobase/que/que0que.cc:1099
#16 0x0000555c1d778239 in que_run_threads (thr=0x555c201a7f68) at /data/src/10.3/storage/innobase/que/que0que.cc:1139
#17 0x0000555c1d8613e3 in srv_task_execute () at /data/src/10.3/storage/innobase/srv/srv0srv.cc:2473
#18 0x0000555c1d8615fb in srv_worker_thread (arg=0x0) at /data/src/10.3/storage/innobase/srv/srv0srv.cc:2521
#19 0x00007f8bc7d414a4 in start_thread (arg=0x7f8ba37fe700) at pthread_create.c:456
#20 0x00007f8bc6289d0f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97

Couldn't reproduce on 10.2 or 10.4.
As mentioned above, couldn't reproduce on ASAN build (although it might be just the matter of luck) or non-debug build.



 Comments   
Comment by jocelyn fournier [ 2019-12-13 ]

Hi!

Is it the same bug than https://bugs.mysql.com/bug.php?id=86485 ?
I've just hit in 10.2.24 a similar bug, after a DELETE on the whole table (20M+ records) following by an insert from mysqldump (hence with index disabled)
I guess it has something to do with the INSERT buffer not being fully applied when trying to insert data (the same than before the DELETE in my case) with disabled indexes.

2019-12-12 19:08:42 140504956139264 [ERROR] InnoDB: tried to purge non-delete-marked record in index `idx_flag` of table `sc_2`.`sc_product`: tuple: TUPLE (info_bits=0, 4 fields): {[1] (0x03),[4]    (0x00000000),[1] (0x06),[4]    (0x0112D1D0)}, record: COMPACT RECORD(info_bits=0, 4 fields): {[1] (0x03),[4]    (0x00000000),[1] (0x06),[4]    (0x0112D1D0)}
2019-12-12 19:08:50 0x7fc9903dd700  InnoDB: Assertion failure in file /home/buildbot/buildbot/build/mariadb-10.2.24/storage/innobase/row/row0ins.cc line 268
InnoDB: Failing assertion: !cursor->index->is_committed()
InnoDB: We intentionally generate a memory trap.
InnoDB: Submit a detailed bug report to https://jira.mariadb.org/
InnoDB: If you get repeated assertion failures or crashes, even
InnoDB: immediately after the mysqld startup, there may be
InnoDB: corruption in the InnoDB tablespace. Please refer to
InnoDB: https://mariadb.com/kb/en/library/innodb-recovery-modes/
InnoDB: about forcing recovery.
191212 19:08:50 [ERROR] mysqld got signal 6 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
 
To report this bug, see https://mariadb.com/kb/en/reporting-bugs
 
We will try our best to scrape up some info that will hopefully help
diagnose the problem, but since we have already crashed, 
something is definitely wrong and this may fail.
 
Server version: 10.2.24-MariaDB-1:10.2.24+maria~bionic-log
key_buffer_size=0
read_buffer_size=2097152
max_used_connections=670
max_threads=1502
thread_count=152
It is possible that mysqld could use up to 
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 6184193 K  bytes of memory
Hope that's ok; if not, decrease some variables in the equation.
 
Thread pointer: 0x7fc66827c488
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 0x7fc9903dcdd8 thread_stack 0x49000
/usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x56395bc503ae]
/usr/sbin/mysqld(handle_fatal_signal+0x513)[0x56395b6ce3a3]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x12890)[0x7fe42d590890]
/lib/x86_64-linux-gnu/libc.so.6(gsignal+0xc7)[0x7fe42caa4e97]
/lib/x86_64-linux-gnu/libc.so.6(abort+0x141)[0x7fe42caa6801]
/usr/sbin/mysqld(+0x410385)[0x56395b480385]
/usr/sbin/mysqld(+0x899160)[0x56395b909160]
/usr/sbin/mysqld(+0x89c0f3)[0x56395b90c0f3]
/usr/sbin/mysqld(+0x89c5d6)[0x56395b90c5d6]
/usr/sbin/mysqld(+0x8ac326)[0x56395b91c326]
/usr/sbin/mysqld(+0x7efc7c)[0x56395b85fc7c]
/usr/sbin/mysqld(_ZN7handler12ha_write_rowEPh+0x157)[0x56395b6d8297]
/usr/sbin/mysqld(_Z12write_recordP3THDP5TABLEP12st_copy_info+0x5d8)[0x56395b511798]
/usr/sbin/mysqld(_Z12mysql_insertP3THDP10TABLE_LISTR4ListI4ItemERS3_IS5_ES6_S6_15enum_duplicatesb+0x123f)[0x56395b51b92f]
/usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x169f)[0x56395b52f1df]
/usr/sbin/mysqld(_Z11mysql_parseP3THDPcjP12Parser_statebb+0x2d2)[0x56395b5365f2]
/usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcjbb+0x1fb5)[0x56395b539435]
/usr/sbin/mysqld(_Z10do_commandP3THD+0x180)[0x56395b53a060]
/usr/sbin/mysqld(_Z24do_handle_one_connectionP7CONNECT+0x20a)[0x56395b603b4a]
/usr/sbin/mysqld(handle_one_connection+0x3d)[0x56395b603d1d]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x76db)[0x7fe42d5856db]
/lib/x86_64-linux-gnu/libc.so.6(clone+0x3f)[0x7fe42cb8788f]
 
Trying to get some variables.
Some pointers may be invalid and cause the dump to abort.
Query (0x7fc4d0622d80): INSERT  IGNORE INTO `sc_product` VALUES (18007916,NULL,NULL,8,'Mary, Did You Know?',NULL,NULL,0,3,'2015-11-07 17:07:07','2017-09-28 20:59:27',0,6,3)

not sure if I should open a separated JIRA for this one?

Jocelyn

Comment by Elena Stepanova [ 2020-11-01 ]

gcol.innodb_virtual_fk failed in buildbot with what appears to be the same problem:
http://buildbot.askmonty.org/buildbot/builders/win32-debug/builds/18380

bb-10.4-release 1565829f76aa2c0ac7b91a76198bb825

gcol.innodb_virtual_fk 'innodb'          w1 [ fail ]
        Test ended at 2020-10-31 19:47:37
 
CURRENT_TEST: gcol.innodb_virtual_fk
mysqltest: At line 639: query 'DROP TABLE child, parent' failed: 2013: Lost connection to MySQL server during query
 
The result from queries just before the failure was:
< snip >
CREATE TABLE child
(
ID int unsigned NOT NULL,
ParentID int unsigned NULL,
Value int unsigned NOT NULL DEFAULT 0,
Flag int unsigned AS (Value) VIRTUAL,
PRIMARY KEY (ID),
KEY (ParentID, Flag),
FOREIGN KEY (ParentID) REFERENCES parent (ID) ON DELETE SET NULL
ON UPDATE CASCADE
);
INSERT INTO parent (ID) VALUES (100);
INSERT INTO child (ID,ParentID,Value) VALUES (123123,100,1);
DELETE FROM parent WHERE ID=100;
select * from child;
ID	ParentID	Value	Flag
123123	NULL	1	1
INSERT INTO parent (ID) VALUES (100);
UPDATE child SET ParentID=100 WHERE ID=123123;
DROP TABLE child, parent;
 
More results from queries before failure can be found in D:\win32-debug\build\mysql-test\var\1\log\innodb_virtual_fk.log
 
 
Server [mysqld.1 - pid: 5016, winpid: 5016, exit: 768] failed during test run
Server log from this test:
----------SERVER LOG START-----------
2020-10-31 19:47:36 1 [ERROR] InnoDB: tried to purge non-delete-marked record in index `ParentID` of table `test`.`child`: tuple: TUPLE (info_bits=0, 3 fields): {[4]   d(0x00000064),[4]    (0x00000001),[4]    (0x0001E0F3)}, record: COMPACT RECORD(info_bits=0, 3 fields): {[4]   d(0x00000064),[4]    (0x00000001),[4]    (0x0001E0F3)}
Assertion failed: 0, file D:\win32-debug\build\src\storage\innobase\row\row0purge.cc, line 599
201031 19:47:36 [ERROR] mysqld got exception 0x80000003 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
 
To report this bug, see https://mariadb.com/kb/en/reporting-bugs
 
We will try our best to scrape up some info that will hopefully help
diagnose the problem, but since we have already crashed, 
something is definitely wrong and this may fail.
 
Server version: 10.4.16-MariaDB-debug-log
key_buffer_size=1048576
read_buffer_size=131072
max_used_connections=2
max_threads=65537
thread_count=7
It is possible that mysqld could use up to 
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 3634 K  bytes of memory
Hope that's ok; if not, decrease some variables in the equation.
 
Thread pointer: 0xb301b90
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
mysqld.exe!my_sigabrt_handler()[my_thr_init.c:485]
ucrtbase.dll!raise()
ucrtbase.dll!abort()
ucrtbase.dll!_get_wpgmptr()
ucrtbase.dll!_get_wpgmptr()
ucrtbase.dll!_wassert()
mysqld.exe!row_purge_remove_sec_if_poss_leaf()[row0purge.cc:599]
mysqld.exe!row_purge_remove_sec_if_poss()[row0purge.cc:695]
mysqld.exe!row_purge_upd_exist_or_extern_func()[row0purge.cc:897]
mysqld.exe!row_purge_record_func()[row0purge.cc:1217]
mysqld.exe!row_purge()[row0purge.cc:1261]
mysqld.exe!row_purge_step()[row0purge.cc:1320]
mysqld.exe!que_thr_step()[que0que.cc:1039]
mysqld.exe!que_run_threads_low()[que0que.cc:1101]
mysqld.exe!que_run_threads()[que0que.cc:1141]
mysqld.exe!trx_purge()[trx0purge.cc:1322]
mysqld.exe!srv_do_purge()[srv0srv.cc:2610]
mysqld.exe!srv_purge_coordinator_thread()[srv0srv.cc:2743]
KERNEL32.DLL!BaseThreadInitThunk()
ntdll.dll!RtlGetAppContainerNamedObjectPath()
ntdll.dll!RtlGetAppContainerNamedObjectPath()
 
Trying to get some variables.
Some pointers may be invalid and cause the dump to abort.
Query (0x0): 
Connection ID (thread ID): 1
Status: NOT_KILLED
 
Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=on,condition_pushdown_from_having=on
 
The manual page at https://mariadb.com/kb/en/how-to-produce-a-full-stack-trace-for-mysqld/ contains
information that should help you find out what is causing the crash.
Writing a core file at D:\win32-debug\build\mysql-test\var\1\mysqld.1\data\
Minidump written to D:\win32-debug\build\mysql-test\var\1\mysqld.1\data\mysqld.dmp
----------SERVER LOG END-------------
 
 
 - found 'mysqld.dmp' (0/5)
 
Trying 'cdb' to get a backtrace
Output from cdb follows. Faulting thread is printed twice,with and without function parameters
Search for STACK_TEXT to see the stack trace of 
the faulting thread. Callstacks of other threads are printed after it.
 
Microsoft (R) Windows Debugger Version 10.0.14321.1024 AMD64
Copyright (c) Microsoft Corporation. All rights reserved.
 
 
Loading Dump File [D:\win32-debug\build\mysql-test\var\1\log\gcol.innodb_virtual_fk-innodb\mysqld.1\data\mysqld.dmp]
User Mini Dump File: Only registers, stack and portions of memory are available
 
 
Response                         Time (ms)     Location
OK                                             C:\Windows\System32
OK                                             D:\win32-debug\build\sql\Debug
OK                                             .
 
Response                         Time (ms)     Location
OK                                             C:\Windows\System32
OK                                             D:\win32-debug\build\sql\Debug
OK                                             .
Deferred                                       SRV*c:\cdb_symbols*http://msdl.microsoft.com/download/symbols
Symbol search path is: C:\Windows\System32;D:\win32-debug\build\sql\Debug;.;SRV*c:\cdb_symbols*http://msdl.microsoft.com/download/symbols
Executable search path is: C:\Windows\System32;D:\win32-debug\build\sql\Debug;.
Windows 10 Version 17763 MP (4 procs) Free x86 compatible
Product: Server, suite: TerminalServer DataCenter SingleUserTS
Built by: 17763.1.x86fre.rs5_release.180914-1434
Machine Name:
Debug session time: Sat Oct 31 19:47:36.000 2020 (UTC + 0:00)
System Uptime: not available
Process Uptime: 0 days 0:00:18.000
....................................
This dump file has a breakpoint exception stored in it.
The stored exception information can be accessed via .ecxr.
eax=00000000 ebx=0f4b7cc0 ecx=00000000 edx=00000000 esi=000002cc edi=0f4b7cb0
eip=77d5136c esp=0c2dcb2c ebp=0c2dcb38 iopl=0         nv up ei pl nz na po nc
cs=0023  ss=002b  ds=002b  es=002b  fs=0053  gs=002b             efl=00000202
ntdll!NtGetContextThread:
         ret     8
0:018> cdb: Reading initial command '!sym prompts off; !analyze -v; .ecxr; !for_each_frame dv /t;!uniqstack -p;q'
quiet mode - symbol prompts off
 
Unable to load image C:\Windows\System32\KERNELBASE.dll, Win32 error 0n2
Unable to load image C:\Windows\System32\user32.dll, Win32 error 0n2
GetUrlPageData2 (WinHttp) failed: 12002.
 
DUMP_CLASS: 2
 
DUMP_QUALIFIER: 400
 
CONTEXT:  (.ecxr)
eax=00000000 ebx=00000008 ecx=00fe92c0 edx=03608000 esi=00000016 edi=00fe92c0
eip=00fe92c3 esp=0c2de80c ebp=0c2de80c iopl=0         nv up ei pl nz ac po nc
cs=0023  ss=002b  ds=002b  es=002b  fs=0053  gs=002b             efl=00000212
mysqld!my_sigabrt_handler:
             int     3
Resetting default scope
 
FAULTING_IP: 
mysqld!my_sigabrt_handler+3 [D:\win32-debug\build\src\mysys\my_thr_init.c @ 485]
             int     3
 
EXCEPTION_RECORD:  (.exr -1)
ExceptionAddress: 00fe92c3 (mysqld!my_sigabrt_handler)
   ExceptionCode: 80000003 (Break instruction exception)
  ExceptionFlags: 00000000
NumberParameters: 1
   Parameter[0]: 00000000
 
DEFAULT_BUCKET_ID:  STATUS_BREAKPOINT
 
PROCESS_NAME:  mysqld.exe
 
ERROR_CODE: (NTSTATUS) 0x80000003 - {EXCEPTION}  Breakpoint  A breakpoint has been reached.
 
EXCEPTION_CODE: (HRESULT) 0x80000003 (2147483651) - One or more arguments are invalid
 
EXCEPTION_CODE_STR:  80000003
 
EXCEPTION_PARAMETER1:  00000000
 
WATSON_BKT_PROCSTAMP:  5f9dbb7e
 
WATSON_BKT_PROCVER:  10.4.16.0
 
WATSON_BKT_MODULE:  mysqld.exe
 
WATSON_BKT_MODSTAMP:  5f9dbb7e
 
WATSON_BKT_MODOFFSET:  dd92c3
 
WATSON_BKT_MODVER:  10.4.16.0
 
BUILD_VERSION_STRING:  17763.1.x86fre.rs5_release.180914-1434
 
MODLIST_WITH_TSCHKSUM_HASH:  07194d4799c4abb08b8eb3015520af528e07aace
 
MODLIST_SHA1_HASH:  34ba4c22fbab5e825ee495efcad53411466925e6
 
DUMP_FLAGS:  0
 
DUMP_TYPE:  2
 
ANALYSIS_SESSION_HOST:  MARIADB-04
 
ANALYSIS_SESSION_TIME:  10-31-2020 19:48:19.0584
 
ANALYSIS_VERSION: 10.0.14321.1024 amd64fre
 
THREAD_ATTRIBUTES: 
PROBLEM_CLASSES: 
 
 
 
 
    Tid    [0x0]
    Frame  [0x00]
    String [STATUS_BREAKPOINT]
    Data Bucketing
 
 
BUGCHECK_STR:  STATUS_BREAKPOINT
 
LAST_CONTROL_TRANSFER:  from 75e13eac to 00fe92c3
 
STACK_TEXT:  
mysqld!my_sigabrt_handler
ucrtbase!raise
ucrtbase!abort
mysqld!row_purge_remove_sec_if_poss_leaf
mysqld!row_purge_remove_sec_if_poss
mysqld!row_purge_upd_exist_or_extern_func
mysqld!row_purge_record_func
mysqld!row_purge
mysqld!row_purge_step
mysqld!que_thr_step
mysqld!que_run_threads_low
mysqld!que_run_threads
mysqld!trx_purge
mysqld!srv_do_purge
mysqld!srv_purge_coordinator_thread
kernel32!BaseThreadInitThunk
ntdll!__RtlUserThreadStart
ntdll!_RtlUserThreadStart
 
 
THREAD_SHA1_HASH_MOD_FUNC:  bc934e3203a69cd9ee603fa9d3a8a474adfcb5f6
 
THREAD_SHA1_HASH_MOD_FUNC_OFFSET:  ee31ac9bd9dbc4b4cf5794ab62b859fdcf4bffe9
 
THREAD_SHA1_HASH_MOD:  d08c78ebfa8ce65f642ffd3a6d5571b639a334b7
 
FOLLOWUP_IP: 
mysqld!my_sigabrt_handler+3 [D:\win32-debug\build\src\mysys\my_thr_init.c @ 485]
             int     3
 
FAULT_INSTR_CODE:  ccc35dcc
 
FAULTING_SOURCE_LINE:  D:\win32-debug\build\src\mysys\my_thr_init.c
 
FAULTING_SOURCE_FILE:  D:\win32-debug\build\src\mysys\my_thr_init.c
 
FAULTING_SOURCE_LINE_NUMBER:  485
 
FAULTING_SOURCE_CODE:  
   481: 
   482: #if (_MSC_VER >= 1400)
   483: static void my_sigabrt_handler(int sig)
   484: {
>  485:   __debugbreak();
   486: }
   487: #endif /*_MSC_VER >=1400 */
   488: 
   489: static void install_sigabrt_handler(void)
   490: {
 
 
SYMBOL_STACK_INDEX:  0
 
SYMBOL_NAME:  mysqld!my_sigabrt_handler+3
 
FOLLOWUP_NAME:  MachineOwner
 
MODULE_NAME: mysqld
 
IMAGE_NAME:  mysqld.exe
 
DEBUG_FLR_IMAGE_TIMESTAMP:  5f9dbb7e
 
STACK_COMMAND:  .ecxr ; kb
 
BUCKET_ID:  STATUS_BREAKPOINT_mysqld!my_sigabrt_handler+3
 
PRIMARY_PROBLEM_CLASS:  STATUS_BREAKPOINT_mysqld!my_sigabrt_handler+3
 
FAILURE_EXCEPTION_CODE:  80000003
 
FAILURE_IMAGE_NAME:  mysqld.exe
 
BUCKET_ID_IMAGE_STR:  mysqld.exe
 
FAILURE_MODULE_NAME:  mysqld
 
BUCKET_ID_MODULE_STR:  mysqld
 
FAILURE_FUNCTION_NAME:  my_sigabrt_handler
 
BUCKET_ID_FUNCTION_STR:  my_sigabrt_handler
 
BUCKET_ID_OFFSET:  3
 
BUCKET_ID_MODTIMEDATESTAMP:  5f9dbb7e
 
BUCKET_ID_MODCHECKSUM:  0
 
BUCKET_ID_MODVER_STR:  10.4.16.0
 
BUCKET_ID_PREFIX_STR:  STATUS_BREAKPOINT_
 
FAILURE_PROBLEM_CLASS:  STATUS_BREAKPOINT
 
FAILURE_SYMBOL_NAME:  mysqld.exe!my_sigabrt_handler
 
FAILURE_BUCKET_ID:  STATUS_BREAKPOINT_80000003_mysqld.exe!my_sigabrt_handler
 
WATSON_STAGEONE_URL:  http://watson.microsoft.com/StageOne/mysqld.exe/10.4.16.0/5f9dbb7e/mysqld.exe/10.4.16.0/5f9dbb7e/80000003/00dd92c3.htm?Retriage=1
 
TARGET_TIME:  2020-10-31T19:47:36.000Z
 
OSBUILD:  17763
 
OSSERVICEPACK:  475
 
SERVICEPACK_NUMBER: 0
 
OS_REVISION: 0
 
SUITE_MASK:  400
 
PRODUCT_TYPE:  3
 
OSPLATFORM_TYPE:  x86
 
OSNAME:  Windows 10
 
OSEDITION:  Windows 10 Server TerminalServer DataCenter SingleUserTS
 
OS_LOCALE:  
 
USER_LCID:  0
 
OSBUILD_TIMESTAMP:  unknown_date
 
BUILDDATESTAMP_STR:  180914-1434
 
BUILDLAB_STR:  rs5_release
 
BUILDOSVER_STR:  10.0.17763.1.x86fre.rs5_release.180914-1434
 
ANALYSIS_SESSION_ELAPSED_TIME: 70aa
 
ANALYSIS_SOURCE:  UM
 
FAILURE_ID_HASH_STRING:  um:status_breakpoint_80000003_mysqld.exe!my_sigabrt_handler
 
FAILURE_ID_HASH:  {f6294d5d-cc99-d322-9f92-e2c8a944cc10}
 
Followup:     MachineOwner
---------
 
eax=00000000 ebx=00000008 ecx=00fe92c0 edx=03608000 esi=00000016 edi=00fe92c0
eip=00fe92c3 esp=0c2de80c ebp=0c2de80c iopl=0         nv up ei pl nz ac po nc
cs=0023  ss=002b  ds=002b  es=002b  fs=0053  gs=002b             efl=00000212
mysqld!my_sigabrt_handler:
             int     3
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
mysqld!my_sigabrt_handler [D:\win32-debug\build\src\mysys\my_thr_init.c @ 485]
int sig = 0n22
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
ucrtbase!raise
Unable to enumerate locals, HRESULT 0x80004005
Private symbols (symbols.pri) are required for locals.
Type ".hh dbgerr005" for details.
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
ucrtbase!abort
Unable to enumerate locals, HRESULT 0x80004005
Private symbols (symbols.pri) are required for locals.
Type ".hh dbgerr005" for details.
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
mysqld!row_purge_remove_sec_if_poss_leaf [D:\win32-debug\build\src\storage\innobase\row\row0purge.cc @ 599]
struct btr_cur_t * btr_cur = 0x0c2dea68
struct purge_node_t * node = 0x0844e338
struct dict_index_t * index = 0x084347f8
struct dtuple_t * entry = 0x0b42c590
bool success = true
struct btr_pcur_t pcur = struct btr_pcur_t
row_search_result search_result = ROW_FOUND (0n0)
struct mtr_t mtr = struct mtr_t
btr_latch_mode mode = BTR_MODIFY_LEAF (0n2)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
mysqld!row_purge_remove_sec_if_poss [D:\win32-debug\build\src\storage\innobase\row\row0purge.cc @ 695]
struct purge_node_t * node = 0x0844e338
struct dict_index_t * index = 0x084347f8
struct dtuple_t * entry = 0x0b42c590
unsigned int n_tries = 0
unsigned int success = 0xcccccccc
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
mysqld!row_purge_upd_exist_or_extern_func [D:\win32-debug\build\src\storage\innobase\row\row0purge.cc @ 897]
struct dtuple_t * entry = 0x0b42c590
struct que_thr_t * thr = 0x0844dec8
struct purge_node_t * node = 0x0844e338
unsigned char * undo_rec = 0x0844ed48 "--- memory read error at address 0x0844ed48 ---"
struct dict_index_t * index = 0xcccccccc
struct mem_block_info_t * heap = 0x0b42c548
struct mtr_t mtr = struct mtr_t
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
mysqld!row_purge_record_func [D:\win32-debug\build\src\storage\innobase\row\row0purge.cc @ 1217]
struct purge_node_t * node = 0x0844e338
unsigned char * undo_rec = 0x0844ed48 "--- memory read error at address 0x0844ed48 ---"
struct que_thr_t * thr = 0x0844dec8
bool updated_extern = false
bool purged = true
struct dict_index_t * clust_index = 0x08435090
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
mysqld!row_purge [D:\win32-debug\build\src\storage\innobase\row\row0purge.cc @ 1261]
bool purged = true
bool updated_extern = false
struct purge_node_t * node = 0x0844e338
unsigned char * undo_rec = 0x0844ed48 "--- memory read error at address 0x0844ed48 ---"
struct que_thr_t * thr = 0x0844dec8
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
mysqld!row_purge_step [D:\win32-debug\build\src\storage\innobase\row\row0purge.cc @ 1320]
struct trx_purge_rec_t * purge_rec = 0x0b459f80
struct que_thr_t * thr = 0x0844dec8
struct purge_node_t * node = 0x0844e338
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
mysqld!que_thr_step [D:\win32-debug\build\src\storage\innobase\que\que0que.cc @ 1039]
struct que_thr_t * thr = 0x0844dec8
struct que_thr_t * old_thr = 0x0844dec8
void * node = 0x0844e338
struct trx_t * trx = 0x0980d028
unsigned int type = 0xd
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
mysqld!que_run_threads_low [D:\win32-debug\build\src\storage\innobase\que\que0que.cc @ 1101]
struct que_thr_t * thr = 0x0844dec8
struct que_thr_t * next_thr = 0x0844dec8
struct trx_t * trx = 0x0980d028
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
mysqld!que_run_threads [D:\win32-debug\build\src\storage\innobase\que\que0que.cc @ 1141]
struct que_thr_t * thr = 0x0844dec8
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
mysqld!trx_purge [D:\win32-debug\build\src\storage\innobase\trx\trx0purge.cc @ 1322]
unsigned int n_purge_threads = 4
bool truncate = false
struct srv_slot_t * slot = 0x02c77430
unsigned int n_pages_handled = 3
struct que_thr_t * thr = 0x0844dec8
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
mysqld!srv_do_purge [D:\win32-debug\build\src\storage\innobase\srv\srv0srv.cc @ 2610]
unsigned int * n_total_purged = 0x0c2dfb2c
struct srv_slot_t * slot = 0x02c77430
unsigned int n_pages_purged = 1
unsigned int rseg_history_len = <Memory access error>
unsigned int count = <Memory access error>
unsigned int n_threads = 4
unsigned int n_use_threads = <Memory access error>
unsigned int old_activity_count = 0x92e
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
mysqld!srv_purge_coordinator_thread [D:\win32-debug\build\src\storage\innobase\srv\srv0srv.cc @ 2743]
void * arg = 0x00000000
unsigned int n_total_purged = 0x1b
struct srv_slot_t * slot = 0x02c77430
unsigned int rseg_history_len = 0x156
class THD * thd = 0x0b301b90
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
kernel32!BaseThreadInitThunk
Unable to enumerate locals, HRESULT 0x80004005
Private symbols (symbols.pri) are required for locals.
Type ".hh dbgerr005" for details.
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
ntdll!__RtlUserThreadStart
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
ntdll!_RtlUserThreadStart
Unable to enumerate locals, HRESULT 0x80004005
Private symbols (symbols.pri) are required for locals.
Type ".hh dbgerr005" for details.
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
mysqld!my_sigabrt_handler [D:\win32-debug\build\src\mysys\my_thr_init.c @ 485]
Processing 30 threads, please wait
 
.  0  Id: 1614.16ac Suspend: 0 Teb: 037cb000 Unfrozen
      Priority: 0  Priority class: 32
  
ntdll!NtWaitForMultipleObjects
KERNELBASE!WaitForMultipleObjects
mysqld!handle_connections_win(void) [D:\win32-debug\build\src\sql\handle_connections_win.cc @ 592]
mysqld!win_main(int argc = 0n146, char ** argv = 0x03c61b68) [D:\win32-debug\build\src\sql\mysqld.cc @ 5926]
mysqld!mysql_service(void * p = 0x00000000) [D:\win32-debug\build\src\sql\mysqld.cc @ 5989]
mysqld!mysqld_main(int argc = 0n23, char ** argv = 0x03c53a80) [D:\win32-debug\build\src\sql\mysqld.cc @ 6181]
mysqld!main(int argc = 0n23, char ** argv = 0x03c53a80) [D:\win32-debug\build\src\sql\main.cc @ 25]
(Inline) -------- mysqld!invoke_main [d:\agent\_work\3\s\src\vctools\crt\vcstartup\src\startup\exe_common.inl @ 78]
mysqld!__scrt_common_main_seh(void) [d:\agent\_work\3\s\src\vctools\crt\vcstartup\src\startup\exe_common.inl @ 288]
kernel32!BaseThreadInitThunk
ntdll!__RtlUserThreadStart
ntdll!_RtlUserThreadStart
 
.  1  Id: 1614.ebc Suspend: 0 Teb: 037ce000 Unfrozen
      Priority: 0  Priority class: 32
  
ntdll!NtWaitForWorkViaWorkerFactory
kernel32!BaseThreadInitThunk
ntdll!__RtlUserThreadStart
ntdll!_RtlUserThreadStart
 
.  3  Id: 1614.9f4 Suspend: 0 Teb: 037d4000 Unfrozen
      Priority: 0  Priority class: 32
  
ntdll!NtWaitForAlertByThreadId
KERNELBASE!SleepConditionVariableCS
mysqld!pthread_cond_timedwait(struct _RTL_CONDITION_VARIABLE * cond = 0x034a0a50, struct _RTL_CRITICAL_SECTION * mutex = 0x034a0a34, struct timespec * abstime = 0x08bdf6f8) [D:\win32-debug\build\src\mysys\my_wincond.c @ 82]
mysqld!inline_mysql_cond_timedwait(struct st_mysql_cond * that = 0x034a0a50, struct st_mysql_mutex * mutex = 0x034a0a34, struct timespec * abstime = 0x08bdf6f8, char * src_file = 0x02a5c368 "--- memory read error at address 0x02a5c368 ---", unsigned int src_line = 0x124) [D:\win32-debug\build\src\include\mysql\psi\mysql_thread.h @ 1211]
mysqld!timer_handler(void * arg = 0x00000000) [D:\win32-debug\build\src\mysys\thr_timer.c @ 293]
mysqld!pfs_spawn_thread(void * arg = 0x03c3e4b8) [D:\win32-debug\build\src\storage\perfschema\pfs.cc @ 1869]
mysqld!pthread_start(void * p = 0x03c536e0) [D:\win32-debug\build\src\mysys\my_winthread.c @ 60]
ucrtbase!thread_start<unsigned int 
kernel32!BaseThreadInitThunk
ntdll!__RtlUserThreadStart
ntdll!_RtlUserThreadStart
 
.  4  Id: 1614.5b8 Suspend: 0 Teb: 037d7000 Unfrozen
      Priority: 0  Priority class: 32
  
ntdll!NtWaitForAlertByThreadId
KERNELBASE!SleepConditionVariableCS
mysqld!pthread_cond_timedwait(struct _RTL_CONDITION_VARIABLE * cond = 0x03490794, struct _RTL_CRITICAL_SECTION * mutex = 0x03490778, struct timespec * abstime = 0x097ffbd4) [D:\win32-debug\build\src\mysys\my_wincond.c @ 82]
mysqld!inline_mysql_cond_timedwait(struct st_mysql_cond * that = 0x03490794, struct st_mysql_mutex * mutex = 0x03490778, struct timespec * abstime = 0x097ffbd4, char * src_file = 0x02a06ea0 "--- memory read error at address 0x02a06ea0 ---", unsigned int src_line = 0x74) [D:\win32-debug\build\src\include\mysql\psi\mysql_thread.h @ 1211]
mysqld!my_service_thread_sleep(struct st_ma_service_thread_control * control = 0x02a06f54, unsigned int64 sleep_time = 0x00000006`c088e200) [D:\win32-debug\build\src\storage\maria\ma_servicethread.c @ 115]
mysqld!ma_checkpoint_background(void * arg = 0x0000001e) [D:\win32-debug\build\src\storage\maria\ma_checkpoint.c @ 707]
mysqld!pfs_spawn_thread(void * arg = 0x03c34970) [D:\win32-debug\build\src\storage\perfschema\pfs.cc @ 1869]
mysqld!pthread_start(void * p = 0x03c53760) [D:\win32-debug\build\src\mysys\my_winthread.c @ 60]
ucrtbase!thread_start<unsigned int 
kernel32!BaseThreadInitThunk
ntdll!__RtlUserThreadStart
ntdll!_RtlUserThreadStart
 
.  5  Id: 1614.1010 Suspend: 0 Teb: 037da000 Unfrozen
      Priority: 0  Priority class: 32
  
ntdll!NtRemoveIoCompletion
mysqld!os_aio_windows_handler(unsigned int segment = 0, unsigned int pos = 0, struct fil_node_t ** m1 = 0x0a94fd70, void ** m2 = 0x0a94fd50, class IORequest * type = 0x0a94fd5c) [D:\win32-debug\build\src\storage\innobase\os\os0file.cc @ 6432]
mysqld!os_aio_handler(unsigned int segment = 0, struct fil_node_t ** m1 = 0x0a94fd70, void ** m2 = 0x0a94fd50, class IORequest * request = 0x0a94fd5c) [D:\win32-debug\build\src\storage\innobase\os\os0file.cc @ 5603]
mysqld!fil_aio_wait(unsigned int segment = 0) [D:\win32-debug\build\src\storage\innobase\fil\fil0fil.cc @ 4316]
mysqld!io_handler_thread(void * arg = 0x02c84fc0) [D:\win32-debug\build\src\storage\innobase\srv\srv0start.cc @ 324]
kernel32!BaseThreadInitThunk
ntdll!__RtlUserThreadStart
ntdll!_RtlUserThreadStart
 
. 11  Id: 1614.1780 Suspend: 0 Teb: 037ec000 Unfrozen
      Priority: 0  Priority class: 32
  
ntdll!NtWaitForAlertByThreadId
KERNELBASE!SleepConditionVariableCS
mysqld!os_event::timed_wait(unsigned long time_in_ms = 0xf1) [D:\win32-debug\build\src\storage\innobase\os\os0event.cc @ 251]
mysqld!os_event::wait_time_low(unsigned int time_in_usec = 0x3ad68, int64 reset_sig_count = 0n3) [D:\win32-debug\build\src\storage\innobase\os\os0event.cc @ 387]
mysqld!os_event_wait_time_low(struct os_event * event = 0x082a9820, unsigned int time_in_usec = 0x3ad68, int64 reset_sig_count = 0n3) [D:\win32-debug\build\src\storage\innobase\os\os0event.cc @ 486]
mysqld!pc_sleep_if_needed(unsigned int next_loop_time = 0x6295438c, int64 sig_count = 0n3, unsigned int cur_time = 0x6295429b) [D:\win32-debug\build\src\storage\innobase\buf\buf0flu.cc @ 2627]
mysqld!buf_flush_page_cleaner_coordinator(void * __formal = 0x00000000) [D:\win32-debug\build\src\storage\innobase\buf\buf0flu.cc @ 3086]
kernel32!BaseThreadInitThunk
ntdll!__RtlUserThreadStart
ntdll!_RtlUserThreadStart
 
. 12  Id: 1614.968 Suspend: 0 Teb: 037f5000 Unfrozen
      Priority: 0  Priority class: 32
  
ntdll!NtWaitForAlertByThreadId
KERNELBASE!SleepConditionVariableCS
mysqld!os_event::timed_wait(unsigned long time_in_ms = 0x3e8) [D:\win32-debug\build\src\storage\innobase\os\os0event.cc @ 251]
mysqld!os_event::wait_time_low(unsigned int time_in_usec = 0xf4240, int64 reset_sig_count = 0n1) [D:\win32-debug\build\src\storage\innobase\os\os0event.cc @ 387]
mysqld!os_event_wait_time_low(struct os_event * event = 0x083981e0, unsigned int time_in_usec = 0xf4240, int64 reset_sig_count = 0n1) [D:\win32-debug\build\src\storage\innobase\os\os0event.cc @ 486]
mysqld!lock_wait_timeout_thread(void * __formal = 0x00000000) [D:\win32-debug\build\src\storage\innobase\lock\lock0wait.cc @ 522]
kernel32!BaseThreadInitThunk
ntdll!__RtlUserThreadStart
ntdll!_RtlUserThreadStart
 
. 13  Id: 1614.ee0 Suspend: 0 Teb: 037f8000 Unfrozen
      Priority: 0  Priority class: 32
  
ntdll!NtWaitForAlertByThreadId
KERNELBASE!SleepConditionVariableCS
mysqld!os_event::timed_wait(unsigned long time_in_ms = 0x3e8) [D:\win32-debug\build\src\storage\innobase\os\os0event.cc @ 251]
mysqld!os_event::wait_time_low(unsigned int time_in_usec = 0xf4240, int64 reset_sig_count = 0n1) [D:\win32-debug\build\src\storage\innobase\os\os0event.cc @ 387]
mysqld!os_event_wait_time_low(struct os_event * event = 0x082a92a8, unsigned int time_in_usec = 0xf4240, int64 reset_sig_count = 0n1) [D:\win32-debug\build\src\storage\innobase\os\os0event.cc @ 486]
mysqld!srv_error_monitor_thread(void * __formal = 0x00000000) [D:\win32-debug\build\src\storage\innobase\srv\srv0srv.cc @ 1854]
kernel32!BaseThreadInitThunk
ntdll!__RtlUserThreadStart
ntdll!_RtlUserThreadStart
 
. 14  Id: 1614.16f4 Suspend: 0 Teb: 037fb000 Unfrozen
      Priority: 0  Priority class: 32
  
ntdll!NtWaitForAlertByThreadId
KERNELBASE!SleepConditionVariableCS
mysqld!os_event::timed_wait(unsigned long time_in_ms = 0x1388) [D:\win32-debug\build\src\storage\innobase\os\os0event.cc @ 251]
mysqld!os_event::wait_time_low(unsigned int time_in_usec = 0x4c4b40, int64 reset_sig_count = 0n1) [D:\win32-debug\build\src\storage\innobase\os\os0event.cc @ 387]
mysqld!os_event_wait_time_low(struct os_event * event = 0x082a9548, unsigned int time_in_usec = 0x4c4b40, int64 reset_sig_count = 0n1) [D:\win32-debug\build\src\storage\innobase\os\os0event.cc @ 486]
mysqld!srv_monitor_thread(void * __formal = 0x00000000) [D:\win32-debug\build\src\storage\innobase\srv\srv0srv.cc @ 1690]
kernel32!BaseThreadInitThunk
ntdll!__RtlUserThreadStart
ntdll!_RtlUserThreadStart
 
. 15  Id: 1614.10f0 Suspend: 0 Teb: 037fe000 Unfrozen
      Priority: 0  Priority class: 32
  
ntdll!NtWaitForAlertByThreadId
KERNELBASE!SleepConditionVariableCS
mysqld!os_event::timed_wait(unsigned long time_in_ms = 0x2710) [D:\win32-debug\build\src\storage\innobase\os\os0event.cc @ 251]
mysqld!os_event::wait_time_low(unsigned int time_in_usec = 0x989680, int64 reset_sig_count = 0n1) [D:\win32-debug\build\src\storage\innobase\os\os0event.cc @ 387]
mysqld!os_event_wait_time_low(struct os_event * event = 0x08398838, unsigned int time_in_usec = 0x989680, int64 reset_sig_count = 0n0) [D:\win32-debug\build\src\storage\innobase\os\os0event.cc @ 486]
mysqld!dict_stats_thread(void * __formal = 0x00000000) [D:\win32-debug\build\src\storage\innobase\dict\dict0stats_bg.cc @ 475]
kernel32!BaseThreadInitThunk
ntdll!__RtlUserThreadStart
ntdll!_RtlUserThreadStart
 
. 16  Id: 1614.18c Suspend: 0 Teb: 03602000 Unfrozen
      Priority: 0  Priority class: 32
  
ntdll!NtWaitForAlertByThreadId
KERNELBASE!SleepConditionVariableCS
mysqld!os_event::timed_wait(unsigned long time_in_ms = 0x1388) [D:\win32-debug\build\src\storage\innobase\os\os0event.cc @ 251]
mysqld!os_event::wait_time_low(unsigned int time_in_usec = 0x4c4b40, int64 reset_sig_count = 0n1) [D:\win32-debug\build\src\storage\innobase\os\os0event.cc @ 387]
mysqld!os_event_wait_time_low(struct os_event * event = 0x083e7a60, unsigned int time_in_usec = 0x4c4b40, int64 reset_sig_count = 0n1) [D:\win32-debug\build\src\storage\innobase\os\os0event.cc @ 486]
mysqld!ib_wqueue_timedwait(struct ib_wqueue_t * wq = 0x08450c20, unsigned int wait_in_usecs = 0x4c4b40) [D:\win32-debug\build\src\storage\innobase\ut\ut0wqueue.cc @ 154]
mysqld!fts_optimize_thread(void * arg = 0x08450c20) [D:\win32-debug\build\src\storage\innobase\fts\fts0opt.cc @ 2829]
kernel32!BaseThreadInitThunk
ntdll!__RtlUserThreadStart
ntdll!_RtlUserThreadStart
 
. 17  Id: 1614.13a4 Suspend: 0 Teb: 03605000 Unfrozen
      Priority: 0  Priority class: 32
  
ntdll!NtDelayExecution
KERNELBASE!Sleep
mysqld!os_thread_sleep(unsigned int tm = 0xf4240) [D:\win32-debug\build\src\storage\innobase\os\os0thread.cc @ 224]
mysqld!srv_master_sleep(void) [D:\win32-debug\build\src\storage\innobase\srv\srv0srv.cc @ 2341]
mysqld!srv_master_thread(void * arg = 0x00000000) [D:\win32-debug\build\src\storage\innobase\srv\srv0srv.cc @ 2383]
kernel32!BaseThreadInitThunk
ntdll!__RtlUserThreadStart
ntdll!_RtlUserThreadStart
 
. 18  Id: 1614.1840 Suspend: 0 Teb: 03608000 Unfrozen
      Priority: 0  Priority class: 32
  
ntdll!NtGetContextThread
 
. 19  Id: 1614.1e4 Suspend: 0 Teb: 0360b000 Unfrozen
      Priority: 0  Priority class: 32
  
ntdll!NtWaitForAlertByThreadId
KERNELBASE!SleepConditionVariableCS
mysqld!os_event::wait(void) [D:\win32-debug\build\src\storage\innobase\os\os0event.cc @ 151]
mysqld!os_event::wait_low(int64 reset_sig_count = 0n605) [D:\win32-debug\build\src\storage\innobase\os\os0event.cc @ 329]
mysqld!os_event_wait_low(struct os_event * event = 0x083989f8, int64 reset_sig_count = 0n605) [D:\win32-debug\build\src\storage\innobase\os\os0event.cc @ 503]
mysqld!sync_array_wait_event(struct sync_array_t * arr = 0x03c34ab8, struct sync_cell_t ** cell = 0x0c41fa18) [D:\win32-debug\build\src\storage\innobase\sync\sync0arr.cc @ 471]
mysqld!rw_lock_s_lock_spin(struct rw_lock_t * lock = 0x02c4c050, unsigned int pass = 0, char * file_name = 0x02319e14 "D:\win32-debug\build\src\storage\innobase\row\row0purge.cc", unsigned int line = 0x42d) [D:\win32-debug\build\src\storage\innobase\sync\sync0rw.cc @ 367]
mysqld!rw_lock_s_lock_func(struct rw_lock_t * lock = 0x02c4c050, unsigned int pass = 0, char * file_name = 0x02319e14 "D:\win32-debug\build\src\storage\innobase\row\row0purge.cc", unsigned int line = 0x42d) [D:\win32-debug\build\src\storage\innobase\include\sync0rw.ic @ 285]
mysqld!pfs_rw_lock_s_lock_func(struct rw_lock_t * lock = 0x02c4c050, unsigned int pass = 0, char * file_name = 0x02319e14 "D:\win32-debug\build\src\storage\innobase\row\row0purge.cc", unsigned int line = 0x42d) [D:\win32-debug\build\src\storage\innobase\include\sync0rw.ic @ 633]
mysqld!row_purge_parse_undo_rec(struct purge_node_t * node = 0x084392b8, unsigned char * undo_rec = 0x08438dd0 "--- memory read error at address 0x08438dd0 ---", bool * updated_extern = 0x0c41fb77, struct que_thr_t * thr = 0x08439250) [D:\win32-debug\build\src\storage\innobase\row\row0purge.cc @ 1069]
mysqld!row_purge(struct purge_node_t * node = 0x084392b8, unsigned char * undo_rec = 0x08438dd0 "--- memory read error at address 0x08438dd0 ---", struct que_thr_t * thr = 0x08439250) [D:\win32-debug\build\src\storage\innobase\row\row0purge.cc @ 1258]
mysqld!row_purge_step(struct que_thr_t * thr = 0x08439250) [D:\win32-debug\build\src\storage\innobase\row\row0purge.cc @ 1320]
mysqld!que_thr_step(struct que_thr_t * thr = 0x08439250) [D:\win32-debug\build\src\storage\innobase\que\que0que.cc @ 1039]
mysqld!que_run_threads_low(struct que_thr_t * thr = 0x08439250) [D:\win32-debug\build\src\storage\innobase\que\que0que.cc @ 1101]
mysqld!que_run_threads(struct que_thr_t * thr = 0x08439250) [D:\win32-debug\build\src\storage\innobase\que\que0que.cc @ 1141]
mysqld!srv_task_execute(struct srv_slot_t * slot = 0x02c77458) [D:\win32-debug\build\src\storage\innobase\srv\srv0srv.cc @ 2471]
mysqld!srv_worker_thread(void * arg = 0x00000000) [D:\win32-debug\build\src\storage\innobase\srv\srv0srv.cc @ 2519]
kernel32!BaseThreadInitThunk
ntdll!__RtlUserThreadStart
ntdll!_RtlUserThreadStart
 
. 22  Id: 1614.1844 Suspend: 0 Teb: 03614000 Unfrozen
      Priority: 0  Priority class: 32
  
ntdll!NtWaitForAlertByThreadId
KERNELBASE!SleepConditionVariableCS
mysqld!os_event::wait(void) [D:\win32-debug\build\src\storage\innobase\os\os0event.cc @ 151]
mysqld!os_event::wait_low(int64 reset_sig_count = 0n1) [D:\win32-debug\build\src\storage\innobase\os\os0event.cc @ 329]
mysqld!os_event_wait_low(struct os_event * event = 0x082a9740, int64 reset_sig_count = 0n0) [D:\win32-debug\build\src\storage\innobase\os\os0event.cc @ 503]
mysqld!buf_dump_thread(void * __formal = 0x00000000) [D:\win32-debug\build\src\storage\innobase\buf\buf0dump.cc @ 832]
kernel32!BaseThreadInitThunk
ntdll!__RtlUserThreadStart
ntdll!_RtlUserThreadStart
 
. 23  Id: 1614.1448 Suspend: 0 Teb: 03617000 Unfrozen
      Priority: 0  Priority class: 32
  
ntdll!NtDelayExecution
KERNELBASE!Sleep
mysqld!os_thread_sleep(unsigned int tm = 0xf4240) [D:\win32-debug\build\src\storage\innobase\os\os0thread.cc @ 224]
mysqld!btr_defragment_thread(void * __formal = 0x00000000) [D:\win32-debug\build\src\storage\innobase\btr\btr0defragment.cc @ 700]
kernel32!BaseThreadInitThunk
ntdll!__RtlUserThreadStart
ntdll!_RtlUserThreadStart
 
. 24  Id: 1614.16e0 Suspend: 0 Teb: 0361a000 Unfrozen
      Priority: 0  Priority class: 32
  
ntdll!NtWaitForAlertByThreadId
KERNELBASE!SleepConditionVariableCS
mysqld!os_event::wait(void) [D:\win32-debug\build\src\storage\innobase\os\os0event.cc @ 151]
mysqld!os_event::wait_low(int64 reset_sig_count = 0n1) [D:\win32-debug\build\src\storage\innobase\os\os0event.cc @ 329]
mysqld!os_event_wait_low(struct os_event * event = 0x082a9778, int64 reset_sig_count = 0n0) [D:\win32-debug\build\src\storage\innobase\os\os0event.cc @ 503]
mysqld!buf_resize_thread(void * __formal = 0x00000000) [D:\win32-debug\build\src\storage\innobase\buf\buf0buf.cc @ 3142]
kernel32!BaseThreadInitThunk
ntdll!__RtlUserThreadStart
ntdll!_RtlUserThreadStart
 
. 25  Id: 1614.17f4 Suspend: 0 Teb: 0361d000 Unfrozen
      Priority: 0  Priority class: 32
  
ntdll!NtWaitForAlertByThreadId
KERNELBASE!SleepConditionVariableCS
mysqld!pthread_cond_timedwait(struct _RTL_CONDITION_VARIABLE * cond = 0x0ca9fd30, struct _RTL_CRITICAL_SECTION * mutex = 0x0ca9fd40, struct timespec * abstime = 0x00000000) [D:\win32-debug\build\src\mysys\my_wincond.c @ 82]
mysqld!pthread_cond_wait(struct _RTL_CONDITION_VARIABLE * cond = 0x0ca9fd30, struct _RTL_CRITICAL_SECTION * mutex = 0x0ca9fd40) [D:\win32-debug\build\src\mysys\my_wincond.c @ 90]
mysqld!inline_mysql_cond_wait(struct st_mysql_cond * that = 0x0ca9fd30, struct st_mysql_mutex * mutex = 0x0ca9fd40, char * src_file = 0x0152f410 "D:\win32-debug\build\src\storage\innobase\handler\ha_innodb.cc", unsigned int src_line = 0x137) [D:\win32-debug\build\src\include\mysql\psi\mysql_thread.h @ 1185]
mysqld!thd_destructor_proxy(void * __formal = 0x00000000) [D:\win32-debug\build\src\storage\innobase\handler\ha_innodb.cc @ 311]
mysqld!pfs_spawn_thread(void * arg = 0x084449a0) [D:\win32-debug\build\src\storage\perfschema\pfs.cc @ 1869]
mysqld!pthread_start(void * p = 0x08464c18) [D:\win32-debug\build\src\mysys\my_winthread.c @ 60]
ucrtbase!thread_start<unsigned int 
kernel32!BaseThreadInitThunk
ntdll!__RtlUserThreadStart
ntdll!_RtlUserThreadStart
 
. 27  Id: 1614.1914 Suspend: 0 Teb: 03623000 Unfrozen
      Priority: 0  Priority class: 32
  
ntdll!NtWaitForAlertByThreadId
KERNELBASE!SleepConditionVariableCS
mysqld!pthread_cond_timedwait(struct _RTL_CONDITION_VARIABLE * cond = 0x02c2c744, struct _RTL_CRITICAL_SECTION * mutex = 0x02c2c59c, struct timespec * abstime = 0x00000000) [D:\win32-debug\build\src\mysys\my_wincond.c @ 82]
mysqld!pthread_cond_wait(struct _RTL_CONDITION_VARIABLE * cond = 0x02c2c744, struct _RTL_CRITICAL_SECTION * mutex = 0x02c2c59c) [D:\win32-debug\build\src\mysys\my_wincond.c @ 90]
mysqld!inline_mysql_cond_wait(struct st_mysql_cond * that = 0x02c2c744, struct st_mysql_mutex * mutex = 0x02c2c59c, char * src_file = 0x0135198c "D:\win32-debug\build\src\sql\slave.cc", unsigned int src_line = 0x20a) [D:\win32-debug\build\src\include\mysql\psi\mysql_thread.h @ 1174]
mysqld!handle_slave_background(void * arg = 0x00000000) [D:\win32-debug\build\src\sql\slave.cc @ 522]
mysqld!pfs_spawn_thread(void * arg = 0x0b3d7938) [D:\win32-debug\build\src\storage\perfschema\pfs.cc @ 1869]
mysqld!pthread_start(void * p = 0x08464b38) [D:\win32-debug\build\src\mysys\my_winthread.c @ 60]
ucrtbase!thread_start<unsigned int 
kernel32!BaseThreadInitThunk
ntdll!__RtlUserThreadStart
ntdll!_RtlUserThreadStart
 
. 28  Id: 1614.1730 Suspend: 0 Teb: 03626000 Unfrozen
      Priority: 0  Priority class: 32
  
ntdll!NtWaitForAlertByThreadId
KERNELBASE!SleepConditionVariableCS
mysqld!os_event::wait(void) [D:\win32-debug\build\src\storage\innobase\os\os0event.cc @ 151]
mysqld!os_event::wait_low(int64 reset_sig_count = 0n445) [D:\win32-debug\build\src\storage\innobase\os\os0event.cc @ 329]
mysqld!os_event_wait_low(struct os_event * event = 0x083983d8, int64 reset_sig_count = 0n445) [D:\win32-debug\build\src\storage\innobase\os\os0event.cc @ 503]
mysqld!sync_array_wait_event(struct sync_array_t * arr = 0x03c34ab8, struct sync_cell_t ** cell = 0x0d0eb020) [D:\win32-debug\build\src\storage\innobase\sync\sync0arr.cc @ 471]
mysqld!rw_lock_x_lock_wait_func(struct rw_lock_t * lock = 0x02c4c050, unsigned int pass = 0, long threshold = 0n0, char * file_name = 0x01d53e60 "D:\win32-debug\build\src\storage\innobase\row\row0mysql.cc", unsigned int line = 0xcf9) [D:\win32-debug\build\src\storage\innobase\sync\sync0rw.cc @ 447]
mysqld!rw_lock_x_lock_low(struct rw_lock_t * lock = 0x02c4c050, unsigned int pass = 0, char * file_name = 0x01d53e60 "D:\win32-debug\build\src\storage\innobase\row\row0mysql.cc", unsigned int line = 0xcf9) [D:\win32-debug\build\src\storage\innobase\sync\sync0rw.cc @ 502]
mysqld!rw_lock_x_lock_func(struct rw_lock_t * lock = 0x02c4c050, unsigned int pass = 0, char * file_name = 0x01d53e60 "D:\win32-debug\build\src\storage\innobase\row\row0mysql.cc", unsigned int line = 0xcf9) [D:\win32-debug\build\src\storage\innobase\sync\sync0rw.cc @ 657]
mysqld!dict_sys_t::lock(char * file = 0x01d53e60 "D:\win32-debug\build\src\storage\innobase\row\row0mysql.cc", unsigned int line = 0xcf9) [D:\win32-debug\build\src\storage\innobase\include\dict0dict.h @ 1553]
mysqld!row_mysql_lock_data_dictionary_func(struct trx_t * trx = 0x0980ebd8, char * file = 0x01d53e60 "D:\win32-debug\build\src\storage\innobase\row\row0mysql.cc", unsigned int line = 0xcf9) [D:\win32-debug\build\src\storage\innobase\row\row0mysql.cc @ 2289]
mysqld!row_drop_table_for_mysql(char * name = 0x0d0ebd68 "test/child", struct trx_t * trx = 0x0980ebd8, enum_sql_command sqlcom = SQLCOM_DROP_TABLE (0n9), bool create_failed = false, bool nonatomic = true) [D:\win32-debug\build\src\storage\innobase\row\row0mysql.cc @ 3321]
mysqld!ha_innobase::delete_table(char * name = 0x0d0ed53c ".\test\child", enum_sql_command sqlcom = SQLCOM_DROP_TABLE (0n9)) [D:\win32-debug\build\src\storage\innobase\handler\ha_innodb.cc @ 13136]
mysqld!ha_innobase::delete_table(char * name = 0x0d0ed53c ".\test\child") [D:\win32-debug\build\src\storage\innobase\handler\ha_innodb.cc @ 13262]
mysqld!handler::ha_delete_table(char * name = 0x0d0ed53c ".\test\child") [D:\win32-debug\build\src\sql\handler.cc @ 4747]
mysqld!ha_delete_table(class THD * thd = 0x0b5dc7c0, struct handlerton * table_type = 0x08311a18, char * path = 0x0d0ed53c ".\test\child", struct st_mysql_const_lex_string * db = 0x0d0ed390, struct st_mysql_const_lex_string * alias = 0x0b447614, bool generate_warning = true) [D:\win32-debug\build\src\sql\handler.cc @ 2624]
mysqld!mysql_rm_table_no_locks(class THD * thd = 0x0b5dc7c0, struct TABLE_LIST * tables = 0x0b447600, bool if_exists = false, bool drop_temporary = false, bool drop_view = false, bool drop_sequence = false, bool dont_log_query = false, bool dont_free_locks = false) [D:\win32-debug\build\src\sql\sql_table.cc @ 2484]
mysqld!mysql_rm_table(class THD * thd = 0x0b5dc7c0, struct TABLE_LIST * tables = 0x0b447600, bool if_exists = false, bool drop_temporary = false, bool drop_sequence = false) [D:\win32-debug\build\src\sql\sql_table.cc @ 2121]
mysqld!mysql_execute_command(class THD * thd = 0x0b5dc7c0) [D:\win32-debug\build\src\sql\sql_parse.cc @ 4897]
mysqld!mysql_parse(class THD * thd = 0x0b5dc7c0, char * rawbuf = 0x0b447548 "--- memory read error at address 0x0b447548 ---", unsigned int length = 0x18, class Parser_state * parser_state = 0x0d0ef36c, bool is_com_multi = false, bool is_next_command = false) [D:\win32-debug\build\src\sql\sql_parse.cc @ 7938]
mysqld!dispatch_command(enum_server_command command = COM_QUERY (0n3), class THD * thd = 0x0b5dc7c0, char * packet = 0x0b5aa9d9 "--- memory read error at address 0x0b5aa9d9 ---", unsigned int packet_length = 0x18, bool is_com_multi = false, bool is_next_command = false) [D:\win32-debug\build\src\sql\sql_parse.cc @ 1839]
mysqld!do_command(class THD * thd = 0x0b5dc7c0) [D:\win32-debug\build\src\sql\sql_parse.cc @ 1357]
mysqld!threadpool_process_request(class THD * thd = 0x0b5dc7c0) [D:\win32-debug\build\src\sql\threadpool_common.cc @ 374]
mysqld!tp_callback(struct TP_connection * c = 0x0b3bad88) [D:\win32-debug\build\src\sql\threadpool_common.cc @ 192]
mysqld!tp_callback(struct _TP_CALLBACK_INSTANCE * instance = 0x0d0efbbc, void * context = 0x0b3bad88) [D:\win32-debug\build\src\sql\threadpool_win.cc @ 368]
mysqld!work_callback(struct _TP_CALLBACK_INSTANCE * instance = 0x0d0efbbc, void * context = 0x0b3bad88, struct _TP_WORK * work = 0x082a7aa8) [D:\win32-debug\build\src\sql\threadpool_win.cc @ 419]
ntdll!TppWorkpExecuteCallback
ntdll!TppWorkerThread
WARNING: Frame IP not in any known module. Following frames may be wrong.
0xd0efcec
 
Total threads: 30
Duplicate callstacks: 10 (windbg thread #s follow):
2, 6, 7, 8, 9, 10, 20, 21, 26, 29
quit:

Comment by Elena Stepanova [ 2021-01-26 ]

Now the same test case from the description fails in even less predictable ways on 10.3.

10.3 4e503aec

Program terminated with signal SIGSEGV, Segmentation fault.
#0  0x00005606a591f82f in rec_init_offsets_comp_ordinary (rec=0x7f1f8400a2f6 "\200", index=0x7f1f500bcc30, offsets=0x7f1f7000e120, n_core=1, def_val=0x0, format=REC_LEAF_ORDINARY) at /data/src/10.3/storage/innobase/rem/rem0rec.cc:379
379			if (!(col->prtype & DATA_NOT_NULL)) {
[Current thread is 1 (Thread 0x7f1f8d7fa700 (LWP 1496946))]
#0  0x00005606a591f82f in rec_init_offsets_comp_ordinary (rec=0x7f1f8400a2f6 "\200", index=0x7f1f500bcc30, offsets=0x7f1f7000e120, n_core=1, def_val=0x0, format=REC_LEAF_ORDINARY) at /data/src/10.3/storage/innobase/rem/rem0rec.cc:379
#1  0x00005606a5920a46 in rec_init_offsets (rec=0x7f1f8400a2f6 "\200", index=0x7f1f500bcc30, leaf=true, offsets=0x7f1f7000e120) at /data/src/10.3/storage/innobase/rem/rem0rec.cc:610
#2  0x00005606a592184b in rec_get_offsets_func (rec=0x7f1f8400a2f6 "\200", index=0x7f1f500bcc30, offsets=0x7f1f7000e120, leaf=true, n_fields=1, file=0x5606a612bb48 "/data/src/10.3/storage/innobase/row/row0purge.cc", line=1362, heap=0x5606a9336148) at /data/src/10.3/storage/innobase/rem/rem0rec.cc:869
#3  0x00005606a59a5ee5 in purge_node_t::validate_pcur (this=0x5606a93360b0) at /data/src/10.3/storage/innobase/row/row0purge.cc:1362
#4  0x00005606a59a19d5 in row_purge_reposition_pcur (mode=1, node=0x5606a93360b0, mtr=0x7f1f8d7f8b40) at /data/src/10.3/storage/innobase/row/row0purge.cc:73
#5  0x00005606a59a24df in row_purge_poss_sec (node=0x5606a93360b0, index=0x7f1f500b8940, entry=0x7f1f8400a3c0, sec_pcur=0x7f1f8d7f9090, sec_mtr=0x7f1f8d7f9330, is_tree=false) at /data/src/10.3/storage/innobase/row/row0purge.cc:316
#6  0x00005606a59a34bb in row_purge_remove_sec_if_poss_leaf (node=0x5606a93360b0, index=0x7f1f500b8940, entry=0x7f1f8400a3c0) at /data/src/10.3/storage/innobase/row/row0purge.cc:579
#7  0x00005606a59a39e7 in row_purge_remove_sec_if_poss (node=0x5606a93360b0, index=0x7f1f500b8940, entry=0x7f1f8400a3c0) at /data/src/10.3/storage/innobase/row/row0purge.cc:692
#8  0x00005606a59a3c0a in row_purge_del_mark (node=0x5606a93360b0) at /data/src/10.3/storage/innobase/row/row0purge.cc:766
#9  0x00005606a59a57f9 in row_purge_record_func (node=0x5606a93360b0, undo_rec=0x7f1f7000df20 "", thr=0x5606a9335ee0, updated_extern=false) at /data/src/10.3/storage/innobase/row/row0purge.cc:1191
#10 0x00005606a59a5b0f in row_purge (node=0x5606a93360b0, undo_rec=0x7f1f7000df20 "", thr=0x5606a9335ee0) at /data/src/10.3/storage/innobase/row/row0purge.cc:1257
#11 0x00005606a59a5d62 in row_purge_step (thr=0x5606a9335ee0) at /data/src/10.3/storage/innobase/row/row0purge.cc:1316
#12 0x00005606a5911636 in que_thr_step (thr=0x5606a9335ee0) at /data/src/10.3/storage/innobase/que/que0que.cc:1038
#13 0x00005606a59118ce in que_run_threads_low (thr=0x5606a9335ee0) at /data/src/10.3/storage/innobase/que/que0que.cc:1100
#14 0x00005606a5911b34 in que_run_threads (thr=0x5606a9335ee0) at /data/src/10.3/storage/innobase/que/que0que.cc:1140
#15 0x00005606a59fbe21 in srv_task_execute (slot=0x5606a66e2740 <srv_sys+384>) at /data/src/10.3/storage/innobase/srv/srv0srv.cc:2491
#16 0x00005606a59fc041 in srv_worker_thread (arg=0x0) at /data/src/10.3/storage/innobase/srv/srv0srv.cc:2539
#17 0x00007f1fb074f609 in start_thread (arg=<optimized out>) at pthread_create.c:477
#18 0x00007f1fb032b293 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

2021-01-26  2:52:38 4 [ERROR] InnoDB: Purge node pcur validation failed
2021-01-26  2:52:38 4 [ERROR] InnoDB: TUPLE (info_bits=0, 1 fields): {[4]    (0x80000002)}
2021-01-26  2:52:38 4 [ERROR] InnoDB: COMPACT RECORD(info_bits=0, 1 fields): {[0](0x)}
2021-01-26  2:52:38 4 [ERROR] InnoDB: Purge node pcur validation failed
2021-01-26  2:52:38 4 [ERROR] InnoDB: TUPLE (info_bits=0, 1 fields): {[4]    (0x80000002)}
2021-01-26  2:52:38 4 [ERROR] InnoDB: COMPACT RECORD(info_bits=0, 1 fields): {[0](0x)}
mysqld: /data/src/10.3/storage/innobase/row/row0purge.cc:73: ulint row_purge_reposition_pcur(ulint, purge_node_t*, mtr_t*): Assertion `node->validate_pcur()' failed.
210126  2:52:38 [ERROR] mysqld got signal 6 ;
 
#3  0x00007fbe39cfdf36 in __GI___assert_fail (assertion=0x556ecce15b30 "node->validate_pcur()", file=0x556ecce15b48 "/data/src/10.3/storage/innobase/row/row0purge.cc", line=73, function=0x556ecce15b80 "ulint row_purge_reposition_pcur(ulint, purge_node_t*, mtr_t*)") at assert.c:101
#4  0x0000556ecc68ba43 in row_purge_reposition_pcur (mode=1, node=0x556ecf2c40b0, mtr=0x7fbe16ffbb40) at /data/src/10.3/storage/innobase/row/row0purge.cc:73
#5  0x0000556ecc68c4df in row_purge_poss_sec (node=0x556ecf2c40b0, index=0x7fbde40b8730, entry=0x7fbe08012430, sec_pcur=0x7fbe16ffc090, sec_mtr=0x7fbe16ffc330, is_tree=false) at /data/src/10.3/storage/innobase/row/row0purge.cc:316
#6  0x0000556ecc68d4bb in row_purge_remove_sec_if_poss_leaf (node=0x556ecf2c40b0, index=0x7fbde40b8730, entry=0x7fbe08012430) at /data/src/10.3/storage/innobase/row/row0purge.cc:579
#7  0x0000556ecc68d9e7 in row_purge_remove_sec_if_poss (node=0x556ecf2c40b0, index=0x7fbde40b8730, entry=0x7fbe08012430) at /data/src/10.3/storage/innobase/row/row0purge.cc:692
#8  0x0000556ecc68dc0a in row_purge_del_mark (node=0x556ecf2c40b0) at /data/src/10.3/storage/innobase/row/row0purge.cc:766
#9  0x0000556ecc68f7f9 in row_purge_record_func (node=0x556ecf2c40b0, undo_rec=0x7fbe0400f170 "", thr=0x556ecf2c3ee0, updated_extern=false) at /data/src/10.3/storage/innobase/row/row0purge.cc:1191
#10 0x0000556ecc68fb0f in row_purge (node=0x556ecf2c40b0, undo_rec=0x7fbe0400f170 "", thr=0x556ecf2c3ee0) at /data/src/10.3/storage/innobase/row/row0purge.cc:1257
#11 0x0000556ecc68fd62 in row_purge_step (thr=0x556ecf2c3ee0) at /data/src/10.3/storage/innobase/row/row0purge.cc:1316
#12 0x0000556ecc5fb636 in que_thr_step (thr=0x556ecf2c3ee0) at /data/src/10.3/storage/innobase/que/que0que.cc:1038
#13 0x0000556ecc5fb8ce in que_run_threads_low (thr=0x556ecf2c3ee0) at /data/src/10.3/storage/innobase/que/que0que.cc:1100
#14 0x0000556ecc5fbb34 in que_run_threads (thr=0x556ecf2c3ee0) at /data/src/10.3/storage/innobase/que/que0que.cc:1140
#15 0x0000556ecc6e5e21 in srv_task_execute (slot=0x556ecd3cc778 <srv_sys+440>) at /data/src/10.3/storage/innobase/srv/srv0srv.cc:2491
#16 0x0000556ecc6e6041 in srv_worker_thread (arg=0x0) at /data/src/10.3/storage/innobase/srv/srv0srv.cc:2539
#17 0x00007fbe3a20d609 in start_thread (arg=<optimized out>) at pthread_create.c:477
#18 0x00007fbe39de9293 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Comment by Marko Mäkelä [ 2021-11-01 ]

I think that the problem could be insufficient MDL protection in purge. The 10.6 branch could be unaffected by this, thanks to the data member dict_table_t::mdl_name, which holds the end-user visible table name also during DDL operations.

Here is a related 10.4 occurrence from buildbot:

10.4 d74d95961a31b47986d943216489513896108782

CURRENT_TEST: gcol.innodb_virtual_basic
mysqltest: At line 65: query 'DROP INDEX idx1 ON t' failed: 2013: Lost connection to MySQL server during query

In the stack trace output, we see that the crashing thread is about to free the table object, while another thread is waiting for MDL, in order to purge an undo log record:

#4  __GI___libc_free (mem=0x80) at malloc.c:3102
#5  0x000056323695679e in dict_free_vc_templ (vc_templ=0x7f745000d560) at /home/buildbot/maria-slave/fulltest-debug-big-only/build/storage/innobase/include/dict0dict.ic:1237
#6  dict_sys_t::remove (this=<optimized out>, table=table@entry=0x7f742c199790, lru=lru@entry=false, keep=keep@entry=false) at /home/buildbot/maria-slave/fulltest-debug-big-only/build/storage/innobase/dict/dict0dict.cc:1778
#7  0x00005632367998a7 in innobase_reload_table (table_name=..., table=0x7f742c199790, thd=0x7f742c000d90) at /home/buildbot/maria-slave/fulltest-debug-big-only/build/storage/innobase/handler/handler0alter.cc:10266
#8  ha_innobase::commit_inplace_alter_table (this=<optimized out>, altered_table=<optimized out>, ha_alter_info=<optimized out>, commit=<optimized out>) at /home/buildbot/maria-slave/fulltest-debug-big-only/build/storage/innobase/handler/handler0alter.cc:11500
#9  0x00005632363e6689 in mysql_inplace_alter_table (thd=0x7f742c000d90, table_list=0x7f742c0c7510, table=0x7f742c079720, altered_table=0x7f747e2533a0, ha_alter_info=0x7f747e253300, alter_ctx=0x7f747e254e30, target_mdl_request=<optimized out>) at /home/buildbot/maria-slave/fulltest-debug-big-only/build/sql/sql_table.cc:7873

This hit SIGSEGV because free() was being invoked on an invalid pointer. Another thread is accessing the same table object 0x7f742c199790:

Thread 22 (Thread 0x7f74667fc700 (LWP 2425116)):
#5  MDL_wait::timed_wait (this=0x7f7450000eb0, owner=0x7f7450000e60, abs_timeout=0x7f74667fafa0, set_status_on_timeout=<optimized out>, wait_state_name=<optimized out>) at /home/buildbot/maria-slave/fulltest-debug-big-only/build/sql/mdl.cc:1176
#6  0x000056323645af2d in MDL_context::acquire_lock (this=this@entry=0x7f7450000eb0, mdl_request=mdl_request@entry=0x7f745000b3a8, lock_wait_timeout=<optimized out>) at /home/buildbot/maria-slave/fulltest-debug-big-only/build/sql/mdl.h:480
#7  0x00005632362cfe8f in open_table_get_mdl_lock (thd=0x7f7450000d90, ot_ctx=0x7f74667fb850, mdl_request=0x7f745000b3a8, flags=<optimized out>, mdl_ticket=0x7f74667fb288) at /home/buildbot/maria-slave/fulltest-debug-big-only/build/sql/sql_base.h:555
#8  0x00005632362d4c54 in open_table (thd=thd@entry=0x7f7450000d90, table_list=table_list@entry=0x7f745000af58, ot_ctx=ot_ctx@entry=0x7f74667fb850) at /home/buildbot/maria-slave/fulltest-debug-big-only/build/sql/sql_base.cc:1915
#9  0x00005632362f344a in open_purge_table (thd=thd@entry=0x7f7450000d90, db=<optimized out>, db@entry=0x7f74667fb910 "test", dblen=4, tb=<optimized out>, tb@entry=0x7f74667fbab0 "t", tblen=1) at /home/buildbot/maria-slave/fulltest-debug-big-only/build/sql/sql_class.cc:4813
#10 0x0000563236764e1c in innodb_acquire_mdl (table=0x7f742c199790, thd=0x7f7450000d90) at /home/buildbot/maria-slave/fulltest-debug-big-only/build/storage/innobase/handler/ha_innodb.cc:20565
#11 innodb_find_table_for_vc (thd=0x7f7450000d90, table=<optimized out>) at /home/buildbot/maria-slave/fulltest-debug-big-only/build/storage/innobase/handler/ha_innodb.cc:20639
#12 0x00005632367798f8 in innobase_init_vc_templ (table=0x7f742c199790) at /home/buildbot/maria-slave/fulltest-debug-big-only/build/include/my_pthread.h:374
#13 0x0000563236877441 in row_purge_parse_undo_rec (thr=0x5632395a1df0, updated_extern=0x7f74667fbd07, undo_rec=0x5632395a2560 "", node=0x5632395a1fc0) at /home/buildbot/maria-slave/fulltest-debug-big-only/build/storage/innobase/row/row0purge.cc:1113
#14 row_purge (thr=0x5632395a1df0, undo_rec=0x5632395a2560 "", node=0x5632395a1fc0) at /home/buildbot/maria-slave/fulltest-debug-big-only/build/storage/innobase/row/row0purge.cc:1261

I would have expected Thread 22 to crash. But, maybe something in that thread corrupted the pointer?

Comment by Marko Mäkelä [ 2021-11-01 ]

For the message tried to purge non-delete-marked record without a crash, MDEV-26951 was filed.

Comment by Roel Van de Paar [ 2022-02-04 ]

joce yours looks to be a different (though possibly related) bug, have a look at MDEV-27744 and bugs linked from that one.

Comment by Marko Mäkelä [ 2022-08-01 ]

I think that the gcol.innodb_virtual_fk crash needs to be fixed before MDEV-29068 and MDEV-29181 can be tested and approved.

Comment by Marko Mäkelä [ 2022-10-06 ]

It looks like after MDEV-29666 was merged to 10.4, the test gcol.innodb_virtual_basic started to fail much more often. I see that this test was marked "big" only to mask failures. As part of the fix, the test must be enabled on all builders.

Comment by Nikita Malyavin [ 2022-12-07 ]

marko why do you mention MDEV-29068 and MDEV-29181 here? The test case in the description has no foreign keys. I'd rather move the focus to MDEV-26951 here

Comment by Nikita Malyavin [ 2023-05-15 ]

I was lucky to reproduce it with ASAN, from the first time.

==58520==ERROR: AddressSanitizer: heap-use-after-free on address 0x618000037928 at pc 0x55fcb7e1ac3f bp 0x7f96eddc4c10 sp 0x7f96eddc4c08
READ of size 8 at 0x618000037928 thread T16
    #0 0x55fcb7e1ac3e in purge_node_t::validate_pcur() /home/nik/mariadb/storage/innobase/row/row0purge.cc:1500:13
    #1 0x55fcb7e191aa in row_purge_reposition_pcur(unsigned long, purge_node_t*, mtr_t*) /home/nik/mariadb/storage/innobase/row/row0purge.cc:74:3
    #2 0x55fcb7e18390 in row_purge_poss_sec(purge_node_t*, dict_index_t*, dtuple_t const*, btr_pcur_t*, mtr_t*, bool) /home/nik/mariadb/storage/innobase/row/row0purge.cc:319:16
    #3 0x55fcb7e2859f in row_purge_remove_sec_if_poss_leaf(purge_node_t*, dict_index_t*, dtuple_t const*) /home/nik/mariadb/storage/innobase/row/row0purge.cc:582:7
    #4 0x55fcb7e27963 in row_purge_remove_sec_if_poss(purge_node_t*, dict_index_t*, dtuple_t const*) /home/nik/mariadb/storage/innobase/row/row0purge.cc:695:6
    #5 0x55fcb7e24aae in row_purge_del_mark(purge_node_t*) /home/nik/mariadb/storage/innobase/row/row0purge.cc:769:4
    #6 0x55fcb7e1e42d in row_purge_record_func(purge_node_t*, unsigned char*, que_thr_t const*, bool) /home/nik/mariadb/storage/innobase/row/row0purge.cc:1320:12
    #7 0x55fcb7e1a3b5 in row_purge(purge_node_t*, unsigned char*, que_thr_t*) /home/nik/mariadb/storage/innobase/row/row0purge.cc:1386:18
    #8 0x55fcb7e19bc2 in row_purge_step(que_thr_t*) /home/nik/mariadb/storage/innobase/row/row0purge.cc:1464:3
    #9 0x55fcb7cb20c9 in que_thr_step(que_thr_t*) /home/nik/mariadb/storage/innobase/que/que0que.cc:966:9
    #10 0x55fcb7cb002a in que_run_threads_low(que_thr_t*) /home/nik/mariadb/storage/innobase/que/que0que.cc:1028:14
    #11 0x55fcb7cafa6f in que_run_threads(que_thr_t*) /home/nik/mariadb/storage/innobase/que/que0que.cc:1068:2
    #12 0x55fcb7fb8a1d in trx_purge(unsigned long, bool, srv_slot_t*) /home/nik/mariadb/storage/innobase/trx/trx0purge.cc:1582:2
    #13 0x55fcb7f256af in srv_do_purge(unsigned long*, srv_slot_t*) /home/nik/mariadb/storage/innobase/srv/srv0srv.cc:2607:20
    #14 0x55fcb7f24535 in srv_purge_coordinator_thread /home/nik/mariadb/storage/innobase/srv/srv0srv.cc:2747:22
    #15 0x7f96fc9d444a  (/usr/lib/libc.so.6+0x8744a) (BuildId: 2f005a79cd1a8e385972f5a102f16adba414d75e)
    #16 0x7f96fca57e3f  (/usr/lib/libc.so.6+0x10ae3f) (BuildId: 2f005a79cd1a8e385972f5a102f16adba414d75e)
 
0x618000037928 is located 168 bytes inside of 800-byte region [0x618000037880,0x618000037ba0)
freed by thread T23 here:
    #0 0x55fcb5ffee92 in __interceptor_free.part.0 (/home/nik/mariadb/bld/sql/mysqld+0x1b1ae92) (BuildId: 846e397c702849e8)
    #1 0x55fcb7bcd1cb in mem_heap_block_free(mem_block_info_t*, mem_block_info_t*) /home/nik/mariadb/storage/innobase/mem/mem0mem.cc:416:3
    #2 0x55fcb768048b in mem_heap_free(mem_block_info_t*) /home/nik/mariadb/storage/innobase/include/mem0mem.inl:417:3
    #3 0x55fcb7687d3f in dict_mem_index_free(dict_index_t*) /home/nik/mariadb/storage/innobase/dict/dict0mem.cc:1108:2
    #4 0x55fcb763f01c in dict_index_remove_from_cache_low(dict_table_t*, dict_index_t*, unsigned long) /home/nik/mariadb/storage/innobase/dict/dict0dict.cc:2171:2
    #5 0x55fcb7636c7d in dict_table_remove_from_cache_low(dict_table_t*, unsigned long) /home/nik/mariadb/storage/innobase/dict/dict0dict.cc:1829:3
    #6 0x55fcb763f698 in dict_table_remove_from_cache(dict_table_t*) /home/nik/mariadb/storage/innobase/dict/dict0dict.cc:1907:2
    #7 0x55fcb7a52d24 in innobase_reload_table(THD*, dict_table_t*, st_mysql_const_lex_string const&) /home/nik/mariadb/storage/innobase/handler/handler0alter.cc:8918:3
    #8 0x55fcb7a4bd44 in ha_innobase::commit_inplace_alter_table(TABLE*, Alter_inplace_info*, bool) /home/nik/mariadb/storage/innobase/handler/handler0alter.cc:10034:23
    #9 0x55fcb63bccb6 in handler::ha_commit_inplace_alter_table(TABLE*, Alter_inplace_info*, bool) /home/nik/mariadb/sql/handler.cc:4618:11
    #10 0x55fcb6e740b3 in mysql_inplace_alter_table(THD*, TABLE_LIST*, TABLE*, TABLE*, Alter_inplace_info*, MDL_request*, Alter_table_ctx*) /home/nik/mariadb/sql/sql_table.cc:7825:22
    #11 0x55fcb6e61cee in mysql_alter_table(THD*, st_mysql_const_lex_string const*, st_mysql_const_lex_string const*, HA_CREATE_INFO*, TABLE_LIST*, Alter_info*, unsigned int, st_order*, bool) /home/nik/mariadb/sql/sql_table.cc:10163:16
    #12 0x55fcb7095b0c in Sql_cmd_alter_table::execute(THD*) /home/nik/mariadb/sql/sql_alter.cc:512:11
    #13 0x55fcb6b10be7 in mysql_execute_command(THD*) /home/nik/mariadb/sql/sql_parse.cc:6076:26
    #14 0x55fcb6af0fba in mysql_parse(THD*, char*, unsigned int, Parser_state*, bool, bool) /home/nik/mariadb/sql/sql_parse.cc:7855:18
    #15 0x55fcb6ae56b9 in dispatch_command(enum_server_command, THD*, char*, unsigned int, bool, bool) /home/nik/mariadb/sql/sql_parse.cc:1852:7
    #16 0x55fcb6aed4f2 in do_command(THD*) /home/nik/mariadb/sql/sql_parse.cc:1398:17
    #17 0x55fcb708075a in do_handle_one_connection(CONNECT*) /home/nik/mariadb/sql/sql_connect.cc:1404:11
    #18 0x55fcb707fe8e in handle_one_connection /home/nik/mariadb/sql/sql_connect.cc:1309:3
    #19 0x55fcb8266f4e in pfs_spawn_thread /home/nik/mariadb/storage/perfschema/pfs.cc:1869:3
    #20 0x7f96fc9d444a  (/usr/lib/libc.so.6+0x8744a) (BuildId: 2f005a79cd1a8e385972f5a102f16adba414d75e)
 
previously allocated by thread T23 here:
    #0 0x55fcb5fffea9 in malloc (/home/nik/mariadb/bld/sql/mysqld+0x1b1bea9) (BuildId: 846e397c702849e8)
    #1 0x55fcb7bcbc62 in mem_heap_create_block_func(mem_block_info_t*, unsigned long, char const*, unsigned int, unsigned long) /home/nik/mariadb/storage/innobase/mem/mem0mem.cc:277:37
    #2 0x55fcb7bccde5 in mem_heap_add_block(mem_block_info_t*, unsigned long) /home/nik/mariadb/storage/innobase/mem/mem0mem.cc:378:14
    #3 0x55fcb767f2c9 in mem_heap_alloc(mem_block_info_t*, unsigned long) /home/nik/mariadb/storage/innobase/include/mem0mem.inl:191:11
    #4 0x55fcb767eeeb in mem_heap_zalloc(mem_block_info_t*, unsigned long) /home/nik/mariadb/storage/innobase/include/mem0mem.inl:160:16
    #5 0x55fcb7685333 in dict_mem_index_create(dict_table_t*, char const*, unsigned long, unsigned long) /home/nik/mariadb/storage/innobase/dict/dict0mem.cc:770:3
    #6 0x55fcb7645706 in dict_index_build_internal_non_clust(dict_index_t*) /home/nik/mariadb/storage/innobase/dict/dict0dict.cc:2641:14
    #7 0x55fcb7640a15 in dict_index_add_to_cache(dict_index_t*&, unsigned long, dict_add_v_col_t const*) /home/nik/mariadb/storage/innobase/dict/dict0dict.cc:2016:6
    #8 0x55fcb79b5378 in dict_create_index_step(que_thr_t*) /home/nik/mariadb/storage/innobase/dict/dict0crea.cc:1361:9
    #9 0x55fcb7cb215a in que_thr_step(que_thr_t*) /home/nik/mariadb/storage/innobase/que/que0que.cc:976:9
    #10 0x55fcb7cb002a in que_run_threads_low(que_thr_t*) /home/nik/mariadb/storage/innobase/que/que0que.cc:1028:14
    #11 0x55fcb7cafa6f in que_run_threads(que_thr_t*) /home/nik/mariadb/storage/innobase/que/que0que.cc:1068:2
    #12 0x55fcb7dbac12 in row_create_index_for_mysql(dict_index_t*, trx_t*, unsigned long const*) /home/nik/mariadb/storage/innobase/row/row0mysql.cc:2559:3
    #13 0x55fcb754d9cb in create_index(trx_t*, TABLE const*, dict_table_t*, unsigned int) /home/nik/mariadb/storage/innobase/handler/ha_innodb.cc:11435:3
    #14 0x55fcb74fb85c in create_table_info_t::create_table(bool) /home/nik/mariadb/storage/innobase/handler/ha_innodb.cc:12581:19
    #15 0x55fcb7550aca in ha_innobase::create(char const*, TABLE*, HA_CREATE_INFO*, bool, trx_t*) /home/nik/mariadb/storage/innobase/handler/ha_innodb.cc:13044:20
    #16 0x55fcb74feaa4 in ha_innobase::create(char const*, TABLE*, HA_CREATE_INFO*) /home/nik/mariadb/storage/innobase/handler/ha_innodb.cc:13095:9
    #17 0x55fcb63bdb16 in handler::ha_create(char const*, TABLE*, HA_CREATE_INFO*) /home/nik/mariadb/sql/handler.cc:4756:14
    #18 0x55fcb63c3f8b in ha_create_table(THD*, char const*, char const*, char const*, HA_CREATE_INFO*, st_mysql_const_unsigned_lex_string*) /home/nik/mariadb/sql/handler.cc:5223:22
    #19 0x55fcb6fa82a0 in rea_create_table(THD*, st_mysql_const_unsigned_lex_string*, char const*, char const*, char const*, HA_CREATE_INFO*, handler*, bool) /home/nik/mariadb/sql/unireg.cc:515:9
    #20 0x55fcb6e4566a in create_table_impl(THD*, st_mysql_const_lex_string const*, st_mysql_const_lex_string const*, st_mysql_const_lex_string const*, st_mysql_const_lex_string const*, char const*, DDL_options_st, HA_CREATE_INFO*, Alter_info*, int, bool*, st_key**, unsigned int*, st_mysql_const_unsigned_lex_string*) /home/nik/mariadb/sql/sql_table.cc:5120:9
    #21 0x55fcb6e43364 in mysql_create_table_no_lock(THD*, st_mysql_const_lex_string const*, st_mysql_const_lex_string const*, Table_specification_st*, Alter_info*, bool*, int, TABLE_LIST*) /home/nik/mariadb/sql/sql_table.cc:5239:8
    #22 0x55fcb6e46d13 in mysql_create_table(THD*, TABLE_LIST*, Table_specification_st*, Alter_info*) /home/nik/mariadb/sql/sql_table.cc:5334:7
    #23 0x55fcb6e81eec in Sql_cmd_create_table_like::execute(THD*) /home/nik/mariadb/sql/sql_table.cc:11503:12
    #24 0x55fcb6b10be7 in mysql_execute_command(THD*) /home/nik/mariadb/sql/sql_parse.cc:6076:26
    #25 0x55fcb6af0fba in mysql_parse(THD*, char*, unsigned int, Parser_state*, bool, bool) /home/nik/mariadb/sql/sql_parse.cc:7855:18
    #26 0x55fcb6ae56b9 in dispatch_command(enum_server_command, THD*, char*, unsigned int, bool, bool) /home/nik/mariadb/sql/sql_parse.cc:1852:7
    #27 0x55fcb6aed4f2 in do_command(THD*) /home/nik/mariadb/sql/sql_parse.cc:1398:17
    #28 0x55fcb708075a in do_handle_one_connection(CONNECT*) /home/nik/mariadb/sql/sql_connect.cc:1404:11
    #29 0x55fcb707fe8e in handle_one_connection /home/nik/mariadb/sql/sql_connect.cc:1309:3
 
Thread T16 created by T0 here:
    #0 0x55fcb5f73a38 in pthread_create (/home/nik/mariadb/bld/sql/mysqld+0x1a8fa38) (BuildId: 846e397c702849e8)
    #1 0x55fcb7c2378b in os_thread_create_func(void* (*)(void*), void*, unsigned long*) /home/nik/mariadb/storage/innobase/os/os0thread.cc:138:8
    #2 0x55fcb7f3a485 in srv_start(bool) /home/nik/mariadb/storage/innobase/srv/srv0start.cc:2442:46
    #3 0x55fcb751c6c0 in innodb_init(void*) /home/nik/mariadb/storage/innobase/handler/ha_innodb.cc:4375:8
    #4 0x55fcb639432b in ha_initialize_handlerton(st_plugin_int*) /home/nik/mariadb/sql/handler.cc:549:31
    #5 0x55fcb6b91ad0 in plugin_initialize(st_mem_root*, st_plugin_int*, int*, char**, bool) /home/nik/mariadb/sql/sql_plugin.cc:1435:9
    #6 0x55fcb6b907b1 in plugin_init(int*, char**, int) /home/nik/mariadb/sql/sql_plugin.cc:1717:15
    #7 0x55fcb6058d6c in init_server_components() /home/nik/mariadb/sql/mysqld.cc:5476:7
    #8 0x55fcb6051fe6 in mysqld_main(int, char**) /home/nik/mariadb/sql/mysqld.cc:6095:7
    #9 0x55fcb6046d31 in main /home/nik/mariadb/sql/main.cc:25:10
    #10 0x7f96fc97084f  (/usr/lib/libc.so.6+0x2384f) (BuildId: 2f005a79cd1a8e385972f5a102f16adba414d75e)

Comment by Nikita Malyavin [ 2023-05-15 ]

I found it reproducing on a simpler (and a frustratingly basic) test case:

create table t1 (pk int primary key, b int as(pk) unique) engine=innodb;
 
insert into t1 (pk) values (1),(2),(3);
 
begin;
delete from t1;
insert into t1 (pk) values (1),(2),(3);
commit;
alter table t1 add column c int;
--sleep 1

To note, only algorithm=instant is vulnerable. Inplace goes fine.

Comment by Nikita Malyavin [ 2024-01-02 ]

Sadly (or not), it is only reproducible on 10.3. I tried 10.4-10.6 and 11.2, it never fails there, so closing it.

For gcol.innodb_virtual_basic failures we have MDEV-16950, MDEV-26951, MDEV-30663

Generated at Thu Feb 08 09:01:02 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.