Details
-
Bug
-
Status: Closed (View Workflow)
-
Critical
-
Resolution: Duplicate
-
10.4.17, 10.5.8
-
Windows, CentOS 7
Description
The following crash happens after upgrade to 10.5.8 (same setup worked in 10.4.11):
|
|
210201 13:42:03 [ERROR] mysqld got exception 0xc0000005 ;
|
This could be because you hit a bug. It is also possible that this binary
|
or one of the libraries it was linked against is corrupt, improperly built,
|
or misconfigured. This error can also be caused by malfunctioning hardware.
|
|
|
To report this bug, see https://mariadb.com/kb/en/reporting-bugs
|
|
|
We will try our best to scrape up some info that will hopefully help
|
diagnose the problem, but since we have already crashed,
|
something is definitely wrong and this may fail.
|
|
|
Server version: 10.5.8-MariaDB-log
|
key_buffer_size=67108864
|
read_buffer_size=1048576
|
max_used_connections=82
|
max_threads=65537
|
thread_count=100
|
It is possible that mysqld could use up to
|
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 88784 K bytes of memory
|
Hope that's ok; if not, decrease some variables in the equation.
|
|
|
Thread pointer: 0x27ed1cb7c88
|
Attempting backtrace. You can use the following information to find out
|
where mysqld died. If you see no messages after this, something went
|
terribly wrong...
|
ha_spider.dll!spider_flush_table_mon_cache_init()
|
ha_spider.dll!spider_ping_table_init()
|
ha_spider.dll!spider_ping_table_init()
|
ha_spider.dll!spider_ping_table_init()
|
server.dll!handler::ha_rnd_next()[handler.cc:3066]
|
server.dll!ha_partition::rnd_next()[ha_partition.cc:5220]
|
server.dll!handler::ha_rnd_next()[handler.cc:3066]
|
server.dll!rr_sequential()[records.cc:519]
|
server.dll!sub_select()[sql_select.cc:20616]
|
server.dll!do_select()[sql_select.cc:20153]
|
server.dll!JOIN::exec_inner()[sql_select.cc:4459]
|
server.dll!JOIN::exec()[sql_select.cc:4241]
|
server.dll!mysql_select()[sql_select.cc:4657]
|
server.dll!handle_select()[sql_select.cc:417]
|
server.dll!execute_sqlcom_select()[sql_parse.cc:6266]
|
server.dll!mysql_execute_command()[sql_parse.cc:3968]
|
server.dll!mysql_parse()[sql_parse.cc:8048]
|
server.dll!dispatch_command()[sql_parse.cc:1875]
|
server.dll!do_command()[sql_parse.cc:1353]
|
server.dll!threadpool_process_request()[threadpool_common.cc:363]
|
server.dll!tp_callback()[threadpool_common.cc:194]
|
ntdll.dll!RtlReleaseSRWLockExclusive()
|
ntdll.dll!RtlReleaseSRWLockExclusive()
|
KERNEL32.DLL!BaseThreadInitThunk()
|
ntdll.dll!RtlUserThreadStart()
|
|
|
Trying to get some variables.
|
Some pointers may be invalid and cause the dump to abort.
|
Query (0x27ed1cc11b0): select * from db.`spider_table
|
|
|
limit 0, 1000
|
Connection ID (thread ID): 25
|
Status: NOT_KILLED
|
...
|
The table has several partitions defined like these:
PARTITION BY RANGE (`ID`)
|
(PARTITION `pt1` VALUES LESS THAN (921201)
|
COMMENT = 'srv "archive", table "data_1900_2005"' ENGINE = SPIDER,
|
...
|
PARTITION `pt10` VALUES LESS THAN (596063838)
|
COMMENT = 'srv "archive", table "data_2017_2019"' ENGINE = SPIDER,
|
PARTITION `pt11` VALUES LESS THAN MAXVALUE
|
COMMENT = 'srv "current", table "db_current"' ENGINE = SPIDER);
|
The crash happens here according to the minidump created:
...
|
CONTEXT: (.ecxr)
|
.ecxr
|
rax=00000000020007d0 rbx=0000027ed23ff060 rcx=0000027ed23ff060
|
rdx=00000000000001b8 rsi=0000000000000000 rdi=00000000020007d0
|
rip=00007ff9e1b07c05 rsp=0000007f3735d5b0 rbp=0000027ed2f59bf0
|
r8=0000000000000001 r9=0000000000000000 r10=0000027ed4a42118
|
r11=0000007f3735d548 r12=0000000000000001 r13=00000000ffffffff
|
r14=0000027ed2f59bf0 r15=00007ff9d6bccc38
|
iopl=0 nv up ei pl zr na po nc
|
cs=0033 ss=002b ds=002b es=002b fs=0053 gs=002b efl=00010246
|
ha_spider!spider_set_direct_limit_offset+0x45:
|
00007ff9`e1b07c05 80b87a06000001 cmp byte ptr [rax+67Ah],1 ds:00000000`02000e4a=??
|
.cxr
|
Resetting default scope
|
|
|
FAULTING_IP:
|
ha_spider!spider_set_direct_limit_offset+45 [D:\winx64-packages\build\src\storage\spider\spd_table.cc @ 9354]
|
00007ff9`e1b07c05 80b87a06000001 cmp byte ptr [rax+67Ah],1
|
|
|
EXCEPTION_RECORD: (.exr -1)
|
.exr -1
|
ExceptionAddress: 00007ff9e1b07c05 (ha_spider!spider_set_direct_limit_offset+0x0000000000000045)
|
ExceptionCode: c0000005 (Access violation)
|
ExceptionFlags: 00000000
|
NumberParameters: 2
|
Parameter[0]: 0000000000000000
|
Parameter[1]: 0000000002000e4a
|
Attempt to read from address 0000000002000e4a
|
|
|
DEFAULT_BUCKET_ID: INVALID_POINTER_READ
|
|
|
PROCESS_NAME: mysqld.exe
|
|
|
ERROR_CODE: (NTSTATUS) 0xc0000005 - <Unable to get error code text>
|
|
|
EXCEPTION_CODE: (NTSTATUS) 0xc0000005 - <Unable to get error code text>
|
|
|
EXCEPTION_CODE_STR: c0000005
|
|
|
EXCEPTION_PARAMETER1: 0000000000000000
|
|
|
EXCEPTION_PARAMETER2: 0000000002000e4a
|
|
|
FOLLOWUP_IP:
|
ha_spider!spider_set_direct_limit_offset+45 [D:\winx64-packages\build\src\storage\spider\spd_table.cc @ 9354]
|
00007ff9`e1b07c05 80b87a06000001 cmp byte ptr [rax+67Ah],1
|
|
|
READ_ADDRESS: 0000000002000e4a
|
...
|
I see nothing special in other threads, but maybe it's only me.
Attachments
Issue Links
- relates to
-
-
- Closed
-
-
-
- Closed
-
-
-
- Closed
-
-
-
- Closed
-
-
-
- Closed
-
-
-
- Closed
-
-
-
- Closed
-
-
-
- Closed
-
-
-
- Closed
-
Activity
In regards spider_check_direct_order_limit, mentioned by nayuta-yanagisawa, in this comment, we still have MDEV-26583 which is currently in review by holyfoot. It would be interesting to know if the customer uses INSERT DELAYED, and -irrespective of whether the customer/users used DELAYED - testing in the future with a build with that patch as well as the various other patches since the work on this bug started, would be a good avenue imho, especially noting that in the original stack spider_set_direct_limit_offset is highlighted as the crashing function, and that bugs MDEV-27522 and MDEV-27388 (and previously MDEV-27171) have this too as the crashing function. Especially the just lodged MDEV-27522 may be of interest, but that one has thus far only been seen in 10.5+. The other two point back to MDEV-27240 which was a core issue. IOW, when all of these tickets have been resolved there is a good possibility that the issue listed here (which may be the same as MDEV-26583, or a duplicate of MDEV-27240 and/or sister bugs) is resolved too.
In recent (note: pre-MDEV-27240 patch) runs, the following issues have shown up:
SIGSEGV|spider_set_direct_limit_offset|ha_spider::check_direct_order_limit|ha_spider::check_direct_order_limit|ha_spider::rnd_next_internal
|
SIGSEGV|spider_set_direct_limit_offset|ha_spider::check_direct_order_limit|ha_spider::rnd_next_internal|ha_spider::pre_rnd_next
|
SIGSEGV|spider_set_direct_limit_offset|ha_spider::check_direct_order_limit|ha_spider::rnd_next_internal|ha_spider::rnd_next
|
Note the similarity with the OP.
FAULTING_IP: ha_spider!spider_set_direct_limit_offset+45
|
server.dll!handler::ha_rnd_next()[handler.cc:3066]
|
server.dll!ha_partition::rnd_next()[ha_partition.cc:5220]
|
server.dll!handler::ha_rnd_next()[handler.cc:3066]
|
As per nayuta-yanagisawa, the UniqueID's in the last comment are likely to be fixed post the MDEV-27240 patch, which was just merged into 10.5 trunk and upmerged all the way to 10.8.
A testcase for this UniqueID:
SIGSEGV|spider_set_direct_limit_offset|ha_spider::check_direct_order_limit|ha_spider::rnd_next_internal|ha_spider::rnd_next
|
Being:
INSTALL PLUGIN spider SONAME 'ha_spider.so'; |
CREATE TABLE t0 (a INT,KEY a (a)) ENGINE=SPIDER; |
CREATE TEMPORARY TABLE sql_temp0_innodb (a INT,b TEXT) ENGINE=SPIDER; |
LOAD DATA INFILE''INTO TABLE t; |
SELECT a,b FROM t0 outer_table WHERE a=(SELECT a FROM t0 WHERE outer_table.a=@arg0 AND a=0) AND b=''; |
ALTER TABLE t0 CHANGE c0 c0 INT UNSIGNED; |
DELETE FROM t0; |
Indeed proves the same as it crashes pre-fix but does not crash post-fix.
There is thus a high or very high possibility that this bug is fixed with the latest source.
I am closing this bug as fixed by the MDEV-27240 patch.
Anyone having run into this bug in production should retry with the latest 10.5 or 10.6 source code, which in due time will be incorporated in the next MariaDB sever release.
If anyone should see this bug, or a similar one, after revisions
10.5 e44439ab7354c5dff20707325694839e9346fb27
|
10.6 bd03c0e51629e1c3969a171137712a6bb854c232
|
10.7 b0998583f8e7331245597a7bf9a878cf6c7fa969 (not for production use yet)
|
10.8 e222e44d1bfc995870430bb90d8ac97e91f66cb4 (not for production use yet)
|
Please let us know and we will re-open the bug.
Various (quite sporadic) testcases can be/have been simplified to short ones, which at least once (in their current form) have produced stacks that include spider_check_direct_order_limit as a prominent frame in the stack. However, when those testcases are replayed manually, I keep running either
MDEV-26539,MDEV-26583orMDEV-26582.It looks like there are conflicting crashes. Fixing or at least understanding
MDEV-26539,MDEV-26583andMDEV-26582may help with this bug in one way or another - either by direct discovery of the underlying issue of this bug (if there are no conflicting crashes but rather two different types of crashes resulting from one core issue), or by clearing the path for a well-working (tough likely very sporadic) testcase for this bug (if there are conflicting crashes with two different underlying issues), or by clarifying that any spider_check_direct_order_limit bugs seen thus far are not related.A few example stacks of what was seen in terms of spider_check_direct_order_limit:
10.7.0 1bc82aaf0a7746c0921a94034aff2d51f0d75cd0 (Optimized)
Core was generated by `/test/MD040921-mariadb-10.7.0-linux-x86_64-opt/bin/mysqld --no-defaults --max_a'.Program terminated with signal SIGSEGV, Segmentation fault.#0 spider_check_index_merge (table=0x14c0b80a0938,select_lex=select_lex@entry=0x14c0b9626458)at /test/10.7_opt/storage/spider/spd_table.cc:9648[Current thread is 1 (Thread 0x14c194464700 (LWP 3272649))](gdb) bt#0 spider_check_index_merge (table=0x14c0b80a0938, select_lex=select_lex@entry=0x14c0b9626458) at /test/10.7_opt/storage/spider/spd_table.cc:9648#1 0x000014c17d7155c9 in spider_check_direct_order_limit (spider=spider@entry=0x14c0b80d2fb0) at /test/10.7_opt/storage/spider/spd_table.cc:9252#2 0x000014c17d732f21 in ha_spider::check_direct_order_limit (this=0x14c0b80d2fb0) at /test/10.7_opt/storage/spider/ha_spider.cc:13096#3 ha_spider::check_direct_order_limit (this=0x14c0b80d2fb0) at /test/10.7_opt/storage/spider/ha_spider.cc:13089#4 0x000014c17d73cfcd in ha_spider::index_last_internal (buf=0x14c0b80c8c28 "", this=0x14c0b80d2fb0) at /test/10.7_opt/storage/spider/ha_spider.cc:3350#5 ha_spider::index_last_internal (this=0x14c0b80d2fb0, buf=0x14c0b80c8c28 "") at /test/10.7_opt/storage/spider/ha_spider.cc:3310#6 0x000014c17d746119 in ha_spider::get_auto_increment (this=0x14c0b80d2fb0, offset=<optimized out>, increment=3, nb_desired_values=1, first_value=0x14c194463ac0, nb_reserved_values=0x14c194463ac8) at /test/10.7_opt/storage/spider/ha_spider.cc:9812#7 0x000056272f1fd848 in handler::update_auto_increment (this=this@entry=0x14c0b80d2fb0) at /test/10.7_opt/sql/handler.cc:3954#8 0x000014c17d735286 in ha_spider::update_auto_increment (this=0x14c0b80d2fb0) at /test/10.7_opt/storage/spider/ha_spider.cc:9755#9 0x000014c17d747345 in ha_spider::write_row (this=0x14c0b80d2fb0, buf=0x14c0b80c8bb8 "\376") at /test/10.7_opt/storage/spider/ha_spider.cc:10014#10 0x000056272f2031f0 in handler::ha_write_row (this=0x14c0b80d2fb0, buf=0x14c0b80c8bb8 "\376") at /test/10.7_opt/sql/handler.cc:7514#11 0x000056272ef79a8d in write_record (thd=thd@entry=0x14c0b971f808, table=0x14c0b80a0938, info=info@entry=0x14c0b97260e0, sink=sink@entry=0x0) at /test/10.7_opt/sql/sql_insert.cc:2135#12 0x000056272ef7c4bd in Delayed_insert::handle_inserts (this=0x14c0b971f7e8) at /test/10.7_opt/sql/sql_insert.cc:3576#13 0x000056272ef83915 in handle_delayed_insert (arg=arg@entry=0x14c0b971f7e8) at /test/10.7_opt/sql/sql_insert.cc:3316#14 0x000056272f427298 in pfs_spawn_thread (arg=0x5627321be8e8) at /test/10.7_opt/storage/perfschema/pfs.cc:2201#15 0x000014c194bae609 in start_thread (arg=<optimized out>) at pthread_create.c:477#16 0x000014c19479c293 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95And
10.7.0 1bc82aaf0a7746c0921a94034aff2d51f0d75cd0 (Optimized)
Core was generated by `/test/MD040921-mariadb-10.7.0-linux-x86_64-opt/bin/mysqld --no-defaults --max_a'.Program terminated with signal SIGSEGV, Segmentation fault.#0 spider_check_index_merge (table=0x1517480274a8,select_lex=select_lex@entry=0x151748011048)at /test/10.7_opt/storage/spider/spd_table.cc:9640[Current thread is 1 (Thread 0x15187006a700 (LWP 2980262))](gdb) bt#0 spider_check_index_merge (table=0x1517480274a8, select_lex=select_lex@entry=0x151748011048) at /test/10.7_opt/storage/spider/spd_table.cc:9640#1 0x00001518645255c9 in spider_check_direct_order_limit (spider=spider@entry=0x15174801ab00) at /test/10.7_opt/storage/spider/spd_table.cc:9252#2 0x0000151864542f21 in ha_spider::check_direct_order_limit (this=0x15174801ab00) at /test/10.7_opt/storage/spider/ha_spider.cc:13096#3 ha_spider::check_direct_order_limit (this=0x15174801ab00) at /test/10.7_opt/storage/spider/ha_spider.cc:13089#4 0x000015186454cfcd in ha_spider::index_last_internal (buf=0x151748028958 "", this=0x15174801ab00) at /test/10.7_opt/storage/spider/ha_spider.cc:3350#5 ha_spider::index_last_internal (this=0x15174801ab00, buf=0x151748028958 "") at /test/10.7_opt/storage/spider/ha_spider.cc:3310#6 0x0000151864556119 in ha_spider::get_auto_increment (this=0x15174801ab00, offset=<optimized out>, increment=1, nb_desired_values=1, first_value=0x151870069ac0, nb_reserved_values=0x151870069ac8) at /test/10.7_opt/storage/spider/ha_spider.cc:9812#7 0x000056017b2af848 in handler::update_auto_increment (this=this@entry=0x15174801ab00) at /test/10.7_opt/sql/handler.cc:3954#8 0x0000151864545286 in ha_spider::update_auto_increment (this=0x15174801ab00) at /test/10.7_opt/storage/spider/ha_spider.cc:9755#9 0x0000151864557345 in ha_spider::write_row (this=0x15174801ab00, buf=0x151748028948 "\376") at /test/10.7_opt/storage/spider/ha_spider.cc:10014#10 0x000056017b2b51f0 in handler::ha_write_row (this=0x15174801ab00, buf=0x151748028948 "\376") at /test/10.7_opt/sql/handler.cc:7514#11 0x000056017b02ba8d in write_record (thd=thd@entry=0x151748063b88, table=0x1517480274a8, info=info@entry=0x15174806a460, sink=sink@entry=0x0) at /test/10.7_opt/sql/sql_insert.cc:2135#12 0x000056017b02e4bd in Delayed_insert::handle_inserts (this=0x151748063b68) at /test/10.7_opt/sql/sql_insert.cc:3576#13 0x000056017b035915 in handle_delayed_insert (arg=arg@entry=0x151748063b68) at /test/10.7_opt/sql/sql_insert.cc:3316#14 0x000056017b4d9298 in pfs_spawn_thread (arg=0x56017d09ead8) at /test/10.7_opt/storage/perfschema/pfs.cc:2201#15 0x0000151879a3a609 in start_thread (arg=<optimized out>) at pthread_create.c:477#16 0x0000151879628293 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95Other uniqueID's observed:
SIGSEGV|spider_get_select_limit_from_select_lex|spider_get_select_limit_from_select_lex|spider_check_direct_order_limit|ha_spider::check_direct_order_limitSIGSEGV|spider_check_index_merge|spider_check_direct_order_limit|ha_spider::check_direct_order_limit|ha_spider::index_last_internalThe first one forms part of
MDEV-26583.