[MDEV-14141] mysqld got exception 0xc0000005 Created: 2017-10-26  Updated: 2019-05-01  Resolved: 2019-05-01

Status: Closed
Project: MariaDB Server
Component/s: Storage Engine - Aria
Affects Version/s: 10.2.9
Fix Version/s: N/A

Type: Bug Priority: Major
Reporter: xOOn Assignee: Unassigned
Resolution: Incomplete Votes: 1
Labels: None
Environment:

Windows 2016 Server

Attachments: Zip Archive mysqld.exe.4576.zip     PNG File screenshot-1.png    
Issue Links:
Relates
relates to MDEV-15237 "Can't write; duplicate key in table"... Closed
relates to MDEV-15304 Server crash in print_keydup_error / ... Closed

 Description   

We use mariaDb 10.2.9 with Aria storage engine, the same stored procedure was working on MySQL 5.1 without any problems over years.

After some day of working (we got the problem multiple times) on the execution of a stored procedure we got

2017-10-26  2:59:34 4592 [ERROR] mysqld.exe: Index for table 'po_data_mem' is corrupt; try to repair it

and after

171026  2:59:34 [ERROR] mysqld got exception 0xc0000005 ;
 
This could be because you hit a bug. It is also possible that this binary
 
or one of the libraries it was linked against is corrupt, improperly built,
 
or misconfigured. This error can also be caused by malfunctioning hardware.
 
 
 
To report this bug, see https://mariadb.com/kb/en/reporting-bugs
 
 
 
We will try our best to scrape up some info that will hopefully help
 
diagnose the problem, but since we have already crashed, 
something is definitely wrong and this may fail.
 
 
 
Server version: 10.2.9-MariaDB
 
key_buffer_size=134217728
 
read_buffer_size=131072
 
max_used_connections=8
 
max_threads=65537
 
thread_count=12
 
It is possible that mysqld could use up to 
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 135244 K  bytes of memory
 
Hope that's ok; if not, decrease some variables in the equation.
 
Thread pointer: 0x60e16c88
 
Attempting backtrace. You can use the following information to find out
 
where mysqld died. If you see no messages after this, something went
 
terribly wrong...
 
mysqld.exe!strnlen()[strnlen.cpp:165]
 
mysqld.exe!process_str_arg()[my_vsnprintf.c:205]
 
mysqld.exe!my_vsnprintf_ex()[my_vsnprintf.c:627]
 
mysqld.exe!my_printf_error()[my_error.c:154]
 
mysqld.exe!print_keydup_error()[handler.cc:3341]
 
mysqld.exe!print_keydup_error()[handler.cc:3354]
 
mysqld.exe!handler::print_error()[handler.cc:3431]
 
mysqld.exe!write_record()[sql_insert.cc:1959]
 
mysqld.exe!mysql_insert()[sql_insert.cc:1055]
 
mysqld.exe!mysql_execute_command()[sql_parse.cc:4411]
 
mysqld.exe!mysql_parse()[sql_parse.cc:7875]
 
mysqld.exe!dispatch_command()[sql_parse.cc:1811]
 
mysqld.exe!do_command()[sql_parse.cc:1359]
 
mysqld.exe!threadpool_process_request()[threadpool_common.cc:366]
 
mysqld.exe!tp_callback()[threadpool_common.cc:192]
 
mysqld.exe!work_callback()[threadpool_win.cc:451]
 
ntdll.dll!TpReleaseTimer()
 
ntdll.dll!TpReleaseTimer()
 
KERNEL32.DLL!BaseThreadInitThunk()
 
ntdll.dll!RtlSubscribeWnfStateChangeNotification()
 
ntdll.dll!RtlSubscribeWnfStateChangeNotification()
 
 
 
Trying to get some variables.
 
Some pointers may be invalid and cause the dump to abort.
 
Query (0x5fac2020): INSERT INTO devicecache_maintenances_operatinghours (DefinitionId, DeviceId, DeviceNameSource, DeviceOphSource, CurrentOperatingHours, LastOperatingHoursMaintained) VALUES ('ta_check_vibration', 2003225, 'Name', 'OperationHours', 61564, 61564) ON DUPLICATE KEY UPDATE CurrentOperatingHours=61564, LastOperatingHoursMaintained=61564
 
Connection ID (thread ID): 2155
 
Status: NOT_KILLED
 
 
 
Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=off,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on
 
 
 
The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
 
information that should help you find out what is causing the crash.



 Comments   
Comment by Elena Stepanova [ 2017-10-30 ]

xoon,

Your server must have created a dmp file upon the crash. Could you please attach it or, if you so prefer, upload it to ftp.askmonty.org/private ?
Please also attach your ini file and paste the output of the following:

SHOW CREATE TABLE devicecache_maintenances_operatinghours;
 
SHOW INDEX IN devicecache_maintenances_operatinghours;
 
SHOW TABLE STATUS LIKE 'devicecache_maintenances_operatinghours';
 
SHOW TRIGGERS LIKE 'devicecache_maintenances_operatinghours';

Comment by xOOn [ 2017-11-02 ]

our my.ini

[mysqld]
datadir=C:/Program Files (x86)/MariaDB 10.2/data
port=3306
innodb_buffer_pool_size=1003M
character-set-server=utf8
[client]
port=3306
plugin-dir=C:/Program Files (x86)/MariaDB 10.2/lib/plugin

Comment by xOOn [ 2017-11-02 ]

CREATE TABLE `devicecache_maintenances_operatinghours` (
`DefinitionId` varchar(60) NOT NULL,
`DeviceId` int(11) unsigned NOT NULL,
`DeviceNameSource` varchar(50) NOT NULL,
`DeviceOphSource` varchar(50) NOT NULL,
`CurrentOperatingHours` double NOT NULL,
`LastOperatingHoursMaintained` double NOT NULL,
PRIMARY KEY (`DeviceId`,`DeviceNameSource`,`DeviceOphSource`,`DefinitionId`)
) ENGINE=Aria DEFAULT CHARSET=utf8 PAGE_CHECKSUM=1 TRANSACTIONAL=1

"Table" "Non_unique" "Key_name" "Seq_in_index" "Column_name" "Collation" "Cardinality" "Sub_part" "Packed" "Null" "Index_type" "Comment" "Index_comment"
"devicecache_maintenances_operatinghours" "0" "PRIMARY" "1" "DeviceId" "A" "1825" \N \N "" "BTREE" "" ""
"devicecache_maintenances_operatinghours" "0" "PRIMARY" "2" "DeviceNameSource" "A" "1825" \N \N "" "BTREE" "" ""
"devicecache_maintenances_operatinghours" "0" "PRIMARY" "3" "DeviceOphSource" "A" "1825" \N \N "" "BTREE" "" ""
"devicecache_maintenances_operatinghours" "0" "PRIMARY" "4" "DefinitionId" "A" "34689" \N \N "" "BTREE" "" ""

"Name" "Engine" "Version" "Row_format" "Rows" "Avg_row_length" "Data_length" "Max_data_length" "Index_length" "Data_free" "Auto_increment" "Create_time" "Update_time" "Check_time" "Collation" "Checksum" "Create_options" "Comment"
"devicecache_maintenances_operatinghours" "Aria" "10" "Page" "34686" "76" "2637824" "17592186011648" "2179072" "0" \N "2017-10-17 14:44:18" "2017-11-02 08:37:40" "2017-11-02 08:21:01" "utf8_general_ci" \N "transactional=1" ""

SHOW TRIGGERS LIKE 'devicecache_maintenances_operatinghours';
is empty

Comment by xOOn [ 2017-11-02 ]

where is located the crash dump?

Comment by Vladislav Vaintroub [ 2017-11-02 ]

if you have core-file entry in [mysqld] section in my.ini, then it would be located in datadir.

Otherwise, Windows Error Reporting, also known under abbreviation WER lets you configure the place to store local dumps.
https://msdn.microsoft.com/en-us/library/windows/desktop/bb787181(v=vs.85).aspx

Comment by xOOn [ 2017-11-02 ]

mysqld.exe.4576.zip

here is the dump file.

Comment by Elena Stepanova [ 2018-01-23 ]

From the dump:

>	mysqld.exe!strnlen(const char * string, unsigned int maximum_count) Line 165	C++
 
 	mysqld.exe!process_str_arg(const charset_info_st * cs, char * to, const char * end, unsigned int width, char * par, unsigned int print_type) Line 205	C
 
 	mysqld.exe!my_vsnprintf_ex(const charset_info_st * cs, char * to, unsigned int n, const char * fmt, char * ap) Line 627	C
 
 	mysqld.exe!my_printf_error(unsigned int error, const char * format, unsigned long MyFlags, ...) Line 154	C
 
 	mysqld.exe!print_keydup_error(TABLE * table, st_key * key, const char * msg, unsigned long errflag) Line 3341	C++
 
 	mysqld.exe!print_keydup_error(TABLE * table, st_key * key, unsigned long errflag) Line 3354	C++
 
 	mysqld.exe!handler::print_error(int error, unsigned long errflag) Line 3431	C++
 
 	mysqld.exe!write_record(THD * thd, TABLE * table, st_copy_info * info) Line 1959	C++
 
 	mysqld.exe!mysql_insert(THD * thd, TABLE_LIST * table_list, List<Item> & fields, List<List<Item> > & values_list, List<Item> & update_fields, List<Item> & update_values, enum_duplicates duplic, bool ignore) Line 1055	C++
 
 	mysqld.exe!mysql_execute_command(THD * thd) Line 4411	C++
 
 	mysqld.exe!mysql_parse(THD * thd, char * rawbuf, unsigned int length, Parser_state * parser_state, bool is_com_multi, bool is_next_command) Line 7875	C++
 
 	mysqld.exe!dispatch_command(enum_server_command command, THD * thd, char * packet, unsigned int packet_length, bool is_com_multi, bool is_next_command) Line 1811	C++
 
 	mysqld.exe!do_command(THD * thd) Line 1359	C++
 
 	mysqld.exe!threadpool_process_request(THD * thd) Line 366	C++
 
 	mysqld.exe!tp_callback(TP_connection * c) Line 192	C++
 
 	mysqld.exe!work_callback(_TP_CALLBACK_INSTANCE * instance, void * context, _TP_WORK * work) Line 451	C++
 
 	ntdll.dll!TppWorkpExecuteCallback()	Unknown
 
 	ntdll.dll!TppWorkerThread()	Unknown
 
 	kernel32.dll!@BaseThreadInitThunk@12()	Unknown
 
 	ntdll.dll!__RtlUserThreadStart()	Unknown
 
 	ntdll.dll!__RtlUserThreadStart@8()	Unknown


>Debug.ListCallStack /AllThreads /ShowExternalCode:no 
Callstack for Thread 1 (Thread Id: 4824 (0x12d8)):
 
 Index  Function
 
--------------------------------------------------------------------------------
 
 1      [External Code]
 
 2      mysqld.exe!_initterm(void(*)() * first=0x000000a4, void(*)() * last=0xffffffff)
 
 3      [External Code]
 
 
 
Callstack for Thread 2 (Thread Id: 5916 (0x171c)):
 
 Index  Function
 
--------------------------------------------------------------------------------
 
 1      [External Code]
 
 
 
Callstack for Thread 3 (Thread Id: 3108 (0xc24)):
 
 Index  Function
 
--------------------------------------------------------------------------------
 
 1      [External Code]
 
 2      mysqld.exe!thread_start<void (__cdecl*)(void *)>(void * const parameter=0x77a4013c)
 
 
 
Callstack for Thread 4 (Thread Id: 5880 (0x16f8)):
 
 Index  Function
 
--------------------------------------------------------------------------------
 
 1      [External Code]
 
 2      mysqld.exe!pthread_start(void * p=0x029ef520)
 
 3      mysqld.exe!thread_start<unsigned int (__stdcall*)(void *)>(void * const parameter=0x029f13d0)
 
 4      [External Code]
 
 
 
Callstack for Thread 5 (Thread Id: 5628 (0x15fc)):
 
 Index  Function
 
--------------------------------------------------------------------------------
 
 1      [External Code]
 
 
 
Callstack for Thread 6 (Thread Id: 6096 (0x17d0)):
 
 Index  Function
 
--------------------------------------------------------------------------------
 
 1      [External Code]
 
 
 
Callstack for Thread 7 (Thread Id: 3976 (0xf88)):
 
 Index  Function
 
--------------------------------------------------------------------------------
 
 1      [External Code]
 
 
 
Callstack for Thread 8 (Thread Id: 6020 (0x1784)):
 
 Index  Function
 
--------------------------------------------------------------------------------
 
 1      [External Code]
 
 
 
Callstack for Thread 9 (Thread Id: 5056 (0x13c0)):
 
 Index  Function
 
--------------------------------------------------------------------------------
 
 1      [External Code]
 
 
 
Callstack for Thread 10 (Thread Id: 5264 (0x1490)):
 
 Index  Function
 
--------------------------------------------------------------------------------
 
 1      [External Code]
 
 
 
Callstack for Thread 11 (Thread Id: 5884 (0x16fc)):
 
 Index  Function
 
--------------------------------------------------------------------------------
 
 1      [External Code]
 
 
 
Callstack for Thread 12 (Thread Id: 1536 (0x600)):
 
 Index  Function
 
--------------------------------------------------------------------------------
 
 1      [External Code]
 
 
 
Callstack for Thread 13 (Thread Id: 2316 (0x90c)):
 
 Index  Function
 
--------------------------------------------------------------------------------
 
 1      [External Code]
 
 
 
Callstack for Thread 14 (Thread Id: 296 (0x128)):
 
 Index  Function
 
--------------------------------------------------------------------------------
 
 1      [External Code]
 
 
 
Callstack for Thread 15 (Thread Id: 4960 (0x1360)):
 
 Index  Function
 
--------------------------------------------------------------------------------
 
 1      [External Code]
 
 
 
Callstack for Thread 16 (Thread Id: 4488 (0x1188)):
 
 Index  Function
 
--------------------------------------------------------------------------------
 
 1      [External Code]
 
 
 
Callstack for Thread 17 (Thread Id: 3092 (0xc14)):
 
 Index  Function
 
--------------------------------------------------------------------------------
 
 1      [External Code]
 
 
 
Callstack for Thread 18 (Thread Id: 5768 (0x1688)):
 
 Index  Function
 
--------------------------------------------------------------------------------
 
 1      [External Code]
 
 
 
Callstack for Thread 19 (Thread Id: 428 (0x1ac)):
 
 Index  Function
 
--------------------------------------------------------------------------------
 
 1      [External Code]
 
 
 
Callstack for Thread 20 (Thread Id: 5036 (0x13ac)):
 
 Index  Function
 
--------------------------------------------------------------------------------
 
 1      [External Code]
 
 
 
Callstack for Thread 21 (Thread Id: 4872 (0x1308)):
 
 Index  Function
 
--------------------------------------------------------------------------------
 
 1      [External Code]
 
 
 
Callstack for Thread 22 (Thread Id: 4908 (0x132c)):
 
 Index  Function
 
--------------------------------------------------------------------------------
 
 1      [External Code]
 
 
 
Callstack for Thread 23 (Thread Id: 1188 (0x4a4)):
 
 Index  Function
 
--------------------------------------------------------------------------------
 
 1      [External Code]
 
 
 
Callstack for Thread 24 (Thread Id: 5104 (0x13f0)):
 
 Index  Function
 
--------------------------------------------------------------------------------
 
 1      [External Code]
 
 
 
Callstack for Thread 25 (Thread Id: 116 (0x74)):
 
 Index  Function
 
--------------------------------------------------------------------------------
 
 1      [External Code]
 
 
 
Callstack for Thread 26 (Thread Id: 4432 (0x1150)):
 
 Index  Function
 
--------------------------------------------------------------------------------
 
 1      [External Code]
 
 
 
Callstack for Thread 27 (Thread Id: 5988 (0x1764)):
 
 Index  Function
 
--------------------------------------------------------------------------------
 
 1      [External Code]
 
 
 
Callstack for Thread 28 (Thread Id: 4008 (0xfa8)):
 
 Index  Function
 
--------------------------------------------------------------------------------
 
 1      [External Code]
 
 
 
Callstack for Thread 29 (Thread Id: 4476 (0x117c)):
 
 Index  Function
 
--------------------------------------------------------------------------------
 
 1      [External Code]
 
 2      mysqld.exe!pthread_start(void * p=0x02ade7a8)
 
 3      mysqld.exe!thread_start<unsigned int (__stdcall*)(void *)>(void * const parameter=0x34ffaa50)
 
 4      [External Code]
 
 
 
Callstack for Thread 30 (Thread Id: 5872 (0x16f0)):
 
 Index  Function
 
--------------------------------------------------------------------------------
 
 1      [External Code]
 
 2      mysqld.exe!my_sync(int fd=1535376172, unsigned long my_flags)
 
 3      mysqld.exe!pthread_start(void * p=0x02ade758)
 
 4      mysqld.exe!thread_start<unsigned int (__stdcall*)(void *)>(void * const parameter=0x34ffa7d0)
 
 5      [External Code]
 
 
 
Callstack for Thread 31 (Thread Id: 4788 (0x12b4)):
 
 Index  Function
 
--------------------------------------------------------------------------------
 
*1      mysqld.exe!strnlen(const char * string=0x00000150, unsigned int maximum_count=192)
 
 2      mysqld.exe!process_str_arg(const charset_info_st * cs=0x018ce298, char * to=0x5b97bf04, const char * end=0x5b97c0e7, unsigned int width=192, char * par=0x00000150, unsigned int print_type=0)
 
 3      mysqld.exe!my_vsnprintf_ex(const charset_info_st * cs=0x018ce298, char * to=0x5b97bf04, unsigned int n=512, const char * fmt=0x02a13b5f, char * ap=0x5b97c108)
 
 4      mysqld.exe!my_printf_error(unsigned int error=1062, const char * format=0x02a13b37, unsigned long MyFlags=0, ...)
 
 5      mysqld.exe!print_keydup_error(TABLE * table=0x5ddd4f70, st_key * key=0x5dc938a8, const char * msg=0x02a13b37, unsigned long errflag=0)
 
 6      mysqld.exe!print_keydup_error(TABLE * table=0x5ddd4f70, st_key * key=0x5dc938a8, unsigned long errflag=0)
 
 7      mysqld.exe!handler::print_error(int error=121, unsigned long errflag=0)
 
 8      mysqld.exe!write_record(THD * thd=0x60e16c88, TABLE * table=0x5ddd4f70, st_copy_info * info=0x5b97ddf0)
 
 9      mysqld.exe!mysql_insert(THD * thd=0x60e16c88, TABLE_LIST * table_list=0x5fac2388, List<Item> & fields, List<List<Item> > & values_list, List<Item> & update_fields, List<Item> & update_values, enum_duplicates duplic=DUP_UPDATE, bool ignore=false)
 
 10     mysqld.exe!mysql_execute_command(THD * thd=0x60e16c88)
 
 11     mysqld.exe!mysql_parse(THD * thd=0x60e16c88, char * rawbuf=0x5fac2020, unsigned int length=331, Parser_state * parser_state=0x5b97ee38, bool is_com_multi=false, bool is_next_command=false)
 
 12     mysqld.exe!dispatch_command(enum_server_command command=COM_QUERY, THD * thd=0x60e16c88, char * packet=0x5d202d81, unsigned int packet_length=331, bool is_com_multi=false, bool is_next_command=false)
 
 13     mysqld.exe!do_command(THD * thd=0x60e16c88)
 
 14     mysqld.exe!threadpool_process_request(THD * thd=0x60e16c88)
 
 15     mysqld.exe!tp_callback(TP_connection * c=0x3515d868)
 
 16     mysqld.exe!work_callback(_TP_CALLBACK_INSTANCE * instance=0x5b97f740, void * context=0x3515d868, _TP_WORK * work=0x02ad8078)
 
 17     [External Code]
 
 
 
Callstack for Thread 32 (Thread Id: 1716 (0x6b4)):
 
 Index  Function
 
--------------------------------------------------------------------------------
 
 1      [External Code]
 
 2      mysqld.exe!pthread_start(void * p=0x00000258)
 
 3      mysqld.exe!thread_start<unsigned int (__stdcall*)(void *)>(void * const parameter=0x00000258)
 
 4      [External Code]
 
 
 
Callstack for Thread 33 (Thread Id: 648 (0x288)):
 
 Index  Function
 
--------------------------------------------------------------------------------
 
 1      [External Code]
 
 2      mysqld.exe!pthread_start(void * p=0x02ade8f8)
 
 3      mysqld.exe!thread_start<unsigned int (__stdcall*)(void *)>(void * const parameter=0x34ffa6b0)
 
 4      [External Code]
 
 
 
Callstack for Thread 34 (Thread Id: 5340 (0x14dc)):
 
 Index  Function
 
--------------------------------------------------------------------------------
 
 1      [External Code]
 
 2      mysqld.exe!pthread_start(void * p=0xcccccccc)
 
 3      [External Code]
 
 
 
Callstack for Thread 35 (Thread Id: 4592 (0x11f0)):
 
 Index  Function
 
--------------------------------------------------------------------------------
 
 1      [External Code]
 
 
 
Callstack for Thread 36 (Thread Id: 5968 (0x1750)):
 
 Index  Function
 
--------------------------------------------------------------------------------
 
 1      [External Code]
 
 
 
Callstack for Thread 37 (Thread Id: 7820 (0x1e8c)):
 
 Index  Function
 
--------------------------------------------------------------------------------
 
 1      [External Code]
 
 
 
Callstack for Thread 38 (Thread Id: 4460 (0x116c)):
 
 Index  Function
 
--------------------------------------------------------------------------------
 
 1      [External Code]

Comment by Michael Widenius [ 2018-01-23 ]

I have no way to repeat the error. However the only logical reason for the crash is that the storage engine somehow returns the wrong key to be used by print_keydup_error().

I have now made a fix that should ensure that print_keydup_error() will never get a reference to an unknown key.

If the storage engine in the future reports a wrong key, the error will be duplicate key error with UNKNOWN key. If this happens, please try to create a test case and we will fix the storage engine code that causes this!

Please comment if this is fixed in the next release so that we can close this issue.

Comment by Elena Stepanova [ 2018-02-13 ]

I have a test case which causes the same crash with RocksDB – and indeed, now it returns ER_DUP_KEY: MDEV-15304
Probably it's somewhat better for production than a crash, but it's highly confusing, because the error happens on a table which has no reason whatsoever to produce a duplicate key error.

Comment by xOOn [ 2018-02-15 ]

Hello,

we had this problem 3 times in 2 weeks. To fix the problem we removed the TRANSACTIONAL flag, we created all tables now with TRANSACTIONAL=0 and it never happens again. We have no idea why.

Comment by Michael Widenius [ 2018-04-06 ]

Which version are you using ?
The fix are in the following versions:
mariadb-10.0.34
mariadb-10.1.31
mariadb-10.2.13
mariadb-10.3.5

Regards,
Monty

Generated at Thu Feb 08 08:11:14 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.