|
Got another crash:
Server version: 10.4.8-MariaDB
|
key_buffer_size=134217728
|
read_buffer_size=67108864
|
max_used_connections=37
|
max_threads=65537
|
thread_count=16
|
It is possible that mysqld could use up to
|
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 266289 K bytes of memory
|
Hope that's ok; if not, decrease some variables in the equation.
|
|
Thread pointer: 0xd5a523ef18
|
Attempting backtrace. You can use the following information to find out
|
where mysqld died. If you see no messages after this, something went
|
terribly wrong...
|
mysqld.exe!btr_cur_latch_leaves()[btr0cur.cc:336]
|
mysqld.exe!btr_cur_search_to_nth_level_func()[btr0cur.cc:1791]
|
mysqld.exe!btr_pcur_restore_position_func()[btr0pcur.cc:384]
|
mysqld.exe!row_purge_reposition_pcur()[row0purge.cc:75]
|
mysqld.exe!row_purge_remove_clust_if_poss_low()[row0purge.cc:115]
|
mysqld.exe!row_purge_remove_clust_if_poss()[row0purge.cc:201]
|
mysqld.exe!row_purge_del_mark()[row0purge.cc:785]
|
mysqld.exe!row_purge_record_func()[row0purge.cc:1195]
|
mysqld.exe!row_purge()[row0purge.cc:1265]
|
mysqld.exe!row_purge_step()[row0purge.cc:1323]
|
mysqld.exe!que_thr_step()[que0que.cc:1037]
|
mysqld.exe!que_run_threads_low()[que0que.cc:1102]
|
mysqld.exe!que_run_threads()[que0que.cc:1141]
|
mysqld.exe!srv_task_execute()[srv0srv.cc:2458]
|
mysqld.exe!srv_worker_thread()[srv0srv.cc:2505]
|
KERNEL32.DLL!BaseThreadInitThunk()
|
ntdll.dll!RtlUserThreadStart()
|
|
Trying to get some variables.
|
Some pointers may be invalid and cause the dump to abort.
|
Query (0x0):
|
Connection ID (thread ID): 2
|
Status: NOT_KILLED
|
|
Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=on,condition_pushdown_from_having=on
|
|
The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
|
information that should help you find out what is causing the crash.
|
Writing a core file at D:\MariaDB 10.4\data\
|
Minidump written to D:\MariaDB 10.4\data\mysqld.dmp
|
|
|
Possibly related to MDEV-20563
|
|
With 10.5.4 i got similar crashes on excessive DELETE and INSERT:
200817 21:46:59 [ERROR] mysqld got exception 0xc0000005 ;
|
This could be because you hit a bug. It is also possible that this binary
|
or one of the libraries it was linked against is corrupt, improperly built,
|
or misconfigured. This error can also be caused by malfunctioning hardware.
|
|
To report this bug, see https://mariadb.com/kb/en/reporting-bugs
|
|
We will try our best to scrape up some info that will hopefully help
|
diagnose the problem, but since we have already crashed,
|
something is definitely wrong and this may fail.
|
|
Server version: 10.5.4-MariaDB
|
key_buffer_size=134217728
|
read_buffer_size=67108864
|
max_used_connections=36
|
max_threads=65537
|
thread_count=9
|
It is possible that mysqld could use up to
|
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 266322 K bytes of memory
|
Hope that's ok; if not, decrease some variables in the equation.
|
|
Thread pointer: 0xf9ea82188
|
Attempting backtrace. You can use the following information to find out
|
where mysqld died. If you see no messages after this, something went
|
terribly wrong...
|
server.dll!TABLE::validate_default_values_of_unset_fields()[table.cc:8928]
|
server.dll!mysql_insert()[sql_insert.cc:945]
|
server.dll!mysql_execute_command()[sql_parse.cc:4553]
|
server.dll!mysql_parse()[sql_parse.cc:7997]
|
server.dll!dispatch_command()[sql_parse.cc:1877]
|
server.dll!do_command()[sql_parse.cc:1355]
|
server.dll!threadpool_process_request()[threadpool_common.cc:354]
|
server.dll!tp_callback()[threadpool_common.cc:194]
|
KERNEL32.DLL!VirtualUnlock()
|
ntdll.dll!RtlGetActiveActivationContext()
|
ntdll.dll!RtlFreeUnicodeString()
|
KERNEL32.DLL!BaseThreadInitThunk()
|
ntdll.dll!RtlUserThreadStart()
|
|
Trying to get some variables.
|
Some pointers may be invalid and cause the dump to abort.
|
Query (0xfb87d2050): insert into CashFlow (flightRecordId, duration, movedNotes, startSeconds, state, cashFlowId) values (3882822, 18, 4, 31831, 'CASH_IN', 316955440)
|
Connection ID (thread ID): 8858331
|
Status: NOT_KILLED
|
|
Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=on,condition_pushdown_from_having=on,not_null_range_scan=off
|
|
The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
|
information that should help you find out what is causing the crash.
|
Writing a core file at D:\MariaDB 10.5\data\
|
200817 22:06:17 [ERROR] mysqld got exception 0xc0000005 ;
|
This could be because you hit a bug. It is also possible that this binary
|
or one of the libraries it was linked against is corrupt, improperly built,
|
or misconfigured. This error can also be caused by malfunctioning hardware.
|
|
To report this bug, see https://mariadb.com/kb/en/reporting-bugs
|
|
We will try our best to scrape up some info that will hopefully help
|
diagnose the problem, but since we have already crashed,
|
something is definitely wrong and this may fail.
|
|
Server version: 10.5.4-MariaDB
|
key_buffer_size=134217728
|
read_buffer_size=67108864
|
max_used_connections=24
|
max_threads=65537
|
thread_count=13
|
It is possible that mysqld could use up to
|
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 266322 K bytes of memory
|
Hope that's ok; if not, decrease some variables in the equation.
|
|
Thread pointer: 0x771da42f88
|
Attempting backtrace. You can use the following information to find out
|
where mysqld died. If you see no messages after this, something went
|
terribly wrong...
|
server.dll!my_free()[my_malloc.c:198]
|
server.dll!my_hash_free()[hash.c:156]
|
server.dll!THD::~THD()[sql_class.cc:1747]
|
server.dll!THD::`scalar deleting destructor'()
|
server.dll!threadpool_remove_connection()[threadpool_common.cc:296]
|
server.dll!tp_callback()[threadpool_common.cc:218]
|
ntdll.dll!RtlFreeUnicodeString()
|
ntdll.dll!RtlFreeUnicodeString()
|
KERNEL32.DLL!BaseThreadInitThunk()
|
ntdll.dll!RtlUserThreadStart()
|
|
Trying to get some variables.
|
Some pointers may be invalid and cause the dump to abort.
|
Query (0x0):
|
Connection ID (thread ID): 64842
|
Status: KILL_CONNECTION
|
|
Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=on,condition_pushdown_from_having=on,not_null_range_scan=off
|
|
|
|
sstamm All reported crashes in the comments look unrelated to the original report and unrelated to each other. I wonder if there are other issues here like physical memory or disk in your system being faulty. Can you please run an exhaustive (offline) memory and disk test please? Can you also upload full error logs to see if there are other obvious issues? One other thing to clarify; what makes you say 'excessive' delete/inserts?
|
|
By excessive I mean we have a InnoDB-Database with 48 cascaded Tables. Server is running under load with approx. 25.000 IOP/s (most of them are bulk insert and some cascaded DELETES).
Server is a Cisco UCS with 256GB of ECC Memory and RAID1 NVMe SAS-drives, which reports none of them as faulty.
So funlikely but I'll try to schedule a test at the next maintenance reboot.
|
|
Thank you. Please also check SMART statistics if available for the drives (tools; https://en.wikipedia.org/wiki/Comparison_of_S.M.A.R.T._tools), as well as dmesg and I/O subsystem logs, especially around the times of the issue. Was MDEV-20897 from the same server? There seems to be I/O issues there ("quite plausible" level).
|
|
@Roel
SMART and Disk looks fine on RAID-Level (for memory we'still have to check offline)
But I got A different crash, does it also looks like a plausible memory problem?
2020-08-26 10:43:43 93336 [ERROR] InnoDB: Unable to decompress .\fcc_1300\refusecashtypes.ibd[page id: space=3540, page number=664534]
|
2020-08-26 10:43:43 93331 [ERROR] InnoDB: Unable to decompress .\fcc_1300\sensorcurrent.ibd[page id: space=3531, page number=4303351]
|
2020-08-26 10:43:43 93336 [ERROR] InnoDB: In cascade of a foreign key op index `FK8hm81oqvmnrcytkomqfcl7jgk` of table `fcc_1300`.`refusecashtypes`
|
2020-08-26 10:43:43 0xc5c InnoDB: Assertion failure in file D:\winx64-packages\build\src\storage\innobase\page\page0page.cc line 803
|
InnoDB: We intentionally generate a memory trap.
|
InnoDB: Submit a detailed bug report to https://jira.mariadb.org/
|
InnoDB: If you get repeated assertion failures or crashes, even
|
InnoDB: immediately after the mysqld startup, there may be
|
InnoDB: corruption in the InnoDB tablespace. Please refer to
|
InnoDB: https://mariadb.com/kb/en/library/innodb-recovery-modes/
|
InnoDB: about forcing recovery.
|
2020-08-26 10:43:43 93331 [Warning] InnoDB: btr_pcur_open_low level: 0 called from file: D:\winx64-packages\build\src\storage\innobase\row\row0row.cc line: 1211 table: `fcc_1300`.`sensorcurrent` index: `PRIMARY` error: Page read from tablespace is corrupted.
|
InnoDB: record PHYSICAL RECORD: n_fields 2; compact format; info bits 0
|
0: len 4; hex 011b5ced; asc \ ;;
|
1: len 3; hex 000001; asc ;;
|
|
InnoDB: clustered record PHYSICAL RECORD: n_fields 3; compact format; info bits 0
|
0: len 3; hex 000001; asc ;;
|
1: len 4; hex 011b5c23; asc \#;;
|
2: len 4; hex 000a23d6; asc # ;;
|
|
InnoDB: Submit a detailed bug report to https://jira.mariadb.org/
|
2020-08-26 10:43:43 93336 [ERROR] InnoDB: Unable to decompress .\fcc_1300\refusecashtypes.ibd[page id: space=3540, page number=664477]
|
2020-08-26 10:43:43 93336 [Warning] InnoDB: btr_pcur_open_low level: 0 called from file: D:\winx64-packages\build\src\storage\innobase\row\row0row.cc line: 1211 table: `fcc_1300`.`refusecashtypes` index: `PRIMARY` error: Page read from tablespace is corrupted.
|
200826 10:43:43 [ERROR] mysqld got exception 0xc0000005 ;
|
This could be because you hit a bug. It is also possible that this binary
|
or one of the libraries it was linked against is corrupt, improperly built,
|
or misconfigured. This error can also be caused by malfunctioning hardware.
|
|
To report this bug, see https://mariadb.com/kb/en/reporting-bugs
|
|
We will try our best to scrape up some info that will hopefully help
|
diagnose the problem, but since we have already crashed,
|
something is definitely wrong and this may fail.
|
|
Server version: 10.5.4-MariaDB
|
key_buffer_size=134217728
|
read_buffer_size=67108864
|
max_used_connections=25
|
max_threads=65537
|
thread_count=14
|
It is possible that mysqld could use up to
|
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 266322 K bytes of memory
|
Hope that's ok; if not, decrease some variables in the equation.
|
|
Thread pointer: 0xf639a0ec88
|
|
|
sstamm, the last occurrence looks like a failure to decompress a ROW_FORMAT=COMPRESSED page. That code has been rather stable since I developed it between 2005 and 2007. Can you please test the memory subsystem?
|
|
@Marko
Performed a full test (with Cisco UCS Server Diagnostics Utility) without any Errors (Memory, Disks and RAID were fine).
But crashes are ongoing: (And yes, mostly on using COMPRESSED Tables)
201029 14:23:11 [ERROR] mysqld got exception 0xc0000005 ;
|
This could be because you hit a bug. It is also possible that this binary
|
or one of the libraries it was linked against is corrupt, improperly built,
|
or misconfigured. This error can also be caused by malfunctioning hardware.
|
|
To report this bug, see https://mariadb.com/kb/en/reporting-bugs
|
|
We will try our best to scrape up some info that will hopefully help
|
diagnose the problem, but since we have already crashed,
|
something is definitely wrong and this may fail.
|
|
Server version: 10.5.5-MariaDB
|
key_buffer_size=134217728
|
read_buffer_size=67108864
|
max_used_connections=25
|
max_threads=65537
|
thread_count=10
|
It is possible that mysqld could use up to
|
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 266324 K bytes of memory
|
Hope that's ok; if not, decrease some variables in the equation.
|
|
Thread pointer: 0xab43ceb038
|
Attempting backtrace. You can use the following information to find out
|
where mysqld died. If you see no messages after this, something went
|
terribly wrong...
|
server.dll!Field_long::store()[field.cc:4341]
|
server.dll!fill_record()[sql_base.cc:8451]
|
server.dll!fill_record_n_invoke_before_triggers()[sql_base.cc:8623]
|
server.dll!mysql_insert()[sql_insert.cc:973]
|
server.dll!mysql_execute_command()[sql_parse.cc:4546]
|
server.dll!mysql_parse()[sql_parse.cc:7998]
|
server.dll!dispatch_command()[sql_parse.cc:1870]
|
server.dll!do_command()[sql_parse.cc:1348]
|
server.dll!threadpool_process_request()[threadpool_common.cc:354]
|
server.dll!tp_callback()[threadpool_common.cc:194]
|
KERNEL32.DLL!VirtualUnlock()
|
ntdll.dll!RtlGetActiveActivationContext()
|
ntdll.dll!RtlFreeUnicodeString()
|
KERNEL32.DLL!BaseThreadInitThunk()
|
ntdll.dll!RtlUserThreadStart()
|
|
Trying to get some variables.
|
Some pointers may be invalid and cause the dump to abort.
|
Query (0xab3a7e6040): insert into CashFlow (flightRecordId, duration, movedNotes, startSeconds, state, cashFlowId) values (25069874, 13, 5, 38747, 'CASH_IN', 2177675658)
|
Connection ID (thread ID): 61043
|
Status: NOT_KILLED
|
|
2020-10-29 14:03:13 0x134c InnoDB: Assertion failure in file D:\winx64-packages\build\src\storage\innobase\page\page0page.cc line 803
|
InnoDB: We intentionally generate a memory trap.
|
InnoDB: Submit a detailed bug report to https://jira.mariadb.org/
|
InnoDB: If you get repeated assertion failures or crashes, even
|
InnoDB: immediately after the mysqld startup, there may be
|
InnoDB: corruption in the InnoDB tablespace. Please refer to
|
InnoDB: https://mariadb.com/kb/en/library/innodb-recovery-modes/
|
InnoDB: about forcing recovery.
|
201029 14:03:13 [ERROR] mysqld got exception 0x80000003 ;
|
This could be because you hit a bug. It is also possible that this binary
|
or one of the libraries it was linked against is corrupt, improperly built,
|
or misconfigured. This error can also be caused by malfunctioning hardware.
|
|
To report this bug, see https://mariadb.com/kb/en/reporting-bugs
|
|
We will try our best to scrape up some info that will hopefully help
|
diagnose the problem, but since we have already crashed,
|
something is definitely wrong and this may fail.
|
|
Server version: 10.5.5-MariaDB
|
key_buffer_size=134217728
|
read_buffer_size=67108864
|
max_used_connections=34
|
max_threads=65537
|
thread_count=9
|
It is possible that mysqld could use up to
|
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 266324 K bytes of memory
|
Hope that's ok; if not, decrease some variables in the equation.
|
|
Thread pointer: 0xe2d26a7048
|
Attempting backtrace. You can use the following information to find out
|
where mysqld died. If you see no messages after this, something went
|
terribly wrong...
|
server.dll!my_parameter_handler()[my_init.c:281]
|
ucrtbase.DLL!raise()
|
ucrtbase.DLL!abort()
|
server.dll!ut_dbg_assertion_failed()[ut0dbg.cc:60]
|
server.dll!page_copy_rec_list_start()[page0page.cc:803]
|
server.dll!btr_compress()[btr0btr.cc:3549]
|
server.dll!btr_cur_compress_if_useful()[btr0cur.cc:5425]
|
server.dll!btr_cur_pessimistic_delete()[btr0cur.cc:5863]
|
server.dll!row_undo_ins_remove_clust_rec()[row0uins.cc:194]
|
server.dll!row_undo_ins()[row0uins.cc:568]
|
server.dll!row_undo()[row0undo.cc:442]
|
server.dll!row_undo_step()[row0undo.cc:492]
|
server.dll!que_thr_step()[que0que.cc:945]
|
server.dll!que_run_threads_low()[que0que.cc:1012]
|
server.dll!que_run_threads()[que0que.cc:1051]
|
server.dll!trx_t::rollback_low()[trx0roll.cc:122]
|
server.dll!trx_rollback_for_mysql_low()[trx0roll.cc:189]
|
server.dll!trx_rollback_for_mysql()[trx0roll.cc:274]
|
server.dll!innobase_rollback()[ha_innodb.cc:4343]
|
server.dll!ha_rollback_trans()[handler.cc:2051]
|
server.dll!trans_rollback()[transaction.cc:374]
|
server.dll!mysql_execute_command()[sql_parse.cc:5623]
|
server.dll!mysql_parse()[sql_parse.cc:7998]
|
server.dll!dispatch_command()[sql_parse.cc:1870]
|
server.dll!do_command()[sql_parse.cc:1348]
|
server.dll!threadpool_process_request()[threadpool_common.cc:354]
|
server.dll!tp_callback()[threadpool_common.cc:194]
|
KERNEL32.DLL!VirtualUnlock()
|
ntdll.dll!RtlGetActiveActivationContext()
|
ntdll.dll!RtlFreeUnicodeString()
|
KERNEL32.DLL!BaseThreadInitThunk()
|
ntdll.dll!RtlUserThreadStart()
|
|
Trying to get some variables.
|
Some pointers may be invalid and cause the dump to abort.
|
Query (0xe2d892b060): ROLLBACK
|
Connection ID (thread ID): 1489668
|
Status: NOT_KILLED
|
|
mysqld.dmp