Details
-
Bug
-
Status: Closed (View Workflow)
-
Critical
-
Resolution: Fixed
-
10.3.11
-
None
-
linux
Description
where to begin.....
Over the last 24 hours i've had 3 (out of 5 total) mariadb production servers crash with similar errors (see attachment).
Each server has about 30 customer databases (identical structure) that customers connect to from an indentical win32 desktop using ODBC.
INNODB tables.
Not aware of any changes to the servers or the databases or the win32.exe.
On each server there has been, since yesterday, one database where if i run a couple of specific but seemingly innocuous queries the server crashes and restarts.
The error mentions an index.... so in each of the queries i (eventually) removed the indexes used by a subquery. and hey presto no crash. slow but no crash.
I must emphasise that i only removed the indexes from the one database on each server that was mentioned in the crash report. The other databases with the same structure on the same servers continue to work just fine with the same queries and the same indexes . And the database that are causing a crash have been same structure running same queries for months and months.
the databases causing the error are no larger than any of the others (a few thousand records in the deal and invoices tables).
i removed the indexes on invoices,dealid invoices.who invoices.term invoices.invoice deal.dealid
.dealid and .invoice and .term are int 11; .who is varchar 10
the other query that causes a crash is an update and uses invoice.invoice in a subquery
again, to emphasise, this query has been run a thousand times a day on 200 or more databases for more than a year with no problem... until yesterday on 3 databases
Attachments
Issue Links
- blocks
-
MDEV-28349 Provide "crash safe" options for CHECK TABLE and ALTER TABLE ... CHECK PARTITION ...
-
- Open
-
- causes
-
-
- Closed
-
- is blocked by
-
-
- Closed
-
- is duplicated by
-
-
- Closed
-
-
-
- Closed
-
-
-
- Closed
-
- relates to
-
SAMU-289 Loading...
(1 is duplicated by, 1 relates to)
Activity
| Field | Original Value | New Value |
|---|---|---|
| Priority | Major [ 3 ] | Critical [ 2 ] |
| Attachment | crashdump.txt [ 49460 ] |
today (but not yesterday) check table is reporting index corruptions.
these were deleted and recreated yesterday on at least one of the tables
From the log:
2019-11-19 09:13:55 0x7fd739d58700 InnoDB: Assertion failure in file /home/buildbot/buildbot/build/mariadb-10.3.11/storage/innobase/rem/rem0rec.cc line 820
|
InnoDB: We intentionally generate a memory trap.
|
InnoDB: Submit a detailed bug report to https://jira.mariadb.org/
|
InnoDB: If you get repeated assertion failures or crashes, even
|
InnoDB: immediately after the mysqld startup, there may be
|
InnoDB: corruption in the InnoDB tablespace. Please refer to
|
InnoDB: https://mariadb.com/kb/en/library/xtradbinnodb-recovery-modes/
|
InnoDB: about forcing recovery.
|
191119 9:13:55 [ERROR] mysqld got signal 6 ;
|
This could be because you hit a bug. It is also possible that this binary
|
or one of the libraries it was linked against is corrupt, improperly built,
|
or misconfigured. This error can also be caused by malfunctioning hardware.
|
To report this bug, see https://mariadb.com/kb/en/reporting-bugs
|
We will try our best to scrape up some info that will hopefully help
|
diagnose the problem, but since we have already crashed,
|
something is definitely wrong and this may fail.
|
Server version: 10.3.11-MariaDB-1:10.3.11+maria~cosmic-log
|
key_buffer_size=134217728
|
read_buffer_size=2097152
|
max_used_connections=60
|
max_threads=502
|
thread_count=67
|
It is possible that mysqld could use up to
|
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 3226530 K bytes of memory
|
Hope that's ok; if not, decrease some variables in the equation.
|
Thread pointer: 0x7fd7242a8df8
|
Attempting backtrace. You can use the following information to find out
|
where mysqld died. If you see no messages after this, something went
|
terribly wrong...
|
stack_bottom = 0x7fd739d57dd8 thread_stack 0x49000
|
/usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x5606afee0c7e]
|
/usr/sbin/mysqld(handle_fatal_signal+0x505)[0x5606afa05575]
|
/lib/x86_64-linux-gnu/libpthread.so.0(+0x12dd0)[0x7fd797324dd0]
|
/lib/x86_64-linux-gnu/libc.so.6(gsignal+0xc7)[0x7fd796e4f077]
|
/lib/x86_64-linux-gnu/libc.so.6(abort+0x121)[0x7fd796e30535]
|
/usr/sbin/mysqld(+0x4c0aec)[0x5606af742aec]
|
/usr/sbin/mysqld(+0x4aa3bb)[0x5606af72c3bb]
|
/usr/sbin/mysqld(+0x9677c2)[0x5606afbe97c2]
|
/usr/sbin/mysqld(+0xa41ea1)[0x5606afcc3ea1]
|
/usr/sbin/mysqld(+0x9cc5de)[0x5606afc4e5de]
|
/usr/sbin/mysqld(+0x9033f1)[0x5606afb853f1]
|
/usr/sbin/mysqld(_ZN7handler17ha_index_read_mapEPhPKhm16ha_rkey_function+0x178)[0x5606afa0a858]
|
/usr/sbin/mysqld(+0x5f5058)[0x5606af877058]
|
/usr/sbin/mysqld(_Z10sub_selectP4JOINP13st_join_tableb+0x18e)[0x5606af86697e]
|
/usr/sbin/mysqld(_ZN4JOIN10exec_innerEv+0xb02)[0x5606af888be2]
|
/usr/sbin/mysqld(_ZN4JOIN4execEv+0x33)[0x5606af888e23]
|
/usr/sbin/mysqld(_ZN30subselect_single_select_engine4execEv+0x145)[0x5606afa9ca75]
|
/usr/sbin/mysqld(_ZN14Item_subselect4execEv+0x4d)[0x5606afa9c35d]
|
/usr/sbin/mysqld(_ZN24Item_singlerow_subselect11val_decimalEP10my_decimal+0x36)[0x5606afa9da36]
|
/usr/sbin/mysqld(_ZN18Item_cache_decimal11cache_valueEv+0x37)[0x5606afa15227]
|
/usr/sbin/mysqld(_ZN18Item_cache_wrapper7val_strEP6String+0x7f)[0x5606afa2a40f]
|
/usr/sbin/mysqld(_ZNK12Type_handler13Item_send_strEP4ItemP8ProtocolP8st_value+0x1c)[0x5606af96a99c]
|
/usr/sbin/mysqld(_ZN8Protocol19send_result_set_rowEP4ListI4ItemE+0xa3)[0x5606af78ba93]
|
/usr/sbin/mysqld(_ZN11select_send9send_dataER4ListI4ItemE+0x5b)[0x5606af7f019b]
|
/usr/sbin/mysqld(+0x5f6be5)[0x5606af878be5]
|
/usr/sbin/mysqld(+0x5da446)[0x5606af85c446]
|
/usr/sbin/mysqld(_Z10sub_selectP4JOINP13st_join_tableb+0x1a7)[0x5606af866997]
|
/usr/sbin/mysqld(+0x5da446)[0x5606af85c446]
|
/usr/sbin/mysqld(_Z10sub_selectP4JOINP13st_join_tableb+0x20f)[0x5606af8669ff]
|
/usr/sbin/mysqld(_ZN4JOIN10exec_innerEv+0xb02)[0x5606af888be2]
|
/usr/sbin/mysqld(_ZN4JOIN4execEv+0x33)[0x5606af888e23]
|
/usr/sbin/mysqld(_Z12mysql_selectP3THDP10TABLE_LISTjR4ListI4ItemEPS4_jP8st_orderS9_S7_S9_yP13select_resultP18st_select_lex_unitP13st_select_lex+0xef)[0x5606af888f6f]
|
/usr/sbin/mysqld(_Z13handle_selectP3THDP3LEXP13select_resultm+0x14d)[0x5606af88986d]
|
/usr/sbin/mysqld(+0x5a61fc)[0x5606af8281fc]
|
/usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x662d)[0x5606af83524d]
|
/usr/sbin/mysqld(_ZN13sp_instr_stmt9exec_coreEP3THDPj+0x15)[0x5606af7a64b5]
|
/usr/sbin/mysqld(_ZN13sp_lex_keeper23reset_lex_and_exec_coreEP3THDPjbP8sp_instr+0x90)[0x5606af7ad030]
|
/usr/sbin/mysqld(_ZN13sp_instr_stmt7executeEP3THDPj+0x647)[0x5606af7adae7]
|
/usr/sbin/mysqld(_ZN7sp_head7executeEP3THDb+0x7fc)[0x5606af7a93cc]
|
/usr/sbin/mysqld(_ZN7sp_head17execute_procedureEP3THDP4ListI4ItemE+0x973)[0x5606af7aa7e3]
|
/usr/sbin/mysqld(+0x5a5fe9)[0x5606af827fe9]
|
/usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x425a)[0x5606af832e7a]
|
/usr/sbin/mysqld(_ZN18Prepared_statement7executeEP6Stringb+0x41e)[0x5606af84b05e]
|
/usr/sbin/mysqld(_ZN18Prepared_statement12execute_loopEP6StringbPhS2_+0x9a)[0x5606af84b24a]
|
/usr/sbin/mysqld(+0x5ca24d)[0x5606af84c24d]
|
/usr/sbin/mysqld(_Z19mysqld_stmt_executeP3THDPcj+0x25)[0x5606af84c2e5]
|
/usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcjbb+0x1a92)[0x5606af839c02]
|
/usr/sbin/mysqld(_Z10do_commandP3THD+0x170)[0x5606af83ac80]
|
/usr/sbin/mysqld(_Z24do_handle_one_connectionP7CONNECT+0x242)[0x5606af90eeb2]
|
/usr/sbin/mysqld(handle_one_connection+0x3d)[0x5606af90f05d]
|
/lib/x86_64-linux-gnu/libpthread.so.0(+0x8164)[0x7fd79731a164]
|
/lib/x86_64-linux-gnu/libc.so.6(clone+0x3f)[0x7fd796f28def]
|
Trying to get some variables.
|
Some pointers may be invalid and cause the dump to abort.
|
Query (0x7fd7247d9dc0): SELECT rental.propref, deal.dealnum AS deal,rental.displayaddress as address,deal.dealid, (SELECT SUM(TERM) FROM invoices WHERE dealid = deal.dealid AND who != 'Tenant' and term > 0 ) AS paidsofar,deal.TERM, (SELECT SUM(TERMdays) FROM invoices WHERE dealid = deal.dealid AND who != 'Tenant' ) AS termdays, deal.startdate AS duedate,rental.building AS building,deal.account AS invoice,deal.startdate, deal.dealnames AS tenant,deal.dealid-deal.dealid as offerid,enddate,actualend,deal.FLWONNOTREQ FROM deal INNER JOIN rental ON rental.propref = deal.propref WHERE deal.cancelled = 0 AND deal.shortlet = 0 AND deal.commrate > 0 and account >= 0
|
Connection ID (thread ID): 99
|
Status: NOT_KILLED
|
Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=off,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on
|
| Component/s | Storage Engine - InnoDB [ 10129 ] | |
| Fix Version/s | 10.3 [ 22126 ] | |
| Assignee | Marko Mäkelä [ marko ] |
problem went away when i exported each table, dropped them and imported back in.
So some sort of data corruption that corrupted the indexes?
Odd that the same 3 tables on 3 different databases (out of 300 or so identically structured databases) got the same error on the same day.
oh well... customers are now happy so i will stop worrying about it unless it reoccurs.
The assertion failed in rec_get_offsets_func(), because rec_get_status() returned an invalid bit combination on a record header. This report is basically duplicating MDEV-13542.
For a while, I was thinking whether MDEV-19916 could explain this, but I do not think so. If you are ever going to execute ALTER TABLE…ADD COLUMN, it would be good to upgrade in any case.
| Link |
This issue relates to |
| Link |
This issue is blocked by |
| Link |
This issue relates to |
| Workflow | MariaDB v3 [ 101287 ] | MariaDB v4 [ 141653 ] |
In MDEV-13542, many crashes were fixed, but mostly at a higher level, preventing access when some page header fields are incorrect. I did not make rec_get_offsets_func() more robust against crashes yet. So, this bug deserves to remain open.
| Link | This issue blocks MDEV-28349 [ MDEV-28349 ] |
It is not easy to fix this. It may be necessary to implement record header validation in specific places that have are (since MDEV-13542) able to return an error. If we added some error return to rec_get_offsets_func() and checks for it in each caller, the code bloat could cause a performance regression.
| Summary | indexes in subquery causing MariaDB crash ??? | Crash in rec_get_offsets_func() due to invalid rec_get_status() |
| Fix Version/s | 10.6 [ 24028 ] | |
| Fix Version/s | 10.7 [ 24805 ] | |
| Fix Version/s | 10.8 [ 26121 ] | |
| Fix Version/s | 10.9 [ 26905 ] | |
| Fix Version/s | 10.3 [ 22126 ] |
| Status | Open [ 1 ] | Confirmed [ 10101 ] |
A possible fix of this could be to introduce some record header validation to the following functions (and make the functions return some special value such as nullptr on corruption, and adjust their callers):
- btr_search_check_guess()
- page_rec_get_next_low(), page_rec_get_next_const(), page_rec_get_next()
- page_dir_slot_get_rec(); also the pointer should be validated to be between the infimum and the PAGE_HEAP_TOP
These measures should guarantee that rec_get_offsets_func() is never invoked on a corrupted record header. For added safety against crashes, the assertion could be changed to one that is only present in debug builds.
| Status | Confirmed [ 10101 ] | In Progress [ 3 ] |
| Status | In Progress [ 3 ] | In Testing [ 10301 ] |
| Assignee | Marko Mäkelä [ marko ] | Matthias Leich [ mleich ] |
| issue.field.resolutiondate | 2022-08-01 08:48:24.0 | 2022-08-01 08:48:24.604 |
| Fix Version/s | 10.6.9 [ 27507 ] | |
| Fix Version/s | 10.7.5 [ 27505 ] | |
| Fix Version/s | 10.8.4 [ 27503 ] | |
| Fix Version/s | 10.9.2 [ 27115 ] | |
| Fix Version/s | 10.10.1 [ 27913 ] | |
| Fix Version/s | 10.6 [ 24028 ] | |
| Fix Version/s | 10.7 [ 24805 ] | |
| Fix Version/s | 10.8 [ 26121 ] | |
| Fix Version/s | 10.9 [ 26905 ] | |
| Assignee | Matthias Leich [ mleich ] | Marko Mäkelä [ marko ] |
| Resolution | Fixed [ 1 ] | |
| Status | In Testing [ 10301 ] | Closed [ 6 ] |
| Link |
This issue is duplicated by |
| Link |
This issue is duplicated by |
| Link |
This issue is duplicated by |
| Zendesk Related Tickets | 110277 |
| Link |
This issue causes |
| Remote Link | This issue links to "SAMU-289 (Jira)" [ 37423 ] |
one server also giving this
2019-11-20 10:36:45 84 [Warning] InnoDB: Table mysql/innodb_table_stats has length mismatch in the column name table_name. Please run mysql_upgrade
2019-11-20 10:36:45 84 [Warning] InnoDB: Table mysql/innodb_index_stats has length mismatch in the column name table_name. Please run mysql_upgrade