[MDEV-28440] MariaDB crashes during select Created: 2022-04-28  Updated: 2022-08-04  Resolved: 2022-08-04

Status: Closed
Project: MariaDB Server
Component/s: Data Manipulation - Update
Affects Version/s: 10.5.15
Fix Version/s: N/A

Type: Bug Priority: Major
Reporter: Willian Vieira Rosa Assignee: Unassigned
Resolution: Incomplete Votes: 0
Labels: crash, innodb
Environment:

CentOS Linux release 7.9.2009 (Core)
Running on esxi 6.7

Attachments: File mariadb.log    

 Description   

MariaDB crashes during queries trying to read, delete or update a row.
For exemple the query:
"select id_versao_secao_documento,id_secao_documento,id_usuario,id_unidade,dth_atualizacao,versao,sin_ultima,conteudo from versao_secao_documento where versao_secao_documento.id_secao_documento IN (5547809) and id_versao_secao_documento = 7160696"

2022-04-28 16:43:41 16 [ERROR] [FATAL] InnoDB: FIL_PAGE_TYPE=0 on BLOB read file ./dbpro_10080_sei/versao_secao_documento.ibd page 13654973
220428 16:43:41 [ERROR] mysqld got signal 6 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.

To report this bug, see https://mariadb.com/kb/en/reporting-bugs

We will try our best to scrape up some info that will hopefully help
diagnose the problem, but since we have already crashed,
something is definitely wrong and this may fail.

Server version: 10.5.15-MariaDB
key_buffer_size=134217728
read_buffer_size=131072
max_used_connections=1
max_threads=153
thread_count=1
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 467872 K bytes of memory
Hope that's ok; if not, decrease some variables in the equation.

Thread pointer: 0x7f1bac0009b8
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 0x7f2040069c90 thread_stack 0x49000
??:0(my_print_stacktrace)[0x55ddf230de3e]
??:0(handle_fatal_signal)[0x55ddf1cff9c7]
sigaction.c:0(__restore_rt)[0x7f204541d630]
:0(__GI_raise)[0x7f2044868387]
:0(__GI_abort)[0x7f2044869a78]
??:0(std::pair<std::_Rb_tree_iterator<unsigned int>, bool> std::_Rb_tree<unsigned int, unsigned int, std::_Identity<unsigned int>, std::less<unsigned int>, std::allocator<unsigned int> >::_M_insert_unique<unsigned int const&>(unsigned int const&))[0x55ddf21648b0]
??:0(std::pair<std::_Rb_tree_iterator<unsigned int>, bool> std::_Rb_tree<unsigned int, unsigned int, std::_Identity<unsigned int>, std::less<unsigned int>, std::allocator<unsigned int> >::_M_insert_unique<unsigned int const&>(unsigned int const&))[0x55ddf2182e45]
??:0(std::pair<std::_Rb_tree_iterator<unsigned int>, bool> std::_Rb_tree<unsigned int, unsigned int, std::_Identity<unsigned int>, std::less<unsigned int>, std::allocator<unsigned int> >::_M_insert_unique<unsigned int const&>(unsigned int const&))[0x55ddf218c332]
/usr/sbin/mariadbd(+0x669398)[0x55ddf19cd398]
/usr/sbin/mariadbd(+0x66965b)[0x55ddf19cd65b]
??:0(void std::__introsort_loop<unsigned char*, long>(unsigned char, unsigned char*, long))[0x55ddf20f85ec]
??:0(wsrep_notify_status(wsrep::server_state::state, wsrep::view const*))[0x55ddf2003d1e]
??:0(handler::index_read_idx_map(unsigned char*, unsigned int, unsigned char const*, unsigned long, ha_rkey_function))[0x55ddf1d00415]
??:0(handler::ha_index_read_idx_map(unsigned char*, unsigned int, unsigned char const*, unsigned long, ha_rkey_function))[0x55ddf1d05653]
??:0(cp_buffer_from_ref(THD*, TABLE*, st_table_ref*))[0x55ddf1b332ec]
/usr/sbin/mariadbd(+0x652668)[0x55ddf19b6668]
??:0(JOIN::optimize_inner())[0x55ddf1b52fc2]
??:0(JOIN::optimize())[0x55ddf1b54685]
??:0(mysql_select(THD*, TABLE_LIST*, List<Item>&, Item*, unsigned int, st_order*, st_order*, Item*, st_order*, unsigned long long, select_result*, st_select_lex_unit*, st_select_lex*))[0x55ddf1b54757]
??:0(handle_select(THD*, LEX*, select_result*, unsigned long))[0x55ddf1b553c4]
/usr/sbin/mariadbd(+0x651836)[0x55ddf19b5836]
??:0(mysql_execute_command(THD*))[0x55ddf1af8e48]
??:0(mysql_parse(THD*, char*, unsigned int, Parser_state*, bool, bool))[0x55ddf1afb2e5]
??:0(dispatch_command(enum_server_command, THD*, char*, unsigned int, bool, bool))[0x55ddf1afdbb5]
??:0(do_command(THD*))[0x55ddf1aff34f]
??:0(do_handle_one_connection(CONNECT*, bool))[0x55ddf1becbc2]
??:0(handle_one_connection)[0x55ddf1bece94]
??:0(MyCTX_nopad::finish(unsigned char*, unsigned int*))[0x55ddf1f551b7]
pthread_create.c:0(start_thread)[0x7f2045415ea5]
??:0(__clone)[0x7f2044930b0d]

Trying to get some variables.
Some pointers may be invalid and cause the dump to abort.
Query (0x7f1bac012860): select id_versao_secao_documento,id_secao_documento,id_usuario,id_unidade,dth_atualizacao,versao,sin_ultima,conteudo from versao_secao_documento where versao_secao_documento.id_secao_documento IN (5547809) and id_versao_secao_documento = 7160696

Connection ID (thread ID): 16
Status: NOT_KILLED

Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=on,condition_pushdown_from_having=on,not_null_range_scan=off

The manual page at https://mariadb.com/kb/en/how-to-produce-a-full-stack-trace-for-mysqld/ contains
information that should help you find out what is causing the crash.
Writing a core file...
Working directory at /opt/dbfiles/mariadb/10/pro_10080_sei/data
Resource Limits:
Limit Soft Limit Hard Limit Units
Max cpu time unlimited unlimited seconds
Max file size unlimited unlimited bytes
Max data size unlimited unlimited bytes
Max stack size 8388608 unlimited bytes
Max core file size 0 unlimited bytes
Max resident set unlimited unlimited bytes
Max processes 95626 95626 processes
Max open files 32768 32768 files
Max locked memory 65536 65536 bytes
Max address space unlimited unlimited bytes
Max file locks unlimited unlimited locks
Max pending signals 95626 95626 signals
Max msgqueue size 819200 819200 bytes
Max nice priority 0 0
Max realtime priority 0 0
Max realtime timeout unlimited unlimited us
Core pattern: core

 Comments   
Comment by Alice Sherepa [ 2022-04-29 ]

Could you please add the output of SHOW CREATE TABLE versao_secao_documento ?
Have you tried to do CHECK TABLE versao_secao_documento? ( from the log - [ERROR] InnoDB: Database page corruption on disk or a failed read of file './dbpro_10080_sei/versao_secao_documento.ibd' page [page id: space=1132, page number=13654965]. You may have to recover from a backup. ) Is the crash reproducible after that?

Comment by Willian Vieira Rosa [ 2022-04-29 ]

This is the result of SHOW CREATE TABLE:

MariaDB [dbpro_10080_sei]> SHOW CREATE TABLE versao_secao_documento;
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

Table Create Table

--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

versao_secao_documento CREATE TABLE `versao_secao_documento` (
`id_versao_secao_documento` bigint(20) NOT NULL,
`id_secao_documento` int(11) NOT NULL,
`id_usuario` int(11) NOT NULL,
`id_unidade` int(11) NOT NULL,
`conteudo` longtext DEFAULT NULL,
`dth_atualizacao` datetime NOT NULL,
`versao` int(11) DEFAULT NULL,
`sin_ultima` char(1) NOT NULL,
PRIMARY KEY (`id_versao_secao_documento`),
KEY `i01_versao_secao_documento` (`id_secao_documento`,`versao`),
KEY `fk_versao_secao_doc_usuario` (`id_usuario`),
KEY `fk_versao_secao_doc_unidade` (`id_unidade`),
KEY `fk_versao_sec_doc_sec_doc` (`id_secao_documento`),
KEY `index_versao_secao_documento_sin_ultima` (`sin_ultima`),
CONSTRAINT `fk_versao_sec_doc_sec_doc` FOREIGN KEY (`id_secao_documento`) REFERENCES `secao_documento` (`id_secao_documento`) ON DELETE NO ACTION ON UPDATE NO ACTION,
CONSTRAINT `fk_versao_secao_doc_unidade` FOREIGN KEY (`id_unidade`) REFERENCES `unidade` (`id_unidade`) ON DELETE NO ACTION ON UPDATE NO ACTION,
CONSTRAINT `fk_versao_secao_doc_usuario` FOREIGN KEY (`id_usuario`) REFERENCES `usuario` (`id_usuario`) ON DELETE NO ACTION ON UPDATE NO ACTION
) ENGINE=InnoDB DEFAULT CHARSET=latin1

--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

Comment by Willian Vieira Rosa [ 2022-04-29 ]

CHECK TABLE don't show errors:

MariaDB [dbpro_10080_sei]> CHECK TABLE versao_secao_documento;
-------------------------------------------------------------+

Table Op Msg_type Msg_text

-------------------------------------------------------------+

dbpro_10080_sei.versao_secao_documento check status OK

-------------------------------------------------------------+
1 row in set (4 min 54.024 sec)

Comment by Willian Vieira Rosa [ 2022-05-26 ]

To show possible solutions to people who might have the same problem, I will show how er recovered the database.
The error os only noted 14 days after the corruption of the database. every time that MariaDB selects a group of registries the database crashes.
I used the command innochecksum on the table to discover which page was corrupted.

"innochecksum -a 3 /opt/dbfiles/mariadb/10/pro_10080_sei/data/dbpro_10080_sei/versao_secao_documento.ibd
Fail: page::13654965 invalid"

The table has 6892719 registries, so I create a Select using the PK trying to identify the group of corrupted registries.

"select id_versao_secao_documento from versao_secao_documento where id_versao_secao_documento > 1 and id_versao_secao_documento < 1000000 ;
select id_versao_secao_documento from versao_secao_documento where id_versao_secao_documento > 1000001 and id_versao_secao_documento < 2000000 ;
"
Until I have only a small group to select and then I have discovered that the corrupted registries are
"
id_versao_secao_documento=7160694
id_versao_secao_documento=7160696
id_versao_secao_documento=7160697
"

Then I created a new table using a select to exclude the corrupted registries.
"create table sei_recover as SELECT * FROM versao_secao_documento WHERE id_versao_secao_documento < 7160693 or id_versao_secao_documento >7160698 ;"

It was really impossible to recovery this 3 registries, but now we have a working dababase.

Comment by Alice Sherepa [ 2022-08-04 ]

There were some bug reports about CHECK TABLE in some cases does not detect table corruption, so it might be also the same problem.
I close the bug as there is no way to repeat this scenario, please comment here if you will know why corruption happened or if there will be a test case to demonstrate the problem.

Generated at Thu Feb 08 10:00:46 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.