[MDEV-26478] Mariadb crashes while accessing a database Created: 2021-08-25  Updated: 2021-12-23

Status: Open
Project: MariaDB Server
Component/s: None
Affects Version/s: 10.4.20
Fix Version/s: 10.4, 10.5, 10.6

Type: Bug Priority: Major
Reporter: Elisabeth Kahnert Assignee: Alexander Barkov
Resolution: Unresolved Votes: 0
Labels: None
Environment:

SLES15 SP2, 12GB RAM, 2 Cores

Attachments: XML File Index.xml    

 Description   

Every operation on a database like "show tables" or "drop database DBNAME" provokes errors ERROR 2013 (HY000): Lost connection to MySQL server during query
or ERROR 2006 (HY000): MySQL server has gone away
No connection. Trying to reconnect...
Current database: *** NONE ***

in the error-log I see:

Version: '10.4.20-MariaDB-log'  socket: '/var/lib/mysql/DB2/mysql.sock'  port: 3316  MariaDB package
 
2021-08-25 11:48:06 0x7f2750151700  InnoDB: Assertion failure in file /home/abuild/rpmbuild/BUILD/mariadb-10.4.20/storage/innobase/handler/ha_innodb.cc line 2132
 
InnoDB: Failing assertion: cset == 0
 
InnoDB: We intentionally generate a memory trap.
 
InnoDB: Submit a detailed bug report to https://jira.mariadb.org/
 
InnoDB: If you get repeated assertion failures or crashes, even
 
InnoDB: immediately after the mysqld startup, there may be
 
InnoDB: corruption in the InnoDB tablespace. Please refer to
 
InnoDB: https://mariadb.com/kb/en/library/innodb-recovery-modes/
 
InnoDB: about forcing recovery.
 
210825 11:48:06 [ERROR] mysqld got signal 6 ;
 
This could be because you hit a bug. It is also possible that this binary
 
or one of the libraries it was linked against is corrupt, improperly built,
 
or misconfigured. This error can also be caused by malfunctioning hardware.
 
 
 
To report this bug, see https://mariadb.com/kb/en/reporting-bugs
 
 
 
We will try our best to scrape up some info that will hopefully help
 
diagnose the problem, but since we have already crashed,
 
something is definitely wrong and this may fail.
 
 
 
Server version: 10.4.20-MariaDB-log
 
key_buffer_size=134217728
 
read_buffer_size=262144
 
max_used_connections=2
 
max_threads=1002
 
thread_count=8
 
It is possible that mysqld could use up to
 
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 2464509 K  bytes of memory
 
Hope that's ok; if not, decrease some variables in the equation.
 
 
 
Thread pointer: 0x7f26b8008118
 
Attempting backtrace. You can use the following information to find out
 
where mysqld died. If you see no messages after this, something went
 
terribly wrong...
 
stack_bottom = 0x7f2750150cf8 thread_stack 0x49000
 
/usr/sbin/mysqld(my_print_stacktrace+0x3c)[0x5609b0aa531c]
 
/usr/sbin/mysqld(handle_fatal_signal+0x535)[0x5609b05b9145]
 
/lib64/libpthread.so.0(+0x132d0)[0x7f277396d2d0]
 
/lib64/libc.so.6(gsignal+0x110)[0x7f2772aa8420]
 
/lib64/libc.so.6(abort+0x151)[0x7f2772aa9a01]
 
/usr/sbin/mysqld(+0x5b8b62)[0x5609b02ddb62]
 
/usr/sbin/mysqld(+0xa214c8)[0x5609b07464c8]
 
/usr/sbin/mysqld(+0xbdd30c)[0x5609b090230c]
 
/usr/sbin/mysqld(+0xbd03d9)[0x5609b08f53d9]
 
/usr/sbin/mysqld(+0xbdb1b9)[0x5609b09001b9]
 
/usr/sbin/mysqld(+0xbd2a30)[0x5609b08f7a30]
 
/usr/sbin/mysqld(+0xbc8593)[0x5609b08ed593]
 
/usr/sbin/mysqld(+0xa23c88)[0x5609b0748c88]
 
/usr/sbin/mysqld(+0xa2f1f9)[0x5609b07541f9]
 
/usr/sbin/mysqld(_ZN7handler7ha_openEP5TABLEPKcijP11st_mem_rootP4ListI6StringE+0x43)[0x5609b05bd803]
 
/usr/sbin/mysqld(_Z21open_table_from_shareP3THDP11TABLE_SHAREPK25st_mysql_const_lex_stringjjjP5TABLEbP4ListI6StringE+0x940)[0x5609b046d090]
 
/usr/sbin/mysqld(_Z10open_tableP3THDP10TABLE_LISTP18Open_table_context+0xb59)[0x5609b0358c49]
 
/usr/sbin/mysqld(_Z11open_tablesP3THDRK14DDL_options_stPP10TABLE_LISTPjjP19Prelocking_strategy+0x929)[0x5609b035b9f9]
 
/usr/sbin/mysqld(_Z30open_normal_and_derived_tablesP3THDP10TABLE_LISTjj+0x6a)[0x5609b035c66a]
 
/usr/sbin/mysqld(_Z18mysqld_list_fieldsP3THDP10TABLE_LISTPKc+0x36)[0x5609b04173e6]
 
/usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcjbb+0x250a)[0x5609b03c0faa]
 
/usr/sbin/mysqld(_Z10do_commandP3THD+0x12f)[0x5609b03c140f]
 
/usr/sbin/mysqld(_Z24do_handle_one_connectionP7CONNECT+0x24e)[0x5609b049656e]
 
/usr/sbin/mysqld(handle_one_connection+0x34)[0x5609b0496644]
 
/lib64/libpthread.so.0(+0x84f9)[0x7f27739624f9]
 
/lib64/libc.so.6(clone+0x3f)[0x7f2772b6aecf]
 
 
 
Trying to get some variables.
 
Some pointers may be invalid and cause the dump to abort.
 
Query (0x7f26b8017dd0):
 
 
 
Connection ID (thread ID): 9
 
Status: NOT_KILLED
 
 
 
Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=on,condition_pushdown_from_having=on
 
 
 
The manual page at https://mariadb.com/kb/en/how-to-produce-a-full-stack-trace-for-mysqld/ contains
 
information that should help you find out what is causing the crash.
 
Writing a core file...
 
Working directory at /var/lib/mysql/DB2
 
Resource Limits:
 
Limit                     Soft Limit           Hard Limit           Units
 
Max cpu time              unlimited            unlimited            seconds
 
Max file size             unlimited            unlimited            bytes
 
Max data size             unlimited            unlimited            bytes
 
Max stack size            8388608              unlimited            bytes
 
Max core file size        unlimited            unlimited            bytes
 
Max resident set          unlimited            unlimited            bytes
 
Max processes             47756                47756                processes
 
Max open files            129033               129033               files
 
Max locked memory         65536                65536                bytes
 
Max address space         unlimited            unlimited            bytes
 
Max file locks            unlimited            unlimited            locks
 
Max pending signals       47756                47756                signals
 
Max msgqueue size         819200               819200               bytes
 
Max nice priority         0                    0
 
Max realtime priority     0                    0
 
Max realtime timeout      unlimited            unlimited            us
 
Core pattern: |/usr/lib/systemd/systemd-coredump %P %u %g %s %t %c %e
 
 
 
210825 11:48:06 mysqld_safe Number of processes running now: 0
 
210825 11:48:06 mysqld_safe mysqld restarted



 Comments   
Comment by Sergei Golubchik [ 2021-09-21 ]

what tables do you have in this database? what files? what storage engines?

Comment by Elisabeth Kahnert [ 2021-09-22 ]

The Engine is innoDB.
The errors were on the second instance ob mariadb-server. We found the cause of these error: Tha arabic collation for the database was gone. Until one of the last updates it was in /the file usr/share/mysql/charsets/Index.xml. After the update this file wasn't used any more, we had to place the collation in /usr/share/mariadb/charsets/Index.xml

Comment by Sergei Golubchik [ 2021-12-21 ]

bar, does it ring any bells?

Comment by Alexander Barkov [ 2021-12-22 ]

serg, yes, I think the current implementation is dangerous indeed:

/usr/share/mysql/charsets/Index.xml is replaced on upgrade. If it contains user-defined collations, they all disappear on upgrade until you edit it manually again.

I think we need to add a way (instead of editing Index.xml) - make it possible to put separate XML files with user-defined collation definitions into /usr/share/mysql/charsets/, and have MariaDB load them all on startup.

nisso, can you please attach your Index.xml file with the Arabic collation? Thanks.

Comment by Elisabeth Kahnert [ 2021-12-22 ]

Hello Alexander,

it's attached.

Best regards,

Elisabeth Z. Kahnert
Universität Hamburg
Regional Computer Centre
Linux server and web services
Schlueterstr. 70
20146 Hamburg
Germany
Phone: +4940/42838 4096
Mail: elisabeth.kahnert@uni-hamburg.de

Comment by Sergei Golubchik [ 2021-12-22 ]

nisso, do you mean that once the stock Index.xml had the Arabic collation and then it disappeared? Or was it something you've added and then it disappeared on upgrade?

Comment by Elisabeth Kahnert [ 2021-12-22 ]

No, the file Index.xml used by mariadb was once in the directory
/usr/share/mysql/charsets/Index.xml, after some update this file was
ignored und the file /usr/share/mariadb/charsets/Index.xml was used
instead. This file had no Arabic collation. Then we copied our file with
this collation from /usr/share/mysql/charsets/Index.xml to
/usr/share/mariadb/charsets/Index.xml and the problem was solved.

Generated at Thu Feb 08 09:45:37 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.