[MDEV-6938] MariaDB 10.0.14 crash (signal 11) Created: 2014-10-24  Updated: 2018-02-01  Due: 2014-11-14  Resolved: 2014-11-15

Status: Closed
Project: MariaDB Server
Component/s: OTHER
Affects Version/s: 10.0.14
Fix Version/s: N/A

Type: Bug Priority: Critical
Reporter: Ján Regeš Assignee: Elena Stepanova
Resolution: Incomplete Votes: 0
Labels: crash, mariadb,
Environment:

Gentoo Linux, MariaDB installed by package system (emerge).
8x Intel E5-2630, 8GB RAM and /var/lib/mysql on SSD harddrive


 Description   

Hi,

our server MariaDB 10.0.14 crashed today, 1 months after upgrade from MySQL 5.1.61, which had 2 years uptime.

This server contains only 2 databases with low load and total size about 1GB. One database with Magento e-shop database and one with our database (100x InnoDB tables, 1x Memory, 1x CONNECT and 1x TokuDB).

I think, that it is not related to CONNECT, or TokuDB engine.

MariaDB crashed during standard mysqldump-based backup proces.

Below is also backtrace and MySQL process-list 1-3 seconds before crash.

There was no load, just backup simple InnoDB tables of Magento database.

Thank you for your help.

141024  2:21:13 [ERROR] mysqld got signal 11 ;
 
This could be because you hit a bug. It is also possible that this binary
 
or one of the libraries it was linked against is corrupt, improperly built,
 
or misconfigured. This error can also be caused by malfunctioning hardware.
 
 
 
To report this bug, see http://kb.askmonty.org/en/reporting-bugs
 
 
 
We will try our best to scrape up some info that will hopefully help
 
diagnose the problem, but since we have already crashed,
 
something is definitely wrong and this may fail.
 
 
 
Server version: 10.0.14-MariaDB
 
key_buffer_size=134217728
 
read_buffer_size=4194304
 
max_used_connections=69
 
max_threads=202
 
thread_count=4
 
It is possible that mysqld could use up to
 
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 4272191 K  bytes of memory
 
Hope that's ok; if not, decrease some variables in the equation.
 
 
 
Thread pointer: 0x0x7fa9570f3008
 
Attempting backtrace. You can use the following information to find out
 
where mysqld died. If you see no messages after this, something went
 
terribly wrong...
 
stack_bottom = 0x7fa954579e28 thread_stack 0x80000
 
/usr/sbin/mysqld(my_print_stacktrace+0x26)[0xb16713]
 
/usr/sbin/mysqld(handle_fatal_signal+0x398)[0x6f3658]
 
/lib64/libpthread.so.0(+0x10150)[0x7fa9fa18b150]
 
/usr/sbin/mysqld[0x88facb]
 
/usr/sbin/mysqld[0x89079f]
 
/usr/sbin/mysqld[0x890a1b]
 
/usr/sbin/mysqld(_ZN7handler22ha_rnd_init_with_errorEb+0x15)[0x6f84a5]
 
/usr/sbin/mysqld(_Z16init_read_recordP11READ_RECORDP3THDP5TABLEP10SQL_SELECTibb+0x4c2)[0x7c9f52]
 
/usr/sbin/mysqld(_Z21join_init_read_recordP13st_join_table+0x70)[0x5f1d10]
 
/usr/sbin/mysqld(_Z10sub_selectP4JOINP13st_join_tableb+0x161)[0x5f1ed1]
 
/usr/sbin/mysqld[0x5fd82d]
 
/usr/sbin/mysqld(_ZN4JOIN10exec_innerEv+0xa08)[0x60e508]
 
/usr/sbin/mysqld(_ZN4JOIN4execEv+0x9)[0x610019]
 
/usr/sbin/mysqld(_Z12mysql_selectP3THDPPP4ItemP10TABLE_LISTjR4ListIS1_ES2_jP8st_orderSB_S2_SB_yP13select_resultP18st_select_lex_unitP13st_select_lex+0x148)[0x60d068]
 
/usr/sbin/mysqld(_Z13handle_selectP3THDP3LEXP13select_resultm+0x294)[0x610314]
 
/usr/sbin/mysqld[0x5bff2b]
 
/usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x50fe)[0x5cac5e]
 
/usr/sbin/mysqld(_Z11mysql_parseP3THDPcjP12Parser_state+0x1fc)[0x5cd46c]
 
/usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcj+0x1530)[0x5ce9f0]
 
/usr/sbin/mysqld(_Z24do_handle_one_connectionP3THD+0x19d)[0x674abd]
 
/usr/sbin/mysqld(handle_one_connection+0x40)[0x674b80]
 
/lib64/libpthread.so.0(+0x8bb5)[0x7fa9fa183bb5]
 
/lib64/libc.so.6(clone+0x6d)[0x7fa9f8e76d7d]
 
 
 
Trying to get some variables.
 
Some pointers may be invalid and cause the dump to abort.
 
Query (0x7fa9634ba020): is an invalid pointer
 
Connection ID (thread ID): 328170
 
Status: NOT_KILLED
 
 
 
Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_me
 
rge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_so
 
rt_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=off,table_elimination=on,extended_keys=on,exists_to_in=on
 
 
 
The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
 
information that should help you find out what is causing the crash.

MySQL process list 1-3 seconds before crash.

+--------+-----------------+-----------+-----------------+---------+------+-----------------------------+--------------------------------------------------------------------+----------+
 
| Id     | User            | Host      | db              | Command | Time | State                       | Info                                                               | Progress |
 
+--------+-----------------+-----------+-----------------+---------+------+-----------------------------+--------------------------------------------------------------------+----------+
 
| 1      | event_scheduler | localhost |                 | Daemon  | 14   | Waiting for next activation |                                                                    | 0.000    |
 
| 328170 | root            | localhost | client_cz_new   | Query   | 1    | Sending data                | SELECT /*!40001 SQL_NO_CACHE */ * FROM `ig_cashondelivery_foreign` | 0.000    |
 
| 328175 | root            | localhost |                 | Query   | 0    | init                        | show processlist                                                   | 0.000    |
 
| 328176 | root            | localhost |                 | Sleep   | 1    |                             |                                                                    | 0.000    |
 
+--------+-----------------+-----------+-----------------+---------+------+-----------------------------+--------------------------------------------------------------------+----------+



 Comments   
Comment by Elena Stepanova [ 2014-10-24 ]

Hi,

How often do you run the same backup procedure? Did it happen before, or was it the first attempt after the upgrade to 10.0.14?

Did you re-run the backup after the crash? If you did, did it succeed?

How confidential is the data? Would you be able to upload it to our ftp.askmonty.org/private? (only MariaDB developers will have access to it)?

If you can't provide all data, maybe at least client_cz_new.ig_cashondelivery_foreign which appears in the processlist – it is no guarantee that it was the last, but there is a chance.

In any case, please also attach your cnf file(s) and the exact mysqldump command your process uses (or the backup script which performs it).

Thank you.

Comment by Ján Regeš [ 2014-10-29 ]

Hi Elena,

backup procedure is executed every 6 hours. It happen just once.

After re-run everything works fine.

I uploaded "my.cnf" and our backup script "make-daily-mysql-backup" into ftp.askmonty.org/private (by jan.reges@siteone.cz).

Btw, table "ig_cashondelivery_foreign" is empty, so it's not used. We store whole process-list every 2-3 seconds and it was last before crash.

Also, maybe it can depends. We have some UDF:

SELECT * FROM mysql.func;
 
 
 
lib_mysqludf_preg_info	0	lib_mysqludf_preg.so	function
 
preg_capture	0	lib_mysqludf_preg.so	function
 
preg_check	2	lib_mysqludf_preg.so	function
 
preg_replace	0	lib_mysqludf_preg.so	function
 
preg_rlike	2	lib_mysqludf_preg.so	function
 
preg_position	2	lib_mysqludf_preg.so	function
 
microtime	1	mysql_udf_microtime.so	function
 
microtime	1	mysql_udf_microtime.so

Comment by Elena Stepanova [ 2014-10-31 ]

Hi Ján,

Thanks for the info.

There are several things here I have difficulties explaining.

First, if the table ig_cashondelivery_foreign is empty, it should not take 1 second or more to select from it, and it should be quite hard to catch it at 'Sending data' phase.
Since you have lots of processlist logs (normal ones, not crash-related) – can you grep them to see if select from this table appears often?

Secondly, I don't see anything in the backup process that should cause the stack trace like that. You had 4 connections, is it possible that one of the two others woke up and ran something? E.g. event scheduler started working? (The time is odd, which makes it rather unlikely but not impossible). Or could it be the other connection, the one that's sleeping in the processlist?

Finally, about your select from mysql.func – it looks strange too. The first column in the table, 'name', is a primary key, so you should not have two functions called 'microtime'. Do you really, or is it a copy-paste error? If you seem to have two, could you please run check table mysql.func extended?

I presume you could not provide the data dump, otherwise you would have, right?
Then, could you please see in one of the dumps which table is queried next after ig_cashondelivery_foreign, and send both tables' structures (SHOW CREATE TABLE .. output and SHOW INDEX IN .. output)? Or, if it's not a table but some other object, it's even more interesting.

Thanks.

Comment by Elena Stepanova [ 2014-11-15 ]

Closing for now as incomplete, please comment to re-open if you have more information.

Generated at Thu Feb 08 07:15:44 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.