[MDEV-6635] MariaDB 10.0.13 unexpected crash Created: 2014-08-25  Updated: 2014-10-29  Due: 2014-10-28  Resolved: 2014-10-29

Status: Closed
Project: MariaDB Server
Component/s: OTHER
Affects Version/s: 10.0.13
Fix Version/s: N/A

Type: Bug Priority: Critical
Reporter: Marek Zakrzewski Assignee: Elena Stepanova
Resolution: Cannot Reproduce Votes: 1
Labels: None
Environment:

CentOS 6.5 64-bit



 Description   

My MariaDB was working fine a few days then it happend:

140825  0:05:42 [ERROR] mysqld got signal 11 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
 
To report this bug, see http://kb.askmonty.org/en/reporting-bugs
 
We will try our best to scrape up some info that will hopefully help
diagnose the problem, but since we have already crashed,
something is definitely wrong and this may fail.
 
Server version: 10.0.13-MariaDB
key_buffer_size=12884901888
read_buffer_size=4194304
max_used_connections=115
max_threads=501
thread_count=38
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 16707820 K  bytes of memory
Hope that's ok; if not, decrease some variables in the equation.
 
Thread pointer: 0x0x7ff042b27008
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 0x7fefcc3b5c58 thread_stack 0x48000
/usr/sbin/mysqld(my_print_stacktrace+0x2b)[0xb7054b]
/usr/sbin/mysqld(handle_fatal_signal+0x398)[0x727588]
/lib64/libpthread.so.0(+0xf710)[0x7ff6bd72d710]
/usr/sbin/mysqld(_ZN8MDL_lock13remove_ticketEMS_NS_11Ticket_listEP10MDL_ticket+0x3d)[0x69fdfd]
/usr/sbin/mysqld(_ZN11MDL_context12acquire_lockEP11MDL_requestm+0x2a7)[0x6a0cc7]
/usr/sbin/mysqld[0x58c4a6]
/usr/sbin/mysqld(_Z10open_tableP3THDP10TABLE_LISTP11st_mem_rootP18Open_table_context+0x4b4)[0x5937c4]
/usr/sbin/mysqld(_Z11open_tablesP3THDPP10TABLE_LISTPjjP19Prelocking_strategy+0x1317)[0x595227]
/usr/sbin/mysqld(_Z20open_and_lock_tablesP3THDP10TABLE_LISTbjP19Prelocking_strategy+0x44)[0x595484]
/usr/sbin/mysqld[0x5ca418]
/usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x4c4f)[0x5d51af]
/usr/sbin/mysqld[0x5d6cc2]
/usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcj+0x1b20)[0x5d8e80]
/usr/sbin/mysqld(_Z26threadpool_process_requestP3THD+0xa7)[0x6cd937]
/usr/sbin/mysqld[0x6fc91d]
/lib64/libpthread.so.0(+0x79d1)[0x7ff6bd7259d1]
/lib64/libc.so.6(clone+0x6d)[0x7ff6bbe42b5d]
 
Trying to get some variables.
Some pointers may be invalid and cause the dump to abort.
Query (0x7ff013397020): is an invalid pointer
Connection ID (thread ID): 6407489
Status: NOT_KILLED
 
Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=off,table_elimination=on,extended_keys=on,exists_to_in=on
 
The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
information that should help you find out what is causing the crash.
140825 00:05:43 mysqld_safe Number of processes running now: 0
140825 00:05:43 mysqld_safe mysqld restarted



 Comments   
Comment by Marek Zakrzewski [ 2014-08-25 ]

My system is Intel(R) Core(TM) i7-3930K CPU @ 3.20GHz, 64 GB DDR3 RAM, 2x 240 GB Intel 520 SSD Software RAID1 (for the system and databases), 2x 3 TB (for the rest) Software RAID1.

Comment by Elena Stepanova [ 2014-08-25 ]

Hi,

Do you have any information on what was being executed in connection 6407489 when the problem occurred?

Comment by Marek Zakrzewski [ 2014-08-25 ]

No - I just found my mail doesn't work which was related to this MariaDB restart.

I've upgraded to 10.0.13 about 2 days ago and added a bit more buffers from 8 GB to 12 GB for key_buffer_size and innodb_buffer_pool_size.

Comment by Elena Stepanova [ 2014-08-25 ]

Is the crash persistent? I mean, does it keep crashing after server restart?

Comment by Marek Zakrzewski [ 2014-08-25 ]

So far working stable. We will see if this happens again.

Comment by Elena Stepanova [ 2014-08-25 ]

You mentioned that your mail stopped working, is this MariaDB instance a part of some email client/server installation? If so, which one is it?

Comment by Marek Zakrzewski [ 2014-08-25 ]

I use QmailToaster + Courier IMAP and Courier IMAP fails when MariaDB is restarted while it's running. It has to be stopped with qmailctl stop then restarted with qmailctl start.

Comment by Takuya Aoki (Inactive) [ 2014-09-10 ]

I also got this bug for over 50 times.
My environment is 10.0.13-MariaDB MariaDB Server on CentOS release 6.5 (X86_64).
I don't know the cause.

140827 16:26:33 [ERROR] mysqld got signal 11 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
 
To report this bug, see http://kb.askmonty.org/en/reporting-bugs
 
We will try our best to scrape up some info that will hopefully help
diagnose the problem, but since we have already crashed, 
something is definitely wrong and this may fail.
 
Server version: 10.0.13-MariaDB
key_buffer_size=16384
read_buffer_size=262144
max_used_connections=1
max_threads=153
thread_count=1
It is possible that mysqld could use up to 
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 52121 K  bytes of memory
Hope that's ok; if not, decrease some variables in the equation.
 
Thread pointer: 0x0x7f9a343f3008
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 0x7f9a3ccc1d10 thread_stack 0x3c000
/usr/sbin/mysqld(my_print_stacktrace+0x2b)[0xb7054b]
/usr/sbin/mysqld(handle_fatal_signal+0x398)[0x727588]
/lib64/libpthread.so.0[0x362140f710]
/usr/lib64/mysql/plugin/ha_connect.so(_ZN10ha_connect8rnd_nextEPh+0x12f)[0x7f9a34d3725f]
/usr/sbin/mysqld(_ZN7handler11ha_rnd_nextEPh+0x16c)[0x72bb5c]
/usr/sbin/mysqld(_Z13rr_sequentialP11READ_RECORD+0x1a)[0x81392a]
/usr/sbin/mysqld(_Z10sub_selectP4JOINP13st_join_tableb+0x161)[0x5f66d1]
/usr/sbin/mysqld[0x60d80d]
/usr/sbin/mysqld(_ZN4JOIN10exec_innerEv+0xa3d)[0x62067d]
/usr/sbin/mysqld(_ZN4JOIN4execEv+0x11)[0x6224f1]
/usr/sbin/mysqld(_Z12mysql_selectP3THDPPP4ItemP10TABLE_LISTjR4ListIS1_ES2_jP8st_orderSB_S2_SB_yP13select_resultP18st_select_lex_unitP13st_select_lex+0x1dd)[0x61f0bd]
/usr/sbin/mysqld(_Z13handle_selectP3THDP3LEXP13select_resultm+0x28d)[0x62284d]
/usr/sbin/mysqld[0x5ca456]
/usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x4c4f)[0x5d51af]
/usr/sbin/mysqld[0x5d6cc2]
/usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcj+0x1b20)[0x5d8e80]
/usr/sbin/mysqld(_Z24do_handle_one_connectionP3THD+0x453)[0x696993]
/usr/sbin/mysqld(handle_one_connection+0x42)[0x696a62]
/lib64/libpthread.so.0[0x36214079d1]
/lib64/libc.so.6(clone+0x6d)[0x36210e8b5d]
 
Trying to get some variables.
Some pointers may be invalid and cause the dump to abort.
Query (0x7f9a22871020): is an invalid pointer
Connection ID (thread ID): 2
Status: NOT_KILLED
 
Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=off,table_elimination=on,extended_keys=on,exists_to_in=on
 
The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
information that should help you find out what is causing the crash.

Comment by Elena Stepanova [ 2014-09-10 ]

Hi,

What makes you think it's the same bug? It looks totally different to me.
The crash happens when you are using the CONNECT engine. Do you use it extensively? Will you be able to enable general log until the next time it crashes?
In any case, please file a new bug report for this.

Comment by Takuya Aoki (Inactive) [ 2014-09-10 ]

Thank you,
I reported it as another bug.

https://mariadb.atlassian.net/browse/MDEV-6722

Comment by Elena Stepanova [ 2014-09-29 ]

Hi Spacedust,

Have you ever encountered the crash again since you submitted the report?

Though even a one-time crash signifies a real bug, there isn't much to go with if we can't narrow it down somehow.

Comment by Marek Zakrzewski [ 2014-09-29 ]

It's working fine so far

Comment by Elena Stepanova [ 2014-09-29 ]

Thanks for the update.
I'll have to close it as 'cannot reproduce' for now, but by all means feel free to re-open if you encounter it again. Maybe we'll see some pattern then.

Comment by Marek Zakrzewski [ 2014-09-29 ]

Another crash when moving whole MariaDB to new machine:

140929 20:23:40 mysqld_safe mysqld from pid file /var/lib/mysql/online.pl.pid ended
140929 20:23:59 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql
140929 20:23:59 [Note] InnoDB: Using mutexes to ref count buffer pool pages
140929 20:23:59 [Note] InnoDB: The InnoDB memory heap is disabled
140929 20:23:59 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
140929 20:23:59 [Note] InnoDB: Compressed tables use zlib 1.2.3
140929 20:23:59 [Note] InnoDB: Using Linux native AIO
140929 20:23:59 [Note] InnoDB: Using CPU crc32 instructions
140929 20:23:59 [Note] InnoDB: Initializing buffer pool, size = 2.0G
140929 20:24:00 [Note] InnoDB: Completed initialization of buffer pool
140929 20:24:00 [Note] InnoDB: Highest supported file format is Barracuda.
140929 20:24:00 [Note] InnoDB: Log scan progressed past the checkpoint lsn 257493140753
140929 20:24:00 [Note] InnoDB: Database was not shutdown normally!
140929 20:24:00 [Note] InnoDB: Starting crash recovery.
140929 20:24:00 [Note] InnoDB: Reading tablespace information from the .ibd files...
140929 20:24:04 [Note] InnoDB: Restoring possible half-written data pages
140929 20:24:04 [Note] InnoDB: from the doublewrite buffer...
InnoDB: Doing recovery: scanned up to log sequence number 257493216556
140929 20:24:04 [Note] InnoDB: Starting an apply batch of log records to the database...
InnoDB: Progress in percent: 1 2 140929 20:24:04 [ERROR] mysqld got signal 11 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
 
To report this bug, see http://kb.askmonty.org/en/reporting-bugs
 
We will try our best to scrape up some info that will hopefully help
diagnose the problem, but since we have already crashed,
something is definitely wrong and this may fail.
 
Server version: 10.0.13-MariaDB
key_buffer_size=2147483640
read_buffer_size=2097152
max_used_connections=0
max_threads=501
thread_count=0
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 4169964 K  bytes of memory
Hope that's ok; if not, decrease some variables in the equation.
 
Thread pointer: 0x0x0
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
3 4 stack_bottom = 0x0 thread_stack 0x48000
5 6 7 8 9 10 /usr/sbin/mysqld(my_print_stacktrace+0x2b)[0xb7054b]
11 /usr/sbin/mysqld(handle_fatal_signal+0x398)[0x727588]
/lib64/libpthread.so.0(+0xf710)[0x7f7b06ace710]
/usr/sbin/mysqld[0x8d8799]
12 /usr/sbin/mysqld[0x8d7802]
/usr/sbin/mysqld[0x8d8035]
13 /usr/sbin/mysqld[0x8be5b1]
/usr/sbin/mysqld[0x8c0294]
14 /usr/sbin/mysqld[0x99cbc6]
/usr/sbin/mysqld[0x9e4ae6]
15 /usr/sbin/mysqld[0x93a455]
/lib64/libpthread.so.0(+0x79d1)[0x7f7b06ac69d1]
/lib64/libc.so.6(clone+0x6d)[0x7f7b051e386d]
The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
information that should help you find out what is causing the crash.
140929 20:24:04 mysqld_safe mysqld from pid file /var/lib/mysql/onlinecity.pl.pid ended

Comment by Elena Stepanova [ 2014-10-14 ]

The crash looks very different though.
It happened on startup, while InnoDB was applying the log after an apparently dirty shutdown. I'm afraid it's going to be impossible to find out what happened there, since the stack trace is useless.
I assume you tried to restart the server after the crash, did it start all right, or did you have to do something to get it up and running?
Do you know what caused the previous dirty shutdown?

Comment by Elena Stepanova [ 2014-10-29 ]

Closing as "Can't reproduce" for now, because, sadly, there's nothing even to start from. If you get any more information, please comment to re-open.

Generated at Thu Feb 08 07:13:23 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.