|
Hi,
The uploaded error log starts from 'signal 11'. Were there any errors/warnings before that? Did server dump InnoDB status in the error log?
Did you issue SHOW ENGINE INNODB STATUS on your own, or was it done by the server?
Do you have any kind of monitoring on the server (SHOW STATUS, SHOW PROCESSLIST and alike)?
Is the crash reproducible?
Thanks.
|
|
Hi,
Yes, mariadb creased with signal 11, but before crashing doesn't were any errors. The server doesn't dumped Innodb status into error log (or other files).
I don't know exactly who called, but I found lot of SHOW /*!50000 ENGINE*/ INNODB STATUS command in munin scripts (we use munin for monitoring ).
We installed mariadb from deb files, do you think if we use binary files (mariadb-5.5.38-linux-x86_64.tar.gz) we get more information from about crash ?
Sorry, but we can't reproduc the crash.
Thanks.
|
|
Hi,
Our server again crashed.
140619 13:10:02 [ERROR] mysqld got signal 11 ;
|
This could be because you hit a bug. It is also possible that this binary
|
or one of the libraries it was linked against is corrupt, improperly built,
|
or misconfigured. This error can also be caused by malfunctioning hardware.
|
|
To report this bug, see http://kb.askmonty.org/en/reporting-bugs
|
|
We will try our best to scrape up some info that will hopefully help
|
diagnose the problem, but since we have already crashed,
|
something is definitely wrong and this may fail.
|
|
Server version: 5.5.38-MariaDB-1~precise-log
|
key_buffer_size=3221225470
|
read_buffer_size=8388608
|
max_used_connections=68
|
max_threads=122
|
thread_count=67
|
It is possible that mysqld could use up to
|
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 7270511 K bytes of memory
|
Hope that's ok; if not, decrease some variables in the equation.
|
|
Thread pointer: 0x0x7f49d5fd0000
|
Attempting backtrace. You can use the following information to find out
|
where mysqld died. If you see no messages after this, something went
|
terribly wrong...
|
stack_bottom = 0x7f4a994dae20 thread_stack 0xc8000
|
/usr/sbin/mysqld(my_print_stacktrace+0x2b)[0xa7499b]
|
/usr/sbin/mysqld(handle_fatal_signal+0x398)[0x6ccf88]
|
/lib/x86_64-linux-gnu/libpthread.so.0(+0xfcb0)[0x7f5288c8ecb0]
|
/lib/x86_64-linux-gnu/libc.so.6(_IO_vfprintf+0x29f1)[0x7f52874793b1]
|
/lib/x86_64-linux-gnu/libc.so.6(__fprintf_chk+0xeb)[0x7f528753780b]
|
/usr/sbin/mysqld[0x846dcc]
|
/usr/sbin/mysqld[0x847fda]
|
/usr/sbin/mysqld[0x849778]
|
/usr/sbin/mysqld[0x83ec6f]
|
/usr/sbin/mysqld[0x8095bc]
|
/usr/sbin/mysqld(_Z14ha_show_statusP3THDP10handlerton12ha_stat_type+0x217)[0x6d4017]
|
/usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x3f13)[0x59fcc3]
|
/usr/sbin/mysqld[0x5a1934]
|
/usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcj+0x159f)[0x5a301f]
|
/usr/sbin/mysqld(_Z24do_handle_one_connectionP3THD+0x1fb)[0x65323b]
|
/usr/sbin/mysqld(handle_one_connection+0x4c)[0x6532bc]
|
/lib/x86_64-linux-gnu/libpthread.so.0(+0x7e9a)[0x7f5288c86e9a]
|
/lib/x86_64-linux-gnu/libc.so.6(clone+0x6d)[0x7f52875223fd]
|
Trying to get some variables.
|
Some pointers may be invalid and cause the dump to abort.
|
Query (0x7f4a3b831018): SHOW /*!50000 ENGINE*/ INNODB STATUS
|
Connection ID (thread ID): 8982
|
Status: NOT_KILLED
|
|
Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,inde
|
|
The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
|
information that should help you find out what is causing the crash.
|
Writing a core file
|
|
|
Could you please get the stack trace from the core you uploaded on the machine where it was generated (or from the new coredump)?
I tried to, but due to environment difference I'm not getting a good one.
gdb --batch --eval-command="thread apply all bt" <mysqld> <coredump> will do.
|
|
Hi,
On the weekend a I will try gdb.
We chage the mariadb verision to mariadb-5.5.38-linux-x86_64 (Generic Linux)
And got signal 11  There is more information abot crash. I upload the core dum file?
140620 9:00:05 [ERROR] mysqld got signal 11 ;
|
This could be because you hit a bug. It is also possible that this binary
|
or one of the libraries it was linked against is corrupt, improperly built,
|
or misconfigured. This error can also be caused by malfunctioning hardware.
|
|
To report this bug, see http://kb.askmonty.org/en/reporting-bugs
|
|
We will try our best to scrape up some info that will hopefully help
|
diagnose the problem, but since we have already crashed,
|
something is definitely wrong and this may fail.
|
|
Server version: 5.5.38-MariaDB-log
|
key_buffer_size=3221225470
|
read_buffer_size=8388608
|
max_used_connections=50
|
max_threads=122
|
thread_count=47
|
It is possible that mysqld could use up to
|
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 7270511 K bytes of memory
|
Hope that's ok; if not, decrease some variables in the equation.
|
|
Thread pointer: 0x0x7fa5e2398000
|
Attempting backtrace. You can use the following information to find out
|
where mysqld died. If you see no messages after this, something went
|
terribly wrong...
|
stack_bottom = 0x7fa6bb560e20 thread_stack 0xc8000
|
mysys/stacktrace.c:247(my_print_stacktrace)[0xaf02be]
|
sql/signal_handler.cc:153(handle_fatal_signal)[0x6da97c]
|
/lib/x86_64-linux-gnu/libpthread.so.0(+0xfcb0)[0x7fae818dacb0]
|
/lib/x86_64-linux-gnu/libc.so.6(_IO_vfprintf+0x29f1)[0x7fae809153b1]
|
/lib/x86_64-linux-gnu/libc.so.6(_IO_fprintf+0x87)[0x7fae8091d837]
|
sync/sync0arr.c:500(sync_array_cell_print)[0x9239f5]
|
sync/sync0arr.c:1058(sync_array_output_info)[0x923dda]
|
sync/sync0sync.c:1602(sync_print_wait_info)[0x925688]
|
include/sync0sync.ic:243(pfs_mutex_enter_func)[0x91dfec]
|
handler/ha_innodb.cc:10864(innodb_show_status)[0x8e6810]
|
handler/ha_innodb.cc:11104(innobase_show_status)[0x8e6b75]
|
sql/handler.cc:4908(ha_show_status(THD*, handlerton*, ha_stat_type))[0x6e0411]
|
sql/sql_parse.cc:2408(mysql_execute_command(THD*))[0x58389e]
|
sql/sql_parse.cc:5799(mysql_parse(THD*, char*, unsigned int, Parser_state*))[0x5866e1]
|
sql/sql_parse.cc:1081(dispatch_command(enum_server_command, THD*, char*, unsigned int))[0x587e0e]
|
sql/sql_parse.cc:793(do_command(THD*))[0x5883b2]
|
sql/sql_connect.cc:1266(do_handle_one_connection(THD*))[0x6427f3]
|
sql/sql_connect.cc:1183(handle_one_connection)[0x64293c]
|
/lib/x86_64-linux-gnu/libpthread.so.0(+0x7e9a)[0x7fae818d2e9a]
|
/lib/x86_64-linux-gnu/libc.so.6(clone+0x6d)[0x7fae809be3fd]
|
|
|
Trying to get some variables.
|
Some pointers may be invalid and cause the dump to abort.
|
Query (0x7fa6b510b018): SHOW /*!50000 ENGINE*/ INNODB STATUS
|
Connection ID (thread ID): 2567
|
Status: NOT_KILLED
|
|
Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index
|
|
The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
|
information that should help you find out what is causing the crash.
|
Writing a core file
|
|
|
Hi,
On the weekend a I will try gdb.
Thanks.
While you're on it, please run both
gdb --batch --eval-command="thread apply all bt" <mysqld> <coredump>
and
gdb --batch --eval-command="thread apply all bt full" <mysqld> <coredump>
just in case.
There is more information abot crash. I upload the core dum file?
What kind of information?
Regarding the coredump, if you can get a decent stack trace from the it, please do so instead of uploading the whole dump. Please also store the dump somewhere if possible, we might ask you to run something else on it later. The problem with coredumps is that often you need the exact same machine where it was created to reveal something useful.
Please also paste or attach at least one full error log, starting from server (re-)start and till "Writing a core file" (rather than starting from signal 11). Even if it doesn't contain any more errors, it might still be useful to see how it starts, how soon after startup the crash occurs, etc.
Do you have any kind of status monitoring (SHOW GLOBAL STATUS)?
Do you happen to have general log enabled, and if not, would you be able to enable it till the next crash and upload it later?
|
|
cd1
output
gdb --batch --eval-command="thread apply all bt" /usr/sbin/mysqld core.mysqld.11108.debrecen.1403176204
/usr/sbin/mysqld Ver 5.5.38-MariaDB-1~precise-log for debian-linux-gnu on x86_64 (mariadb.org binary distribution) from deb
|
|
cd2
output :
gdb --batch --eval-command="thread apply all bt full"
/usr/sbin/mysqld Ver 5.5.38-MariaDB-1~precise-log for debian-linux-gnu on x86_64 (mariadb.org binary distribution)
|
|
cd5
output
gdb --batch --eval-command="thread apply all bt" /usr/local/mariadb-5.5.38-linux-x86_64/bin/mysqld core.mysqld.23201.debrecen.1403247607
Generic mariadb binary
/usr/local/mariadb-5.5.38-linux-x86_64/bin/mysqld Ver 5.5.38-MariaDB-log for Linux on x86_64 (MariaDB Server)
|
|
cd6
output:
gdb --batch --eval-command="thread apply all bt full" /usr/local/mariadb-5.5.38-linux-x86_64/bin/mysqld core.mysqld.23201.debrecen.1403247607
Generic mariadb binary
/usr/local/mariadb-5.5.38-linux-x86_64/bin/mysqld Ver 5.5.38-MariaDB-log for Linux on x86_64 (MariaDB Server)
|
|
File log
5.5.38-MariaDB-1~precise-log log from start
|
Do you have any kind of status monitoring (SHOW GLOBAL STATUS)?
Yes we use munin, and it's use run SHOW /!50000 ENGINE/ INNODB STATUS command
Do you happen to have general log enabled, and if not, would you be able to enable it till the next crash and upload it later?
Sorry but in this week was 3 crash, and we chage back to 5.5.33a
|
|
Thanks a lot for provided stack traces, we'll see if we can figure out the problem from there.
Please do attach the log that munin produced if you still have it, and also your cnf file(s), I suppose they haven't changed after the downgrade to 5.5.33a,
By SHOW GLOBAL STATUS I meant the literal command, it's different from SHOW ENGINE INNODB STATUS, and provides information about what's going on with the server in general. If you can enable it (e.g. if munin allows it), please do, at least for a day, and attach the log, it might be useful even though you are not getting crashes. If you can't do that, we'll try to go without it.
|
|
Hi,
Error: 140630 19:03:41 InnoDB: compressed page checksum mismatch (space 112227 page 33844): 33554432 != 4127160225
Is indication of database corruption on either in-memory or on storage. This is bad, but there have been several complains about these compressed tables, that there is someting wrong with current implementation. Can you avoid using them ?
The other crashes are related to long semaphore waits, there is something wrong with info output code, I will try to find out where and fix it.
R: Jan
|
|
Hi,
After Sig 6 I got InnoDB: Warning: CHECK TABLE on index `PRIMARY` of table `merleg`.`lista` returned 12.
merleg.lista is a compressed table.
After
alter table merleg.lista engine innodb;
table is fine.
|
|
revno: 4225
committer: Jan Lindström <jplindst@mariadb.org>
branch nick: 5.5
timestamp: Tue 2014-07-08 17:21:13 +0300
message:
MDEV-6348: mariadb crash signal 11
Analysis: sync array output function, should make sure that all
used pointers are valid before using them.
|
|
revno: 4274
committer: Jan Lindström <jplindst@mariadb.org>
branch nick: 10.0-innodb
timestamp: Tue 2014-07-08 18:51:34 +0300
message:
MDEV-6348: mariadb crash signal 11
Analysis: sync array output function, should make sure that all
used pointers are valid before using them.
Merge revision 4225 from lp:maria/5.5.
|
cd1