Details
-
Bug
-
Status: Closed (View Workflow)
-
Major
-
Resolution: Fixed
-
10.1.10
-
None
-
Linux 3.16.0-4-amd64 #1 SMP Debian 3.16.7-ckt20-1+deb8u1 (2015-12-14) x86_64 GNU/Linux
Supermicro Server Intel Xeon X3430 @ 2.40GHz Cache Size 8192 KB
RAM: 16GB
Samsung SSD 850
Description
This server is running MariaDB in a cluster with other 3 servers and I've FreeRadius running on the same server
Today at about 6:35AM GMT -2 the server crashed and I could see the follow log:
Jan 27 06:25:01 davinci3 CRON[3059]: (root) CMD (test -x /usr/sbin/anacron || ( cd / && run-parts --report /etc/cron.daily ))
|
Jan 27 06:25:02 davinci3 mysqld[19220]: 160127 6:25:02 [ERROR] mysqld got signal 11 ;
|
Jan 27 06:25:02 davinci3 mysqld[19220]: This could be because you hit a bug. It is also possible that this binary
|
Jan 27 06:25:02 davinci3 mysqld[19220]: or one of the libraries it was linked against is corrupt, improperly built,
|
Jan 27 06:25:02 davinci3 mysqld[19220]: or misconfigured. This error can also be caused by malfunctioning hardware.
|
Jan 27 06:25:02 davinci3 mysqld[19220]: To report this bug, see http://kb.askmonty.org/en/reporting-bugs
|
Jan 27 06:25:02 davinci3 mysqld[19220]: We will try our best to scrape up some info that will hopefully help
|
Jan 27 06:25:02 davinci3 mysqld[19220]: diagnose the problem, but since we have already crashed,
|
Jan 27 06:25:02 davinci3 mysqld[19220]: something is definitely wrong and this may fail.
|
Jan 27 06:25:02 davinci3 mysqld[19220]: Server version: 10.1.10-MariaDB-1~jessie-log
|
Jan 27 06:25:02 davinci3 mysqld[19220]: key_buffer_size=134217728
|
Jan 27 06:25:02 davinci3 mysqld[19220]: read_buffer_size=2097152
|
Jan 27 06:25:02 davinci3 mysqld[19220]: max_used_connections=301
|
Jan 27 06:25:02 davinci3 mysqld[19220]: max_threads=302
|
Jan 27 06:25:02 davinci3 mysqld[19220]: thread_count=23
|
Jan 27 06:25:02 davinci3 mysqld[19220]: It is possible that mysqld could use up to
|
Jan 27 06:25:02 davinci3 mysqld[19220]: key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 1992709 K bytes of memory
|
Jan 27 06:25:02 davinci3 mysqld[19220]: Hope that's ok; if not, decrease some variables in the equation.
|
Jan 27 06:25:02 davinci3 mysqld[19220]: Thread pointer: 0x0x7fc1a5462008
|
Jan 27 06:25:02 davinci3 mysqld[19220]: Attempting backtrace. You can use the following information to find out
|
Jan 27 06:25:02 davinci3 mysqld[19220]: where mysqld died. If you see no messages after this, something went
|
Jan 27 06:25:02 davinci3 mysqld[19220]: terribly wrong...
|
Jan 27 06:25:02 davinci3 mysqld[19220]: stack_bottom = 0x7fc1fc178df8 thread_stack 0x48000
|
Jan 27 06:25:02 davinci3 mysqld[19220]: /usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x7fc20101c14e]
|
Jan 27 06:25:02 davinci3 mysqld[19220]: /usr/sbin/mysqld(handle_fatal_signal+0x34d)[0x7fc200b5fadd]
|
Jan 27 06:25:02 davinci3 mysqld[19220]: /lib/x86_64-linux-gnu/libpthread.so.0(+0xf8d0)[0x7fc2001908d0]
|
Jan 27 06:25:02 davinci3 mysqld[19220]: /usr/sbin/mysqld(_ZN13MYSQL_BIN_LOG21do_checkpoint_requestEm+0x98)[0x7fc200c14b68]
|
Jan 27 06:25:02 davinci3 mysqld[19220]: /usr/sbin/mysqld(_ZN13MYSQL_BIN_LOG20checkpoint_and_purgeEm+0x11)[0x7fc200c14b91]
|
Jan 27 06:25:02 davinci3 mysqld[19220]: /usr/sbin/mysqld(_ZN13MYSQL_BIN_LOG16rotate_and_purgeEb+0x7e)[0x7fc200c1703e]
|
Jan 27 06:25:02 davinci3 mysqld[19220]: /usr/sbin/mysqld(_Z20reload_acl_and_cacheP3THDyP10TABLE_LISTPi+0x131)[0x7fc200ac4af1]
|
Jan 27 06:25:02 davinci3 mysqld[19220]: /usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x1408)[0x7fc2009d7558]
|
Jan 27 06:25:02 davinci3 mysqld[19220]: /usr/sbin/mysqld(_Z11mysql_parseP3THDPcjP12Parser_state+0x20e)[0x7fc2009df5fe]
|
Jan 27 06:25:02 davinci3 mysqld[19220]: /usr/sbin/mysqld(+0x41edf9)[0x7fc2009dfdf9]
|
Jan 27 06:25:02 davinci3 mysqld[19220]: /usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcj+0x195c)[0x7fc2009e1d8c]
|
Jan 27 06:25:02 davinci3 mysqld[19220]: /usr/sbin/mysqld(_Z10do_commandP3THD+0x16e)[0x7fc2009e2b4e]
|
Jan 27 06:25:02 davinci3 mysqld[19220]: /usr/sbin/mysqld(_Z24do_handle_one_connectionP3THD+0x182)[0x7fc200aab602]
|
Jan 27 06:25:02 davinci3 mysqld[19220]: /usr/sbin/mysqld(handle_one_connection+0x40)[0x7fc200aab7c0]
|
Jan 27 06:25:02 davinci3 mysqld[19220]: /lib/x86_64-linux-gnu/libpthread.so.0(+0x80a4)[0x7fc2001890a4]
|
Jan 27 06:25:02 davinci3 mysqld[19220]: /lib/x86_64-linux-gnu/libc.so.6(clone+0x6d)[0x7fc1fe33504d]
|
Jan 27 06:25:02 davinci3 mysqld[19220]: Trying to get some variables.
|
Jan 27 06:25:02 davinci3 mysqld[19220]: Some pointers may be invalid and cause the dump to abort.
|
Jan 27 06:25:02 davinci3 mysqld[19220]: Query (0x7fc1b1c87020): flush logs
|
Jan 27 06:25:02 davinci3 mysqld[19220]: Connection ID (thread ID): 3009977
|
Jan 27 06:25:02 davinci3 mysqld[19220]: Status: NOT_KILLED
|
Jan 27 06:25:02 davinci3 mysqld[19220]: Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushd
|
own=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on
|
,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=o
|
ff,table_elimination=on,extended_keys=on,exists_to_in=on
|
Jan 27 06:25:02 davinci3 mysqld[19220]: The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
|
Jan 27 06:25:02 davinci3 mysqld[19220]: information that should help you find out what is causing the crash.
|
Jan 27 06:25:02 davinci3 systemd[1]: mariadb.service: main process exited, code=exited, status=1/FAILURE
|
Jan 27 06:25:02 davinci3 systemd[1]: Unit mariadb.service entered failed state.
|
ffkammer,
Is the problem reproducible?
Did you run flush logs manually, or did it come from RADIUS? Could you please paste the output of ls -l <datadir>, or wherever your binary logs reside (or upload to ftp.askmonty.org/private if you so prefer)?
What do you mean by cluster? Is it a Galera cluster, or regular replication, or some other setup? Could you please describe the topology?
I'm not familiar with details of FreeRADIUS technology, do you know whether it only uses SQL server (MariaDB in this case) via a regular client connection, or does it have direct access to on-disk MariaDB files, e.g. logs?
Please also attach your cnf file(s). If you happen to have the general log enabled, it would also be a big help.