[MDEV-11550] Crash when running flush logs Created: 2016-12-12  Updated: 2020-08-25  Resolved: 2017-12-13

Status: Closed
Project: MariaDB Server
Component/s: Galera, Replication
Affects Version/s: 10.1.18
Fix Version/s: 10.1.30, 10.2.11

Type: Bug Priority: Major
Reporter: Nathan Madpsy Assignee: Sachin Setiya (Inactive)
Resolution: Duplicate Votes: 0
Labels: galera
Environment:

Ubuntu 16.04 (official image on Azure)


Issue Links:
Duplicate
duplicates MDEV-10276 Crash signal 11 during logrotate Closed
duplicates MDEV-11860 Signal 11 - Buffer overflow detected Closed
Relates
relates to MDEV-11610 Logrotate to only FLUSH LOCAL ERROR L... Closed

 Description   

First time I have seen this on my 3 node Galera cluster. It appears when the daily cron ran to flush logs via logrotate something caused maraidb to crash.

I don't have much more information except the command which was run by logrotate (below) and the trace from maridadb.

mysqladmin --defaults-file=/etc/mysql/debian.cnf flush-logs

Dec 10 06:25:04 ams-prod1-db03 mysqld[2001]: 161210  6:25:04 [ERROR] mysqld got signal 11 ;
Dec 10 06:25:04 ams-prod1-db03 mysqld[2001]: This could be because you hit a bug. It is also possible that this binary
Dec 10 06:25:04 ams-prod1-db03 mysqld[2001]: or one of the libraries it was linked against is corrupt, improperly built,
Dec 10 06:25:04 ams-prod1-db03 mysqld[2001]: or misconfigured. This error can also be caused by malfunctioning hardware.
Dec 10 06:25:04 ams-prod1-db03 mysqld[2001]: To report this bug, see https://mariadb.com/kb/en/reporting-bugs
Dec 10 06:25:04 ams-prod1-db03 mysqld[2001]: We will try our best to scrape up some info that will hopefully help
Dec 10 06:25:04 ams-prod1-db03 mysqld[2001]: diagnose the problem, but since we have already crashed,
Dec 10 06:25:04 ams-prod1-db03 mysqld[2001]: something is definitely wrong and this may fail.
Dec 10 06:25:04 ams-prod1-db03 mysqld[2001]: Server version: 10.1.18-MariaDB-1~xenial
Dec 10 06:25:04 ams-prod1-db03 mysqld[2001]: key_buffer_size=134217728
Dec 10 06:25:04 ams-prod1-db03 mysqld[2001]: read_buffer_size=2097152
Dec 10 06:25:04 ams-prod1-db03 mysqld[2001]: max_used_connections=65
Dec 10 06:25:04 ams-prod1-db03 mysqld[2001]: max_threads=502
Dec 10 06:25:04 ams-prod1-db03 mysqld[2001]: thread_count=8
Dec 10 06:25:04 ams-prod1-db03 mysqld[2001]: It is possible that mysqld could use up to
Dec 10 06:25:04 ams-prod1-db03 mysqld[2001]: key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 3225650 K  bytes of memory
Dec 10 06:25:04 ams-prod1-db03 mysqld[2001]: Hope that's ok; if not, decrease some variables in the equation.
Dec 10 06:25:04 ams-prod1-db03 mysqld[2001]: Thread pointer: 0x0x7f571d4fe008
Dec 10 06:25:04 ams-prod1-db03 mysqld[2001]: Attempting backtrace. You can use the following information to find out
Dec 10 06:25:04 ams-prod1-db03 mysqld[2001]: where mysqld died. If you see no messages after this, something went
Dec 10 06:25:04 ams-prod1-db03 mysqld[2001]: terribly wrong...
Dec 10 06:25:04 ams-prod1-db03 mysqld[2001]: stack_bottom = 0x7f578bd8c088 thread_stack 0x48400
Dec 10 06:25:04 ams-prod1-db03 mysqld[2001]: (my_addr_resolve failure: fork)
Dec 10 06:25:04 ams-prod1-db03 mysqld[2001]: /usr/sbin/mysqld(my_print_stacktrace+0x2e) [0x561e94e7562e]
Dec 10 06:25:04 ams-prod1-db03 mysqld[2001]: /usr/sbin/mysqld(handle_fatal_signal+0x2d5) [0x561e949c43d5]
Dec 10 06:25:04 ams-prod1-db03 mysqld[2001]: /lib/x86_64-linux-gnu/libpthread.so.0(+0x113d0) [0x7f583594f3d0]
Dec 10 06:25:04 ams-prod1-db03 mysqld[2001]: /usr/sbin/mysqld(MYSQL_BIN_LOG::do_checkpoint_request(unsigned long)+0x4a) [0x561e94a7957a]
Dec 10 06:25:04 ams-prod1-db03 mysqld[2001]: /usr/sbin/mysqld(MYSQL_BIN_LOG::checkpoint_and_purge(unsigned long)+0x11) [0x561e94a795f1]
Dec 10 06:25:04 ams-prod1-db03 mysqld[2001]: /usr/sbin/mysqld(MYSQL_BIN_LOG::rotate_and_purge(bool)+0x7e) [0x561e94a7baae]
Dec 10 06:25:04 ams-prod1-db03 mysqld[2001]: /usr/sbin/mysqld(reload_acl_and_cache(THD*, unsigned long long, TABLE_LIST*, int*)+0x135) [0x561e94925045]
Dec 10 06:25:04 ams-prod1-db03 mysqld[2001]: /usr/sbin/mysqld(mysql_execute_command(THD*)+0x1365) [0x561e948360f5]
Dec 10 06:25:04 ams-prod1-db03 mysqld[2001]: /usr/sbin/mysqld(mysql_parse(THD*, char*, unsigned int, Parser_state*)+0x311) [0x561e9483e031]
Dec 10 06:25:04 ams-prod1-db03 mysqld[2001]: /usr/sbin/mysqld(+0x40f899) [0x561e9483e899]
Dec 10 06:25:04 ams-prod1-db03 mysqld[2001]: /usr/sbin/mysqld(dispatch_command(enum_server_command, THD*, char*, unsigned int)+0x1c3b) [0x561e94840b8b]
Dec 10 06:25:04 ams-prod1-db03 mysqld[2001]: /usr/sbin/mysqld(do_command(THD*)+0x16d) [0x561e948419ad]
Dec 10 06:25:04 ams-prod1-db03 mysqld[2001]: /usr/sbin/mysqld(do_handle_one_connection(THD*)+0x182) [0x561e9490af52]
Dec 10 06:25:04 ams-prod1-db03 mysqld[2001]: /usr/sbin/mysqld(handle_one_connection+0x40) [0x561e9490b110]
Dec 10 06:25:04 ams-prod1-db03 mysqld[2001]: /usr/sbin/mysqld(+0x705fb1) [0x561e94b34fb1]
Dec 10 06:25:04 ams-prod1-db03 mysqld[2001]: /lib/x86_64-linux-gnu/libpthread.so.0(+0x76fa) [0x7f58359456fa]
Dec 10 06:25:04 ams-prod1-db03 mysqld[2001]: /lib/x86_64-linux-gnu/libc.so.6(clone+0x6d) [0x7f5834ff0b5d]
Dec 10 06:25:04 ams-prod1-db03 mysqld[2001]: Trying to get some variables.
Dec 10 06:25:04 ams-prod1-db03 mysqld[2001]: Some pointers may be invalid and cause the dump to abort.
Dec 10 06:25:04 ams-prod1-db03 mysqld[2001]: Query (0x7f57060e1020): flush logs
Dec 10 06:25:04 ams-prod1-db03 mysqld[2001]: Connection ID (thread ID): 20185572
Dec 10 06:25:04 ams-prod1-db03 mysqld[2001]: Status: NOT_KILLED
Dec 10 06:25:04 ams-prod1-db03 mysqld[2001]: Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=off,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=off
Dec 10 06:25:04 ams-prod1-db03 mysqld[2001]: The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
Dec 10 06:25:04 ams-prod1-db03 mysqld[2001]: information that should help you find out what is causing the crash.



 Comments   
Comment by Florian Engelmann [ 2016-12-16 ]

Could you please check if flush-logs get stuck if started manual? We have got the same issue and to me it looks like signal 11 is caused by multiple hanging flush-logs.

Comment by Florian Engelmann [ 2016-12-16 ]

It doesn't get stuck if started frequently after the DB was started. It looks like it gets stuck after binlog has grown to a certain amount of queries?

Comment by Nathan Madpsy [ 2016-12-16 ]

It's only happened once and it runs every night so I would imagine running manually would be fine.

Comment by Hartmut Holzgraefe [ 2017-05-15 ]

Isn't this actually a duplicate of earlier MDEV-10276, too?

Comment by Sachin Setiya (Inactive) [ 2017-12-12 ]

This one is related to MDEV-9510, Elkin patch should have solved this issue.

Comment by Andrei Elkin [ 2017-12-13 ]

Fixed by MDEV-9510.

Generated at Thu Feb 08 07:50:49 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.