[MDEV-9624] mysqld crash Created: 2016-02-24  Updated: 2016-03-30  Resolved: 2016-03-30

Status: Closed
Project: MariaDB Server
Component/s: Galera, Replication
Affects Version/s: 10.1.11
Fix Version/s: 10.1.13

Type: Bug Priority: Major
Reporter: Karl E. Jørgensen Assignee: Nirbhay Choubey (Inactive)
Resolution: Fixed Votes: 1
Labels: None
Environment:

CentOS Linux release 7.2.1511 (Core), 64-bit

Attachments: Text File crashlog.txt    
Issue Links:
Relates
relates to MDEV-9498 MariaDB server with Galera replicatio... Closed
relates to MDEV-9478 MariaDB 10.1.10 crashs on Debian Jessie Closed

 Description   

We had a mysqld crash which appears to be during binlog rotation:

160224  9:51:59 [ERROR] mysqld got signal 11 ;
 
...
 
stack_bottom = 0x7fc43aa2d8d0 thread_stack 0x48400
 
/usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x7fc541c0ccce]
 
/usr/sbin/mysqld(handle_fatal_signal+0x38d)[0x7fc54173a49d]
 
/lib64/libpthread.so.0(+0xf100)[0x7fc540d5a100]
 
/usr/sbin/mysqld(_ZN13MYSQL_BIN_LOG21do_checkpoint_requestEm+0x9d)[0x7fc5417fb90d]
 
/usr/sbin/mysqld(_ZN13MYSQL_BIN_LOG20checkpoint_and_purgeEm+0x11)[0x7fc5417fb9a1]
 
/usr/sbin/mysqld(_ZN13MYSQL_BIN_LOG23trx_group_commit_leaderEPNS_18group_commit_entryE+0x50a)[0x7fc5417fe95a]
 
/usr/sbin/mysqld(_ZN13MYSQL_BIN_LOG34write_transaction_to_binlog_eventsEPNS_18group_commit_entryE+0x93)[0x7fc5417fec63]
 
/usr/sbin/mysqld(_ZN13MYSQL_BIN_LOG27write_transaction_to_binlogEP3THDP17binlog_cache_mngrP9Log_eventbbb+0xcf)[0x7fc5417fef7f]
 
/usr/sbin/mysqld(+0x6740f8)[0x7fc5417ff0f8]
 
/usr/sbin/mysqld(+0x67465c)[0x7fc5417ff65c]
 
/usr/sbin/mysqld(_ZN13MYSQL_BIN_LOG13log_and_orderEP3THDybbb+0x44)[0x7fc5417ff6f4]
 
/usr/sbin/mysqld(_Z15ha_commit_transP3THDb+0x482)[0x7fc54173d782]
 
/usr/sbin/mysqld(_Z12trans_commitP3THD+0x5b)[0x7fc54169753b]
 
/usr/sbin/mysqld(_ZN13Xid_log_event14do_apply_eventEP14rpl_group_info+0xba)[0x7fc541807f2a]
 
/usr/sbin/mysqld(_Z26apply_event_and_update_posP9Log_eventP3THDP14rpl_group_infoP19rpl_parallel_thread+0x1e1)[0x7fc541545441]
 
/usr/sbin/mysqld(handle_slave_sql+0x261b)[0x7fc541548e7b]
 
/lib64/libpthread.so.0(+0x7dc5)[0x7fc540d52dc5]
 
/lib64/libc.so.6(clone+0x6d)[0x7fc53f17628d]

(full log is attached in crashlog.txt as it appeared in /var/log/messages.

From a cursory investigation, these observations may be relevant:

  • The crashing mysqld instance was part of a 2-node galera cluster. (The cluster was under construction, and a 3rd node was due to join it later).
  • A new file with a timestamp of the crash appeared in /var/lib/mysql: 0x009fe611_data.000000. The contents appears to be binary of some sort, and it contains what appears to be schema/tablenames towards the beginning. I suspect the contents may aid diagnostics
  • The node was running inbound replication - no other activity on the node yet (or elsewhere in the cluster)
  • Replication is "traditional" - i.e. not using GTID
  • Parallel replication is not configured - i.e. defaults to single thread
  • It appears to have happened during (or immediately after) binlog rotation: A new bin log file had been created - 405 bytes long, and with no transactions yet according to mysqlbinlog. The timestamp is in the same second as the crash.
  • Other nodes (elsewhere in our estate) replicating off the same master did not encounter any problems

 Comments   
Comment by Karl E. Jørgensen [ 2016-02-24 ]

The stack traces also appear in MDEV-9498 and MDEV-9478 ...

Comment by Elena Stepanova [ 2016-02-24 ]

nirbhay_c, I think the fact that all three reports concern instances running Galera cluster is notable. Probably these issues should be analyzed together as a whole.

Generated at Thu Feb 08 07:36:05 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.