[MDEV-9250] new server crashed Created: 2015-12-09  Updated: 2016-07-18  Resolved: 2016-07-12

Status: Closed
Project: MariaDB Server
Component/s: Events
Affects Version/s: 10.0.12, 5.5, 10.0, 10.1, 10.2
Fix Version/s: 10.0.22, 10.1.9

Type: Bug Priority: Major
Reporter: Zhoukai Lin Assignee: Sergei Golubchik
Resolution: Fixed Votes: 0
Labels: upstream-fixed
Environment:

redhat 6.5

Attachments: File my.cnf    

 Description   

Server crashed at the starting time, just after is has been installed.

error log:

151119 17:18:34 [Note] InnoDB: Using mutexes to ref count buffer pool pages
 
151119 17:18:34 [Note] InnoDB: The InnoDB memory heap is disabled
 
151119 17:18:34 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
 
151119 17:18:34 [Note] InnoDB: Compressed tables use zlib 1.2.3
 
151119 17:18:34 [Note] InnoDB: Using CPU crc32 instructions
 
151119 17:18:34 [Note] InnoDB: Initializing buffer pool, size = 600.0M
 
151119 17:18:34 [Note] InnoDB: Completed initialization of buffer pool
 
151119 17:18:34 [Note] InnoDB: Highest supported file format is Barracuda.
 
151119 17:18:34 [Note] InnoDB: 128 rollback segment(s) are active.
 
151119 17:18:34 [Note] InnoDB: Waiting for purge to start
 
151119 17:18:34 [Note] InnoDB:  Percona XtraDB (http://www.percona.com) 5.6.17-65.0 started; log sequence number 1616718
 
151119 17:18:34 [Note] Semi-sync replication initialized for transactions.
 
151119 17:18:34 [Note] Semi-sync replication enabled on the master.
 
151119 17:18:34 server_audit: MariaDB Audit Plugin version 1.1.7 STARTED.
 
151119 17:18:34 [Note] Server socket created on IP: '10.46.178.76'.
 
 
 
Status information:
 
 
 
Current dir: /home/maria76/data/data/
 
Running threads: 0  Stack size: 294912
 
Current locks:
 
lock: 0x7f5472c60d88:
 
 
 
lock: 0x7f53c8018508:
 
 
 
lock: 0x18f1760:
 
 
 
lock: 0x18f1680:
 
 
 
lock: 0x18f06c0:
 
 
 
lock: 0x18f05e0:
 
 
 
lock: 0x18f0500:
 
 
 
lock: 0x18f0880:
 
 
 
lock: 0x18eeac0:
 
 
 
lock: 0x18f15a0:
 
 
 
lock: 0x18ef1c0:
 
 
 
lock: 0x18ef460:
 
 
 
lock: 0x18ef0e0:
 
 
 
lock: 0x18ef380:
 
 
 
lock: 0x18ef000:
 
 
 
lock: 0x18ef2a0:
 
 
 
lock: 0x18ef9a0:
 
 
 
lock: 0x18ef8c0:
 
 
 
lock: 0x18ef7e0:
 
 
 
lock: 0x18eef20:
 
 
 
lock: 0x18eec80:
 
 
 
lock: 0x18eee40:
 
 
 
lock: 0x18eeba0:
 
 
 
lock: 0x18eed60:
 
 
 
lock: 0x18ef700:
 
 
 
lock: 0x18ef620:
 
 
 
lock: 0x18ef540:
 
 
 
lock: 0x18f1220:
 
 
 
lock: 0x18f14c0:
 
 
 
lock: 0x18f13e0:
 
 
 
lock: 0x18f1300:
 
 
 
lock: 0x18f0ea0:
 
 
 
lock: 0x18f0dc0:
 
 
 
lock: 0x18f0ce0:
 
 
 
lock: 0x18f0c00:
 
 
 
lock: 0x18f0b20:
 
 
 
lock: 0x18f1060:
 
 
 
lock: 0x18f0a40:
 
 
 
lock: 0x18f0960:
 
 
 
lock: 0x18f0f80:
 
 
 
lock: 0x18f07a0:
 
 
 
lock: 0x18f0340:
 
 
 
lock: 0x18f0420:
 
 
 
lock: 0x18f0260:
 
 
 
lock: 0x18f0180:
 
 
 
lock: 0x18efe00:
 
 
 
lock: 0x18f00a0:
 
 
 
lock: 0x18effc0:
 
 
 
lock: 0x18efd20:
 
 
 
lock: 0x18efee0:
 
 
 
lock: 0x18efc40:
 
 
 
lock: 0x18efb60:
 
 
 
lock: 0x18efa80:
 
 
 
lock: 0x18f1140:
 
 
 
151119 17:18:34 [ERROR] mysqld got signal 11 ;
 
This could be because you hit a bug. It is also possible that this binary
 
or one of the libraries it was linked against is corrupt, improperly built,
 
or misconfigured. This error can also be caused by malfunctioning hardware.
 
 
 
To report this bug, see http://kb.askmonty.org/en/reporting-bugs
 
 
 
We will try our best to scrape up some info that will hopefully help
 
diagnose the problem, but since we have already crashed, 
something is definitely wrong and this may fail.
 
 
 
Server version: 10.0.12-MariaDB-log
 
key_buffer_size=16777216
 
read_buffer_size=262144
 
max_used_connections=0
 
max_threads=153
 
thread_count=0
 
It is possible that mysqld could use up to 
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 137032 K  bytes of memory
 
Hope that's ok; if not, decrease some variables in the equation.
 
 
 
Thread pointer: 0x0x0
 
Attempting backtrace. You can use the following information to find out
 
where mysqld died. If you see no messages after this, something went
 
terribly wrong...
 
stack_bottom = 0x0 thread_stack 0x48000
 
mysys/stacktrace.c:247(my_print_stacktrace)[0xa9fc89]
 
sql/signal_handler.cc:153(handle_fatal_signal)[0x6dd61a]
 
/lib64/libpthread.so.0[0x3bfcc0f710]
 
sql/event_scheduler.cc:835(Event_scheduler::dump_internal_status())[0x7fdfbb]
 
sql/events.cc:1035(Events::dump_internal_status())[0x639e2a]
 
sql/sql_test.cc:648(mysql_print_status())[0x612ca4]
 
sql/mysqld.cc:3366(signal_hand)[0x51aa25]
 
/lib64/libpthread.so.0[0x3bfcc079d1]
 
/lib64/libc.so.6(clone+0x6d)[0x3bfc4e8b6d]
 
The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
 
information that should help you find out what is causing the crash.



 Comments   
Comment by Daniel Black [ 2015-12-22 ]

is this really 10.0.12? or is this mean to be 10.0.21? I see you have semisync enabled. What is the rest of the configuration? What is the semisync slave version?

Comment by Elena Stepanova [ 2015-12-29 ]

danblack, it is really 10.0.12, you can see it in the log.

Comment by Elena Stepanova [ 2015-12-29 ]

linzhoukai,
As Daniel has already requested above, please attach your cnf file(s).
Please also specify which packages you are using, and whether they were installed on a clean system, or had there been a MySQL/MariaDB/Percona server before.

Comment by Zhoukai Lin [ 2016-03-31 ]

I have a lot of servers useing the same cnf file.All compiled from 10.0.12 code.
cmake .. \
-DCMAKE_INSTALL_PREFIX=/home/maria76 \
-DDEFAULT_CHARSET=utf8 \
-DDEFAULT_COLLATION=utf8_bin \
-DWITH_EXTRA_CHARSETS:STRING=all \
-DWITH_MYISAM_STORAGE_ENGINE=1 \
-DWITH_INNOBASE_STORAGE_ENGINE=1 \
-DWITH_NDBCLUSTER_STORAGE_ENGINE=0 \
-DENABLED_LOCAL_INFILE=1

Only one crashed after install. This server has installed several MariaDBs. But the crash one were using a new system user.

my.cnf

Comment by Elena Stepanova [ 2016-06-11 ]

This is (was) an upstream problem, bug
https://bugs.mysql.com/bug.php?id=62311

When server receives SIGHUP on startup, it can crash.

In the provided error log we see that the server did receive the SIGHUP and started printing the status (I don't know where the signal came from in that installation, but it's not important).

The problem can be reproduced easily on non-debug builds of 10.0 before 10.0.22, by sending SIGHUP immediately after starting the server with event_scheduler=ON.

From 10.0.22 the crash stopped happening due to this change:
https://github.com/MariaDB/server/commit/18f7dfe

commit 18f7dfed179204dcfc02a27790e22bb9cc4e2e32
 
Author: Michael Widenius <monty@mariadb.org>
 
Date:   Thu Oct 15 12:11:17 2015 +0300
 
 
 
    Allow mysql_upgrade to enable event after table is corrected
 
    
    new features:
 
    set event_scheduler=ON|OFF will now try to init event scheduler
 
    if it's not enabled
 
    set event_scheduler=default will try to enable it based on
 
    the value of the event_scheduler when mysqld was started

Particularly, due to this change:

sql/events.cc

 
@@ -1028,7 +1040,7 @@ Events::dump_internal_status()
 
     holding LOCK_global_system_variables.
 
   */
 
   mysql_mutex_lock(&LOCK_global_system_variables);
 
-  if (opt_event_scheduler == EVENTS_DISABLED)
 
+  if (!inited)
 
     puts("The Event Scheduler is disabled");
 
   else
 
   {

If the change is reverted on the current trees, the crash starts happening again.

However, the upstream fix looks much more global:
https://github.com/mysql/mysql-server/commit/c6b299db9c3f04207972075c7ac5f172a471c085

I'm not sure whether disabling only event_scheduler dump until it's fully initialized fixes everything, so before closing the issue I'd like somebody to have a look at the upstream fix and decide whether we need it.

5.5 is still affected, both upstream and MariaDB. Probably it isn't worth backporting the fix to MariaDB 5.5, just like Oracle decided not to backport theirs to MySQL 5.5.

Generated at Thu Feb 08 07:33:16 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.