[MDEV-8044] sigsegv during XtraDB shutdown Created: 2015-04-24  Updated: 2015-10-21  Resolved: 2015-06-17

Status: Closed
Project: MariaDB Server
Component/s: Storage Engine - XtraDB
Affects Version/s: 10.1.4
Fix Version/s: N/A

Type: Bug Priority: Minor
Reporter: Kolbe Kegel (Inactive) Assignee: Jan Lindström (Inactive)
Resolution: Cannot Reproduce Votes: 0
Labels: None
Environment:

CentOS 6

Sprint: 10.1.6-1

 Description   

There's clearly not much info to go on here, but this task can act as a stub in case we find more to work with. The user is seeing sigsegv each time the server shuts down, after "InnoDB: Starting shutdown" and before "InnoDB: Shutdown completed". There is no useful backtrace.

150423 17:12:28 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql
 
150423 17:12:28 mysqld_safe WSREP: Running position recovery with --log_error='/var/lib/mysql/wsrep_recovery.kYi8Y1' --pid-file='...'
 
150423 17:12:41 mysqld_safe WSREP: Position recovery skipped
 
150423 17:12:42 [Note] InnoDB: Using mutexes to ref count buffer pool pages
 
150423 17:12:42 [Note] InnoDB: The InnoDB memory heap is disabled
 
150423 17:12:42 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
 
150423 17:12:42 [Note] InnoDB: Memory barrier is not used
 
150423 17:12:42 [Note] InnoDB: Compressed tables use zlib 1.2.3
 
150423 17:12:42 [Note] InnoDB: Using Linux native AIO
 
150423 17:12:42 [Note] InnoDB: Using CPU crc32 instructions
 
150423 17:12:42 [Note] InnoDB: Initializing buffer pool, size = 200.0G
 
150423 17:12:50 [Note] InnoDB: Completed initialization of buffer pool
 
150423 17:12:51 [Note] InnoDB: Highest supported file format is Barracuda.
 
150423 17:12:51 [Note] InnoDB: Redo log crypto: unencrypted key ver.
 
150423 17:12:51 [Note] InnoDB: Log scan progressed past the checkpoint lsn 121695848220106
 
150423 17:12:51 [Note] InnoDB: Database was not shutdown normally!
 
150423 17:12:51 [Note] InnoDB: Starting crash recovery.
 
150423 17:12:51 [Note] InnoDB: Reading tablespace information from the .ibd files...
 
150423 17:12:51 [Note] InnoDB: Restoring possible half-written data pages
 
150423 17:12:51 [Note] InnoDB: from the doublewrite buffer...
 
InnoDB: Doing recovery: scanned up to log sequence number 121695848220116
 
InnoDB: In a MySQL replication slave the last master binlog file
 
InnoDB: position 0 401474766, file name mysql-bin.084373
 
InnoDB: Last MySQL binlog file position 0 150702, file name ./mysql-bin.025303
 
150423 17:12:54 [Note] InnoDB: 128 rollback segment(s) are active.
 
150423 17:12:54 [Note] InnoDB: Waiting for purge to start
 
150423 17:12:54 [Note] InnoDB: Percona XtraDB (http://www.percona.com) 5.6.22-72.0 started; log sequence number 121695848220116
 
150423 17:12:54 [Note] Server socket created on IP: '0.0.0.0'.
 
150423 17:12:54 [Note] WSREP: Read nil XID from storage engines, skipping position init
 
150423 17:12:54 [Note] WSREP: wsrep_load(): loading provider library 'none'
 
150423 17:12:54 [Note] /usr/sbin/mysqld: ready for connections.
 
Version: '10.1.4-MariaDB-wsrep-log' socket: '/var/lib/mysql/mysql.sock' port: 3306 MariaDB Server, wsrep_25.10.r4144
 
150423 17:12:54 [Note] Slave I/O thread: connected to master 'replic@venom:3306',replication starts at GTID position '205-37100-103818655,0-37100-1297299746,504-37100-775544,503-37100-697202,81-37100-1979,805-37100-169,206-37100-106185681,309-37100-941659,807-37100-164,308-37100-597508,207-37100-103810546,311-37100-846659,314-37100-587201,806-37100-184,302-37100-71829638,214-37100-1094,500-37100-772002,208-37100-28120,513-37100-550392,904-37100-784,209-37100-962,36-37100-109836652,907-37100-696,30-37100-140906688,502-37100-749228,310-37100-442922,31-37100-142509720,210-37100-1081,912-37100-249,810-37100-27,211-37100-962,313-37100-625695,913-37100-277,32-37100-135991140,501-37100-778681,312-37100-486452,33-37100-135071016,801-37100-7498,517-37100-563773,304-37100-72239237,213-37100-891,2001-37100-1739,306-37100-70223908,34-37100-133513877,201-37100-105166098,902-37100-906,35-37100-132901879,514-37100-950440,519-37100-550327,908-37100-727,510-37100-1087405,202-37100-107537995,512-37100-492831,905-37100-557,203-37100-105579213,515-37
 
150423 17:13:47 [Note] Slave: received end packet from server, apparent master shutdown:
 
150423 17:13:47 [Note] Slave I/O thread: Failed reading log event, reconnecting to retry, log 'mysql-bin.025219' at position 40124; GTID position '205-37100-103818655,0-37100-1297301770,504-37100-775544,503-37100-697202,81-37100-1979,805-37100-169,206-37100-106185681,309-37100-
 
150423 17:13:47 [ERROR] Slave I/O: error reconnecting to master 'replic@venom:3306' - retry-time: 60 retries: 86400 message: Can't connect to MySQL server on 'venom' (111 "Connection refused"), Internal MariaDB error code: 2003
 
150423 17:13:51 [Note] /usr/sbin/mysqld: Normal shutdown
 
 
 
150423 17:13:51 [Note] Event Scheduler: Purging the queue. 0 events
 
150423 17:13:51 [Note] Slave I/O thread exiting, read up to log 'mysql-bin.025219', position 40124; GTID position 205-37100-103818655,0-37100-1297301770,504-37100-775544,503-37100-697202,81-37100-1979,805-37100-169,206-37100-106185681,309-37100-941659,807-37100-164,308-37100-597508,207-37100-103810546,311-37100-846659,314-37100-587201,806-37100-184,302-37100-71829638,214-37100-1094,500-37100-772002,208-37100-28120,513-37100-550392,904-37100-784,209-37100-962,36-37100-109836652,907-37100-696,30-37100-140906688,502-37100-749228,310-37100-442922,31-37100-142509720,210-37100-1081,912-37100-249,810-37100-27,211-37100-962,313-37100-625695,913-37100-277,32-37100-135991140,501-37100-778681,312-37100-486452,33-37100-135071016,801-37100-7498,517-37100-563773,304-37100-72239237,213-37100-891,2001-37100-1739,306-37100-70223908,34-37100-133513877,201-37100-105166098,902-37100-906,35-37100-132901879,514-37100-950440,519-37100-550327,908-37100-727,510-37100-1087405,202-37100-107537995,512-37100-492831,905-37100-557,203-37100-105579213,515-37100-
 
150423 17:13:51 [Note] InnoDB: FTS optimize thread exiting.
 
150423 17:13:51 [Note] InnoDB: Starting shutdown...
 
150423 17:13:51 [ERROR] mysqld got signal 11 ;
 
This could be because you hit a bug. It is also possible that this binary
 
or one of the libraries it was linked against is corrupt, improperly built,
 
or misconfigured. This error can also be caused by malfunctioning hardware.
 
 
 
To report this bug, see http://kb.askmonty.org/en/reporting-bugs
 
 
 
We will try our best to scrape up some info that will hopefully help
 
diagnose the problem, but since we have already crashed,
 
something is definitely wrong and this may fail.
 
 
 
Server version: 10.1.4-MariaDB-wsrep-log
 
key_buffer_size=2147483648
 
read_buffer_size=8388608
 
max_used_connections=27
 
max_threads=402
 
thread_count=0
 
It is possible that mysqld could use up to
 
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 11985247 K bytes of memory
 
Hope that's ok; if not, decrease some variables in the equation.
 
 
 
Thread pointer: 0x0x0
 
Attempting backtrace. You can use the following information to find out
 
where mysqld died. If you see no messages after this, something went
 
terribly wrong...
 
stack_bottom = 0x0 thread_stack 0x48000
 
150423 17:13:52 mysqld_safe Number of processes running now: 0
 
150423 17:13:52 mysqld_safe mysqld restarted



 Comments   
Comment by Kolbe Kegel (Inactive) [ 2015-04-24 ]

This occurred when 10.1.4 was running on tables created under a 10.1.3 server.

Comment by Kolbe Kegel (Inactive) [ 2015-04-24 ]

This sigsegv-on-shutdown issue eventually was replaced by an assertion on crash recovery:

InnoDB: Apply batch completed
 
InnoDB: In a MySQL replication slave the last master binlog file
 
InnoDB: position 0 401474766, file name mysql-bin.084373
 
InnoDB: Last MySQL binlog file position 0 71036301, file name ./mysql-bin.025331
 
InnoDB: Cleaning up trx with id 158495032187
 
2015-04-23 17:43:17 7f3479664820  InnoDB: Assertion failure in thread 139863351773216 in file trx0trx.cc line 467
 
InnoDB: Failing assertion: trx->update_undo == NULL
 
InnoDB: We intentionally generate a memory trap.
 
InnoDB: Submit a detailed bug report to http://bugs.mysql.com.
 
InnoDB: If you get repeated assertion failures or crashes, even
 
InnoDB: immediately after the mysqld startup, there may be
 
InnoDB: corruption in the InnoDB tablespace. Please refer to
 
InnoDB: http://dev.mysql.com/doc/refman/5.6/en/forcing-innodb-recovery.html
 
InnoDB: about forcing recovery.
 
150423 17:43:17 [ERROR] mysqld got signal 6 ;

Comment by Jan Lindström (Inactive) [ 2015-05-11 ]

Can't repeat with this information, I would either need a database that crashes on crash recovery or more detailed steps how to repeat.

Comment by Elena Stepanova [ 2015-10-21 ]

See MDEV-8337 – looks like the same assertion failure.

Generated at Thu Feb 08 07:24:13 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.