[MDEV-9177] upgrade from 10.1.8 to 10.1.9 crashes server Created: 2015-11-24  Updated: 2020-12-08  Resolved: 2015-12-15

Status: Closed
Project: MariaDB Server
Component/s: Storage Engine - InnoDB, Storage Engine - XtraDB
Affects Version/s: 10.1.9
Fix Version/s: 10.1.9

Type: Bug Priority: Major
Reporter: Thomas Stein Assignee: Jan Lindström (Inactive)
Resolution: Fixed Votes: 1
Labels: None
Environment:

Gentoo Linux. Kernel 4.1.12


Issue Links:
Duplicate
is duplicated by MDEV-9272 apt-get upgrade to 10.1.9 fails on De... Closed

 Description   

Hello Devs.

After upgrading to 10.1.9 mariadb crashes hard. Downgrade to 10.1.8 solves the issue. Here the crash log.

2015-11-24 13:48:45 140258989979520 [Note] InnoDB: Using mutexes to ref count buffer pool pages
2015-11-24 13:48:45 140258989979520 [Note] InnoDB: The InnoDB memory heap is disabled
2015-11-24 13:48:45 140258989979520 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
2015-11-24 13:48:45 140258989979520 [Note] InnoDB: Memory barrier is not used
2015-11-24 13:48:45 140258989979520 [Note] InnoDB: Compressed tables use zlib 1.2.8
2015-11-24 13:48:45 140258989979520 [Note] InnoDB: Using Linux native AIO
2015-11-24 13:48:45 140258989979520 [Note] InnoDB: Using CPU crc32 instructions
2015-11-24 13:48:45 140258989979520 [Note] InnoDB: Initializing buffer pool, size = 2.0G
2015-11-24 13:48:45 140258989979520 [Note] InnoDB: Completed initialization of buffer pool
2015-11-24 13:48:45 140258989979520 [Note] InnoDB: Highest supported file format is Barracuda.
2015-11-24 13:48:46 140258989979520 [Note] InnoDB: 128 rollback segment(s) are active.
2015-11-24 13:48:46 140258989979520 [Note] InnoDB: Waiting for purge to start
2015-11-24 13:48:46 140258989979520 [Note] InnoDB:  Percona XtraDB (http://www.percona.com) 5.6.26-74.0 started; log sequence number 1555843502
2015-11-24 13:48:46 140256120637184 [Note] InnoDB: Dumping buffer pool(s) not yet started
2015-11-24 13:48:46 140258989979520 [Note] Plugin 'FEEDBACK' is disabled.
2015-11-24 13:48:46 140258989979520 [Note] Server socket created on IP: '127.0.0.1'.
2015-11-24 13:48:46 140258989979520 [Note] /usr/sbin/mysqld: ready for connections.
Version: '10.1.9-MariaDB-log'  socket: '/var/run/mysqld/mysqld.sock'  port: 3306  Source distribution
151124 13:48:46 [ERROR] mysqld got signal 11 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
 
To report this bug, see http://kb.askmonty.org/en/reporting-bugs
 
We will try our best to scrape up some info that will hopefully help
diagnose the problem, but since we have already crashed, 
something is definitely wrong and this may fail.
 
Server version: 10.1.9-MariaDB-log
key_buffer_size=33554432
read_buffer_size=249856
max_used_connections=0
max_threads=52
thread_count=0
It is possible that mysqld could use up to 
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 59167 K  bytes of memory
Hope that's ok; if not, decrease some variables in the equation.
 
Thread pointer: 0x0x0
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 0x0 thread_stack 0x48000
/usr/sbin/mysqld(my_print_stacktrace+0x29)[0x557fc0208e89]
/usr/sbin/mysqld(handle_fatal_signal+0x355)[0x557fbfdebd15]
/lib64/libpthread.so.0(+0x10220)[0x7f9095e00220]
/usr/sbin/mysqld(+0x7d3dfa)[0x557fc0078dfa]
/usr/sbin/mysqld(+0x7a84fa)[0x557fc004d4fa]
/usr/sbin/mysqld(+0x776640)[0x557fc001b640]
/usr/sbin/mysqld(+0x7d11b6)[0x557fc00761b6]
/usr/sbin/mysqld(+0x7c1b67)[0x557fc0066b67]
/lib64/libpthread.so.0(+0x7314)[0x7f9095df7314]
/lib64/libc.so.6(clone+0x6d)[0x7f9094c7051d]
The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
information that should help you find out what is causing the crash.{quote}

Any hints how to solve this?

thanks and cheers



 Comments   
Comment by Elena Stepanova [ 2015-11-24 ]

himpierre,

Is it a production server?
How big is the datadir? Is it possible to upload it so we could investigate? (If you upload it to ftp.askmonty.org/private, only MariaDB developers will have access to it).
Do you use a binary build, or do you build from sources? Would you be able to run a debug build to get a proper stack trace?

In any case, please also attach your cnf file(s).

Thanks.

Comment by Thomas Stein [ 2015-11-24 ]

Hello.

It's a productions server, yes. Build from source via gentoo ebuild. Datadir is 1.6G. Probably i am able to have a debug build. I have to look into it.

And for your upload question. I guess i could do that. Can i have login credentials?

Here the my.cnf

# /etc/mysql/my.cnf: The global mysql configuration file.
# $Header: /var/cvsroot/gentoo-x86/dev-db/mariadb/files/my.cnf-5.6,v 1.1 2014/10/08 16:41:46 grknight Exp $
 
# The following options will be passed to all MySQL clients
[client]
#password					= your_password
port						= 3306
socket						= /var/run/mysqld/mysqld.sock
 
[mysql]
character-sets-dir=/usr/share/mysql/charsets
default-character-set=utf8
 
[mysqladmin]
character-sets-dir=/usr/share/mysql/charsets
default-character-set=utf8
 
[mysqlcheck]
character-sets-dir=/usr/share/mysql/charsets
default-character-set=utf8
 
[mysqldump]
character-sets-dir=/usr/share/mysql/charsets
default-character-set=utf8
 
[mysqlimport]
character-sets-dir=/usr/share/mysql/charsets
default-character-set=utf8
 
[mysqlshow]
character-sets-dir=/usr/share/mysql/charsets
default-character-set=utf8
 
[myisamchk]
character-sets-dir=/usr/share/mysql/charsets
 
[myisampack]
character-sets-dir=/usr/share/mysql/charsets
 
# use [safe_mysqld] with mysql-3
[mysqld_safe]
err-log						= /var/log/mysql/mysql.err
 
# add a section [mysqld-4.1] or [mysqld-5.0] for specific configurations
[mysqld]
character-set-server		= utf8
user 						= mysql
port 						= 3306
socket 						= /var/run/mysqld/mysqld.sock
pid-file 					= /var/run/mysqld/mysqld.pid
log-error 					= /var/log/mysql/mysqld.err
basedir 					= /usr
datadir 					= /var/lib/mysql
skip-external-locking
key_buffer_size				= 32M
max_allowed_packet 			= 8M
net_buffer_length 			= 16K
myisam_sort_buffer_size 	= 8M
lc_messages_dir			= /usr/share/mysql
#Set this to your desired error message language
lc_messages			= en_US
 
max_connections = 50
 
# security:
# using "localhost" in connects uses sockets by default
# skip-networking
bind-address				= 127.0.0.1
 
server-id 					= 1
 
# point the following paths to different dedicated disks
tmpdir 						= /tmp/
#log-update 				= /path-to-dedicated-directory/hostname
 
join_buffer_size = 2047K
sort_buffer_size = 244K
thread_cache_size = 16
read_buffer_size = 244K
read_rnd_buffer_size = 244K
 
query_cache_limit=1M
query_cache_size=32M
 
tmp_table_size=64M
max_heap_table_size=64M
 
table_cache=256
table_open_cache=400
 
 
slow_query_log = On
slow_query_log_file = /var/log/mysql/zabbix-slow.log
# log_queries_not_using_indexes = On
 
# you need the debug USE flag enabled to use the following directives,
# if needed, uncomment them, start the server and issue
# #tail -f /tmp/mysqld.sql /tmp/mysqld.trace
# this will show you *exactly* what's happening in your server ;)
 
#log						= /tmp/mysqld.sql
#gdb
#debug						= d:t:i:o,/tmp/mysqld.trace
#one-thread
 
# the rest of the innodb config follows:
# don't eat too much memory, we're trying to be safe on 64Mb boxes
# you might want to bump this up a bit on boxes with more RAM
innodb_buffer_pool_size = 2G
#
# i'd like to use /var/lib/mysql/innodb, but that is seen as a database :-(
# and upstream wants things to be under /var/lib/mysql/, so that's the route
# we have to take for the moment
#innodb_data_home_dir		= /var/lib/mysql/
#innodb_log_arch_dir		= /var/lib/mysql/
#innodb_log_group_home_dir	= /var/lib/mysql/
# you may wish to change this size to be more suitable for your system
# the max is there to avoid run-away growth on your machine
innodb_data_file_path = ibdata1:10M:autoextend
# we keep this at around 25% of of innodb_buffer_pool_size
# sensible values range from 1MB to (1/innodb_log_files_in_group*innodb_buffer_pool_size)
innodb_log_file_size = 256M
# this is the default, increase it if you have very large transactions going on
innodb_log_buffer_size = 8M
# this is the default and won't hurt you
# you shouldn't need to tweak it
innodb_log_files_in_group=2
# see the innodb config docs, the other options are not always safe
innodb_flush_log_at_trx_commit = 2
innodb_lock_wait_timeout = 50
innodb_file_per_table
 
innodb_flush_method=O_DIRECT
 
# Uncomment this to get FEDERATED engine support
#plugin-load=federated=ha_federated.so
#loose-federated
 
[mysqldump]
quick
max_allowed_packet 			= 16M
 
[mysql]
# uncomment the next directive if you are not familiar with SQL
#safe-updates
 
[isamchk]
key_buffer_size				= 20M
sort_buffer_size 			= 20M
read_buffer 				= 2M
write_buffer 				= 2M
 
[myisamchk]
key_buffer_size				= 20M
sort_buffer_size 			= 20M
read_buffer_size			= 2M
write_buffer_size			= 2M
 
[mysqlhotcopy]
interactive-timeout
 
[mariadb]

thanks and best regards

Comment by Elena Stepanova [ 2015-11-24 ]

And for your upload question. I guess i could do that. Can i have login credentials?

You don't need credentials to upload to ftp.askmonty.org/private , everyone can do that. Nobody can download from that section.

Comment by Thomas Stein [ 2015-11-24 ]

Hello.

I guess before i upload the datadir i'll try the make a debug enabled build. It's a simple useflag in gentoo. Could you please give me a hint what to do afterwards?

thanks and cheers
t.

Comment by Elena Stepanova [ 2015-11-24 ]

himpierre,
Please get the coredump created, collect all threads' stack trace from the debug build and send it (both "thread apply all bt" and "thread apply all bt full"), and store the coredump together with the mysqld binary it was created by, so we could open it later if needed.

Comment by Thomas Stein [ 2015-11-24 ]

collect all threads' stack trace from the debug build and send it

Forgive me asking but how do i do that?

best regards

Comment by Elena Stepanova [ 2015-11-24 ]

himpierre, see https://mariadb.com/kb/en/mariadb/how-to-produce-a-full-stack-trace-for-mysqld/
The important part is to produce the coredump and store it together with mysqld binary; after it's done, if something is unclear in instructions above, we'll get you through them.

Comment by Thomas Stein [ 2015-11-24 ]

Never mind. I take a look at:

https://mariadb.com/kb/en/mariadb/how-to-produce-a-full-stack-trace-for-mysqld/

Comment by Thomas Stein [ 2015-11-24 ]

Hello Elena.

Now i have a almost 3G core file. I guess i am not done, right?

 mysql # ls -la core 
-rw------- 1 mysql mysql 2796331008 Nov 24 15:13 core
 mysql #

Comment by Thomas Stein [ 2015-11-24 ]

mysql # gdb /usr/sbin/mysqld core 
GNU gdb (Gentoo 7.10 vanilla) 7.10
Copyright (C) 2015 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-pc-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<https://bugs.gentoo.org/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from /usr/sbin/mysqld...(no debugging symbols found)...done.
 
warning: exec file is newer than core file.
[New LWP 20882]
[New LWP 20852]
[New LWP 20853]
[New LWP 20854]
[New LWP 20855]
[New LWP 20856]
[New LWP 20857]
[New LWP 20848]
[New LWP 20858]
[New LWP 20859]
[New LWP 20878]
[New LWP 20879]
[New LWP 20860]
[New LWP 20861]
[New LWP 20862]
[New LWP 20863]
[New LWP 20881]
[New LWP 20880]
 
warning: Could not load shared library symbols for linux-vdso.so.1.
Do you need "set solib-search-path" or "set sysroot"?
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
Core was generated by `/usr/sbin/mysqld --defaults-file=/etc/mysql/my.cnf'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0  0x00007f99b81d89cc in pthread_kill () from /lib64/libpthread.so.0
[Current thread is 1 (Thread 0x7f990f7fc700 (LWP 20882))]
(gdb) backtrace full
#0  0x00007f99b81d89cc in pthread_kill () from /lib64/libpthread.so.0
No symbol table info available.
#1  0x0000561fa5513d9d in handle_fatal_signal ()
No symbol table info available.
#2  <signal handler called>
No symbol table info available.
#3  0x0000561fa57a0d9a in ?? ()
No symbol table info available.
#4  0x0000561fa577549a in ?? ()
No symbol table info available.
#5  0x0000561fa57435e0 in ?? ()
No symbol table info available.
#6  0x0000561fa579e156 in ?? ()
No symbol table info available.
#7  0x0000561fa578eb07 in ?? ()
No symbol table info available.
#8  0x00007f99b81d2314 in start_thread () from /lib64/libpthread.so.0
No symbol table info available.
#9  0x00007f99b704b51d in clone () from /lib64/libc.so.6
No symbol table info available.
(gdb) 

(gdb) thread apply all bt
 
Thread 18 (Thread 0x7f99107fe700 (LWP 20880)):
#0  0x00007f99b81d7c98 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x0000561fa5729153 in ?? ()
#2  0x0000561fa578c01c in ?? ()
#3  0x00007f99b81d2314 in start_thread () from /lib64/libpthread.so.0
#4  0x00007f99b704b51d in clone () from /lib64/libc.so.6
 
Thread 17 (Thread 0x7f990fffd700 (LWP 20881)):
#0  0x00007f99b7044453 in select () from /lib64/libc.so.6
#1  0x0000561fa5729d0b in ?? ()
#2  0x0000561fa578da03 in ?? ()
#3  0x00007f99b81d2314 in start_thread () from /lib64/libpthread.so.0
#4  0x00007f99b704b51d in clone () from /lib64/libc.so.6
 
Thread 16 (Thread 0x7f9916bf6700 (LWP 20863)):
#0  0x00007f99b8ffb6fa in ?? () from /lib64/libaio.so.1
#1  0x0000561fa5728133 in ?? ()
#2  0x0000561fa5840ed8 in ?? ()
#3  0x0000561fa5790198 in ?? ()
#4  0x00007f99b81d2314 in start_thread () from /lib64/libpthread.so.0
#5  0x00007f99b704b51d in clone () from /lib64/libc.so.6
 
Thread 15 (Thread 0x7f99173f7700 (LWP 20862)):
#0  0x00007f99b8ffb6fa in ?? () from /lib64/libaio.so.1
#1  0x0000561fa5728133 in ?? ()
#2  0x0000561fa5840ed8 in ?? ()
#3  0x0000561fa5790198 in ?? ()
#4  0x00007f99b81d2314 in start_thread () from /lib64/libpthread.so.0
#5  0x00007f99b704b51d in clone () from /lib64/libc.so.6
 
Thread 14 (Thread 0x7f9917bf8700 (LWP 20861)):
#0  0x00007f99b8ffb6fa in ?? () from /lib64/libaio.so.1
#1  0x0000561fa5728133 in ?? ()
#2  0x0000561fa5840ed8 in ?? ()
#3  0x0000561fa5790198 in ?? ()
#4  0x00007f99b81d2314 in start_thread () from /lib64/libpthread.so.0
#5  0x00007f99b704b51d in clone () from /lib64/libc.so.6
 
Thread 13 (Thread 0x7f99183f9700 (LWP 20860)):
---Type <return> to continue, or q <return> to quit---
#0  0x00007f99b8ffb6fa in ?? () from /lib64/libaio.so.1
#1  0x0000561fa5728133 in ?? ()
#2  0x0000561fa5840ed8 in ?? ()
#3  0x0000561fa5790198 in ?? ()
#4  0x00007f99b81d2314 in start_thread () from /lib64/libpthread.so.0
#5  0x00007f99b704b51d in clone () from /lib64/libc.so.6
 
Thread 12 (Thread 0x7f9910fff700 (LWP 20879)):
#0  0x00007f99b81d7c98 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x0000561fa5729153 in ?? ()
#2  0x0000561fa578c62f in ?? ()
#3  0x00007f99b81d2314 in start_thread () from /lib64/libpthread.so.0
#4  0x00007f99b704b51d in clone () from /lib64/libc.so.6
 
Thread 11 (Thread 0x7f9912fff700 (LWP 20878)):
#0  0x00007f99b81d7c98 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x0000561fa5729153 in ?? ()
#2  0x0000561fa5702715 in ?? ()
#3  0x00007f99b81d2314 in start_thread () from /lib64/libpthread.so.0
#4  0x00007f99b704b51d in clone () from /lib64/libc.so.6
 
Thread 10 (Thread 0x7f9918bfa700 (LWP 20859)):
#0  0x00007f99b8ffb6fa in ?? () from /lib64/libaio.so.1
#1  0x0000561fa5728133 in ?? ()
#2  0x0000561fa5840ed8 in ?? ()
#3  0x0000561fa5790198 in ?? ()
#4  0x00007f99b81d2314 in start_thread () from /lib64/libpthread.so.0
#5  0x00007f99b704b51d in clone () from /lib64/libc.so.6
 
Thread 9 (Thread 0x7f99193fb700 (LWP 20858)):
#0  0x00007f99b8ffb6fa in ?? () from /lib64/libaio.so.1
#1  0x0000561fa5728133 in ?? ()
#2  0x0000561fa5840ed8 in ?? ()
#3  0x0000561fa5790198 in ?? ()
#4  0x00007f99b81d2314 in start_thread () from /lib64/libpthread.so.0
#5  0x00007f99b704b51d in clone () from /lib64/libc.so.6
 
Thread 8 (Thread 0x7f99b983e780 (LWP 20848)):
#0  0x00007f99b7044453 in select () from /lib64/libc.so.6
#1  0x0000561fa5729d0b in ?? ()
---Type <return> to continue, or q <return> to quit---
#2  0x0000561fa57933e4 in ?? ()
#3  0x0000561fa56d1cff in ?? ()
#4  0x0000561fa551615d in ha_initialize_handlerton(st_plugin_int*) ()
#5  0x0000561fa53d2f9a in ?? ()
#6  0x0000561fa53d3fca in plugin_init(int*, char**, int) ()
#7  0x0000561fa534426d in ?? ()
#8  0x0000561fa5349150 in mysqld_main(int, char**) ()
#9  0x00007f99b6f82fa0 in __libc_start_main () from /lib64/libc.so.6
#10 0x0000561fa533dc18 in _start ()
 
Thread 7 (Thread 0x7f9919bfc700 (LWP 20857)):
#0  0x00007f99b8ffb6fa in ?? () from /lib64/libaio.so.1
#1  0x0000561fa5728133 in ?? ()
#2  0x0000561fa5840ed8 in ?? ()
#3  0x0000561fa5790198 in ?? ()
#4  0x00007f99b81d2314 in start_thread () from /lib64/libpthread.so.0
#5  0x00007f99b704b51d in clone () from /lib64/libc.so.6
 
Thread 6 (Thread 0x7f991a3fd700 (LWP 20856)):
#0  0x00007f99b8ffb6fa in ?? () from /lib64/libaio.so.1
#1  0x0000561fa5728133 in ?? ()
#2  0x0000561fa5840ed8 in ?? ()
#3  0x0000561fa5790198 in ?? ()
#4  0x00007f99b81d2314 in start_thread () from /lib64/libpthread.so.0
#5  0x00007f99b704b51d in clone () from /lib64/libc.so.6
 
Thread 5 (Thread 0x7f991abfe700 (LWP 20855)):
#0  0x00007f99b8ffb6fa in ?? () from /lib64/libaio.so.1
#1  0x0000561fa5728133 in ?? ()
#2  0x0000561fa5840ed8 in ?? ()
#3  0x0000561fa5790198 in ?? ()
#4  0x00007f99b81d2314 in start_thread () from /lib64/libpthread.so.0
#5  0x00007f99b704b51d in clone () from /lib64/libc.so.6
 
Thread 4 (Thread 0x7f991b3ff700 (LWP 20854)):
#0  0x00007f99b8ffb6fa in ?? () from /lib64/libaio.so.1
#1  0x0000561fa5728133 in ?? ()
#2  0x0000561fa5840ed8 in ?? ()
#3  0x0000561fa5790198 in ?? ()
#4  0x00007f99b81d2314 in start_thread () from /lib64/libpthread.so.0
---Type <return> to continue, or q <return> to quit---
#5  0x00007f99b704b51d in clone () from /lib64/libc.so.6
 
Thread 3 (Thread 0x7f99a7bff700 (LWP 20853)):
#0  0x00007f99b81d7c98 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x0000561fa56621ae in ?? ()
#2  0x0000561fa565b9c7 in ?? ()
#3  0x00007f99b81d2314 in start_thread () from /lib64/libpthread.so.0
#4  0x00007f99b704b51d in clone () from /lib64/libc.so.6
 
Thread 2 (Thread 0x7f99b37ff700 (LWP 20852)):
#0  0x00007f99b81d7c98 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x0000561fa5934c16 in ?? ()
#2  0x00007f99b81d2314 in start_thread () from /lib64/libpthread.so.0
#3  0x00007f99b704b51d in clone () from /lib64/libc.so.6
 
Thread 1 (Thread 0x7f990f7fc700 (LWP 20882)):
#0  0x00007f99b81d89cc in pthread_kill () from /lib64/libpthread.so.0
#1  0x0000561fa5513d9d in handle_fatal_signal ()
#2  <signal handler called>
#3  0x0000561fa57a0d9a in ?? ()
#4  0x0000561fa577549a in ?? ()
#5  0x0000561fa57435e0 in ?? ()
#6  0x0000561fa579e156 in ?? ()
#7  0x0000561fa578eb07 in ?? ()
#8  0x00007f99b81d2314 in start_thread () from /lib64/libpthread.so.0
#9  0x00007f99b704b51d in clone () from /lib64/libc.so.6
(gdb) 
 
 
 
{code:java}
(gdb) thread apply all bt full
 
Thread 18 (Thread 0x7f99107fe700 (LWP 20880)):
#0  0x00007f99b81d7c98 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
No symbol table info available.
#1  0x0000561fa5729153 in ?? ()
No symbol table info available.
#2  0x0000561fa578c01c in ?? ()
No symbol table info available.
#3  0x00007f99b81d2314 in start_thread () from /lib64/libpthread.so.0
No symbol table info available.
#4  0x00007f99b704b51d in clone () from /lib64/libc.so.6
No symbol table info available.
 
Thread 17 (Thread 0x7f990fffd700 (LWP 20881)):
#0  0x00007f99b7044453 in select () from /lib64/libc.so.6
No symbol table info available.
#1  0x0000561fa5729d0b in ?? ()
No symbol table info available.
#2  0x0000561fa578da03 in ?? ()
No symbol table info available.
#3  0x00007f99b81d2314 in start_thread () from /lib64/libpthread.so.0
No symbol table info available.
#4  0x00007f99b704b51d in clone () from /lib64/libc.so.6
No symbol table info available.
 
Thread 16 (Thread 0x7f9916bf6700 (LWP 20863)):
#0  0x00007f99b8ffb6fa in ?? () from /lib64/libaio.so.1
No symbol table info available.
#1  0x0000561fa5728133 in ?? ()
No symbol table info available.
#2  0x0000561fa5840ed8 in ?? ()
No symbol table info available.
#3  0x0000561fa5790198 in ?? ()
No symbol table info available.
#4  0x00007f99b81d2314 in start_thread () from /lib64/libpthread.so.0
No symbol table info available.
#5  0x00007f99b704b51d in clone () from /lib64/libc.so.6
No symbol table info available.
 
Thread 15 (Thread 0x7f99173f7700 (LWP 20862)):
---Type <return> to continue, or q <return> to quit---
#0  0x00007f99b8ffb6fa in ?? () from /lib64/libaio.so.1
No symbol table info available.
#1  0x0000561fa5728133 in ?? ()
No symbol table info available.
#2  0x0000561fa5840ed8 in ?? ()
No symbol table info available.
#3  0x0000561fa5790198 in ?? ()
No symbol table info available.
#4  0x00007f99b81d2314 in start_thread () from /lib64/libpthread.so.0
No symbol table info available.
#5  0x00007f99b704b51d in clone () from /lib64/libc.so.6
No symbol table info available.
 
Thread 14 (Thread 0x7f9917bf8700 (LWP 20861)):
#0  0x00007f99b8ffb6fa in ?? () from /lib64/libaio.so.1
No symbol table info available.
#1  0x0000561fa5728133 in ?? ()
No symbol table info available.
#2  0x0000561fa5840ed8 in ?? ()
No symbol table info available.
#3  0x0000561fa5790198 in ?? ()
No symbol table info available.
#4  0x00007f99b81d2314 in start_thread () from /lib64/libpthread.so.0
No symbol table info available.
#5  0x00007f99b704b51d in clone () from /lib64/libc.so.6
No symbol table info available.
 
Thread 13 (Thread 0x7f99183f9700 (LWP 20860)):
#0  0x00007f99b8ffb6fa in ?? () from /lib64/libaio.so.1
No symbol table info available.
#1  0x0000561fa5728133 in ?? ()
No symbol table info available.
#2  0x0000561fa5840ed8 in ?? ()
No symbol table info available.
#3  0x0000561fa5790198 in ?? ()
No symbol table info available.
#4  0x00007f99b81d2314 in start_thread () from /lib64/libpthread.so.0
No symbol table info available.
#5  0x00007f99b704b51d in clone () from /lib64/libc.so.6
No symbol table info available.
---Type <return> to continue, or q <return> to quit---
 
Thread 12 (Thread 0x7f9910fff700 (LWP 20879)):
#0  0x00007f99b81d7c98 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
No symbol table info available.
#1  0x0000561fa5729153 in ?? ()
No symbol table info available.
#2  0x0000561fa578c62f in ?? ()
No symbol table info available.
#3  0x00007f99b81d2314 in start_thread () from /lib64/libpthread.so.0
No symbol table info available.
#4  0x00007f99b704b51d in clone () from /lib64/libc.so.6
No symbol table info available.
 
Thread 11 (Thread 0x7f9912fff700 (LWP 20878)):
#0  0x00007f99b81d7c98 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
No symbol table info available.
#1  0x0000561fa5729153 in ?? ()
No symbol table info available.
#2  0x0000561fa5702715 in ?? ()
No symbol table info available.
#3  0x00007f99b81d2314 in start_thread () from /lib64/libpthread.so.0
No symbol table info available.
#4  0x00007f99b704b51d in clone () from /lib64/libc.so.6
No symbol table info available.
 
Thread 10 (Thread 0x7f9918bfa700 (LWP 20859)):
#0  0x00007f99b8ffb6fa in ?? () from /lib64/libaio.so.1
No symbol table info available.
#1  0x0000561fa5728133 in ?? ()
No symbol table info available.
#2  0x0000561fa5840ed8 in ?? ()
No symbol table info available.
#3  0x0000561fa5790198 in ?? ()
No symbol table info available.
#4  0x00007f99b81d2314 in start_thread () from /lib64/libpthread.so.0
No symbol table info available.
#5  0x00007f99b704b51d in clone () from /lib64/libc.so.6
No symbol table info available.
 
Thread 9 (Thread 0x7f99193fb700 (LWP 20858)):
---Type <return> to continue, or q <return> to quit---
#0  0x00007f99b8ffb6fa in ?? () from /lib64/libaio.so.1
No symbol table info available.
#1  0x0000561fa5728133 in ?? ()
No symbol table info available.
#2  0x0000561fa5840ed8 in ?? ()
No symbol table info available.
#3  0x0000561fa5790198 in ?? ()
No symbol table info available.
#4  0x00007f99b81d2314 in start_thread () from /lib64/libpthread.so.0
No symbol table info available.
#5  0x00007f99b704b51d in clone () from /lib64/libc.so.6
No symbol table info available.
 
Thread 8 (Thread 0x7f99b983e780 (LWP 20848)):
#0  0x00007f99b7044453 in select () from /lib64/libc.so.6
No symbol table info available.
#1  0x0000561fa5729d0b in ?? ()
No symbol table info available.
#2  0x0000561fa57933e4 in ?? ()
No symbol table info available.
#3  0x0000561fa56d1cff in ?? ()
No symbol table info available.
#4  0x0000561fa551615d in ha_initialize_handlerton(st_plugin_int*) ()
No symbol table info available.
#5  0x0000561fa53d2f9a in ?? ()
No symbol table info available.
#6  0x0000561fa53d3fca in plugin_init(int*, char**, int) ()
No symbol table info available.
#7  0x0000561fa534426d in ?? ()
No symbol table info available.
#8  0x0000561fa5349150 in mysqld_main(int, char**) ()
No symbol table info available.
#9  0x00007f99b6f82fa0 in __libc_start_main () from /lib64/libc.so.6
No symbol table info available.
#10 0x0000561fa533dc18 in _start ()
No symbol table info available.
 
Thread 7 (Thread 0x7f9919bfc700 (LWP 20857)):
#0  0x00007f99b8ffb6fa in ?? () from /lib64/libaio.so.1
No symbol table info available.
---Type <return> to continue, or q <return> to quit---
#1  0x0000561fa5728133 in ?? ()
No symbol table info available.
#2  0x0000561fa5840ed8 in ?? ()
No symbol table info available.
#3  0x0000561fa5790198 in ?? ()
No symbol table info available.
#4  0x00007f99b81d2314 in start_thread () from /lib64/libpthread.so.0
No symbol table info available.
#5  0x00007f99b704b51d in clone () from /lib64/libc.so.6
No symbol table info available.
 
Thread 6 (Thread 0x7f991a3fd700 (LWP 20856)):
#0  0x00007f99b8ffb6fa in ?? () from /lib64/libaio.so.1
No symbol table info available.
#1  0x0000561fa5728133 in ?? ()
No symbol table info available.
#2  0x0000561fa5840ed8 in ?? ()
No symbol table info available.
#3  0x0000561fa5790198 in ?? ()
No symbol table info available.
#4  0x00007f99b81d2314 in start_thread () from /lib64/libpthread.so.0
No symbol table info available.
#5  0x00007f99b704b51d in clone () from /lib64/libc.so.6
No symbol table info available.
 
Thread 5 (Thread 0x7f991abfe700 (LWP 20855)):
#0  0x00007f99b8ffb6fa in ?? () from /lib64/libaio.so.1
No symbol table info available.
#1  0x0000561fa5728133 in ?? ()
No symbol table info available.
#2  0x0000561fa5840ed8 in ?? ()
No symbol table info available.
#3  0x0000561fa5790198 in ?? ()
No symbol table info available.
#4  0x00007f99b81d2314 in start_thread () from /lib64/libpthread.so.0
No symbol table info available.
#5  0x00007f99b704b51d in clone () from /lib64/libc.so.6
No symbol table info available.
 
Thread 4 (Thread 0x7f991b3ff700 (LWP 20854)):
---Type <return> to continue, or q <return> to quit---
#0  0x00007f99b8ffb6fa in ?? () from /lib64/libaio.so.1
No symbol table info available.
#1  0x0000561fa5728133 in ?? ()
No symbol table info available.
#2  0x0000561fa5840ed8 in ?? ()
No symbol table info available.
#3  0x0000561fa5790198 in ?? ()
No symbol table info available.
#4  0x00007f99b81d2314 in start_thread () from /lib64/libpthread.so.0
No symbol table info available.
#5  0x00007f99b704b51d in clone () from /lib64/libc.so.6
No symbol table info available.
 
Thread 3 (Thread 0x7f99a7bff700 (LWP 20853)):
#0  0x00007f99b81d7c98 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
No symbol table info available.
#1  0x0000561fa56621ae in ?? ()
No symbol table info available.
#2  0x0000561fa565b9c7 in ?? ()
No symbol table info available.
#3  0x00007f99b81d2314 in start_thread () from /lib64/libpthread.so.0
No symbol table info available.
#4  0x00007f99b704b51d in clone () from /lib64/libc.so.6
No symbol table info available.
 
Thread 2 (Thread 0x7f99b37ff700 (LWP 20852)):
#0  0x00007f99b81d7c98 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
No symbol table info available.
#1  0x0000561fa5934c16 in ?? ()
No symbol table info available.
#2  0x00007f99b81d2314 in start_thread () from /lib64/libpthread.so.0
No symbol table info available.
#3  0x00007f99b704b51d in clone () from /lib64/libc.so.6
No symbol table info available.
 
Thread 1 (Thread 0x7f990f7fc700 (LWP 20882)):
#0  0x00007f99b81d89cc in pthread_kill () from /lib64/libpthread.so.0
No symbol table info available.
#1  0x0000561fa5513d9d in handle_fatal_signal ()
No symbol table info available.
---Type <return> to continue, or q <return> to quit---
#2  <signal handler called>
No symbol table info available.
#3  0x0000561fa57a0d9a in ?? ()
No symbol table info available.
#4  0x0000561fa577549a in ?? ()
No symbol table info available.
#5  0x0000561fa57435e0 in ?? ()
No symbol table info available.
#6  0x0000561fa579e156 in ?? ()
No symbol table info available.
#7  0x0000561fa578eb07 in ?? ()
No symbol table info available.
#8  0x00007f99b81d2314 in start_thread () from /lib64/libpthread.so.0
No symbol table info available.
#9  0x00007f99b704b51d in clone () from /lib64/libc.so.6
No symbol table info available.
(gdb) 

 

Comment by Jan Lindström (Inactive) [ 2015-11-24 ]

Can you resolve the stack trace or open that core using ddd and do bt apply all bt

Comment by Jan Lindström (Inactive) [ 2015-11-24 ]

Thanks, does not much help, can you use debug build ?

Comment by Thomas Stein [ 2015-11-24 ]

Can i upload the core file? It was created with a debug enabled 10.1.9 mariadb binary.

thanks and best regards
t.

Comment by Jan Lindström (Inactive) [ 2015-11-24 ]

If you can just send thread apply bt all from that core with debug 10.1.9 it would be enough, above is from product build.

Comment by Thomas Stein [ 2015-11-24 ]

Okay. I'll try that tomorrow.

Comment by Thomas Stein [ 2015-11-24 ]

Uhh. Did you delete all the other comments? Just compiled mariadb with DCMAKE_BUILD_TYPE=Debug but i get nothing out of:

gdb /usr/sbin/mysqld core 
GNU gdb (Gentoo 7.10 vanilla) 7.10
Copyright (C) 2015 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-pc-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<https://bugs.gentoo.org/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from /usr/sbin/mysqld...(no debugging symbols found)...done.
 
warning: exec file is newer than core file.
[New LWP 20882]
[New LWP 20852]
[New LWP 20853]
[New LWP 20854]
[New LWP 20855]
[New LWP 20856]
[New LWP 20857]
[New LWP 20848]
[New LWP 20858]
[New LWP 20859]
[New LWP 20878]
[New LWP 20879]
[New LWP 20860]
[New LWP 20861]
[New LWP 20862]
[New LWP 20863]
[New LWP 20881]
[New LWP 20880]
Core was generated by `/usr/sbin/mysqld --defaults-file=/etc/mysql/my.cnf'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0  0x00007f99b81d89cc in ?? ()
[Current thread is 1 (LWP 20882)]

Comment by Thomas Stein [ 2015-11-24 ]

(gdb) thread apply bt all
(gdb) backtrace full
#0 0x00007f99b81d89cc in ?? ()
No symbol table info available.
#1 0x0000561fa5513d9d in ?? ()
No symbol table info available.
#2 0x00007f991106d180 in ?? ()
No symbol table info available.
#3 0x0000000b1106d180 in ?? ()
No symbol table info available.
#4 0x0000000000000020 in ?? ()
No symbol table info available.
#5 0x0000000056547093 in ?? ()
No symbol table info available.
#6 0x0000000d00000027 in ?? ()
No symbol table info available.
#7 0x000000180000000f in ?? ()
No symbol table info available.
#8 0x000000730000000a in ?? ()
No symbol table info available.
#9 0x0000014700000002 in ?? ()
No symbol table info available.
#10 0x00007f9900000000 in ?? ()
No symbol table info available.
#11 0x0000000000000e10 in ?? ()
No symbol table info available.
#12 0x00007f99b6435050 in ?? ()
No symbol table info available.
#13 0x0000000000000000 in ?? ()
No symbol table info available.
(gdb)

Comment by Jan Lindström (Inactive) [ 2015-11-24 ]

You can't open core with that build, try running the actual server, if it produces a new core then you can open it with this build.

Comment by pierre C [ 2015-11-25 ]

Hi,
i upgraded this morning, as well, and i got a similar error.
I'm using Debian GNU/Linux 7.9 (wheezy)
Here is the syslog related to the update :

Nov 25 09:47:33 demeter mysqld: 2015-11-25  9:47:33 140236347524864 [Note] /usr/sbin/mysqld: Normal shutdown
Nov 25 09:47:33 demeter mysqld:
Nov 25 09:47:33 demeter mysqld: 2015-11-25  9:47:33 140236347524864 [Note] Event Scheduler: Purging the queue. 0 events
Nov 25 09:47:33 demeter mysqld: 2015-11-25  9:47:33 140235430127360 [Note] InnoDB: FTS optimize thread exiting.
Nov 25 09:47:33 demeter mysqld: 2015-11-25  9:47:33 140236347524864 [Note] InnoDB: Starting shutdown...
Nov 25 09:47:35 demeter mysqld: 2015-11-25  9:47:35 140236347524864 [Note] InnoDB: Shutdown completed; log sequence number 357910866
Nov 25 09:47:35 demeter mysqld: 2015-11-25  9:47:35 140236347524864 [Note] /usr/sbin/mysqld: Shutdown complete
Nov 25 09:47:35 demeter mysqld:
Nov 25 09:47:35 demeter mysqld_safe: mysqld from pid file /var/run/mysqld/mysqld.pid ended
Nov 25 09:47:38 demeter mysqld_safe[9542]: 2015-11-25  9:47:38 139864863786848 [Note] /usr/sbin/mysqld (mysqld 10.1.9-MariaDB-1~wheezy) starting as process 9541 ...
Nov 25 09:47:38 demeter mysqld_safe[9542]: 2015-11-25  9:47:38 139864863786848 [Note] Using unique option prefix 'myisam_recover' is error-prone and can break in the future. Please use the full name 'myisam-recover-options' instead.
Nov 25 09:47:38 demeter mysqld_safe[9542]: 2015-11-25  9:47:38 139864863786848 [Note] InnoDB: Using mutexes to ref count buffer pool pages
Nov 25 09:47:38 demeter mysqld_safe[9542]: 2015-11-25  9:47:38 139864863786848 [Note] InnoDB: The InnoDB memory heap is disabled
Nov 25 09:47:38 demeter mysqld_safe[9542]: 2015-11-25  9:47:38 139864863786848 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
Nov 25 09:47:38 demeter mysqld_safe[9542]: 2015-11-25  9:47:38 139864863786848 [Note] InnoDB: Memory barrier is not used
Nov 25 09:47:38 demeter mysqld_safe[9542]: 2015-11-25  9:47:38 139864863786848 [Note] InnoDB: Compressed tables use zlib 1.2.7
Nov 25 09:47:38 demeter mysqld_safe[9542]: 2015-11-25  9:47:38 139864863786848 [Note] InnoDB: Using Linux native AIO
Nov 25 09:47:38 demeter mysqld_safe[9542]: 2015-11-25  9:47:38 139864863786848 [Note] InnoDB: Not using CPU crc32 instructions
Nov 25 09:47:38 demeter mysqld_safe[9542]: 2015-11-25  9:47:38 139864863786848 [Note] InnoDB: Initializing buffer pool, size = 256.0M
Nov 25 09:47:38 demeter mysqld_safe[9542]: 2015-11-25  9:47:38 139864863786848 [Note] InnoDB: Completed initialization of buffer pool
Nov 25 09:47:38 demeter mysqld_safe[9542]: 2015-11-25  9:47:38 139864863786848 [Note] InnoDB: Highest supported file format is Barracuda.
Nov 25 09:47:38 demeter mysqld_safe[9542]: 2015-11-25  9:47:38 139864863786848 [Note] InnoDB: 128 rollback segment(s) are active.
Nov 25 09:47:38 demeter mysqld_safe[9542]: 2015-11-25  9:47:38 139864863786848 [Note] InnoDB: Waiting for purge to start
Nov 25 09:47:38 demeter mysqld_safe[9542]: 151125  9:47:38 [ERROR] mysqld got signal 11 ;
Nov 25 09:47:38 demeter mysqld_safe[9542]: This could be because you hit a bug. It is also possible that this binary
Nov 25 09:47:38 demeter mysqld_safe[9542]: or one of the libraries it was linked against is corrupt, improperly built,
Nov 25 09:47:38 demeter mysqld_safe[9542]: or misconfigured. This error can also be caused by malfunctioning hardware.
Nov 25 09:47:38 demeter mysqld_safe[9542]:
Nov 25 09:47:38 demeter mysqld_safe[9542]: To report this bug, see http://kb.askmonty.org/en/reporting-bugs
Nov 25 09:47:38 demeter mysqld_safe[9542]:
Nov 25 09:47:38 demeter mysqld_safe[9542]: We will try our best to scrape up some info that will hopefully help
Nov 25 09:47:38 demeter mysqld_safe[9542]: diagnose the problem, but since we have already crashed,
Nov 25 09:47:38 demeter mysqld_safe[9542]: something is definitely wrong and this may fail.
Nov 25 09:47:38 demeter mysqld_safe[9542]:
Nov 25 09:47:38 demeter mysqld_safe[9542]: Server version: 10.1.9-MariaDB-1~wheezy
Nov 25 09:47:38 demeter mysqld_safe[9542]: key_buffer_size=134217728
Nov 25 09:47:38 demeter mysqld_safe[9542]: read_buffer_size=2097152
Nov 25 09:47:38 demeter mysqld_safe[9542]: max_used_connections=0
Nov 25 09:47:38 demeter mysqld_safe[9542]: max_threads=102
Nov 25 09:47:38 demeter mysqld_safe[9542]: thread_count=0
Nov 25 09:47:38 demeter mysqld_safe[9542]: It is possible that mysqld could use up to
Nov 25 09:47:38 demeter mysqld_safe[9542]: key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 759822 K  bytes of memory
Nov 25 09:47:38 demeter mysqld_safe[9542]: Hope that's ok; if not, decrease some variables in the equation.
Nov 25 09:47:38 demeter mysqld_safe[9542]:
Nov 25 09:47:38 demeter mysqld_safe[9542]: Thread pointer: 0x0x0
Nov 25 09:47:38 demeter mysqld_safe[9542]: Attempting backtrace. You can use the following information to find out
Nov 25 09:47:38 demeter mysqld_safe[9542]: where mysqld died. If you see no messages after this, something went
Nov 25 09:47:38 demeter mysqld_safe[9542]: terribly wrong...
Nov 25 09:47:38 demeter mysqld_safe[9542]: stack_bottom = 0x0 thread_stack 0x48000
Nov 25 09:47:38 demeter mysqld_safe[9542]: /usr/sbin/mysqld(my_print_stacktrace+0x2b)[0x7f34d42e61ab]
Nov 25 09:47:38 demeter mysqld_safe[9542]: /usr/sbin/mysqld(handle_fatal_signal+0x475)[0x7f34d3e471e5]
Nov 25 09:47:38 demeter mysqld_safe[9542]: /lib/x86_64-linux-gnu/libpthread.so.0(+0xf0a0)[0x7f34d34400a0]
Nov 25 09:47:38 demeter mysqld_safe[9542]: /usr/sbin/mysqld(+0x85e5b7)[0x7f34d40cd5b7]
Nov 25 09:47:38 demeter mysqld_safe[9542]: /usr/sbin/mysqld(+0x82da7f)[0x7f34d409ca7f]
Nov 25 09:47:38 demeter mysqld_safe[9542]: /usr/sbin/mysqld(+0x7f3f6f)[0x7f34d4062f6f]
Nov 25 09:47:38 demeter mysqld_safe[9542]: /usr/sbin/mysqld(+0x85ba71)[0x7f34d40caa71]
Nov 25 09:47:38 demeter mysqld_safe[9542]: /usr/sbin/mysqld(+0x849445)[0x7f34d40b8445]
Nov 25 09:47:38 demeter mysqld_safe[9542]: /lib/x86_64-linux-gnu/libpthread.so.0(+0x6b50)[0x7f34d3437b50]
Nov 25 09:47:38 demeter mysqld_safe[9542]: /lib/x86_64-linux-gnu/libc.so.6(clone+0x6d)[0x7f34d190295d]
Nov 25 09:47:38 demeter mysqld_safe[9542]: The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
Nov 25 09:47:38 demeter mysqld_safe[9542]: information that should help you find out what is causing the crash.
Nov 25 09:47:38 demeter mysqld_safe: Starting mysqld daemon with databases from /var/lib/mysql
Nov 25 09:47:38 demeter mysqld: 2015-11-25  9:47:38 140616633907040 [Note] /usr/sbin/mysqld (mysqld 10.1.9-MariaDB-1~wheezy-log) starting as process 9821 ...
Nov 25 09:47:38 demeter mysqld: 2015-11-25  9:47:38 140616633907040 [Note] Using unique option prefix 'myisam_recover' is error-prone and can break in the future. Please use the full name 'myisam-recover-options' instead.
Nov 25 09:47:38 demeter mysqld: 2015-11-25  9:47:38 140616633907040 [Note] InnoDB: Using mutexes to ref count buffer pool pages
Nov 25 09:47:38 demeter mysqld: 2015-11-25  9:47:38 140616633907040 [Note] InnoDB: The InnoDB memory heap is disabled
Nov 25 09:47:38 demeter mysqld: 2015-11-25  9:47:38 140616633907040 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
Nov 25 09:47:38 demeter mysqld: 2015-11-25  9:47:38 140616633907040 [Note] InnoDB: Memory barrier is not used
Nov 25 09:47:38 demeter mysqld: 2015-11-25  9:47:38 140616633907040 [Note] InnoDB: Compressed tables use zlib 1.2.7
Nov 25 09:47:38 demeter mysqld: 2015-11-25  9:47:38 140616633907040 [Note] InnoDB: Using Linux native AIO
Nov 25 09:47:38 demeter mysqld: 2015-11-25  9:47:38 140616633907040 [Note] InnoDB: Not using CPU crc32 instructions
Nov 25 09:47:38 demeter mysqld: 2015-11-25  9:47:38 140616633907040 [Note] InnoDB: Initializing buffer pool, size = 256.0M
Nov 25 09:47:38 demeter mysqld: 2015-11-25  9:47:38 140616633907040 [Note] InnoDB: Completed initialization of buffer pool
Nov 25 09:47:38 demeter mysqld: 2015-11-25  9:47:38 140616633907040 [Note] InnoDB: Highest supported file format is Barracuda.
Nov 25 09:47:39 demeter mysqld: 2015-11-25  9:47:39 140616633907040 [Note] InnoDB: 128 rollback segment(s) are active.
Nov 25 09:47:39 demeter mysqld: 2015-11-25  9:47:39 140616633907040 [Note] InnoDB: Waiting for purge to start
Nov 25 09:47:39 demeter mysqld: 151125  9:47:39 [ERROR] mysqld got signal 11 ;
Nov 25 09:47:39 demeter mysqld: This could be because you hit a bug. It is also possible that this binary
Nov 25 09:47:39 demeter mysqld: or one of the libraries it was linked against is corrupt, improperly built,
Nov 25 09:47:39 demeter mysqld: or misconfigured. This error can also be caused by malfunctioning hardware.
Nov 25 09:47:39 demeter mysqld:
Nov 25 09:47:39 demeter mysqld: To report this bug, see http://kb.askmonty.org/en/reporting-bugs
Nov 25 09:47:39 demeter mysqld:
Nov 25 09:47:39 demeter mysqld: We will try our best to scrape up some info that will hopefully help
Nov 25 09:47:39 demeter mysqld: diagnose the problem, but since we have already crashed,
Nov 25 09:47:39 demeter mysqld: something is definitely wrong and this may fail.
Nov 25 09:47:39 demeter mysqld:
Nov 25 09:47:39 demeter mysqld: Server version: 10.1.9-MariaDB-1~wheezy-log
Nov 25 09:47:39 demeter mysqld: key_buffer_size=134217728
Nov 25 09:47:39 demeter mysqld: read_buffer_size=2097152
Nov 25 09:47:39 demeter mysqld: max_used_connections=0
Nov 25 09:47:39 demeter mysqld: max_threads=102
Nov 25 09:47:39 demeter mysqld: thread_count=0
Nov 25 09:47:39 demeter mysqld: It is possible that mysqld could use up to
Nov 25 09:47:39 demeter mysqld: key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 759822 K  bytes of memory
Nov 25 09:47:39 demeter mysqld: Hope that's ok; if not, decrease some variables in the equation.
Nov 25 09:47:39 demeter mysqld:
Nov 25 09:47:39 demeter mysqld: Thread pointer: 0x0x0
Nov 25 09:47:39 demeter mysqld: Attempting backtrace. You can use the following information to find out
Nov 25 09:47:39 demeter mysqld: where mysqld died. If you see no messages after this, something went
Nov 25 09:47:39 demeter mysqld: terribly wrong...
Nov 25 09:47:39 demeter mysqld: stack_bottom = 0x0 thread_stack 0x48000
Nov 25 09:47:39 demeter mysqld: /usr/sbin/mysqld(my_print_stacktrace+0x2b)[0x7fe3dd2c11ab]
Nov 25 09:47:39 demeter mysqld: /usr/sbin/mysqld(handle_fatal_signal+0x475)[0x7fe3dce221e5]
Nov 25 09:47:39 demeter mysqld: /lib/x86_64-linux-gnu/libpthread.so.0(+0xf0a0)[0x7fe3dc41b0a0]
Nov 25 09:47:39 demeter mysqld: /usr/sbin/mysqld(+0x85e5b7)[0x7fe3dd0a85b7]
Nov 25 09:47:39 demeter mysqld: /usr/sbin/mysqld(+0x82da7f)[0x7fe3dd077a7f]
Nov 25 09:47:39 demeter mysqld: /usr/sbin/mysqld(+0x7f3f6f)[0x7fe3dd03df6f]
Nov 25 09:47:39 demeter mysqld: /usr/sbin/mysqld(+0x85ba71)[0x7fe3dd0a5a71]
Nov 25 09:47:39 demeter mysqld: /usr/sbin/mysqld(+0x849445)[0x7fe3dd093445]
Nov 25 09:47:39 demeter mysqld: /lib/x86_64-linux-gnu/libpthread.so.0(+0x6b50)[0x7fe3dc412b50]
Nov 25 09:47:39 demeter mysqld: /lib/x86_64-linux-gnu/libc.so.6(clone+0x6d)[0x7fe3da8dd95d]
Nov 25 09:47:39 demeter mysqld: The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
Nov 25 09:47:39 demeter mysqld: information that should help you find out what is causing the crash.
Nov 25 09:47:39 demeter mysqld_safe: mysqld from pid file /var/run/mysqld/mysqld.pid ended
Nov 25 09:48:09 demeter /etc/init.d/mysql[10153]: 0 processes alive and '/usr/bin/mysqladmin --defaults-file=/etc/mysql/debian.cnf ping' resulted in
Nov 25 09:48:09 demeter /etc/init.d/mysql[10153]: #007/usr/bin/mysqladmin: connect to server at 'localhost' failed
Nov 25 09:48:09 demeter /etc/init.d/mysql[10153]: error: 'Can't connect to local MySQL server through socket '/var/run/mysqld/mysqld.sock' (2 "No such file or directory")'
Nov 25 09:48:09 demeter /etc/init.d/mysql[10153]: Check that mysqld is running and that the socket: '/var/run/mysqld/mysqld.sock' exists!
Nov 25 09:48:09 demeter /etc/init.d/mysql[10153]:

Please let me know how i can help. Meanwhile, i will follow this ticket.
Thanks.
p.

Comment by Jan Lindström (Inactive) [ 2015-11-25 ]

Hi, there are few ways you may help. (1) use debug build and start server and if it crashes attach debugger to core and send me output of thread apply all bt
(2) try your database before you migrated 10.1.8 and see if that helps

R: Jan

Comment by Thomas Stein [ 2015-11-25 ]

Hello Jan.

Here the backtrace.

(gdb) thread apply all bt
 
Thread 24 (Thread 0x7f4806836780 (LWP 18076)):
#0  0x00007f48055015ad in write () from /lib64/libpthread.so.0
#1  0x00005591f1c35aca in my_write ()
#2  0x00005591f1c184d3 in ?? ()
#3  0x00005591f1c1ab51 in my_b_flush_io_cache ()
#4  0x00005591f1794011 in MYSQL_LOG::open(unsigned int, char const*, enum_log_type, char const*, unsigned long, cache_type) ()
#5  0x00005591f1644e44 in MYSQL_QUERY_LOG::open_slow_log(char const*) ()
#6  0x00005591f1794076 in ?? ()
#7  0x00005591f179412a in LOGGER::init_log_tables() ()
#8  0x00005591f14a7307 in ?? ()
#9  0x00005591f14ac8f6 in mysqld_main(int, char**) ()
#10 0x00007f48042b0855 in __libc_start_main () from /lib64/libc.so.6
#11 0x00005591f149f479 in _start ()
 
Thread 23 (Thread 0x7f4755bfa700 (LWP 18104)):
#0  0x00007f480436fd73 in select () from /lib64/libc.so.6
#1  0x00005591f18cfcfb in ?? ()
#2  0x00005591f1a1f43f in ?? ()
#3  0x00007f48054fa4b4 in start_thread () from /lib64/libpthread.so.0
#4  0x00007f4804376cfd in clone () from /lib64/libc.so.6
 
Thread 22 (Thread 0x7f47563fb700 (LWP 18103)):
#0  0x00007f480436fd73 in select () from /lib64/libc.so.6
#1  0x00005591f18cfcfb in ?? ()
#2  0x00005591f1a1e08e in ?? ()
#3  0x00007f48054fa4b4 in start_thread () from /lib64/libpthread.so.0
#4  0x00007f4804376cfd in clone () from /lib64/libc.so.6
 
Thread 21 (Thread 0x7f475a3ff700 (LWP 18097)):
#0  0x00007f48054fed08 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00005591f18cf0d3 in ?? ()
#2  0x00005591f189ee95 in ?? ()
#3  0x00007f48054fa4b4 in start_thread () from /lib64/libpthread.so.0
#4  0x00007f4804376cfd in clone () from /lib64/libc.so.6
 
Thread 20 (Thread 0x7f47583ff700 (LWP 18098)):
#0  0x00007f48054fed08 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00005591f18cf0d3 in ?? ()
#2  0x00005591f1979559 in ?? ()
---Type <return> to continue, or q <return> to quit---
#3  0x00007f48054fa4b4 in start_thread () from /lib64/libpthread.so.0
#4  0x00007f4804376cfd in clone () from /lib64/libc.so.6
 
Thread 19 (Thread 0x7f4800bff700 (LWP 18077)):
#0  0x00007f48054fed08 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00005591f1c3c117 in safe_cond_timedwait ()
#2  0x00005591f1c3c84d in ?? ()
#3  0x00007f48054fa4b4 in start_thread () from /lib64/libpthread.so.0
#4  0x00007f4804376cfd in clone () from /lib64/libc.so.6
 
Thread 18 (Thread 0x7f4757bfe700 (LWP 18099)):
#0  0x00007f48054fed08 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00005591f18cf0d3 in ?? ()
#2  0x00005591f1978f14 in ?? ()
#3  0x00007f48054fa4b4 in start_thread () from /lib64/libpthread.so.0
#4  0x00007f4804376cfd in clone () from /lib64/libc.so.6
 
Thread 17 (Thread 0x7f47f4fff700 (LWP 18078)):
#0  0x00007f48054fed08 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00005591f1c3c117 in safe_cond_timedwait ()
#2  0x00005591f1b08fcb in ?? ()
#3  0x00005591f1b0115f in ?? ()
#4  0x00007f48054fa4b4 in start_thread () from /lib64/libpthread.so.0
#5  0x00007f4804376cfd in clone () from /lib64/libc.so.6
 
Thread 16 (Thread 0x7f47627ff700 (LWP 18079)):
#0  0x00007f4805ff364a in ?? () from /lib64/libaio.so.1
#1  0x00005591f18cdf9b in ?? ()
#2  0x00005591f1a768c1 in ?? ()
#3  0x00005591f197e7f0 in ?? ()
#4  0x00007f48054fa4b4 in start_thread () from /lib64/libpthread.so.0
#5  0x00007f4804376cfd in clone () from /lib64/libc.so.6
 
Thread 15 (Thread 0x7f47573fd700 (LWP 18100)):
#0  0x00007f480436fd73 in select () from /lib64/libc.so.6
#1  0x00005591f18cfcfb in ?? ()
#2  0x00005591f197ae76 in ?? ()
#3  0x00007f48054fa4b4 in start_thread () from /lib64/libpthread.so.0
#4  0x00007f4804376cfd in clone () from /lib64/libc.so.6
 
---Type <return> to continue, or q <return> to quit---
Thread 14 (Thread 0x7f4761ffe700 (LWP 18080)):
#0  0x00007f4805ff364a in ?? () from /lib64/libaio.so.1
#1  0x00005591f18cdf9b in ?? ()
#2  0x00005591f1a768c1 in ?? ()
#3  0x00005591f197e7f0 in ?? ()
#4  0x00007f48054fa4b4 in start_thread () from /lib64/libpthread.so.0
#5  0x00007f4804376cfd in clone () from /lib64/libc.so.6
 
Thread 13 (Thread 0x7f47617fd700 (LWP 18081)):
#0  0x00007f4805ff364a in ?? () from /lib64/libaio.so.1
#1  0x00005591f18cdf9b in ?? ()
#2  0x00005591f1a768c1 in ?? ()
#3  0x00005591f197e7f0 in ?? ()
#4  0x00007f48054fa4b4 in start_thread () from /lib64/libpthread.so.0
#5  0x00007f4804376cfd in clone () from /lib64/libc.so.6
 
Thread 12 (Thread 0x7f47607fb700 (LWP 18083)):
#0  0x00007f4805ff364a in ?? () from /lib64/libaio.so.1
#1  0x00005591f18cdf9b in ?? ()
#2  0x00005591f1a768c1 in ?? ()
#3  0x00005591f197e7f0 in ?? ()
#4  0x00007f48054fa4b4 in start_thread () from /lib64/libpthread.so.0
#5  0x00007f4804376cfd in clone () from /lib64/libc.so.6
 
Thread 11 (Thread 0x7f475fffa700 (LWP 18084)):
#0  0x00007f4805ff364a in ?? () from /lib64/libaio.so.1
#1  0x00005591f18cdf9b in ?? ()
#2  0x00005591f1a768c1 in ?? ()
#3  0x00005591f197e7f0 in ?? ()
#4  0x00007f48054fa4b4 in start_thread () from /lib64/libpthread.so.0
#5  0x00007f4804376cfd in clone () from /lib64/libc.so.6
 
Thread 10 (Thread 0x7f475f7f9700 (LWP 18085)):
#0  0x00007f4805ff364a in ?? () from /lib64/libaio.so.1
#1  0x00005591f18cdf9b in ?? ()
#2  0x00005591f1a768c1 in ?? ()
#3  0x00005591f197e7f0 in ?? ()
#4  0x00007f48054fa4b4 in start_thread () from /lib64/libpthread.so.0
#5  0x00007f4804376cfd in clone () from /lib64/libc.so.6
 
---Type <return> to continue, or q <return> to quit---
Thread 9 (Thread 0x7f475eff8700 (LWP 18086)):
#0  0x00007f4805ff364a in ?? () from /lib64/libaio.so.1
#1  0x00005591f18cdf9b in ?? ()
#2  0x00005591f1a768c1 in ?? ()
#3  0x00005591f197e7f0 in ?? ()
#4  0x00007f48054fa4b4 in start_thread () from /lib64/libpthread.so.0
#5  0x00007f4804376cfd in clone () from /lib64/libc.so.6
 
Thread 8 (Thread 0x7f475e7f7700 (LWP 18087)):
#0  0x00007f4805ff364a in ?? () from /lib64/libaio.so.1
#1  0x00005591f18cdf9b in ?? ()
#2  0x00005591f1a768c1 in ?? ()
#3  0x00005591f197e7f0 in ?? ()
#4  0x00007f48054fa4b4 in start_thread () from /lib64/libpthread.so.0
#5  0x00007f4804376cfd in clone () from /lib64/libc.so.6
 
Thread 7 (Thread 0x7f475dff6700 (LWP 18088)):
#0  0x00007f4805ff364a in ?? () from /lib64/libaio.so.1
#1  0x00005591f18cdf9b in ?? ()
#2  0x00005591f1a768c1 in ?? ()
#3  0x00005591f197e7f0 in ?? ()
#4  0x00007f48054fa4b4 in start_thread () from /lib64/libpthread.so.0
#5  0x00007f4804376cfd in clone () from /lib64/libc.so.6
 
Thread 6 (Thread 0x7f4754bf8700 (LWP 18107)):
#0  0x00007f48054fed08 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00005591f18cf0d3 in ?? ()
#2  0x00005591f1a6b738 in ?? ()
#3  0x00007f48054fa4b4 in start_thread () from /lib64/libpthread.so.0
#4  0x00007f4804376cfd in clone () from /lib64/libc.so.6
 
Thread 5 (Thread 0x7f4753bf6700 (LWP 18109)):
#0  0x00007f480436fd73 in select () from /lib64/libc.so.6
#1  0x00005591f18cfcfb in ?? ()
#2  0x00005591f19febce in ?? ()
#3  0x00007f48054fa4b4 in start_thread () from /lib64/libpthread.so.0
#4  0x00007f4804376cfd in clone () from /lib64/libc.so.6
 
Thread 4 (Thread 0x7f4760ffc700 (LWP 18082)):
#0  0x00007f4805ff364a in ?? () from /lib64/libaio.so.1
---Type <return> to continue, or q <return> to quit---
#1  0x00005591f18cdf9b in ?? ()
#2  0x00005591f1a768c1 in ?? ()
#3  0x00005591f197e7f0 in ?? ()
#4  0x00007f48054fa4b4 in start_thread () from /lib64/libpthread.so.0
#5  0x00007f4804376cfd in clone () from /lib64/libc.so.6
 
Thread 3 (Thread 0x7f47543f7700 (LWP 18108)):
#0  0x00007f48054fed08 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00005591f18cf0d3 in ?? ()
#2  0x00005591f19c073d in ?? ()
#3  0x00005591f1ab4876 in ?? ()
#4  0x00007f48054fa4b4 in start_thread () from /lib64/libpthread.so.0
#5  0x00007f4804376cfd in clone () from /lib64/libc.so.6
 
Thread 2 (Thread 0x7f47553f9700 (LWP 18106)):
#0  0x00007f48054fe95f in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00005591f18cef81 in ?? ()
#2  0x00005591f1a1577b in ?? ()
#3  0x00007f48054fa4b4 in start_thread () from /lib64/libpthread.so.0
#4  0x00007f4804376cfd in clone () from /lib64/libc.so.6
 
Thread 1 (Thread 0x7f4756bfc700 (LWP 18101)):
#0  0x00007f48054ff9b1 in pthread_kill () from /lib64/libpthread.so.0
#1  0x00005591f16d59dd in handle_fatal_signal ()
#2  <signal handler called>
#3  0x00005591f199c65a in ?? ()
#4  0x00005591f194d64c in ?? ()
#5  0x00005591f18fada8 in ?? ()
#6  0x00005591f1995163 in ?? ()
#7  0x00005591f197c3d9 in ?? ()
#8  0x00007f48054fa4b4 in start_thread () from /lib64/libpthread.so.0
#9  0x00007f4804376cfd in clone () from /lib64/libc.so.6
(gdb) 
 
Hope it's usefull this time.
 
cheers
t.

Comment by Jan Lindström (Inactive) [ 2015-11-25 ]

Hi, is this really from debug build? Most of the symbols are not there.

Comment by Thomas Stein [ 2015-11-25 ]

Hi.

I build 10.1.9 with debug and did a "gdb /usr/sbin/mysqld/ core". The core was created with the very same build, yes.

Comment by Jan Lindström (Inactive) [ 2015-11-25 ]

I see, can you provide a database ?

Comment by Thomas Stein [ 2015-11-25 ]

No sorry. I'm afraid i can't do that. I can apply test patches if you have some at one point. Maybe i can give you ssh access to the test machine where i run the databases now.

thanks and cheers

Comment by Jan Lindström (Inactive) [ 2015-11-25 ]

I do not have any patches, there was a bug on 10.1.8 that has now fixed on 10.1.9, however this bug was such that mostly migrate from earlier versions failed but if it somehow did succeed, still the fix will cause the crash on migration from 10.1.8. Better to use you db before 10.1.8 migration.

Comment by pierre C [ 2015-11-25 ]

On my side, i managed to create the mysqld debug version. When i start it, it still fail.

Following the tutorial, i couldn't find any "core" file in my data directory (/var/lib/mysql/), but I noticed that more debug data are now available in the syslog:

Nov 25 21:36:17 demeter mysqld_safe: Starting mysqld daemon with databases from /var/lib/mysql
Nov 25 21:36:17 demeter mysqld: 2015-11-25 21:36:17 139926935971616 [Note] /usr/sbin/mysqld (mysqld 10.1.9-MariaDB-debug-log) starting as process 1472 ...
Nov 25 21:36:18 demeter mysqld: 2015-11-25 21:36:18 139926935971616 [Note] Using unique option prefix 'myisam_recover' is error-prone and can break in the future. Please use the full name 'myisam-recover-options' instead.
Nov 25 21:36:18 demeter mysqld: 2015-11-25 21:36:18 139926935971616 [ERROR] mysqld: file-key-management-filename is not set
Nov 25 21:36:18 demeter mysqld: 2015-11-25 21:36:18 139926935971616 [ERROR] Plugin 'file_key_management' init function returned error.
Nov 25 21:36:18 demeter mysqld: 2015-11-25 21:36:18 139926935971616 [ERROR] Plugin 'file_key_management' registration as a ENCRYPTION failed.
Nov 25 21:36:18 demeter mysqld: 2015-11-25 21:36:18 139926935971616 [Note] InnoDB: Using mutexes to ref count buffer pool pages
Nov 25 21:36:18 demeter mysqld: 2015-11-25 21:36:18 139926935971616 [Note] InnoDB:  InnoDB: !!!!!!!! UNIV_DEBUG switched on !!!!!!!!!
Nov 25 21:36:18 demeter mysqld: 2015-11-25 21:36:18 139926935971616 [Note] InnoDB:  InnoDB: !!!!!!!! UNIV_SYNC_DEBUG switched on !!!!!!!!!
Nov 25 21:36:18 demeter mysqld: 2015-11-25 21:36:18 139926935971616 [Note] InnoDB: The InnoDB memory heap is disabled
Nov 25 21:36:18 demeter mysqld: 2015-11-25 21:36:18 139926935971616 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
Nov 25 21:36:18 demeter mysqld: 2015-11-25 21:36:18 139926935971616 [Note] InnoDB: Memory barrier is not used
Nov 25 21:36:18 demeter mysqld: 2015-11-25 21:36:18 139926935971616 [Note] InnoDB: Compressed tables use zlib 1.2.7
Nov 25 21:36:18 demeter mysqld: 2015-11-25 21:36:18 139926935971616 [Note] InnoDB: Not using CPU crc32 instructions
Nov 25 21:36:18 demeter mysqld: 2015-11-25 21:36:18 139926935971616 [Note] InnoDB: Initializing buffer pool, size = 256.0M
Nov 25 21:36:18 demeter mysqld: 2015-11-25 21:36:18 139926935971616 [Note] InnoDB: Completed initialization of buffer pool
Nov 25 21:36:18 demeter mysqld: 2015-11-25 21:36:18 139926935971616 [Note] InnoDB: Highest supported file format is Barracuda.
Nov 25 21:36:18 demeter mysqld: 2015-11-25 21:36:18 139926935971616 [Note] InnoDB: The log sequence numbers 357910866 and 357910866 in ibdata files do not match the log sequence number 357910896 in the ib_logfiles!
Nov 25 21:36:18 demeter mysqld: 2015-11-25 21:36:18 139926935971616 [Note] InnoDB: Database was not shutdown normally!
Nov 25 21:36:18 demeter mysqld: 2015-11-25 21:36:18 139926935971616 [Note] InnoDB: Starting crash recovery.
Nov 25 21:36:18 demeter mysqld: 2015-11-25 21:36:18 139926935971616 [Note] InnoDB: Reading tablespace information from the .ibd files...
Nov 25 21:36:18 demeter mysqld: 2015-11-25 21:36:18 139926935971616 [Note] InnoDB: Restoring possible half-written data pages
Nov 25 21:36:18 demeter mysqld: 2015-11-25 21:36:18 139926935971616 [Note] InnoDB: from the doublewrite buffer...
Nov 25 21:36:18 demeter mysqld: InnoDB: Last MySQL binlog file position 0 274917, file name /var/log/mysql/mariadb-bin.000015
Nov 25 21:36:19 demeter mysqld: 2015-11-25 21:36:19 139926935971616 [Note] InnoDB: 128 rollback segment(s) are active.
Nov 25 21:36:19 demeter mysqld: 2015-11-25 21:36:19 139926935971616 [Note] InnoDB: Waiting for purge to start
Nov 25 21:36:19 demeter mysqld: 151125 21:36:19 [ERROR] mysqld got signal 11 ;
Nov 25 21:36:19 demeter mysqld: This could be because you hit a bug. It is also possible that this binary
Nov 25 21:36:19 demeter mysqld: or one of the libraries it was linked against is corrupt, improperly built,
Nov 25 21:36:19 demeter mysqld: or misconfigured. This error can also be caused by malfunctioning hardware.
Nov 25 21:36:19 demeter mysqld:
Nov 25 21:36:19 demeter mysqld: To report this bug, see http://kb.askmonty.org/en/reporting-bugs
Nov 25 21:36:19 demeter mysqld:
Nov 25 21:36:19 demeter mysqld: We will try our best to scrape up some info that will hopefully help
Nov 25 21:36:19 demeter mysqld: diagnose the problem, but since we have already crashed,
Nov 25 21:36:19 demeter mysqld: something is definitely wrong and this may fail.
Nov 25 21:36:19 demeter mysqld:
Nov 25 21:36:19 demeter mysqld: Server version: 10.1.9-MariaDB-debug-log
Nov 25 21:36:19 demeter mysqld: key_buffer_size=134217728
Nov 25 21:36:19 demeter mysqld: read_buffer_size=2097152
Nov 25 21:36:19 demeter mysqld: max_used_connections=0
Nov 25 21:36:19 demeter mysqld: max_threads=102
Nov 25 21:36:19 demeter mysqld: thread_count=0
Nov 25 21:36:19 demeter mysqld: It is possible that mysqld could use up to
Nov 25 21:36:19 demeter mysqld: key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 759879 K  bytes of memory
Nov 25 21:36:19 demeter mysqld: Hope that's ok; if not, decrease some variables in the equation.
Nov 25 21:36:19 demeter mysqld:
Nov 25 21:36:19 demeter mysqld: Thread pointer: 0x0x0
Nov 25 21:36:19 demeter mysqld: Attempting backtrace. You can use the following information to find out
Nov 25 21:36:19 demeter mysqld: where mysqld died. If you see no messages after this, something went
Nov 25 21:36:19 demeter mysqld: terribly wrong...
Nov 25 21:36:19 demeter mysqld: stack_bottom = 0x0 thread_stack 0x48000
Nov 25 21:36:19 demeter mysqld: 2015-11-25 21:36:19 139926935971616 [Note] InnoDB:  Percona XtraDB (http://www.percona.com) 5.6.26-74.0 started; log sequence number 357910896
Nov 25 21:36:19 demeter mysqld: 2015-11-25 21:36:19 139926935971616 [Note] CONNECT: Version 1.04.0003 October 25, 2015
Nov 25 21:36:19 demeter mysqld: 2015-11-25 21:36:19 139926241908480 [Note] InnoDB: Dumping buffer pool(s) not yet started
Nov 25 21:36:20 demeter mysqld: mysys/stacktrace.c:246(my_print_stacktrace)[0xfc1272]
Nov 25 21:36:20 demeter mysqld: sql/signal_handler.cc:162(handle_fatal_signal)[0x8c8810]
Nov 25 21:36:20 demeter mysqld: /lib/x86_64-linux-gnu/libpthread.so.0(+0xf0a0)[0x7f43470e40a0]
Nov 25 21:36:20 demeter mysqld: include/mach0data.ic:56(mach_read_from_1)[0xbe72f9]
Nov 25 21:36:20 demeter mysqld: include/mach0data.ic:266(mach_read_compressed)[0xbe773e]
Nov 25 21:36:20 demeter mysqld: trx/trx0rec.cc:907(trx_undo_update_rec_get_field_no)[0xbee13e]
Nov 25 21:36:20 demeter mysqld: trx/trx0rec.cc:992(trx_undo_update_rec_get_update(unsigned char*, dict_index_t*, unsigned long, unsigned long, unsigned long, unsigned long, trx_t*, mem_block_info_t*, upd_t**))[0xbee324]
Nov 25 21:36:20 demeter mysqld: row/row0purge.cc:816(row_purge_parse_undo_rec)[0xb97b42]
Nov 25 21:36:20 demeter mysqld: row/row0purge.cc:909(row_purge)[0xb97e61]
Nov 25 21:36:20 demeter mysqld: row/row0purge.cc:994(row_purge_step(que_thr_t*))[0xb980b8]
Nov 25 21:36:20 demeter mysqld: que/que0que.cc:1089(que_thr_step)[0xb38434]
Nov 25 21:36:20 demeter mysqld: que/que0que.cc:1151(que_run_threads_low)[0xb38635]
Nov 25 21:36:20 demeter mysqld: que/que0que.cc:1194(que_run_threads(que_thr_t*))[0xb387c7]
Nov 25 21:36:20 demeter mysqld: trx/trx0purge.cc:1253(trx_purge(unsigned long, unsigned long, bool))[0xbe696d]
Nov 25 21:36:20 demeter mysqld: srv/srv0srv.cc:3436(srv_do_purge)[0xbc9fe9]
Nov 25 21:36:20 demeter mysqld: srv/srv0srv.cc:3622(srv_purge_coordinator_thread)[0xbca701]
Nov 25 21:36:20 demeter mysqld: /lib/x86_64-linux-gnu/libpthread.so.0(+0x6b50)[0x7f43470dbb50]
Nov 25 21:36:20 demeter mysqld: /lib/x86_64-linux-gnu/libc.so.6(clone+0x6d)[0x7f4345e2c95d]
Nov 25 21:36:20 demeter mysqld: The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
Nov 25 21:36:20 demeter mysqld: information that should help you find out what is causing the crash.
Nov 25 21:36:20 demeter mysqld_safe: mysqld from pid file /var/run/mysqld/mysqld.pid ended
Nov 25 21:36:48 demeter /etc/init.d/mysql[1815]: 0 processes alive and '/us1       r/bin/mysqladmin --defaults-file=/etc/mysql/debian.cnf ping' resulted in
Nov 25 21:36:48 demeter /etc/init.d/mysql[1815]: #007/usr/bin/mysqladmin: connect to server at 'localhost' failed
Nov 25 21:36:48 demeter /etc/init.d/mysql[1815]: error: 'Can't connect to local MySQL server through socket '/var/run/mysqld/mysqld.sock' (2 "No such file or directory")'
Nov 25 21:36:48 demeter /etc/init.d/mysql[1815]: Check that mysqld is running and that the socket: '/var/run/mysqld/mysqld.sock' exists!
Nov 25 21:36:48 demeter /etc/init.d/mysql[1815]:

thanks.
p.

Comment by Jan Lindström (Inactive) [ 2015-11-25 ]

Hi,

Yes, this is exactly the problem there was in 10.1.8, you need to migrate your database from state before migrate to 10.1.8 and migrate that to 10.1.9.

Comment by Elena Stepanova [ 2015-11-25 ]

If 10.1.8 somehow magically works with these tables, I would still suggest at least trying to dump/restore the current database, rather than using the old backup. If dump doesn't work, the old backup will always be the plan B.

Comment by Thomas Stein [ 2015-11-25 ]

So you suggesting dumping all databases, delete them, upgrade to 10.1.9 and import them again?

thanks and cheers

Comment by Jan Lindström (Inactive) [ 2015-11-25 ]

What I would suggest is to first dump all databases from a backup before you migrated 10.1.8 and then dump all databases from your 10.1.8 version and then make sure (diff) that result is correct before upgrading to 10.1.9 and importing whatever is correct.

Comment by Thomas Stein [ 2015-11-25 ]

Well that's not really an option. The db's are running a few weeks now with 10.1.8. But i was able to dump with 10.1.8 and import them with 10.1.9. For now everything is working as expected.

Comment by John Jetmore [ 2015-12-17 ]

Jan, can you elaborate on this comment:

"Yes, this is exactly the problem there was in 10.1.8, you need to migrate your database from state before migrate to 10.1.8 and migrate that to 10.1.9."

I had the problem in 5.5.45 -> 10.1.8 upgrade intermittently. Opened a case on it and confirmed that a dev build of 10.1.9 was able to run with the databases that 10.1.8 was crashing on. Now we are trying 10.1.8 -> 10.1.9 upgrades and we are seeing these crashes again. Many of our sites have already updated to 10.1.8, so doing something before that upgrade isn't really an option. Our software also does unattended upgrades at client sites, often on very large databases. Is there something I can do on a 10.1.8 install to prepare it for the upgrade to 10.1.9 to minimize the risk of this bug? Or can I run something to see if there's a risk of this problem, and then just not upgrade?

Comment by Jan Lindström (Inactive) [ 2015-12-17 ]

Hi, 10.1.8 and earlier 10.1 versions have a bug that makes migration from any previous MariaDB version either crash or if migration is successful it could crash when you migrate to 10.1.9 that contains the fix. So if you have databases using 5.5 or 10.0, migrate to 10.1.9. If you have database using 10.1.8 try to migrate to 10.1.9 and if that fails, you need to use mysqldump and dump your database, and import it to 10.1.9. Bug was such that there is no way to avoid the crash if you are affected (we have not yet found a simple rules when you are affected). But main point is do not use 10.1.8.

Comment by Daniel Norton [ 2016-02-01 ]

This doesn't seem properly resolved, as there are two bugs:

  1. The bug that corrupts the database
  2. The bug that crashes the entire server.

Part 1 might have been fixed, but part 2 definitely persists in 10.1.11. It seems to me that a proper fix to #2 is to not crash the server. At worst, it might simply not open the offending database, but the other databases should remain fully operational.

Generated at Thu Feb 08 07:32:42 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.