my.cnf:
[mysqld]
|
datadir=/base/mysql
|
socket=/base/mysql/mysql.sock
|
character-set-server=cp1251
|
|
|
[server]
|
datadir=/base/mysql
|
tmpdir=/tmp
|
|
|
concurrent_insert=2
|
|
|
max_connections=740
|
max_user_connections=700
|
max_connect_errors=50
|
open-files-limit=20000
|
|
|
join_buffer_size=2M
|
key_buffer_size=256M
|
myisam_sort_buffer_size=64M
|
read_buffer_size=128K
|
read_rnd_buffer_size=32M
|
sort_buffer_size=32M
|
|
|
max_allowed_packet=32M
|
max_heap_table_size=512M
|
max_tmp_tables=1000
|
table_cache=10000
|
tmp_table_size=512M
|
|
|
query_cache_limit=131072
|
query_cache_size=33554432
|
query_cache_min_res_unit=1536
|
query_cache_strip_comments=ON
|
|
|
thread_cache_size=400
|
|
|
innodb_buffer_pool_size=100G
|
transaction-isolation=READ-COMMITTED
|
innodb_flush_log_at_trx_commit=0
|
innodb_flush_method=O_DIRECT
|
innodb_data_file_path=ibdata1:128M:autoextend
|
innodb_log_buffer_size = 256M
|
innodb_log_file_size = 1024M
|
innodb_log_files_in_group = 2
|
|
|
innodb_file_per_table=1
|
innodb_file_format=barracuda
|
innodb_io_capacity=5000
|
innodb_read_io_threads=12
|
innodb_write_io_threads=12
|
|
|
long_query_time=3
|
log-warnings
|
back_log=500
|
|
|
interactive_timeout=320
|
wait_timeout=320
|
net_read_timeout=320
|
net_write_timeout=420
|
|
|
server-id=79
|
log-bin=/base/binlog/bin
|
log_slave_updates
|
relay-log=/base/binlog/relay-bin
|
skip-name-resolve
|
expire_logs_days=4
|
binlog_format=ROW
|
|
|
replicate-wild-ignore-table=tmp.%
|
|
|
init-connect='SET NAMES cp1251'
|
|
|
performance_schema='on'
|
Similar issues on our backup server (centos5 and old hardware):
140212 0:42:20 [ERROR] mysqld got signal 11 ;
|
This could be because you hit a bug. It is also possible that this binary
|
or one of the libraries it was linked against is corrupt, improperly built,
|
or misconfigured. This error can also be caused by malfunctioning hardware.
|
|
|
To report this bug, see http://kb.askmonty.org/en/reporting-bugs
|
|
|
We will try our best to scrape up some info that will hopefully help
|
diagnose the problem, but since we have already crashed,
|
something is definitely wrong and this may fail.
|
|
|
Server version: 5.5.35-MariaDB-log
|
key_buffer_size=134217728
|
read_buffer_size=131072
|
max_used_connections=11
|
max_threads=502
|
thread_count=8
|
It is possible that mysqld could use up to
|
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 2260475 K bytes of memory
|
Hope that's ok; if not, decrease some variables in the equation.
|
|
|
Thread pointer: 0x0x2aaefc6b8e60
|
Attempting backtrace. You can use the following information to find out
|
where mysqld died. If you see no messages after this, something went
|
terribly wrong...
|
stack_bottom = 0x415d80a0 thread_stack 0x48000
|
(my_addr_resolve failure: fork)
|
/usr/libexec/mysqld(my_print_stacktrace+0x2e) [0x2b8ddd8a89ee]
|
/usr/libexec/mysqld(handle_fatal_signal+0x4d1) [0x2b8ddd4d8b01]
|
/lib64/libpthread.so.0 [0x2b8dde9e2b10]
|
/usr/libexec/mysqld [0x2b8ddd7b44a0]
|
/usr/libexec/mysqld [0x2b8ddd7b4b4f]
|
/usr/libexec/mysqld [0x2b8ddd7a6e28]
|
/usr/libexec/mysqld [0x2b8ddd7b5626]
|
/usr/libexec/mysqld [0x2b8ddd7b6d1f]
|
/usr/libexec/mysqld [0x2b8ddd7a2458]
|
/usr/libexec/mysqld [0x2b8ddd795e56]
|
/usr/libexec/mysqld [0x2b8ddd748ebc]
|
/usr/libexec/mysqld [0x2b8ddd7163d7]
|
/usr/libexec/mysqld(rr_sequential(READ_RECORD*)+0xba) [0x2b8ddd5ea01a]
|
/usr/libexec/mysqld(sub_select(JOIN*, st_join_table*, bool)+0x137) [0x2b8ddd3cb067]
|
/usr/libexec/mysqld [0x2b8ddd3cb4f2]
|
/usr/libexec/mysqld(JOIN::exec()+0xa81) [0x2b8ddd3da611]
|
/usr/libexec/mysqld(mysql_select(THD*, Item***, TABLE_LIST*, unsigned int, List<Item>&, Item*, unsigned int, st_order*, st_order*, Item*, st_order*, unsigned long long, select_result*, st_select_lex_unit*, st_select_lex*)+0x1a9) [0x2b8ddd3dcb19]
|
/usr/libexec/mysqld(handle_select(THD*, LEX*, select_result*, unsigned long)+0x2b6) [0x2b8ddd3dd616]
|
/usr/libexec/mysqld [0x2b8ddd389d96]
|
/usr/libexec/mysqld(mysql_execute_command(THD*)+0x423f) [0x2b8ddd38f1cf]
|
/usr/libexec/mysqld(mysql_parse(THD*, char*, unsigned int, Parser_state*)+0x1cc) [0x2b8ddd391bac]
|
/usr/libexec/mysqld(dispatch_command(enum_server_command, THD*, char*, unsigned int)+0x1829) [0x2b8ddd3933e9]
|
/usr/libexec/mysqld(do_command(THD*)+0xd2) [0x2b8ddd393c82]
|
/usr/libexec/mysqld(do_handle_one_connection(THD*)+0x12f) [0x2b8ddd44959f]
|
/usr/libexec/mysqld(handle_one_connection+0x54) [0x2b8ddd4496b4]
|
/lib64/libpthread.so.0 [0x2b8dde9da73d]
|
/lib64/libc.so.6(clone+0x6d) [0x2b8de003ed1d]
|
|
|
Trying to get some variables.
|
Some pointers may be invalid and cause the dump to abort.
|
Query (0x2b8e04b6a0e8): SELECT /*!40001 SQL_NO_CACHE */ * FROM `education`
|
|
|
Connection ID (thread ID): 8430
|
Status: NOT_KILLED
|
|
|
Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=off,table_elimination=on,extended_keys=off
|
|
|
The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
|
information that should help you find out what is causing the crash.
|
140212 00:42:22 mysqld_safe Number of processes running now: 0
|
140212 00:42:22 mysqld_safe mysqld restarted
|
140212 0:42:22 InnoDB: The InnoDB memory heap is disabled
|
140212 0:42:22 InnoDB: Mutexes and rw_locks use GCC atomic builtins
|
140212 0:42:22 InnoDB: Compressed tables use zlib 1.2.3
|
140212 0:42:22 InnoDB: Using Linux native AIO
|
140212 0:42:22 InnoDB: Initializing buffer pool, size = 16.0G
|
140212 0:42:23 InnoDB: Completed initialization of buffer pool
|
140212 0:42:23 InnoDB: highest supported file format is Barracuda.
|
InnoDB: The log sequence number in ibdata files does not match
|
InnoDB: the log sequence number in the ib_logfiles!
|
140212 0:42:23 InnoDB: Database was not shut down normally!
|
InnoDB: Starting crash recovery.
|
InnoDB: Reading tablespace information from the .ibd files...
|
InnoDB: Restoring possible half-written data pages from the doublewrite
|
InnoDB: buffer...
|
InnoDB: In a MySQL replication slave the last master binlog file
|
InnoDB: position 668119808, file name dbs01-06-bin.001802
|
InnoDB: and relay log file
|
InnoDB: position 668120095, file name /base/binlog/dbs01-02-relay-bin.003436
|
InnoDB: Last MySQL binlog file position 0 147687341, file name /base/binlog/dbs01-02-bin.002047
|
140212 0:42:28 InnoDB: Waiting for the background threads to start
|
140212 0:42:29 Percona XtraDB (http://www.percona.com) 5.5.35-MariaDB-33.0 started; log sequence number 20309702665164
|
140212 0:42:29 [Note] Plugin 'FEEDBACK' is disabled.
|
140212 0:42:29 [Note] Recovering after a crash using /base/binlog/dbs01-02-bin
|
140212 0:42:30 [Note] Starting crash recovery...
|
140212 0:42:30 [Note] Crash recovery finished.
|
140212 0:42:31 [Note] Server socket created on IP: '10.10.1.26'.
|
140212 0:42:31 [Note] Slave SQL thread initialized, starting replication in log 'dbs01-06-bin.001802' at position 668119808, relay log '/base/binlog/dbs01-02-relay-bin.003436' position: 668120095
|
140212 0:42:31 [Note] Slave I/O thread: connected to master 'mirror@dbs01-06.sj.local:3306',replication started in log 'dbs01-06-bin.001802' at position 668121990
|
140212 0:42:31 [Note] Event Scheduler: Loaded 0 events
|
140212 0:42:31 [Note] /usr/libexec/mysqld: ready for connections.
|
Version: '5.5.35-MariaDB-log' socket: '/base/mysql/mysql.sock' port: 3306 MariaDB Server
|
Config:
[mysqld]
|
datadir=/base/mysql
|
socket=/base/mysql/mysql.sock
|
character-set-server=cp1251
|
bind-address=10.10.1.26
|
|
|
[server]
|
datadir=/base/mysql
|
concurrent_insert=2
|
max_connections=500
|
max_user_connections=490
|
max_connect_errors=50
|
open-files-limit=20000
|
join_buffer_size=2M
|
key_buffer_size=128M
|
myisam_sort_buffer_size=64M
|
read_buffer_size=128K
|
read_rnd_buffer_size=32M
|
sort_buffer_size=4M
|
max_allowed_packet=32M
|
max_heap_table_size=512M
|
max_tmp_tables=1000
|
table_cache=10000
|
tmp_table_size=512M
|
query_cache_limit=131072
|
query_cache_size=0
|
query_cache_min_res_unit=1536
|
thread_cache_size=20
|
innodb_buffer_pool_size=16G
|
transaction-isolation=READ-COMMITTED
|
innodb_flush_log_at_trx_commit=1
|
innodb_flush_method=O_DIRECT
|
innodb_data_file_path=ibdata1:128M:autoextend
|
innodb_log_file_size=1024M
|
innodb_log_buffer_size=128M
|
innodb_file_per_table=1
|
innodb_file_format=barracuda
|
innodb_io_capacity=2000
|
long_query_time=3
|
log-warnings
|
back_log=500
|
interactive_timeout=320
|
wait_timeout=320
|
net_read_timeout=320
|
net_write_timeout=420
|
server-id=26
|
log-bin=/base/binlog/dbs01-02-bin
|
log_slave_updates
|
relay-log=/base/binlog/dbs01-02-relay-bin
|
skip-name-resolve
|
expire_logs_days=1
|
binlog_format=ROW
|
|
|
replicate-wild-ignore-table=tmp.%
|
init-connect='SET NAMES cp1251'
|
Table schema:
CREATE TABLE `education` ( |
`id_user` int(10) unsigned NOT NULL, |
`monthend` tinyint(3) unsigned NOT NULL, |
`yearend` year(4) DEFAULT NULL, |
`name` varchar(255) NOT NULL, |
`institut` varchar(255) NOT NULL, |
`town` varchar(50) NOT NULL, |
`treningtime` varchar(20) NOT NULL, |
`id_education` int(11) NOT NULL DEFAULT '0', |
PRIMARY KEY (`id_user`,`id_education`) |
) ENGINE=InnoDB DEFAULT CHARSET=utf8 ROW_FORMAT=DYNAMIC |
And two days earlier:
140210 0:45:02 [ERROR] mysqld got signal 11 ;
|
This could be because you hit a bug. It is also possible that this binary
|
or one of the libraries it was linked against is corrupt, improperly built,
|
or misconfigured. This error can also be caused by malfunctioning hardware.
|
|
|
To report this bug, see http://kb.askmonty.org/en/reporting-bugs
|
|
|
We will try our best to scrape up some info that will hopefully help
|
diagnose the problem, but since we have already crashed,
|
something is definitely wrong and this may fail.
|
|
|
Server version: 5.5.35-MariaDB-log
|
key_buffer_size=134217728
|
read_buffer_size=131072
|
max_used_connections=11
|
max_threads=502
|
thread_count=8
|
It is possible that mysqld could use up to
|
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 2260475 K bytes of memory
|
Hope that's ok; if not, decrease some variables in the equation.
|
|
|
Thread pointer: 0x0x2aaf0c434200
|
Attempting backtrace. You can use the following information to find out
|
where mysqld died. If you see no messages after this, something went
|
terribly wrong...
|
stack_bottom = 0x4b9120a0 thread_stack 0x48000
|
(my_addr_resolve failure: fork)
|
/usr/libexec/mysqld(my_print_stacktrace+0x2e) [0x2b8591b929ee]
|
/usr/libexec/mysqld(handle_fatal_signal+0x4d1) [0x2b85917c2b01]
|
/lib64/libpthread.so.0 [0x2b8592cccb10]
|
/usr/libexec/mysqld [0x2b8591a9e4a0]
|
/usr/libexec/mysqld [0x2b8591a9eb4f]
|
/usr/libexec/mysqld [0x2b8591a90e28]
|
/usr/libexec/mysqld [0x2b8591a9f626]
|
/usr/libexec/mysqld [0x2b8591aa0d1f]
|
/usr/libexec/mysqld [0x2b8591a8c458]
|
/usr/libexec/mysqld [0x2b8591a7fe56]
|
/usr/libexec/mysqld [0x2b8591a32ebc]
|
/usr/libexec/mysqld [0x2b8591a003d7]
|
/usr/libexec/mysqld(rr_sequential(READ_RECORD*)+0xba) [0x2b85918d401a]
|
/usr/libexec/mysqld(sub_select(JOIN*, st_join_table*, bool)+0x137) [0x2b85916b5067]
|
/usr/libexec/mysqld [0x2b85916b54f2]
|
/usr/libexec/mysqld(JOIN::exec()+0xa81) [0x2b85916c4611]
|
/usr/libexec/mysqld(mysql_select(THD*, Item***, TABLE_LIST*, unsigned int, List<Item>&, Item*, unsigned int, st_order*, st_order*, Item*, st_order*, unsigned long long, select_res
|
/usr/libexec/mysqld(handle_select(THD*, LEX*, select_result*, unsigned long)+0x2b6) [0x2b85916c7616]
|
/usr/libexec/mysqld [0x2b8591673d96]
|
/usr/libexec/mysqld(mysql_execute_command(THD*)+0x423f) [0x2b85916791cf]
|
/usr/libexec/mysqld(mysql_parse(THD*, char*, unsigned int, Parser_state*)+0x1cc) [0x2b859167bbac]
|
/usr/libexec/mysqld(dispatch_command(enum_server_command, THD*, char*, unsigned int)+0x1829) [0x2b859167d3e9]
|
/usr/libexec/mysqld(do_command(THD*)+0xd2) [0x2b859167dc82]
|
/usr/libexec/mysqld(do_handle_one_connection(THD*)+0x12f) [0x2b859173359f]
|
/usr/libexec/mysqld(handle_one_connection+0x54) [0x2b85917336b4]
|
/lib64/libpthread.so.0 [0x2b8592cc473d]
|
/lib64/libc.so.6(clone+0x6d) [0x2b8594328d1d]
|
|
|
Trying to get some variables.
|
Some pointers may be invalid and cause the dump to abort.
|
Query (0x2b85bc50e488): SELECT /*!40001 SQL_NO_CACHE */ * FROM `received_resumes`
|
|
|
Connection ID (thread ID): 18168
|
Status: NOT_KILLED
|
|
|
Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_c
|
|
|
The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
|
information that should help you find out what is causing the crash.
|
Table schema:
CREATE TABLE `received_resumes` ( |
`id` int(10) unsigned NOT NULL DEFAULT '0', |
`id_client` int(10) unsigned NOT NULL DEFAULT '0', |
`id_user` int(10) unsigned NOT NULL DEFAULT '0', |
`id_resume` int(10) unsigned NOT NULL DEFAULT '0', |
`status` tinyint(3) unsigned NOT NULL DEFAULT '0', |
`id_folder` int(10) unsigned NOT NULL DEFAULT '0', |
`new_event` tinyint(3) unsigned NOT NULL DEFAULT '0', |
`date_last_event` int(10) unsigned NOT NULL DEFAULT '0', |
`id_last_vac` int(10) unsigned NOT NULL, |
`display_contacts` tinyint(1) NOT NULL DEFAULT '0' COMMENT 'censored', |
UNIQUE KEY `id_user` (`id_user`,`id_resume`), |
KEY `id` (`id`), |
KEY `id_resume` (`id_resume`) |
) ENGINE=InnoDB DEFAULT CHARSET=utf8 |
I had tried to reproduce bug on backup server (mysqldump, 8 simultaneous threads) and catch signal 11 with core file:
# previous part of log with "Assertion failure in file buf0lru.c line 2355" you can see
|
# in https://mariadb.atlassian.net/browse/MDEV-5670 :)
|
140213 01:25:51 mysqld_safe mysqld restarted
|
140213 1:25:54 InnoDB: The InnoDB memory heap is disabled
|
140213 1:25:54 InnoDB: Mutexes and rw_locks use GCC atomic builtins
|
140213 1:25:54 InnoDB: Compressed tables use zlib 1.2.3
|
140213 1:25:54 InnoDB: Using Linux native AIO
|
140213 1:25:55 InnoDB: Initializing buffer pool, size = 16.0G
|
140213 1:25:55 InnoDB: Completed initialization of buffer pool
|
140213 1:25:55 InnoDB: highest supported file format is Barracuda.
|
InnoDB: The log sequence number in ibdata files does not match
|
InnoDB: the log sequence number in the ib_logfiles!
|
140213 1:25:56 InnoDB: Database was not shut down normally!
|
InnoDB: Starting crash recovery.
|
InnoDB: Reading tablespace information from the .ibd files...
|
InnoDB: Restoring possible half-written data pages from the doublewrite
|
InnoDB: buffer...
|
InnoDB: In a MySQL replication slave the last master binlog file
|
InnoDB: position 918533120, file name dbs01-06-bin.001815
|
InnoDB: and relay log file
|
InnoDB: position 918533407, file name /base/binlog/dbs01-02-relay-bin.003478
|
InnoDB: Last MySQL binlog file position 0 327998402, file name /base/binlog/dbs01-02-bin.002061
|
140213 1:26:06 InnoDB: Waiting for the background threads to start
|
140213 1:26:07 Percona XtraDB (http://www.percona.com) 5.5.35-MariaDB-33.0 started; log sequence number 20326976545089
|
140213 1:26:07 [Note] Plugin 'FEEDBACK' is disabled.
|
140213 1:26:07 [Note] Recovering after a crash using /base/binlog/dbs01-02-bin
|
140213 1:26:11 [Note] Starting crash recovery...
|
140213 1:26:11 [Note] Crash recovery finished.
|
140213 1:26:11 [Note] Server socket created on IP: '10.10.1.26'.
|
140213 1:26:11 [Note] Slave SQL thread initialized, starting replication in log 'dbs01-06-bin.001815' at position 918533120, relay log '/base/binlog/dbs01-02-relay-bin.003478' position: 918533407
|
140213 1:26:11 [Note] Slave I/O thread: connected to master 'mirror@dbs01-06:3306',replication started in log 'dbs01-06-bin.001817' at position 780439587
|
140213 1:26:11 [Note] Event Scheduler: Loaded 0 events
|
140213 1:26:11 [Note] /usr/libexec/mysqld: ready for connections.
|
Version: '5.5.35-MariaDB-log' socket: '/base/mysql/mysql.sock' port: 3306 MariaDB Server
|
140213 16:45:59 [ERROR] Invalid (old?) table or database name 'lost+found'
|
140213 16:48:13 [Note] Slave SQL thread exiting, replication stopped in log 'dbs01-06-bin.001828' at position 37353415
|
140213 16:48:13 [ERROR] Error reading packet from server: Lost connection to MySQL server during query ( server_errno=2013)
|
140213 16:48:13 [Note] Slave I/O thread killed while reading event
|
140213 16:48:13 [Note] Slave I/O thread exiting, read up to log 'dbs01-06-bin.001828', position 37808759
|
140213 17:34:05 [ERROR] mysqld got signal 11 ;
|
This could be because you hit a bug. It is also possible that this binary
|
or one of the libraries it was linked against is corrupt, improperly built,
|
or misconfigured. This error can also be caused by malfunctioning hardware.
|
|
|
To report this bug, see http://kb.askmonty.org/en/reporting-bugs
|
|
|
We will try our best to scrape up some info that will hopefully help
|
diagnose the problem, but since we have already crashed,
|
something is definitely wrong and this may fail.
|
|
|
Server version: 5.5.35-MariaDB-log
|
key_buffer_size=134217728
|
read_buffer_size=131072
|
max_used_connections=11
|
max_threads=502
|
thread_count=8
|
It is possible that mysqld could use up to
|
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 2260475 K bytes of memory
|
Hope that's ok; if not, decrease some variables in the equation.
|
|
|
Thread pointer: 0x0x2aaef4ca7df0
|
Attempting backtrace. You can use the following information to find out
|
where mysqld died. If you see no messages after this, something went
|
terribly wrong...
|
stack_bottom = 0x417650a0 thread_stack 0x48000
|
(my_addr_resolve failure: fork)
|
/usr/libexec/mysqld(my_print_stacktrace+0x2e) [0x2aea6a9aa9ee]
|
/usr/libexec/mysqld(handle_fatal_signal+0x4d1) [0x2aea6a5dab01]
|
/lib64/libpthread.so.0 [0x2aea6bae4b10]
|
/usr/libexec/mysqld [0x2aea6a8b64a0]
|
/usr/libexec/mysqld [0x2aea6a8b6b4f]
|
/usr/libexec/mysqld [0x2aea6a8a45df]
|
/usr/libexec/mysqld [0x2aea6a897e56]
|
/usr/libexec/mysqld [0x2aea6a84aebc]
|
/usr/libexec/mysqld [0x2aea6a8183d7]
|
/usr/libexec/mysqld(rr_sequential(READ_RECORD*)+0xba) [0x2aea6a6ec01a]
|
/usr/libexec/mysqld(sub_select(JOIN*, st_join_table*, bool)+0x137) [0x2aea6a4cd067]
|
/usr/libexec/mysqld [0x2aea6a4cd4f2]
|
/usr/libexec/mysqld(JOIN::exec()+0xa81) [0x2aea6a4dc611]
|
/usr/libexec/mysqld(mysql_select(THD*, Item***, TABLE_LIST*, unsigned int, List<Item>&, Item*, unsigned int, st_order*, st_order*, Item*, st_order*, unsigned long long, select_result*, st_select_lex_unit*, st_select_lex*)+0x1a9) [0x2aea6a4deb19]
|
/usr/libexec/mysqld(handle_select(THD*, LEX*, select_result*, unsigned long)+0x2b6) [0x2aea6a4df616]
|
/usr/libexec/mysqld [0x2aea6a48bd96]
|
/usr/libexec/mysqld(mysql_execute_command(THD*)+0x423f) [0x2aea6a4911cf]
|
/usr/libexec/mysqld(mysql_parse(THD*, char*, unsigned int, Parser_state*)+0x1cc) [0x2aea6a493bac]
|
/usr/libexec/mysqld(dispatch_command(enum_server_command, THD*, char*, unsigned int)+0x1829) [0x2aea6a4953e9]
|
/usr/libexec/mysqld(do_command(THD*)+0xd2) [0x2aea6a495c82]
|
/usr/libexec/mysqld(do_handle_one_connection(THD*)+0x12f) [0x2aea6a54b59f]
|
/usr/libexec/mysqld(handle_one_connection+0x54) [0x2aea6a54b6b4]
|
/lib64/libpthread.so.0 [0x2aea6badc73d]
|
/lib64/libc.so.6(clone+0x6d) [0x2aea6d140d1d]
|
|
|
Trying to get some variables.
|
Some pointers may be invalid and cause the dump to abort.
|
Query (0x2aeaa88f1938): SELECT /*!40001 SQL_NO_CACHE */ * FROM `resume_sphinx`
|
Connection ID (thread ID): 5743
|
Status: NOT_KILLED
|
|
|
Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=off,table_elimination=on,extended_keys=off
|
|
|
The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
|
information that should help you find out what is causing the crash.
|
Writing a core file
|
140213 17:35:08 mysqld_safe Number of processes running now: 0
|
140213 17:35:09 mysqld_safe mysqld restarted
|
# gdb /usr/libexec/mysqld core.13369
|
GNU gdb (GDB) Red Hat Enterprise Linux (7.0.1-23.el5_5.1)
|
Copyright (C) 2009 Free Software Foundation, Inc.
|
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
|
This is free software: you are free to change and redistribute it.
|
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
|
and "show warranty" for details.
|
This GDB was configured as "x86_64-redhat-linux-gnu".
|
For bug reporting instructions, please see:
|
<http://www.gnu.org/software/gdb/bugs/>...
|
Reading symbols from /usr/libexec/mysqld...done.
|
[New Thread 30146]
|
[New Thread 30145]
|
[New Thread 29854]
|
[New Thread 29594]
|
[New Thread 29593]
|
[New Thread 29592]
|
[New Thread 29560]
|
[New Thread 13984]
|
[New Thread 13983]
|
[New Thread 13982]
|
[New Thread 13389]
|
[New Thread 13388]
|
[New Thread 13387]
|
[New Thread 13386]
|
[New Thread 13385]
|
[New Thread 13384]
|
[New Thread 13383]
|
[New Thread 13381]
|
[New Thread 13380]
|
[New Thread 13379]
|
[New Thread 13378]
|
[New Thread 13377]
|
[New Thread 13376]
|
[New Thread 13375]
|
[New Thread 13374]
|
[New Thread 13373]
|
[New Thread 13372]
|
[New Thread 13371]
|
[New Thread 13369]
|
Reading symbols from /lib64/libpthread.so.0...(no debugging symbols found)...done.
|
Loaded symbols for /lib64/libpthread.so.0
|
Reading symbols from /usr/lib64/libz.so.1...(no debugging symbols found)...done.
|
Loaded symbols for /usr/lib64/libz.so.1
|
Reading symbols from /lib64/librt.so.1...(no debugging symbols found)...done.
|
Loaded symbols for /lib64/librt.so.1
|
Reading symbols from /lib64/libcrypt.so.1...(no debugging symbols found)...done.
|
Loaded symbols for /lib64/libcrypt.so.1
|
Reading symbols from /lib64/libdl.so.2...(no debugging symbols found)...done.
|
Loaded symbols for /lib64/libdl.so.2
|
Reading symbols from /lib64/libssl.so.6...(no debugging symbols found)...done.
|
Loaded symbols for /lib64/libssl.so.6
|
Reading symbols from /lib64/libcrypto.so.6...(no debugging symbols found)...done.
|
Loaded symbols for /lib64/libcrypto.so.6
|
Reading symbols from /usr/lib64/libstdc++.so.6...(no debugging symbols found)...done.
|
Loaded symbols for /usr/lib64/libstdc++.so.6
|
Reading symbols from /lib64/libm.so.6...(no debugging symbols found)...done.
|
Loaded symbols for /lib64/libm.so.6
|
Reading symbols from /lib64/libc.so.6...(no debugging symbols found)...done.
|
Loaded symbols for /lib64/libc.so.6
|
Reading symbols from /lib64/ld-linux-x86-64.so.2...(no debugging symbols found)...done.
|
Loaded symbols for /lib64/ld-linux-x86-64.so.2
|
Reading symbols from /usr/lib64/libgssapi_krb5.so.2...(no debugging symbols found)...done.
|
Loaded symbols for /usr/lib64/libgssapi_krb5.so.2
|
Reading symbols from /usr/lib64/libkrb5.so.3...(no debugging symbols found)...done.
|
Loaded symbols for /usr/lib64/libkrb5.so.3
|
Reading symbols from /lib64/libcom_err.so.2...(no debugging symbols found)...done.
|
Loaded symbols for /lib64/libcom_err.so.2
|
Reading symbols from /usr/lib64/libk5crypto.so.3...(no debugging symbols found)...done.
|
Loaded symbols for /usr/lib64/libk5crypto.so.3
|
Reading symbols from /lib64/libgcc_s.so.1...(no debugging symbols found)...done.
|
Loaded symbols for /lib64/libgcc_s.so.1
|
Reading symbols from /usr/lib64/libkrb5support.so.0...(no debugging symbols found)...done.
|
Loaded symbols for /usr/lib64/libkrb5support.so.0
|
Reading symbols from /lib64/libkeyutils.so.1...(no debugging symbols found)...done.
|
Loaded symbols for /lib64/libkeyutils.so.1
|
Reading symbols from /lib64/libresolv.so.2...(no debugging symbols found)...done.
|
Loaded symbols for /lib64/libresolv.so.2
|
Reading symbols from /lib64/libselinux.so.1...(no debugging symbols found)...done.
|
Loaded symbols for /lib64/libselinux.so.1
|
Reading symbols from /lib64/libsepol.so.1...(no debugging symbols found)...done.
|
Loaded symbols for /lib64/libsepol.so.1
|
Reading symbols from /lib64/libnss_files.so.2...(no debugging symbols found)...done.
|
Loaded symbols for /lib64/libnss_files.so.2
|
Reading symbols from /lib64/libnss_dns.so.2...(no debugging symbols found)...done.
|
Loaded symbols for /lib64/libnss_dns.so.2
|
Core was generated by `/usr/libexec/mysqld --basedir=/usr --datadir=/base/mysql --plugin-dir=/usr/lib6'.
|
Program terminated with signal 11, Segmentation fault.
|
#0 0x00002aea6bae1d02 in pthread_kill () from /lib64/libpthread.so.0
|
(gdb) bt
|
#0 0x00002aea6bae1d02 in pthread_kill () from /lib64/libpthread.so.0
|
#1 0x00002aea6a5daa8b in handle_fatal_signal (sig=11) at /usr/src/redhat/BUILD/mariadb-5.5.35/sql/signal_handler.cc:262
|
#2 <signal handler called>
|
#3 buf_pool_from_bpage (buf_pool=0x2aea8c416fd8, n_iterations=1048575) at /usr/src/redhat/BUILD/mariadb-5.5.35/storage/xtradb/include/buf0buf.ic:77
|
#4 buf_page_get_mutex (buf_pool=0x2aea8c416fd8, n_iterations=1048575) at /usr/src/redhat/BUILD/mariadb-5.5.35/storage/xtradb/include/buf0buf.ic:340
|
#5 buf_page_get_mutex_enter (buf_pool=0x2aea8c416fd8, n_iterations=1048575) at /usr/src/redhat/BUILD/mariadb-5.5.35/storage/xtradb/include/buf0buf.ic:372
|
#6 buf_LRU_free_from_common_LRU_list (buf_pool=0x2aea8c416fd8, n_iterations=1048575) at /usr/src/redhat/BUILD/mariadb-5.5.35/storage/xtradb/buf/buf0lru.c:1039
|
#7 buf_LRU_search_and_free_block (buf_pool=0x2aea8c416fd8, n_iterations=1048575) at /usr/src/redhat/BUILD/mariadb-5.5.35/storage/xtradb/buf/buf0lru.c:1104
|
#8 0x00002aea6a8b6b4f in buf_LRU_get_free_block (buf_pool=0x2aea8c416fd8) at /usr/src/redhat/BUILD/mariadb-5.5.35/storage/xtradb/buf/buf0lru.c:1343
|
#9 0x00002aea6a8a45df in buf_page_get_gen (space=1575020, zip_size=8192, offset=996276, rw_latch=1, guess=0x0, mode=10,
|
file=0x2aea6aa78458 "/usr/src/redhat/BUILD/mariadb-5.5.35/storage/xtradb/btr/btr0pcur.c", line=429, mtr=0x41762380)
|
at /usr/src/redhat/BUILD/mariadb-5.5.35/storage/xtradb/buf/buf0buf.c:2692
|
#10 0x00002aea6a897e56 in btr_block_get_func (cursor=0x2aaef47537b8, mtr=0x41762380) at /usr/src/redhat/BUILD/mariadb-5.5.35/storage/xtradb/include/btr0btr.ic:59
|
#11 btr_pcur_move_to_next_page (cursor=0x2aaef47537b8, mtr=0x41762380) at /usr/src/redhat/BUILD/mariadb-5.5.35/storage/xtradb/btr/btr0pcur.c:427
|
#12 0x00002aea6a84aebc in btr_pcur_move_to_next (buf=0x2aaef4847988 "\377Γ\211\001\270\210", mode=1, prebuilt=0x2aaef4753748, match_mode=0, direction=1)
|
at /usr/src/redhat/BUILD/mariadb-5.5.35/storage/xtradb/include/btr0pcur.ic:360
|
#13 row_search_for_mysql (buf=0x2aaef4847988 "\377Γ\211\001\270\210", mode=1, prebuilt=0x2aaef4753748, match_mode=0, direction=1)
|
at /usr/src/redhat/BUILD/mariadb-5.5.35/storage/xtradb/row/row0sel.c:4887
|
#14 0x00002aea6a8183d7 in ha_innobase::general_fetch (this=0x2aaef4947d08, buf=0x2aaef4847988 "\377Γ\211\001\270\210", direction=1, match_mode=0)
|
at /usr/src/redhat/BUILD/mariadb-5.5.35/storage/xtradb/handler/ha_innodb.cc:7326
|
#15 0x00002aea6a6ec01a in ha_rnd_next (info=0x2aaef464de68) at /usr/src/redhat/BUILD/mariadb-5.5.35/sql/sql_class.h:4313
|
#16 rr_sequential (info=0x2aaef464de68) at /usr/src/redhat/BUILD/mariadb-5.5.35/sql/records.cc:467
|
#17 0x00002aea6a4cd067 in sub_select (join=0x2aeaa88f2228, join_tab=0x2aaef464ddb8, end_of_records=<value optimized out>)
|
at /usr/src/redhat/BUILD/mariadb-5.5.35/sql/sql_select.cc:16808
|
#18 0x00002aea6a4cd4f2 in do_select (join=0x2aeaa88f2228, fields=0x2aaef4cab778, table=0x0, procedure=0x0) at /usr/src/redhat/BUILD/mariadb-5.5.35/sql/sql_select.cc:16451
|
#19 0x00002aea6a4dc611 in JOIN::exec (this=0x2aeaa88f2228) at /usr/src/redhat/BUILD/mariadb-5.5.35/sql/sql_select.cc:2859
|
#20 0x00002aea6a4deb19 in mysql_select (thd=0x2aaef4ca7df0, rref_pointer_array=0x2aaef4cab8d0, tables=0x2aeaa88f1b80, wild_num=1, fields=<value optimized out>, conds=0x0,
|
og_num=0, order=0x0, group=0x0, having=0x0, proc_param=0x0, select_options=2147781376, result=0x2aeaa88f2208, unit=0x2aaef4caaf88, select_lex=0x2aaef4cab660)
|
at /usr/src/redhat/BUILD/mariadb-5.5.35/sql/sql_select.cc:3079
|
#21 0x00002aea6a4df616 in handle_select (thd=0x2aaef4ca7df0, lex=0x2aaef4caaed8, result=0x2aeaa88f2208, setup_tables_done_option=0)
|
at /usr/src/redhat/BUILD/mariadb-5.5.35/sql/sql_select.cc:319
|
#22 0x00002aea6a48bd96 in execute_sqlcom_select (thd=0x2aaef4ca7df0, all_tables=0x2aeaa88f1b80) at /usr/src/redhat/BUILD/mariadb-5.5.35/sql/sql_parse.cc:4688
|
#23 0x00002aea6a4911cf in mysql_execute_command (thd=0x2aaef4ca7df0) at /usr/src/redhat/BUILD/mariadb-5.5.35/sql/sql_parse.cc:2232
|
#24 0x00002aea6a493bac in mysql_parse (thd=0x2aaef4ca7df0, rawbuf=<value optimized out>, length=<value optimized out>, parser_state=0x41764e80)
|
at /usr/src/redhat/BUILD/mariadb-5.5.35/sql/sql_parse.cc:5799
|
#25 0x00002aea6a4953e9 in dispatch_command (command=COM_QUERY, thd=0x2aaef4ca7df0,
|
packet=0x2aaef4cac581 "\270хся объектах;\n- Представление директору актов, справок, докладных записок об отклонении сроков и объемов вып", <incomplete sequence \320>..., packet_length=54) at /usr/src/redhat/BUILD/mariadb-5.5.35/sql/sql_parse.cc:1078
|
#26 0x00002aea6a495c82 in do_command (thd=0x2aaef4ca7df0) at /usr/src/redhat/BUILD/mariadb-5.5.35/sql/sql_parse.cc:793
|
#27 0x00002aea6a54b59f in do_handle_one_connection (thd_arg=0x2aaef4ca7df0) at /usr/src/redhat/BUILD/mariadb-5.5.35/sql/sql_connect.cc:1266
|
#28 0x00002aea6a54b6b4 in handle_one_connection (arg=<value optimized out>) at /usr/src/redhat/BUILD/mariadb-5.5.35/sql/sql_connect.cc:1181
|
#29 0x00002aea6badc73d in start_thread () from /lib64/libpthread.so.0
|
#30 0x00002aea6d140d1d in clone () from /lib64/libc.so.6
|
(gdb) q
|
mysql crashes on various tables:
egrep '^14|SQL_NO_CACHE' dbs01-02.err | grep SQL_NO_CACHE -B1
|
140131 0:44:43 [ERROR] mysqld got signal 11 ;
|
Query (0x2b51a86f2b98): SELECT /*!40001 SQL_NO_CACHE */ * FROM `resume_sphinx`
|
--
|
140210 0:45:02 [ERROR] mysqld got signal 11 ;
|
Query (0x2b85bc50e488): SELECT /*!40001 SQL_NO_CACHE */ * FROM `received_resumes`
|
--
|
140212 0:42:20 [ERROR] mysqld got signal 11 ;
|
Query (0x2b8e04b6a0e8): SELECT /*!40001 SQL_NO_CACHE */ * FROM `education`
|
--
|
140213 1:24:45 [ERROR] mysqld got signal 6 ; # this is described in MDEV-5670
|
Query (0x2aaf185facf8): SELECT /*!40001 SQL_NO_CACHE */ * FROM `received_resumes_events`
|
--
|
140213 17:34:05 [ERROR] mysqld got signal 11 ;
|
Query (0x2aeaa88f1938): SELECT /*!40001 SQL_NO_CACHE */ * FROM `resume_sphinx`
|
Thank you for all the data.
The more I'm looking at it the more I am inclined to think that the queries that the server prints upon the crash in this case are irrelevant (unfortunately). Stack traces just don't make sense for SELECT * queries from a single table, there should be no ranges, joins and subqueries that the stack trace pretends to be processing.
I'll get the second opinion in this regard from optimizer team, but I think the better lead so far is the error message and the assertion failure in the other report, MDEV-5670. They most likely have the same root cause, obviously a race condition of some sort, and it's just the matter of probability whether the server is shutdown by InnoDB due to the assertion failure, or crashes hard with the segfault.
But until it's confirmed somehow, I will keep both reports open.
Elena, thank you for fast feedback.
Is it need to upload core dump for this crash to ftp.askmonty.org? Compressed tarball is big enough, about 5Gb. mysqld same as in https://mariadb.atlassian.net/browse/MDEV-5670.
Could you please take a look at the error logs in this bug report?
Before I assign it to Jan as a pure InnoDB failure, I want to confirm that the crashes have nothing to do with optimizer.
I can't see any correlation between stack traces and the query that the server allegedly crashes on, but maybe you will.
Elena, thank you for fast feedback.
Is it need to upload core dump for this crash to ftp.askmonty.org? Compressed tarball is big enough, about 5Gb. mysqld same as in https://mariadb.atlassian.net/browse/MDEV-5670.
Hi Sergey,
If possible, please keep it for now. I think for the initial quick analysis the stack traces you pasted should be enough; when/if it comes to the point when the coredump is needed, we'll know that you have it and will ask to upload.
Thanks.
Ok, I will keep it.
As discussed with psergey, the problem is most likely unrelated to optimizer, since the crash is happening deep inside XtraDB, and also taking into account the assertion failure in MDEV-5670.
Reassigning to Jan to take a look from InnoDB side.
Yet another crash on our backup server (see https://mariadb.atlassian.net/browse/MDEV-5670?focusedCommentId=41630&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-41630 ):
140215 1:10:42 [ERROR] mysqld got signal 11 ;
|
This could be because you hit a bug. It is also possible that this binary
|
or one of the libraries it was linked against is corrupt, improperly built,
|
or misconfigured. This error can also be caused by malfunctioning hardware.
|
|
|
To report this bug, see http://kb.askmonty.org/en/reporting-bugs
|
|
|
We will try our best to scrape up some info that will hopefully help
|
diagnose the problem, but since we have already crashed,
|
something is definitely wrong and this may fail.
|
|
|
Server version: 5.5.35-MariaDB-log
|
key_buffer_size=134217728
|
read_buffer_size=131072
|
max_used_connections=11
|
max_threads=502
|
thread_count=8
|
It is possible that mysqld could use up to
|
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 2260475 K bytes of memory
|
Hope that's ok; if not, decrease some variables in the equation.
|
|
|
Thread pointer: 0x0x2ab5116fd0f0
|
Attempting backtrace. You can use the following information to find out
|
where mysqld died. If you see no messages after this, something went
|
terribly wrong...
|
stack_bottom = 0x4c3fc0a0 thread_stack 0x48000
|
(my_addr_resolve failure: fork)
|
/usr/libexec/mysqld(my_print_stacktrace+0x2e) [0x2ab4d3b119ee]
|
/usr/libexec/mysqld(handle_fatal_signal+0x4d1) [0x2ab4d3741b01]
|
/lib64/libpthread.so.0 [0x2ab4d4c4bb10]
|
/usr/lib64/libz.so.1(adler32+0x21b) [0x2ab4d4e5a16b]
|
/usr/libexec/mysqld [0x2ab4d3aa4f76]
|
/usr/libexec/mysqld [0x2ab4d3a1c69b]
|
/usr/libexec/mysqld [0x2ab4d3a1d407]
|
/usr/libexec/mysqld [0x2ab4d3a1db4f]
|
/usr/libexec/mysqld [0x2ab4d3a0b5df]
|
/usr/libexec/mysqld [0x2ab4d39fee56]
|
/usr/libexec/mysqld [0x2ab4d39b1ebc]
|
/usr/libexec/mysqld [0x2ab4d397f3d7]
|
/usr/libexec/mysqld(rr_sequential(READ_RECORD*)+0xba) [0x2ab4d385301a]
|
/usr/libexec/mysqld(sub_select(JOIN*, st_join_table*, bool)+0x137) [0x2ab4d3634067]
|
/usr/libexec/mysqld [0x2ab4d36344f2]
|
/usr/libexec/mysqld(JOIN::exec()+0xa81) [0x2ab4d3643611]
|
/usr/libexec/mysqld(mysql_select(THD*, Item***, TABLE_LIST*, unsigned int, List<Item>&, Item*, unsigned int, st_order*, st_order*, Item*, st_order*, unsigned long long, select_result*, st_select_lex_unit*, st_select_lex*)+0x1a9) [0x2ab4d3645b19]
|
/usr/libexec/mysqld(handle_select(THD*, LEX*, select_result*, unsigned long)+0x2b6) [0x2ab4d3646616]
|
/usr/libexec/mysqld [0x2ab4d35f2d96]
|
/usr/libexec/mysqld(mysql_execute_command(THD*)+0x423f) [0x2ab4d35f81cf]
|
/usr/libexec/mysqld(mysql_parse(THD*, char*, unsigned int, Parser_state*)+0x1cc) [0x2ab4d35fabac]
|
/usr/libexec/mysqld(dispatch_command(enum_server_command, THD*, char*, unsigned int)+0x1829) [0x2ab4d35fc3e9]
|
/usr/libexec/mysqld(do_command(THD*)+0xd2) [0x2ab4d35fcc82]
|
/usr/libexec/mysqld(do_handle_one_connection(THD*)+0x12f) [0x2ab4d36b259f]
|
/usr/libexec/mysqld(handle_one_connection+0x54) [0x2ab4d36b26b4]
|
/lib64/libpthread.so.0 [0x2ab4d4c4373d]
|
/lib64/libc.so.6(clone+0x6d) [0x2ab4d62a7d1d]
|
|
|
Trying to get some variables.
|
Some pointers may be invalid and cause the dump to abort.
|
Query (0x2aaef93c7b78): SELECT /*!40001 SQL_NO_CACHE */ * FROM `resume_sphinx`
|
Connection ID (thread ID): 11820
|
Status: NOT_KILLED
|
|
|
Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=off,table_elimination=on,extended_keys=off
|
|
|
The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
|
information that should help you find out what is causing the crash.
|
Writing a core file
|
140215 01:11:46 mysqld_safe Number of processes running now: 0
|
140215 01:11:47 mysqld_safe mysqld restarted
|
# gdb /usr/libexec/mysqld core.24334
|
GNU gdb (GDB) Red Hat Enterprise Linux (7.0.1-23.el5_5.1)
|
Copyright (C) 2009 Free Software Foundation, Inc.
|
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
|
This is free software: you are free to change and redistribute it.
|
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
|
and "show warranty" for details.
|
This GDB was configured as "x86_64-redhat-linux-gnu".
|
For bug reporting instructions, please see:
|
<http://www.gnu.org/software/gdb/bugs/>...
|
Reading symbols from /usr/libexec/mysqld...done.
|
[New Thread 8207]
|
[New Thread 7916]
|
[New Thread 7915]
|
[New Thread 7914]
|
[New Thread 7913]
|
[New Thread 7912]
|
[New Thread 7064]
|
[New Thread 26688]
|
[New Thread 24955]
|
[New Thread 24954]
|
[New Thread 24662]
|
[New Thread 24661]
|
[New Thread 24660]
|
[New Thread 24659]
|
[New Thread 24658]
|
[New Thread 24657]
|
[New Thread 24656]
|
[New Thread 24645]
|
[New Thread 24644]
|
[New Thread 24643]
|
[New Thread 24642]
|
[New Thread 24641]
|
[New Thread 24640]
|
[New Thread 24639]
|
[New Thread 24638]
|
[New Thread 24637]
|
[New Thread 24636]
|
[New Thread 24613]
|
[New Thread 24334]
|
Reading symbols from /lib64/libpthread.so.0...(no debugging symbols found)...done.
|
Loaded symbols for /lib64/libpthread.so.0
|
Reading symbols from /usr/lib64/libz.so.1...(no debugging symbols found)...done.
|
Loaded symbols for /usr/lib64/libz.so.1
|
Reading symbols from /lib64/librt.so.1...(no debugging symbols found)...done.
|
Loaded symbols for /lib64/librt.so.1
|
Reading symbols from /lib64/libcrypt.so.1...(no debugging symbols found)...done.
|
Loaded symbols for /lib64/libcrypt.so.1
|
Reading symbols from /lib64/libdl.so.2...(no debugging symbols found)...done.
|
Loaded symbols for /lib64/libdl.so.2
|
Reading symbols from /lib64/libssl.so.6...(no debugging symbols found)...done.
|
Loaded symbols for /lib64/libssl.so.6
|
Reading symbols from /lib64/libcrypto.so.6...(no debugging symbols found)...done.
|
Loaded symbols for /lib64/libcrypto.so.6
|
Reading symbols from /usr/lib64/libstdc++.so.6...(no debugging symbols found)...done.
|
Loaded symbols for /usr/lib64/libstdc++.so.6
|
Reading symbols from /lib64/libm.so.6...(no debugging symbols found)...done.
|
Loaded symbols for /lib64/libm.so.6
|
Reading symbols from /lib64/libc.so.6...(no debugging symbols found)...done.
|
Loaded symbols for /lib64/libc.so.6
|
Reading symbols from /lib64/ld-linux-x86-64.so.2...(no debugging symbols found)...done.
|
Loaded symbols for /lib64/ld-linux-x86-64.so.2
|
Reading symbols from /usr/lib64/libgssapi_krb5.so.2...(no debugging symbols found)...done.
|
Loaded symbols for /usr/lib64/libgssapi_krb5.so.2
|
Reading symbols from /usr/lib64/libkrb5.so.3...(no debugging symbols found)...done.
|
Loaded symbols for /usr/lib64/libkrb5.so.3
|
Reading symbols from /lib64/libcom_err.so.2...(no debugging symbols found)...done.
|
Loaded symbols for /lib64/libcom_err.so.2
|
Reading symbols from /usr/lib64/libk5crypto.so.3...(no debugging symbols found)...done.
|
Loaded symbols for /usr/lib64/libk5crypto.so.3
|
Reading symbols from /lib64/libgcc_s.so.1...(no debugging symbols found)...done.
|
Loaded symbols for /lib64/libgcc_s.so.1
|
Reading symbols from /usr/lib64/libkrb5support.so.0...(no debugging symbols found)...done.
|
Loaded symbols for /usr/lib64/libkrb5support.so.0
|
Reading symbols from /lib64/libkeyutils.so.1...(no debugging symbols found)...done.
|
Loaded symbols for /lib64/libkeyutils.so.1
|
Reading symbols from /lib64/libresolv.so.2...(no debugging symbols found)...done.
|
Loaded symbols for /lib64/libresolv.so.2
|
Reading symbols from /lib64/libselinux.so.1...(no debugging symbols found)...done.
|
Loaded symbols for /lib64/libselinux.so.1
|
Reading symbols from /lib64/libsepol.so.1...(no debugging symbols found)...done.
|
Loaded symbols for /lib64/libsepol.so.1
|
Reading symbols from /lib64/libnss_files.so.2...(no debugging symbols found)...done.
|
Loaded symbols for /lib64/libnss_files.so.2
|
Reading symbols from /lib64/libnss_dns.so.2...(no debugging symbols found)...done.
|
Loaded symbols for /lib64/libnss_dns.so.2
|
Core was generated by `/usr/libexec/mysqld --basedir=/usr --datadir=/base/mysql --plugin-dir=/usr/lib6'.
|
Program terminated with signal 11, Segmentation fault.
|
#0 0x00002ab4d4c48d02 in pthread_kill () from /lib64/libpthread.so.0
|
(gdb) bt
|
#0 0x00002ab4d4c48d02 in pthread_kill () from /lib64/libpthread.so.0
|
#1 0x00002ab4d3741a8b in handle_fatal_signal (sig=11) at /usr/src/redhat/BUILD/mariadb-5.5.35/sql/signal_handler.cc:262
|
#2 <signal handler called>
|
#3 0x00002ab4d4e5a16b in adler32 () from /usr/lib64/libz.so.1
|
#4 0x00002ab4d3aa4f76 in page_zip_calc_checksum (data=0x0, size=0) at /usr/src/redhat/BUILD/mariadb-5.5.35/storage/xtradb/page/page0zip.c:4784
|
#5 0x00002ab4d3a1c69b in buf_LRU_free_block (bpage=0x2aaac4c288c0, zip=0, have_LRU_mutex=0x4c3f8f60) at /usr/src/redhat/BUILD/mariadb-5.5.35/storage/xtradb/buf/buf0lru.c:2096
|
#6 0x00002ab4d3a1d407 in buf_LRU_free_from_unzip_LRU_list (buf_pool=0x2ab4f5e47fd8, n_iterations=1) at /usr/src/redhat/BUILD/mariadb-5.5.35/storage/xtradb/buf/buf0lru.c:997
|
#7 buf_LRU_search_and_free_block (buf_pool=0x2ab4f5e47fd8, n_iterations=1) at /usr/src/redhat/BUILD/mariadb-5.5.35/storage/xtradb/buf/buf0lru.c:1101
|
#8 0x00002ab4d3a1db4f in buf_LRU_get_free_block (buf_pool=0x2ab4f5e47fd8) at /usr/src/redhat/BUILD/mariadb-5.5.35/storage/xtradb/buf/buf0lru.c:1343
|
#9 0x00002ab4d3a0b5df in buf_page_get_gen (space=1575020, zip_size=8192, offset=1146045, rw_latch=1, guess=0x0, mode=10,
|
file=0x2ab4d3bdf458 "/usr/src/redhat/BUILD/mariadb-5.5.35/storage/xtradb/btr/btr0pcur.c", line=429, mtr=0x4c3f9380)
|
at /usr/src/redhat/BUILD/mariadb-5.5.35/storage/xtradb/buf/buf0buf.c:2692
|
#10 0x00002ab4d39fee56 in btr_block_get_func (cursor=0x2aaef818b478, mtr=0x4c3f9380) at /usr/src/redhat/BUILD/mariadb-5.5.35/storage/xtradb/include/btr0btr.ic:59
|
#11 btr_pcur_move_to_next_page (cursor=0x2aaef818b478, mtr=0x4c3f9380) at /usr/src/redhat/BUILD/mariadb-5.5.35/storage/xtradb/btr/btr0pcur.c:427
|
#12 0x00002ab4d39b1ebc in btr_pcur_move_to_next (buf=0x2aaef816ccc8 "\377\261v\177\001", mode=1, prebuilt=0x2aaef818b408, match_mode=0, direction=1)
|
at /usr/src/redhat/BUILD/mariadb-5.5.35/storage/xtradb/include/btr0pcur.ic:360
|
#13 row_search_for_mysql (buf=0x2aaef816ccc8 "\377\261v\177\001", mode=1, prebuilt=0x2aaef818b408, match_mode=0, direction=1)
|
at /usr/src/redhat/BUILD/mariadb-5.5.35/storage/xtradb/row/row0sel.c:4887
|
#14 0x00002ab4d397f3d7 in ha_innobase::general_fetch (this=0x2aaef82397c8, buf=0x2aaef816ccc8 "\377\261v\177\001", direction=1, match_mode=0)
|
at /usr/src/redhat/BUILD/mariadb-5.5.35/storage/xtradb/handler/ha_innodb.cc:7326
|
#15 0x00002ab4d385301a in ha_rnd_next (info=0x2aaef838c548) at /usr/src/redhat/BUILD/mariadb-5.5.35/sql/sql_class.h:4313
|
#16 rr_sequential (info=0x2aaef838c548) at /usr/src/redhat/BUILD/mariadb-5.5.35/sql/records.cc:467
|
#17 0x00002ab4d3634067 in sub_select (join=0x2aaef93c8468, join_tab=0x2aaef838c498, end_of_records=<value optimized out>)
|
at /usr/src/redhat/BUILD/mariadb-5.5.35/sql/sql_select.cc:16808
|
#18 0x00002ab4d36344f2 in do_select (join=0x2aaef93c8468, fields=0x2ab511700a78, table=0x0, procedure=0x0) at /usr/src/redhat/BUILD/mariadb-5.5.35/sql/sql_select.cc:16451
|
#19 0x00002ab4d3643611 in JOIN::exec (this=0x2aaef93c8468) at /usr/src/redhat/BUILD/mariadb-5.5.35/sql/sql_select.cc:2859
|
#20 0x00002ab4d3645b19 in mysql_select (thd=0x2ab5116fd0f0, rref_pointer_array=0x2ab511700bd0, tables=0x2aaef93c7dc0, wild_num=1, fields=<value optimized out>, conds=0x0,
|
og_num=0, order=0x0, group=0x0, having=0x0, proc_param=0x0, select_options=2147781376, result=0x2aaef93c8448, unit=0x2ab511700288, select_lex=0x2ab511700960)
|
at /usr/src/redhat/BUILD/mariadb-5.5.35/sql/sql_select.cc:3079
|
#21 0x00002ab4d3646616 in handle_select (thd=0x2ab5116fd0f0, lex=0x2ab5117001d8, result=0x2aaef93c8448, setup_tables_done_option=0)
|
at /usr/src/redhat/BUILD/mariadb-5.5.35/sql/sql_select.cc:319
|
#22 0x00002ab4d35f2d96 in execute_sqlcom_select (thd=0x2ab5116fd0f0, all_tables=0x2aaef93c7dc0) at /usr/src/redhat/BUILD/mariadb-5.5.35/sql/sql_parse.cc:4688
|
#23 0x00002ab4d35f81cf in mysql_execute_command (thd=0x2ab5116fd0f0) at /usr/src/redhat/BUILD/mariadb-5.5.35/sql/sql_parse.cc:2232
|
#24 0x00002ab4d35fabac in mysql_parse (thd=0x2ab5116fd0f0, rawbuf=<value optimized out>, length=<value optimized out>, parser_state=0x4c3fbe80)
|
at /usr/src/redhat/BUILD/mariadb-5.5.35/sql/sql_parse.cc:5799
|
#25 0x00002ab4d35fc3e9 in dispatch_command (command=COM_QUERY, thd=0x2ab5116fd0f0,
|
packet=0x2ab511a6f401 "589\n1390220989\n1391283511\001\060\001\061\001\064\001\061\001\060\002\067\060\001\060\001\060\001\060", packet_length=54)
|
at /usr/src/redhat/BUILD/mariadb-5.5.35/sql/sql_parse.cc:1078
|
#26 0x00002ab4d35fcc82 in do_command (thd=0x2ab5116fd0f0) at /usr/src/redhat/BUILD/mariadb-5.5.35/sql/sql_parse.cc:793
|
#27 0x00002ab4d36b259f in do_handle_one_connection (thd_arg=0x2aaef4286a90) at /usr/src/redhat/BUILD/mariadb-5.5.35/sql/sql_connect.cc:1266
|
#28 0x00002ab4d36b26b4 in handle_one_connection (arg=<value optimized out>) at /usr/src/redhat/BUILD/mariadb-5.5.35/sql/sql_connect.cc:1181
|
#29 0x00002ab4d4c4373d in start_thread () from /lib64/libpthread.so.0
|
#30 0x00002ab4d62a7d1d in clone () from /lib64/libc.so.6
|
(gdb) q
|
CREATE TABLE `resume_sphinx` (
|
`id` int(10) unsigned NOT NULL DEFAULT '0',
|
`payment` int(10) unsigned NOT NULL DEFAULT '0',
|
`town` smallint(5) unsigned NOT NULL DEFAULT '0',
|
`type_of_work` tinyint(3) unsigned NOT NULL DEFAULT '0',
|
`place_of_work` tinyint(3) unsigned NOT NULL DEFAULT '0',
|
`firstname` varchar(20) NOT NULL DEFAULT '',
|
`lastname` varchar(30) NOT NULL DEFAULT '',
|
`middlname` varchar(20) NOT NULL DEFAULT '',
|
`pol` tinyint(3) unsigned NOT NULL DEFAULT '0',
|
`birthday` tinyint(3) unsigned NOT NULL DEFAULT '0',
|
`birthmonth` tinyint(3) unsigned NOT NULL DEFAULT '0',
|
`birthyear` year(4) NOT NULL DEFAULT '0000',
|
`age` tinyint(3) unsigned NOT NULL DEFAULT '0',
|
`maritalstatus` tinyint(3) unsigned NOT NULL DEFAULT '0',
|
`children` tinyint(3) unsigned NOT NULL DEFAULT '0',
|
`address` varchar(90) NOT NULL DEFAULT '',
|
`phone1` varchar(35) NOT NULL,
|
`timebeg1` varchar(5) NOT NULL DEFAULT '',
|
`timeend1` varchar(5) NOT NULL DEFAULT '',
|
`phone2` varchar(35) NOT NULL,
|
`timebeg2` varchar(5) NOT NULL DEFAULT '',
|
`timeend2` varchar(5) NOT NULL DEFAULT '',
|
`receive_sms` tinyint(3) unsigned NOT NULL,
|
`email1` varchar(100) NOT NULL DEFAULT '',
|
`computer` mediumtext NOT NULL,
|
`best` mediumtext NOT NULL,
|
`dop` mediumtext NOT NULL,
|
`driving_licence` set('A','B','C','D','E') NOT NULL,
|
`region` tinyint(3) unsigned NOT NULL DEFAULT '0',
|
`date1` int(11) NOT NULL DEFAULT '0',
|
`lastedit` int(11) NOT NULL DEFAULT '0',
|
`datepub` int(11) NOT NULL DEFAULT '0',
|
`date_closed` int(10) unsigned NOT NULL DEFAULT '0',
|
`published` tinyint(4) NOT NULL DEFAULT '0',
|
`education` tinyint(3) unsigned NOT NULL DEFAULT '0',
|
`comp` tinyint(3) unsigned NOT NULL DEFAULT '0',
|
`archive` tinyint(4) NOT NULL DEFAULT '0',
|
`rating` tinyint(4) NOT NULL DEFAULT '0',
|
`moveable` tinyint(3) unsigned NOT NULL DEFAULT '0',
|
`business_trip` tinyint(3) unsigned NOT NULL DEFAULT '0',
|
`id_vacancy` int(10) unsigned NOT NULL DEFAULT '0',
|
`other_contacts` varchar(200) NOT NULL DEFAULT '',
|
`recomendations` mediumtext NOT NULL,
|
`base_education` mediumtext NOT NULL,
|
`work` mediumtext NOT NULL,
|
`eduform` tinyint(3) unsigned NOT NULL DEFAULT '0',
|
`workname` mediumtext NOT NULL,
|
`employed` tinyint(4) NOT NULL,
|
`reg_user_id` int(10) unsigned NOT NULL,
|
`portfolio` tinyint(3) unsigned NOT NULL,
|
`has_photo` tinyint(3) unsigned NOT NULL,
|
`other_text` mediumtext NOT NULL,
|
`town_sph` int(10) unsigned NOT NULL DEFAULT '0',
|
`driving_licence_sph` tinyint(3) unsigned NOT NULL DEFAULT '0',
|
`last_profession` varchar(127) NOT NULL DEFAULT '',
|
`desired_profession` mediumtext NOT NULL,
|
`all_achievments` mediumtext NOT NULL,
|
`not_last_professions` mediumtext,
|
PRIMARY KEY (`id`)
|
) ENGINE=InnoDB DEFAULT CHARSET=utf8 ROW_FORMAT=COMPRESSED KEY_BLOCK_SIZE=8
|
Hi,
After fist look, it looks like that database could be corrupted. It tries to calculate checksum for a null pointer and zero size. As far as I understand this piece of code, that should not be possible.
R: Jan
Hi,
what is possible root of database corruption? Is it corrupted buffer_pool or ibd-files? At some days backup finished successful without crashing:
140131 0:44:43 [ERROR] mysqld got signal 11 ;
|
140202 0:36:56 [ERROR] mysqld got signal 11 ;
|
140208 0:37:10 [ERROR] mysqld got signal 11 ;
|
140210 0:45:02 [ERROR] mysqld got signal 11 ;
|
140211 2:19:29 [ERROR] mysqld got signal 11 ;
|
140212 0:42:20 [ERROR] mysqld got signal 11 ;
|
140213 1:24:45 [ERROR] mysqld got signal 6 ;
|
140213 17:34:05 [ERROR] mysqld got signal 11 ;
|
140215 1:10:42 [ERROR] mysqld got signal 11 ;
|
140216 1:18:29 [ERROR] mysqld got signal 11 ;
|
As we see 1,3-7,9,14,17 Febuary mysqld has not crashed (we do backup every day at night). So I think, ibd-files is not corrupted. How I can check its for corruption?
I do not think it is probable that pages on buffer pool only are corrupted, You have compressed table but when it tries to free a block code does not find compressed data and its size. You could try check table <table_name> extended;
R: Jan
Jan, thank you for answer.
Command "check table <table> extended" not found any errors, but innocheck found:
innochecksum -v -d resume_sphinx.ibd
|
file resume_sphinx.ibd = 9554624512 bytes (583168 pages)...
|
checking pages in range 0 to 583167
|
page 0: log sequence number: first = 1745798819; second = 0
|
page 0 invalid (fails log sequence number check)
|
Also i found some small tables with similar results, but select * from <table> return all data without crash. How is it possible?
And main question - how i can fix these errors?
If checksum mismatches are found, you would normally restore the tablespace from backup or start the server and attempt to use mysqldump to make a backup of the tables within the tablespace. If mysqldump does not work, and select * does, you can alway construct a SQL-clause to dump all rows directly to insert format. I.e.
SELECT 'INSERT INTO TABLE VALUES('||num_column||,'||char_column||',...
I have been reproducing similar crashes. Just turning on Barracuda and testing with compression, so I assume it's more related to that than 5.5.35. Servers without Barracuda format have been running fine.
140219 12:14:37 [ERROR] mysqld got signal 11 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
To report this bug, see http://kb.askmonty.org/en/reporting-bugs
We will try our best to scrape up some info that will hopefully help
diagnose the problem, but since we have already crashed,
something is definitely wrong and this may fail.
Server version: 5.5.35-MariaDB
key_buffer_size=33554432
read_buffer_size=131072
max_used_connections=143
max_threads=5002
thread_count=124
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 11006512 K bytes of memory
Hope that's ok; if not, decrease some variables in the equation.
Thread pointer: 0x0x7f0c6c433000
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 0x7f0c74030d58 thread_stack 0x40000
(my_addr_resolve failure: fork)
/usr/sbin/mysqld(my_print_stacktrace+0x2e) [0xa8c62e]
/usr/sbin/mysqld(handle_fatal_signal+0x40b) [0x6d2f8b]
/lib64/libpthread.so.0(+0xf710) [0x7f1339a75710]
/lib64/libz.so.1(adler32+0x54) [0x7f1339852244]
/usr/sbin/mysqld() [0x953ff6]
/usr/sbin/mysqld() [0x8d2378]
/usr/sbin/mysqld() [0x8d2fe7]
/usr/sbin/mysqld() [0x8d3544]
/usr/sbin/mysqld() [0x8c9a54]
/usr/sbin/mysqld() [0x908248]
/usr/sbin/mysqld() [0x90b5d3]
/usr/sbin/mysqld() [0x90bf85]
/usr/sbin/mysqld() [0x89c01a]
/usr/sbin/mysqld() [0x88bb1d]
/usr/sbin/mysqld() [0x8b28ba]
/usr/sbin/mysqld() [0x964fdd]
/usr/sbin/mysqld() [0x9670ef]
/usr/sbin/mysqld() [0x85d7b6]
/usr/sbin/mysqld() [0x84589e]
/usr/sbin/mysqld(handler::ha_write_row(unsigned char*)+0x6d) [0x6d854d]
/usr/sbin/mysqld() [0xa68f01]
/usr/sbin/mysqld(handler::ha_write_row(unsigned char*)+0x6d) [0x6d854d]
/usr/sbin/mysqld(write_record(THD*, TABLE*, st_copy_info*)+0x61) [0x5704c1]
/usr/sbin/mysqld(select_insert::send_data(List<Item>&)+0xaf) [0x570eef]
/usr/sbin/mysqld() [0x5b9304]
/usr/sbin/mysqld() [0x5ace1f]
/usr/sbin/mysqld(sub_select(JOIN*, st_join_table*, bool)+0x213) [0x5b23b3]
/usr/sbin/mysqld() [0x5ba2ed]
/usr/sbin/mysqld(JOIN::exec()+0xe84) [0x5db724]
/usr/sbin/mysqld(mysql_select(THD*, Item**, TABLE_LIST, unsigned int, List<Item>&, Item*, unsigned int, st_order*, st_order*, Item*, st_order*, unsigned long long, select_result*, st_select_lex_unit*, st_select_lex*)+0x128) [0x5dd098]
/usr/sbin/mysqld(handle_select(THD*, LEX*, select_result*, unsigned long)+0x2b3) [0x5ddd33]
/usr/sbin/mysqld(mysql_execute_command(THD*)+0x515a) [0x58f13a]
/usr/sbin/mysqld(mysql_parse(THD*, char*, unsigned int, Parser_state*)+0x299) [0x58fd39]
/usr/sbin/mysqld(dispatch_command(enum_server_command, THD*, char*, unsigned int)+0x14b0) [0x5912b0]
/usr/sbin/mysqld(do_handle_one_connection(THD*)+0x3ef) [0x64ccbf]
/usr/sbin/mysqld(handle_one_connection+0x4c) [0x64cd5c]
/lib64/libpthread.so.0(+0x79d1) [0x7f1339a6d9d1]
/lib64/libc.so.6(clone+0x6d) [0x7f133838cb6d]
Trying to get some variables.
Some pointers may be invalid and cause the dump to abort.
Query (0x7f0c63bfc018): is an invalid pointer
Connection ID (thread ID): 23261
Status: NOT_KILLED
Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=off,table_elimination=on,extended_keys=off
The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
information that should help you find out what is causing the crash.
140219 12:14:39 mysqld_safe Number of processes running now: 0
140219 12:14:39 mysqld_safe mysqld restarted
140219 12:14:39 InnoDB: The InnoDB memory heap is disabled
140219 12:14:39 InnoDB: Mutexes and rw_locks use GCC atomic builtins
140219 12:14:39 InnoDB: Compressed tables use zlib 1.2.3
140219 12:14:39 InnoDB: Using Linux native AIO
140219 12:14:39 InnoDB: Initializing buffer pool, size = 24.4G
140219 12:14:40 InnoDB: Completed initialization of buffer pool
140219 12:14:40 InnoDB: highest supported file format is Barracuda.
InnoDB: Log scan progressed past the checkpoint lsn 41729726673238
140219 12:14:40 InnoDB: Database was not shut down normally!
InnoDB: Starting crash recovery.
...
This table is about over 20gb with 17 partitions by hash, but all the DB on production is about 1tb.
Busy production server, but nothing else should be going against the tables testing with compression. Tests were typically alter table changing key_block_size or insert into newtable select * from oldtable.
Unfortunately it is reproducible on production (several crashes last 24 hours). Will try to reproduce on a test box...
test server failed, but crashed differently...
...
InnoDB: ###### Diagnostic info printed to the standard error stream
InnoDB: Error: semaphore wait has lasted > 600 seconds
InnoDB: We intentionally crash the server, because it appears to be hung.
140219 14:31:38 InnoDB: Assertion failure in thread 140211126265600 in file srv0srv.c line 2976
InnoDB: We intentionally generate a memory trap.
InnoDB: Submit a detailed bug report to http://bugs.mysql.com.
InnoDB: If you get repeated assertion failures or crashes, even
InnoDB: immediately after the mysqld startup, there may be
InnoDB: corruption in the InnoDB tablespace. Please refer to
InnoDB: http://dev.mysql.com/doc/refman/5.5/en/forcing-innodb-recovery.html
InnoDB: about forcing recovery.
140219 14:31:38 [ERROR] mysqld got signal 6 ;
This could be because you hit a bug. It is also possible that this binary
...
Table was fresh from mysqldump, so any corruption should be cleaned up (unless hardware failure, however hardware problems unlikely as everything is ecc, etc).
Forgot to mention what I did for the test...
Imported the large table into the test server with a KEY_BLOCK_SIZE=8. That went fine.
Created empty tables with KEY_BLOCK_SIZE=4 and no compression or KEY_BLOCK_SIZE.
Simultaneously ran INSERT into newtable select * from importedtable;
It did finish cloning the data from the 8k to uncompressed. After that copy finished, I added another table with KEY_BLOCK_SIZE 8k and did the insert for that 4th table.
When it crashed it was on p3 of the copy to table with KEY_BLOCK_SIZE=4, and p5 of the copy to the 4th table.
Using KEY_BLOCK_SIZE=4k seems to increase the instability (and be unacceptably slower), but seems to happen with 8k too. Test server has 5000M for innodb_buffer_pool_size and the uncompressed table is 23gb over 17 partitions, with 8K block, it's 12gb.
innochecksum does not work with compressed tables (all tables in crashes are compressed):
http://bugs.mysql.com/bug.php?id=66779 - fixed in 5.7.2, but not in 5.5
https://bugs.launchpad.net/percona-server/+bug/1100652
So I suppose all ibd-files are consistent ("check table <table> extended" not found any errors).
Yesterday I did convertation of all compressed tables to ROW_FORMAT=DYNAMIC in sequence without any other queries, in other words "ALTER TABLE <table> ROW_FORMAT=Dynamic" was single running query. All tables were converted successfully. Nightly backup finished without crashes. I did manual backup this morning, it finished successfully too.
Today I did another test: on another server with original compressed tables I did run four simultaneous "ALTER TABLE <table> ROW_FORMAT=Dynamic" (one for each compressed table), and in 10-20 minutes mysqld crashed.
As I see mysqld crashed on simultaneous queries to big compressed tables in circumstances of intensively repopulating of buffer_pool. May be it race condition as in https://bugs.launchpad.net/percona-server/+bug/1227581 ?
Jan,
I have continued tests on backup server and converted tables (which were converted from compresses to dynamic earlier) today from ROW_FORMAT=Dynamic to ROW_FORMAT=Compressed. After that I have run backup utility in eight threads and mysqld crashed on just created compressed tables.
So I think corruption of ibd-files is not root of this issue.
I agree, corruption is not the root cause, or simply using compression causes corruption. Would it be worth testing earlier versions and/or 10.x?
Yes, I shall try to test other versions later.
Hi,
Based on tests it is clear that the problem is caused on some bug on xtradb somewhere on buf0buf, buf0lru files.
R: Jan
May problem be caused by any background threads?
Hi,
I do not think so, somehow memory image of one/or all compressed tables becomes corrupted. This is due a bug on
buffer pool handling.
R: Jan
Hi,
I have wrote script that reproduces crash:
$ cat test.sh
|
#!/bin/bash
|
|
|
host="-hlocalhost "
|
mcmd="mysql $host "
|
|
|
case "$1" in
|
c|create)
|
(
|
|
|
echo "use test;"
|
|
|
for j in 1 2 3 4; do
|
cat <<EOF
|
DROP TABLE IF EXISTS test.t$j ;
|
CREATE TABLE t$j (
|
id int(10) unsigned NOT NULL AUTO_INCREMENT,
|
a mediumtext,
|
PRIMARY KEY (id)
|
) ENGINE=innodb DEFAULT CHARSET=utf8 ROW_FORMAT=COMPRESSED KEY_BLOCK_SIZE=8;
|
EOF
|
|
|
done
|
|
|
echo -e "insert into t1 (a) values ('"
|
cat /dev/urandom | dd bs=8k count=1 2>&1 | hexdump
|
echo "'),(';"
|
cat /dev/urandom | dd bs=8k count=1 2>&1 | hexdump
|
echo "');"
|
) | $mcmd
|
|
|
for i in `seq 1 8`; do
|
(
|
echo "insert into t2 select 0,a from t1; insert into t2 select 0,a from t1;"
|
echo "insert into t1 select 0,a from t2; insert into t1 select 0,a from t2;"
|
|
|
echo "insert into t3 select 0,a from t2; insert into t3 select 0,a from t2;"
|
echo "insert into t4 select 0,a from t2; insert into t4 select 0,a from t2;"
|
) | $mcmd test
|
echo -n .
|
done
|
|
|
;;
|
l|load)
|
|
|
for i in 1 2 3; do
|
echo "step $i"
|
echo "first";
|
for j in 1 2 3 4; do
|
mysqldump $host test t$j >/dev/null &
|
done
|
|
|
sleep 120
|
echo "second";
|
for j in 1 2 3 4; do
|
mysqldump $host test t$j >/dev/null &
|
done
|
|
|
wait
|
done
|
;;
|
esac
|
At first, it need to populate four table in database test with random data:
- ./test.sh create
Second, run load (eight threads of mysqldump):
- ./test.sh load
If you're lucky mysqld crashed at some minutes. Else just run it again 
I have drop all database and innodb files except database mysql before start test. Log:
140224 11:07:28 mysqld_safe Starting mysqld daemon with databases from /base/mysql
|
140224 11:07:29 InnoDB: The InnoDB memory heap is disabled
|
140224 11:07:29 InnoDB: Mutexes and rw_locks use GCC atomic builtins
|
140224 11:07:29 InnoDB: Compressed tables use zlib 1.2.3
|
140224 11:07:29 InnoDB: Using Linux native AIO
|
140224 11:07:29 InnoDB: Initializing buffer pool, size = 16.0G
|
140224 11:07:30 InnoDB: Completed initialization of buffer pool
|
InnoDB: The first specified data file ./ibdata1 did not exist:
|
InnoDB: a new database to be created!
|
140224 11:07:30 InnoDB: Setting file ./ibdata1 size to 128 MB
|
InnoDB: Database physically writes the file full: wait...
|
InnoDB: Progress in MB: 100
|
140224 11:07:30 InnoDB: Log file ./ib_logfile0 did not exist: new to be created
|
InnoDB: Setting log file ./ib_logfile0 size to 1024 MB
|
InnoDB: Database physically writes the file full: wait...
|
InnoDB: Progress in MB: 100 200 300 400 500 600 700 800 900 1000
|
140224 11:07:31 InnoDB: Log file ./ib_logfile1 did not exist: new to be created
|
InnoDB: Setting log file ./ib_logfile1 size to 1024 MB
|
InnoDB: Database physically writes the file full: wait...
|
InnoDB: Progress in MB: 100 200 300 400 500 600 700 800 900 1000
|
InnoDB: Doublewrite buffer not found: creating new
|
InnoDB: Doublewrite buffer created
|
InnoDB: 127 rollback segment(s) active.
|
InnoDB: Creating foreign key constraint system tables
|
InnoDB: Foreign key constraint system tables created
|
140224 11:07:33 InnoDB: Waiting for the background threads to start
|
140224 11:07:34 Percona XtraDB (http://www.percona.com) 5.5.35-MariaDB-33.0 started; log sequence number 0
|
140224 11:07:34 [Note] Plugin 'FEEDBACK' is disabled.
|
140224 11:07:34 [Note] Server socket created on IP: '0.0.0.0'.
|
140224 11:07:34 [ERROR] Native table 'performance_schema'.'events_waits_current' has the wrong structure
|
140224 11:07:34 [ERROR] Native table 'performance_schema'.'events_waits_history' has the wrong structure
|
140224 11:07:34 [ERROR] Native table 'performance_schema'.'events_waits_history_long' has the wrong structure
|
140224 11:07:34 [ERROR] Native table 'performance_schema'.'setup_consumers' has the wrong structure
|
140224 11:07:34 [ERROR] Native table 'performance_schema'.'setup_instruments' has the wrong structure
|
140224 11:07:34 [ERROR] Native table 'performance_schema'.'setup_timers' has the wrong structure
|
140224 11:07:34 [ERROR] Native table 'performance_schema'.'performance_timers' has the wrong structure
|
140224 11:07:34 [ERROR] Native table 'performance_schema'.'threads' has the wrong structure
|
140224 11:07:34 [ERROR] Native table 'performance_schema'.'events_waits_summary_by_thread_by_event_name' has the wrong structure
|
140224 11:07:34 [ERROR] Native table 'performance_schema'.'events_waits_summary_by_instance' has the wrong structure
|
140224 11:07:34 [ERROR] Native table 'performance_schema'.'events_waits_summary_global_by_event_name' has the wrong structure
|
140224 11:07:34 [ERROR] Native table 'performance_schema'.'file_summary_by_event_name' has the wrong structure
|
140224 11:07:34 [ERROR] Native table 'performance_schema'.'file_summary_by_instance' has the wrong structure
|
140224 11:07:34 [ERROR] Native table 'performance_schema'.'mutex_instances' has the wrong structure
|
140224 11:07:34 [ERROR] Native table 'performance_schema'.'rwlock_instances' has the wrong structure
|
140224 11:07:34 [ERROR] Native table 'performance_schema'.'cond_instances' has the wrong structure
|
140224 11:07:34 [ERROR] Native table 'performance_schema'.'file_instances' has the wrong structure
|
140224 11:07:34 [Note] Event Scheduler: Loaded 0 events
|
140224 11:07:34 [Note] /usr/libexec/mysqld: ready for connections.
|
Version: '5.5.35-MariaDB-log' socket: '/base/mysql/mysql.sock' port: 3306 MariaDB Server
|
140224 14:59:48 [ERROR] mysqld got signal 11 ;
|
This could be because you hit a bug. It is also possible that this binary
|
or one of the libraries it was linked against is corrupt, improperly built,
|
or misconfigured. This error can also be caused by malfunctioning hardware.
|
|
|
To report this bug, see http://kb.askmonty.org/en/reporting-bugs
|
|
|
We will try our best to scrape up some info that will hopefully help
|
diagnose the problem, but since we have already crashed,
|
something is definitely wrong and this may fail.
|
|
|
Server version: 5.5.35-MariaDB-log
|
key_buffer_size=134217728
|
read_buffer_size=131072
|
max_used_connections=9
|
max_threads=502
|
thread_count=8
|
It is possible that mysqld could use up to
|
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 16653819 K bytes of memory
|
Hope that's ok; if not, decrease some variables in the equation.
|
|
|
Thread pointer: 0x0x7f0ec9b00c30
|
Attempting backtrace. You can use the following information to find out
|
where mysqld died. If you see no messages after this, something went
|
terribly wrong...
|
stack_bottom = 0x7f09dfffed58 thread_stack 0x48000
|
/usr/libexec/mysqld(my_print_stacktrace+0x2e)[0x7f0eaa9acc5e]
|
/usr/libexec/mysqld(handle_fatal_signal+0x4ac)[0x7f0eaa5f551c]
|
/lib64/libpthread.so.0(+0x3fe0c0f710)[0x7f0ea9d21710]
|
/usr/libexec/mysqld(+0x764454)[0x7f0eaa8b5454]
|
/usr/libexec/mysqld(+0x76530c)[0x7f0eaa8b630c]
|
/usr/libexec/mysqld(+0x7665bc)[0x7f0eaa8b75bc]
|
/usr/libexec/mysqld(+0x76684c)[0x7f0eaa8b784c]
|
/usr/libexec/mysqld(+0x758d1c)[0x7f0eaa8a9d1c]
|
/usr/libexec/mysqld(+0x749b7a)[0x7f0eaa89ab7a]
|
/usr/libexec/mysqld(+0x6f7ca7)[0x7f0eaa848ca7]
|
/usr/libexec/mysqld(+0x6f8487)[0x7f0eaa849487]
|
/usr/libexec/mysqld(+0x6c9984)[0x7f0eaa81a984]
|
/usr/libexec/mysqld(_Z13rr_sequentialP11READ_RECORD+0x2f)[0x7f0eaa6e8c9f]
|
/usr/libexec/mysqld(_Z10sub_selectP4JOINP13st_join_tableb+0x1ee)[0x7f0eaa4ea75e]
|
/usr/libexec/mysqld(+0x39ca05)[0x7f0eaa4eda05]
|
/usr/libexec/mysqld(_ZN4JOIN4execEv+0xe83)[0x7f0eaa507d83]
|
/usr/libexec/mysqld(_Z12mysql_selectP3THDPPP4ItemP10TABLE_LISTjR4ListIS1_ES2_jP8st_orderSB_S2_SB_yP13select_resultP18st_select_lex_unitP13st_select_lex+0x128)[0x7f0eaa5096b8]
|
/usr/libexec/mysqld(_Z13handle_selectP3THDP3LEXP13select_resultm+0x2b3)[0x7f0eaa50a2d3]
|
/usr/libexec/mysqld(+0x36535f)[0x7f0eaa4b635f]
|
/usr/libexec/mysqld(_Z21mysql_execute_commandP3THD+0x30b5)[0x7f0eaa4bd435]
|
/usr/libexec/mysqld(_Z11mysql_parseP3THDPcjP12Parser_state+0x142)[0x7f0eaa4c05e2]
|
/usr/libexec/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcj+0x175b)[0x7f0eaa4c1ddb]
|
/usr/libexec/mysqld(_Z24do_handle_one_connectionP3THD+0xff)[0x7f0eaa57599f]
|
/usr/libexec/mysqld(handle_one_connection+0x51)[0x7f0eaa575ae1]
|
/usr/libexec/mysqld(+0x829ed8)[0x7f0eaa97aed8]
|
/lib64/libpthread.so.0(+0x3fe0c079d1)[0x7f0ea9d199d1]
|
/lib64/libc.so.6(clone+0x6d)[0x7f0ea8638b6d]
|
|
|
Trying to get some variables.
|
Some pointers may be invalid and cause the dump to abort.
|
Query (0x7f09f4004c18): SELECT /*!40001 SQL_NO_CACHE */ * FROM `t2`
|
Connection ID (thread ID): 51
|
Status: NOT_KILLED
|
|
|
Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=off,table_elimination=on,extended_keys=off
|
|
|
The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
|
information that should help you find out what is causing the crash.
|
Writing a core file
|
140224 15:00:27 mysqld_safe Number of processes running now: 0
|
140224 15:00:27 mysqld_safe mysqld restarted
|
my.cnf:
[mysqld]
|
datadir=/base/mysql
|
socket=/base/mysql/mysql.sock
|
character-set-server=cp1251
|
stack-trace
|
core-file
|
|
|
[mysqld-safe]
|
core-file-size=unlimited
|
|
|
[server]
|
tmpdir=/base/tmp
|
datadir=/base/mysql
|
slave_transaction_retries=10000
|
skip-slave-start
|
|
|
concurrent_insert=2
|
|
|
max_connections=500
|
max_user_connections=490
|
max_connect_errors=50
|
open-files-limit=20000
|
|
|
join_buffer_size=2M
|
key_buffer_size=128M
|
myisam_sort_buffer_size=64M
|
read_buffer_size=128K
|
read_rnd_buffer_size=32M
|
sort_buffer_size=32M
|
|
|
max_allowed_packet=32M
|
max_heap_table_size=512M
|
max_tmp_tables=1000
|
table_cache=10000
|
tmp_table_size=512M
|
|
|
query_cache_limit=131072
|
query_cache_size=33554432
|
query_cache_min_res_unit=1536
|
query_cache_strip_comments=ON
|
|
|
thread_cache_size=400
|
|
|
#InnoDb
|
innodb_buffer_pool_size=16G
|
transaction-isolation=READ-COMMITTED
|
innodb_flush_log_at_trx_commit=0
|
innodb_flush_method=O_DIRECT
|
innodb_data_file_path=ibdata1:128M:autoextend
|
innodb_log_buffer_size = 256M
|
innodb_log_file_size = 1024M
|
innodb_log_files_in_group = 2
|
|
|
innodb_file_per_table=1
|
innodb_file_format=barracuda
|
innodb_io_capacity=5000
|
innodb_read_io_threads=12
|
innodb_write_io_threads=12
|
|
|
long_query_time=3
|
log-warnings
|
back_log=500
|
|
|
interactive_timeout=320
|
wait_timeout=320
|
net_read_timeout=320
|
net_write_timeout=420
|
|
|
server-id=2084
|
log-bin=/base/binlog/bin
|
log_slave_updates
|
relay-log=/base/binlog/relay-bin
|
skip-name-resolve
|
expire_logs_days=1
|
binlog_format=ROW
|
|
|
replicate-wild-ignore-table=tmp.%
|
|
|
init-connect='SET NAMES cp1251'
|
|
|
performance_schema='on'
|
|
|
[mysql]
|
socket=/base/mysql/mysql.sock
|
|
|
[mysqldump]
|
socket=/base/mysql/mysql.sock
|
|
|
[mysql.server]
|
datadir=/base/mysql
|
socket=/base/mysql/mysql.sock
|
|
|
[mysqladmin]
|
socket=/base/mysql/mysql.sock
|
|
|
[mysqlshow]
|
socket=/base/mysql/mysql.sock
|
I can upload core file (or gdb bt) of this crash if it need.
Hi,
It seems Oracle's MySQL-server-5.5.36 is not affected, the test was run during 16 hours in loop and mysqld not crashed.
I have used MySQL-5.5.36-1.el6.x86_64.rpm-bundle.tar from mysql.com.
Hi,
Thanks for testing with that version. I could not repeat on my machine but this might be too slow, I will try with better machine. Can you verify is MariaDB with innodb storage engine affected ?
R: Jan
Hi,
I cannot reproduce crash with MariaDB-10.0.8.
I'll test innodb storage engine today.
I cannot reproduce crash with MariaDB-5.5.36
Sorry to report I was able to reproduce with 5.5.36. (well, not signal 11, but signal 6...)
140228 0:21:26 InnoDB: Assertion failure in thread 139881564858112 in file btr0cur.c line 289
InnoDB: Failing assertion: page_is_comp(get_block->frame) == page_is_comp(page)
InnoDB: We intentionally generate a memory trap.
InnoDB: Submit a detailed bug report to http://bugs.mysql.com.
InnoDB: If you get repeated assertion failures or crashes, even
InnoDB: immediately after the mysqld startup, there may be
InnoDB: corruption in the InnoDB tablespace. Please refer to
InnoDB: http://dev.mysql.com/doc/refman/5.5/en/forcing-innodb-recovery.html
InnoDB: about forcing recovery.
140228 0:21:26 [ERROR] mysqld got signal 6 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
To report this bug, see http://kb.askmonty.org/en/reporting-bugs
We will try our best to scrape up some info that will hopefully help
diagnose the problem, but since we have already crashed,
something is definitely wrong and this may fail.
Server version: 5.5.36-MariaDB
key_buffer_size=33554432
read_buffer_size=131072
max_used_connections=173
max_threads=5002
thread_count=132
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 11006512 K bytes of memory
Hope that's ok; if not, decrease some variables in the equation.
Thread pointer: 0x0x7f38baf95000
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 0x7f38b6fbdd30 thread_stack 0x40000
/usr/sbin/mysqld(my_print_stacktrace+0x2b)[0xa6c18b]
/usr/sbin/mysqld(handle_fatal_signal+0x398)[0x6cdb98]
/lib64/libpthread.so.0(+0xf710)[0x7f3d055b5710]
/lib64/libc.so.6(gsignal+0x35)[0x7f3d03e16925]
/lib64/libc.so.6(abort+0x175)[0x7f3d03e18105]
/usr/sbin/mysqld[0x87654a]
/usr/sbin/mysqld[0x876df3]
/usr/sbin/mysqld[0x93d402]
... (more stack available if useful)
I'll try to test 10.0.8, but might be a few days before I can. Will not be able to stress test it as much as with 5.5...
John,
what kind of binaries you are use? Are you built it from sources or got from https://downloads.mariadb.org?
It's surprisingly for me, but it seems cause of my problems is in my building procedure... 
I'm use slightly modified src.rpm from fedora - I did changes in some paths, RPM-tags and so on, but didn't touch building commands.
Yesterday I did tests with original 5.5.35 and 5.5.36 binaries from https://downloads.mariadb.org and... no crashes.
Jan, could you guess which building conditions may influence to mysqld stability?
Build options from spec:
cmake . -DBUILD_CONFIG=mysql_release \
|
-DFEATURE_SET="community" \
|
-DINSTALL_LAYOUT=RPM \
|
-DCMAKE_INSTALL_PREFIX="%{_prefix}" \
|
-DINSTALL_INCLUDEDIR=include/mysql \
|
-DINSTALL_INFODIR=share/info \
|
-DINSTALL_LIBDIR="%{_lib}/mysql" \
|
-DINSTALL_MANDIR=share/man \
|
-DINSTALL_MYSQLSHAREDIR=share/mysql-5.5 \
|
-DINSTALL_MYSQLTESTDIR=share/mysql-test \
|
-DINSTALL_PLUGINDIR="%{_lib}/mysql/plugin" \
|
-DINSTALL_SBINDIR=libexec \
|
-DINSTALL_SCRIPTDIR=bin \
|
-DINSTALL_SQLBENCHDIR=share \
|
-DINSTALL_SUPPORTFILESDIR=share/mysql-5.5 \
|
-DMYSQL_DATADIR="%{_localstatedir}/lib/mysql" \
|
-DMYSQL_UNIX_ADDR="%{_localstatedir}/lib/mysql/mysql.sock" \
|
-DENABLED_LOCAL_INFILE=ON \
|
-DENABLE_DTRACE=ON \
|
-DWITH_EMBEDDED_SERVER=ON \
|
-DWITH_READLINE=ON \
|
-DWITH_SSL=system \
|
-DWITH_ZLIB=system \
|
-DWITH_JEMALLOC=no \
|
-DTMPDIR=%{_localstatedir}/tmp \
|
-DWITH_MYSQLD_LDFLAGS="-Wl,-z,relro,-z,now"
|
Building environment has been updated to last version of centos6 and has not been applied any patches from original src.rpm.
I am using the binaries that were delivered from yum (baseurl = http://yum.mariadb.org/5.5/rhel6-amd64).
If you were not able to reproduce with 5.5.35 it sounds like you are either not stressing it enough, or your build differences is just different enough to avoid the problem.
I think the table was clean, but I didn't start fresh so it's possible I have corruption from previous testing with 5.5.35. I'll do a mysqldump and reimport the table to make sure it's clean.
I retested 5.5.36 on a test server with a fresh table import and was able to reproduce the problem again.
Running tests on 10.0.8 now, and so far have not been able to reproduce any problems. It's somewhat intermittent so it's not conclusive yet, but it's been running over twice as long as 5.5.35/36 ever lasted for this test. Let me know if it would be useful to test any specific versions (such as that standard mysql 5.5 or earlier 5.5). I assume knowing it's probably already fixed in 10.0.8, makes it less important to fix for 5.5...
John,
I retested (with 12 threads) original mariadb 5.5.35 build during two days on the same server where my build crashed, but can't reproduce problem.
This is what I do for testing that seems to crash reliably 5.3.35 and 5.3.36 (but no crash for 10.0.8).
I have a large table (format compact). Disk is all ssd in case that makes a difference.
tbl_source is actual production data with a couple of indexes and has lots of short rows, and lots of rows over 10k of varchar.
create table t1 like tbl_source;
create table t2 like tbl_source;
create table t3 like tbl_source;
create table t4 like tbl_source;
create table t5 like tbl_source;
alter table t1 key_block_size=8;
alter table t2 key_block_size=8;
alter table t3 key_block_size=4;
alter table t4 key_block_size=16
Verify t1, t2, t3, t4 all show compressed for row_format and t4 compressed (show table status).
You should now have 6 test tables each with indexes
First, I do:
insert into t1 select * from tbl_source;
After that is done, then the following seems to reliably kill it: Run each of these simultaneously in a separate session...
insert into t2 select * from t1;
insert into t3 select * from t1;
insert into t4 select * from t1;
insert into t5 select * from t1;
Hi,
Does it repeat i you attach gdb or ddd to mysqld process and then run the above commands? If it does can you provide me the output from thread apply all bt and actual assertion or line where it crashes and variable values at that line. I can't repeat with my machine, it could be that there is not enough data, raw power etc to reproduce.
R: Jan
I'll give it a try. Right now my test machine was migrated to slower storage, so don't know if that impacts it or not. I downgraded to 5.5.36 for 10.0.8 (has warnings now about InnoDB: in InnoDB data dictionary has unknown flags 10, but made sure it's not complaining about the tables I am working with).
Test is running now with gdb attached, but might take a long long time now... if it fails to crash I'll move it back to fast storage and retest which could take a few days.
That didn't take long.
(gdb) continue
Continuing.
[New Thread 0x7f747e324700 (LWP 26740)]
Program received signal SIGABRT, Aborted.
[Switching to Thread 0x7f74a6ffa700 (LWP 26331)]
0x00007f7613945925 in raise () from /lib64/libc.so.6
(gdb)
(gdb) bt
#0 0x00007f7613945925 in raise () from /lib64/libc.so.6
#1 0x00007f7613947105 in abort () from /lib64/libc.so.6
#2 0x000000000089829c in buf_flush_buffered_writes.part.7 ()
#3 0x000000000089add6 in buf_flush_try_neighbors ()
#4 0x000000000089b64b in buf_flush_flush_list_batch ()
#5 0x000000000089be5d in buf_flush_list ()
#6 0x0000000000843f14 in srv_master_thread ()
#7 0x00007f76150dc9d1 in start_thread () from /lib64/libpthread.so.0
#8 0x00007f76139fbb6d in clone () from /lib64/libc.so.6
Not sure how to do what else you said. don't really use gdb...
(This part is from the error log. 16k buffer removed for potential privacy issues, say if you need it. Don't recall the packet dump before).
InnoDB: End of page dump
140307 22:21:31 InnoDB: Page checksum 1450486885 (32bit_calc: 2292805437), prior-to-4.0.14-form checksum 3931726525
InnoDB: stored checksum 1450486885, prior-to-4.0.14-form stored checksum 3931726525
InnoDB: Page lsn 321 3503936644, low 4 bytes of lsn at page end 3503936644
InnoDB: Page number (if stored to page already) 26343,
InnoDB: space id (if created with >= MySQL-4.1.1 and stored already) 13984
InnoDB: Page may be an index page where index id is 637
InnoDB: (index "PRIMARY" of table "jltest"."t2" /* Partition "p5" */)
140307 22:21:31 InnoDB: Apparent corruption of an index page n:o 26343 in space 13984
InnoDB: to be written to data file. We intentionally crash server
InnoDB: to prevent corrupt data from ending up in data
InnoDB: files.
140307 22:21:31 InnoDB: Assertion failure in thread 140138994706176 in file buf0flu.c line 806
InnoDB: We intentionally generate a memory trap.
...
This seems to like to crash different every time I test it... but fine with 10.0.8 and or I don't use compressed tables with 5.5.x is also fine.
Hi,
About the stack, it does not look correct.
#2 0x000000000089829c in buf_flush_buffered_writes.part.7 ()
|
#3 0x000000000089add6 in buf_flush_try_neighbors ()
|
There is no function called buf_flush_buffered_writes.part.7 and buf_flush_buffered_writes is not called from buf_flush_try_neighbors. Could you try again and please send me output from following command from gdb/ddd:
thread apply all bt
|
Sorry for the delay... Wasn't able to do right away, and forgot about this for a bit.
Program received signal SIGABRT, Aborted.
[Switching to Thread 0x7fb2287fd700 (LWP 2715)]
0x00007fb396d37925 in raise () from /lib64/libc.so.6
(gdb) thread apply all bt
Thread 27 (Thread 0x7fb394bff700 (LWP 2698)):
#0 0x000000000094bf74 in __io_getevents_0_4 ()
#1 0x0000000000922a74 in os_aio_linux_handle ()
#2 0x00000000008d5793 in fil_aio_wait ()
#3 0x0000000000844e68 in io_handler_thread ()
#4 0x00007fb3984ce9d1 in start_thread () from /lib64/libpthread.so.0
#5 0x00007fb396dedb6d in clone () from /lib64/libc.so.6
Thread 26 (Thread 0x7fb23bbfe700 (LWP 2699)):
#0 0x000000000094bf74 in __io_getevents_0_4 ()
#1 0x0000000000922a74 in os_aio_linux_handle ()
#2 0x00000000008d5793 in fil_aio_wait ()
#3 0x0000000000844e68 in io_handler_thread ()
#4 0x00007fb3984ce9d1 in start_thread () from /lib64/libpthread.so.0
#5 0x00007fb396dedb6d in clone () from /lib64/libc.so.6
Thread 25 (Thread 0x7fb23b1fd700 (LWP 2700)):
#0 0x000000000094bf74 in __io_getevents_0_4 ()
#1 0x0000000000922a74 in os_aio_linux_handle ()
#2 0x00000000008d5793 in fil_aio_wait ()
#3 0x0000000000844e68 in io_handler_thread ()
#4 0x00007fb3984ce9d1 in start_thread () from /lib64/libpthread.so.0
#5 0x00007fb396dedb6d in clone () from /lib64/libc.so.6
Thread 24 (Thread 0x7fb23a7fc700 (LWP 2701)):
#0 0x000000000094bf74 in __io_getevents_0_4 ()
#1 0x0000000000922a74 in os_aio_linux_handle ()
#2 0x00000000008d5793 in fil_aio_wait ()
#3 0x0000000000844e68 in io_handler_thread ()
#4 0x00007fb3984ce9d1 in start_thread () from /lib64/libpthread.so.0
#5 0x00007fb396dedb6d in clone () from /lib64/libc.so.6
Thread 23 (Thread 0x7fb239dfb700 (LWP 2702)):
#0 0x000000000094bf74 in __io_getevents_0_4 ()
#1 0x0000000000922a74 in os_aio_linux_handle ()
#2 0x00000000008d5793 in fil_aio_wait ()
--Type <return> to continue, or q <return> to quit--
#3 0x0000000000844e68 in io_handler_thread ()
#4 0x00007fb3984ce9d1 in start_thread () from /lib64/libpthread.so.0
#5 0x00007fb396dedb6d in clone () from /lib64/libc.so.6
Thread 22 (Thread 0x7fb2393fa700 (LWP 2703)):
#0 0x000000000094bf74 in __io_getevents_0_4 ()
#1 0x0000000000922a74 in os_aio_linux_handle ()
#2 0x00000000008d5793 in fil_aio_wait ()
#3 0x0000000000844e68 in io_handler_thread ()
#4 0x00007fb3984ce9d1 in start_thread () from /lib64/libpthread.so.0
#5 0x00007fb396dedb6d in clone () from /lib64/libc.so.6
Thread 21 (Thread 0x7fb2389f9700 (LWP 2704)):
#0 0x000000000094bf74 in __io_getevents_0_4 ()
#1 0x0000000000922a74 in os_aio_linux_handle ()
#2 0x00000000008d5793 in fil_aio_wait ()
#3 0x0000000000844e68 in io_handler_thread ()
#4 0x00007fb3984ce9d1 in start_thread () from /lib64/libpthread.so.0
#5 0x00007fb396dedb6d in clone () from /lib64/libc.so.6
Thread 20 (Thread 0x7fb237ff8700 (LWP 2705)):
#0 0x000000000094bf74 in __io_getevents_0_4 ()
#1 0x0000000000922a74 in os_aio_linux_handle ()
#2 0x00000000008d5793 in fil_aio_wait ()
#3 0x0000000000844e68 in io_handler_thread ()
#4 0x00007fb3984ce9d1 in start_thread () from /lib64/libpthread.so.0
#5 0x00007fb396dedb6d in clone () from /lib64/libc.so.6
Thread 19 (Thread 0x7fb2375f7700 (LWP 2706)):
#0 0x000000000094bf74 in __io_getevents_0_4 ()
#1 0x0000000000922a74 in os_aio_linux_handle ()
#2 0x00000000008d5793 in fil_aio_wait ()
#3 0x0000000000844e68 in io_handler_thread ()
#4 0x00007fb3984ce9d1 in start_thread () from /lib64/libpthread.so.0
#5 0x00007fb396dedb6d in clone () from /lib64/libc.so.6
Thread 18 (Thread 0x7fb236bf6700 (LWP 2707)):
--Type <return> to continue, or q <return> to quit--
#0 0x000000000094bf74 in __io_getevents_0_4 ()
#1 0x0000000000922a74 in os_aio_linux_handle ()
#2 0x00000000008d5793 in fil_aio_wait ()
#3 0x0000000000844e68 in io_handler_thread ()
#4 0x00007fb3984ce9d1 in start_thread () from /lib64/libpthread.so.0
#5 0x00007fb396dedb6d in clone () from /lib64/libc.so.6
Thread 17 (Thread 0x7fb2361f5700 (LWP 2708)):
#0 0x000000000094bf74 in __io_getevents_0_4 ()
#1 0x0000000000922a74 in os_aio_linux_handle ()
#2 0x00000000008d5793 in fil_aio_wait ()
#3 0x0000000000844e68 in io_handler_thread ()
#4 0x00007fb3984ce9d1 in start_thread () from /lib64/libpthread.so.0
#5 0x00007fb396dedb6d in clone () from /lib64/libc.so.6
Thread 16 (Thread 0x7fb2357f4700 (LWP 2709)):
#0 0x000000000094bf74 in __io_getevents_0_4 ()
#1 0x0000000000922a74 in os_aio_linux_handle ()
#2 0x00000000008d5793 in fil_aio_wait ()
#3 0x0000000000844e68 in io_handler_thread ()
#4 0x00007fb3984ce9d1 in start_thread () from /lib64/libpthread.so.0
#5 0x00007fb396dedb6d in clone () from /lib64/libc.so.6
Thread 15 (Thread 0x7fb234df3700 (LWP 2710)):
#0 0x000000000094bf74 in __io_getevents_0_4 ()
#1 0x0000000000922a74 in os_aio_linux_handle ()
#2 0x00000000008d5793 in fil_aio_wait ()
#3 0x0000000000844e68 in io_handler_thread ()
#4 0x00007fb3984ce9d1 in start_thread () from /lib64/libpthread.so.0
#5 0x00007fb396dedb6d in clone () from /lib64/libc.so.6
Thread 14 (Thread 0x7fb2343f2700 (LWP 2711)):
#0 0x000000000094bf74 in __io_getevents_0_4 ()
#1 0x0000000000922a74 in os_aio_linux_handle ()
#2 0x00000000008d5793 in fil_aio_wait ()
#3 0x0000000000844e68 in io_handler_thread ()
#4 0x00007fb3984ce9d1 in start_thread () from /lib64/libpthread.so.0
--Type <return> to continue, or q <return> to quit--
#5 0x00007fb396dedb6d in clone () from /lib64/libc.so.6
Thread 13 (Thread 0x7fb2291fe700 (LWP 2714)):
#0 0x00007fb3984d298e in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x000000000092482b in os_event_wait_time_low ()
#2 0x0000000000841890 in srv_lock_timeout_thread ()
#3 0x00007fb3984ce9d1 in start_thread () from /lib64/libpthread.so.0
#4 0x00007fb396dedb6d in clone () from /lib64/libc.so.6
Thread 12 (Thread 0x7fb2287fd700 (LWP 2715)):
#0 0x00007fb396d37925 in raise () from /lib64/libc.so.6
#1 0x00007fb396d39105 in abort () from /lib64/libc.so.6
#2 0x00000000008420e3 in srv_error_monitor_thread ()
#3 0x00007fb3984ce9d1 in start_thread () from /lib64/libpthread.so.0
#4 0x00007fb396dedb6d in clone () from /lib64/libc.so.6
Thread 11 (Thread 0x7fb227dfc700 (LWP 2716)):
#0 0x00007fb3984d298e in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x000000000092482b in os_event_wait_time_low ()
#2 0x00000000008415a3 in srv_monitor_thread ()
#3 0x00007fb3984ce9d1 in start_thread () from /lib64/libpthread.so.0
#4 0x00007fb396dedb6d in clone () from /lib64/libc.so.6
Thread 10 (Thread 0x7fb2273fb700 (LWP 2717)):
#0 0x00007fb3984d298e in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x000000000092482b in os_event_wait_time_low ()
#2 0x0000000000842163 in srv_LRU_dump_restore_thread ()
#3 0x00007fb3984ce9d1 in start_thread () from /lib64/libpthread.so.0
#4 0x00007fb396dedb6d in clone () from /lib64/libc.so.6
Thread 9 (Thread 0x7fb2269fa700 (LWP 2718)):
#0 0x00007fb3984d25bc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x0000000000924721 in os_event_wait_low ()
#2 0x000000000089b850 in buf_flush_wait_batch_end ()
#3 0x000000000084360c in srv_master_thread ()
#4 0x00007fb3984ce9d1 in start_thread () from /lib64/libpthread.so.0
#5 0x00007fb396dedb6d in clone () from /lib64/libc.so.6
--Type <return> to continue, or q <return> to quit--
Thread 8 (Thread 0x7fb225ff9700 (LWP 2719)):
#0 0x00007fb3984d25bc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x0000000000924721 in os_event_wait_low ()
#2 0x0000000000844aae in srv_purge_thread ()
#3 0x00007fb3984ce9d1 in start_thread () from /lib64/libpthread.so.0
#4 0x00007fb396dedb6d in clone () from /lib64/libc.so.6
Thread 7 (Thread 0x7fb21d7ff700 (LWP 2720)):
#0 0x00007fb3984d298e in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x000000000099f5c7 in my_service_thread_sleep ()
#2 0x0000000000997de2 in ma_checkpoint_background ()
#3 0x00007fb3984ce9d1 in start_thread () from /lib64/libpthread.so.0
#4 0x00007fb396dedb6d in clone () from /lib64/libc.so.6
Thread 6 (Thread 0x7fb21cdfe700 (LWP 2721)):
#0 0x00007fb3984d64b5 in sigwait () from /lib64/libpthread.so.0
#1 0x000000000051f76f in signal_hand ()
#2 0x00007fb3984ce9d1 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fb396dedb6d in clone () from /lib64/libc.so.6
Thread 5 (Thread 0x7fb21bfff700 (LWP 3123)):
#0 0x00007fb3984d575d in read () from /lib64/libpthread.so.0
#1 0x0000000000a9f558 in vio_read ()
#2 0x0000000000527b81 in my_real_read(st_net*, unsigned long*) ()
#3 0x0000000000528a49 in my_net_read ()
#4 0x00000000005a4012 in do_command(THD*) ()
#5 0x00000000006533cb in do_handle_one_connection(THD*) ()
#6 0x000000000065344c in handle_one_connection ()
#7 0x00007fb3984ce9d1 in start_thread () from /lib64/libpthread.so.0
#8 0x00007fb396dedb6d in clone () from /lib64/libc.so.6
Thread 4 (Thread 0x7fb21bfb6700 (LWP 3125)):
#0 0x0000000000a7273c in my_pthread_fastmutex_lock ()
#1 0x000000000092469d in os_event_reset ()
#2 0x00000000008487ac in sync_array_reserve_cell ()
#3 0x000000000084a07e in mutex_spin_wait ()
--Type <return> to continue, or q <return> to quit--
#4 0x00000000008a37a3 in buf_LRU_free_from_common_LRU_list ()
#5 0x00000000008a532f in buf_LRU_get_free_block ()
#6 0x0000000000894133 in buf_page_create ()
#7 0x00000000008dabc8 in fsp_page_create ()
#8 0x00000000008de6b8 in fseg_alloc_free_page_low ()
#9 0x00000000008e155b in fseg_alloc_free_page_general ()
#10 0x000000000086e000 in btr_page_split_and_insert ()
#11 0x00000000008793d6 in btr_cur_pessimistic_insert ()
#12 0x000000000093d655 in row_ins_index_entry_low ()
#13 0x0000000000940f30 in row_ins_step ()
#14 0x0000000000825f00 in row_insert_for_mysql ()
#15 0x000000000080e830 in ha_innobase::write_row(unsigned char*) ()
#16 0x00000000006d4e88 in handler::ha_write_row(unsigned char*) ()
#17 0x0000000000a4444f in ha_partition::write_row(unsigned char*) ()
#18 0x00000000006d4e88 in handler::ha_write_row(unsigned char*) ()
#19 0x00000000005856a2 in write_record(THD*, TABLE*, st_copy_info*) ()
#20 0x0000000000585f83 in select_insert::send_data(List<Item>&) ()
#21 0x00000000005bc7b6 in end_send(JOIN*, st_join_table*, bool) ()
#22 0x00000000005bcb22 in evaluate_join_record(JOIN*, st_join_table*, int) ()
#23 0x00000000005bcdf9 in sub_select(JOIN*, st_join_table*, bool) ()
#24 0x00000000005d3acd in do_select(JOIN*, List<Item>, TABLE, Procedure*) ()
#25 0x00000000005e8ccc in JOIN::exec() ()
#26 0x00000000005e4095 in mysql_select(THD*, Item**, TABLE_LIST, unsigned int, List<Item>&, Item*, unsigned int, st_order*, st_order*, Item*, st_order*, unsigned long long, select_result*, st_select_lex_unit*, st_select_lex*) ()
#27 0x00000000005eac83 in handle_select(THD*, LEX*, select_result*, unsigned long) ()
#28 0x00000000005a0e76 in mysql_execute_command(THD*) ()
#29 0x00000000005a1ab4 in mysql_parse ()
#30 0x00000000005a3f2e in dispatch_command(enum_server_command, THD*, char*, unsigned int) ()
#31 0x00000000006533cb in do_handle_one_connection(THD*) ()
#32 0x000000000065344c in handle_one_connection ()
#33 0x00007fb3984ce9d1 in start_thread () from /lib64/libpthread.so.0
#34 0x00007fb396dedb6d in clone () from /lib64/libc.so.6
Thread 3 (Thread 0x7fb21bf6d700 (LWP 3126)):
#0 0x00007fb3984d25bc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x0000000000924721 in os_event_wait_low ()
#2 0x0000000000848bae in sync_array_wait_event ()
--Type <return> to continue, or q <return> to quit--
#3 0x0000000000849900 in rw_lock_s_lock_spin ()
#4 0x000000000089aefe in buf_flush_try_neighbors ()
#5 0x000000000089b64b in buf_flush_flush_list_batch ()
#6 0x000000000089be5d in buf_flush_list ()
#7 0x000000000090d9de in log_check_margins ()
#8 0x000000000093d317 in row_ins_index_entry_low ()
#9 0x0000000000940e68 in row_ins_step ()
#10 0x0000000000825f00 in row_insert_for_mysql ()
#11 0x000000000080e830 in ha_innobase::write_row(unsigned char*) ()
#12 0x00000000006d4e88 in handler::ha_write_row(unsigned char*) ()
#13 0x0000000000a4444f in ha_partition::write_row(unsigned char*) ()
#14 0x00000000006d4e88 in handler::ha_write_row(unsigned char*) ()
#15 0x00000000005856a2 in write_record(THD*, TABLE*, st_copy_info*) ()
#16 0x0000000000585f83 in select_insert::send_data(List<Item>&) ()
#17 0x00000000005bc7b6 in end_send(JOIN*, st_join_table*, bool) ()
#18 0x00000000005bcb22 in evaluate_join_record(JOIN*, st_join_table*, int) ()
#19 0x00000000005bcdf9 in sub_select(JOIN*, st_join_table*, bool) ()
#20 0x00000000005d3acd in do_select(JOIN*, List<Item>, TABLE, Procedure*) ()
#21 0x00000000005e8ccc in JOIN::exec() ()
#22 0x00000000005e4095 in mysql_select(THD*, Item**, TABLE_LIST, unsigned int, List<Item>&, Item*, unsigned int, st_order*, st_order*, Item*, st_order*, unsigned long long, select_result*, st_select_lex_unit*, st_select_lex*) ()
#23 0x00000000005eac83 in handle_select(THD*, LEX*, select_result*, unsigned long) ()
#24 0x00000000005a0e76 in mysql_execute_command(THD*) ()
#25 0x00000000005a1ab4 in mysql_parse ()
#26 0x00000000005a3f2e in dispatch_command(enum_server_command, THD*, char*, unsigned int) ()
#27 0x00000000006533cb in do_handle_one_connection(THD*) ()
#28 0x000000000065344c in handle_one_connection ()
#29 0x00007fb3984ce9d1 in start_thread () from /lib64/libpthread.so.0
#30 0x00007fb396dedb6d in clone () from /lib64/libc.so.6
Thread 2 (Thread 0x7fb21bf24700 (LWP 3127)):
#0 0x00007fb3984d25bc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x0000000000924721 in os_event_wait_low ()
#2 0x000000000089b850 in buf_flush_wait_batch_end ()
#3 0x000000000090de63 in log_check_margins ()
#4 0x000000000093d317 in row_ins_index_entry_low ()
#5 0x0000000000940e68 in row_ins_step ()
--Type <return> to continue, or q <return> to quit--
#6 0x0000000000825f00 in row_insert_for_mysql ()
#7 0x000000000080e830 in ha_innobase::write_row(unsigned char*) ()
#8 0x00000000006d4e88 in handler::ha_write_row(unsigned char*) ()
#9 0x0000000000a4444f in ha_partition::write_row(unsigned char*) ()
#10 0x00000000006d4e88 in handler::ha_write_row(unsigned char*) ()
#11 0x00000000005856a2 in write_record(THD*, TABLE*, st_copy_info*) ()
#12 0x0000000000585f83 in select_insert::send_data(List<Item>&) ()
#13 0x00000000005bc7b6 in end_send(JOIN*, st_join_table*, bool) ()
#14 0x00000000005bcb22 in evaluate_join_record(JOIN*, st_join_table*, int) ()
#15 0x00000000005bcdf9 in sub_select(JOIN*, st_join_table*, bool) ()
#16 0x00000000005d3acd in do_select(JOIN*, List<Item>, TABLE, Procedure*) ()
#17 0x00000000005e8ccc in JOIN::exec() ()
#18 0x00000000005e4095 in mysql_select(THD*, Item**, TABLE_LIST, unsigned int, List<Item>&, Item*, unsigned int, st_order*, st_order*, Item*, st_order*, unsigned long long, select_result*, st_select_lex_unit*, st_select_lex*) ()
#19 0x00000000005eac83 in handle_select(THD*, LEX*, select_result*, unsigned long) ()
#20 0x00000000005a0e76 in mysql_execute_command(THD*) ()
#21 0x00000000005a1ab4 in mysql_parse ()
#22 0x00000000005a3f2e in dispatch_command(enum_server_command, THD*, char*, unsigned int) ()
#23 0x00000000006533cb in do_handle_one_connection(THD*) ()
#24 0x000000000065344c in handle_one_connection ()
#25 0x00007fb3984ce9d1 in start_thread () from /lib64/libpthread.so.0
#26 0x00007fb396dedb6d in clone () from /lib64/libc.so.6
Thread 1 (Thread 0x7fb3988f1840 (LWP 2696)):
#0 0x00007fb396de4343 in poll () from /lib64/libc.so.6
#1 0x0000000000523073 in handle_connections_sockets() ()
#2 0x0000000000526db2 in mysqld_main(int, char**) ()
#3 0x00007fb396d23d1d in __libc_start_main () from /lib64/libc.so.6
#4 0x000000000051d149 in _start ()
(gdb)
This does not look similar to original problem, it looks like more a long semaphore wait problem.
I don't know, it seems to crash differently each time, but if we are getting random corruption is that unexpected? It is on slower storage right now than when I was testing originally. Tight on SSD storage right now so don't know if I'll be able to move it to fast storage again (but we did just order some new servers, not sure how many weeks that might take).
Even if it's just a semaphore wait problem in this case, shouldn't it do a deadlock detection or something and abort some threads and not crash mysql? or is gdb intercepting that?
There is no deadlock detection on semaphore waits, only timeout and then it stops