[MDEV-5695] Crash after debian MariaDB 10.0.x upgrade Created: 2014-02-17  Updated: 2014-03-11  Due: 2014-03-20  Resolved: 2014-03-11

Status: Closed
Project: MariaDB Server
Component/s: None
Affects Version/s: 10.0.7, 10.0.8
Fix Version/s: 10.0.9

Type: Bug Priority: Critical
Reporter: jocelyn fournier Assignee: Jan Lindström (Inactive)
Resolution: Fixed Votes: 0
Labels: None


 Description   

Hi,

This crash already occurs for me when I upgraded from 10.0.6 to 10.0.7.
It also occured when I upgraded from 10.0.7 to 10.0.8.

The upgrade is done with a classical apt-get upgrade.

Content of the log file :

131229 23:40:34 [Note] /usr/sbin/mysqld: ready for connections.
Version: '10.0.7-MariaDB-1~wheezy' socket: '/var/run/mysqld/mysqld.sock' port: 3306 mariadb.org binary distribution
140211 0:00:19 [Note] /usr/sbin/mysqld: Normal shutdown

140211 0:00:19 [Note] Event Scheduler: Purging the queue. 0 events
140211 0:00:22 [Note] InnoDB: FTS optimize thread exiting.
140211 0:00:22 [Note] InnoDB: Starting shutdown...
140211 0:00:25 [Note] InnoDB: Shutdown completed; log sequence number 3396091272693
140211 0:00:25 [Note] /usr/sbin/mysqld: Shutdown complete

140211 00:00:25 mysqld_safe mysqld from pid file /var/run/mysqld/mysqld.pid ended
140211 0:00:53 [Note] InnoDB: The InnoDB memory heap is disabled
140211 0:00:53 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
140211 0:00:53 [Note] InnoDB: Compressed tables use zlib 1.2.7
140211 0:00:53 [Note] InnoDB: Using Linux native AIO
140211 0:00:53 [Note] InnoDB: Using CPU crc32 instructions
140211 0:00:53 [Note] InnoDB: Initializing buffer pool, size = 4.0G
140211 0:00:53 [Note] InnoDB: Completed initialization of buffer pool
140211 0:00:53 [Note] InnoDB: Highest supported file format is Barracuda.
140211 0:00:54 [Note] InnoDB: 128 rollback segment(s) are active.
140211 0:00:54 [Note] InnoDB: Waiting for purge to start
140211 0:00:54 [Note] InnoDB: 5.6.14 started; log sequence number 3396091272693
140211 0:01:00 [Note] InnoDB: FTS optimize thread exiting.
140211 0:01:00 [Note] InnoDB: Starting shutdown...
140211 0:01:01 [Note] InnoDB: Shutdown completed; log sequence number 3396091274630
140211 0:01:03 [Note] InnoDB: The InnoDB memory heap is disabled
140211 0:01:03 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
140211 0:01:03 [Note] InnoDB: Compressed tables use zlib 1.2.7
140211 0:01:03 [Note] InnoDB: Using Linux native AIO
140211 0:01:03 [Note] InnoDB: Using CPU crc32 instructions
140211 0:01:03 [Note] InnoDB: Initializing buffer pool, size = 4.0G
140211 0:01:03 [Note] InnoDB: Completed initialization of buffer pool
140211 0:01:03 [Note] InnoDB: Highest supported file format is Barracuda.
140211 0:01:04 [Note] InnoDB: 128 rollback segment(s) are active.
140211 0:01:05 [Note] InnoDB: Waiting for purge to start
140211 0:01:05 [Note] InnoDB: 5.6.14 started; log sequence number 3396091274630
140211 0:01:06 [Note] InnoDB: FTS optimize thread exiting.
140211 0:01:06 [Note] InnoDB: Starting shutdown...
140211 0:01:08 [Note] InnoDB: Shutdown completed; log sequence number 3396091277535
140211 0:01:09 [Note] InnoDB: The InnoDB memory heap is disabled
140211 0:01:09 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
140211 0:01:09 [Note] InnoDB: Compressed tables use zlib 1.2.7
140211 0:01:09 [Note] InnoDB: Using Linux native AIO
140211 0:01:09 [Note] InnoDB: Using CPU crc32 instructions
140211 0:01:09 [Note] InnoDB: Initializing buffer pool, size = 4.0G
140211 0:01:09 [Note] InnoDB: Completed initialization of buffer pool
140211 0:01:09 [Note] InnoDB: Highest supported file format is Barracuda.
140211 0:01:09 [Note] InnoDB: 128 rollback segment(s) are active.
140211 0:01:10 [Note] InnoDB: Waiting for purge to start
140211 0:01:10 [Note] InnoDB: 5.6.14 started; log sequence number 3396091277535
140211 0:01:10 [Note] Plugin 'FEEDBACK' is disabled.
140211 0:01:11 [Note] InnoDB: FTS optimize thread exiting.
140211 0:01:11 [Note] InnoDB: Starting shutdown...
140211 0:01:13 [Note] InnoDB: Shutdown completed; log sequence number 3396091282907
140211 0:01:14 [Note] InnoDB: The InnoDB memory heap is disabled
140211 0:01:14 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
140211 0:01:14 [Note] InnoDB: Compressed tables use zlib 1.2.7
140211 0:01:14 [Note] InnoDB: Using Linux native AIO
140211 0:01:14 [Note] InnoDB: Using CPU crc32 instructions
140211 0:01:14 [Note] InnoDB: Initializing buffer pool, size = 4.0G
140211 0:01:14 [Note] InnoDB: Completed initialization of buffer pool
140211 0:01:14 [Note] InnoDB: Highest supported file format is Barracuda.
140211 0:01:15 [Note] InnoDB: 128 rollback segment(s) are active.
140211 0:01:15 [Note] InnoDB: Waiting for purge to start
140211 0:01:15 [Note] InnoDB: 5.6.14 started; log sequence number 3396091282907
140211 0:01:15 [Note] Plugin 'FEEDBACK' is disabled.
140211 0:01:16 [Note] InnoDB: FTS optimize thread exiting.
140211 0:01:16 [Note] InnoDB: Starting shutdown...
140211 0:01:18 [Note] InnoDB: Shutdown completed; log sequence number 3396091282917
140211 0:01:18 [Note] InnoDB: The InnoDB memory heap is disabled
140211 0:01:18 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
140211 0:01:18 [Note] InnoDB: Compressed tables use zlib 1.2.7
140211 0:01:18 [Note] InnoDB: Using Linux native AIO
140211 0:01:18 [Note] InnoDB: Using CPU crc32 instructions
140211 0:01:18 [Note] InnoDB: Initializing buffer pool, size = 4.0G
140211 0:01:19 [Note] InnoDB: Completed initialization of buffer pool
140211 0:01:19 [Note] InnoDB: Highest supported file format is Barracuda.
140211 0:01:19 [Note] InnoDB: 128 rollback segment(s) are active.
140211 0:01:19 [Note] InnoDB: Waiting for purge to start
140211 0:01:19 [Note] InnoDB: 5.6.14 started; log sequence number 3396091282917
140211 0:01:19 [Note] Plugin 'FEEDBACK' is disabled.
140211 0:01:21 [Note] InnoDB: FTS optimize thread exiting.
140211 0:01:21 [Note] InnoDB: Starting shutdown...
140211 0:01:23 [Note] InnoDB: Shutdown completed; log sequence number 3396091282927
140211 00:01:23 mysqld_safe Starting mysqld daemon with databases from /home/mysql/data
140211 0:01:23 [Note] InnoDB: The InnoDB memory heap is disabled
140211 0:01:23 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
140211 0:01:23 [Note] InnoDB: Compressed tables use zlib 1.2.7
140211 0:01:23 [Note] InnoDB: Using Linux native AIO
140211 0:01:23 [Note] InnoDB: Using CPU crc32 instructions
140211 0:01:23 [Note] InnoDB: Initializing buffer pool, size = 4.0G
140211 0:01:24 [Note] InnoDB: Completed initialization of buffer pool
140211 0:01:24 [Note] InnoDB: Highest supported file format is Barracuda.
140211 0:01:24 [Note] InnoDB: 128 rollback segment(s) are active.
140211 0:01:24 [Note] InnoDB: Waiting for purge to start
140211 0:01:24 [Note] InnoDB: 5.6.14 started; log sequence number 3396091282927
140211 0:01:24 [Note] Plugin 'FEEDBACK' is disabled.
140211 0:01:24 [ERROR] Function 'TokuDB_fractal_tree_block_map' already exists
140211 0:01:24 [Warning] Couldn't load plugin named 'TokuDB_fractal_tree_block_map' with soname 'ha_tokudb.so'.
140211 0:01:24 [ERROR] Function 'TokuDB_fractal_tree_info' already exists
140211 0:01:24 [Warning] Couldn't load plugin named 'TokuDB_fractal_tree_info' with soname 'ha_tokudb.so'.
140211 0:01:24 [ERROR] Function 'TokuDB_file_map' already exists
140211 0:01:24 [Warning] Couldn't load plugin named 'TokuDB_file_map' with soname 'ha_tokudb.so'.
140211 0:01:24 [ERROR] Can't find symbol 'TokuDB_user_data_exact' in library
140211 0:01:24 [Warning] Couldn't load plugin named 'TokuDB_user_data_exact' with soname 'ha_tokudb.so'.
140211 0:01:24 [ERROR] Can't find symbol 'TokuDB_user_data' in library
140211 0:01:24 [Warning] Couldn't load plugin named 'TokuDB_user_data' with soname 'ha_tokudb.so'.
140211 0:01:25 [Note] Server socket created on IP: '127.0.0.1'.
140211 0:01:25 [Warning] 'user' entry 'prestashop@@ocalhost' ignored in --skip-name-resolve mode.
140211 0:01:25 [Warning] 'db' entry 'prestashop prestashop@@ocalhost' ignored in --skip-name-resolve mode.
140211 0:01:25 [Note] Event Scheduler: Loaded 0 events
140211 0:01:25 [Note] /usr/sbin/mysqld: ready for connections.
Version: '10.0.8-MariaDB-1~wheezy' socket: '/var/run/mysqld/mysqld.sock' port: 3306 mariadb.org binary distribution

Status information:

Current dir: /home/mysql/data/
Running threads: 1 Stack size: 294912
Current locks:
lock: 0x7ff1f582d198:

lock: 0x7ff1f5818508:

lock: 0x7ff326f67f88:

lock: 0x7ff326f5bf88:

lock: 0x7ff32f3ee988:

lock: 0x7ff326f5b388:

lock: 0x7ff32f3eb788:

lock: 0x7ff32f3ea388:

lock: 0x7ff341c9eb88:

lock: 0x7ff20032a388:

lock: 0x7ff326f58388:

lock: 0x7ff32f3e7188:

lock: 0x7ff3458bafc0:

lock: 0x7ff3458baec0:

lock: 0x7ff3458b9cc0:

lock: 0x7ff3458b9bc0:

lock: 0x7ff3458b9ac0:

lock: 0x7ff3458b9ec0:

lock: 0x7ff3458b7cc0:

lock: 0x7ff3458badc0:

lock: 0x7ff3458b84c0:

lock: 0x7ff3458b87c0:

lock: 0x7ff3458b83c0:

lock: 0x7ff3458b86c0:

lock: 0x7ff3458b82c0:

lock: 0x7ff3458b85c0:

lock: 0x7ff3458b8bc0:

lock: 0x7ff3458b8cc0:

lock: 0x7ff3458b8dc0:

lock: 0x7ff3458b81c0:

lock: 0x7ff3458b7ec0:

lock: 0x7ff3458b80c0:

lock: 0x7ff3458b7dc0:

lock: 0x7ff3458b7fc0:

lock: 0x7ff3458b88c0:

lock: 0x7ff3458b89c0:

lock: 0x7ff3458b8ac0:

lock: 0x7ff3458ba9c0:

lock: 0x7ff3458bacc0:

lock: 0x7ff3458babc0:

lock: 0x7ff3458baac0:

lock: 0x7ff3458ba5c0:

lock: 0x7ff3458ba4c0:

lock: 0x7ff3458ba3c0:

lock: 0x7ff3458ba2c0:

lock: 0x7ff3458ba1c0:

lock: 0x7ff3458ba7c0:

lock: 0x7ff3458ba0c0:

lock: 0x7ff3458b9fc0:

lock: 0x7ff3458ba8c0:

lock: 0x7ff3458b9dc0:

lock: 0x7ff3458b98c0:

lock: 0x7ff3458b99c0:

lock: 0x7ff3458b97c0:

lock: 0x7ff3458b96c0:

lock: 0x7ff3458b92c0:

lock: 0x7ff3458b95c0:

lock: 0x7ff3458b94c0:

lock: 0x7ff3458b91c0:

lock: 0x7ff3458b93c0:

lock: 0x7ff3458b8ec0:

lock: 0x7ff3458b8fc0:

lock: 0x7ff3458b90c0:

lock: 0x7ff3458ba6c0:

Key caches:
default
Buffer_size: 134217728
Block_size: 1024
Division_limit: 100
Age_threshold: 300
Partitions: 0
blocks used: 4
not flushed: 0
w_requests: 0
writes: 0
r_requests: 8
reads: 4

handler status:
read_key: 0
read_next: 0
read_rnd 0
read_first: 0
write: 0
delete 0
update: 0

Table status:
Opened tables: 0
Open tables: 63
Open files: 26
Open streams: 0

Alarm status:
Active alarms: 0
Max used alarms: 0
Next alarm time: 0
140211 0:01:27 [Warning] 'user' entry 'prestashop@@ocalhost' ignored in --skip-name-resolve mode.
140211 0:01:27 [Warning] 'db' entry 'prestashop prestashop@@ocalhost' ignored in --skip-name-resolve mode.

Memory status:
Non-mmapped space allocated from system: 0
Number of free chunks: 1
Number of fastbin blocks: 0
Number of mmapped regions: 0
Space in mmapped regions: 0
Maximum total allocated space: 0
Space available in freed fastbin blocks: 0
Total allocated space: 0
Total free space: 0
Top-most, releasable space: 0
Estimated memory (with thread stack): 294912

Events status:
LLA = Last Locked At LUA = Last Unlocked At
WOC = Waiting On Condition DL = Data Locked

Event scheduler status:
State : INITIALIZED
Thread id : 0
LLA : n/a:0
LUA : n/a:0
WOC : NO
Workers : 0
Executed : 0
Data locked: NO

Event queue status:
Element count : 0
Data locked : NO
Attempting lock : NO
LLA : init_queue:139
LUA : init_queue:149
WOC : NO
Next activation : never
InnoDB: Error: trying to access page number 32 in space 326262,
InnoDB: space name MQA/errors,
InnoDB: which is outside the tablespace bounds.
InnoDB: Byte offset 0, len 16384, i/o type 10.
InnoDB: If you get this error at mysqld startup, please check that
InnoDB: your my.cnf matches the ibdata files that you have in the
InnoDB: MySQL server.
2014-02-11 00:01:35 7ff344631700 InnoDB: Assertion failure in thread 140682801125120 in file fil0fil.cc line 5445
InnoDB: We intentionally generate a memory trap.
InnoDB: Submit a detailed bug report to http://bugs.mysql.com.
InnoDB: If you get repeated assertion failures or crashes, even
InnoDB: immediately after the mysqld startup, there may be
InnoDB: corruption in the InnoDB tablespace. Please refer to
InnoDB: http://dev.mysql.com/doc/refman/5.6/en/forcing-innodb-recovery.html
InnoDB: about forcing recovery.
140211 0:01:35 [ERROR] mysqld got signal 6 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.

To report this bug, see http://kb.askmonty.org/en/reporting-bugs

We will try our best to scrape up some info that will hopefully help
diagnose the problem, but since we have already crashed,
something is definitely wrong and this may fail.

Server version: 10.0.8-MariaDB-1~wheezy
key_buffer_size=134217728
read_buffer_size=4194304
max_used_connections=1
max_threads=402
thread_count=1
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 2609233 K bytes of memory
Hope that's ok; if not, decrease some variables in the equation.

Thread pointer: 0x0x7ff1f4993008
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 0x7ff344630dc0 thread_stack 0x48000
/usr/sbin/mysqld(my_print_stacktrace+0x2b)[0x7ff34507808b]
/usr/sbin/mysqld(handle_fatal_signal+0x422)[0x7ff344c2b902]
/lib/x86_64-linux-gnu/libpthread.so.0(+0xf030)[0x7ff34429d030]
/lib/x86_64-linux-gnu/libc.so.6(gsignal+0x35)[0x7ff3428f9475]
/lib/x86_64-linux-gnu/libc.so.6(abort+0x180)[0x7ff3428fc6f0]
/usr/sbin/mysqld(+0x93a025)[0x7ff345006025]
/usr/sbin/mysqld(+0x8ff6d6)[0x7ff344fcb6d6]
/usr/sbin/mysqld(+0x900024)[0x7ff344fcc024]
/usr/sbin/mysqld(+0x8e7899)[0x7ff344fb3899]
/usr/sbin/mysqld(+0x8ccb61)[0x7ff344f98b61]
/usr/sbin/mysqld(+0x87cc64)[0x7ff344f48c64]
/usr/sbin/mysqld(+0x7d854c)[0x7ff344ea454c]
/usr/sbin/mysqld(+0x7dd720)[0x7ff344ea9720]
/usr/sbin/mysqld(_ZN7handler7ha_openEP5TABLEPKcij+0x3e)[0x7ff344c2fa7e]
/usr/sbin/mysqld(_Z21open_table_from_shareP3THDP11TABLE_SHAREPKcjjjP5TABLEb+0x8f1)[0x7ff344b725a1]
/usr/sbin/mysqld(_Z10open_tableP3THDP10TABLE_LISTP11st_mem_rootP18Open_table_context+0x79f)[0x7ff344a9530f]
/usr/sbin/mysqld(_Z11open_tablesP3THDPP10TABLE_LISTPjjP19Prelocking_strategy+0x12c1)[0x7ff344a96951]
/usr/sbin/mysqld(_Z20open_and_lock_tablesP3THDP10TABLE_LISTbjP19Prelocking_strategy+0x44)[0x7ff344a96ba4]
/usr/sbin/mysqld(+0x4dacab)[0x7ff344ba6cab]
/usr/sbin/mysqld(_ZN19Sql_cmd_check_table7executeEP3THD+0xb7)[0x7ff344ba8627]
/usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x1a18)[0x7ff344ad10f8]
/usr/sbin/mysqld(+0x409cba)[0x7ff344ad5cba]
/usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcj+0x14f8)[0x7ff344ad7298]
/usr/sbin/mysqld(_Z24do_handle_one_connectionP3THD+0x47b)[0x7ff344b996fb]
/usr/sbin/mysqld(handle_one_connection+0x47)[0x7ff344b997d7]
/usr/sbin/mysqld(+0x6d2ab0)[0x7ff344d9eab0]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x6b50)[0x7ff344294b50]
/lib/x86_64-linux-gnu/libc.so.6(clone+0x6d)[0x7ff3429a30ed]

Trying to get some variables.
Some pointers may be invalid and cause the dump to abort.
Query (0x7ff1f5872020): is an invalid pointer
Connection ID (thread ID): 7
Status: NOT_KILLED

Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=off,table_elimination=on,extended_keys=off,exists_to_in=off

The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
information that should help you find out what is causing the crash.

140211 00:04:29 mysqld_safe Number of processes running now: 0
140211 00:04:29 mysqld_safe mysqld restarted
140211 0:04:30 [Note] InnoDB: The InnoDB memory heap is disabled
140211 0:04:30 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
140211 0:04:30 [Note] InnoDB: Compressed tables use zlib 1.2.7
140211 0:04:30 [Note] InnoDB: Using Linux native AIO
140211 0:04:30 [Note] InnoDB: Using CPU crc32 instructions
140211 0:04:30 [Note] InnoDB: Initializing buffer pool, size = 4.0G
140211 0:04:30 [Note] InnoDB: Completed initialization of buffer pool
140211 0:04:30 [Note] InnoDB: Highest supported file format is Barracuda.
140211 0:04:30 [Note] InnoDB: The log sequence numbers 3396091282927 and 3396091282927 in ibdata files do not match the log sequence number 3396091284662 in the ib_logfiles!
140211 0:04:30 [Note] InnoDB: Database was not shutdown normally!
140211 0:04:30 [Note] InnoDB: Starting crash recovery.
140211 0:04:30 [Note] InnoDB: Reading tablespace information from the .ibd files...
140211 0:04:30 [Note] InnoDB: Restoring possible half-written data pages
140211 0:04:30 [Note] InnoDB: from the doublewrite buffer...
InnoDB: Last MySQL binlog file position 0 21857709, file name /home/mysql/log/mysql-bin.000011
140211 0:04:31 [Note] InnoDB: 128 rollback segment(s) are active.
140211 0:04:31 [Note] InnoDB: Waiting for purge to start
140211 0:04:31 [Note] InnoDB: 5.6.14 started; log sequence number 3396091284662
140211 0:04:31 [Note] Plugin 'FEEDBACK' is disabled.
140211 0:04:31 [ERROR] Function 'TokuDB_fractal_tree_block_map' already exists
140211 0:04:31 [Warning] Couldn't load plugin named 'TokuDB_fractal_tree_block_map' with soname 'ha_tokudb.so'.
140211 0:04:31 [ERROR] Function 'TokuDB_fractal_tree_info' already exists
140211 0:04:31 [Warning] Couldn't load plugin named 'TokuDB_fractal_tree_info' with soname 'ha_tokudb.so'.
140211 0:04:31 [ERROR] Function 'TokuDB_file_map' already exists
140211 0:04:31 [Warning] Couldn't load plugin named 'TokuDB_file_map' with soname 'ha_tokudb.so'.
140211 0:04:31 [ERROR] Can't find symbol 'TokuDB_user_data_exact' in library
140211 0:04:31 [Warning] Couldn't load plugin named 'TokuDB_user_data_exact' with soname 'ha_tokudb.so'.
140211 0:04:31 [ERROR] Can't find symbol 'TokuDB_user_data' in library
140211 0:04:31 [Warning] Couldn't load plugin named 'TokuDB_user_data' with soname 'ha_tokudb.so'.
Tue Feb 11 00:04:31 2014 Tokudb recovery starting in env /home/mysql/data/
Tue Feb 11 00:04:31 2014 Tokudb recovery scanning backward from 4948343327
Tue Feb 11 00:04:31 2014 Tokudb recovery bw_end_checkpoint at 4948343327 timestamp 1392073431618680 xid 4948343296 (bw_newer)
Tue Feb 11 00:04:31 2014 Tokudb recovery bw_begin_checkpoint at 4948343296 timestamp 1392073431114318 (bw_between)
Tue Feb 11 00:04:31 2014 Tokudb recovery turning around at begin checkpoint 4948343296 time 504362
Tue Feb 11 00:04:31 2014 Tokudb recovery starts scanning forward to 4948343327 from 4948343296 left 31 (fw_between)
Tue Feb 11 00:04:31 2014 Tokudb recovery closing 29 dictionaries
Tue Feb 11 00:04:31 2014 Tokudb recovery making a checkpoint
Tue Feb 11 00:04:31 2014 Tokudb recovery done
140211 0:04:31 [Note] Recovering after a crash using tc.log
140211 0:04:31 [Note] Starting crash recovery...
140211 0:04:31 [Note] Crash recovery finished.
140211 0:04:31 [Note] Server socket created on IP: '127.0.0.1'.
140211 0:04:31 [Warning] 'user' entry 'prestashop@@ocalhost' ignored in --skip-name-resolve mode.
140211 0:04:31 [Warning] 'db' entry 'prestashop prestashop@@ocalhost' ignored in --skip-name-resolve mode.
140211 0:04:31 [Note] Event Scheduler: Loaded 0 events
140211 0:04:32 [Note] /usr/sbin/mysqld: ready for connections.
Version: '10.0.8-MariaDB-1~wheezy' socket: '/var/run/mysqld/mysqld.sock' port: 3306 mariadb.org binary distribution

Each time, the error occurs in the same table I have to drop and recreate to fix the issue (which is a little bit annoying) :

CREATE TABLE `errors` (
`id_error` int(11) unsigned NOT NULL AUTO_INCREMENT,
`type` varchar(20) NOT NULL DEFAULT '',
`priority` smallint(5) unsigned NOT NULL DEFAULT '0',
`message` text NOT NULL,
`file` varchar(200) NOT NULL DEFAULT '',
`line` smallint(5) unsigned NOT NULL DEFAULT '0',
`url` varchar(200) NOT NULL DEFAULT '',
`referer` varchar(200) NOT NULL DEFAULT '',
`id_user` mediumint(8) unsigned NOT NULL DEFAULT '0',
`server_web` varchar(100) NOT NULL DEFAULT '',
`server_sql` varchar(100) NOT NULL DEFAULT '',
`date` datetime NOT NULL DEFAULT '0000-00-00 00:00:00',
`count` smallint(5) unsigned NOT NULL DEFAULT '1',
`trace` text NOT NULL,
`md5_message` varchar(32) NOT NULL DEFAULT '',
`id_license_domain` int(10) unsigned NOT NULL DEFAULT '0',
`id_license` int(10) unsigned NOT NULL DEFAULT '0',
`extra` varchar(50) NOT NULL DEFAULT '',
PRIMARY KEY (`id_error`),
UNIQUE KEY `error` (`type`,`priority`,`md5_message`,`file`,`line`,`date`,`id_license_domain`),
KEY `date` (`date`),
KEY `md5_message` (`md5_message`,`date`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8

My my.cnf :

transaction-isolation="READ-UNCOMMITTED"
innodb_use_fallocate=1
plugin-load-add=ha_tokudb
tokudb_commit_sync="OFF"
tokudb_cache_size=14G
tokudb_lock_timeout=120000
tokudb_row_format="tokudb_small"
innodb_lock_wait_timeout=120
default_storage_engine=InnoDB
max_allowed_packet=16M
key_buffer_size=128M
table_cache=4096
read_buffer_size=4M
read_rnd_buffer_size=2M
thread_cache_size=80
delayed_insert_limit=500
max_delayed_threads=60
join_buffer_size=2M
sort_buffer_size=2M
delayed_insert_timeout=20
wait_timeout=1800
max_connections=400
back_log=1300
myisam_sort_buffer_size=512M
thread_concurrency=8
tmp_table_size=1G
table_definition_cache = 8192
concurrent_insert=2
innodb_buffer_pool_size=4G
innodb_log_file_size=128M
innodb_log_files_in_group=2
innodb_log_buffer_size=8M
innodb_flush_log_at_trx_commit=2
innodb_thread_concurrency=16
innodb_read_io_threads=8
innodb_write_io_threads=8
innodb_flush_method=O_DIRECT
innodb_file_per_table=1
innodb_io_capacity=2000
innodb_file_format=Barracuda
myisam-recover = BACKUP
query_cache_limit=2M
query_cache_size=32M

They were no issue with MariaDB 5.5.x

Thanks and regards,
Jocelyn Fournier



 Comments   
Comment by Elena Stepanova [ 2014-02-20 ]

jplindst,

This is for your collection of possible data corruption issues; maybe looking at them together will give a new perspective (or maybe they are completely unrelated).

Comment by Jan Lindström (Inactive) [ 2014-02-24 ]

Hi,

I tried to reproduce with creating mentioned table using 10.0.6, upgraded to 10.0.7 without any problems and then upgrade to 10.0.8 and no problems. What kind of device you have tables stored and can you explain what commands you did to see a crash ?

R: Jan

Comment by jocelyn fournier [ 2014-02-24 ]

Hi Jan,

The device is a classical harddrive on a ext4 FS.
I haven't done anything specific other than executing the debian "apt-get upgrade" command.

Note there are several consecutive "[Note] InnoDB: Starting shutdown..." in the log during the standard shutdown, with different log sequence number between
140211 00:00:25 mysqld_safe mysqld from pid file /var/run/mysqld/mysqld.pid ended
and
140211 00:01:23 mysqld_safe Starting mysqld daemon with databases from /home/mysql/data
which seem to be specific to the DB upgrade; when doing a classical /etc/init.d/mysql restart, I do not see this kind of behaviour :

140224 12:45:01 [Note] /usr/sbin/mysqld: Normal shutdown

140224 12:45:01 [Note] Event Scheduler: Purging the queue. 0 events
140224 12:45:03 [Note] InnoDB: FTS optimize thread exiting.
140224 12:45:03 [Note] InnoDB: Starting shutdown...
140224 12:45:06 [Note] InnoDB: Shutdown completed; log sequence number 3396586217139
140224 12:45:06 [Note] /usr/sbin/mysqld: Shutdown complete

140224 12:45:06 mysqld_safe mysqld from pid file /var/run/mysqld/mysqld.pid ended
140224 12:45:07 mysqld_safe Starting mysqld daemon with databases from /home/mysql/data
140224 12:45:07 [Note] InnoDB: The InnoDB memory heap is disabled
140224 12:45:07 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
140224 12:45:07 [Note] InnoDB: Compressed tables use zlib 1.2.7
140224 12:45:07 [Note] InnoDB: Using Linux native AIO
140224 12:45:07 [Note] InnoDB: Using CPU crc32 instructions
140224 12:45:07 [Note] InnoDB: Initializing buffer pool, size = 4.0G
140224 12:45:07 [Note] InnoDB: Completed initialization of buffer pool
140224 12:45:07 [Note] InnoDB: Highest supported file format is Barracuda.
140224 12:45:10 [Note] InnoDB: 128 rollback segment(s) are active.
140224 12:45:10 [Note] InnoDB: Waiting for purge to start
140224 12:45:10 [Note] InnoDB: 5.6.14 started; log sequence number 3396586217139
140224 12:45:10 [Note] Plugin 'FEEDBACK' is disabled.
140224 12:45:10 [ERROR] Function 'TokuDB_fractal_tree_block_map' already exists
140224 12:45:10 [Warning] Couldn't load plugin named 'TokuDB_fractal_tree_block_map' with soname 'ha_tokudb.so'.
140224 12:45:10 [ERROR] Function 'TokuDB_fractal_tree_info' already exists
140224 12:45:10 [Warning] Couldn't load plugin named 'TokuDB_fractal_tree_info' with soname 'ha_tokudb.so'.
140224 12:45:10 [ERROR] Function 'TokuDB_file_map' already exists
140224 12:45:10 [Warning] Couldn't load plugin named 'TokuDB_file_map' with soname 'ha_tokudb.so'.
140224 12:45:10 [ERROR] Can't find symbol 'TokuDB_user_data_exact' in library
140224 12:45:10 [Warning] Couldn't load plugin named 'TokuDB_user_data_exact' with soname 'ha_tokudb.so'.
140224 12:45:10 [ERROR] Can't find symbol 'TokuDB_user_data' in library
140224 12:45:10 [Warning] Couldn't load plugin named 'TokuDB_user_data' with soname 'ha_tokudb.so'.
140224 12:45:10 [Note] Server socket created on IP: '127.0.0.1'.
140224 12:45:10 [Warning] 'user' entry 'prestashop@@ocalhost' ignored in --skip-name-resolve mode.
140224 12:45:10 [Warning] 'db' entry 'prestashop prestashop@@ocalhost' ignored in --skip-name-resolve mode.
140224 12:45:10 [Note] Event Scheduler: Loaded 0 events
140224 12:45:10 [Note] /usr/sbin/mysqld: ready for connections.
Version: '10.0.8-MariaDB-1~wheezy' socket: '/var/run/mysqld/mysqld.sock' port: 3306 mariadb.org binary distribution

Thanks,
Jocelyn

Comment by jocelyn fournier [ 2014-03-07 ]

Just got a new crash not related to any upgrade, and MariaDB seems to be completely unusable this time, I'm not able to restart mysql anymore
FYI, this crash occured during a classic backup. I'll definitly switch back to 5.5 after regenerating the DB from a backup.

140307 1:38:37 [ERROR] InnoDB: File ./a_build/ip2c.ibd: 'Linux aio' returned OS error 0. Cannot continue operation
140307 01:38:39 mysqld_safe Number of processes running now: 0
140307 01:38:39 mysqld_safe mysqld restarted
140307 1:38:40 [Note] InnoDB: The InnoDB memory heap is disabled
140307 1:38:40 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
140307 1:38:40 [Note] InnoDB: Compressed tables use zlib 1.2.7
140307 1:38:40 [Note] InnoDB: Using Linux native AIO
140307 1:38:40 [Note] InnoDB: Using CPU crc32 instructions
140307 1:38:40 [Note] InnoDB: Initializing buffer pool, size = 4.0G
140307 1:38:40 [Note] InnoDB: Completed initialization of buffer pool
140307 1:38:40 [Note] InnoDB: Highest supported file format is Barracuda.
140307 1:38:40 [Note] InnoDB: Log scan progressed past the checkpoint lsn 3397107988944
140307 1:38:40 [Note] InnoDB: Database was not shutdown normally!
140307 1:38:40 [Note] InnoDB: Starting crash recovery.
140307 1:38:40 [Note] InnoDB: Reading tablespace information from the .ibd files...
140307 1:39:00 [Note] InnoDB: Restoring possible half-written data pages
140307 1:39:00 [Note] InnoDB: from the doublewrite buffer...
InnoDB: Doing recovery: scanned up to log sequence number 3397107989420
InnoDB: Last MySQL binlog file position 0 21857709, file name /home/mysql/log/mysql-bin.000011
140307 1:39:01 [Note] InnoDB: 128 rollback segment(s) are active.
140307 1:39:01 [Note] InnoDB: Waiting for purge to start
140307 1:39:01 [Note] InnoDB: 5.6.14 started; log sequence number 3397107989420
140307 1:39:01 [Note] Plugin 'FEEDBACK' is disabled.
140307 1:39:01 [ERROR] Function 'TokuDB_fractal_tree_block_map' already exists
140307 1:39:01 [Warning] Couldn't load plugin named 'TokuDB_fractal_tree_block_map' with soname 'ha_tokudb.so'.
140307 1:39:01 [ERROR] Function 'TokuDB_fractal_tree_info' already exists
140307 1:39:01 [Warning] Couldn't load plugin named 'TokuDB_fractal_tree_info' with soname 'ha_tokudb.so'.
140307 1:39:01 [ERROR] Function 'TokuDB_file_map' already exists
140307 1:39:01 [Warning] Couldn't load plugin named 'TokuDB_file_map' with soname 'ha_tokudb.so'.
140307 1:39:01 [ERROR] Can't find symbol 'TokuDB_user_data_exact' in library
140307 1:39:01 [Warning] Couldn't load plugin named 'TokuDB_user_data_exact' with soname 'ha_tokudb.so'.
140307 1:39:01 [ERROR] Can't find symbol 'TokuDB_user_data' in library
140307 1:39:01 [Warning] Couldn't load plugin named 'TokuDB_user_data' with soname 'ha_tokudb.so'.
InnoDB: Error: trying to access page number 42 in space 326272,
InnoDB: space name jiradb/jiraissue,
InnoDB: which is outside the tablespace bounds.
InnoDB: Byte offset 0, len 16384, i/o type 10.
InnoDB: If you get this error at mysqld startup, please check that
InnoDB: your my.cnf matches the ibdata files that you have in the
InnoDB: MySQL server.
2014-03-07 01:39:01 7eff2dbfc700 InnoDB: Assertion failure in thread 139634449303296 in file fil0fil.cc line 5445
InnoDB: We intentionally generate a memory trap.
InnoDB: Submit a detailed bug report to http://bugs.mysql.com.
InnoDB: If you get repeated assertion failures or crashes, even
InnoDB: immediately after the mysqld startup, there may be
InnoDB: corruption in the InnoDB tablespace. Please refer to
InnoDB: http://dev.mysql.com/doc/refman/5.6/en/forcing-innodb-recovery.html
InnoDB: about forcing recovery.
140307 1:39:01 [ERROR] mysqld got signal 6 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.

To report this bug, see http://kb.askmonty.org/en/reporting-bugs

We will try our best to scrape up some info that will hopefully help
diagnose the problem, but since we have already crashed,
something is definitely wrong and this may fail.

Server version: 10.0.8-MariaDB-1~wheezy
key_buffer_size=134217728
read_buffer_size=4194304
max_used_connections=0
max_threads=402
thread_count=0
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 2609233 K bytes of memory
Hope that's ok; if not, decrease some variables in the equation.

Thread pointer: 0x0x0
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 0x0 thread_stack 0x48000
/usr/sbin/mysqld(my_print_stacktrace+0x2b)[0x7f008555e08b]
/usr/sbin/mysqld(handle_fatal_signal+0x422)[0x7f0085111902]
/lib/x86_64-linux-gnu/libpthread.so.0(+0xf030)[0x7f0084783030]
/lib/x86_64-linux-gnu/libc.so.6(gsignal+0x35)[0x7f0082ddf475]
/lib/x86_64-linux-gnu/libc.so.6(abort+0x180)[0x7f0082de26f0]
/usr/sbin/mysqld(+0x93a025)[0x7f00854ec025]
/usr/sbin/mysqld(+0x8ff6d6)[0x7f00854b16d6]
/usr/sbin/mysqld(+0x900024)[0x7f00854b2024]
/usr/sbin/mysqld(+0x8e7899)[0x7f0085499899]
/usr/sbin/mysqld(+0x8cbc3b)[0x7f008547dc3b]
/usr/sbin/mysqld(+0x875438)[0x7f0085427438]
/usr/sbin/mysqld(+0x8727f3)[0x7f00854247f3]
/usr/sbin/mysqld(+0x8cc16a)[0x7f008547e16a]
/usr/sbin/mysqld(+0x8757bd)[0x7f00854277bd]
/usr/sbin/mysqld(+0x8729ae)[0x7f00854249ae]
/usr/sbin/mysqld(+0x8734da)[0x7f00854254da]
/usr/sbin/mysqld(+0x873a82)[0x7f0085425a82]
/usr/sbin/mysqld(+0x840a7b)[0x7f00853f2a7b]
/usr/sbin/mysqld(+0x899111)[0x7f008544b111]
/usr/sbin/mysqld(+0x88bb9d)[0x7f008543db9d]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x6b50)[0x7f008477ab50]
/lib/x86_64-linux-gnu/libc.so.6(clone+0x6d)[0x7f0082e890ed]
The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
information that should help you find out what is causing the crash.
140307 01:39:02 mysqld_safe mysqld from pid file /var/run/mysqld/mysqld.pid ended

Comment by Jan Lindström (Inactive) [ 2014-03-07 ]

Hi,

What kind of storage you have? Is it SSD ? You could try by setting innodb_use_fallocate=0 and recreate db if that helps, but your database is now broken.

R: Jan

Comment by jocelyn fournier [ 2014-03-07 ]

Hi,

It's a good old classical harddrive (ATA TOSHIBA DT01ACA2), and no smart errors reported (no read error, no pending sector, ...).

I'll try to change innodb_use_fallocate=0. FYI, I'm able to start MariaDB with innodb_force_recovery = 2.
I'll see if dropping the DB "a_build" changes anything

Comment by jocelyn fournier [ 2014-03-07 ]

BTW, FYI the only thing I've done yesterday is creating/dropping a few new databases, tables and users.

Comment by jocelyn fournier [ 2014-03-07 ]

It seems I can work with my DB (no more crash), but I've InnoDB error in the log I don't like at all

I've dropped the table jiradb/jiraissue mentionned in the first error in innodb_force_recovery mode.
Then I've deleted the innodb log file, and restart mariadb, and reimport the table :

140307 8:54:01 [Note] /usr/sbin/mysqld: Normal shutdown

140307 8:54:01 [Note] Event Scheduler: Purging the queue. 0 events
140307 8:54:02 [Note] InnoDB: FTS optimize thread exiting.
140307 8:54:02 [Note] InnoDB: Starting shutdown...
140307 8:54:04 [Note] InnoDB: Shutdown completed; log sequence number 3396587590267
140307 8:54:04 [Note] /usr/sbin/mysqld: Shutdown complete

140307 08:54:04 mysqld_safe mysqld from pid file /var/run/mysqld/mysqld.pid ended
140307 08:54:26 mysqld_safe Starting mysqld daemon with databases from /home/mysql/data
140307 8:54:26 [Note] InnoDB: The InnoDB memory heap is disabled
140307 8:54:26 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
140307 8:54:26 [Note] InnoDB: Compressed tables use zlib 1.2.7
140307 8:54:26 [Note] InnoDB: Using Linux native AIO
140307 8:54:26 [Note] InnoDB: Using CPU crc32 instructions
140307 8:54:26 [Note] InnoDB: Initializing buffer pool, size = 4.0G
140307 8:54:26 [Note] InnoDB: Completed initialization of buffer pool
140307 8:54:26 [Note] InnoDB: Setting log file ./ib_logfile101 size to 128 MB
140307 8:54:26 [Note] InnoDB: Setting log file ./ib_logfile1 size to 128 MB
140307 8:54:26 [Note] InnoDB: Renaming log file ./ib_logfile101 to ./ib_logfile0
140307 8:54:26 [Warning] InnoDB: New log files created, LSN=3396587590267
140307 8:54:26 [Note] InnoDB: Highest supported file format is Barracuda.
2014-03-07 08:54:26 7fac8afc8780 InnoDB: Error: page 7 log sequence number 3397094069371
InnoDB: is in the future! Current system log sequence number 3396587590668.
InnoDB: Your database may be corrupt or you may have copied the InnoDB
InnoDB: tablespace but not the InnoDB log files. See
InnoDB: http://dev.mysql.com/doc/refman/5.6/en/forcing-innodb-recovery.html
InnoDB: for more information.
2014-03-07 08:54:26 7fac8afc8780 InnoDB: Error: page 11 log sequence number 3397093899144
InnoDB: is in the future! Current system log sequence number 3396587590668.
InnoDB: Your database may be corrupt or you may have copied the InnoDB
InnoDB: tablespace but not the InnoDB log files. See
InnoDB: http://dev.mysql.com/doc/refman/5.6/en/forcing-innodb-recovery.html
InnoDB: for more information.
2014-03-07 08:54:26 7fac8afc8780 InnoDB: Error: page 32795 log sequence number 3397107877727
InnoDB: is in the future! Current system log sequence number 3396587590668.
InnoDB: Your database may be corrupt or you may have copied the InnoDB
InnoDB: tablespace but not the InnoDB log files. See
InnoDB: http://dev.mysql.com/doc/refman/5.6/en/forcing-innodb-recovery.html
InnoDB: for more information.

Do you think I've no other solution than recreating all the tables / DB from the backup ?

Thanks and regards,
Jocelyn

Comment by jocelyn fournier [ 2014-03-07 ]

After purging the datas, and especially the innodb_table_stats & innodb_index_stats in the mysql database, and trying to reimport the schema with empty datas, I've just figured out than one (and only one) of the table, the one involved in the first crash needs infos from those tables (I forgot to recreate them) :

2014-03-07 09:35:02 7f442677f700 InnoDB: Error: Fetch of persistent statistics requested for table "MQA"."errors" but the required system tables mysql.innodb_table_stats and mysql.innodb_index_stats are not present or have unexpected structure. Using transient stats instead.

The only thing I've done so far is recreating the schema of all my tables, without any datas.

I'll of course recreate properly the innodb_table_stats & innodb_index_stats, but I want to mention it since this feature is perhaps linked to my original crash ?
Do you know why this table explicitly needs statistics, even without any datas in it ?

The structure of the table :

CREATE TABLE `errors` (
`id_error` int(11) unsigned NOT NULL AUTO_INCREMENT,
`type` varchar(20) NOT NULL DEFAULT '',
`priority` smallint(5) unsigned NOT NULL DEFAULT '0',
`message` text NOT NULL,
`file` varchar(200) NOT NULL DEFAULT '',
`line` smallint(5) unsigned NOT NULL DEFAULT '0',
`url` varchar(200) NOT NULL DEFAULT '',
`referer` varchar(200) NOT NULL DEFAULT '',
`id_user` mediumint(8) unsigned NOT NULL DEFAULT '0',
`server_web` varchar(100) NOT NULL DEFAULT '',
`server_sql` varchar(100) NOT NULL DEFAULT '',
`date` datetime NOT NULL DEFAULT '0000-00-00 00:00:00',
`count` smallint(5) unsigned NOT NULL DEFAULT '1',
`trace` text NOT NULL,
`md5_message` varchar(32) NOT NULL DEFAULT '',
`id_license_domain` int(10) unsigned NOT NULL DEFAULT '0',
`id_license` int(10) unsigned NOT NULL DEFAULT '0',
`extra` varchar(50) NOT NULL DEFAULT '',
PRIMARY KEY (`id_error`),
UNIQUE KEY `error` (`type`,`priority`,`md5_message`,`file`,`line`,`date`,`id_license_domain`),
KEY `date` (`date`),
KEY `md5_message` (`md5_message`,`date`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8

Extract of the log :

140307 09:31:23 mysqld_safe Starting mysqld daemon with databases from /home/mysql/data
140307 9:31:23 [Note] InnoDB: The InnoDB memory heap is disabled
140307 9:31:23 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
140307 9:31:23 [Note] InnoDB: Compressed tables use zlib 1.2.7
140307 9:31:23 [Note] InnoDB: Using Linux native AIO
140307 9:31:23 [Note] InnoDB: Using CPU crc32 instructions
140307 9:31:23 [Note] InnoDB: Initializing buffer pool, size = 4.0G
140307 9:31:23 [Note] InnoDB: Completed initialization of buffer pool
140307 9:31:23 [Note] InnoDB: The first specified data file ./ibdata1 did not exist: a new database to be created!
140307 9:31:23 [Note] InnoDB: Setting file ./ibdata1 size to 12 MB
140307 9:31:23 [Note] InnoDB: Database physically writes the file full: wait...
140307 9:31:23 [Note] InnoDB: Setting log file ./ib_logfile101 size to 128 MB
InnoDB: Progress in MB: 100
140307 9:31:24 [Note] InnoDB: Setting log file ./ib_logfile1 size to 128 MB
InnoDB: Progress in MB: 100
140307 9:31:26 [Note] InnoDB: Renaming log file ./ib_logfile101 to ./ib_logfile0
140307 9:31:26 [Warning] InnoDB: New log files created, LSN=45781
140307 9:31:26 [Note] InnoDB: Doublewrite buffer not found: creating new
140307 9:31:26 [Note] InnoDB: Doublewrite buffer created
140307 9:31:26 [Note] InnoDB: 128 rollback segment(s) are active.
140307 9:31:26 [Warning] InnoDB: Creating foreign key constraint system tables.
140307 9:31:26 [Note] InnoDB: Foreign key constraint system tables created
140307 9:31:26 [Note] InnoDB: Creating tablespace and datafile system tables.
140307 9:31:26 [Note] InnoDB: Tablespace and datafile system tables created.
140307 9:31:26 [Note] InnoDB: Waiting for purge to start
140307 9:31:26 [Note] InnoDB: 5.6.14 started; log sequence number 0
140307 9:31:26 [Note] Plugin 'FEEDBACK' is disabled.
140307 9:31:26 [ERROR] Function 'TokuDB_fractal_tree_block_map' already exists
140307 9:31:26 [Warning] Couldn't load plugin named 'TokuDB_fractal_tree_block_map' with soname 'ha_tokudb.so'.
140307 9:31:26 [ERROR] Function 'TokuDB_fractal_tree_info' already exists
140307 9:31:26 [Warning] Couldn't load plugin named 'TokuDB_fractal_tree_info' with soname 'ha_tokudb.so'.
140307 9:31:26 [ERROR] Function 'TokuDB_file_map' already exists
140307 9:31:26 [Warning] Couldn't load plugin named 'TokuDB_file_map' with soname 'ha_tokudb.so'.
140307 9:31:26 [ERROR] Can't find symbol 'TokuDB_user_data_exact' in library
140307 9:31:26 [Warning] Couldn't load plugin named 'TokuDB_user_data_exact' with soname 'ha_tokudb.so'.
140307 9:31:26 [ERROR] Can't find symbol 'TokuDB_user_data' in library
140307 9:31:26 [Warning] Couldn't load plugin named 'TokuDB_user_data' with soname 'ha_tokudb.so'.
140307 9:31:26 [Note] Server socket created on IP: '127.0.0.1'.
140307 9:31:26 [Warning] 'db' entry 'prestashop prestashop@@ocalhost' ignored in --skip-name-resolve mode.
140307 9:31:26 [Note] Event Scheduler: Loaded 0 events
140307 9:31:26 [Warning] Failed to load slave replication state from table mysql.gtid_slave_pos: 1146: Table 'mysql.gtid_slave_pos' doesn't exist
140307 9:31:26 [Note] /usr/sbin/mysqld: ready for connections.
Version: '10.0.8-MariaDB-1~wheezy' socket: '/var/run/mysqld/mysqld.sock' port: 3306 mariadb.org binary distribution
2014-03-07 09:33:43 7f44267c8700 InnoDB: Error: Table "mysql"."innodb_table_stats" not found.
2014-03-07 09:33:43 7f44267c8700 InnoDB: Error: Table "mysql"."innodb_table_stats" not found.
2014-03-07 09:33:44 7f44267c8700 InnoDB: Error: Table "mysql"."innodb_table_stats" not found.
2014-03-07 09:33:44 7f44267c8700 InnoDB: Error: Table "mysql"."innodb_table_stats" not found.
2014-03-07 09:33:44 7f44267c8700 InnoDB: Error: Table "mysql"."innodb_table_stats" not found.
2014-03-07 09:33:45 7f44267c8700 InnoDB: Error: Table "mysql"."innodb_table_stats" not found.
2014-03-07 09:33:45 7f44267c8700 InnoDB: Error: Table "mysql"."innodb_table_stats" not found.
2014-03-07 09:33:45 7f44267c8700 InnoDB: Error: Table "mysql"."innodb_table_stats" not found.
[...]
2014-03-07 09:35:02 7f442677f700 InnoDB: Error: Fetch of persistent statistics requested for table "MQA"."errors" but the required system tables mysql.innodb_table_stats and mysql.innodb_index_stats are not present or have unexpected structure. Using transient stats instead.
2014-03-07 09:37:05 7f44267c8700 InnoDB: Error: Table "mysql"."innodb_table_stats" not found.
2014-03-07 09:37:05 7f44267c8700 InnoDB: Error: Table "mysql"."innodb_table_stats" not found.
2014-03-07 09:37:05 7f44267c8700 InnoDB: Error: Table "mysql"."innodb_table_stats" not found.
2014-03-07 09:37:06 7f44267c8700 InnoDB: Error: Table "mysql"."innodb_table_stats" not found.
2014-03-07 09:37:06 7f44267c8700 InnoDB: Error: Table "mysql"."innodb_table_stats" not found.
2014-03-07 09:37:06 7f44267c8700 InnoDB: Error: Table "mysql"."innodb_table_stats" not found.
[...]

Comment by Jan Lindström (Inactive) [ 2014-03-07 ]

Hi,

That error is not critical but some features that use these table statistics might not be optimal especially after shutdown, as noted above it will use transient (in-memory) version as persistent table stats can't be stored.

R: Jan

Comment by jocelyn fournier [ 2014-03-07 ]

Hi Jan,

Yes I've recreated those tables, but is this possible there's a bug somewhere in the code using those stats tables which could perhaps explain my initial crash with MQA.errors ?

Thanks and regards,
Jocelyn

Comment by jocelyn fournier [ 2014-03-10 ]

Hi,

FYI, no problem with the 10.0.9 upgrade. However, I still don't really understand why it seems there are several call to start & shutdown during this process ? :

140310 22:10:50 [Note] /usr/sbin/mysqld: Normal shutdown
 
140310 22:10:51 [Note] Event Scheduler: Purging the queue. 0 events
140310 22:10:54 [Note] InnoDB: FTS optimize thread exiting.
140310 22:10:54 [Note] InnoDB: Starting shutdown...
140310 22:10:56 [Note] InnoDB: Shutdown completed; log sequence number 8845526156
140310 22:10:57 [Note] /usr/sbin/mysqld: Shutdown complete
140310 22:10:57 mysqld_safe mysqld from pid file /var/run/mysqld/mysqld.pid ended
140310 22:11:10 [Note] InnoDB: The InnoDB memory heap is disabled
140310 22:11:10 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
140310 22:11:10 [Note] InnoDB: Compressed tables use zlib 1.2.7
140310 22:11:10 [Note] InnoDB: Using Linux native AIO
140310 22:11:10 [Note] InnoDB: Using CPU crc32 instructions
140310 22:11:10 [Note] InnoDB: Initializing buffer pool, size = 4.0G
140310 22:11:11 [Note] InnoDB: Completed initialization of buffer pool
140310 22:11:11 [Note] InnoDB: Highest supported file format is Barracuda.
140310 22:11:13 [Note] InnoDB: 128 rollback segment(s) are active.
140310 22:11:13 [Note] InnoDB: Waiting for purge to start
140310 22:11:13 [Note] InnoDB:  Percona XtraDB (http://www.percona.com) 5.6.15-63.0 started; log sequence number 8845526156
140310 22:11:23 [Note] InnoDB: FTS optimize thread exiting.
140310 22:11:23 [Note] InnoDB: Starting shutdown...
140310 22:11:25 [Note] InnoDB: Shutdown completed; log sequence number 8845528348
140310 22:11:25 [Note] InnoDB: The InnoDB memory heap is disabled
140310 22:11:25 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
140310 22:11:25 [Note] InnoDB: Compressed tables use zlib 1.2.7
140310 22:11:25 [Note] InnoDB: Using Linux native AIO
140310 22:11:25 [Note] InnoDB: Using CPU crc32 instructions
140310 22:11:25 [Note] InnoDB: Initializing buffer pool, size = 4.0G
140310 22:11:25 [Note] InnoDB: Completed initialization of buffer pool
140310 22:11:25 [Note] InnoDB: Highest supported file format is Barracuda.
140310 22:11:26 [Note] InnoDB: 128 rollback segment(s) are active.
140310 22:11:27 [Note] InnoDB: Waiting for purge to start
140310 22:11:27 [Note] InnoDB:  Percona XtraDB (http://www.percona.com) 5.6.15-63.0 started; log sequence number 8845528348
140310 22:11:28 [Note] InnoDB: FTS optimize thread exiting.
140310 22:11:28 [Note] InnoDB: Starting shutdown...
140310 22:11:30 [Note] InnoDB: Shutdown completed; log sequence number 8845531481
140310 22:11:31 [Note] InnoDB: The InnoDB memory heap is disabled
140310 22:11:31 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
140310 22:11:31 [Note] InnoDB: Compressed tables use zlib 1.2.7
140310 22:11:31 [Note] InnoDB: Using Linux native AIO
140310 22:11:31 [Note] InnoDB: Using CPU crc32 instructions
140310 22:11:31 [Note] InnoDB: Initializing buffer pool, size = 4.0G
140310 22:11:31 [Note] InnoDB: Completed initialization of buffer pool
140310 22:11:31 [Note] InnoDB: Highest supported file format is Barracuda.
140310 22:11:32 [Note] InnoDB: 128 rollback segment(s) are active.
140310 22:11:33 [Note] InnoDB: Waiting for purge to start
140310 22:11:33 [Note] InnoDB:  Percona XtraDB (http://www.percona.com) 5.6.15-63.0 started; log sequence number 8845531481
140310 22:11:33 [Note] Plugin 'FEEDBACK' is disabled.
140310 22:11:34 [Note] InnoDB: FTS optimize thread exiting.
140310 22:11:34 [Note] InnoDB: Starting shutdown...
140310 22:11:36 [Note] InnoDB: Shutdown completed; log sequence number 8845535312
140310 22:11:37 [Note] InnoDB: The InnoDB memory heap is disabled
140310 22:11:37 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
140310 22:11:37 [Note] InnoDB: Compressed tables use zlib 1.2.7
140310 22:11:37 [Note] InnoDB: Using Linux native AIO
140310 22:11:37 [Note] InnoDB: Using CPU crc32 instructions
140310 22:11:37 [Note] InnoDB: Initializing buffer pool, size = 4.0G
140310 22:11:37 [Note] InnoDB: Completed initialization of buffer pool
140310 22:11:37 [Note] InnoDB: Highest supported file format is Barracuda.
140310 22:11:38 [Note] InnoDB: 128 rollback segment(s) are active.
140310 22:11:38 [Note] InnoDB: Waiting for purge to start
140310 22:11:38 [Note] InnoDB:  Percona XtraDB (http://www.percona.com) 5.6.15-63.0 started; log sequence number 8845535312
140310 22:11:38 [Note] Plugin 'FEEDBACK' is disabled.
140310 22:11:40 [Note] InnoDB: FTS optimize thread exiting.
140310 22:11:40 [Note] InnoDB: Starting shutdown...
140310 22:11:42 [Note] InnoDB: Shutdown completed; log sequence number 8845539392
140310 22:11:42 [Note] InnoDB: The InnoDB memory heap is disabled
140310 22:11:42 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
140310 22:11:42 [Note] InnoDB: Compressed tables use zlib 1.2.7
140310 22:11:42 [Note] InnoDB: Using Linux native AIO
140310 22:11:42 [Note] InnoDB: Using CPU crc32 instructions
140310 22:11:42 [Note] InnoDB: Initializing buffer pool, size = 4.0G
140310 22:11:42 [Note] InnoDB: Completed initialization of buffer pool
140310 22:11:43 [Note] InnoDB: Highest supported file format is Barracuda.
140310 22:11:44 [Note] InnoDB: 128 rollback segment(s) are active.
140310 22:11:44 [Note] InnoDB: Waiting for purge to start
140310 22:11:44 [Note] InnoDB:  Percona XtraDB (http://www.percona.com) 5.6.15-63.0 started; log sequence number 8845539392
140310 22:11:44 [Note] Plugin 'FEEDBACK' is disabled.
140310 22:11:46 [Note] InnoDB: FTS optimize thread exiting.
140310 22:11:46 [Note] InnoDB: Starting shutdown...
140310 22:11:48 [Note] InnoDB: Shutdown completed; log sequence number 8845543718
140310 22:11:48 mysqld_safe Starting mysqld daemon with databases from /home/mysql/data
140310 22:11:49 [Note] InnoDB: The InnoDB memory heap is disabled
140310 22:11:49 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
140310 22:11:49 [Note] InnoDB: Compressed tables use zlib 1.2.7
140310 22:11:49 [Note] InnoDB: Using Linux native AIO
140310 22:11:49 [Note] InnoDB: Using CPU crc32 instructions
140310 22:11:49 [Note] InnoDB: Initializing buffer pool, size = 4.0G
140310 22:11:49 [Note] InnoDB: Completed initialization of buffer pool
140310 22:11:49 [Note] InnoDB: Highest supported file format is Barracuda.
140310 22:11:50 [Note] InnoDB: 128 rollback segment(s) are active.
140310 22:11:50 [Note] InnoDB: Waiting for purge to start
140310 22:11:50 [Note] InnoDB:  Percona XtraDB (http://www.percona.com) 5.6.15-63.0 started; log sequence number 8845543718
140310 22:11:50 [Note] Plugin 'FEEDBACK' is disabled.
140310 22:11:50 [ERROR] Function 'TokuDB_fractal_tree_block_map' already exists
140310 22:11:50 [Warning] Couldn't load plugin named 'TokuDB_fractal_tree_block_map' with soname 'ha_tokudb.so'.
140310 22:11:50 [ERROR] Function 'TokuDB_fractal_tree_info' already exists
140310 22:11:50 [Warning] Couldn't load plugin named 'TokuDB_fractal_tree_info' with soname 'ha_tokudb.so'.
140310 22:11:50 [ERROR] Function 'TokuDB_file_map' already exists
140310 22:11:50 [Warning] Couldn't load plugin named 'TokuDB_file_map' with soname 'ha_tokudb.so'.
140310 22:11:50 [ERROR] Can't find symbol 'TokuDB_user_data_exact' in library
140310 22:11:50 [Warning] Couldn't load plugin named 'TokuDB_user_data_exact' with soname 'ha_tokudb.so'.
140310 22:11:50 [ERROR] Can't find symbol 'TokuDB_user_data' in library
140310 22:11:50 [Warning] Couldn't load plugin named 'TokuDB_user_data' with soname 'ha_tokudb.so'.
140310 22:11:51 [Note] Server socket created on IP: '127.0.0.1'.
140310 22:11:51 [Warning] 'db' entry 'prestashop prestashop@@ocalhost' ignored in --skip-name-resolve mode.
140310 22:11:51 [Note] Event Scheduler: Loaded 0 events
140310 22:11:51 [Note] /usr/sbin/mysqld: ready for connections.
Version: '10.0.9-MariaDB-1~wheezy'  socket: '/var/run/mysqld/mysqld.sock'  port: 3306  mariadb.org binary distribution
 
Status information:
 
Current dir: /home/mysql/data/
Running threads: 1  Stack size: 294912
Current locks:
lock: 0x7f506f82d198:
 
lock: 0x7f506f818508:
 
lock: 0x7f5198b6f788:
 
lock: 0x7f5198b63788:
 
lock: 0x7f51a0fe8f88:
 
lock: 0x7f5198b62b88:
 
lock: 0x7f51a0fe6788:
 
lock: 0x7f51a0fe5388:
 
lock: 0x7f51b389eb88:
 
lock: 0x7f5198be6388:
 
lock: 0x7f5198b5fb88:
 
lock: 0x7f51a0fe2188:
 
lock: 0x7f51b7445b40:
 
lock: 0x7f51b7445a40:
 
lock: 0x7f51b7444840:
 
lock: 0x7f51b7444740:
 
lock: 0x7f51b7444640:
 
lock: 0x7f51b7444a40:
 
lock: 0x7f51b7442840:
 
lock: 0x7f51b7445940:
 
lock: 0x7f51b7443040:
 
lock: 0x7f51b7443340:
 
lock: 0x7f51b7442f40:
 
lock: 0x7f51b7443240:
 
lock: 0x7f51b7442e40:
 
lock: 0x7f51b7443140:
 
lock: 0x7f51b7443740:
 
lock: 0x7f51b7443840:
 
lock: 0x7f51b7443940:
 
lock: 0x7f51b7442d40:
 
lock: 0x7f51b7442a40:
 
lock: 0x7f51b7442c40:
 
lock: 0x7f51b7442940:
 
lock: 0x7f51b7442b40:
 
lock: 0x7f51b7443440:
 
lock: 0x7f51b7443540:
 
lock: 0x7f51b7443640:
 
lock: 0x7f51b7445540:
 
lock: 0x7f51b7445840:
 
lock: 0x7f51b7445740:
 
lock: 0x7f51b7445640:
 
lock: 0x7f51b7445140:
 
lock: 0x7f51b7445040:
 
lock: 0x7f51b7444f40:
 
lock: 0x7f51b7444e40:
 
lock: 0x7f51b7444d40:
 
lock: 0x7f51b7445340:
 
lock: 0x7f51b7444c40:
 
lock: 0x7f51b7444b40:
 
lock: 0x7f51b7445440:
 
lock: 0x7f51b7444940:
 
lock: 0x7f51b7444440:
 
lock: 0x7f51b7444540:
 
lock: 0x7f51b7444340:
 
lock: 0x7f51b7444240:
 
lock: 0x7f51b7443e40:
 
lock: 0x7f51b7444140:
 
lock: 0x7f51b7444040:
 
lock: 0x7f51b7443d40:
 
lock: 0x7f51b7443f40:
 
lock: 0x7f51b7443a40:
 
lock: 0x7f51b7443b40:
 
lock: 0x7f51b7443c40:
 
lock: 0x7f51b7445240:
 
 
Key caches:
default
Buffer_size:     134217728
Block_size:           1024
Division_limit:        100
Age_threshold:         300
Partitions:              0
blocks used:             4
not flushed:             0
w_requests:              0
writes:                  0
r_requests:              8
reads:                   4
 
 
handler status:
read_key:            0
read_next:           0
read_rnd             0
read_first:          0
write:               0
delete               0
update:              0
 
Table status:
Opened tables:          0
Open tables:           63
Open files:            26
Open streams:           0
 
Alarm status:
Active alarms:   0
Max used alarms: 0
Next alarm time: 0
140310 22:11:54 [Warning] 'db' entry 'prestashop prestashop@@ocalhost' ignored in --skip-name-resolve mode.
 
Memory status:
Non-mmapped space allocated from system: 0
Number of free chunks:			 1
Number of fastbin blocks:		 0
Number of mmapped regions:		 0
Space in mmapped regions:		 0
Maximum total allocated space:		 0
Space available in freed fastbin blocks: 0
Total allocated space:			 0
Total free space:			 0
Top-most, releasable space:		 0
Estimated memory (with thread stack):    294912
 
 
 
Events status:
LLA = Last Locked At  LUA = Last Unlocked At
WOC = Waiting On Condition  DL = Data Locked
 
Event scheduler status:
State      : INITIALIZED
Thread id  : 0
LLA        : n/a:0
LUA        : n/a:0
WOC        : NO
Workers    : 0
Executed   : 0
Data locked: NO
 
Event queue status:
Element count   : 0
Data locked     : NO
Attempting lock : NO
LLA             : init_queue:139
LUA             : init_queue:149
WOC             : NO
Next activation : never
140310 22:12:15 [Warning] 'db' entry 'prestashop prestashop@@ocalhost' ignored in --skip-name-resolve mode.

The output during the upgrade :

Pr?paration du remplacement de mariadb-server 10.0.8+maria-1~wheezy (en utilisant .../mariadb-server_10.0.9+maria-1~wheezy_all.deb) ...
D?paquetage de la mise ? jour de mariadb-server ...
Pr?paration du remplacement de mariadb-client 10.0.8+maria-1~wheezy (en utilisant .../mariadb-client_10.0.9+maria-1~wheezy_all.deb) ...
D?paquetage de la mise ? jour de mariadb-client ...
Pr?paration du remplacement de mysql-common 10.0.8+maria-1~wheezy (en utilisant .../mysql-common_10.0.9+maria-1~wheezy_all.deb) ...
D?paquetage de la mise ? jour de mysql-common ...
Pr?paration du remplacement de mariadb-common 10.0.8+maria-1~wheezy (en utilisant .../mariadb-common_10.0.9+maria-1~wheezy_all.deb) ...
D?paquetage de la mise ? jour de mariadb-common ...
Pr?paration du remplacement de libmariadbclient18 10.0.8+maria-1~wheezy (en utilisant .../libmariadbclient18_10.0.9+maria-1~wheezy_amd64.deb) ...
D?paquetage de la mise ? jour de libmariadbclient18 ...
Pr?paration du remplacement de libmysqlclient18 10.0.8+maria-1~wheezy (en utilisant .../libmysqlclient18_10.0.9+maria-1~wheezy_amd64.deb) ...
D?paquetage de la mise ? jour de libmysqlclient18 ...
Pr?paration du remplacement de mariadb-client-core-10.0 10.0.8+maria-1~wheezy (en utilisant .../mariadb-client-core-10.0_10.0.9+maria-1~wheezy_amd64.deb) ...
D?paquetage de la mise ? jour de mariadb-client-core-10.0 ...
Pr?paration du remplacement de mariadb-client-10.0 10.0.8+maria-1~wheezy (en utilisant .../mariadb-client-10.0_10.0.9+maria-1~wheezy_amd64.deb) ...
D?paquetage de la mise ? jour de mariadb-client-10.0 ...
Pr?paration du remplacement de mariadb-server-core-10.0 10.0.8+maria-1~wheezy (en utilisant .../mariadb-server-core-10.0_10.0.9+maria-1~wheezy_amd64.deb) ...
D?paquetage de la mise ? jour de mariadb-server-core-10.0 ...
Traitement des actions diff?r?es (??triggers??) pour ??man-db??...
Param?trage de mysql-common (10.0.9+maria-1~wheezy) ...
Param?trage de mariadb-common (10.0.9+maria-1~wheezy) ...
(Lecture de la base de donn?es... 97806 fichiers et r?pertoires d?j? install?s.)
Pr?paration du remplacement de mariadb-server-10.0 10.0.8+maria-1~wheezy (en utilisant .../mariadb-server-10.0_10.0.9+maria-1~wheezy_amd64.deb) ...
Stopping MariaDB database server: mysqld.
Stopping MariaDB database server: mysqld.
D?paquetage de la mise ? jour de mariadb-server-10.0 ...
Pr?paration du remplacement de libmariadbclient-dev 10.0.8+maria-1~wheezy (en utilisant .../libmariadbclient-dev_10.0.9+maria-1~wheezy_amd64.deb) ...
D?paquetage de la mise ? jour de libmariadbclient-dev ...
Pr?paration du remplacement de libmariadbd-dev 10.0.8+maria-1~wheezy (en utilisant .../libmariadbd-dev_10.0.9+maria-1~wheezy_amd64.deb) ...
D?paquetage de la mise ? jour de libmariadbd-dev ...
Pr?paration du remplacement de udisks 1.0.4-7 (en utilisant .../udisks_1.0.4-7wheezy1_amd64.deb) ...
D?paquetage de la mise ? jour de udisks ...
Traitement des actions diff?r?es (??triggers??) pour ??man-db??...
Param?trage de udisks (1.0.4-7wheezy1) ...
Param?trage de libmysqlclient18 (10.0.9+maria-1~wheezy) ...
Param?trage de libmariadbclient18 (10.0.9+maria-1~wheezy) ...
Param?trage de mariadb-client-core-10.0 (10.0.9+maria-1~wheezy) ...
Param?trage de mariadb-client-10.0 (10.0.9+maria-1~wheezy) ...
Param?trage de mariadb-server-core-10.0 (10.0.9+maria-1~wheezy) ...
Param?trage de mariadb-server-10.0 (10.0.9+maria-1~wheezy) ...
Stopping MariaDB database server: mysqld.
Starting MariaDB database server: mysqld . ..
Checking for corrupt, not cleanly closed and upgrade needing tables..
Param?trage de mariadb-server (10.0.9+maria-1~wheezy) ...
Param?trage de mariadb-client (10.0.9+maria-1~wheezy) ...
Param?trage de libmariadbclient-dev (10.0.9+maria-1~wheezy) ...
Param?trage de libmariadbd-dev (10.0.9+maria-1~wheezy) ...

Comment by Jan Lindström (Inactive) [ 2014-03-11 ]

I'm not familiar with debian upgrade process but based on log these were normal shutdowns i.e. external shutdown requests not any crashes or unexpected shutdowns. Now that upgrade does work, I will close the issue.

Comment by jocelyn fournier [ 2014-03-11 ]

Hi Jan,

Not sure the initial problem is really fixed : I've disabled innodb_use_fallocate before the upgrade.
I'll let you know if I have any other similar corruption.

Thanks,
Jocelyn

Generated at Thu Feb 08 07:06:20 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.