Details
-
Bug
-
Status: Closed (View Workflow)
-
Major
-
Resolution: Fixed
-
10.1.13
-
None
-
10.1.19
Description
We're use mariadb 10.1.13 as a main database of our system.
We migrated data from develpment server to product server a week ago.
Below is our error case :
MariaDB shutdown with signal 6 when some query executes. (insert or update...)
The error log is below.
...
|
2016-07-18 9:44:14 139882584566528 [ERROR] InnoDB: Block in space_id 0 in file /data001/masvc01/ibdata1 encrypted.
|
2016-07-18 9:44:14 139882584566528 [ERROR] InnoDB: However key management plugin or used key_id 286 is not found or used encryption algorithm or method does not match.
|
2016-07-18 9:44:14 139882584566528 [ERROR] InnoDB: Marking tablespace as missing. You may drop this table or install correct key management plugin and key file.
|
2016-07-18 9:44:14 139882584566528 [ERROR] InnoDB: Block in space_id 0 in file /data001/masvc01/ibdata1 encrypted.
|
2016-07-18 9:44:14 139882584566528 [ERROR] InnoDB: However key management plugin or used key_id 286 is not found or used encryption algorithm or method does not match.
|
2016-07-18 9:44:14 139882584566528 [ERROR] InnoDB: Marking tablespace as missing. You may drop this table or install correct key management plugin and key file.
|
2016-07-18 09:44:14 7f38f3c36b00 InnoDB: Assertion failure in thread 139882584566528 in file buf0buf.cc line 4803
|
...
|
We check out system innodb table space with innochecksum.
The result is below.
/engn001/masvc01/mysql/bin/innochecksum -v ibdata1
|
 |
InnoDB offline file checksum utility.
|
|
Variables (--variable-name=value)
|
and boolean options {FALSE|TRUE} Value (after reading options)
|
--------------------------------- ----------------------------------------
|
verbose TRUE
|
debug FALSE
|
skip-corrupt FALSE
|
count FALSE
|
start-page 0
|
end-page 0
|
page 0
|
per-page-details FALSE
|
leaf FALSE
|
merge 0
|
Table is uncompressed
|
Page size is 16384
|
file ibdata1_20160718 = 3221225472 bytes (196608 pages)...
|
InnoChecksum; checking pages in range 0 to 196607
|
Fail; page 0 invalid (fails innodb and crc32 checksum)
|
We hava ibdata1, ibdata2, ibdata3 and ibdata4 and results of innochecksum are same.
We compiled mariadb with debug option and test under same enviroment.
Below is the mysql.trace when db crash.
==========================================================================================================================
|
T@10 : | | | | | | | | | | | | | | | | | >print_buffer_to_file
|
T@10 : | | | | | | | | | | | | | | | | | | enter: buffer: InnoDB: Marking tablespace as missing. You may drop this table or install correct key management plugin and key file.
|
T@10 : | | | | | | | | | | | | | | | | | | mutex: LOCK_error_log (0x15b4b60) locking
|
T@10 : | | | | | | | | | | | | | | | | | | mutex: LOCK_error_log (0x15b4b60) locked
|
T@10 : | | | | | | | | | | | | | | | | | | mutex: LOCK_error_log (0x15b4b60) unlocking
|
T@10 : | | | | | | | | | | | | | | | | | <print_buffer_to_file
|
T@10 : | | | | | | | | | | | | | | | | <vprint_msg_to_log
|
T@10 : | | | | | | | | | | | | | | | <sql_print_error
|
T@10 : | | | | | | | | | | | | | | | >my_malloc
|
T@10 : | | | | | | | | | | | | | | | | my: size: 4096 my_flags: 16
|
T@10 : | | | | | | | | | | | | | | | | exit: ptr: 0x7f9e84159690
|
T@10 : | | | | | | | | | | | | | | | <my_malloc
|
T@10 : | | | | | | | | | | | | | | | >push_warning_printf
|
T@10 : | | | | | | | | | | | | | | | | enter: warning: 192
|
T@10 : | | | | | | | | | | | | | | | | >push_warning
|
T@10 : | | | | | | | | | | | | | | | | | enter: code: 192, msg: Table in tablespace 0 encrypted.However key management plugin or used key_id 290 is not found or used encryption algorithm or method does not match. Can't continue opening the table.
|
T@10 : | | | | | | | | | | | | | | | | | >THD::raise_condition
|
T@10 : | | | | | | | | | | | | | | | | | | >mysql_audit_acquire_plugins
|
T@10 : | | | | | | | | | | | | | | | | | | <mysql_audit_acquire_plugins
|
T@10 : | | | | | | | | | | | | | | | | | | mutex: l_perm->lock (0x5dcaeb00) locking
|
T@10 : | | | | | | | | | | | | | | | | | | >my_multi_malloc
|
T@10 : | | | | | | | | | | | | | | | | | | | >my_malloc
|
T@10 : | | | | | | | | | | | | | | | | | | | | my: size: 208 my_flags: 24
|
T@10 : | | | | | | | | | | | | | | | | | | | | exit: ptr: 0x7f9e8415a710
|
T@10 : | | | | | | | | | | | | | | | | | | | <my_malloc
|
T@10 : | | | | | | | | | | | | | | | | | | <my_multi_malloc
|
T@10 : | | | | | | | | | | | | | | | | | | >my_hash_init
|
T@10 : | | | | | | | | | | | | | | | | | | | enter: hash: 0x7f9e8415a710 size: 128
|
T@10 : | | | | | | | | | | | | | | | | | | | >init_dynamic_array2
|
T@10 : | | | | | | | | | | | | | | | | | | | | >my_malloc
|
So we export data from mariadb and import to new table space.
After migration, we shutdown mariadb and restart. but signal 11 occurred and db crash.
Below is our error log.
2016-07-19 8:55:09 140556344748800 [Note] /engn001/masvc01/mysql/bin/mysqld: Normal shutdown
|
 |
2016-07-19 8:55:09 140556344748800 [Note] Event Scheduler: Killing the scheduler thread, thread id 1
|
2016-07-19 8:55:09 140556344748800 [Note] Event Scheduler: Waiting for the scheduler thread to reply
|
2016-07-19 8:55:09 140556344748800 [Note] Event Scheduler: Stopped
|
2016-07-19 8:55:09 140556344748800 [Note] Event Scheduler: Purging the queue. 0 events
|
160719 8:55:09 server_audit: STOPPED
|
2016-07-19 8:55:09 140556627830528 [Note] InnoDB: FTS optimize thread exiting.
|
2016-07-19 8:55:09 140556344748800 [Note] InnoDB: Starting shutdown...
|
2016-07-19 8:55:09 140556648810240 [Warning] InnoDB: Dumping buffer pool(s) to /data001/masvc01/ib_buffer_pool
|
2016-07-19 8:55:10 140556648810240 [Warning] InnoDB: Buffer pool(s) dump completed at 160719 8:55:10
|
2016-07-19 8:55:15 140556344748800 [Note] InnoDB: Shutdown completed; log sequence number 399145649446
|
2016-07-19 8:55:15 140556344748800 [Note] /engn001/masvc01/mysql/bin/mysqld: Shutdown complete
|
 |
160719 08:55:16 mysqld_safe mysqld from pid file /engn001/masvc01/mysql/mysqld.pid ended
|
160719 08:57:05 mysqld_safe Starting mysqld daemon with databases from /data001/masvc01
|
2016-07-19 8:57:05 140304014350176 [Note] /engn001/masvc01/mysql/bin/mysqld (mysqld 10.1.14-MariaDB) starting as process 2608 ...
|
2016-07-19 8:57:06 140304014350176 [Note] InnoDB: Using mutexes to ref count buffer pool pages
|
2016-07-19 8:57:06 140304014350176 [Note] InnoDB: The InnoDB memory heap is disabled
|
2016-07-19 8:57:06 140304014350176 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
|
2016-07-19 8:57:06 140304014350176 [Note] InnoDB: Memory barrier is not used
|
2016-07-19 8:57:06 140304014350176 [Note] InnoDB: Compressed tables use zlib 1.2.3
|
2016-07-19 8:57:06 140304014350176 [Note] InnoDB: Using Linux native AIO
|
2016-07-19 8:57:06 140304014350176 [Note] InnoDB: Using SSE crc32 instructions
|
2016-07-19 8:57:06 140304014350176 [Note] InnoDB: Initializing buffer pool, size = 24.0G
|
2016-07-19 8:57:08 140304014350176 [Note] InnoDB: Completed initialization of buffer pool
|
2016-07-19 8:57:10 140304014350176 [Note] InnoDB: Highest supported file format is Barracuda.
|
2016-07-19 8:57:13 140304014350176 [Note] InnoDB: 128 rollback segment(s) are active.
|
2016-07-19 8:57:13 140304014350176 [Note] InnoDB: Waiting for purge to start
|
2016-07-19 8:57:13 140304014350176 [Note] InnoDB: Percona XtraDB (http://www.percona.com) 5.6.29-76.2 started; log sequence number 399145649446
|
2016-07-19 8:57:13 140273638147840 [Note] InnoDB: Dumping buffer pool(s) not yet started
|
2016-07-19 08:57:13 7f94005f8700 InnoDB: Loading buffer pool(s) from /data001/masvc01/ib_buffer_pool
|
2016-07-19 8:57:13 140304014350176 [Note] Plugin 'FEEDBACK' is disabled.
|
160719 8:57:13 server_audit: MariaDB Audit Plugin version 1.4.0 STARTED.
|
2016-07-19 8:57:13 140304014350176 [Note] Server socket created on IP: '::'.
|
2016-07-19 8:57:13 140304014350176 [Warning] 'user' entry 'root@lgedglap02v' ignored in --skip-name-resolve mode.
|
2016-07-19 8:57:13 140304014350176 [Warning] 'user' entry '@lgedglap02v' ignored in --skip-name-resolve mode.
|
2016-07-19 8:57:13 140304014350176 [Warning] 'proxies_priv' entry '@% root@lgedglap02v' ignored in --skip-name-resolve mode.
|
2016-07-19 8:57:13 140273595853568 [Note] Event Scheduler: scheduler thread started with id 1
|
2016-07-19 8:57:13 140304014350176 [Note] Reading of all Master_info entries succeded
|
2016-07-19 8:57:13 140304014350176 [Note] Added new Master_info '' to hash table
|
2016-07-19 8:57:13 140304014350176 [Note] /engn001/masvc01/mysql/bin/mysqld: ready for connections.
|
Version: '10.1.14-MariaDB' socket: '/engn001/masvc01/mysql/mysqld.sock' port: 3310 MariaDB Server
|
2016-07-19 8:57:47 140273638147840 [ERROR] InnoDB: Block in space_id 0 in file /data001/masvc01/ibdata1 encrypted.
|
2016-07-19 8:57:47 140273638147840 [ERROR] InnoDB: However key management plugin or used key_id 92 is not found or used encryption algorithm or method does not match.
|
2016-07-19 8:57:47 140273638147840 [ERROR] InnoDB: Marking tablespace as missing. You may drop this table or install correct key management plugin and key file.
|
2016-07-19 8:57:47 140273638147840 [ERROR] InnoDB: Block in space_id 0 in file /data001/masvc01/ibdata1 encrypted.
|
2016-07-19 8:57:47 140273638147840 [ERROR] InnoDB: However key management plugin or used key_id 92 is not found or used encryption algorithm or method does not match.
|
2016-07-19 8:57:47 140273638147840 [ERROR] InnoDB: Marking tablespace as missing. You may drop this table or install correct key management plugin and key file.
|
160719 8:57:47 [ERROR] mysqld got signal 11 ;
|
This could be because you hit a bug. It is also possible that this binary
|
or one of the libraries it was linked against is corrupt, improperly built,
|
or misconfigured. This error can also be caused by malfunctioning hardware.
|
 |
To report this bug, see https://mariadb.com/kb/en/reporting-bugs
|
 |
We will try our best to scrape up some info that will hopefully help
|
diagnose the problem, but since we have already crashed,
|
something is definitely wrong and this may fail.
|
 |
Server version: 10.1.14-MariaDB
|
key_buffer_size=67108864
|
read_buffer_size=50331648
|
max_used_connections=3
|
max_threads=102
|
thread_count=4
|
It is possible that mysqld could use up to
|
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 10094612 K bytes of memory
|
Hope that's ok; if not, decrease some variables in the equation.
|
 |
Thread pointer: 0x0x0
|
Attempting backtrace. You can use the following information to find out
|
where mysqld died. If you see no messages after this, something went
|
terribly wrong...
|
stack_bottom = 0x0 thread_stack 0x48400
|
/engn001/masvc01/mysql/bin/mysqld(my_print_stacktrace+0x2e)[0xc028ae]
|
/engn001/masvc01/mysql/bin/mysqld(handle_fatal_signal+0x464)[0x7651f4]
|
/lib64/libpthread.so.0[0x3ffe40f7e0]
|
/engn001/masvc01/mysql/bin/mysqld[0x8fa6d3]
|
/engn001/masvc01/mysql/bin/mysqld[0xa3ba47]
|
/engn001/masvc01/mysql/bin/mysqld[0xa51a2e]
|
/engn001/masvc01/mysql/bin/mysqld[0xa52bc9]
|
/engn001/masvc01/mysql/bin/mysqld[0xa433ff]
|
/engn001/masvc01/mysql/bin/mysqld[0xa440b7]
|
/lib64/libpthread.so.0[0x3ffe407aa1]
|
/lib64/libc.so.6(clone+0x6d)[0x3ffe0e893d]
|
The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
|
information that should help you find out what is causing the crash.
|
.....
|
|
I can't find solution for it. I attached my.cnf.
Is it a bug or hardware fault ?
Attachments
Issue Links
- includes
-
-
- Closed
-
- relates to
-
-
- Closed
-