Details
-
Bug
-
Status: Closed (View Workflow)
-
Major
-
Resolution: Fixed
-
10.1.13
-
None
-
10.1.19
Description
We're use mariadb 10.1.13 as a main database of our system.
We migrated data from develpment server to product server a week ago.
Below is our error case :
MariaDB shutdown with signal 6 when some query executes. (insert or update...)
The error log is below.
...
|
2016-07-18 9:44:14 139882584566528 [ERROR] InnoDB: Block in space_id 0 in file /data001/masvc01/ibdata1 encrypted.
|
2016-07-18 9:44:14 139882584566528 [ERROR] InnoDB: However key management plugin or used key_id 286 is not found or used encryption algorithm or method does not match.
|
2016-07-18 9:44:14 139882584566528 [ERROR] InnoDB: Marking tablespace as missing. You may drop this table or install correct key management plugin and key file.
|
2016-07-18 9:44:14 139882584566528 [ERROR] InnoDB: Block in space_id 0 in file /data001/masvc01/ibdata1 encrypted.
|
2016-07-18 9:44:14 139882584566528 [ERROR] InnoDB: However key management plugin or used key_id 286 is not found or used encryption algorithm or method does not match.
|
2016-07-18 9:44:14 139882584566528 [ERROR] InnoDB: Marking tablespace as missing. You may drop this table or install correct key management plugin and key file.
|
2016-07-18 09:44:14 7f38f3c36b00 InnoDB: Assertion failure in thread 139882584566528 in file buf0buf.cc line 4803
|
...
|
We check out system innodb table space with innochecksum.
The result is below.
/engn001/masvc01/mysql/bin/innochecksum -v ibdata1
|
|
|
InnoDB offline file checksum utility.
|
|
Variables (--variable-name=value)
|
and boolean options {FALSE|TRUE} Value (after reading options)
|
--------------------------------- ----------------------------------------
|
verbose TRUE
|
debug FALSE
|
skip-corrupt FALSE
|
count FALSE
|
start-page 0
|
end-page 0
|
page 0
|
per-page-details FALSE
|
leaf FALSE
|
merge 0
|
Table is uncompressed
|
Page size is 16384
|
file ibdata1_20160718 = 3221225472 bytes (196608 pages)...
|
InnoChecksum; checking pages in range 0 to 196607
|
Fail; page 0 invalid (fails innodb and crc32 checksum)
|
We hava ibdata1, ibdata2, ibdata3 and ibdata4 and results of innochecksum are same.
We compiled mariadb with debug option and test under same enviroment.
Below is the mysql.trace when db crash.
==========================================================================================================================
|
T@10 : | | | | | | | | | | | | | | | | | >print_buffer_to_file
|
T@10 : | | | | | | | | | | | | | | | | | | enter: buffer: InnoDB: Marking tablespace as missing. You may drop this table or install correct key management plugin and key file.
|
T@10 : | | | | | | | | | | | | | | | | | | mutex: LOCK_error_log (0x15b4b60) locking
|
T@10 : | | | | | | | | | | | | | | | | | | mutex: LOCK_error_log (0x15b4b60) locked
|
T@10 : | | | | | | | | | | | | | | | | | | mutex: LOCK_error_log (0x15b4b60) unlocking
|
T@10 : | | | | | | | | | | | | | | | | | <print_buffer_to_file
|
T@10 : | | | | | | | | | | | | | | | | <vprint_msg_to_log
|
T@10 : | | | | | | | | | | | | | | | <sql_print_error
|
T@10 : | | | | | | | | | | | | | | | >my_malloc
|
T@10 : | | | | | | | | | | | | | | | | my: size: 4096 my_flags: 16
|
T@10 : | | | | | | | | | | | | | | | | exit: ptr: 0x7f9e84159690
|
T@10 : | | | | | | | | | | | | | | | <my_malloc
|
T@10 : | | | | | | | | | | | | | | | >push_warning_printf
|
T@10 : | | | | | | | | | | | | | | | | enter: warning: 192
|
T@10 : | | | | | | | | | | | | | | | | >push_warning
|
T@10 : | | | | | | | | | | | | | | | | | enter: code: 192, msg: Table in tablespace 0 encrypted.However key management plugin or used key_id 290 is not found or used encryption algorithm or method does not match. Can't continue opening the table.
|
T@10 : | | | | | | | | | | | | | | | | | >THD::raise_condition
|
T@10 : | | | | | | | | | | | | | | | | | | >mysql_audit_acquire_plugins
|
T@10 : | | | | | | | | | | | | | | | | | | <mysql_audit_acquire_plugins
|
T@10 : | | | | | | | | | | | | | | | | | | mutex: l_perm->lock (0x5dcaeb00) locking
|
T@10 : | | | | | | | | | | | | | | | | | | >my_multi_malloc
|
T@10 : | | | | | | | | | | | | | | | | | | | >my_malloc
|
T@10 : | | | | | | | | | | | | | | | | | | | | my: size: 208 my_flags: 24
|
T@10 : | | | | | | | | | | | | | | | | | | | | exit: ptr: 0x7f9e8415a710
|
T@10 : | | | | | | | | | | | | | | | | | | | <my_malloc
|
T@10 : | | | | | | | | | | | | | | | | | | <my_multi_malloc
|
T@10 : | | | | | | | | | | | | | | | | | | >my_hash_init
|
T@10 : | | | | | | | | | | | | | | | | | | | enter: hash: 0x7f9e8415a710 size: 128
|
T@10 : | | | | | | | | | | | | | | | | | | | >init_dynamic_array2
|
T@10 : | | | | | | | | | | | | | | | | | | | | >my_malloc
|
So we export data from mariadb and import to new table space.
After migration, we shutdown mariadb and restart. but signal 11 occurred and db crash.
Below is our error log.
2016-07-19 8:55:09 140556344748800 [Note] /engn001/masvc01/mysql/bin/mysqld: Normal shutdown
|
|
|
2016-07-19 8:55:09 140556344748800 [Note] Event Scheduler: Killing the scheduler thread, thread id 1
|
2016-07-19 8:55:09 140556344748800 [Note] Event Scheduler: Waiting for the scheduler thread to reply
|
2016-07-19 8:55:09 140556344748800 [Note] Event Scheduler: Stopped
|
2016-07-19 8:55:09 140556344748800 [Note] Event Scheduler: Purging the queue. 0 events
|
160719 8:55:09 server_audit: STOPPED
|
2016-07-19 8:55:09 140556627830528 [Note] InnoDB: FTS optimize thread exiting.
|
2016-07-19 8:55:09 140556344748800 [Note] InnoDB: Starting shutdown...
|
2016-07-19 8:55:09 140556648810240 [Warning] InnoDB: Dumping buffer pool(s) to /data001/masvc01/ib_buffer_pool
|
2016-07-19 8:55:10 140556648810240 [Warning] InnoDB: Buffer pool(s) dump completed at 160719 8:55:10
|
2016-07-19 8:55:15 140556344748800 [Note] InnoDB: Shutdown completed; log sequence number 399145649446
|
2016-07-19 8:55:15 140556344748800 [Note] /engn001/masvc01/mysql/bin/mysqld: Shutdown complete
|
|
|
160719 08:55:16 mysqld_safe mysqld from pid file /engn001/masvc01/mysql/mysqld.pid ended
|
160719 08:57:05 mysqld_safe Starting mysqld daemon with databases from /data001/masvc01
|
2016-07-19 8:57:05 140304014350176 [Note] /engn001/masvc01/mysql/bin/mysqld (mysqld 10.1.14-MariaDB) starting as process 2608 ...
|
2016-07-19 8:57:06 140304014350176 [Note] InnoDB: Using mutexes to ref count buffer pool pages
|
2016-07-19 8:57:06 140304014350176 [Note] InnoDB: The InnoDB memory heap is disabled
|
2016-07-19 8:57:06 140304014350176 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
|
2016-07-19 8:57:06 140304014350176 [Note] InnoDB: Memory barrier is not used
|
2016-07-19 8:57:06 140304014350176 [Note] InnoDB: Compressed tables use zlib 1.2.3
|
2016-07-19 8:57:06 140304014350176 [Note] InnoDB: Using Linux native AIO
|
2016-07-19 8:57:06 140304014350176 [Note] InnoDB: Using SSE crc32 instructions
|
2016-07-19 8:57:06 140304014350176 [Note] InnoDB: Initializing buffer pool, size = 24.0G
|
2016-07-19 8:57:08 140304014350176 [Note] InnoDB: Completed initialization of buffer pool
|
2016-07-19 8:57:10 140304014350176 [Note] InnoDB: Highest supported file format is Barracuda.
|
2016-07-19 8:57:13 140304014350176 [Note] InnoDB: 128 rollback segment(s) are active.
|
2016-07-19 8:57:13 140304014350176 [Note] InnoDB: Waiting for purge to start
|
2016-07-19 8:57:13 140304014350176 [Note] InnoDB: Percona XtraDB (http://www.percona.com) 5.6.29-76.2 started; log sequence number 399145649446
|
2016-07-19 8:57:13 140273638147840 [Note] InnoDB: Dumping buffer pool(s) not yet started
|
2016-07-19 08:57:13 7f94005f8700 InnoDB: Loading buffer pool(s) from /data001/masvc01/ib_buffer_pool
|
2016-07-19 8:57:13 140304014350176 [Note] Plugin 'FEEDBACK' is disabled.
|
160719 8:57:13 server_audit: MariaDB Audit Plugin version 1.4.0 STARTED.
|
2016-07-19 8:57:13 140304014350176 [Note] Server socket created on IP: '::'.
|
2016-07-19 8:57:13 140304014350176 [Warning] 'user' entry 'root@lgedglap02v' ignored in --skip-name-resolve mode.
|
2016-07-19 8:57:13 140304014350176 [Warning] 'user' entry '@lgedglap02v' ignored in --skip-name-resolve mode.
|
2016-07-19 8:57:13 140304014350176 [Warning] 'proxies_priv' entry '@% root@lgedglap02v' ignored in --skip-name-resolve mode.
|
2016-07-19 8:57:13 140273595853568 [Note] Event Scheduler: scheduler thread started with id 1
|
2016-07-19 8:57:13 140304014350176 [Note] Reading of all Master_info entries succeded
|
2016-07-19 8:57:13 140304014350176 [Note] Added new Master_info '' to hash table
|
2016-07-19 8:57:13 140304014350176 [Note] /engn001/masvc01/mysql/bin/mysqld: ready for connections.
|
Version: '10.1.14-MariaDB' socket: '/engn001/masvc01/mysql/mysqld.sock' port: 3310 MariaDB Server
|
2016-07-19 8:57:47 140273638147840 [ERROR] InnoDB: Block in space_id 0 in file /data001/masvc01/ibdata1 encrypted.
|
2016-07-19 8:57:47 140273638147840 [ERROR] InnoDB: However key management plugin or used key_id 92 is not found or used encryption algorithm or method does not match.
|
2016-07-19 8:57:47 140273638147840 [ERROR] InnoDB: Marking tablespace as missing. You may drop this table or install correct key management plugin and key file.
|
2016-07-19 8:57:47 140273638147840 [ERROR] InnoDB: Block in space_id 0 in file /data001/masvc01/ibdata1 encrypted.
|
2016-07-19 8:57:47 140273638147840 [ERROR] InnoDB: However key management plugin or used key_id 92 is not found or used encryption algorithm or method does not match.
|
2016-07-19 8:57:47 140273638147840 [ERROR] InnoDB: Marking tablespace as missing. You may drop this table or install correct key management plugin and key file.
|
160719 8:57:47 [ERROR] mysqld got signal 11 ;
|
This could be because you hit a bug. It is also possible that this binary
|
or one of the libraries it was linked against is corrupt, improperly built,
|
or misconfigured. This error can also be caused by malfunctioning hardware.
|
|
|
To report this bug, see https://mariadb.com/kb/en/reporting-bugs
|
|
|
We will try our best to scrape up some info that will hopefully help
|
diagnose the problem, but since we have already crashed,
|
something is definitely wrong and this may fail.
|
|
|
Server version: 10.1.14-MariaDB
|
key_buffer_size=67108864
|
read_buffer_size=50331648
|
max_used_connections=3
|
max_threads=102
|
thread_count=4
|
It is possible that mysqld could use up to
|
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 10094612 K bytes of memory
|
Hope that's ok; if not, decrease some variables in the equation.
|
|
|
Thread pointer: 0x0x0
|
Attempting backtrace. You can use the following information to find out
|
where mysqld died. If you see no messages after this, something went
|
terribly wrong...
|
stack_bottom = 0x0 thread_stack 0x48400
|
/engn001/masvc01/mysql/bin/mysqld(my_print_stacktrace+0x2e)[0xc028ae]
|
/engn001/masvc01/mysql/bin/mysqld(handle_fatal_signal+0x464)[0x7651f4]
|
/lib64/libpthread.so.0[0x3ffe40f7e0]
|
/engn001/masvc01/mysql/bin/mysqld[0x8fa6d3]
|
/engn001/masvc01/mysql/bin/mysqld[0xa3ba47]
|
/engn001/masvc01/mysql/bin/mysqld[0xa51a2e]
|
/engn001/masvc01/mysql/bin/mysqld[0xa52bc9]
|
/engn001/masvc01/mysql/bin/mysqld[0xa433ff]
|
/engn001/masvc01/mysql/bin/mysqld[0xa440b7]
|
/lib64/libpthread.so.0[0x3ffe407aa1]
|
/lib64/libc.so.6(clone+0x6d)[0x3ffe0e893d]
|
The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
|
information that should help you find out what is causing the crash.
|
.....
|
|
I can't find solution for it. I attached my.cnf.
Is it a bug or hardware fault ?
Attachments
Issue Links
- includes
-
-
- Closed
-
- relates to
-
-
- Closed
-
Activity
Hi,
Can you try with 10.1.16, there was some bad bugs on 10.1.13.
I got the same error on current 10.1. The difference is that instead of an assertion failure I'm getting SIGSEGV.
2016-09-05 16:08:20 140463950751616 [ERROR] InnoDB: Space id in fsp header 131344,but in the page header 0
|
2016-09-05 16:08:20 140463950751616 [Note] InnoDB: Highest supported file format is Barracuda.
|
2016-09-05 16:08:21 140463950751616 [ERROR] InnoDB: Block in space_id 0 in file ./ibdata1 encrypted.
|
2016-09-05 16:08:21 140463950751616 [ERROR] InnoDB: However key management plugin or used key_id 5 is not found or used encryption algorithm or method does not match.
|
2016-09-05 16:08:21 140463950751616 [ERROR] InnoDB: Marking tablespace as missing. You may drop this table or install correct key management plugin and key file.
|
2016-09-05 16:08:21 140463950751616 [ERROR] InnoDB: Block in space_id 0 in file ./ibdata1 encrypted.
|
2016-09-05 16:08:21 140463950751616 [ERROR] InnoDB: However key management plugin or used key_id 5 is not found or used encryption algorithm or method does not match.
|
2016-09-05 16:08:21 140463950751616 [ERROR] InnoDB: Marking tablespace as missing. You may drop this table or install correct key management plugin and key file.
|
160905 16:08:21 [ERROR] mysqld got signal 11 ;
|
|
Stack trace from 10.1 747893a85451 |
Thread 1 (Thread 0x7fc04fe36780 (LWP 22646)):
|
#0 0x00007fc04e74af8c in pthread_kill () from /lib/x86_64-linux-gnu/libpthread.so.0
|
#1 0x00007fc0503e6005 in handle_fatal_signal (sig=11) at /home/elenst/mdev10499/10.1.16/sql/signal_handler.cc:294
|
#2 <signal handler called>
|
#3 ib_push_warning (trx=0x0, error=error@entry=67, format=format@entry=0x7fc05094b030 "Table in tablespace %lu encrypted.However key management plugin or used key_id %u is not found or used encryption algorithm or method does not match. Can't continue opening the table.") at /home/elenst/mdev10499/10.1.16/storage/xtradb/handler/ha_innodb.cc:22138
|
#4 0x00007fc05072bfc6 in buf_page_io_complete (bpage=bpage@entry=0x7fbf73af95c0) at /home/elenst/mdev10499/10.1.16/storage/xtradb/buf/buf0buf.cc:4817
|
#5 0x00007fc0507411d8 in buf_read_page_low (err=err@entry=0x7ffffd8e790c, sync=sync@entry=true, mode=mode@entry=132, space=space@entry=0, zip_size=zip_size@entry=0, unzip=unzip@entry=0, tablespace_version=1, offset=offset@entry=320, trx=trx@entry=0x0, rbpage=rbpage@entry=0x7ffffd8e79c8) at /home/elenst/mdev10499/10.1.16/storage/xtradb/buf/buf0rea.cc:262
|
#6 0x00007fc050741f20 in buf_read_page (space=space@entry=0, zip_size=zip_size@entry=0, offset=offset@entry=320, trx=trx@entry=0x0, bpage=bpage@entry=0x7ffffd8e79c8) at /home/elenst/mdev10499/10.1.16/storage/xtradb/buf/buf0rea.cc:474
|
#7 0x00007fc0507236bf in buf_page_get_gen (space=<optimised out>, zip_size=<optimised out>, offset=offset@entry=320, rw_latch=rw_latch@entry=2, guess=<optimised out>, guess@entry=0x0, mode=mode@entry=10, file=file@entry=0x7fc050944f28 "/home/elenst/mdev10499/10.1.16/storage/xtradb/include/trx0undo.ic", line=line@entry=171, mtr=mtr@entry=0x7ffffd8e7ba0, err=err@entry=0x0) at /home/elenst/mdev10499/10.1.16/storage/xtradb/buf/buf0buf.cc:2962
|
#8 0x00007fc0506e7750 in trx_undo_page_get (mtr=0x7ffffd8e7ba0, page_no=320, zip_size=<optimised out>, space=<optimised out>) at /home/elenst/mdev10499/10.1.16/storage/xtradb/include/trx0undo.ic:170
|
#9 trx_undo_mem_create_at_db_start (mtr=0x7ffffd8e7ba0, page_no=320, id=0, rseg=0x7fbf1de12268) at /home/elenst/mdev10499/10.1.16/storage/xtradb/trx/trx0undo.cc:1292
|
#10 trx_undo_lists_init (rseg=rseg@entry=0x7fbf1de12268) at /home/elenst/mdev10499/10.1.16/storage/xtradb/trx/trx0undo.cc:1424
|
#11 0x00007fc0506db057 in trx_rseg_mem_create (id=id@entry=82, space=<optimised out>, zip_size=<optimised out>, page_no=page_no@entry=255, ib_bh=ib_bh@entry=0x7fc04d05d400, mtr=mtr@entry=0x7ffffd8e81b0) at /home/elenst/mdev10499/10.1.16/storage/xtradb/trx/trx0rseg.cc:209
|
#12 0x00007fc0506dba3c in trx_rseg_create_instance (mtr=0x7ffffd8e81b0, ib_bh=0x7fc04d05d400, sys_header=0x7ffffd8e81b0 "") at /home/elenst/mdev10499/10.1.16/storage/xtradb/trx/trx0rseg.cc:287
|
#13 trx_rseg_array_init (sys_header=sys_header@entry=0x7fc03c3dc026 "", ib_bh=ib_bh@entry=0x7fc04d05d400, mtr=mtr@entry=0x7ffffd8e81b0) at /home/elenst/mdev10499/10.1.16/storage/xtradb/trx/trx0rseg.cc:358
|
#14 0x00007fc0506dd33e in trx_sys_init_at_db_start () at /home/elenst/mdev10499/10.1.16/storage/xtradb/trx/trx0sys.cc:660
|
#15 0x00007fc0506c4ca5 in innobase_start_or_create_for_mysql () at /home/elenst/mdev10499/10.1.16/storage/xtradb/srv/srv0start.cc:2540
|
#16 0x00007fc0505fe211 in innobase_init (p=<optimised out>) at /home/elenst/mdev10499/10.1.16/storage/xtradb/handler/ha_innodb.cc:4394
|
#17 0x00007fc0503e8471 in ha_initialize_handlerton (plugin=0x7fc04d108790) at /home/elenst/mdev10499/10.1.16/sql/handler.cc:513
|
#18 0x00007fc050290d48 in plugin_initialize (tmp_root=tmp_root@entry=0x7ffffd8ec650, plugin=plugin@entry=0x7fc04d108790, argc=argc@entry=0x7fc051091fb8 <remaining_argc>, argv=argv@entry=0x7fc04d02ba58, options_only=options_only@entry=false) at /home/elenst/mdev10499/10.1.16/sql/sql_plugin.cc:1408
|
#19 0x00007fc050291e8a in plugin_init (argc=argc@entry=0x7fc051091fb8 <remaining_argc>, argv=0x7fc04d02ba58, flags=2) at /home/elenst/mdev10499/10.1.16/sql/sql_plugin.cc:1686
|
#20 0x00007fc0501fbea2 in init_server_components () at /home/elenst/mdev10499/10.1.16/sql/mysqld.cc:5190
|
#21 0x00007fc050200d0f in mysqld_main (argc=50, argv=0x7fc04d02ba58) at /home/elenst/mdev10499/10.1.16/sql/mysqld.cc:5768
|
#22 0x00007fc04dda076d in __libc_start_main () from /lib/x86_64-linux-gnu/libc.so.6
|
#23 0x00007fc0501f4ef9 in _start ()
|
My scenario was simple, but time-consuming, so it will take some time to retry.
- i started the server with some non-default options (no encryption, no encryption plugin) on a clean datadir
- created one table
- populated it with some 120M rows
- shut down the server with normal shutdown
- tried to restart the server with the same configuration
=> crash (persistent)
I just got the same error after upgrading on ubuntu from 10.1.17+maria-1~trusty to 10.1.18+maria-1~trusty.
Restart gives the
2016-10-19 21:43:18 139834710702848 [ERROR] InnoDB: Block in space_id 0 in file ./ibdata1 encrypted.
|
2016-10-19 21:43:18 139834710702848 [ERROR] InnoDB: However key management plugin or used key_id 5808 is not found or used encryption algorithm or method does not match.
|
2016-10-19 21:43:18 139834710702848 [ERROR] InnoDB: Marking tablespace as missing. You may drop this table or install correct key management plugin and key file.
|
Anything I can provide to track down the issue and maybe solve this?
Is the tablespace now really corrupted and I need to rebuild the database server or any solution?
commit bc323727de312b32e80ae9590e2346414746a594
Author: Jan Lindström <jan.lindstrom@mariadb.com>
Date: Thu Oct 27 08:18:14 2016 +0300
MDEV-10977: [ERROR] InnoDB: Block in space_id 0 in file ibdata1 encrypted.
MDEV-10394: Innodb system table space corrupted
Analysis: After we have read the page in buf_page_io_complete try to
find if the page is encrypted or corrupted. Encryption was determined
by reading FIL_PAGE_FILE_FLUSH_LSN_OR_KEY_VERSION field from FIL-header
as a key_version. However, this field is not always zero even when
encryption is not used. Thus, incorrect key_version could lead situation where
decryption is tried to page that is not encrypted.
Fix: We still read key_version information from FIL_PAGE_FILE_FLUSH_LSN_OR_KEY_VERSION
field but also check if tablespace has encryption information before trying
encrypt the page.
Yes, We also getting the InnoDB assertion failure error. In that case, server started sucessfuly while having innodb table space corruption. but the server crashed with followed error log as I mentioned.
...........................................................................
2016-07-18 9:44:14 139882584566528 [ERROR] InnoDB: Block in space_id 0 in file /data001/masvc01/ibdata1 encrypted.
2016-07-18 9:44:14 139882584566528 [ERROR] InnoDB: However key management plugin or used key_id 286 is not found or used encryption algorithm or method does not match.
2016-07-18 9:44:14 139882584566528 [ERROR] InnoDB: Marking tablespace as missing. You may drop this table or install correct key management plugin and key file.
2016-07-18 09:44:14 7f38f3c36b00 InnoDB: Assertion failure in thread 139882584566528 in file buf0buf.cc line 4803
.......................................................