[MDEV-10394] Innodb system table space corrupted Created: 2016-07-19  Updated: 2020-12-08  Resolved: 2016-10-29

Status: Closed
Project: MariaDB Server
Component/s: Storage Engine - InnoDB, Storage Engine - XtraDB
Affects Version/s: 10.1.13
Fix Version/s: 10.1.19

Type: Bug Priority: Major
Reporter: KIM KYUNG NAM Assignee: Jan Lindström (Inactive)
Resolution: Fixed Votes: 1
Labels: None
Environment:

SB Version: :base-4.0-amd64:base-4.0-noarch:core-4.0-amd64:core-4.0-noarch:graphics-4.0-amd64:graphics-4.0-noarch:printing-4.0-amd64:printing-4.0-noarch
Distributor ID: CentOS
Description: CentOS release 6.7 (Final)
Release: 6.7
Codename: Final

Attachments: Text File bugtest_schema.sql     File my.cnf    
Issue Links:
PartOf
includes MDEV-11182 InnoDB: Assertion failure in file buf... Closed
Relates
relates to MDEV-10977 [ERROR] InnoDB: Block in space_id 0 i... Closed
Sprint: 10.1.19

 Description   

We're use mariadb 10.1.13 as a main database of our system.
We migrated data from develpment server to product server a week ago.

Below is our error case :

MariaDB shutdown with signal 6 when some query executes. (insert or update...)

The error log is below.

...
 
2016-07-18  9:44:14 139882584566528 [ERROR] InnoDB: Block in space_id 0 in file /data001/masvc01/ibdata1 encrypted.
 
2016-07-18  9:44:14 139882584566528 [ERROR] InnoDB: However key management plugin or used key_id 286 is not found or used encryption algorithm or method does not match.
 
2016-07-18  9:44:14 139882584566528 [ERROR] InnoDB: Marking tablespace as missing. You may drop this table or install correct key management plugin and key file.
 
2016-07-18  9:44:14 139882584566528 [ERROR] InnoDB: Block in space_id 0 in file /data001/masvc01/ibdata1 encrypted.
 
2016-07-18  9:44:14 139882584566528 [ERROR] InnoDB: However key management plugin or used key_id 286 is not found or used encryption algorithm or method does not match.
 
2016-07-18  9:44:14 139882584566528 [ERROR] InnoDB: Marking tablespace as missing. You may drop this table or install correct key management plugin and key file.
 
2016-07-18 09:44:14 7f38f3c36b00  InnoDB: Assertion failure in thread 139882584566528 in file buf0buf.cc line 4803
 
...

We check out system innodb table space with innochecksum.
The result is below.

 /engn001/masvc01/mysql/bin/innochecksum -v ibdata1
 
 
 
 InnoDB offline file checksum utility.
 
 
 Variables (--variable-name=value)
 
 and boolean options {FALSE|TRUE}  Value (after reading options)
 
 --------------------------------- ----------------------------------------
 
 verbose                           TRUE
 
 debug                             FALSE
 
 skip-corrupt                      FALSE
 
 count                             FALSE
 
 start-page                        0
 
 end-page                          0
 
 page                              0
 
 per-page-details                  FALSE
 
 leaf                              FALSE
 
 merge                             0
 
 Table is uncompressed
 
 Page size is 16384
 
 file ibdata1_20160718 = 3221225472 bytes (196608 pages)...
 
 InnoChecksum; checking pages in range 0 to 196607
 
 Fail; page 0 invalid (fails innodb and crc32 checksum)

We hava ibdata1, ibdata2, ibdata3 and ibdata4 and results of innochecksum are same.

We compiled mariadb with debug option and test under same enviroment.
Below is the mysql.trace when db crash.

==========================================================================================================================
 
T@10   : | | | | | | | | | | | | | | | | | >print_buffer_to_file
 
T@10   : | | | | | | | | | | | | | | | | | | enter: buffer: InnoDB: Marking tablespace as missing. You may drop this table or install correct key management plugin and key file.
 
T@10   : | | | | | | | | | | | | | | | | | | mutex: LOCK_error_log (0x15b4b60) locking
 
T@10   : | | | | | | | | | | | | | | | | | | mutex: LOCK_error_log (0x15b4b60) locked
 
T@10   : | | | | | | | | | | | | | | | | | | mutex: LOCK_error_log (0x15b4b60) unlocking
 
T@10   : | | | | | | | | | | | | | | | | | <print_buffer_to_file
 
T@10   : | | | | | | | | | | | | | | | | <vprint_msg_to_log
 
T@10   : | | | | | | | | | | | | | | | <sql_print_error
 
T@10   : | | | | | | | | | | | | | | | >my_malloc
 
T@10   : | | | | | | | | | | | | | | | | my: size: 4096  my_flags: 16
 
T@10   : | | | | | | | | | | | | | | | | exit: ptr: 0x7f9e84159690
 
T@10   : | | | | | | | | | | | | | | | <my_malloc
 
T@10   : | | | | | | | | | | | | | | | >push_warning_printf
 
T@10   : | | | | | | | | | | | | | | | | enter: warning: 192
 
T@10   : | | | | | | | | | | | | | | | | >push_warning
 
T@10   : | | | | | | | | | | | | | | | | | enter: code: 192, msg: Table in tablespace 0 encrypted.However key management plugin or used key_id 290 is not found or used encryption algorithm or method does not match. Can't continue opening the table.
 
T@10   : | | | | | | | | | | | | | | | | | >THD::raise_condition
 
T@10   : | | | | | | | | | | | | | | | | | | >mysql_audit_acquire_plugins
 
T@10   : | | | | | | | | | | | | | | | | | | <mysql_audit_acquire_plugins
 
T@10   : | | | | | | | | | | | | | | | | | | mutex: l_perm->lock (0x5dcaeb00) locking
 
T@10   : | | | | | | | | | | | | | | | | | | >my_multi_malloc
 
T@10   : | | | | | | | | | | | | | | | | | | | >my_malloc
 
T@10   : | | | | | | | | | | | | | | | | | | | | my: size: 208  my_flags: 24
 
T@10   : | | | | | | | | | | | | | | | | | | | | exit: ptr: 0x7f9e8415a710
 
T@10   : | | | | | | | | | | | | | | | | | | | <my_malloc
 
T@10   : | | | | | | | | | | | | | | | | | | <my_multi_malloc
 
T@10   : | | | | | | | | | | | | | | | | | | >my_hash_init
 
T@10   : | | | | | | | | | | | | | | | | | | | enter: hash: 0x7f9e8415a710  size: 128
 
T@10   : | | | | | | | | | | | | | | | | | | | >init_dynamic_array2
 
T@10   : | | | | | | | | | | | | | | | | | | | | >my_malloc

So we export data from mariadb and import to new table space.
After migration, we shutdown mariadb and restart. but signal 11 occurred and db crash.
Below is our error log.

2016-07-19  8:55:09 140556344748800 [Note] /engn001/masvc01/mysql/bin/mysqld: Normal shutdown
 
 
 
2016-07-19  8:55:09 140556344748800 [Note] Event Scheduler: Killing the scheduler thread, thread id 1
 
2016-07-19  8:55:09 140556344748800 [Note] Event Scheduler: Waiting for the scheduler thread to reply
 
2016-07-19  8:55:09 140556344748800 [Note] Event Scheduler: Stopped
 
2016-07-19  8:55:09 140556344748800 [Note] Event Scheduler: Purging the queue. 0 events
 
160719  8:55:09 server_audit: STOPPED
 
2016-07-19  8:55:09 140556627830528 [Note] InnoDB: FTS optimize thread exiting.
 
2016-07-19  8:55:09 140556344748800 [Note] InnoDB: Starting shutdown...
 
2016-07-19  8:55:09 140556648810240 [Warning] InnoDB: Dumping buffer pool(s) to /data001/masvc01/ib_buffer_pool
 
2016-07-19  8:55:10 140556648810240 [Warning] InnoDB: Buffer pool(s) dump completed at 160719  8:55:10
 
2016-07-19  8:55:15 140556344748800 [Note] InnoDB: Shutdown completed; log sequence number 399145649446
 
2016-07-19  8:55:15 140556344748800 [Note] /engn001/masvc01/mysql/bin/mysqld: Shutdown complete
 
 
 
160719 08:55:16 mysqld_safe mysqld from pid file /engn001/masvc01/mysql/mysqld.pid ended
 
160719 08:57:05 mysqld_safe Starting mysqld daemon with databases from /data001/masvc01
 
2016-07-19  8:57:05 140304014350176 [Note] /engn001/masvc01/mysql/bin/mysqld (mysqld 10.1.14-MariaDB) starting as process 2608 ...
 
2016-07-19  8:57:06 140304014350176 [Note] InnoDB: Using mutexes to ref count buffer pool pages
 
2016-07-19  8:57:06 140304014350176 [Note] InnoDB: The InnoDB memory heap is disabled
 
2016-07-19  8:57:06 140304014350176 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
 
2016-07-19  8:57:06 140304014350176 [Note] InnoDB: Memory barrier is not used
 
2016-07-19  8:57:06 140304014350176 [Note] InnoDB: Compressed tables use zlib 1.2.3
 
2016-07-19  8:57:06 140304014350176 [Note] InnoDB: Using Linux native AIO
 
2016-07-19  8:57:06 140304014350176 [Note] InnoDB: Using SSE crc32 instructions
 
2016-07-19  8:57:06 140304014350176 [Note] InnoDB: Initializing buffer pool, size = 24.0G
 
2016-07-19  8:57:08 140304014350176 [Note] InnoDB: Completed initialization of buffer pool
 
2016-07-19  8:57:10 140304014350176 [Note] InnoDB: Highest supported file format is Barracuda.
 
2016-07-19  8:57:13 140304014350176 [Note] InnoDB: 128 rollback segment(s) are active.
 
2016-07-19  8:57:13 140304014350176 [Note] InnoDB: Waiting for purge to start
 
2016-07-19  8:57:13 140304014350176 [Note] InnoDB:  Percona XtraDB (http://www.percona.com) 5.6.29-76.2 started; log sequence number 399145649446
 
2016-07-19  8:57:13 140273638147840 [Note] InnoDB: Dumping buffer pool(s) not yet started
 
2016-07-19 08:57:13 7f94005f8700 InnoDB: Loading buffer pool(s) from /data001/masvc01/ib_buffer_pool
 
2016-07-19  8:57:13 140304014350176 [Note] Plugin 'FEEDBACK' is disabled.
 
160719  8:57:13 server_audit: MariaDB Audit Plugin version 1.4.0 STARTED.
 
2016-07-19  8:57:13 140304014350176 [Note] Server socket created on IP: '::'.
 
2016-07-19  8:57:13 140304014350176 [Warning] 'user' entry 'root@lgedglap02v' ignored in --skip-name-resolve mode.
 
2016-07-19  8:57:13 140304014350176 [Warning] 'user' entry '@lgedglap02v' ignored in --skip-name-resolve mode.
 
2016-07-19  8:57:13 140304014350176 [Warning] 'proxies_priv' entry '@% root@lgedglap02v' ignored in --skip-name-resolve mode.
 
2016-07-19  8:57:13 140273595853568 [Note] Event Scheduler: scheduler thread started with id 1
 
2016-07-19  8:57:13 140304014350176 [Note] Reading of all Master_info entries succeded
 
2016-07-19  8:57:13 140304014350176 [Note] Added new Master_info '' to hash table
 
2016-07-19  8:57:13 140304014350176 [Note] /engn001/masvc01/mysql/bin/mysqld: ready for connections.
 
Version: '10.1.14-MariaDB'  socket: '/engn001/masvc01/mysql/mysqld.sock'  port: 3310  MariaDB Server
 
2016-07-19  8:57:47 140273638147840 [ERROR] InnoDB: Block in space_id 0 in file /data001/masvc01/ibdata1 encrypted.
 
2016-07-19  8:57:47 140273638147840 [ERROR] InnoDB: However key management plugin or used key_id 92 is not found or used encryption algorithm or method does not match.
 
2016-07-19  8:57:47 140273638147840 [ERROR] InnoDB: Marking tablespace as missing. You may drop this table or install correct key management plugin and key file.
 
2016-07-19  8:57:47 140273638147840 [ERROR] InnoDB: Block in space_id 0 in file /data001/masvc01/ibdata1 encrypted.
 
2016-07-19  8:57:47 140273638147840 [ERROR] InnoDB: However key management plugin or used key_id 92 is not found or used encryption algorithm or method does not match.
 
2016-07-19  8:57:47 140273638147840 [ERROR] InnoDB: Marking tablespace as missing. You may drop this table or install correct key management plugin and key file.
 
160719  8:57:47 [ERROR] mysqld got signal 11 ;
 
This could be because you hit a bug. It is also possible that this binary
 
or one of the libraries it was linked against is corrupt, improperly built,
 
or misconfigured. This error can also be caused by malfunctioning hardware.
 
 
 
To report this bug, see https://mariadb.com/kb/en/reporting-bugs
 
 
 
We will try our best to scrape up some info that will hopefully help
 
diagnose the problem, but since we have already crashed, 
something is definitely wrong and this may fail.
 
 
 
Server version: 10.1.14-MariaDB
 
key_buffer_size=67108864
 
read_buffer_size=50331648
 
max_used_connections=3
 
max_threads=102
 
thread_count=4
 
It is possible that mysqld could use up to 
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 10094612 K  bytes of memory
 
Hope that's ok; if not, decrease some variables in the equation.
 
 
 
Thread pointer: 0x0x0
 
Attempting backtrace. You can use the following information to find out
 
where mysqld died. If you see no messages after this, something went
 
terribly wrong...
 
stack_bottom = 0x0 thread_stack 0x48400
 
/engn001/masvc01/mysql/bin/mysqld(my_print_stacktrace+0x2e)[0xc028ae]
 
/engn001/masvc01/mysql/bin/mysqld(handle_fatal_signal+0x464)[0x7651f4]
 
/lib64/libpthread.so.0[0x3ffe40f7e0]
 
/engn001/masvc01/mysql/bin/mysqld[0x8fa6d3]
 
/engn001/masvc01/mysql/bin/mysqld[0xa3ba47]
 
/engn001/masvc01/mysql/bin/mysqld[0xa51a2e]
 
/engn001/masvc01/mysql/bin/mysqld[0xa52bc9]
 
/engn001/masvc01/mysql/bin/mysqld[0xa433ff]
 
/engn001/masvc01/mysql/bin/mysqld[0xa440b7]
 
/lib64/libpthread.so.0[0x3ffe407aa1]
 
/lib64/libc.so.6(clone+0x6d)[0x3ffe0e893d]
 
The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
 
information that should help you find out what is causing the crash.
 
.....

I can't find solution for it. I attached my.cnf.
Is it a bug or hardware fault ?



 Comments   
Comment by Elena Stepanova [ 2016-07-19 ]

windfree,

First, could you please clarify the versions? In 'Affects version/s' you put 10.1.13, but your description starts with you using 10.1.3; and the error log is from 10.1.14.
Is 10.1.3 a typo, or did you actually migrate from 10.1.3 to 10.1.13/14?

Further, when you say

So we export data from mariadb and import to new table space.
After migration, we shutdown mariadb and restart.

what do you mean by that? Do you export/import one table only? Can you specify what exactly you did to import data to a new table space (how you made sure it's really new)?

The error log insists you have something encrypted, but your has no sign of it. Are you using, or have you ever used, any encryption on this server?

Thanks.

Comment by KIM KYUNG NAM [ 2016-07-19 ]

>> First, could you please clarify the versions? In 'Affects version/s' you put 10.1.13, but your description starts with you using 10.1.3; and the error log is >> from 10.1.14.
>> Is 10.1.3 a typo, or did you actually migrate from 10.1.3 to 10.1.13/14?

I mistyped. correct version is 10.1.13 and We tested with 10.1.13 and 10.1.14.
So above error log is mariadb 10.1.14's log.

>> Further, when you say
>> So we export data from mariadb and import to new table space.
>> After migration, we shutdown mariadb and restart.

>> what do you mean by that? Do you export/import one table only? Can you specify what exactly you did to import data to a new table space (how you made sure it's really new)?

We reinstalled mariadb and export our database from test server and imported to new mariadb.

>>The error log insists you have something encrypted, but your has no sign of it. Are you using, or have you ever used, any encryption on this server?

No, I attaced my,cnf file. We don't use any encryption option.

Finally, We downgraded mariadb version from 10.1.13 to 10.0.22 and DB doesn't crash any more under same test.
I assume that it's bug of mariadb 10.1.xx .

Thanks.

Comment by Elena Stepanova [ 2016-07-19 ]

Yes, it must be a 10.1-specific problem, since 10.0 does not have encryption, which is clearly involved here – somehow InnoDB (mis)interprets the data as being encrypted.

When you just re-install MariaDB, the datadir is preserved, so unless you removed it (or InnoDB files from it) manually, it's possible that even after re-installation the server was still using the old (corrupted?) system table space.

Do you have that data stored, by any chance? I mean the InnoDB files itself, not the data dump.

Comment by KIM KYUNG NAM [ 2016-07-19 ]

>>When you just re-install MariaDB, the datadir is preserved, so unless you removed it (or InnoDB files from it) manually, it's >>possible that even after re-installation the server was still using the old (corrupted?) system table space.

>>Do you have that data stored, by any chance? I mean the InnoDB files itself, not the data dump.

When I reinstalled MariaDB, I changed datadir to another position.
Also I installed MariaDB another server and got same result under test.

Our test is very simple.
1. Install MariaDB 10.1.13 or 10.1.14
2. Start MariaDB
3. Shutdown MariaDB
4. Check innodb tablespace by innochecksum. the result is ok.
5. Start MariaDB
6. Import database from our backup file which is created by mysqldump. ( about 15G)
7. Shutdown MariaDB. After shutdown, there is no error message. it's normal shutdown.
8. Check innodb tablespace by innochecksum. the result is below.
--> Fail; page 0 invalid (fails innodb and crc32 checksum)

So we changed MariaDB 10.1.13 to 10.0.22 and do same test.
There is no invalid page in innodb tablespace with 10.0.22

Thanks

Comment by Elena Stepanova [ 2016-07-20 ]

windfree, thanks for clarification.

Would you be able to provide this data dump? You could upload it to our ftp.askmonty.org/private, this way only MariaDB developers will have access to it.
15 GB is a lot, but we would really want to find the reason of the problem, and if it's reliably reproducible with your dump, it might be our real chance.
Besides, if it's 15 GB uncompressed, it should be much less in an archive, dumps usually compress very well.

Comment by KIM KYUNG NAM [ 2016-07-20 ]

Thanks for response.
Ok, I'll check whether it is possible or not.

Comment by KIM KYUNG NAM [ 2016-07-23 ]

I can't real data because of our client's security policy.
But I made a test data and do same test and found a innodb system table space corrupted.
I uploaded our test files to ftp.askmonty.org/private. ( mdev-10394-bugtest-schema.sql, mdev-10394-data.zip ) .
Below is my test scenario.

1. create dababase .
2. create table using mdev-10394-bugtest-schema.sql.
3. bin/mysql --defaults-file=./my.cnf < data.sql ( bugtest table's row count is 7388877)
4. shutdown database.
5. check innodb table space. ( bin/innochecksum data/ibdata1 .
6 . At this time, innodb tablespace has no problem.
7. start database.
8. execute step3 again --> bin/mysql --defaults-file=./my.cnf < data.sql ( bugtest table's row count is twice of 14777754 )
9 .shutdown databse.
10. check innodb table space. ( bin/innochecksum data/ibdata1 .
11. Now, innodb tablespace corrupted. ( message - > Fail; page 0 invalid (fails innodb and crc32 checksum))

I wanna know the reason of this bug.

Comment by Elena Stepanova [ 2016-07-29 ]

Mr. Kim,

With the test scenario described above, are you also getting the InnoDB complaints and assertion failure which you had in the description?
I've been trying to reproduce the issue, and with your test I'm getting exactly what you said in the previous comment, innochecksum complaining about the tablespace being corrputed; but further attempt to start the server again and to use the table does not cause any visible problem, no InnoDB errors in the log, no assertion, and InnoDB is able to continue working with the table. So, while innochecksum complaint definitely needs to be investigated, I think maybe there is something more to it when it comes to the original problem.

Comment by KIM KYUNG NAM [ 2016-08-01 ]

Yes, We also getting the InnoDB assertion failure error. In that case, server started sucessfuly while having innodb table space corruption. but the server crashed with followed error log as I mentioned.
...........................................................................
2016-07-18 9:44:14 139882584566528 [ERROR] InnoDB: Block in space_id 0 in file /data001/masvc01/ibdata1 encrypted.
2016-07-18 9:44:14 139882584566528 [ERROR] InnoDB: However key management plugin or used key_id 286 is not found or used encryption algorithm or method does not match.
2016-07-18 9:44:14 139882584566528 [ERROR] InnoDB: Marking tablespace as missing. You may drop this table or install correct key management plugin and key file.
2016-07-18 09:44:14 7f38f3c36b00 InnoDB: Assertion failure in thread 139882584566528 in file buf0buf.cc line 4803
.......................................................

Comment by Jan Lindström (Inactive) [ 2016-08-01 ]

Hi,

Can you try with 10.1.16, there was some bad bugs on 10.1.13.

Comment by Elena Stepanova [ 2016-09-05 ]

I got the same error on current 10.1. The difference is that instead of an assertion failure I'm getting SIGSEGV.

2016-09-05 16:08:20 140463950751616 [ERROR] InnoDB: Space id in fsp header 131344,but in the page header 0
 
2016-09-05 16:08:20 140463950751616 [Note] InnoDB: Highest supported file format is Barracuda.
 
2016-09-05 16:08:21 140463950751616 [ERROR] InnoDB: Block in space_id 0 in file ./ibdata1 encrypted.
 
2016-09-05 16:08:21 140463950751616 [ERROR] InnoDB: However key management plugin or used key_id 5 is not found or used encryption algorithm or method does not match.
 
2016-09-05 16:08:21 140463950751616 [ERROR] InnoDB: Marking tablespace as missing. You may drop this table or install correct key management plugin and key file.
 
2016-09-05 16:08:21 140463950751616 [ERROR] InnoDB: Block in space_id 0 in file ./ibdata1 encrypted.
 
2016-09-05 16:08:21 140463950751616 [ERROR] InnoDB: However key management plugin or used key_id 5 is not found or used encryption algorithm or method does not match.
 
2016-09-05 16:08:21 140463950751616 [ERROR] InnoDB: Marking tablespace as missing. You may drop this table or install correct key management plugin and key file.
 
160905 16:08:21 [ERROR] mysqld got signal 11 ;

Stack trace from 10.1 747893a85451

 
Thread 1 (Thread 0x7fc04fe36780 (LWP 22646)):
 
#0  0x00007fc04e74af8c in pthread_kill () from /lib/x86_64-linux-gnu/libpthread.so.0
 
#1  0x00007fc0503e6005 in handle_fatal_signal (sig=11) at /home/elenst/mdev10499/10.1.16/sql/signal_handler.cc:294
 
#2  <signal handler called>
 
#3  ib_push_warning (trx=0x0, error=error@entry=67, format=format@entry=0x7fc05094b030 "Table in tablespace %lu encrypted.However key management plugin or used key_id %u is not found or used encryption algorithm or method does not match. Can't continue opening the table.") at /home/elenst/mdev10499/10.1.16/storage/xtradb/handler/ha_innodb.cc:22138
 
#4  0x00007fc05072bfc6 in buf_page_io_complete (bpage=bpage@entry=0x7fbf73af95c0) at /home/elenst/mdev10499/10.1.16/storage/xtradb/buf/buf0buf.cc:4817
 
#5  0x00007fc0507411d8 in buf_read_page_low (err=err@entry=0x7ffffd8e790c, sync=sync@entry=true, mode=mode@entry=132, space=space@entry=0, zip_size=zip_size@entry=0, unzip=unzip@entry=0, tablespace_version=1, offset=offset@entry=320, trx=trx@entry=0x0, rbpage=rbpage@entry=0x7ffffd8e79c8) at /home/elenst/mdev10499/10.1.16/storage/xtradb/buf/buf0rea.cc:262
 
#6  0x00007fc050741f20 in buf_read_page (space=space@entry=0, zip_size=zip_size@entry=0, offset=offset@entry=320, trx=trx@entry=0x0, bpage=bpage@entry=0x7ffffd8e79c8) at /home/elenst/mdev10499/10.1.16/storage/xtradb/buf/buf0rea.cc:474
 
#7  0x00007fc0507236bf in buf_page_get_gen (space=<optimised out>, zip_size=<optimised out>, offset=offset@entry=320, rw_latch=rw_latch@entry=2, guess=<optimised out>, guess@entry=0x0, mode=mode@entry=10, file=file@entry=0x7fc050944f28 "/home/elenst/mdev10499/10.1.16/storage/xtradb/include/trx0undo.ic", line=line@entry=171, mtr=mtr@entry=0x7ffffd8e7ba0, err=err@entry=0x0) at /home/elenst/mdev10499/10.1.16/storage/xtradb/buf/buf0buf.cc:2962
 
#8  0x00007fc0506e7750 in trx_undo_page_get (mtr=0x7ffffd8e7ba0, page_no=320, zip_size=<optimised out>, space=<optimised out>) at /home/elenst/mdev10499/10.1.16/storage/xtradb/include/trx0undo.ic:170
 
#9  trx_undo_mem_create_at_db_start (mtr=0x7ffffd8e7ba0, page_no=320, id=0, rseg=0x7fbf1de12268) at /home/elenst/mdev10499/10.1.16/storage/xtradb/trx/trx0undo.cc:1292
 
#10 trx_undo_lists_init (rseg=rseg@entry=0x7fbf1de12268) at /home/elenst/mdev10499/10.1.16/storage/xtradb/trx/trx0undo.cc:1424
 
#11 0x00007fc0506db057 in trx_rseg_mem_create (id=id@entry=82, space=<optimised out>, zip_size=<optimised out>, page_no=page_no@entry=255, ib_bh=ib_bh@entry=0x7fc04d05d400, mtr=mtr@entry=0x7ffffd8e81b0) at /home/elenst/mdev10499/10.1.16/storage/xtradb/trx/trx0rseg.cc:209
 
#12 0x00007fc0506dba3c in trx_rseg_create_instance (mtr=0x7ffffd8e81b0, ib_bh=0x7fc04d05d400, sys_header=0x7ffffd8e81b0 "") at /home/elenst/mdev10499/10.1.16/storage/xtradb/trx/trx0rseg.cc:287
 
#13 trx_rseg_array_init (sys_header=sys_header@entry=0x7fc03c3dc026 "", ib_bh=ib_bh@entry=0x7fc04d05d400, mtr=mtr@entry=0x7ffffd8e81b0) at /home/elenst/mdev10499/10.1.16/storage/xtradb/trx/trx0rseg.cc:358
 
#14 0x00007fc0506dd33e in trx_sys_init_at_db_start () at /home/elenst/mdev10499/10.1.16/storage/xtradb/trx/trx0sys.cc:660
 
#15 0x00007fc0506c4ca5 in innobase_start_or_create_for_mysql () at /home/elenst/mdev10499/10.1.16/storage/xtradb/srv/srv0start.cc:2540
 
#16 0x00007fc0505fe211 in innobase_init (p=<optimised out>) at /home/elenst/mdev10499/10.1.16/storage/xtradb/handler/ha_innodb.cc:4394
 
#17 0x00007fc0503e8471 in ha_initialize_handlerton (plugin=0x7fc04d108790) at /home/elenst/mdev10499/10.1.16/sql/handler.cc:513
 
#18 0x00007fc050290d48 in plugin_initialize (tmp_root=tmp_root@entry=0x7ffffd8ec650, plugin=plugin@entry=0x7fc04d108790, argc=argc@entry=0x7fc051091fb8 <remaining_argc>, argv=argv@entry=0x7fc04d02ba58, options_only=options_only@entry=false) at /home/elenst/mdev10499/10.1.16/sql/sql_plugin.cc:1408
 
#19 0x00007fc050291e8a in plugin_init (argc=argc@entry=0x7fc051091fb8 <remaining_argc>, argv=0x7fc04d02ba58, flags=2) at /home/elenst/mdev10499/10.1.16/sql/sql_plugin.cc:1686
 
#20 0x00007fc0501fbea2 in init_server_components () at /home/elenst/mdev10499/10.1.16/sql/mysqld.cc:5190
 
#21 0x00007fc050200d0f in mysqld_main (argc=50, argv=0x7fc04d02ba58) at /home/elenst/mdev10499/10.1.16/sql/mysqld.cc:5768
 
#22 0x00007fc04dda076d in __libc_start_main () from /lib/x86_64-linux-gnu/libc.so.6
 
#23 0x00007fc0501f4ef9 in _start ()

My scenario was simple, but time-consuming, so it will take some time to retry.

  • i started the server with some non-default options (no encryption, no encryption plugin) on a clean datadir
  • created one table
  • populated it with some 120M rows
  • shut down the server with normal shutdown
  • tried to restart the server with the same configuration
    => crash (persistent)
Comment by Thomas Seifert [ 2016-10-19 ]

I just got the same error after upgrading on ubuntu from 10.1.17+maria-1~trusty to 10.1.18+maria-1~trusty.
Restart gives the 

2016-10-19 21:43:18 139834710702848 [ERROR] InnoDB: Block in space_id 0 in file ./ibdata1 encrypted.
 
2016-10-19 21:43:18 139834710702848 [ERROR] InnoDB: However key management plugin or used key_id 5808 is not found or used encryption algorithm or method does not match.
 
2016-10-19 21:43:18 139834710702848 [ERROR] InnoDB: Marking tablespace as missing. You may drop this table or install correct key management plugin and key file.

Anything I can provide to track down the issue and maybe solve this?
Is the tablespace now really corrupted and I need to rebuild the database server or any solution?

Comment by Jan Lindström (Inactive) [ 2016-10-29 ]

commit bc323727de312b32e80ae9590e2346414746a594
Author: Jan Lindström <jan.lindstrom@mariadb.com>
Date: Thu Oct 27 08:18:14 2016 +0300

MDEV-10977: [ERROR] InnoDB: Block in space_id 0 in file ibdata1 encrypted.
MDEV-10394: Innodb system table space corrupted

Analysis: After we have read the page in buf_page_io_complete try to
find if the page is encrypted or corrupted. Encryption was determined
by reading FIL_PAGE_FILE_FLUSH_LSN_OR_KEY_VERSION field from FIL-header
as a key_version. However, this field is not always zero even when
encryption is not used. Thus, incorrect key_version could lead situation where
decryption is tried to page that is not encrypted.

Fix: We still read key_version information from FIL_PAGE_FILE_FLUSH_LSN_OR_KEY_VERSION
field but also check if tablespace has encryption information before trying
encrypt the page.

Generated at Thu Feb 08 07:41:54 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.