Index page corruption with data-at-rest encryption

2018-03-08 15:37:22 140038675761920 [ERROR] Master 'dc1master': InnoDB:  Page 3759119 in space N/A (148162) maybe corrupted. Post encryption checksum 1117819446 stored [0:2893804962] key_version 1

InnoDB: Page directory corruption: infimum not pointed to

2018-03-08 15:37:22 7f5d4b860b00 InnoDB: Page dump in ascii and hex (16384 bytes):

 len 16384; hex 

...

InnoDB: End of page dump

2018-03-08 15:37:22 7f5d4b860b00 InnoDB: uncompressed page, stored checksum in field1 0, calculated checksums for field1: crc32 1117819446, innodb 996923242, none 3735928559, stored checksum in field2 2893804962, calculated checksums for field2: crc32 1117819446, innodb 2893804962, none 3735928559, page LSN 4285 488303025, low 4 bytes of LSN at page end 488303025, page number (if stored to page already) 3759119, space id (if created with >= MySQL-4.1.1 and stored already) 148162

InnoDB: page type 17855 meaning INDEX

InnoDB: Page may be an index page where index id is 7476710334411351822

InnoDB: Page directory corruption: supremum not pointed to

2018-03-08 15:37:22 7f5d4b860b00 InnoDB: Page dump in ascii and hex (16384 bytes):

 len 16384; hex 

...

InnoDB: End of page dump

2018-03-08 15:37:22 7f5d4b860b00 InnoDB: uncompressed page, stored checksum in field1 0, calculated checksums for field1: crc32 1117819446, innodb 996923242, none 3735928559, stored checksum in field2 2893804962, calculated checksums for field2: crc32 1117819446, innodb 2893804962, none 3735928559, page LSN 4285 488303025, low 4 bytes of LSN at page end 488303025, page number (if stored to page already) 3759119, space id (if created with >= MySQL-4.1.1 and stored already) 148162

InnoDB: page type 17855 meaning INDEX

InnoDB: Page may be an index page where index id is 7476710334411351822

2018-03-08 15:37:22 7f5d4b860b00  InnoDB: Assertion failure in thread 140038675761920 in file rem0rec.cc line 581

InnoDB: We intentionally generate a memory trap.

InnoDB: Submit a detailed bug report to https://jira.mariadb.org/

InnoDB: If you get repeated assertion failures or crashes, even

InnoDB: immediately after the mysqld startup, there may be

InnoDB: corruption in the InnoDB tablespace. Please refer to

InnoDB: http://dev.mysql.com/doc/refman/5.6/en/forcing-innodb-recovery.html

InnoDB: about forcing recovery.

180308 15:37:22 [ERROR] mysqld got signal 6 ;

This could be because you hit a bug. It is also possible that this binary

or one of the libraries it was linked against is corrupt, improperly built,

or misconfigured. This error can also be caused by malfunctioning hardware.

To report this bug, see https://mariadb.com/kb/en/reporting-bugs

We will try our best to scrape up some info that will hopefully help

diagnose the problem, but since we have already crashed, 

something is definitely wrong and this may fail.

Server version: 10.1.31-MariaDB-1~trusty

key_buffer_size=134217728

read_buffer_size=2097152

max_used_connections=5

max_threads=2502

thread_count=2

It is possible that mysqld could use up to 

key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 15555412 K  bytes of memory

Hope that's ok; if not, decrease some variables in the equation.

Thread pointer: 0x7f26e282b008

Attempting backtrace. You can use the following information to find out

where mysqld died. If you see no messages after this, something went

terribly wrong...

stack_bottom = 0x7f5d4b85f990 thread_stack 0x48400

/usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x56472af6122e]

/usr/sbin/mysqld(handle_fatal_signal+0x305)[0x56472aa833e5]

/lib/x86_64-linux-gnu/libpthread.so.0(+0x10330)[0x7f5d4c867330]

/lib/x86_64-linux-gnu/libc.so.6(gsignal+0x37)[0x7f5d4bcb6c37]

/lib/x86_64-linux-gnu/libc.so.6(abort+0x148)[0x7f5d4bcba028]

/usr/sbin/mysqld(+0x89f7bb)[0x56472ad607bb]

/usr/sbin/mysqld(+0x87fddc)[0x56472ad40ddc]

/usr/sbin/mysqld(+0x93d8f2)[0x56472adfe8f2]

/usr/sbin/mysqld(+0x8b316e)[0x56472ad7416e]

/usr/sbin/mysqld(+0x8b5bbb)[0x56472ad76bbb]

/usr/sbin/mysqld(+0x8b5f7b)[0x56472ad76f7b]

/usr/sbin/mysqld(+0x8c2597)[0x56472ad83597]

/usr/sbin/mysqld(+0x812bb3)[0x56472acd3bb3]

/usr/sbin/mysqld(_ZN7handler12ha_write_rowEPh+0x483)[0x56472aa8e153]

/usr/sbin/mysqld(_ZN14Rows_log_event9write_rowEP14rpl_group_infob+0x16d)[0x56472ab62cbd]

/usr/sbin/mysqld(_ZN20Write_rows_log_event11do_exec_rowEP14rpl_group_info+0x7d)[0x56472ab630cd]

/usr/sbin/mysqld(_ZN14Rows_log_event14do_apply_eventEP14rpl_group_info+0x31c)[0x56472ab55c0c]

/usr/sbin/mysqld(+0x3b528b)[0x56472a87628b]

/usr/sbin/mysqld(handle_slave_sql+0x27b9)[0x56472a8824d9]

/lib/x86_64-linux-gnu/libpthread.so.0(+0x8184)[0x7f5d4c85f184]

/lib/x86_64-linux-gnu/libc.so.6(clone+0x6d)[0x7f5d4bd7e03d]

Trying to get some variables.

Some pointers may be invalid and cause the dump to abort.

Query (0x0): is an invalid pointer

Connection ID (thread ID): 4

Status: NOT_KILLED

Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=off,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=off

The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains

information that should help you find out what is causing the crash.

We think the query pointer is invalid, but we will try to print it anyway. 

Query: 

MariaDB [(none)]> SELECT * FROM information_schema.INNODB_SYS_TABLESPACES WHERE SPACE=148162;

+--------+-----------------------------------+------+-------------+----------------------+-----------+---------------+

| SPACE | NAME | FLAG | FILE_FORMAT | ROW_FORMAT | PAGE_SIZE | ZIP_PAGE_SIZE |

+--------+-----------------------------------+------+-------------+----------------------+-----------+---------------+

| 148162 | db1/tab | 0 | Antelope | Compact or Redundant | 16384 | 0 |

+--------+-----------------------------------+------+-------------+----------------------+-----------+---------------+

2018-03-13 21:34:34 7f47dd5d5b00  InnoDB: Warning: CHECK TABLE on index `PRIMARY` of table `db1`.`tab` returned 12

2018-03-14 12:29:15 139946633288448 [ERROR] InnoDB:  Page 3759119 in space N/A (148162) maybe corrupted. Post encryption checksum 1117819446 stored [0:2893804962] key_version 1

InnoDB: Error in pages 20929391 and 3759119 of index "idx1" of table "db1"."tab"

InnoDB: 'compact' flag mismatch

2018-03-14 12:29:15 7f47dd5d5b00 InnoDB: Page dump in ascii and hex (16384 bytes):

 len 16384; hex 

...

InnoDB: End of page dump

2018-03-14 12:29:15 7f47dd5d5b00 InnoDB: uncompressed page, stored checksum in field1 1851218656, calculated checksums for field1: crc32 3816730793, innodb 1851218656, none 3735928559, stored checksum in field2 2568069479, calculated checksums for field2: crc32 3816730793, innodb 2568069479, none 3735928559, page LSN 4340 1363275092, low 4 bytes of LSN at page end 1363275092, page number (if stored to page already) 20929391, space id (if created with >= MySQL-4.1.1 and stored already) 148162

InnoDB: page type 17855 meaning INDEX

InnoDB: Page may be an index page where index id is 89357

InnoDB: (index "idx1" of table "db1"."tab")

2018-03-14 12:29:15 7f47dd5d5b00 InnoDB: Page dump in ascii and hex (16384 bytes):

 len 16384; hex 

...

InnoDB: End of page dump

2018-03-14 12:29:15 7f47dd5d5b00 InnoDB: uncompressed page, stored checksum in field1 0, calculated checksums for field1: crc32 1117819446, innodb 996923242, none 3735928559, stored checksum in field2 2893804962, calculated checksums for field2: crc32 1117819446, innodb 2893804962, none 3735928559, page LSN 4285 488303025, low 4 bytes of LSN at page end 488303025, page number (if stored to page already) 3759119, space id (if created with >= MySQL-4.1.1 and stored already) 148162

InnoDB: page type 17855 meaning INDEX

InnoDB: Page may be an index page where index id is 7476710334411351822

2018-03-14 12:29:15 139946633288448 [ERROR] InnoDB: Page index id 7476710334411351822 != data dictionary index id 89357

2018-03-14 12:29:15 7f47dd5d5b00  InnoDB: Assertion failure in thread 139946633288448 in file btr0btr.cc line 4904

InnoDB: Failing assertion: btr_page_get_level(page, &mtr) == level

InnoDB: We intentionally generate a memory trap.

InnoDB: Submit a detailed bug report to https://jira.mariadb.org/

InnoDB: If you get repeated assertion failures or crashes, even

InnoDB: immediately after the mysqld startup, there may be

InnoDB: corruption in the InnoDB tablespace. Please refer to

InnoDB: http://dev.mysql.com/doc/refman/5.6/en/forcing-innodb-recovery.html

InnoDB: about forcing recovery.

180314 12:29:15 [ERROR] mysqld got signal 6 ;

This could be because you hit a bug. It is also possible that this binary

or one of the libraries it was linked against is corrupt, improperly built,

or misconfigured. This error can also be caused by malfunctioning hardware.

To report this bug, see https://mariadb.com/kb/en/reporting-bugs

We will try our best to scrape up some info that will hopefully help

diagnose the problem, but since we have already crashed, 

something is definitely wrong and this may fail.

Server version: 10.1.31-MariaDB-1~trusty

key_buffer_size=67108864

read_buffer_size=2097152

max_used_connections=3

max_threads=502

thread_count=1

It is possible that mysqld could use up to 

key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 3160251 K  bytes of memory

Hope that's ok; if not, decrease some variables in the equation.

Thread pointer: 0x7f3e533f9008

Attempting backtrace. You can use the following information to find out

where mysqld died. If you see no messages after this, something went

terribly wrong...

stack_bottom = 0x7f47dd5d51f0 thread_stack 0x48400

/usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x55620797122e]

/usr/sbin/mysqld(handle_fatal_signal+0x305)[0x5562074933e5]

/lib/x86_64-linux-gnu/libpthread.so.0(+0x10330)[0x7f47dc24f330]

/lib/x86_64-linux-gnu/libc.so.6(gsignal+0x37)[0x7f47db69fc37]

/lib/x86_64-linux-gnu/libc.so.6(abort+0x148)[0x7f47db6a3028]

/usr/sbin/mysqld(+0x939e34)[0x55620780ae34]

/usr/sbin/mysqld(+0x811d2a)[0x5562076e2d2a]

/usr/sbin/mysqld(_ZN7handler8ha_checkEP3THDP15st_ha_check_opt+0x7a)[0x55620749ac6a]

/usr/sbin/mysqld(+0x51afc8)[0x5562073ebfc8]

/usr/sbin/mysqld(_ZN19Sql_cmd_check_table7executeEP3THD+0xb0)[0x5562073ecd60]

/usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x14ac)[0x5562073072ac]

/usr/sbin/mysqld(_Z11mysql_parseP3THDPcjP12Parser_state+0x321)[0x55620730f611]

/usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcj+0x2693)[0x556207312e03]

/usr/sbin/mysqld(_Z10do_commandP3THD+0x136)[0x556207313686]

/usr/sbin/mysqld(_Z24do_handle_one_connectionP3THD+0x18a)[0x5562073df75a]

/usr/sbin/mysqld(handle_one_connection+0x40)[0x5562073df900]

/lib/x86_64-linux-gnu/libpthread.so.0(+0x8184)[0x7f47dc247184]

/lib/x86_64-linux-gnu/libc.so.6(clone+0x6d)[0x7f47db766bed]

Trying to get some variables.

Some pointers may be invalid and cause the dump to abort.

Query (0x7f3a2f034020): is an invalid pointer

Connection ID (thread ID): 43

Status: NOT_KILLED

Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=off,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=off

The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains

information that should help you find out what is causing the crash.

We think the query pointer is invalid, but we will try to print it anyway. 

Query: CHECK TABLE db1.tab

INDEX_ID: 89357

NAME: idx1

TABLE_ID: 148176

TYPE: 2

N_FIELDS: 3

PAGE_NO: 4

SPACE: 148162

CREATE TABLE `tab` (

`col1` bigint(20) NOT NULL AUTO_INCREMENT,

`col2` varchar(64) COLLATE utf8_unicode_ci NOT NULL,

`col3` varchar(64) COLLATE utf8_unicode_ci NOT NULL,

`col4` varchar(64) COLLATE utf8_unicode_ci NOT NULL,

`col5` varchar(50) COLLATE utf8_unicode_ci DEFAULT NULL,

`col6` varchar(100) COLLATE utf8_unicode_ci DEFAULT NULL,

`col7` int(11) DEFAULT '0',

`col8` bit(1) DEFAULT b'0',

`col9` smallint(6) DEFAULT NULL,

`col10` datetime NOT NULL DEFAULT CURRENT_TIMESTAMP,

`col11` varchar(50) COLLATE utf8_unicode_ci DEFAULT NULL,

`col12` datetime NOT NULL DEFAULT CURRENT_TIMESTAMP,

`col13` int(11) DEFAULT NULL,

PRIMARY KEY (`col1`),

UNIQUE KEY `idx1` (`col2`,`col3`,`col4`),

UNIQUE KEY `idx2` (`col2`,`col6`) USING BTREE,

KEY `idx3` (`col2`,`col4`) USING BTREE,

KEY `idx4` (`col4`) USING BTREE,

KEY `idx5` (`col10`)

) ENGINE=InnoDB AUTO_INCREMENT=5032253394 DEFAULT CHARSET=utf8 COLLATE=utf8_unicode_ci DATA DIRECTORY='./'