[MDEV-13939] InnoDB: Page directory corruption: infimum not pointed to Created: 2017-09-28  Updated: 2017-10-26  Resolved: 2017-10-26

Status: Closed
Project: MariaDB Server
Component/s: Storage Engine - InnoDB, Storage Engine - XtraDB
Affects Version/s: 5.5.54
Fix Version/s: N/A

Type: Bug Priority: Major
Reporter: Max Assignee: Marko Mäkelä
Resolution: Cannot Reproduce Votes: 0
Labels: None
Environment:

synology DSM 6.1.3.15152 Update 5



 Description   

Trying to start MariaDB fails. The hostname.err file contains the following:

170928 10:32:34 mysqld_safe mysqld from pid file /run/mysqld/mysqld.pid ended
170928 10:37:38 mysqld_safe Starting mysqld daemon with databases from /var/packages/MariaDB/target/mysql
170928 10:37:39 [Note] /usr/bin/mysqld (mysqld 5.5.54-MariaDB) starting as process 20074 ...
170928 10:37:39 InnoDB: The InnoDB memory heap is disabled
170928 10:37:39 InnoDB: Mutexes and rw_locks use InnoDB's own implementation
170928 10:37:39 InnoDB: Compressed tables use zlib 1.2.8
170928 10:37:39 InnoDB: Using Linux native AIO
170928 10:37:39 InnoDB: Initializing buffer pool, size = 16.0M
170928 10:37:39 InnoDB: Completed initialization of buffer pool
170928 10:37:39 InnoDB: highest supported file format is Barracuda.
InnoDB: The log sequence number in ibdata files does not match
InnoDB: the log sequence number in the ib_logfiles!
170928 10:37:39  InnoDB: Database was not shut down normally!
InnoDB: Starting crash recovery.
InnoDB: Reading tablespace information from the .ibd files...
InnoDB: Restoring possible half-written data pages from the doublewrite
InnoDB: buffer...
170928 10:37:44  InnoDB: Waiting for the background threads to start
170928 10:37:45 Percona XtraDB (http://www.percona.com) 5.5.52-MariaDB-38.3 started; log sequence number 1598363888
170928 10:37:45 [Note] Plugin 'FEEDBACK' is disabled.
170928 10:37:45 [Note] Server socket created on IP: '0.0.0.0'.
170928 10:37:47 [Note] Event Scheduler: Loaded 0 events
170928 10:37:47 [Note] /usr/bin/mysqld: ready for connections.
Version: '5.5.54-MariaDB'  socket: '/run/mysqld/mysqld.sock'  port: 3306  Source distribution
InnoDB: Page directory corruption: infimum not pointed to
170928 10:37:47  InnoDB: Page dump in ascii and hex (16384 bytes):
 len 16384; hex 000000000 [REOMVED MANY ZEROS]; asc                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                 ;
InnoDB: End of page dump
170928 10:37:47  InnoDB: Page checksum 1575996416 (32bit_calc: 2253597184), prior-to-4.0.14-form checksum 1371122432
InnoDB: stored checksum 0, prior-to-4.0.14-form stored checksum 0
InnoDB: Page lsn 0 0, low 4 bytes of lsn at page end 0
InnoDB: Page number (if stored to page already) 0,
InnoDB: space id (if created with >= MySQL-4.1.1 and stored already) 0
InnoDB: Page may be a freshly allocated page
InnoDB: Page directory corruption: supremum not pointed to
170928 10:37:47  InnoDB: Page dump in ascii and hex (16384 bytes):
 len 16384; hex 000000000; asc                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                 ;
InnoDB: End of page dump
170928 10:37:47  InnoDB: Page checksum 1575996416 (32bit_calc: 2253597184), prior-to-4.0.14-form checksum 1371122432
InnoDB: stored checksum 0, prior-to-4.0.14-form stored checksum 0
InnoDB: Page lsn 0 0, low 4 bytes of lsn at page end 0
InnoDB: Page number (if stored to page already) 0,
InnoDB: space id (if created with >= MySQL-4.1.1 and stored already) 0
InnoDB: Page may be a freshly allocated page
170928 10:37:47 [ERROR] mysqld got signal 11 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
 
To report this bug, see http://kb.askmonty.org/en/reporting-bugs
 
We will try our best to scrape up some info that will hopefully help
diagnose the problem, but since we have already crashed, 
something is definitely wrong and this may fail.
 
Server version: 5.5.54-MariaDB
key_buffer_size=16384
read_buffer_size=262144
max_used_connections=1
max_threads=153
thread_count=0
It is possible that mysqld could use up to 
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 354234 K  bytes of memory
Hope that's ok; if not, decrease some variables in the equation.
 
Thread pointer: 0x0
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 0x0 thread_stack 0x3c000
The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
information that should help you find out what is causing the crash.
170928 10:37:47 mysqld_safe Number of processes running now: 0
170928 10:37:47 mysqld_safe mysqld restarted
170928 10:37:48 [Note] /usr/bin/mysqld (mysqld 5.5.54-MariaDB) starting as process 20164 ...
170928 10:37:48 InnoDB: The InnoDB memory heap is disabled
170928 10:37:48 InnoDB: Mutexes and rw_locks use InnoDB's own implementation
170928 10:37:48 InnoDB: Compressed tables use zlib 1.2.8
170928 10:37:48 InnoDB: Using Linux native AIO
170928 10:37:48 InnoDB: Initializing buffer pool, size = 16.0M
170928 10:37:48 InnoDB: Completed initialization of buffer pool
170928 10:37:48 InnoDB: highest supported file format is Barracuda.
InnoDB: The log sequence number in ibdata files does not match
InnoDB: the log sequence number in the ib_logfiles!
170928 10:37:48  InnoDB: Database was not shut down normally!
InnoDB: Starting crash recovery.
InnoDB: Reading tablespace information from the .ibd files...
InnoDB: Restoring possible half-written data pages from the doublewrite
InnoDB: buffer...
170928 10:37:48  InnoDB: Waiting for the background threads to start
InnoDB: Page directory corruption: infimum not pointed to
170928 10:37:48  InnoDB: Page dump in ascii and hex (16384 bytes):
 len 16384; hex 000000000; asc                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                 ;
InnoDB: End of page dump
170928 10:37:48  InnoDB: Page checksum 1575996416 (32bit_calc: 2253597184), prior-to-4.0.14-form checksum 1371122432
InnoDB: stored checksum 0, prior-to-4.0.14-form stored checksum 0
InnoDB: Page lsn 0 0, low 4 bytes of lsn at page end 0
InnoDB: Page number (if stored to page already) 0,
InnoDB: space id (if created with >= MySQL-4.1.1 and stored already) 0
InnoDB: Page may be a freshly allocated page
InnoDB: Page directory corruption: supremum not pointed to
170928 10:37:48  InnoDB: Page dump in ascii and hex (16384 bytes):
 len 16384; hex 000000000; asc                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                 ;
InnoDB: End of page dump
170928 10:37:48  InnoDB: Page checksum 1575996416 (32bit_calc: 2253597184), prior-to-4.0.14-form checksum 1371122432
InnoDB: stored checksum 0, prior-to-4.0.14-form stored checksum 0
InnoDB: Page lsn 0 0, low 4 bytes of lsn at page end 0
InnoDB: Page number (if stored to page already) 0,
InnoDB: space id (if created with >= MySQL-4.1.1 and stored already) 0
InnoDB: Page may be a freshly allocated page
170928 10:37:48 [ERROR] mysqld got signal 11 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
 
To report this bug, see http://kb.askmonty.org/en/reporting-bugs
 
We will try our best to scrape up some info that will hopefully help
diagnose the problem, but since we have already crashed, 
something is definitely wrong and this may fail.
 
Server version: 5.5.54-MariaDB
key_buffer_size=16384
read_buffer_size=262144
max_used_connections=0
max_threads=153
thread_count=0
It is possible that mysqld could use up to 
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 354234 K  bytes of memory
Hope that's ok; if not, decrease some variables in the equation.
 
Thread pointer: 0x0
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 0x0 thread_stack 0x3c000
The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
information that should help you find out what is causing the crash.
170928 10:37:48 mysqld_safe mysqld from pid file /run/mysqld/mysqld.pid ended

Tried updates. Out if ideas. Any help is much appreciated!
Max



 Comments   
Comment by Elena Stepanova [ 2017-09-30 ]

marko, do you know if there is a way to recover from this?

Comment by Marko Mäkelä [ 2017-10-02 ]

I changed the bug title to the relevant error message:

InnoDB: Page directory corruption: infimum not pointed to

According to the log messages, the server did start up, but probably crashed on the first access to a table. At least one index of the table is corrupted. Apparently zero-byte-filled page is supposed to be a B-tree index page.

If the access to the corrupted table was not initiated from SQL, then it should originate from InnoDB background tasks, such as rolling back recovered incomplete transactions, or purging the history. You can try to start up with --innodb-force-recovery=3 to prevent those tasks from running, and then extract the data with mysqldump or similar. You may have to skip certain tables or ranges of primary keys in case they lead to a crash due to the corrupted page. With luck, the corruption is only in a secondary index page.

In any case, it is safest to dump and restore the data.

Comment by Max [ 2017-10-02 ]

Managed to start MariaDB with these settings in /etc/mysql/my.cnf

[mysqld]
innodb_purge_threads=0
innodb_force_recovery = 3

Then restarting the service (on a synology system):

/usr/syno/bin/synopkg restart MariaDB
package MariaDB restart successfully

Then dumping the database:

mysqldump -p owncloud > own.sql
Enter password: 

Dropping it:

mysql -p
Enter password: 
Welcome to the MariaDB monitor.  Commands end with ; or \g.
Your MariaDB connection id is 6
Server version: 5.5.54-MariaDB Source distribution
 
Copyright (c) 2000, 2016, Oracle, MariaDB Corporation Ab and others.
 
Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.
 
MariaDB [(none)]> DROP DATABASE owncloud;
Query OK, 53 rows affected (0.61 sec)
 
MariaDB [(none)]> Ctrl-C -- exit!
Aborted

Then put innodb settings back to default in /etc/mysql/my.cnf:

[mysqld]
#innodb_purge_threads=0
#innodb_force_recovery = 3

restarting the MariaDB service:

/usr/syno/bin/synopkg restart MariaDB
package MariaDB restart successfully

Create the empty database again:

mysql -p 
Enter password: 
Welcome to the MariaDB monitor.  Commands end with ; or \g.
Your MariaDB connection id is 5
Server version: 5.5.54-MariaDB Source distribution
 
Copyright (c) 2000, 2016, Oracle, MariaDB Corporation Ab and others.
 
Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.
 
MariaDB [(none)]> CREATE DATABASE owncloud;
Query OK, 1 row affected (0.02 sec)
 
MariaDB [(none)]> Ctrl-C -- exit!
Aborted

dump previous data back into it:

mysql -p owncloud < own.sql
Enter password: 

run a check:

mysqlcheck --user root --password -B owncloud
Enter password: 
owncloud.oc_activity                               OK
owncloud.oc_activity_mq                            OK
owncloud.oc_addressbookchanges                     OK
owncloud.oc_addressbooks                           OK
owncloud.oc_appconfig                              OK
owncloud.oc_audioplayer_album_artists              OK
owncloud.oc_audioplayer_albums                     OK
owncloud.oc_audioplayer_artists                    OK
owncloud.oc_audioplayer_genre                      OK
owncloud.oc_audioplayer_playlist_tracks            OK
owncloud.oc_audioplayer_playlists                  OK
owncloud.oc_audioplayer_tracks                     OK
owncloud.oc_authtoken                              OK
owncloud.oc_calendarchanges                        OK
owncloud.oc_calendarobjects                        OK
owncloud.oc_calendars                              OK
owncloud.oc_calendarsubscriptions                  OK
owncloud.oc_cards                                  OK
owncloud.oc_cards_properties                       OK
owncloud.oc_comments                               OK
owncloud.oc_comments_read_markers                  OK
owncloud.oc_credentials                            OK
owncloud.oc_dav_shares                             OK
owncloud.oc_documents_invite                       OK
owncloud.oc_documents_member                       OK
owncloud.oc_documents_op                           OK
owncloud.oc_documents_revisions                    OK
owncloud.oc_documents_session                      OK
owncloud.oc_federated_reshares                     OK
owncloud.oc_file_locks                             OK
owncloud.oc_filecache                              OK
owncloud.oc_files_trash                            OK
owncloud.oc_group_admin                            OK
owncloud.oc_group_user                             OK
owncloud.oc_groups                                 OK
owncloud.oc_jobs                                   OK
owncloud.oc_mimetypes                              OK
owncloud.oc_mounts                                 OK
owncloud.oc_notifications                          OK
owncloud.oc_preferences                            OK
owncloud.oc_privatedata                            OK
owncloud.oc_properties                             OK
owncloud.oc_schedulingobjects                      OK
owncloud.oc_share                                  OK
owncloud.oc_share_external                         OK
owncloud.oc_storages                               OK
owncloud.oc_systemtag                              OK
owncloud.oc_systemtag_group                        OK
owncloud.oc_systemtag_object_mapping               OK
owncloud.oc_trusted_servers                        OK
owncloud.oc_users                                  OK
owncloud.oc_vcategory                              OK
owncloud.oc_vcategory_to_object                    OK

Restart all necessary services (phpmyadmin in my case)

All set!
Thanks!

Comment by Marko Mäkelä [ 2017-10-26 ]

MaxC, I am glad that my advice was helpful.
I will close this as "Cannot Reproduce" instead of "Not a Bug", because I cannot be 100% sure that the corruption was not caused by code running in the MariaDB server.

Generated at Thu Feb 08 08:09:35 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.