Details
-
Bug
-
Status: Closed (View Workflow)
-
Critical
-
Resolution: Fixed
-
10.0.28, 10.0.29, 10.0.30
-
RHEL6, x86_64, VMWare, wheezy
Description
3 times now in the last 6 months or so we've had a random signal 6 crash corrupt a single ibd file by actually overwriting the top it with the error message. (which I assume was meant for the log file instead)
I've attached the corrupted file as an example. I've confirmed this occurs in 10.0.28 and 10.0.29. Here's the error data from the correct log file:
2017-02-27 01:25:07 7f50466df700 InnoDB: Assertion failure in thread 139982755723008 in file buf0lru.cc line 2245
|
InnoDB: We intentionally generate a memory trap.
|
InnoDB: Submit a detailed bug report to http://bugs.mysql.com.
|
InnoDB: If you get repeated assertion failures or crashes, even
|
InnoDB: immediately after the mysqld startup, there may be
|
InnoDB: corruption in the InnoDB tablespace. Please refer to
|
InnoDB: http://dev.mysql.com/doc/refman/5.6/en/forcing-innodb-recovery.html
|
InnoDB: about forcing recovery.
|
/usr/sbin/mysqld(my_print_stacktrace+0x2b)[0xb924ab]
|
/usr/sbin/mysqld(handle_fatal_signal+0x3b6)[0x739136]
|
/lib64/libpthread.so.0[0x323120f7e0]
|
/lib64/libc.so.6(gsignal+0x35)[0x3230e325e5]
|
/lib64/libc.so.6(abort+0x175)[0x3230e33dc5]
|
/usr/sbin/mysqld[0x9c86c9]
|
/usr/sbin/mysqld[0x8db7d5]
|
/usr/sbin/mysqld[0x9280e8]
|
/usr/sbin/mysqld[0x8835ff]
|
/usr/sbin/mysqld(_Z8closefrmP5TABLEb+0x3a)[0x67c43a]
|
/usr/sbin/mysqld(_Z18intern_close_tableP5TABLE+0x59)[0x59a609]
|
/usr/sbin/mysqld(_Z8tc_purgeb+0x285)[0x6e88f5]
|
/usr/sbin/mysqld(_Z19close_cached_tablesP3THDP10TABLE_LISTbm+0x8f)[0x5a317f]
|
/usr/sbin/mysqld(_Z20reload_acl_and_cacheP3THDyP10TABLE_LISTPi+0x3ce)[0x6c023e]
|
/usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x4889)[0x5e3899]
|
/usr/sbin/mysqld[0x5e5a97]
|
/usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcj+0x1c6c)[0x5e7e6c]
|
/usr/sbin/mysqld(_Z24do_handle_one_connectionP3THD+0x453)[0x6a6f13]
|
/usr/sbin/mysqld(handle_one_connection+0x42)[0x6a6fe2]
|
/lib64/libpthread.so.0[0x3231207aa1]
|
/lib64/libc.so.6(clone+0x6d)[0x3230ee8aad]
|
170227 01:25:09 mysqld_safe Number of processes running now: 0
|
170227 01:25:09 mysqld_safe mysqld restarted
|
170227 1:25:09 [Note] /usr/sbin/mysqld (mysqld 10.0.29-MariaDB) starting as process 23465 ...
|
170227 1:25:09 [Note] InnoDB: Using mutexes to ref count buffer pool pages
|
170227 1:25:09 [Note] InnoDB: The InnoDB memory heap is disabled
|
170227 1:25:09 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
|
170227 1:25:09 [Note] InnoDB: GCC builtin __atomic_thread_fence() is used for memory barrier
|
170227 1:25:09 [Note] InnoDB: Compressed tables use zlib 1.2.3
|
170227 1:25:09 [Note] InnoDB: Using Linux native AIO
|
170227 1:25:10 [Note] InnoDB: Using CPU crc32 instructions
|
170227 1:25:10 [Note] InnoDB: Initializing buffer pool, size = 7.0G
|
170227 1:25:10 [Note] InnoDB: Completed initialization of buffer pool
|
170227 1:25:10 [Note] InnoDB: Highest supported file format is Barracuda.
|
170227 1:25:10 [Note] InnoDB: Log scan progressed past the checkpoint lsn 6877178025086
|
170227 1:25:10 [Note] InnoDB: Database was not shutdown normally!
|
170227 1:25:10 [Note] InnoDB: Starting crash recovery.
|
170227 1:25:10 [Note] InnoDB: Reading tablespace information from the .ibd files...
|
170227 1:28:01 [ERROR] InnoDB: Space id in fsp header 1851878432,but in the page header 544434535
|
170227 1:28:01 [ERROR] InnoDB: innodb-page-size mismatch in tablespace ./optometrydb/page_manager_handlers.ibd (table optometrydb/page_manager_handlers)
|
170227 1:28:01 [Note] InnoDB: Page size:1024 Pages to analyze:64
|
170227 1:28:01 [Note] InnoDB: Page size: 1024, Possible space_id count:0
|
170227 1:28:01 [Note] InnoDB: Page size:2048 Pages to analyze:64
|
170227 1:28:01 [Note] InnoDB: Page size: 2048, Possible space_id count:0
|
170227 1:28:01 [Note] InnoDB: Page size:4096 Pages to analyze:32
|
170227 1:28:01 [Note] InnoDB: Page size: 4096, Possible space_id count:0
|
170227 1:28:01 [Note] InnoDB: Page size:8192 Pages to analyze:16
|
170227 1:28:01 [Note] InnoDB: Page size: 8192, Possible space_id count:0
|
170227 1:28:01 [Note] InnoDB: Page size:16384 Pages to analyze:8
|
170227 1:28:01 [Note] InnoDB: VALID: space:1859997 page_no:1 page_size:16384
|
170227 1:28:01 [Note] InnoDB: VALID: space:1859997 page_no:2 page_size:16384
|
170227 1:28:01 [Note] InnoDB: VALID: space:1859997 page_no:3 page_size:16384
|
170227 1:28:01 [Note] InnoDB: VALID: space:1859997 page_no:4 page_size:16384
|
170227 1:28:01 [Note] InnoDB: VALID: space:1859997 page_no:5 page_size:16384
|
170227 1:28:01 [Note] InnoDB: Page size: 16384, Possible space_id count:1
|
170227 1:28:01 [Note] InnoDB: space_id:1859997, Number of pages matched: 5/5 (16384)
|
170227 1:28:01 [Note] InnoDB: Chosen space:1859997
|
 |
170227 1:28:01 [Note] InnoDB: Restoring page 0 of tablespace 1859997
|
170227 1:28:01 [Warning] InnoDB: Doublewrite does not have page_no=0 of space: 1859997
|
2017-02-27 01:28:01 7f7346996820 InnoDB: Operating system error number 2 in a file operation.
|
InnoDB: The error means the system cannot find the path specified.
|
InnoDB: If you are installing InnoDB, remember that you must create
|
InnoDB: directories yourself, InnoDB does not create them.
|
InnoDB: Error: could not open single-table tablespace file ./optometrydb/page_manager_handlers.ibd
|
InnoDB: We do not continue the crash recovery, because the table may become
|
InnoDB: corrupt if we cannot apply the log records in the InnoDB log to it.
|
InnoDB: To fix the problem and start mysqld:
|
InnoDB: 1) If there is a permission problem in the file and mysqld cannot
|
InnoDB: open the file, you should modify the permissions.
|
InnoDB: 2) If the table is not needed, or you can restore it from a backup,
|
InnoDB: then you can remove the .ibd file, and InnoDB will do a normal
|
InnoDB: crash recovery and ignore that table.
|
InnoDB: 3) If the file system or the disk is broken, and you cannot remove
|
InnoDB: the .ibd file, you can set innodb_force_recovery > 0 in my.cnf
|
InnoDB: and force InnoDB to continue crash recovery here.
|
170227 01:28:01 mysqld_safe mysqld from pid file /var/lib/mysqldata/mysql/DATABASES/whsqlprd05.pid ended
|