Edward Hicks added a comment - 2017-03-28 14:29 - edited Ours is: $ uname -a Linux dbserver 2.6.32-642.13.1.el6.x86_64 #1 SMP Wed Nov 23 16:03:01 EST 2016 x86_64 x86_64 x86_64 GNU/Linux $ cat /etc/redhat-release Red Hat Enterprise Linux Server release 6.8 (Santiago)

Edward Hicks added a comment - 2017-03-29 16:01 - edited We had another event last night which was triggered by the disk running out of free space. The crash was therefore not a surprise; however, in the process another ibd file was corrupted. You can add 10.0.30 to the list of affected versions. Unfortunately I cannot attach it this time as it holds potentially sensitive info; however, this time the top of the overwritten file only contained the string "pure virtual method called". 2017-03-29 08:14:05 7f05ac913700 InnoDB: Error: Write to file ./database/table.ibd failed at offset 131072. InnoDB: 32768 bytes should have been written, only 0 were written. InnoDB: Operating system error number 28. InnoDB: Check that your OS and file system support files of this size. InnoDB: Check also that the disk is not full or a disk quota exceeded. InnoDB: Error number 28 means 'No space left on device'. InnoDB: Some operating system error numbers are described at InnoDB: http://dev.mysql.com/doc/refman/5.6/en/operating-system-error-codes.html FAIL170329 8:14:05 [ERROR] mysqld: The table 'table' is full FAIL170329 8:14:05 [ERROR] mysqld: The table 'table' is full FAILFAILFAILFAILFAILFAILFAILFAILFAILFAILFAILFAILFAILFAILFAILFAILFAILFAILFAILFAILFAILFAILFAILFAILFAILFAILFAILFAILFAILFAILFAILFAILFAILFAILFAILFAILFAILFAILFAILFAILFAILFAILFAIL170329 8:14:07 [ERROR] mysqld: The table 'table' is full FAIL170329 8:14:07 [ERROR] mysqld: The table 'table' is full FAILFAILFAILFAILFAILFAILFAILFAILFAILFAILFAILFAILFAILFAILFAILFAILFAILFAILFAILFAILFAILFAILFAILFAILFAILFAILFAILFAILFAILFAILFAILFAILFAILFAILFAILFAILFAILFAILFAILFAILFAILFAILFAILFAILFAILFAILFAILFAILFAILFAILFAILFAILFAILFAILFAILFAILFAILFAILFAILFAILFAILFAILFAILFAILFAILFAILFAILFAILFAILFAILFAILFAILFAILFAILFAILFAILFAILFAILFAILFAILFAILFAILFAILFAILFAILFAILFAILFAILFAILFAILFAILFAILFAILFAILFAILFAILFAILFAILFAILFAILFAILFAILFAILFAILFAILFAILFAILFAILFAILFAILFAILFAILFAILFAILFAILFAILFAILFAILFAILFAILInnoDB: Fatal error (Out of disk space) in rollback. InnoDB: Out of tablespace. InnoDB: Consider increasing your tablespace. FAILFAILFAILFAIL170329 08:14:12 mysqld_safe Number of processes running now: 0 170329 08:14:12 mysqld_safe mysqld restarted 170329 8:14:12 [Note] /usr/sbin/mysqld (mysqld 10.0.30-MariaDB) starting as process 9407 ... 170329 8:14:13 [Note] InnoDB: Using mutexes to ref count buffer pool pages 170329 8:14:13 [Note] InnoDB: The InnoDB memory heap is disabled 170329 8:14:13 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins 170329 8:14:13 [Note] InnoDB: GCC builtin __atomic_thread_fence() is used for memory barrier 170329 8:14:13 [Note] InnoDB: Compressed tables use zlib 1.2.3 170329 8:14:13 [Note] InnoDB: Using Linux native AIO 170329 8:14:13 [Note] InnoDB: Using CPU crc32 instructions 170329 8:14:13 [Note] InnoDB: Initializing buffer pool, size = 7.0G 170329 8:14:13 [Note] InnoDB: Completed initialization of buffer pool 170329 8:14:13 [Note] InnoDB: Highest supported file format is Barracuda. 170329 8:14:13 [Note] InnoDB: Log scan progressed past the checkpoint lsn 7020651191165 170329 8:14:13 [Note] InnoDB: Database was not shutdown normally! 170329 8:14:13 [Note] InnoDB: Starting crash recovery. 170329 8:14:13 [Note] InnoDB: Reading tablespace information from the .ibd files... 170329 8:16:28 [ERROR] InnoDB: Space id in fsp header 876228913,but in the page header 540555825 170329 8:16:28 [ERROR] InnoDB: checksum mismatch in tablespace ./meddb/anothertable.ibd (table meddb/anothertable) 170329 8:16:28 [Note] InnoDB: Page size:1024 Pages to analyze:64 170329 8:16:28 [Note] InnoDB: Page size: 1024, Possible space_id count:0 170329 8:16:28 [Note] InnoDB: Page size:2048 Pages to analyze:64 170329 8:16:28 [Note] InnoDB: Page size: 2048, Possible space_id count:0 170329 8:16:28 [Note] InnoDB: Page size:4096 Pages to analyze:56 170329 8:16:28 [Note] InnoDB: Page size: 4096, Possible space_id count:0 170329 8:16:28 [Note] InnoDB: Page size:8192 Pages to analyze:28 170329 8:16:28 [Note] InnoDB: Page size: 8192, Possible space_id count:0 170329 8:16:28 [Note] InnoDB: Page size:16384 Pages to analyze:14 170329 8:16:28 [Note] InnoDB: VALID: space:1894080 page_no:1 page_size:16384 170329 8:16:28 [Note] InnoDB: VALID: space:1894080 page_no:2 page_size:16384 170329 8:16:28 [Note] InnoDB: VALID: space:1894080 page_no:3 page_size:16384 170329 8:16:28 [Note] InnoDB: VALID: space:1894080 page_no:4 page_size:16384 170329 8:16:28 [Note] InnoDB: VALID: space:1894080 page_no:5 page_size:16384 170329 8:16:28 [Note] InnoDB: VALID: space:1894080 page_no:6 page_size:16384 170329 8:16:28 [Note] InnoDB: VALID: space:1894080 page_no:7 page_size:16384 170329 8:16:28 [Note] InnoDB: VALID: space:1894080 page_no:8 page_size:16384 170329 8:16:28 [Note] InnoDB: VALID: space:1894080 page_no:9 page_size:16384 170329 8:16:28 [Note] InnoDB: VALID: space:1894080 page_no:10 page_size:16384 170329 8:16:28 [Note] InnoDB: VALID: space:1894080 page_no:11 page_size:16384 170329 8:16:28 [Note] InnoDB: VALID: space:1894080 page_no:12 page_size:16384 170329 8:16:28 [Note] InnoDB: Page size: 16384, Possible space_id count:1 170329 8:16:28 [Note] InnoDB: space_id:1894080, Number of pages matched: 12/12 (16384) 170329 8:16:28 [Note] InnoDB: Chosen space:1894080   170329 8:16:28 [Note] InnoDB: Restoring page 0 of tablespace 1894080 170329 8:16:28 [Warning] InnoDB: Doublewrite does not have page_no=0 of space: 1894080 2017-03-29 08:16:28 7f1ca3eec820 InnoDB: Operating system error number 2 in a file operation. InnoDB: The error means the system cannot find the path specified. InnoDB: If you are installing InnoDB, remember that you must create InnoDB: directories yourself, InnoDB does not create them. InnoDB: Error: could not open single-table tablespace file ./meddb/anothertable.ibd InnoDB: We do not continue the crash recovery, because the table may become InnoDB: corrupt if we cannot apply the log records in the InnoDB log to it. InnoDB: To fix the problem and start mysqld: InnoDB: 1) If there is a permission problem in the file and mysqld cannot InnoDB: open the file, you should modify the permissions. InnoDB: 2) If the table is not needed, or you can restore it from a backup, InnoDB: then you can remove the .ibd file, and InnoDB will do a normal InnoDB: crash recovery and ignore that table. InnoDB: 3) If the file system or the disk is broken, and you cannot remove InnoDB: the .ibd file, you can set innodb_force_recovery > 0 in my.cnf InnoDB: and force InnoDB to continue crash recovery here. 170329 08:16:28 mysqld_safe mysqld from pid file /var/lib/mysqldata/mysql/DATABASES/whsqlprd05.pid ended

Jean Weisbuch added a comment - 2017-07-06 16:25 Any news on this, is it still not possible to upgrade to 10.0.31?

Sergey Vojtovich added a comment - 2017-07-14 11:24 mysqld_safe starts mysqld with with "1" and "2" file descriptors closed. This was introduced in https://github.com/MariaDB/server/commit/7497ebf8a49bfe30bb4110f2ac20a30f804b7946 This means that descriptors "1" and "2" are free for use for any file. But then signal handler writes it's messages to a hardcoded file descriptor STDERR_FILENO (which is "2"). In both reports InnoDB data file header was overrider exactly with signal handler messages. I'm reassigning to serg and raising priority to critical, since it causes data loss.

Sergey Vojtovich added a comment - 2017-07-14 11:29 Please also note that early server error messages (before freopen(stderr)) may also go to undesired location. Same for setups where log_error is not defined.