Uploaded image for project: 'MariaDB Server'
  1. MariaDB Server
  2. MDEV-12144

Signal 6 crash corrupts ibd files

    XMLWordPrintable

Details

    Description

      3 times now in the last 6 months or so we've had a random signal 6 crash corrupt a single ibd file by actually overwriting the top it with the error message. (which I assume was meant for the log file instead)

      I've attached the corrupted file as an example. I've confirmed this occurs in 10.0.28 and 10.0.29. Here's the error data from the correct log file:

      2017-02-27 01:25:07 7f50466df700  InnoDB: Assertion failure in thread 139982755723008 in file buf0lru.cc line 2245
      InnoDB: We intentionally generate a memory trap.
      InnoDB: Submit a detailed bug report to http://bugs.mysql.com.
      InnoDB: If you get repeated assertion failures or crashes, even
      InnoDB: immediately after the mysqld startup, there may be
      InnoDB: corruption in the InnoDB tablespace. Please refer to
      InnoDB: http://dev.mysql.com/doc/refman/5.6/en/forcing-innodb-recovery.html
      InnoDB: about forcing recovery.
      /usr/sbin/mysqld(my_print_stacktrace+0x2b)[0xb924ab]
      /usr/sbin/mysqld(handle_fatal_signal+0x3b6)[0x739136]
      /lib64/libpthread.so.0[0x323120f7e0]
      /lib64/libc.so.6(gsignal+0x35)[0x3230e325e5]
      /lib64/libc.so.6(abort+0x175)[0x3230e33dc5]
      /usr/sbin/mysqld[0x9c86c9]
      /usr/sbin/mysqld[0x8db7d5]
      /usr/sbin/mysqld[0x9280e8]
      /usr/sbin/mysqld[0x8835ff]
      /usr/sbin/mysqld(_Z8closefrmP5TABLEb+0x3a)[0x67c43a]
      /usr/sbin/mysqld(_Z18intern_close_tableP5TABLE+0x59)[0x59a609]
      /usr/sbin/mysqld(_Z8tc_purgeb+0x285)[0x6e88f5]
      /usr/sbin/mysqld(_Z19close_cached_tablesP3THDP10TABLE_LISTbm+0x8f)[0x5a317f]
      /usr/sbin/mysqld(_Z20reload_acl_and_cacheP3THDyP10TABLE_LISTPi+0x3ce)[0x6c023e]
      /usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x4889)[0x5e3899]
      /usr/sbin/mysqld[0x5e5a97]
      /usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcj+0x1c6c)[0x5e7e6c]
      /usr/sbin/mysqld(_Z24do_handle_one_connectionP3THD+0x453)[0x6a6f13]
      /usr/sbin/mysqld(handle_one_connection+0x42)[0x6a6fe2]
      /lib64/libpthread.so.0[0x3231207aa1]
      /lib64/libc.so.6(clone+0x6d)[0x3230ee8aad]
      170227 01:25:09 mysqld_safe Number of processes running now: 0
      170227 01:25:09 mysqld_safe mysqld restarted
      170227  1:25:09 [Note] /usr/sbin/mysqld (mysqld 10.0.29-MariaDB) starting as process 23465 ...
      170227  1:25:09 [Note] InnoDB: Using mutexes to ref count buffer pool pages
      170227  1:25:09 [Note] InnoDB: The InnoDB memory heap is disabled
      170227  1:25:09 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
      170227  1:25:09 [Note] InnoDB: GCC builtin __atomic_thread_fence() is used for memory barrier
      170227  1:25:09 [Note] InnoDB: Compressed tables use zlib 1.2.3
      170227  1:25:09 [Note] InnoDB: Using Linux native AIO
      170227  1:25:10 [Note] InnoDB: Using CPU crc32 instructions
      170227  1:25:10 [Note] InnoDB: Initializing buffer pool, size = 7.0G
      170227  1:25:10 [Note] InnoDB: Completed initialization of buffer pool
      170227  1:25:10 [Note] InnoDB: Highest supported file format is Barracuda.
      170227  1:25:10 [Note] InnoDB: Log scan progressed past the checkpoint lsn 6877178025086
      170227  1:25:10 [Note] InnoDB: Database was not shutdown normally!
      170227  1:25:10 [Note] InnoDB: Starting crash recovery.
      170227  1:25:10 [Note] InnoDB: Reading tablespace information from the .ibd files...
      170227  1:28:01 [ERROR] InnoDB: Space id in fsp header 1851878432,but in the page header 544434535
      170227  1:28:01 [ERROR] InnoDB: innodb-page-size mismatch in tablespace ./optometrydb/page_manager_handlers.ibd (table optometrydb/page_manager_handlers)
      170227  1:28:01 [Note] InnoDB: Page size:1024 Pages to analyze:64
      170227  1:28:01 [Note] InnoDB: Page size: 1024, Possible space_id count:0
      170227  1:28:01 [Note] InnoDB: Page size:2048 Pages to analyze:64
      170227  1:28:01 [Note] InnoDB: Page size: 2048, Possible space_id count:0
      170227  1:28:01 [Note] InnoDB: Page size:4096 Pages to analyze:32
      170227  1:28:01 [Note] InnoDB: Page size: 4096, Possible space_id count:0
      170227  1:28:01 [Note] InnoDB: Page size:8192 Pages to analyze:16
      170227  1:28:01 [Note] InnoDB: Page size: 8192, Possible space_id count:0
      170227  1:28:01 [Note] InnoDB: Page size:16384 Pages to analyze:8
      170227  1:28:01 [Note] InnoDB: VALID: space:1859997 page_no:1 page_size:16384
      170227  1:28:01 [Note] InnoDB: VALID: space:1859997 page_no:2 page_size:16384
      170227  1:28:01 [Note] InnoDB: VALID: space:1859997 page_no:3 page_size:16384
      170227  1:28:01 [Note] InnoDB: VALID: space:1859997 page_no:4 page_size:16384
      170227  1:28:01 [Note] InnoDB: VALID: space:1859997 page_no:5 page_size:16384
      170227  1:28:01 [Note] InnoDB: Page size: 16384, Possible space_id count:1
      170227  1:28:01 [Note] InnoDB: space_id:1859997, Number of pages matched: 5/5 (16384)
      170227  1:28:01 [Note] InnoDB: Chosen space:1859997
       
      170227  1:28:01 [Note] InnoDB: Restoring page 0 of tablespace 1859997
      170227  1:28:01 [Warning] InnoDB: Doublewrite does not have page_no=0 of space: 1859997
      2017-02-27 01:28:01 7f7346996820  InnoDB: Operating system error number 2 in a file operation.
      InnoDB: The error means the system cannot find the path specified.
      InnoDB: If you are installing InnoDB, remember that you must create
      InnoDB: directories yourself, InnoDB does not create them.
      InnoDB: Error: could not open single-table tablespace file ./optometrydb/page_manager_handlers.ibd
      InnoDB: We do not continue the crash recovery, because the table may become
      InnoDB: corrupt if we cannot apply the log records in the InnoDB log to it.
      InnoDB: To fix the problem and start mysqld:
      InnoDB: 1) If there is a permission problem in the file and mysqld cannot
      InnoDB: open the file, you should modify the permissions.
      InnoDB: 2) If the table is not needed, or you can restore it from a backup,
      InnoDB: then you can remove the .ibd file, and InnoDB will do a normal
      InnoDB: crash recovery and ignore that table.
      InnoDB: 3) If the file system or the disk is broken, and you cannot remove
      InnoDB: the .ibd file, you can set innodb_force_recovery > 0 in my.cnf
      InnoDB: and force InnoDB to continue crash recovery here.
      170227 01:28:01 mysqld_safe mysqld from pid file /var/lib/mysqldata/mysql/DATABASES/whsqlprd05.pid ended
      

      Attachments

        1. fdmain.c
          0.3 kB
          Daniel Black
        2. page_manager_handlers.ibd
          128 kB
          Edward Hicks

        Activity

          People

            serg Sergei Golubchik
            eshicks4 Edward Hicks
            Votes:
            2 Vote for this issue
            Watchers:
            10 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Git Integration

                Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.