Details

    • Bug
    • Status: Closed (View Workflow)
    • Major
    • Resolution: Incomplete
    • 10.0.28
    • N/A
    • None
    • CentOS 6.9

    Description

      10.0.28

      InnoDB: Page may be a freshly allocated page
      InnoDB: Database page corruption on disk or a failed
      InnoDB: file read of page 12862421.
      InnoDB: You may have to recover from a backup.
      InnoDB: It is also possible that your operating
      InnoDB: system has corrupted its own file cache
      InnoDB: and rebooting your computer removes the
      InnoDB: error.
      InnoDB: If the corrupt page is an index page
      InnoDB: you can also try to fix the corruption
      InnoDB: by dumping, dropping, and reimporting
      InnoDB: the corrupt table. You can use CHECK
      InnoDB: TABLE to scan your table for corruption.
      InnoDB: See also http://dev.mysql.com/doc/refman/5.6/en/forcing-innodb-recovery.html
      InnoDB: about forcing recovery.
      InnoDB: Error: Unable to read tablespace 61037 page no 12862421 into the buffer pool after 100 attempts
      InnoDB: The most probable cause of this error may be that the table has been corrupted.
      InnoDB: You can try to fix this problem by using innodb_force_recovery.
      InnoDB: Please see reference manual for more details.
      InnoDB: Aborting...
      2018-07-14 19:25:31 7edabadfa700  InnoDB: Assertion failure in thread 139477903189760 in file buf0buf.cc line 2895
      InnoDB: We intentionally generate a memory trap.
      InnoDB: Submit a detailed bug report to http://bugs.mysql.com.
      InnoDB: If you get repeated assertion failures or crashes, even
      InnoDB: immediately after the mysqld startup, there may be
      InnoDB: corruption in the InnoDB tablespace. Please refer to
      InnoDB: http://dev.mysql.com/doc/refman/5.6/en/forcing-innodb-recovery.html
      InnoDB: about forcing recovery.
      180714 19:25:31 [ERROR] mysqld got signal 6 ;
      This could be because you hit a bug. It is also possible that this binary
      or one of the libraries it was linked against is corrupt, improperly built,
      or misconfigured. This error can also be caused by malfunctioning hardware.
       
      To report this bug, see https://mariadb.com/kb/en/reporting-bugs
       
      We will try our best to scrape up some info that will hopefully help
      diagnose the problem, but since we have already crashed,
      something is definitely wrong and this may fail.
       
      Server version: 10.0.28-MariaDB
      key_buffer_size=25165824
      read_buffer_size=2097152
      max_used_connections=1911
      max_threads=2002
      thread_count=101
      It is possible that mysqld could use up to
      key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 8264428 K  bytes of memory
      Hope that's ok; if not, decrease some variables in the equation.
       
      Thread pointer: 0x0x0
      Attempting backtrace. You can use the following information to find out
      where mysqld died. If you see no messages after this, something went
      terribly wrong...
      stack_bottom = 0x0 thread_stack 0x48000
      (my_addr_resolve failure: fork)
      /engn001/masvc01/GMRPP/mysql/bin/mysqld(my_print_stacktrace+0x2e) [0xbda1ae]
      /engn001/masvc01/GMRPP/mysql/bin/mysqld(handle_fatal_signal+0x49a) [0x722a9a]
      /lib64/libpthread.so.0() [0x3063c0f790]
      /lib64/libc.so.6(gsignal+0x35) [0x3063832625]
      /lib64/libc.so.6(abort+0x175) [0x3063833e05]
      /engn001/masvc01/GMRPP/mysql/bin/mysqld() [0x9d135f]
      /engn001/masvc01/GMRPP/mysql/bin/mysqld() [0x9b498e]
      /engn001/masvc01/GMRPP/mysql/bin/mysqld() [0x9b69b7]
      /engn001/masvc01/GMRPP/mysql/bin/mysqld() [0x9590ea]
      /engn001/masvc01/GMRPP/mysql/bin/mysqld() [0x9572f3]
      /engn001/masvc01/GMRPP/mysql/bin/mysqld() [0x958c93]
      /engn001/masvc01/GMRPP/mysql/bin/mysqld() [0x925ef8]
      /engn001/masvc01/GMRPP/mysql/bin/mysqld() [0x982228]
      /engn001/masvc01/GMRPP/mysql/bin/mysqld() [0x96d40e]
      /engn001/masvc01/GMRPP/mysql/bin/mysqld() [0x9733e0]
      /lib64/libpthread.so.0() [0x3063c07a51]
      /lib64/libc.so.6(clone+0x6d) [0x30638e896d]
      The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
      information that should help you find out what is causing the crash.
       
      ...
       
      180714 19:32:48 server_audit: MariaDB Audit Plugin version 1.4.0 STARTED.
      180714 19:32:48 server_audit: logging started to the file server_audit.log.
      180714 19:32:48 [Note] Server socket created on IP: '::'.
      180714 19:32:48 [ERROR] mysqld: Table './mysql/user' is marked as crashed and should be repaired
      180714 19:32:48 [ERROR] mysqld: Table 'user' is marked as crashed and should be repaired
      180714 19:32:48 [Warning] Checking table:   './mysql/user'
      180714 19:32:48 [ERROR] mysql.user: 1 client is using or hasn't closed the table properly
      180714 19:32:48 [ERROR] mysqld: Table './mysql/db' is marked as crashed and should be repaired
      180714 19:32:48 [ERROR] mysqld: Table 'db' is marked as crashed and should be repaired
      180714 19:32:48 [Warning] Checking table:   './mysql/db'
      180714 19:32:48 [ERROR] mysql.db: 1 client is using or hasn't closed the table properly
      180714 19:32:48 [ERROR] mysqld: Table './mysql/tables_priv' is marked as crashed and should be repaired
      180714 19:32:48 [ERROR] mysqld: Table 'tables_priv' is marked as crashed and should be repaired
      180714 19:32:48 [Warning] Checking table:   './mysql/tables_priv'
      180714 19:32:48 [ERROR] mysql.tables_priv: 1 client is using or hasn't closed the table properly
      180714 19:32:48 [ERROR] mysqld: Table './mysql/procs_priv' is marked as crashed and should be repaired
      

      We have confirmed that the db is restarting indefinitely after signal 6 in the error log.
      Can you tell us the cause of signal 6?

      Attachments

        1. cpu.txt
          57 kB
        2. disk.txt
          0.7 kB
        3. engine.txt
          35 kB
        4. gmrpe_table_size.txt
          5 kB
        5. gmrpp_table_size.txt
          21 kB
        6. memory.txt
          0.2 kB
        7. plugin.txt
          5 kB
        8. sts.txt
          11 kB
        9. var.txt
          15 kB

        Issue Links

          Activity

            ossk_db ossk_db added a comment - cpu.txt disk.txt engine.txt gmrpe_table_size.txt gmrpp_table_size.txt memory.txt plugin.txt sts.txt var.txt

            Back to your original question about the cause of signal 6, InnoDB itself rather verbosely explains it in the error log: it's been trying to read a page, fails to do so, apparently due to data corruption, and after a number of attempts gives up.
            Assigning to marko for further advice/ideas/etc.

            elenst Elena Stepanova added a comment - Back to your original question about the cause of signal 6, InnoDB itself rather verbosely explains it in the error log: it's been trying to read a page, fails to do so, apparently due to data corruption, and after a number of attempts gives up. Assigning to marko for further advice/ideas/etc.
            ossk_db ossk_db added a comment -

            Hi~ Team.

            1. Do you know if this text has been patched in the latest version?

            2. Is the obvious data corruption a problem with the disk HW?

            ossk_db ossk_db added a comment - Hi~ Team. 1. Do you know if this text has been patched in the latest version? 2. Is the obvious data corruption a problem with the disk HW?

            Hi ossk_db, my simplest explanation would be to blame the hardware.
            I did not see any messages about the page LSN being in the future. Such messages could be displayed when something wrong was done, such as killing the server, deleting ib_logfile* and restarting.

            That said, there is some room for improvement for InnoDB could in this area: MDEV-13542 should propagate read errors up to the client (so that the user would know which table or SQL statement is in question), and the InnoDB doublewrite buffer could be improved as well (MDEV-11799, MDEV-12905).

            The doublewrite buffer only matters if the mysqld process was killed (or the whole system was killed) while InnoDB was in the middle of writing a data page.

            marko Marko Mäkelä added a comment - Hi ossk_db , my simplest explanation would be to blame the hardware. I did not see any messages about the page LSN being in the future. Such messages could be displayed when something wrong was done, such as killing the server, deleting ib_logfile* and restarting. That said, there is some room for improvement for InnoDB could in this area: MDEV-13542 should propagate read errors up to the client (so that the user would know which table or SQL statement is in question), and the InnoDB doublewrite buffer could be improved as well ( MDEV-11799 , MDEV-12905 ). The doublewrite buffer only matters if the mysqld process was killed (or the whole system was killed) while InnoDB was in the middle of writing a data page.

            One thing that could be useful is to pay attention to the diagnostics provided by the storage layer, to get an early warning of upcoming failures. Modern hard disks and SSDs provide the S.M.A.R.T. interface. On GNU/Linux, there is the smartmontools package. If I were running an important server, I would try adding some diagnostics around smartctl -A /dev/sda or similar.

            marko Marko Mäkelä added a comment - One thing that could be useful is to pay attention to the diagnostics provided by the storage layer, to get an early warning of upcoming failures. Modern hard disks and SSDs provide the S.M.A.R.T. interface. On GNU/Linux, there is the smartmontools package. If I were running an important server, I would try adding some diagnostics around smartctl -A /dev/sda or similar.

            People

              marko Marko Mäkelä
              kyoung-yeon ssauravy
              Votes:
              1 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Git Integration

                  Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.