Uploaded image for project: 'MariaDB Server'
  1. MariaDB Server
  2. MDEV-22056

Rocks db corrupts data when disk is out of space

Details

    Description

      How do we eliminate this risk and how do we fix this issue after it happens?
      I issued a command to truncate a table, and since it was stuck, I rebooted.

      2020-03-27 5:18:17 0 [Warning] RocksDB: Schema mismatch - A .frm file exists for table asterisk.acc, but that table is not registered in RocksDB
      2020-03-27 5:18:17 0 [ERROR] RocksDB: Problems validating data dictionary against .frm files, exiting
      2020-03-27 5:18:17 0 [ERROR] RocksDB: Failed to initialize DDL manager.
      2020-03-27 5:18:17 0 [ERROR] Plugin 'ROCKSDB' init function returned error.
      2020-03-27 5:18:17 0 [ERROR] Plugin 'ROCKSDB' registration as a STORAGE ENGINE failed.
      2020-03-27 5:18:17 0 [ERROR] Unknown/unsupported storage engine: rocksdb

      Attachments

        Issue Links

          relates to

          Task - A task that needs to be done. MDEV-17567 Atomic DDL

          • Critical - Crashes, loss of data, severe memory leak.
          • Closed

          Activity

            Ascending order - Click to sort in descending order
            marko Marko Mäkelä added a comment -

            philip_38, from the messages it would appear to me that you might be able to work around the problem by deleting the file asterisk/acc.frm from the data directory. Once the server has started up, you should be able to issue CREATE TABLE.

            TRUNCATE is a bit special case of a DDL operation, because it will not modify the data dictionary on the SQL layer, or touch the .frm file. Apparently, the problem is that MyRocks will essentially drop the table when it runs out of space. InnoDB could be even worse in such scenarios, ever eager to commit suicide when it encounters a fatal error. In MDEV-13564 I rewrote its TRUNCATE so that the InnoDB in MariaDB will internally rename the table, create a new one, and drop the original table.

            Could you create a minimal test case for this? It could involve creating a tiny file system on a loopback device. Such a test could also be helpful when we develop MDEV-17567.

            marko Marko Mäkelä added a comment - philip_38 , from the messages it would appear to me that you might be able to work around the problem by deleting the file asterisk/acc.frm from the data directory. Once the server has started up, you should be able to issue CREATE TABLE . TRUNCATE is a bit special case of a DDL operation, because it will not modify the data dictionary on the SQL layer, or touch the .frm file. Apparently, the problem is that MyRocks will essentially drop the table when it runs out of space. InnoDB could be even worse in such scenarios, ever eager to commit suicide when it encounters a fatal error. In MDEV-13564 I rewrote its TRUNCATE so that the InnoDB in MariaDB will internally rename the table, create a new one, and drop the original table. Could you create a minimal test case for this? It could involve creating a tiny file system on a loopback device. Such a test could also be helpful when we develop MDEV-17567 .
            philip_38 Philip orleans added a comment -

            Could you elaborate on how to create a small file system on a loopback
            device?

            On Fri, Mar 27, 2020 at 3:19 AM Marko Mäkelä (Jira) <jira@mariadb.org>

            philip_38 Philip orleans added a comment - Could you elaborate on how to create a small file system on a loopback device? On Fri, Mar 27, 2020 at 3:19 AM Marko Mäkelä (Jira) <jira@mariadb.org>
            elenst Elena Stepanova added a comment -

            If a test case is really needed for debugging/fixing this, a better way to do it would be introducing a debug injection to imitate disk space outage (if applicable to this particular failure). Only then will the test case be portable and usable for the regression suite.

            elenst Elena Stepanova added a comment - If a test case is really needed for debugging/fixing this, a better way to do it would be introducing a debug injection to imitate disk space outage (if applicable to this particular failure). Only then will the test case be portable and usable for the regression suite.

            People

              psergei Sergei Petrunia
              philip_38 Philip orleans
              Votes:
              1 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

                Created:
                Updated:

                Git Integration

                  Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.