Uploaded image for project: 'MariaDB Server'
  1. MariaDB Server
  2. MDEV-31156

Innodb unrepairable table with innodb_encrypt_tables

Details

    Description

      We have master-salve replication with table encryption and with aws-key-management plugin

      On the master :

      innodb_encrypt_tables = Off
      		 
      innodb_encrypt_log = Off
      		 
      innodb_encryption_threads = 0

      on the slave:

      innodb_encrypt_tables = On
      		 
      innodb_encrypt_log = Off
      		 
      innodb_encryption_threads = 4

      First we got this messages :
      [Note] InnoDB: Cannot close file ./database/table.ibd because of pending fsync

      After this the server crashed with this error message

      [ERROR] InnoDB: Operating system error number 9 in a file operation.
      		 
      [ERROR] InnoDB: Error number 9 means 'Bad file descriptor'
      		 
      [Note] InnoDB: Some operating system error numbers are described at https://mariadb.com/kb/en/library/operating-system-error-codes/
      		 
       [ERROR] InnoDB: File (unknown): 'close' returned OS error 209. Cannot continue operation
      		 
      230420 20:18:35 [ERROR] mysqld got signal 6 ;
      		 
      This could be because you hit a bug. It is also possible that this binary
      		 
      or one of the libraries it was linked against is corrupt, improperly built,
      		 
      or misconfigured. This error can also be caused by malfunctioning hardware.

      Atfer start up

      [ERROR] InnoDB: Cannot decrypt [page id: space=2890523, page number=384]

      and another time:

      [Warning] InnoDB: 16384 bytes should have been read at 1174798336 from ./database/table.ibd, but got only 0. Retrying.

      After this the table was corrupted, can't acess, alter, can't repair table.

      Our solution is on the slave (after reload databse from master) :
      innodb_encryption_threads = 0

      Attachments

        Issue Links

          relates to

          Bug - A problem which impairs or prevents the functions of the product. MDEV-31347 fil_ibd_create() may hijack the file handle of an old file

          • Critical - Crashes, loss of data, severe memory leak.
          • Closed

          Activity

            Ascending order - Click to sort in descending order
            bulepage bulepage added a comment - - edited

            After few days quiet running, again on the slave:

            2023-05-02 2:05:26 7 [Note] InnoDB: Cannot close file ./database/table1.ibd because of pending fsync
            2023-05-02 2:08:24 7 [Note] InnoDB: Cannot close file ./database/table1.ibd because of pending fsync
            2023-05-02 2:08:32 7 [Note] InnoDB: Cannot close file ./database/table1.ibd because of pending fsync
            2023-05-02 4:24:10 7 [Note] InnoDB: Cannot close file ./database/table1.ibd because of pending fsync
            2023-05-02 4:31:05 7 [Note] InnoDB: Cannot close file ./database/table1.ibd because of pending fsync
            2023-05-02 12:26:20 7 [Note] InnoDB: Cannot close file ./anotherdb/ReportLang.ibd because of pending fsync
            2023-05-02 12:29:39 7 [Note] InnoDB: Cannot close file ./anotherdb/ReportLang.ibd because of pending fsync
            2023-05-02 12:30:14 23810 [Note] InnoDB: Cannot close file ./anotherdb/ReportLang.ibd because of pending fsync

            bulepage bulepage added a comment - - edited After few days quiet running, again on the slave: 2023-05-02 2:05:26 7 [Note] InnoDB: Cannot close file ./database/table1.ibd because of pending fsync 2023-05-02 2:08:24 7 [Note] InnoDB: Cannot close file ./database/table1.ibd because of pending fsync 2023-05-02 2:08:32 7 [Note] InnoDB: Cannot close file ./database/table1.ibd because of pending fsync 2023-05-02 4:24:10 7 [Note] InnoDB: Cannot close file ./database/table1.ibd because of pending fsync 2023-05-02 4:31:05 7 [Note] InnoDB: Cannot close file ./database/table1.ibd because of pending fsync 2023-05-02 12:26:20 7 [Note] InnoDB: Cannot close file ./anotherdb/ReportLang.ibd because of pending fsync 2023-05-02 12:29:39 7 [Note] InnoDB: Cannot close file ./anotherdb/ReportLang.ibd because of pending fsync 2023-05-02 12:30:14 23810 [Note] InnoDB: Cannot close file ./anotherdb/ReportLang.ibd because of pending fsync
            serg Sergei Golubchik added a comment -

            A combination of

            [ERROR] InnoDB: Operating system error number 9 in a file operation.
            		 
            [ERROR] InnoDB: Error number 9 means 'Bad file descriptor'
            		 
            ...
            		 
             [ERROR] InnoDB: File (unknown): 'close' returned OS error 209. Cannot continue operation

            was also observed in MDEV-31347. Please try a fixed version (10.5.22, planned release date — end of July)

            serg Sergei Golubchik added a comment - A combination of [ERROR] InnoDB: Operating system error number 9 in a file operation. [ERROR] InnoDB: Error number 9 means 'Bad file descriptor' ... [ERROR] InnoDB: File (unknown): 'close' returned OS error 209. Cannot continue operation was also observed in MDEV-31347 . Please try a fixed version (10.5.22, planned release date — end of July)
            elenst Elena Stepanova added a comment -

            As a comment above says, the possibly related MDEV-31347 fix will be released with 10.5.22 / 10.6.15, planned release date — end of July.

            elenst Elena Stepanova added a comment - As a comment above says, the possibly related MDEV-31347 fix will be released with 10.5.22 / 10.6.15, planned release date — end of July.
            bulepage bulepage added a comment - - edited

            After upgrade to 10.6.15 only we got notice
            [Note] InnoDB: Cannot close file ./db/table.ibd because of 1 pending operations

            I think the problem is solved.
            Thanks

            bulepage bulepage added a comment - - edited After upgrade to 10.6.15 only we got notice [Note] InnoDB: Cannot close file ./db/table.ibd because of 1 pending operations I think the problem is solved. Thanks

            People

              Unassigned Unassigned
              bulepage bulepage
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Git Integration

                  Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.