[MDEV-31156] Innodb unrepairable table with innodb_encrypt_tables Created: 2023-04-30  Updated: 2023-08-18  Resolved: 2023-08-18

Status: Closed
Project: MariaDB Server
Component/s: Encryption, Plugin - AWS key management, Storage Engine - InnoDB
Affects Version/s: 10.6.12, 10.6.14
Fix Version/s: 10.6.15, 10.9.8, 10.10.6, 10.11.5, 11.0.3

Type: Bug Priority: Critical
Reporter: bulepage Assignee: Unassigned
Resolution: Fixed Votes: 0
Labels: None
Environment:

unbuntu 20.04

Attachments: File mysqld-1.err     File mysqld.err    
Issue Links:
Relates
relates to MDEV-31347 fil_ibd_create() may hijack the file ... Closed

 Description   

We have master-salve replication with table encryption and with aws-key-management plugin

On the master :

innodb_encrypt_tables = Off
 
innodb_encrypt_log = Off
 
innodb_encryption_threads = 0

on the slave:

innodb_encrypt_tables = On
 
innodb_encrypt_log = Off
 
innodb_encryption_threads = 4

First we got this messages :
[Note] InnoDB: Cannot close file ./database/table.ibd because of pending fsync

After this the server crashed with this error message

[ERROR] InnoDB: Operating system error number 9 in a file operation.
 
[ERROR] InnoDB: Error number 9 means 'Bad file descriptor'
 
[Note] InnoDB: Some operating system error numbers are described at https://mariadb.com/kb/en/library/operating-system-error-codes/
 
 [ERROR] InnoDB: File (unknown): 'close' returned OS error 209. Cannot continue operation
 
230420 20:18:35 [ERROR] mysqld got signal 6 ;
 
This could be because you hit a bug. It is also possible that this binary
 
or one of the libraries it was linked against is corrupt, improperly built,
 
or misconfigured. This error can also be caused by malfunctioning hardware.

Atfer start up

[ERROR] InnoDB: Cannot decrypt [page id: space=2890523, page number=384]

and another time:

[Warning] InnoDB: 16384 bytes should have been read at 1174798336 from ./database/table.ibd, but got only 0. Retrying.

After this the table was corrupted, can't acess, alter, can't repair table.

Our solution is on the slave (after reload databse from master) :
innodb_encryption_threads = 0

 Comments   
Comment by bulepage [ 2023-05-04 ]

After few days quiet running, again on the slave:

2023-05-02 2:05:26 7 [Note] InnoDB: Cannot close file ./database/table1.ibd because of pending fsync
2023-05-02 2:08:24 7 [Note] InnoDB: Cannot close file ./database/table1.ibd because of pending fsync
2023-05-02 2:08:32 7 [Note] InnoDB: Cannot close file ./database/table1.ibd because of pending fsync
2023-05-02 4:24:10 7 [Note] InnoDB: Cannot close file ./database/table1.ibd because of pending fsync
2023-05-02 4:31:05 7 [Note] InnoDB: Cannot close file ./database/table1.ibd because of pending fsync
2023-05-02 12:26:20 7 [Note] InnoDB: Cannot close file ./anotherdb/ReportLang.ibd because of pending fsync
2023-05-02 12:29:39 7 [Note] InnoDB: Cannot close file ./anotherdb/ReportLang.ibd because of pending fsync
2023-05-02 12:30:14 23810 [Note] InnoDB: Cannot close file ./anotherdb/ReportLang.ibd because of pending fsync

Comment by Sergei Golubchik [ 2023-06-06 ]

A combination of

[ERROR] InnoDB: Operating system error number 9 in a file operation.
 
[ERROR] InnoDB: Error number 9 means 'Bad file descriptor'
 
...
 
 [ERROR] InnoDB: File (unknown): 'close' returned OS error 209. Cannot continue operation

was also observed in MDEV-31347. Please try a fixed version (10.5.22, planned release date — end of July)

Comment by Elena Stepanova [ 2023-07-09 ]

As a comment above says, the possibly related MDEV-31347 fix will be released with 10.5.22 / 10.6.15, planned release date — end of July.

Comment by bulepage [ 2023-08-18 ]

After upgrade to 10.6.15 only we got notice
[Note] InnoDB: Cannot close file ./db/table.ibd because of 1 pending operations

I think the problem is solved.
Thanks

Generated at Thu Feb 08 10:21:42 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.