[MDEV-31156] Innodb unrepairable table with innodb_encrypt_tables - Jira

bulepage created issue - 2023-04-30 10:49

bulepage made changes - 2023-04-30 10:51

Field	Original Value	New Value
Description	We have master-salve replication with table encryption with aws-key-management plugin On the master : innodb_encrypt_tables = Off innodb_encrypt_log = Off innodb_encryption_threads = 0 on the slave: innodb_encrypt_tables = On innodb_encrypt_log = Off innodb_encryption_threads = 4 First we got this messages : [Note] InnoDB: Cannot close file ./database/table.ibd because of pending fsync After this the server crashed with this error message <pre> [ERROR] InnoDB: Operating system error number 9 in a file operation. [ERROR] InnoDB: Error number 9 means 'Bad file descriptor' [Note] InnoDB: Some operating system error numbers are described at https://mariadb.com/kb/en/library/operating-system-error-codes/ [ERROR] InnoDB: File (unknown): 'close' returned OS error 209. Cannot continue operation 230420 20:18:35 [ERROR] mysqld got signal 6 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. </pre> Atfer start up <pre> [ERROR] InnoDB: Cannot decrypt [page id: space=2890523, page number=384] <pre> and another time: <pre> [Warning] InnoDB: 16384 bytes should have been read at 1174798336 from ./database/table.ibd, but got only 0. Retrying. </pre> After this the table was corrupted, can't acess or alter. Our solution is pn the slave : innodb_encryption_threads = 0	We have master-salve replication with table encryption with aws-key-management plugin On the master : {code:log} innodb_encrypt_tables = Off innodb_encrypt_log = Off innodb_encryption_threads = 0 {code} on the slave: innodb_encrypt_tables = On innodb_encrypt_log = Off innodb_encryption_threads = 4 First we got this messages : [Note] InnoDB: Cannot close file ./database/table.ibd because of pending fsync After this the server crashed with this error message <pre> [ERROR] InnoDB: Operating system error number 9 in a file operation. [ERROR] InnoDB: Error number 9 means 'Bad file descriptor' [Note] InnoDB: Some operating system error numbers are described at https://mariadb.com/kb/en/library/operating-system-error-codes/ [ERROR] InnoDB: File (unknown): 'close' returned OS error 209. Cannot continue operation 230420 20:18:35 [ERROR] mysqld got signal 6 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. </pre> Atfer start up <pre> [ERROR] InnoDB: Cannot decrypt [page id: space=2890523, page number=384] <pre> and another time: <pre> [Warning] InnoDB: 16384 bytes should have been read at 1174798336 from ./database/table.ibd, but got only 0. Retrying. </pre> After this the table was corrupted, can't acess or alter. Our solution is pn the slave : innodb_encryption_threads = 0

bulepage made changes - 2023-04-30 10:51

Description

We have master-salve replication with table encryption with aws-key-management plugin

On the master :

{code:log}

innodb_encrypt_tables = Off
innodb_encrypt_log = Off
*innodb_encryption_threads = 0*
{code}

on the slave:
innodb_encrypt_tables = On
innodb_encrypt_log = Off
*innodb_encryption_threads = 4*

First we got this messages :
[Note] InnoDB: Cannot close file ./database/table.ibd because of pending fsync

After this the server crashed with this error message
<pre>
[ERROR] InnoDB: Operating system error number 9 in a file operation.
[ERROR] InnoDB: Error number 9 means 'Bad file descriptor'
[Note] InnoDB: Some operating system error numbers are described at https://mariadb.com/kb/en/library/operating-system-error-codes/
[ERROR] InnoDB: File (unknown): 'close' returned OS error 209. Cannot continue operation
230420 20:18:35 [ERROR] mysqld got signal 6 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.

</pre>
Atfer start up
<pre>
[ERROR] InnoDB: Cannot decrypt [page id: space=2890523, page number=384]
<pre>
and another time:
<pre>
[Warning] InnoDB: 16384 bytes should have been read at 1174798336 from ./database/table.ibd, but got only 0. Retrying.
</pre>

After this the table was corrupted, can't acess or alter.

Our solution is pn the slave :
innodb_encryption_threads = 0

We have master-salve replication with table encryption with aws-key-management plugin

On the master :

{code:log}
innodb_encrypt_tables = Off
innodb_encrypt_log = Off
*innodb_encryption_threads = 0*
{code}

on the slave:
innodb_encrypt_tables = On
innodb_encrypt_log = Off
*innodb_encryption_threads = 4*

First we got this messages :
[Note] InnoDB: Cannot close file ./database/table.ibd because of pending fsync

After this the server crashed with this error message
<pre>
[ERROR] InnoDB: Operating system error number 9 in a file operation.
[ERROR] InnoDB: Error number 9 means 'Bad file descriptor'
[Note] InnoDB: Some operating system error numbers are described at https://mariadb.com/kb/en/library/operating-system-error-codes/
[ERROR] InnoDB: File (unknown): 'close' returned OS error 209. Cannot continue operation
230420 20:18:35 [ERROR] mysqld got signal 6 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.

</pre>
Atfer start up
<pre>
[ERROR] InnoDB: Cannot decrypt [page id: space=2890523, page number=384]
<pre>
and another time:
<pre>
[Warning] InnoDB: 16384 bytes should have been read at 1174798336 from ./database/table.ibd, but got only 0. Retrying.
</pre>

After this the table was corrupted, can't acess or alter.

Our solution is pn the slave :
innodb_encryption_threads = 0

bulepage made changes - 2023-04-30 10:51

Description

We have master-salve replication with table encryption with aws-key-management plugin

On the master :

{code:log}
innodb_encrypt_tables = Off
innodb_encrypt_log = Off
*innodb_encryption_threads = 0*
{code}

on the slave:
innodb_encrypt_tables = On
innodb_encrypt_log = Off
*innodb_encryption_threads = 4*

First we got this messages :
[Note] InnoDB: Cannot close file ./database/table.ibd because of pending fsync

After this the server crashed with this error message
<pre>
[ERROR] InnoDB: Operating system error number 9 in a file operation.
[ERROR] InnoDB: Error number 9 means 'Bad file descriptor'
[Note] InnoDB: Some operating system error numbers are described at https://mariadb.com/kb/en/library/operating-system-error-codes/
[ERROR] InnoDB: File (unknown): 'close' returned OS error 209. Cannot continue operation
230420 20:18:35 [ERROR] mysqld got signal 6 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.

</pre>
Atfer start up
<pre>
[ERROR] InnoDB: Cannot decrypt [page id: space=2890523, page number=384]
<pre>
and another time:
<pre>
[Warning] InnoDB: 16384 bytes should have been read at 1174798336 from ./database/table.ibd, but got only 0. Retrying.
</pre>

After this the table was corrupted, can't acess or alter.

Our solution is pn the slave :
innodb_encryption_threads = 0

We have master-salve replication with table encryption with aws-key-management plugin

On the master :

{code:log}
innodb_encrypt_tables = Off
innodb_encrypt_log = Off
*innodb_encryption_threads = 0*
{code}

on the slave:
{code:log}
innodb_encrypt_tables = On
innodb_encrypt_log = Off
*innodb_encryption_threads = 4*
{code}

First we got this messages :
[Note] InnoDB: Cannot close file ./database/table.ibd because of pending fsync

After this the server crashed with this error message
<pre>
[ERROR] InnoDB: Operating system error number 9 in a file operation.
[ERROR] InnoDB: Error number 9 means 'Bad file descriptor'
[Note] InnoDB: Some operating system error numbers are described at https://mariadb.com/kb/en/library/operating-system-error-codes/
[ERROR] InnoDB: File (unknown): 'close' returned OS error 209. Cannot continue operation
230420 20:18:35 [ERROR] mysqld got signal 6 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.

</pre>
Atfer start up
<pre>
[ERROR] InnoDB: Cannot decrypt [page id: space=2890523, page number=384]
<pre>
and another time:
<pre>
[Warning] InnoDB: 16384 bytes should have been read at 1174798336 from ./database/table.ibd, but got only 0. Retrying.
</pre>

After this the table was corrupted, can't acess or alter.

Our solution is pn the slave :
innodb_encryption_threads = 0

bulepage made changes - 2023-04-30 10:53

Description

We have master-salve replication with table encryption with aws-key-management plugin

On the master :

{code:log}
innodb_encrypt_tables = Off
innodb_encrypt_log = Off
*innodb_encryption_threads = 0*
{code}

on the slave:
{code:log}
innodb_encrypt_tables = On
innodb_encrypt_log = Off
*innodb_encryption_threads = 4*
{code}

First we got this messages :
[Note] InnoDB: Cannot close file ./database/table.ibd because of pending fsync

After this the server crashed with this error message
<pre>
[ERROR] InnoDB: Operating system error number 9 in a file operation.
[ERROR] InnoDB: Error number 9 means 'Bad file descriptor'
[Note] InnoDB: Some operating system error numbers are described at https://mariadb.com/kb/en/library/operating-system-error-codes/
[ERROR] InnoDB: File (unknown): 'close' returned OS error 209. Cannot continue operation
230420 20:18:35 [ERROR] mysqld got signal 6 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.

</pre>
Atfer start up
<pre>
[ERROR] InnoDB: Cannot decrypt [page id: space=2890523, page number=384]
<pre>
and another time:
<pre>
[Warning] InnoDB: 16384 bytes should have been read at 1174798336 from ./database/table.ibd, but got only 0. Retrying.
</pre>

After this the table was corrupted, can't acess or alter.

Our solution is pn the slave :
innodb_encryption_threads = 0

We have master-salve replication with table encryption with aws-key-management plugin

On the master :

{code:log}
innodb_encrypt_tables = Off
innodb_encrypt_log = Off
*innodb_encryption_threads = 0*
{code}

on the slave:
{code:log}
innodb_encrypt_tables = On
innodb_encrypt_log = Off
*innodb_encryption_threads = 4*
{code}

First we got this messages :
[Note] InnoDB: Cannot close file ./database/table.ibd because of pending fsync

After this the server crashed with this error message
{code:log}
[ERROR] InnoDB: Operating system error number 9 in a file operation.
[ERROR] InnoDB: Error number 9 means 'Bad file descriptor'
[Note] InnoDB: Some operating system error numbers are described at https://mariadb.com/kb/en/library/operating-system-error-codes/
[ERROR] InnoDB: File (unknown): 'close' returned OS error 209. Cannot continue operation
230420 20:18:35 [ERROR] mysqld got signal 6 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
{code}

Atfer start up

[ERROR] InnoDB: Cannot decrypt [page id: space=2890523, page number=384]

and another time:

[Warning] InnoDB: 16384 bytes should have been read at 1174798336 from ./database/table.ibd, but got only 0. Retrying.

After this the table was corrupted, can't acess or alter.

Our solution is pn the slave :
innodb_encryption_threads = 0

bulepage made changes - 2023-04-30 10:53

Description

We have master-salve replication with table encryption with aws-key-management plugin

On the master :

{code:log}
innodb_encrypt_tables = Off
innodb_encrypt_log = Off
*innodb_encryption_threads = 0*
{code}

on the slave:
{code:log}
innodb_encrypt_tables = On
innodb_encrypt_log = Off
*innodb_encryption_threads = 4*
{code}

First we got this messages :
[Note] InnoDB: Cannot close file ./database/table.ibd because of pending fsync

After this the server crashed with this error message
{code:log}
[ERROR] InnoDB: Operating system error number 9 in a file operation.
[ERROR] InnoDB: Error number 9 means 'Bad file descriptor'
[Note] InnoDB: Some operating system error numbers are described at https://mariadb.com/kb/en/library/operating-system-error-codes/
[ERROR] InnoDB: File (unknown): 'close' returned OS error 209. Cannot continue operation
230420 20:18:35 [ERROR] mysqld got signal 6 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
{code}

Atfer start up

[ERROR] InnoDB: Cannot decrypt [page id: space=2890523, page number=384]

and another time:

[Warning] InnoDB: 16384 bytes should have been read at 1174798336 from ./database/table.ibd, but got only 0. Retrying.

After this the table was corrupted, can't acess or alter.

Our solution is pn the slave :
innodb_encryption_threads = 0

We have master-salve replication with table encryption with aws-key-management plugin

On the master :

{code:log}
innodb_encrypt_tables = Off
innodb_encrypt_log = Off
innodb_encryption_threads = 0
{code}

on the slave:
{code:log}
innodb_encrypt_tables = On
innodb_encrypt_log = Off
innodb_encryption_threads = 4
{code}

First we got this messages :
[Note] InnoDB: Cannot close file ./database/table.ibd because of pending fsync

After this the server crashed with this error message
{code:log}
[ERROR] InnoDB: Operating system error number 9 in a file operation.
[ERROR] InnoDB: Error number 9 means 'Bad file descriptor'
[Note] InnoDB: Some operating system error numbers are described at https://mariadb.com/kb/en/library/operating-system-error-codes/
[ERROR] InnoDB: File (unknown): 'close' returned OS error 209. Cannot continue operation
230420 20:18:35 [ERROR] mysqld got signal 6 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
{code}

Atfer start up

[ERROR] InnoDB: Cannot decrypt [page id: space=2890523, page number=384]

and another time:

[Warning] InnoDB: 16384 bytes should have been read at 1174798336 from ./database/table.ibd, but got only 0. Retrying.

After this the table was corrupted, can't acess or alter.

Our solution is pn the slave :
innodb_encryption_threads = 0

bulepage made changes - 2023-04-30 10:54

Description

We have master-salve replication with table encryption with aws-key-management plugin

On the master :

{code:log}
innodb_encrypt_tables = Off
innodb_encrypt_log = Off
innodb_encryption_threads = 0
{code}

on the slave:
{code:log}
innodb_encrypt_tables = On
innodb_encrypt_log = Off
innodb_encryption_threads = 4
{code}

First we got this messages :
[Note] InnoDB: Cannot close file ./database/table.ibd because of pending fsync

After this the server crashed with this error message
{code:log}
[ERROR] InnoDB: Operating system error number 9 in a file operation.
[ERROR] InnoDB: Error number 9 means 'Bad file descriptor'
[Note] InnoDB: Some operating system error numbers are described at https://mariadb.com/kb/en/library/operating-system-error-codes/
[ERROR] InnoDB: File (unknown): 'close' returned OS error 209. Cannot continue operation
230420 20:18:35 [ERROR] mysqld got signal 6 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
{code}

Atfer start up

[ERROR] InnoDB: Cannot decrypt [page id: space=2890523, page number=384]

and another time:

[Warning] InnoDB: 16384 bytes should have been read at 1174798336 from ./database/table.ibd, but got only 0. Retrying.

After this the table was corrupted, can't acess or alter.

Our solution is pn the slave :
innodb_encryption_threads = 0

We have master-salve replication with table encryption and with aws-key-management plugin

On the master :

{code:log}
innodb_encrypt_tables = Off
innodb_encrypt_log = Off
innodb_encryption_threads = 0
{code}

on the slave:
{code:log}
innodb_encrypt_tables = On
innodb_encrypt_log = Off
innodb_encryption_threads = 4
{code}

First we got this messages :
[Note] InnoDB: Cannot close file ./database/table.ibd because of pending fsync

After this the server crashed with this error message
{code:log}
[ERROR] InnoDB: Operating system error number 9 in a file operation.
[ERROR] InnoDB: Error number 9 means 'Bad file descriptor'
[Note] InnoDB: Some operating system error numbers are described at https://mariadb.com/kb/en/library/operating-system-error-codes/
[ERROR] InnoDB: File (unknown): 'close' returned OS error 209. Cannot continue operation
230420 20:18:35 [ERROR] mysqld got signal 6 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
{code}

Atfer start up

[ERROR] InnoDB: Cannot decrypt [page id: space=2890523, page number=384]

and another time:

[Warning] InnoDB: 16384 bytes should have been read at 1174798336 from ./database/table.ibd, but got only 0. Retrying.

After this the table was corrupted, can't acess or alter.

Our solution is pn the slave :
innodb_encryption_threads = 0

bulepage made changes - 2023-04-30 10:55

Description

We have master-salve replication with table encryption and with aws-key-management plugin

On the master :

{code:log}
innodb_encrypt_tables = Off
innodb_encrypt_log = Off
innodb_encryption_threads = 0
{code}

on the slave:
{code:log}
innodb_encrypt_tables = On
innodb_encrypt_log = Off
innodb_encryption_threads = 4
{code}

First we got this messages :
[Note] InnoDB: Cannot close file ./database/table.ibd because of pending fsync

After this the server crashed with this error message
{code:log}
[ERROR] InnoDB: Operating system error number 9 in a file operation.
[ERROR] InnoDB: Error number 9 means 'Bad file descriptor'
[Note] InnoDB: Some operating system error numbers are described at https://mariadb.com/kb/en/library/operating-system-error-codes/
[ERROR] InnoDB: File (unknown): 'close' returned OS error 209. Cannot continue operation
230420 20:18:35 [ERROR] mysqld got signal 6 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
{code}

Atfer start up

[ERROR] InnoDB: Cannot decrypt [page id: space=2890523, page number=384]

and another time:

[Warning] InnoDB: 16384 bytes should have been read at 1174798336 from ./database/table.ibd, but got only 0. Retrying.

After this the table was corrupted, can't acess or alter.

Our solution is pn the slave :
innodb_encryption_threads = 0

We have master-salve replication with table encryption and with aws-key-management plugin

On the master :

{code:log}
innodb_encrypt_tables = Off
innodb_encrypt_log = Off
innodb_encryption_threads = 0
{code}

on the slave:
{code:log}
innodb_encrypt_tables = On
innodb_encrypt_log = Off
innodb_encryption_threads = 4
{code}

First we got this messages :
[Note] InnoDB: Cannot close file ./database/table.ibd because of pending fsync

After this the server crashed with this error message
{code:log}
[ERROR] InnoDB: Operating system error number 9 in a file operation.
[ERROR] InnoDB: Error number 9 means 'Bad file descriptor'
[Note] InnoDB: Some operating system error numbers are described at https://mariadb.com/kb/en/library/operating-system-error-codes/
[ERROR] InnoDB: File (unknown): 'close' returned OS error 209. Cannot continue operation
230420 20:18:35 [ERROR] mysqld got signal 6 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
{code}

Atfer start up

[ERROR] InnoDB: Cannot decrypt [page id: space=2890523, page number=384]

and another time:

[Warning] InnoDB: 16384 bytes should have been read at 1174798336 from ./database/table.ibd, but got only 0. Retrying.

After this the table was corrupted, can't acess or alter.

Our solution is on the slave :
innodb_encryption_threads = 0

bulepage made changes - 2023-04-30 11:02

Description

We have master-salve replication with table encryption and with aws-key-management plugin

On the master :

{code:log}
innodb_encrypt_tables = Off
innodb_encrypt_log = Off
innodb_encryption_threads = 0
{code}

on the slave:
{code:log}
innodb_encrypt_tables = On
innodb_encrypt_log = Off
innodb_encryption_threads = 4
{code}

First we got this messages :
[Note] InnoDB: Cannot close file ./database/table.ibd because of pending fsync

After this the server crashed with this error message
{code:log}
[ERROR] InnoDB: Operating system error number 9 in a file operation.
[ERROR] InnoDB: Error number 9 means 'Bad file descriptor'
[Note] InnoDB: Some operating system error numbers are described at https://mariadb.com/kb/en/library/operating-system-error-codes/
[ERROR] InnoDB: File (unknown): 'close' returned OS error 209. Cannot continue operation
230420 20:18:35 [ERROR] mysqld got signal 6 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
{code}

Atfer start up

[ERROR] InnoDB: Cannot decrypt [page id: space=2890523, page number=384]

and another time:

[Warning] InnoDB: 16384 bytes should have been read at 1174798336 from ./database/table.ibd, but got only 0. Retrying.

After this the table was corrupted, can't acess or alter.

Our solution is on the slave :
innodb_encryption_threads = 0

We have master-salve replication with table encryption and with aws-key-management plugin

On the master :

{code:log}
innodb_encrypt_tables = Off
innodb_encrypt_log = Off
innodb_encryption_threads = 0
{code}

on the slave:
{code:log}
innodb_encrypt_tables = On
innodb_encrypt_log = Off
innodb_encryption_threads = 4
{code}

First we got this messages :
{{[Note] InnoDB: Cannot close file ./database/table.ibd because of pending fsync}}

After this the server crashed with this error message
{code:log}
[ERROR] InnoDB: Operating system error number 9 in a file operation.
[ERROR] InnoDB: Error number 9 means 'Bad file descriptor'
[Note] InnoDB: Some operating system error numbers are described at https://mariadb.com/kb/en/library/operating-system-error-codes/
[ERROR] InnoDB: File (unknown): 'close' returned OS error 209. Cannot continue operation
230420 20:18:35 [ERROR] mysqld got signal 6 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
{code}

Atfer start up

[ERROR] InnoDB: Cannot decrypt [page id: space=2890523, page number=384]

and another time:

[Warning] InnoDB: 16384 bytes should have been read at 1174798336 from ./database/table.ibd, but got only 0. Retrying.

After this the table was corrupted, can't acess or alter.

Our solution is on the slave :
innodb_encryption_threads = 0

bulepage made changes - 2023-04-30 11:17

Description

We have master-salve replication with table encryption and with aws-key-management plugin

On the master :

{code:log}
innodb_encrypt_tables = Off
innodb_encrypt_log = Off
innodb_encryption_threads = 0
{code}

on the slave:
{code:log}
innodb_encrypt_tables = On
innodb_encrypt_log = Off
innodb_encryption_threads = 4
{code}

First we got this messages :
{{[Note] InnoDB: Cannot close file ./database/table.ibd because of pending fsync}}

After this the server crashed with this error message
{code:log}
[ERROR] InnoDB: Operating system error number 9 in a file operation.
[ERROR] InnoDB: Error number 9 means 'Bad file descriptor'
[Note] InnoDB: Some operating system error numbers are described at https://mariadb.com/kb/en/library/operating-system-error-codes/
[ERROR] InnoDB: File (unknown): 'close' returned OS error 209. Cannot continue operation
230420 20:18:35 [ERROR] mysqld got signal 6 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
{code}

Atfer start up

[ERROR] InnoDB: Cannot decrypt [page id: space=2890523, page number=384]

and another time:

[Warning] InnoDB: 16384 bytes should have been read at 1174798336 from ./database/table.ibd, but got only 0. Retrying.

After this the table was corrupted, can't acess or alter.

Our solution is on the slave :
innodb_encryption_threads = 0

We have master-salve replication with table encryption and with aws-key-management plugin

On the master :

{code:log}
innodb_encrypt_tables = Off
innodb_encrypt_log = Off
innodb_encryption_threads = 0
{code}

on the slave:
{code:log}
innodb_encrypt_tables = On
innodb_encrypt_log = Off
innodb_encryption_threads = 4
{code}

First we got this messages :
{{[Note] InnoDB: Cannot close file ./database/table.ibd because of pending fsync}}

After this the server crashed with this error message
{code:log}
[ERROR] InnoDB: Operating system error number 9 in a file operation.
[ERROR] InnoDB: Error number 9 means 'Bad file descriptor'
[Note] InnoDB: Some operating system error numbers are described at https://mariadb.com/kb/en/library/operating-system-error-codes/
[ERROR] InnoDB: File (unknown): 'close' returned OS error 209. Cannot continue operation
230420 20:18:35 [ERROR] mysqld got signal 6 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
{code}

Atfer start up

[ERROR] InnoDB: Cannot decrypt [page id: space=2890523, page number=384]

and another time:

[Warning] InnoDB: 16384 bytes should have been read at 1174798336 from ./database/table.ibd, but got only 0. Retrying.

After this the table was corrupted, can't acess, alter, can't repair table.

Our solution is on the slave :
innodb_encryption_threads = 0

bulepage made changes - 2023-04-30 13:43

Description

We have master-salve replication with table encryption and with aws-key-management plugin

On the master :

{code:log}
innodb_encrypt_tables = Off
innodb_encrypt_log = Off
innodb_encryption_threads = 0
{code}

on the slave:
{code:log}
innodb_encrypt_tables = On
innodb_encrypt_log = Off
innodb_encryption_threads = 4
{code}

First we got this messages :
{{[Note] InnoDB: Cannot close file ./database/table.ibd because of pending fsync}}

After this the server crashed with this error message
{code:log}
[ERROR] InnoDB: Operating system error number 9 in a file operation.
[ERROR] InnoDB: Error number 9 means 'Bad file descriptor'
[Note] InnoDB: Some operating system error numbers are described at https://mariadb.com/kb/en/library/operating-system-error-codes/
[ERROR] InnoDB: File (unknown): 'close' returned OS error 209. Cannot continue operation
230420 20:18:35 [ERROR] mysqld got signal 6 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
{code}

Atfer start up

[ERROR] InnoDB: Cannot decrypt [page id: space=2890523, page number=384]

and another time:

[Warning] InnoDB: 16384 bytes should have been read at 1174798336 from ./database/table.ibd, but got only 0. Retrying.

After this the table was corrupted, can't acess, alter, can't repair table.

Our solution is on the slave :
innodb_encryption_threads = 0

We have master-salve replication with table encryption and with aws-key-management plugin

On the master :

{code:log}
innodb_encrypt_tables = Off
innodb_encrypt_log = Off
innodb_encryption_threads = 0
{code}

on the slave:
{code:log}
innodb_encrypt_tables = On
innodb_encrypt_log = Off
innodb_encryption_threads = 4
{code}

First we got this messages :
{{[Note] InnoDB: Cannot close file ./database/table.ibd because of pending fsync}}

After this the server crashed with this error message
{code:log}
[ERROR] InnoDB: Operating system error number 9 in a file operation.
[ERROR] InnoDB: Error number 9 means 'Bad file descriptor'
[Note] InnoDB: Some operating system error numbers are described at https://mariadb.com/kb/en/library/operating-system-error-codes/
[ERROR] InnoDB: File (unknown): 'close' returned OS error 209. Cannot continue operation
230420 20:18:35 [ERROR] mysqld got signal 6 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
{code}

Atfer start up

[ERROR] InnoDB: Cannot decrypt [page id: space=2890523, page number=384]

and another time:

[Warning] InnoDB: 16384 bytes should have been read at 1174798336 from ./database/table.ibd, but got only 0. Retrying.

After this the table was corrupted, can't acess, alter, can't repair table.

Our solution is on the slave (after reload databse from master) :
innodb_encryption_threads = 0

bulepage made changes - 2023-04-30 16:54

Summary

Innodb unrepairable table with encrypted table

Innodb unrepairable table with innodb_encrypt_tables

bulepage made changes - 2023-04-30 16:56

Description

We have master-salve replication with table encryption and with aws-key-management plugin

On the master :

{code:log}
innodb_encrypt_tables = Off
innodb_encrypt_log = Off
innodb_encryption_threads = 0
{code}

on the slave:
{code:log}
innodb_encrypt_tables = On
innodb_encrypt_log = Off
innodb_encryption_threads = 4
{code}

First we got this messages :
{{[Note] InnoDB: Cannot close file ./database/table.ibd because of pending fsync}}

After this the server crashed with this error message
{code:log}
[ERROR] InnoDB: Operating system error number 9 in a file operation.
[ERROR] InnoDB: Error number 9 means 'Bad file descriptor'
[Note] InnoDB: Some operating system error numbers are described at https://mariadb.com/kb/en/library/operating-system-error-codes/
[ERROR] InnoDB: File (unknown): 'close' returned OS error 209. Cannot continue operation
230420 20:18:35 [ERROR] mysqld got signal 6 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
{code}

Atfer start up

[ERROR] InnoDB: Cannot decrypt [page id: space=2890523, page number=384]

and another time:

[Warning] InnoDB: 16384 bytes should have been read at 1174798336 from ./database/table.ibd, but got only 0. Retrying.

After this the table was corrupted, can't acess, alter, can't repair table.

Our solution is on the slave (after reload databse from master) :
innodb_encryption_threads = 0

We have master-salve replication with table encryption and with aws-key-management plugin

On the master :

{code:log}
innodb_encrypt_tables = Off
innodb_encrypt_log = Off
innodb_encryption_threads = 0
{code}

on the slave:
{code:log}
innodb_encrypt_tables = On
innodb_encrypt_log = Off
innodb_encryption_threads = 4
{code}

First we got this messages :
{{[Note] InnoDB: Cannot close file ./database/table.ibd because of pending fsync}}

After this the server crashed with this error message
{code:log}
[ERROR] InnoDB: Operating system error number 9 in a file operation.
[ERROR] InnoDB: Error number 9 means 'Bad file descriptor'
[Note] InnoDB: Some operating system error numbers are described at https://mariadb.com/kb/en/library/operating-system-error-codes/
[ERROR] InnoDB: File (unknown): 'close' returned OS error 209. Cannot continue operation
230420 20:18:35 [ERROR] mysqld got signal 6 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
{code}

Atfer start up

{{[ERROR] InnoDB: Cannot decrypt [page id: space=2890523, page number=384]}}

and another time:

{{[Warning] InnoDB: 16384 bytes should have been read at 1174798336 from ./database/table.ibd, but got only 0. Retrying.}}

After this the table was corrupted, can't acess, alter, can't repair table.

Our solution is on the slave (after reload databse from master) :
{{innodb_encryption_threads = 0}}

bulepage made changes - 2023-05-04 13:25

Priority

Major [ 3 ]

Critical [ 2 ]

bulepage added a comment - 2023-05-04 13:32 - edited

After few days quiet running, again on the slave:

2023-05-02 2:05:26 7 [Note] InnoDB: Cannot close file ./database/table1.ibd because of pending fsync
2023-05-02 2:08:24 7 [Note] InnoDB: Cannot close file ./database/table1.ibd because of pending fsync
2023-05-02 2:08:32 7 [Note] InnoDB: Cannot close file ./database/table1.ibd because of pending fsync
2023-05-02 4:24:10 7 [Note] InnoDB: Cannot close file ./database/table1.ibd because of pending fsync
2023-05-02 4:31:05 7 [Note] InnoDB: Cannot close file ./database/table1.ibd because of pending fsync
2023-05-02 12:26:20 7 [Note] InnoDB: Cannot close file ./anotherdb/ReportLang.ibd because of pending fsync
2023-05-02 12:29:39 7 [Note] InnoDB: Cannot close file ./anotherdb/ReportLang.ibd because of pending fsync
2023-05-02 12:30:14 23810 [Note] InnoDB: Cannot close file ./anotherdb/ReportLang.ibd because of pending fsync

bulepage added a comment - 2023-05-04 13:32 - edited After few days quiet running, again on the slave: 2023-05-02 2:05:26 7 [Note] InnoDB: Cannot close file ./database/table1.ibd because of pending fsync 2023-05-02 2:08:24 7 [Note] InnoDB: Cannot close file ./database/table1.ibd because of pending fsync 2023-05-02 2:08:32 7 [Note] InnoDB: Cannot close file ./database/table1.ibd because of pending fsync 2023-05-02 4:24:10 7 [Note] InnoDB: Cannot close file ./database/table1.ibd because of pending fsync 2023-05-02 4:31:05 7 [Note] InnoDB: Cannot close file ./database/table1.ibd because of pending fsync 2023-05-02 12:26:20 7 [Note] InnoDB: Cannot close file ./anotherdb/ReportLang.ibd because of pending fsync 2023-05-02 12:29:39 7 [Note] InnoDB: Cannot close file ./anotherdb/ReportLang.ibd because of pending fsync 2023-05-02 12:30:14 23810 [Note] InnoDB: Cannot close file ./anotherdb/ReportLang.ibd because of pending fsync

Sergei Golubchik added a comment - 2023-06-06 10:19

A combination of

[ERROR] InnoDB: Operating system error number 9 in a file operation.

[ERROR] InnoDB: Error number 9 means 'Bad file descriptor'

...

 [ERROR] InnoDB: File (unknown): 'close' returned OS error 209. Cannot continue operation

was also observed in ~~MDEV-31347~~. Please try a fixed version (10.5.22, planned release date — end of July)

Sergei Golubchik added a comment - 2023-06-06 10:19 A combination of [ERROR] InnoDB: Operating system error number 9 in a file operation. [ERROR] InnoDB: Error number 9 means 'Bad file descriptor' ... [ERROR] InnoDB: File (unknown): 'close' returned OS error 209. Cannot continue operation was also observed in MDEV-31347 . Please try a fixed version (10.5.22, planned release date — end of July)

Sergei Golubchik made changes - 2023-06-06 10:20

Link

This issue relates to ~~MDEV-31347~~ [ ~~MDEV-31347~~ ]

Sergei Golubchik made changes - 2023-06-06 10:20

Status

Open [ 1 ]

Needs Feedback [ 10501 ]

bulepage made changes - 2023-06-10 08:22

Attachment

mysqld.err [ 70482 ]

bulepage made changes - 2023-06-10 08:22

Attachment

mysqld-1.err [ 70483 ]

bulepage made changes - 2023-06-10 08:25

Affects Version/s

10.6.14 [ 28914 ]

Elena Stepanova added a comment - 2023-07-09 15:36

As a comment above says, the possibly related ~~MDEV-31347~~ fix will be released with 10.5.22 / 10.6.15, planned release date — end of July.

Elena Stepanova added a comment - 2023-07-09 15:36 As a comment above says, the possibly related MDEV-31347 fix will be released with 10.5.22 / 10.6.15, planned release date — end of July.

Sergei Golubchik made changes - 2023-08-07 11:25

Fix Version/s		10.6 [ 24028 ]
Fix Version/s		10.9 [ 26905 ]
Fix Version/s		10.10 [ 27530 ]
Fix Version/s		10.11 [ 27614 ]
Fix Version/s		11.0 [ 28320 ]
Fix Version/s		11.1 [ 28549 ]

bulepage added a comment - 2023-08-18 08:48 - edited

After upgrade to 10.6.15 only we got notice
[Note] InnoDB: Cannot close file ./db/table.ibd because of 1 pending operations

I think the problem is solved.
Thanks

bulepage added a comment - 2023-08-18 08:48 - edited After upgrade to 10.6.15 only we got notice [Note] InnoDB: Cannot close file ./db/table.ibd because of 1 pending operations I think the problem is solved. Thanks

Sergei Golubchik made changes - 2023-08-18 09:54

Fix Version/s		11.0.3 [ 28920 ]
Fix Version/s		10.11.5 [ 29019 ]
Fix Version/s		10.10.6 [ 29017 ]
Fix Version/s		10.9.8 [ 29015 ]
Fix Version/s		10.6.15 [ 29013 ]
Fix Version/s	10.6 [ 24028 ]
Fix Version/s	10.9 [ 26905 ]
Fix Version/s	10.10 [ 27530 ]
Fix Version/s	10.11 [ 27614 ]
Fix Version/s	11.0 [ 28320 ]
Fix Version/s	11.1 [ 28549 ]

Sergei Golubchik made changes - 2023-08-18 09:55

Status

Needs Feedback [ 10501 ]

Open [ 1 ]

Sergei Golubchik made changes - 2023-08-18 09:56

Resolution		Fixed [ 1 ]
Status	Open [ 1 ]	Closed [ 6 ]

MariaDB Server

Innodb unrepairable table with innodb_encrypt_tables

Details

Description

Attachments

Attachments

Issue Links

Activity

People

Dates

Git Integration