[MDEV-33140] transaction failed and "flush logs" statement failed. Could not use for logging (error 2) Created: 2023-12-29  Updated: 2024-01-09  Resolved: 2024-01-03

Status: Closed
Project: MariaDB Server
Component/s: Replication
Affects Version/s: None
Fix Version/s: N/A

Type: Bug Priority: Major
Reporter: R S N Murthy Assignee: Roel Van de Paar
Resolution: Not a Bug Votes: 0
Labels: None
Environment:

OS : Red Hat Enterprise Linux 8.3, @@version: 10.5.9-MariaDB-log, Linux duuscn1ahxunwcadbs1001 4.18.0-513.9.1.el8_9.x86_64 #1 SMP Thu Nov 16 10:29:04 EST 2023 x86_64 x86_64 x86_64 GNU/Linux

Attachments: File duuscn1ahxunwcadbs1001.err    

 Description   

The application transaction failed with below error 

*Caused by: java.sql.SQLTransientConnectionException: (conn=173325) Error writing file '/mariadb-binlog/hxun' (errno: 11 "Resource temporarily unavailable")*
 
20231229 191909.014 [] ERROR : .DatabaseHelperService-exception while saving  {}
 
org.springframework.orm.jpa.JpaSystemException: could not execute batch; nested exception is org.hibernate.exception.GenericJDBCException: could not execute batch
 
*Caused by: java.sql.BatchUpdateException: (conn=173323) Writing one row to the row-based binary log failed

The DB has enough free space to handle the transaction, but still it was not proceeding with transaciton.

When tryied flusing the logs it throwed error. Noticed Below in mariadb error log file

[unicauser@duuscn1ahxunwcadbs1001 ~]$ tail -f /mariadb/data/duuscn1ahxunwcadbs1001.err
 
2023-12-29 11:32:49 173937 [ERROR] Could not use /mariadb/mariadb-binlog/hxun for logging (error 2). Turning logging off for the whole duration of the MariaDB server process. To turn it on again: fix the cause, shutdown the MariaDB server and restart it.
 
2023-12-29 11:32:49 173937 [ERROR] Could not use /mariadb/mariadb-binlog/hxun.000369 for logging (error 0). Turning logging off for the whole duration of the MariaDB server process. To turn it on again: fix the cause, shutdown the MariaDB server and restart it.
 
2023-12-29 11:48:13 174188 [Warning] Access denied for user '' (using password: YES)



 Comments   
Comment by Roel Van de Paar [ 2024-01-03 ]

MurthyRamachandrula Hi! 

$ ./bin/perror 2
 
OS error code   2:  No such file or directory

The application error shows:

/mariadb-binlog/hxun' (errno: 11 "Resource temporarily unavailable")

The error log in the original description shows:

Could not use /mariadb/mariadb-binlog/hxun for logging

And the uploaded (and edited) error log shows:

Could not use /data/XXXX/mariadb/mariadb-binlog/hxun for logging (error 2)

Which are all inconsistent with each other.

It looks like a directory change/disk resource management mishap has happened, and the subsequently executed log flush (likely at 2023-12-29 11:32:49) then triggered the issue (and/or OS file level disconnect, i.e. if they were in use and moved, Linux would keep the link open).

To turn logging back on again please fix the cause, ensure logs are available and configured correctly, shutdown the MariaDB server and then restart it.

Comment by Roel Van de Paar [ 2024-01-03 ]

MurthyRamachandrula Btw, there are apparently some of your internal IP addresses left in the uploaded error log; you may want to remove it from this ticket.

Comment by Roel Van de Paar [ 2024-01-03 ]

MurthyRamachandrula Btw, https://mariadb.com/services/technical-support-services/ may be of interest if you would like to explore Enterprise Database Technical Support Services. All the best for 2024 also.

Comment by R S N Murthy [ 2024-01-04 ]

Thank you for Wishes and Wish you a Happy New Year.

We did a bounce on mariadb service on the same day, it came up and is healthy.

I came accross https://jira.mariadb.org/browse/MDEV-24625 that has a similar issue. The Database is enabled with "Data Encryption At REST and "Data Encryption in Transit" . The configuration was done by following the document
https://mariadb.com/kb/en/file-key-management-encryption-plugin/

MariaDB [(none)]> show global variables like '%encry%';
-------------------------------------------------+

Variable_name Value

-------------------------------------------------+

aria_encrypt_tables ON
encrypt_binlog ON
encrypt_tmp_disk_tables ON
encrypt_tmp_files ON
file_key_management_encryption_algorithm aes_cbc
innodb_default_encryption_key_id 1
innodb_encrypt_log ON
innodb_encrypt_tables FORCE
innodb_encrypt_temporary_tables ON
innodb_encryption_rotate_key_age 0
innodb_encryption_rotation_iops 2000
innodb_encryption_threads 4

-------------------------------------------------+
12 rows in set (0.002 sec)

MariaDB [(none)]> show global variables like '%ssl%';
------------------------------------------------+

Variable_name Value

------------------------------------------------+

have_openssl NO
have_ssl YES
ssl_ca /data/HCL/Certs/ca.cert
ssl_capath  
ssl_cert /data/HCL/Certs/server.cert
ssl_cipher  
ssl_crl  
ssl_crlpath  
ssl_key /data/HCL/Certs/server.key
version_ssl_library WolfSSL 4.6.0

------------------------------------------------+
10 rows in set (0.001 sec)

Need guidence on how to avoid the issues. please guide

Comment by Roel Van de Paar [ 2024-01-05 ]

MurthyRamachandrula Hi again. MDEV-24625 Does not look related to the issue you observed as in your case the directory path seems to have been changed.

Comment by R S N Murthy [ 2024-01-09 ]

Please help in understanding a situation. Does OS patching updates will make any difference to encryption since we are using OpenSSL to setup Data Encryption.

Generated at Thu Feb 08 10:36:41 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.