[MDEV-33379] innodb_log_file_buffering=OFF causes corruption on bcachefs Created: 2024-02-03 Updated: 2024-02-05 |
|
| Status: | In Progress |
| Project: | MariaDB Server |
| Component/s: | Storage Engine - InnoDB |
| Affects Version/s: | 11.2.2 |
| Fix Version/s: | 10.11, 11.0 |
| Type: | Bug | Priority: | Critical |
| Reporter: | Miroslav Rýzek | Assignee: | Marko Mäkelä |
| Resolution: | Unresolved | Votes: | 0 |
| Labels: | Compatibility | ||
| Environment: |
Archlinux, bcachefs. 2 background HDD devices, 2 foreground SSD devices. |
||
| Issue Links: |
|
||||||||||||
| Description |
|
Hello, When I installed MariaDB and try to init databases using Your assistance in bug reporting will enable us to fix this for the next release. We will try our best to scrape up some info that will hopefully help Server version: 11.2.2-MariaDB source revision: 929532a9426d085111c24c63de9c23cc54382259 Thread pointer: 0x0 Kernel version: Linux version 6.7.3-arch1-1 (linux@archlinux) (gcc (GCC) 13.2.1 20230801, GNU ld (GNU Binutils) 2.42.0) #1 SMP PREEMPT_DYNAMIC Thu, 01 Feb 2024 10:30:35 +0000 /usr/bin/mariadb-install-db: line 581: 28383 Aborted (core dumped) "$mysqld_bootstrap" $defaults $defaults_group_suffix "$mysqld_opt" - Installation of system tables failed! Examine the logs in The problem could be conflicting information in an external shell> /usr/bin/mariadb-install-db --defaults-file=~/.my.cnf You can also try to start the mariadbd daemon with: shell> /usr/bin/mariadbd --skip-grant-tables --general-log & and use the command line tool /usr/bin/mariadb shell> /usr/bin/mariadb -u root mysql Try '/usr/bin/mariadbd --help' if you have problems with paths. Using The latest information about mariadb-install-db is available at Please check all of the above before submitting a bug report I dont know whether is it mariaDB or bcachefs issue (as used filesystem). But I would like to help here so I found out some rules for working configurations that "solve" this issue. Working configuration is: `innodb_flush_log_at_trx_commit` must always be `= 2` regardless of `innodb_flush_method`. If I set 0 or 1 it just does not work also. With this configuration `mariadb-install-db` just works and I can start `mariadb.service`. I did not try real functionality of the mysql server but I expect that as I started without any errors then it could work. Could you please navigate me to find out what is wrong? Maybe it is a mariadb issue itself but maybe a filesystem issue. I would like to find out with your support what part of filesystem does not work as expected from mariadb point of view and then I can open an issue in bcachefs if you confirm it is not mariadb issue. Just for reference, my bcachefs volume is created using: |
| Comments |
| Comment by Marko Mäkelä [ 2024-02-03 ] | |||||||||||||||||||||||||||||||||||||||||||||
|
The parameter innodb_flush_method was deprecated in Can you please try differentn values of the 4 Boolean parameters:
I would expect O_DIRECT to not be supported on many copy-on-write file systems that support transparent compression. Ideally, the file system should return an error from a system call that tries to enable O_DIRECT, instead of corrupting any data. As you can read in issues linked from | |||||||||||||||||||||||||||||||||||||||||||||
| Comment by Miroslav Rýzek [ 2024-02-03 ] | |||||||||||||||||||||||||||||||||||||||||||||
|
Thank you for your quick support. These are results from testing. I used default configuration (removed innodb_flush_log_at_trx_commit and innodb_flush_method).
I opened an issue in bcachefs and will link this for reference. It looks like some issue with O_DIRECT. | |||||||||||||||||||||||||||||||||||||||||||||
| Comment by Marko Mäkelä [ 2024-02-05 ] | |||||||||||||||||||||||||||||||||||||||||||||
|
kezyr, thank you for testing this. Your table does not quite look like a truth table with 4 inputs, because there are 5 instead of 4 lines innodb_log_file_buffering=OFF. In any case, the error seems to correlate exactly with that. One more difference between the log and data files is that the ib_logfile0 is being written synchronously (we may revisit MDEV-26603 some day) while the data files are typically written asynchronously via io_uring or libaio (io_setup(), io_submit(), io_getevents()). But, it does not look like this would be a problem. After all, we are using the simpler (synchronous) API for log file I/O. I see that on Microsoft Windows, we pass the flags FILE_FLAG_NO_BUFFERING (corresponding to O_DIRECT) and FILE_FLAG_WRITE_THROUGH (corresponding to O_DSYNC or O_SYNC) to the CreateFile() call, which corresponds to open(2) in POSIX environments. Before
I think that we should be able to fix this in MariaDB Server, by removing the function os_file_set_nocache() and always trying to open the files in O_DIRECT mode when needed. We probably should fix this in 10.11 as well, because it is the newest long-term supported (LTS) release. Fixing this in older LTS (10.5 or 10.6) could require more extensive changes. |