[MDEV-21584] Linux aio returned OS error 22 Created: 2020-01-28 Updated: 2021-04-19 Resolved: 2020-10-14 |
|
| Status: | Closed |
| Project: | MariaDB Server |
| Component/s: | Storage Engine - InnoDB |
| Affects Version/s: | 10.4.10 |
| Fix Version/s: | 10.2.35, 10.3.26, 10.4.16, 10.5.7 |
| Type: | Bug | Priority: | Blocker |
| Reporter: | Eugene | Assignee: | Eugene Kosov (Inactive) |
| Resolution: | Fixed | Votes: | 1 |
| Labels: | innodb | ||
| Environment: |
Linux 4.19.98-gentoo x86_64 AMD EPYC 7451, XFS on software RAID1 of two NVMe drives |
||
| Issue Links: |
|
||||||||
| Description |
|
Server has following configuration:
More intensively it's used ("intensive" means 20-30 select queries in processlist, 1-2 insert queries updating multiple rows), more frequent following situation is:
and corresponding system message:
This happens once in several hours. However, this is an issue as node is a part of galera cluster and crash renders into snapshot transfer to failed node. Changing configuration to
"solves" stability issue with setting flush method to default fsync() one. Expected behavior would be stable mariadb operation with O_DIRECT. |
| Comments |
| Comment by Andrew Sun [ 2020-03-04 ] |
|
I have reproduced this crash on Linux 5.5.4, 2x Xeon X5670, XFS on Intel DC P4600 NVMe SSD with MariaDB 10.4.12. |
| Comment by Eugene Kosov (Inactive) [ 2020-07-09 ] |
|
Hi. We're aware of that bug but we weren't able to reproduce it. And because of that we can't fix it still. Can you help us with reproduction? |
| Comment by Eugene Kosov (Inactive) [ 2020-08-17 ] |
|
Please, provide us with such a data. Run this /sbin/blockdev --getss /dev/nvme0n1p2 on a disk device where InnoDB data is located. |
| Comment by Andrew Sun [ 2020-08-17 ] |
|
@kevg blockdev --getss /dev/nvme0n1p1 returns 4096 |
| Comment by Eugene Kosov (Inactive) [ 2020-10-08 ] |
|
InnoDB was written with the assumption that block size is always 512. And this won't be fixed in GA versions. We know that ROW_FORMAT=COMPRESSED definitely doesn't work with block size of 4096. The fix in 10.2 is do disable O_DIRECT. In the best case it will be disabled partially, in the worst case it will be disabled completely. |
| Comment by Nick (Inactive) [ 2020-10-08 ] |
|
kevg Does this mean customers should use a different innodb_flush_method ? If yes, do we have one that's recommended? Is this something that will be fixed eventually in a later release? |
| Comment by Marko Mäkelä [ 2020-10-14 ] |
|
The simple solution of disabling O_DIRECT on page_compressed tables, or on ROW_FORMAT=COMPRESSED tables with a page size smaller than 4KiB, seems to work fine. I tested it on my SSD (512-byte block size) and an instrumentation patch by kevg that would complain about not-4KiB-aligned writes on O_DIRECT files. |
| Comment by Marko Mäkelä [ 2021-03-17 ] |
|
As noted in |
| Comment by Vladislav Vaintroub [ 2021-03-17 ] |
|
Also, there was no OS_DATA_FILE_NO_O_DIRECT on Windows, eventhough alignment, offset, and size requirements are well documented on Windows (for at least 20 years if not more) |