[MDEV-10852] Server hangs after running xtrabackup with MariaDB 10.0 and innodb_flush_method = O_DIRECT Created: 2016-09-21 Updated: 2022-11-10 Resolved: 2022-11-10 |
|
| Status: | Closed |
| Project: | MariaDB Server |
| Component/s: | Storage Engine - XtraDB |
| Affects Version/s: | 10.0.26, 10.0.27, 10.0 |
| Fix Version/s: | N/A |
| Type: | Bug | Priority: | Major |
| Reporter: | Tomas Mozes | Assignee: | Jan Lindström (Inactive) |
| Resolution: | Cannot Reproduce | Votes: | 0 |
| Labels: | None | ||
| Environment: |
Server virtualized in Xen domU, 230GB RAM, Linux kernel 4.4.21, hardware raid 10 on hdd, Xtrabackup 2.4.4. |
||
| Attachments: |
|
| Description |
|
After backing up something around 15-25GB data with xtrabackup: the server starts acting strange. The xtrabackup process only prints "log scanned up to" messages with same lsn: ... MariaDB cannot be stopped nor xtrabackup can be killed. Some files cannot be read: During this time, the sever iowait is high (also load) as some files cannot be even read. Iotop shows no activity for xtrabackup (normally it reads around 200-300MB/s). There is nothing in the system logs, nor in MariaDB. This machine is running virtualized in Xen and there is nothing in dom0 logs also. A normal shutdown fails, only a forced machine shutdown works. After reboot, the files are readable normally. Under normal circumstances, all MariaDB data files can be read: After unsetting innodb_flush_method in MariaDB my.cnf the backup completes normally and system continues to work normally. $ xfs_info /var/lib/mysql/ $ find /var/lib/mysql -type f | wc -l $ du -hs /var/lib/mysql/ |
| Comments |
| Comment by Tomas Mozes [ 2016-10-28 ] |
|
I've been doing some tests to discover which part is having problems. Seems like it's xfs on kernel > 4.1: kernel 4.1 + xfs = pass kernel 4.4 + ext4 = pass |
| Comment by Tomas Mozes [ 2017-03-21 ] |
|
It also happens with ext4, but it took more time to show up. So kernel 4.4, either xfs/ext4 with O_DIRECT has this problem. On kernel 4.1 it cannot be reproduced. |
| Comment by Tomas Mozes [ 2017-10-27 ] |
|
Cannot reproduce any more on MariaDB 10.1 and Linux Kernel 4.12+. |