Type:
Bug
Priority:
Major
Resolution:
Cannot Reproduce
Affects Version/s:
10.0.26 , 10.0.27 , 10.0(EOL)
Environment:
Server virtualized in Xen domU, 230GB RAM, Linux kernel 4.4.21, hardware raid 10 on hdd, Xtrabackup 2.4.4.
After backing up something around 15-25GB data with xtrabackup:
$ xtrabackup --backup --stream=xbstream --parallel=4 --compress --compress-threads=12
the server starts acting strange. The xtrabackup process only prints "log scanned up to" messages with same lsn:
...
160921 09:09:54 >> log scanned up to (7313323134056)
160921 09:09:55 >> log scanned up to (7313323134056)
160921 09:09:56 >> log scanned up to (7313323134056)
160921 09:09:57 >> log scanned up to (7313323134056)
160921 09:09:58 >> log scanned up to (7313323134056)
160921 09:09:59 >> log scanned up to (7313323134056)
...
MariaDB cannot be stopped nor xtrabackup can be killed. Some files cannot be read:
$ cat /var/lib/mysql/backup-my.cnf -> hangs.
During this time, the sever iowait is high (also load) as some files cannot be even read. Iotop shows no activity for xtrabackup (normally it reads around 200-300MB/s). There is nothing in the system logs, nor in MariaDB. This machine is running virtualized in Xen and there is nothing in dom0 logs also.
A normal shutdown fails, only a forced machine shutdown works. After reboot, the files are readable normally.
Under normal circumstances, all MariaDB data files can be read:
$ find /var/lib/mysql -type f -exec cat {} \; > /dev/null
After unsetting innodb_flush_method in MariaDB my.cnf the backup completes normally and system continues to work normally.
$ xfs_info /var/lib/mysql/
meta-data=/dev/xvda7 isize=512 agcount=23, agsize=7864256 blks
= sectsz=512 attr=2, projid32bit=1
= crc=1 finobt=1 spinodes=0
data = bsize=4096 blocks=178257920, imaxpct=25
= sunit=64 swidth=64 blks
naming =version 2 bsize=4096 ascii-ci=0 ftype=1
log =internal bsize=4096 blocks=61440, version=2
= sectsz=512 sunit=64 blks, lazy-count=1
realtime =none extsz=4096 blocks=0, rtextents=0
$ find /var/lib/mysql -type f | wc -l
624
$ du -hs /var/lib/mysql/
454G /var/lib/mysql/
{"report":{"fcp":884.9000000059605,"ttfb":182.70000000298023,"pageVisibility":"visible","entityId":58158,"key":"jira.project.issue.view-issue","isInitial":true,"threshold":1000,"elementTimings":{},"userDeviceMemory":8,"userDeviceProcessors":64,"apdex":1,"journeyId":"c18d4233-ecc2-47f1-aab5-01165b3625d4","navigationType":0,"readyForUser":963.4000000059605,"redirectCount":0,"resourceLoadedEnd":1072.7000000029802,"resourceLoadedStart":188.79999999701977,"resourceTiming":[{"duration":78.5,"initiatorType":"link","name":"https://jira.mariadb.org/s/2c21342762a6a02add1c328bed317ffd-CDN/lu2cib/820016/12ta74/0a8bac35585be7fc6c9cc5a0464cd4cf/_/download/contextbatch/css/_super/batch.css","startTime":188.79999999701977,"connectEnd":0,"connectStart":0,"domainLookupEnd":0,"domainLookupStart":0,"fetchStart":188.79999999701977,"redirectEnd":0,"redirectStart":0,"requestStart":0,"responseEnd":267.29999999701977,"responseStart":0,"secureConnectionStart":0},{"duration":170.29999999701977,"initiatorType":"link","name":"https://jira.mariadb.org/s/7ebd35e77e471bc30ff0eba799ebc151-CDN/lu2cib/820016/12ta74/494e4c556ecbb29f90a3d3b4f09cb99c/_/download/contextbatch/css/jira.browse.project,project.issue.navigator,jira.view.issue,jira.general,jira.global,atl.general,-_super/batch.css?agile_global_admin_condition=true&jag=true&jira.create.linked.issue=true&slack-enabled=true&whisper-enabled=true","startTime":189,"connectEnd":0,"connectStart":0,"domainLookupEnd":0,"domainLookupStart":0,"fetchStart":189,"redirectEnd":0,"redirectStart":0,"requestStart":0,"responseEnd":359.29999999701977,"responseStart":0,"secureConnectionStart":0},{"duration":179.79999999701977,"initiatorType":"script","name":"https://jira.mariadb.org/s/0917945aaa57108d00c5076fea35e069-CDN/lu2cib/820016/12ta74/0a8bac35585be7fc6c9cc5a0464cd4cf/_/download/contextbatch/js/_super/batch.js?locale=en","startTime":189.20000000298023,"connectEnd":189.20000000298023,"connectStart":189.20000000298023,"domainLookupEnd":189.20000000298023,"domainLookupStart":189.20000000298023,"fetchStart":189.20000000298023,"redirectEnd":0,"redirectStart":0,"requestStart":189.20000000298023,"responseEnd":369,"responseStart":369,"secureConnectionStart":189.20000000298023},{"duration":247.09999999403954,"initiatorType":"script","name":"https://jira.mariadb.org/s/2d8175ec2fa4c816e8023260bd8c1786-CDN/lu2cib/820016/12ta74/494e4c556ecbb29f90a3d3b4f09cb99c/_/download/contextbatch/js/jira.browse.project,project.issue.navigator,jira.view.issue,jira.general,jira.global,atl.general,-_super/batch.js?agile_global_admin_condition=true&jag=true&jira.create.linked.issue=true&locale=en&slack-enabled=true&whisper-enabled=true","startTime":189.40000000596046,"connectEnd":189.40000000596046,"connectStart":189.40000000596046,"domainLookupEnd":189.40000000596046,"domainLookupStart":189.40000000596046,"fetchStart":189.40000000596046,"redirectEnd":0,"redirectStart":0,"requestStart":189.40000000596046,"responseEnd":436.5,"responseStart":436.5,"secureConnectionStart":189.40000000596046},{"duration":250.20000000298023,"initiatorType":"script","name":"https://jira.mariadb.org/s/a9324d6758d385eb45c462685ad88f1d-CDN/lu2cib/820016/12ta74/c92c0caa9a024ae85b0ebdbed7fb4bd7/_/download/contextbatch/js/atl.global,-_super/batch.js?locale=en","startTime":189.70000000298023,"connectEnd":189.70000000298023,"connectStart":189.70000000298023,"domainLookupEnd":189.70000000298023,"domainLookupStart":189.70000000298023,"fetchStart":189.70000000298023,"redirectEnd":0,"redirectStart":0,"requestStart":189.70000000298023,"responseEnd":439.90000000596046,"responseStart":439.90000000596046,"secureConnectionStart":189.70000000298023},{"duration":250.79999999701977,"initiatorType":"script","name":"https://jira.mariadb.org/s/d41d8cd98f00b204e9800998ecf8427e-CDN/lu2cib/820016/12ta74/1.0/_/download/batch/jira.webresources:calendar-en/jira.webresources:calendar-en.js","startTime":189.90000000596046,"connectEnd":189.90000000596046,"connectStart":189.90000000596046,"domainLookupEnd":189.90000000596046,"domainLookupStart":189.90000000596046,"fetchStart":189.90000000596046,"redirectEnd":0,"redirectStart":0,"requestStart":189.90000000596046,"responseEnd":440.70000000298023,"responseStart":440.70000000298023,"secureConnectionStart":189.90000000596046},{"duration":251.40000000596046,"initiatorType":"script","name":"https://jira.mariadb.org/s/d41d8cd98f00b204e9800998ecf8427e-CDN/lu2cib/820016/12ta74/1.0/_/download/batch/jira.webresources:calendar-localisation-moment/jira.webresources:calendar-localisation-moment.js","startTime":190,"connectEnd":190,"connectStart":190,"domainLookupEnd":190,"domainLookupStart":190,"fetchStart":190,"redirectEnd":0,"redirectStart":0,"requestStart":190,"responseEnd":441.40000000596046,"responseStart":441.40000000596046,"secureConnectionStart":190},{"duration":322.59999999403954,"initiatorType":"link","name":"https://jira.mariadb.org/s/b04b06a02d1959df322d9cded3aeecc1-CDN/lu2cib/820016/12ta74/a2ff6aa845ffc9a1d22fe23d9ee791fc/_/download/contextbatch/css/jira.global.look-and-feel,-_super/batch.css","startTime":190.20000000298023,"connectEnd":0,"connectStart":0,"domainLookupEnd":0,"domainLookupStart":0,"fetchStart":190.20000000298023,"redirectEnd":0,"redirectStart":0,"requestStart":0,"responseEnd":512.7999999970198,"responseStart":0,"secureConnectionStart":0},{"duration":251.70000000298023,"initiatorType":"script","name":"https://jira.mariadb.org/rest/api/1.0/shortcuts/820016/47140b6e0a9bc2e4913da06536125810/shortcuts.js?context=issuenavigation&context=issueaction","startTime":190.40000000596046,"connectEnd":190.40000000596046,"connectStart":190.40000000596046,"domainLookupEnd":190.40000000596046,"domainLookupStart":190.40000000596046,"fetchStart":190.40000000596046,"redirectEnd":0,"redirectStart":0,"requestStart":190.40000000596046,"responseEnd":442.1000000089407,"responseStart":442.1000000089407,"secureConnectionStart":190.40000000596046},{"duration":322.3999999910593,"initiatorType":"link","name":"https://jira.mariadb.org/s/3ac36323ba5e4eb0af2aa7ac7211b4bb-CDN/lu2cib/820016/12ta74/d176f0986478cc64f24226b3d20c140d/_/download/contextbatch/css/com.atlassian.jira.projects.sidebar.init,-_super,-project.issue.navigator,-jira.view.issue/batch.css?jira.create.linked.issue=true","startTime":190.6000000089407,"connectEnd":0,"connectStart":0,"domainLookupEnd":0,"domainLookupStart":0,"fetchStart":190.6000000089407,"redirectEnd":0,"redirectStart":0,"requestStart":0,"responseEnd":513,"responseStart":0,"secureConnectionStart":0},{"duration":252.29999999701977,"initiatorType":"script","name":"https://jira.mariadb.org/s/5d5e8fe91fbc506585e83ea3b62ccc4b-CDN/lu2cib/820016/12ta74/d176f0986478cc64f24226b3d20c140d/_/download/contextbatch/js/com.atlassian.jira.projects.sidebar.init,-_super,-project.issue.navigator,-jira.view.issue/batch.js?jira.create.linked.issue=true&locale=en","startTime":190.70000000298023,"connectEnd":190.70000000298023,"connectStart":190.70000000298023,"domainLookupEnd":190.70000000298023,"domainLookupStart":190.70000000298023,"fetchStart":190.70000000298023,"redirectEnd":0,"redirectStart":0,"requestStart":190.70000000298023,"responseEnd":443,"responseStart":443,"secureConnectionStart":190.70000000298023},{"duration":453.40000000596046,"initiatorType":"script","name":"https://jira.mariadb.org/s/d41d8cd98f00b204e9800998ecf8427e-CDN/lu2cib/820016/12ta74/1.0/_/download/batch/jira.webresources:bigpipe-js/jira.webresources:bigpipe-js.js","startTime":191.5,"connectEnd":191.5,"connectStart":191.5,"domainLookupEnd":191.5,"domainLookupStart":191.5,"fetchStart":191.5,"redirectEnd":0,"redirectStart":0,"requestStart":191.5,"responseEnd":644.9000000059605,"responseStart":644.9000000059605,"secureConnectionStart":191.5},{"duration":881.0999999940395,"initiatorType":"script","name":"https://jira.mariadb.org/s/d41d8cd98f00b204e9800998ecf8427e-CDN/lu2cib/820016/12ta74/1.0/_/download/batch/jira.webresources:bigpipe-init/jira.webresources:bigpipe-init.js","startTime":191.6000000089407,"connectEnd":191.6000000089407,"connectStart":191.6000000089407,"domainLookupEnd":191.6000000089407,"domainLookupStart":191.6000000089407,"fetchStart":191.6000000089407,"redirectEnd":0,"redirectStart":0,"requestStart":191.6000000089407,"responseEnd":1072.7000000029802,"responseStart":1072.7000000029802,"secureConnectionStart":191.6000000089407},{"duration":116.70000000298023,"initiatorType":"xmlhttprequest","name":"https://jira.mariadb.org/rest/webResources/1.0/resources","startTime":528.7999999970198,"connectEnd":528.7999999970198,"connectStart":528.7999999970198,"domainLookupEnd":528.7999999970198,"domainLookupStart":528.7999999970198,"fetchStart":528.7999999970198,"redirectEnd":0,"redirectStart":0,"requestStart":528.7999999970198,"responseEnd":645.5,"responseStart":645.5,"secureConnectionStart":528.7999999970198}],"fetchStart":1,"domainLookupStart":1,"domainLookupEnd":1,"connectStart":1,"connectEnd":1,"requestStart":23,"responseStart":183,"responseEnd":188,"domLoading":187,"domInteractive":1094,"domContentLoadedEventStart":1094,"domContentLoadedEventEnd":1135,"domComplete":1778,"loadEventStart":1778,"loadEventEnd":1778,"userAgent":"Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; ClaudeBot/1.0; +claudebot@anthropic.com)","marks":[{"name":"bigPipe.sidebar-id.start","time":1074.7000000029802},{"name":"bigPipe.sidebar-id.end","time":1075.5},{"name":"bigPipe.activity-panel-pipe-id.start","time":1075.7000000029802},{"name":"bigPipe.activity-panel-pipe-id.end","time":1077.1000000089407},{"name":"activityTabFullyLoaded","time":1149.7000000029802}],"measures":[],"correlationId":"3e70dbc7a4c7f4","effectiveType":"4g","downlink":10,"rtt":0,"serverDuration":101,"dbReadsTimeInMs":9,"dbConnsTimeInMs":18,"applicationHash":"9d11dbea5f4be3d4cc21f03a88dd11d8c8687422","experiments":[]}}
I've been doing some tests to discover which part is having problems. Seems like it's xfs on kernel > 4.1:
kernel 4.1 + xfs = pass
kernel 4.4 + ext4 = pass
kernel 4.4 + xfs - O_DIRECT = pass
kernel 4.4 + xfs + O_DIRECT = FAIL