mysqltest: In included file "./include/sync_with_master_gtid.inc":
included from /usr/local/share/mysql-test/suite/rpl/t/rpl_xa.inc at line 350:
included from /usr/local/share/mysql-test/suite/rpl/t/rpl_xa.test at line 4:
At line 48: Failed to sync with master
The result from queries just before the failure was:
< snip >
connection master;
xa recover;
formatID gtrid_length bqual_length data
1 11 0 skip_binlog
set @@session.sql_log_bin = OFF;
xa rollback 'skip_binlog';
set @@session.sql_log_bin = ON;
include/save_master_gtid.inc
*** Zero must be in the list:
connection master;
xa recover;
formatID gtrid_length bqual_length data
*** At the end of skip_log_binb section gtid list has 0 more:
flush logs;
show binlog events in 'master-bin.000007' limit 1,1;
Log_name Pos Event_type Server_id End_log_pos Info
master-bin.000007 # Gtid_list 1 # [0-1-25]
connection slave;
include/sync_with_master_gtid.inc
Timeout in master_gtid_wait('0-1-25', 120), current slave GTID position is: 0-1-11.
The replica error log shows:
2022-06-28 18:56:39 7 [ERROR] Error reading packet from server: bogus data in log event; the first event 'master-bin.000001' at 4, the last event read from 'master-bin.000002' at 343, the last byte read from 'master-bin.000002' at 362. (server_errno=1236)
2022-06-28 18:56:39 7 [ERROR] Slave I/O: Got fatal error 1236 from master when reading data from binary log: 'bogus data in log event; the first event 'master-bin.000001' at 4, the last event read from 'master-bin.000002' at 343, the last byte read from 'master-bin.000002' at 362.', Internal MariaDB error code: 1236
2022-06-28 18:56:39 7 [Note] Slave I/O thread exiting, read up to log 'master-bin.000002', position 343, master 127.0.0.1:16020
Here we clearly see that the dump thread is reading the file in parallel with the binlog background thread writing the second binlog checkpoint. The read sees the length as if the second write had completed, but it does not see the correct data, it reads all zeros.
So this is not a bug in MariaDB, it's a problem with FreeBSD.
We can solve this by not using tmpfs for /tmp/var on the FreeBSD 13.0 builder.
Kristian Nielsen
added a comment - I reproduced the failure using FreeBSD ktrace. This shows that the FreeBSD kernel is returning incorrect data:
# File open for write:
33358 898673 mariadbd 8.128300 CALL openat(AT_FDCWD,0x1842d40,0x100201<O_WRONLY|O_CREAT|O_CLOEXEC>,0660<S_IRUSR|S_IWUSR|S_IRGRP|S_IWGRP>)
33358 898673 mariadbd 8.128324 NAMI "./master-bin.000002"
33358 898673 mariadbd 8.128358 RET openat 84/0x54
# File write first part:
33358 898673 mariadbd 8.128619 CALL write(0x54,0x808cf03d8,0x157)
33358 898673 mariadbd 8.128651 GIO fd 84 wrote 343 bytes
0x0000 fe62 696e f23e bc64 0f01 0000 00fc 0000 |.bin.>.d........|
0x0010 0000 0100 0001 0004 0031 312e 322e 302d |.........11.2.0-|
0x0020 4d61 7269 6144 422d 6c6f 6700 0000 0000 |MariaDB-log.....|
0x0030 0000 0000 0000 0000 0000 0000 0000 0000 |................|
0x0040 0000 0000 0000 0000 0000 0000 0000 0013 |................|
0x0050 380d 0008 0012 0004 0404 0412 0000 e400 |8...............|
0x0060 041a 0800 0000 0808 0802 0000 000a 0a0a |................|
0x0070 0000 0000 0000 0000 0000 0000 0000 0000 |................|
0x0080 0000 0000 0000 0000 0000 0000 0000 0000 |................|
0x0090 0000 0000 0000 0000 0000 0000 0000 0000 |................|
0x00a0 0000 0000 0000 0000 0000 0000 0000 0000 |................|
0x00b0 0000 0000 0000 0000 0000 0000 0000 0000 |................|
0x00c0 0000 0000 0000 0000 0000 0000 0000 0000 |................|
0x00d0 0000 0000 0000 0000 0000 0000 0000 0000 |................|
0x00e0 0000 0000 0000 0000 0000 0000 0000 0000 |................|
0x00f0 0413 0400 0d08 0808 0a0a 0a01 f184 ad09 |................|
0x0100 f23e bc64 a301 0000 002b 0000 002b 0100 |.>.d.....+...+..|
0x0110 0000 0001 0000 0000 0000 0001 0000 0007 |................|
0x0120 0000 0000 0000 004b c30b 8ff2 3ebc 64a1 |.......K....>.d.|
0x0130 0100 0000 2c00 0000 5701 0000 0000 1100 |....,...W.......|
0x0140 0000 6d61 7374 6572 2d62 696e 2e30 3030 |..master-bin.000|
0x0150 3030 31d1 53cd ff |001.S..|
33358 898673 mariadbd 8.128848 RET write 343/0x157
# File open for read:
33358 898672 mariadbd 8.130722 CALL openat(AT_FDCWD,0x7fffdec20010,0x100000<O_RDONLY|O_CLOEXEC>)
33358 898672 mariadbd 8.130747 NAMI "./master-bin.000002"
33358 898672 mariadbd 8.130772 RET openat 23/0x17
# Write write second part starts:
33358 898649 mariadbd 8.130846 CALL write(0x54,0x808cf03d8,0x2c)
# Read of file starts concurrently:
33358 898672 mariadbd 8.131108 CALL read(0x17,0x807e31558,0x4000)
33358 898672 mariadbd 8.131192 GIO fd 23 read 387 bytes
0x0000 fe62 696e f23e bc64 0f01 0000 00fc 0000 |.bin.>.d........|
0x0010 0000 0100 0001 0004 0031 312e 322e 302d |.........11.2.0-|
0x0020 4d61 7269 6144 422d 6c6f 6700 0000 0000 |MariaDB-log.....|
0x0030 0000 0000 0000 0000 0000 0000 0000 0000 |................|
0x0040 0000 0000 0000 0000 0000 0000 0000 0013 |................|
0x0050 380d 0008 0012 0004 0404 0412 0000 e400 |8...............|
0x0060 041a 0800 0000 0808 0802 0000 000a 0a0a |................|
0x0070 0000 0000 0000 0000 0000 0000 0000 0000 |................|
0x0080 0000 0000 0000 0000 0000 0000 0000 0000 |................|
0x0090 0000 0000 0000 0000 0000 0000 0000 0000 |................|
0x00a0 0000 0000 0000 0000 0000 0000 0000 0000 |................|
0x00b0 0000 0000 0000 0000 0000 0000 0000 0000 |................|
0x00c0 0000 0000 0000 0000 0000 0000 0000 0000 |................|
0x00d0 0000 0000 0000 0000 0000 0000 0000 0000 |................|
0x00e0 0000 0000 0000 0000 0000 0000 0000 0000 |................|
0x00f0 0413 0400 0d08 0808 0a0a 0a01 f184 ad09 |................|
0x0100 f23e bc64 a301 0000 002b 0000 002b 0100 |.>.d.....+...+..|
0x0110 0000 0001 0000 0000 0000 0001 0000 0007 |................|
0x0120 0000 0000 0000 004b c30b 8ff2 3ebc 64a1 |.......K....>.d.|
0x0130 0100 0000 2c00 0000 5701 0000 0000 1100 |....,...W.......|
0x0140 0000 6d61 7374 6572 2d62 696e 2e30 3030 |..master-bin.000|
0x0150 3030 31d1 53cd ff00 0000 0000 0000 0000 |001.S...........|
0x0160 0000 0000 0000 0000 0000 0000 0000 0000 |................|
0x0170 0000 0000 0000 0000 0000 0000 0000 0000 |................|
0x0180 0000 00 |...|
33358 898672 mariadbd 8.131301 RET read 387/0x183
# Second write completes. The read sees the length of the file including the
# second write, but the data is missing, all zeros.
33358 898649 mariadbd 8.131540 GIO fd 84 wrote 44 bytes
0x0000 f23e bc64 a101 0000 002c 0000 0083 0100 |.>.d.....,......|
0x0010 0000 0011 0000 006d 6173 7465 722d 6269 |.......master-bi|
0x0020 6e2e 3030 3030 3032 0539 6465 |n.000002.9de|
33358 898649 mariadbd 8.131596 RET write 44/0x2c
Here we clearly see that the dump thread is reading the file in parallel with the binlog background thread writing the second binlog checkpoint. The read sees the length as if the second write had completed, but it does not see the correct data, it reads all zeros.
So this is not a bug in MariaDB, it's a problem with FreeBSD.
We can solve this by not using tmpfs for /tmp/var on the FreeBSD 13.0 builder.
Kristian Nielsen
added a comment - Fix for buildbot: https://github.com/MariaDB/mariadb.org-tools/pull/176
elenst are you the one who can update the old buildbot to include this fix?
It should be effective in buildbot starting from build 4870 on FreeBSD 130 bintar builder.
If it doesn't help, we'll merge the pull request above.
Elena Stepanova
added a comment - As discussed elsewhere, we are going to try their suggested workaround first,
sudo sysctl debug.vn_io_pgcache_read_enable=0
It should be effective in buildbot starting from build 4870 on FreeBSD 130 bintar builder.
If it doesn't help, we'll merge the pull request above.
Kristian Nielsen
added a comment - Fixed by using a work-around in buildbot for the FreeBSD bug:
https://github.com/MariaDB/mariadb.org-tools/commit/ecbc909d5bdd0838be28ca61014857fadd70cd67
People
Kristian Nielsen
Angelique Sklavounos (Inactive)
Votes:
0Vote for this issue
Watchers:
5Start watching this issue
Dates
Created:
Updated:
Resolved:
Git Integration
Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.
{"report":{"fcp":924.7999997138977,"ttfb":315.2999997138977,"pageVisibility":"visible","entityId":112377,"key":"jira.project.issue.view-issue","isInitial":true,"threshold":1000,"elementTimings":{},"userDeviceMemory":8,"userDeviceProcessors":64,"apdex":0.5,"journeyId":"a1b512ce-2c0a-4ac8-92bf-86c1bc73674c","navigationType":0,"readyForUser":1023.1999998092651,"redirectCount":0,"resourceLoadedEnd":1206.6999998092651,"resourceLoadedStart":320.5,"resourceTiming":[{"duration":134.40000009536743,"initiatorType":"link","name":"https://jira.mariadb.org/s/2c21342762a6a02add1c328bed317ffd-CDN/lu2cib/820016/12ta74/0a8bac35585be7fc6c9cc5a0464cd4cf/_/download/contextbatch/css/_super/batch.css","startTime":320.5,"connectEnd":0,"connectStart":0,"domainLookupEnd":0,"domainLookupStart":0,"fetchStart":320.5,"redirectEnd":0,"redirectStart":0,"requestStart":0,"responseEnd":454.90000009536743,"responseStart":0,"secureConnectionStart":0},{"duration":134.2999997138977,"initiatorType":"link","name":"https://jira.mariadb.org/s/7ebd35e77e471bc30ff0eba799ebc151-CDN/lu2cib/820016/12ta74/2bf333562ca6724060a9d5f1535471f6/_/download/contextbatch/css/jira.browse.project,project.issue.navigator,jira.view.issue,jira.general,jira.global,atl.general,-_super/batch.css?agile_global_admin_condition=true&jag=true&jira.create.linked.issue=true&slack-enabled=true","startTime":320.90000009536743,"connectEnd":0,"connectStart":0,"domainLookupEnd":0,"domainLookupStart":0,"fetchStart":320.90000009536743,"redirectEnd":0,"redirectStart":0,"requestStart":0,"responseEnd":455.19999980926514,"responseStart":0,"secureConnectionStart":0},{"duration":143.09999990463257,"initiatorType":"script","name":"https://jira.mariadb.org/s/0917945aaa57108d00c5076fea35e069-CDN/lu2cib/820016/12ta74/0a8bac35585be7fc6c9cc5a0464cd4cf/_/download/contextbatch/js/_super/batch.js?locale=en","startTime":321,"connectEnd":321,"connectStart":321,"domainLookupEnd":321,"domainLookupStart":321,"fetchStart":321,"redirectEnd":0,"redirectStart":0,"requestStart":321,"responseEnd":464.09999990463257,"responseStart":464.09999990463257,"secureConnectionStart":321},{"duration":214.59999990463257,"initiatorType":"script","name":"https://jira.mariadb.org/s/2d8175ec2fa4c816e8023260bd8c1786-CDN/lu2cib/820016/12ta74/2bf333562ca6724060a9d5f1535471f6/_/download/contextbatch/js/jira.browse.project,project.issue.navigator,jira.view.issue,jira.general,jira.global,atl.general,-_super/batch.js?agile_global_admin_condition=true&jag=true&jira.create.linked.issue=true&locale=en&slack-enabled=true","startTime":321.19999980926514,"connectEnd":321.19999980926514,"connectStart":321.19999980926514,"domainLookupEnd":321.19999980926514,"domainLookupStart":321.19999980926514,"fetchStart":321.19999980926514,"redirectEnd":0,"redirectStart":0,"requestStart":321.19999980926514,"responseEnd":535.7999997138977,"responseStart":535.7999997138977,"secureConnectionStart":321.19999980926514},{"duration":218,"initiatorType":"script","name":"https://jira.mariadb.org/s/a9324d6758d385eb45c462685ad88f1d-CDN/lu2cib/820016/12ta74/c92c0caa9a024ae85b0ebdbed7fb4bd7/_/download/contextbatch/js/atl.global,-_super/batch.js?locale=en","startTime":321.40000009536743,"connectEnd":321.40000009536743,"connectStart":321.40000009536743,"domainLookupEnd":321.40000009536743,"domainLookupStart":321.40000009536743,"fetchStart":321.40000009536743,"redirectEnd":0,"redirectStart":0,"requestStart":321.40000009536743,"responseEnd":539.4000000953674,"responseStart":539.4000000953674,"secureConnectionStart":321.40000009536743},{"duration":218.30000019073486,"initiatorType":"script","name":"https://jira.mariadb.org/s/d41d8cd98f00b204e9800998ecf8427e-CDN/lu2cib/820016/12ta74/1.0/_/download/batch/jira.webresources:calendar-en/jira.webresources:calendar-en.js","startTime":321.59999990463257,"connectEnd":321.59999990463257,"connectStart":321.59999990463257,"domainLookupEnd":321.59999990463257,"domainLookupStart":321.59999990463257,"fetchStart":321.59999990463257,"redirectEnd":0,"redirectStart":0,"requestStart":321.59999990463257,"responseEnd":539.9000000953674,"responseStart":539.9000000953674,"secureConnectionStart":321.59999990463257},{"duration":218.5,"initiatorType":"script","name":"https://jira.mariadb.org/s/d41d8cd98f00b204e9800998ecf8427e-CDN/lu2cib/820016/12ta74/1.0/_/download/batch/jira.webresources:calendar-localisation-moment/jira.webresources:calendar-localisation-moment.js","startTime":321.7999997138977,"connectEnd":321.7999997138977,"connectStart":321.7999997138977,"domainLookupEnd":321.7999997138977,"domainLookupStart":321.7999997138977,"fetchStart":321.7999997138977,"redirectEnd":0,"redirectStart":0,"requestStart":321.7999997138977,"responseEnd":540.2999997138977,"responseStart":540.2999997138977,"secureConnectionStart":321.7999997138977},{"duration":285.2999997138977,"initiatorType":"link","name":"https://jira.mariadb.org/s/b04b06a02d1959df322d9cded3aeecc1-CDN/lu2cib/820016/12ta74/a2ff6aa845ffc9a1d22fe23d9ee791fc/_/download/contextbatch/css/jira.global.look-and-feel,-_super/batch.css","startTime":322,"connectEnd":0,"connectStart":0,"domainLookupEnd":0,"domainLookupStart":0,"fetchStart":322,"redirectEnd":0,"redirectStart":0,"requestStart":0,"responseEnd":607.2999997138977,"responseStart":0,"secureConnectionStart":0},{"duration":218.7999997138977,"initiatorType":"script","name":"https://jira.mariadb.org/rest/api/1.0/shortcuts/820016/47140b6e0a9bc2e4913da06536125810/shortcuts.js?context=issuenavigation&context=issueaction","startTime":322,"connectEnd":322,"connectStart":322,"domainLookupEnd":322,"domainLookupStart":322,"fetchStart":322,"redirectEnd":0,"redirectStart":0,"requestStart":322,"responseEnd":540.7999997138977,"responseStart":540.7999997138977,"secureConnectionStart":322},{"duration":285.30000019073486,"initiatorType":"link","name":"https://jira.mariadb.org/s/3ac36323ba5e4eb0af2aa7ac7211b4bb-CDN/lu2cib/820016/12ta74/d176f0986478cc64f24226b3d20c140d/_/download/contextbatch/css/com.atlassian.jira.projects.sidebar.init,-_super,-project.issue.navigator,-jira.view.issue/batch.css?jira.create.linked.issue=true","startTime":322.19999980926514,"connectEnd":0,"connectStart":0,"domainLookupEnd":0,"domainLookupStart":0,"fetchStart":322.19999980926514,"redirectEnd":0,"redirectStart":0,"requestStart":0,"responseEnd":607.5,"responseStart":0,"secureConnectionStart":0},{"duration":218.89999961853027,"initiatorType":"script","name":"https://jira.mariadb.org/s/5d5e8fe91fbc506585e83ea3b62ccc4b-CDN/lu2cib/820016/12ta74/d176f0986478cc64f24226b3d20c140d/_/download/contextbatch/js/com.atlassian.jira.projects.sidebar.init,-_super,-project.issue.navigator,-jira.view.issue/batch.js?jira.create.linked.issue=true&locale=en","startTime":322.40000009536743,"connectEnd":322.40000009536743,"connectStart":322.40000009536743,"domainLookupEnd":322.40000009536743,"domainLookupStart":322.40000009536743,"fetchStart":322.40000009536743,"redirectEnd":0,"redirectStart":0,"requestStart":322.40000009536743,"responseEnd":541.2999997138977,"responseStart":541.2999997138977,"secureConnectionStart":322.40000009536743},{"duration":877.7999997138977,"initiatorType":"script","name":"https://jira.mariadb.org/s/d41d8cd98f00b204e9800998ecf8427e-CDN/lu2cib/820016/12ta74/1.0/_/download/batch/jira.webresources:bigpipe-js/jira.webresources:bigpipe-js.js","startTime":328.40000009536743,"connectEnd":328.40000009536743,"connectStart":328.40000009536743,"domainLookupEnd":328.40000009536743,"domainLookupStart":328.40000009536743,"fetchStart":328.40000009536743,"redirectEnd":0,"redirectStart":0,"requestStart":328.40000009536743,"responseEnd":1206.1999998092651,"responseStart":1206.1999998092651,"secureConnectionStart":328.40000009536743},{"duration":878.1999998092651,"initiatorType":"script","name":"https://jira.mariadb.org/s/d41d8cd98f00b204e9800998ecf8427e-CDN/lu2cib/820016/12ta74/1.0/_/download/batch/jira.webresources:bigpipe-init/jira.webresources:bigpipe-init.js","startTime":328.5,"connectEnd":328.5,"connectStart":328.5,"domainLookupEnd":328.5,"domainLookupStart":328.5,"fetchStart":328.5,"redirectEnd":0,"redirectStart":0,"requestStart":328.5,"responseEnd":1206.6999998092651,"responseStart":1206.6999998092651,"secureConnectionStart":328.5},{"duration":413.7000002861023,"initiatorType":"xmlhttprequest","name":"https://jira.mariadb.org/rest/webResources/1.0/resources","startTime":634.1999998092651,"connectEnd":634.1999998092651,"connectStart":634.1999998092651,"domainLookupEnd":634.1999998092651,"domainLookupStart":634.1999998092651,"fetchStart":634.1999998092651,"redirectEnd":0,"redirectStart":0,"requestStart":634.1999998092651,"responseEnd":1047.9000000953674,"responseStart":1047.9000000953674,"secureConnectionStart":634.1999998092651}],"fetchStart":0,"domainLookupStart":0,"domainLookupEnd":0,"connectStart":0,"connectEnd":0,"requestStart":140,"responseStart":316,"responseEnd":324,"domLoading":319,"domInteractive":1241,"domContentLoadedEventStart":1241,"domContentLoadedEventEnd":1290,"domComplete":1918,"loadEventStart":1918,"loadEventEnd":1919,"userAgent":"Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; ClaudeBot/1.0; +claudebot@anthropic.com)","marks":[{"name":"bigPipe.sidebar-id.start","time":1212.9000000953674},{"name":"bigPipe.sidebar-id.end","time":1213.6999998092651},{"name":"bigPipe.activity-panel-pipe-id.start","time":1213.7999997138977},{"name":"bigPipe.activity-panel-pipe-id.end","time":1217.1999998092651},{"name":"activityTabFullyLoaded","time":1305.4000000953674}],"measures":[],"correlationId":"a88bd099f355c1","effectiveType":"4g","downlink":10,"rtt":0,"serverDuration":102,"dbReadsTimeInMs":9,"dbConnsTimeInMs":16,"applicationHash":"9d11dbea5f4be3d4cc21f03a88dd11d8c8687422","experiments":[]}}
Bug
Major
I reproduced the failure using FreeBSD ktrace. This shows that the FreeBSD kernel is returning incorrect data:
# File open for write:33358 898673 mariadbd 8.128300 CALL openat(AT_FDCWD,0x1842d40,0x100201<O_WRONLY|O_CREAT|O_CLOEXEC>,0660<S_IRUSR|S_IWUSR|S_IRGRP|S_IWGRP>)33358 898673 mariadbd 8.128324 NAMI "./master-bin.000002"33358 898673 mariadbd 8.128358 RET openat 84/0x54# File write first part:33358 898673 mariadbd 8.128619 CALL write(0x54,0x808cf03d8,0x157)33358 898673 mariadbd 8.128651 GIO fd 84 wrote 343 bytes0x0000 fe62 696e f23e bc64 0f01 0000 00fc 0000 |.bin.>.d........|0x0010 0000 0100 0001 0004 0031 312e 322e 302d |.........11.2.0-|0x0020 4d61 7269 6144 422d 6c6f 6700 0000 0000 |MariaDB-log.....|0x0030 0000 0000 0000 0000 0000 0000 0000 0000 |................|0x0040 0000 0000 0000 0000 0000 0000 0000 0013 |................|0x0050 380d 0008 0012 0004 0404 0412 0000 e400 |8...............|0x0060 041a 0800 0000 0808 0802 0000 000a 0a0a |................|0x0070 0000 0000 0000 0000 0000 0000 0000 0000 |................|0x0080 0000 0000 0000 0000 0000 0000 0000 0000 |................|0x0090 0000 0000 0000 0000 0000 0000 0000 0000 |................|0x00a0 0000 0000 0000 0000 0000 0000 0000 0000 |................|0x00b0 0000 0000 0000 0000 0000 0000 0000 0000 |................|0x00c0 0000 0000 0000 0000 0000 0000 0000 0000 |................|0x00d0 0000 0000 0000 0000 0000 0000 0000 0000 |................|0x00e0 0000 0000 0000 0000 0000 0000 0000 0000 |................|0x00f0 0413 0400 0d08 0808 0a0a 0a01 f184 ad09 |................|0x0100 f23e bc64 a301 0000 002b 0000 002b 0100 |.>.d.....+...+..|0x0110 0000 0001 0000 0000 0000 0001 0000 0007 |................|0x0120 0000 0000 0000 004b c30b 8ff2 3ebc 64a1 |.......K....>.d.|0x0130 0100 0000 2c00 0000 5701 0000 0000 1100 |....,...W.......|0x0140 0000 6d61 7374 6572 2d62 696e 2e30 3030 |..master-bin.000|0x0150 3030 31d1 53cd ff |001.S..|33358 898673 mariadbd 8.128848 RET write 343/0x157# File open for read:33358 898672 mariadbd 8.130722 CALL openat(AT_FDCWD,0x7fffdec20010,0x100000<O_RDONLY|O_CLOEXEC>)33358 898672 mariadbd 8.130747 NAMI "./master-bin.000002"33358 898672 mariadbd 8.130772 RET openat 23/0x17# Write write second part starts:33358 898649 mariadbd 8.130846 CALL write(0x54,0x808cf03d8,0x2c)# Read of file starts concurrently:33358 898672 mariadbd 8.131108 CALL read(0x17,0x807e31558,0x4000)33358 898672 mariadbd 8.131192 GIO fd 23 read 387 bytes0x0000 fe62 696e f23e bc64 0f01 0000 00fc 0000 |.bin.>.d........|0x0010 0000 0100 0001 0004 0031 312e 322e 302d |.........11.2.0-|0x0020 4d61 7269 6144 422d 6c6f 6700 0000 0000 |MariaDB-log.....|0x0030 0000 0000 0000 0000 0000 0000 0000 0000 |................|0x0040 0000 0000 0000 0000 0000 0000 0000 0013 |................|0x0050 380d 0008 0012 0004 0404 0412 0000 e400 |8...............|0x0060 041a 0800 0000 0808 0802 0000 000a 0a0a |................|0x0070 0000 0000 0000 0000 0000 0000 0000 0000 |................|0x0080 0000 0000 0000 0000 0000 0000 0000 0000 |................|0x0090 0000 0000 0000 0000 0000 0000 0000 0000 |................|0x00a0 0000 0000 0000 0000 0000 0000 0000 0000 |................|0x00b0 0000 0000 0000 0000 0000 0000 0000 0000 |................|0x00c0 0000 0000 0000 0000 0000 0000 0000 0000 |................|0x00d0 0000 0000 0000 0000 0000 0000 0000 0000 |................|0x00e0 0000 0000 0000 0000 0000 0000 0000 0000 |................|0x00f0 0413 0400 0d08 0808 0a0a 0a01 f184 ad09 |................|0x0100 f23e bc64 a301 0000 002b 0000 002b 0100 |.>.d.....+...+..|0x0110 0000 0001 0000 0000 0000 0001 0000 0007 |................|0x0120 0000 0000 0000 004b c30b 8ff2 3ebc 64a1 |.......K....>.d.|0x0130 0100 0000 2c00 0000 5701 0000 0000 1100 |....,...W.......|0x0140 0000 6d61 7374 6572 2d62 696e 2e30 3030 |..master-bin.000|0x0150 3030 31d1 53cd ff00 0000 0000 0000 0000 |001.S...........|0x0160 0000 0000 0000 0000 0000 0000 0000 0000 |................|0x0170 0000 0000 0000 0000 0000 0000 0000 0000 |................|0x0180 0000 00 |...|33358 898672 mariadbd 8.131301 RET read 387/0x183# Second write completes. The read sees the length of the file including the# second write, but the data is missing, all zeros.33358 898649 mariadbd 8.131540 GIO fd 84 wrote 44 bytes0x0000 f23e bc64 a101 0000 002c 0000 0083 0100 |.>.d.....,......|0x0010 0000 0011 0000 006d 6173 7465 722d 6269 |.......master-bi|0x0020 6e2e 3030 3030 3032 0539 6465 |n.000002.9de|33358 898649 mariadbd 8.131596 RET write 44/0x2cHere we clearly see that the dump thread is reading the file in parallel with the binlog background thread writing the second binlog checkpoint. The read sees the length as if the second write had completed, but it does not see the correct data, it reads all zeros.
So this is not a bug in MariaDB, it's a problem with FreeBSD.
We can solve this by not using tmpfs for /tmp/var on the FreeBSD 13.0 builder.