Background
|
|
A real issue bit me in 10.6, so I tried to reproduce it: a disk-full
|
test, real ENOSPC, concurrent writers, no crash/restart involved.
|
While 10.6 is now EOL, there may be some people still using it, and I
|
also wanted to see if this was fixed in later code. That led to a
|
mixed picture: one of the two defects below is fixed by 11.8, the
|
other still looks present through 12.3, from source inspection only
|
(with the help of AI). So: one issue confirmed by reproduction on
|
10.6, a second believed still present in current code but not tested.
|
|
Pinned commits (verify with `git show <hash>:path`, not branch tip)
|
10.6 cb32aa9aa85b243bead6ed47653db0f0ffc218e8 (10.6.25-39-g..., 2026-03-31)
|
11.8 b3a5d24ef499f1f963aaa3507082afd6505e7019 (11.8.6-...)
|
12.3 2f4743f1515d241ed0c903e376759291c9c41800 (12.3.1-147-g..., 2026-03-15)
|
origin/10.6 and origin/12.3 have moved past these since (210ab6e.../
|
881b64e... as of 2026-09-17); nothing relevant changed in that drift.
|
|
------------------------------------------------------------------
|
TL;DR
|
|
Under group commit, when one session's binlog write fails partway
|
(e.g. ENOSPC):
|
|
1. Failure 1 (10.6 only, fixed by 11.8): the write's own error is
|
discarded. CacheWriter::write() always returns 0.
|
2. Failure 2 (10.6, 11.8, 12.3 - all checked, current included):
|
even when the failure is detected, the group-commit loop does
|
not stop. It writes the next queued session's transaction into
|
the same file immediately after the torn one, no marker.
|
|
Reproduced in 10.6 (real ENOSPC, 3 concurrent writers, no
|
crash/restart): clean GTID/Table_map, then ~3.8KB of garbled event
|
data, then a 45,155,721-byte hole of physically unwritten zero bytes,
|
then a different valid transaction (GTID 14621) resuming cleanly.
|
Cross-checked 3 ways (our scanner, mariadb-binlog -c, a replica's
|
checksum failure) - offsets agree. Note: the exact mechanism that
|
produced a 45MB hole specifically (vs. a small torn region) is not
|
established; see "what actually ends up on disk" below.
|
|
------------------------------------------------------------------
|
Failure 1: CacheWriter::write() eats the error (10.6 only)
|
|
sql/log.cc, class CacheWriter::write(), ~line 7421-7450
|
|
if (first) write_header(pos, len); // return value discarded
|
else write_data(pos, len); // return value discarded
|
remains -= len; // advances regardless
|
if ((first = !remains)) write_footer(); // discarded
|
DBUG_RETURN(0); // always "success"
|
|
write_header/write_data/write_footer -> Log_event_writer::
|
write_internal() (log_event_server.cc:817-827) does check
|
my_b_safe_write() and returns 1 on error correctly. CacheWriter
|
just never looks at it.
|
|
Caller MYSQL_BIN_LOG::write_cache() has 4 guard sites (~7536, 7562,
|
7574, 7604) of the form `if (writer.write(...)) return ER_ERROR_
|
ON_WRITE;` - dead code, since writer.write() can't return nonzero.
|
|
Consequence: `remains` decrements as if bytes landed even when they
|
didn't. current->error never gets set from this path, so
|
run_commit_ordered() still runs -> storage engine commits while the
|
binlog image for that same txn is torn.
|
|
Fixed between 10.6 and 11.8 by commit 24c923d4984 (MDEV-31273,
|
"Refactor write_cache()", a checksum-precompute prep refactor).
|
Commit message explicitly notes it as a fix "bonus". Not a
|
deliberate targeted fix. 11.8's Event_log::write_cache() checks each
|
write_header/write_data/write_footer call individually and
|
propagates ER_ERROR_ON_WRITE correctly. Verified same in 12.3.
|
|
------------------------------------------------------------------
|
Failure 2: group-commit loop doesn't stop on a failed entry
|
(10.6, 11.8, 12.3 - unfixed in all)
|
|
sql/log.cc, commit-ordering loop over the queued-txn linked list
|
10.6: ~8469-8488 11.8: ~9295-9320 12.3: 10834-10858
|
(line numbers move, same shape all three)
|
|
for (current = queue; current; current = current->next) {
|
if (unlikely((current->error = write_transaction_or_stmt(...))))
|
current->commit_errno = errno;
|
// no break, no return, no skip - falls straight through
|
strmake_buf(cache_mngr->last_commit_pos_file, log_file_name);
|
commit_offset = my_b_write_tell(&log_file);
|
...
|
// continues to current->next regardless
|
}
|
|
queue spans multiple client sessions being flushed into one shared
|
file in this group-commit cycle. On failure: error is recorded on
|
`current` (for later delivery to that session - can't call
|
my_error() here per the code's own comment), then the loop
|
immediately proceeds to write the next session's txn into the same
|
file from wherever my_b_write_tell() currently claims to be. No
|
break, no rotate, no incident flag.
|
|
12.3 detail: commit_offset/update_gtid_index bookkeeping is gated by
|
`if (!opt_binlog_engine_hton)`; with the opt-in binlog storage
|
engine active, tracking happens elsewhere - doesn't change the
|
defect, loop still doesn't break/skip either way.
|
|
Not fixed in any version checked, including 12.3 (latest stable at
|
time of check).
|
|
------------------------------------------------------------------
|
What actually ends up on disk
|
|
Observed on 10.6: a torn/garbled transaction, immediately followed
|
by a gap of bytes that were never actually written, then a
|
different, later transaction's data landing right on top - with
|
nothing anywhere flagging that any of it happened. Failure 2
|
explains why writing continues past the tear; the hole's exact
|
size isn't explained yet (see below). This is the part that matters
|
for anyone relying on the binlog as a source of truth, especially
|
if it's being fed to downstream replicas.
|
|
Reproduced in 10.6 (binlog.000002, ENOSPC during a 10,109-row
|
DELETE, 3 concurrent writers, zero crashes/restarts):
|
32735543 Table_map ends cleanly
|
32735543-32739328 ~3.8KB garbled (CRC fail; includes leaked
|
payload - literal repeated 'x' from this
|
session's own VARCHAR, in event-framing
|
position)
|
32739328-77895049 45,155,721 zero bytes, confirmed unwritten
|
77895049+ GTID 14621, different transaction, fully valid
|
Cross-validated: our scanner, mariadb-binlog -c, replica checksum
|
failure - all agree on tear offset.
|
|
Blocking open question: my_write() (mysys/my_write.c) is documented
|
to block and retry under MY_WAIT_IF_FULL for plain ENOSPC, via
|
wait_for_free_space() - so a binlog write hitting disk-full should
|
normally just stall, not fail. What actually made write_header/
|
write_data return failure here, rather than blocking the way
|
my_write() does, hasn't been isolated - and it's likely the missing
|
piece that would explain why the gap is exactly 45,155,721 bytes
|
rather than some smaller torn region. Failure 1's own bookkeeping
|
(`remains -= len`) is local event-framing logic that doesn't feed
|
the file offset, so it isn't what walks the position forward by
|
45MB either. Doesn't change the finding: the file ends up with a
|
real hole followed by someone else's data either way.
|
|
Also unconfirmed: whether GTID 14621 (the transaction right after
|
the gap) was the next session already queued in the same group-
|
commit round, or the same session committing again in a later
|
round - both leave an identical trace, since nothing marks the file
|
as damaged either way. Doesn't matter for the finding: the loop not
|
stopping on a failed write is what allows it, regardless of whose
|
transaction lands there.
|
|
------------------------------------------------------------------
|
Version matrix
|
|
Ver Failure 1 (error discarded) Failure 2 (loop doesn't stop)
|
---- ----------------------------- -----------------------------
|
10.6 present present
|
11.8 fixed present, unchanged
|
12.3 fixed, same as 11.8 present, unchanged
|
(verified sql/log.cc:10848)
|
|
10.6's status is from reproducing it; 11.8's and 12.3's are from
|
reading the pinned source only, not reproduced there.
|
|
Headline, and the reason for this report: this is not a 10.6-only,
|
already-fixed bug that upgrading makes moot. Failure 1 (the error
|
getting thrown away) is fixed in 11.8+. Failure 2 - the part that
|
actually lets a torn transaction be silently followed by someone
|
else's data in the same file - is present, unchanged, in every
|
version checked, current stable (12.3) included. A 10.6 user with
|
replication who upgrades to 11.8+ gets correct failure detection
|
back, but the file can still end up silently corrupted the same way
|
once a write fails.
|
|
From a DBA/DBRE's perspective: a replica picks up the corrupted
|
event from the master and can't process it. The master's own binlog
|
was never written properly, so there's nothing to recover from -
|
the only way to fix the replicas is to reclone them all from the
|
master. That can be painful on a busy cluster.
|
|
------------------------------------------------------------------
|
Related: InnoDB tablespace growth doesn't retry ENOSPC at all
|
|
storage/innobase/os/os0file.cc, os_file_set_size(), 10.6:1531-1589
|
|
do {
|
...
|
if (!fallocate(file, 0, current_size, size - current_size)) {
|
err = 0; break;
|
}
|
err = errno;
|
} while (err == EINTR && srv_shutdown_state <= SRV_SHUTDOWN_INITIATED);
|
switch (err) {
|
case 0: return true;
|
default: sql_print_error(...); /* fall through */
|
case EINTR: errno = err; return false; // ENOSPC lands here
|
}
|
|
Retry condition is EINTR only - ENOSPC isn't retried, fails on
|
first attempt, propagates as ERROR 1114 (table full) immediately.
|
Reproduced: 3 sessions got the error instantly, no wait,
|
even though freeing disk space a moment later would let the same
|
INSERT succeed.
|
|
Asymmetry: my_write()/wait_for_free_space() (the binlog write path)
|
blocks transparently on ENOSPC and completes once space returns -
|
client sees a slow commit, nothing else. InnoDB's os_file_set_size()
|
hits the identical OS condition on the same disk and fails hard
|
immediately. No principled reason for the difference (fd stays open
|
and valid in both cases).
|
|
Unchanged in 11.8 and 12.3 (identical EINTR-only retry condition).
|
12.3 adds a DBUG_EXECUTE_IF("ib_alloc_file_disk_full", ...) fault-
|
injection hook just before the loop (mirrors binlog side's
|
fail_binlog_write_1/inject_error_writing_xid) but retry logic itself
|
is untouched.
|
|
Still open: how does Aria handle the same condition? It's the
|
other major engine actually in play here (aria_log.0000000N present
|
in every datadir seen), with its own independent file-growth and
|
logging code, not examined at all in this investigation - may share
|
this same fail-hard-no-retry gap, may not.
|
|
------------------------------------------------------------------
|
What seems to be required to fix this
|
|
Minimum safe: stop on a failed write. Don't advance the position
|
tracker past unwritten bytes; don't let the group-commit loop write
|
the next session's txn into the file until the failure is handled.
|
Turns silent corruption into a loud, recoverable error - but the
|
failed txn is lost and every other queued session in that round has
|
to be told their commit failed too, even though their own write was
|
fine. In reality that's not practical.
|
|
Ideal: don't just detect and abort, finish the write. my_write()
|
(mysys/my_write.c) already retries a partial write correctly,
|
resuming at the exact byte offset, blocking via
|
wait_for_free_space() until space returns. That discipline isn't
|
carried up to where it matters: pre-11.8, CacheWriter ignores the
|
failure outright (Failure 1); post-fix, write_cache() detects it
|
correctly but treats it as terminal rather than resumable. Either
|
way, and in the group-commit loop too, nothing waits or retries
|
like my_write() does. Correct fix:
|
hold the transaction (and the group-commit batch behind it) exactly
|
where it is, wait for space the same way my_write() does, and
|
finish writing the rest of that same event before any subsequent
|
transaction is allowed to write into the file.
|
Bug
Minor
