Details
-
Bug
-
Status: Closed (View Workflow)
-
Critical
-
Resolution: Fixed
-
12.3
-
Can result in hang or crash
-
Q1/2027 Server Maintenance
Description
innodb engine crashed with signal 11 in some case , it looks like a timerace.
In good cases, the binlog waits until disk space is free again
2026-10-09 8:17:55 0 [ERROR] InnoDB: preallocating 33554432 bytes for file ./binlog-000003.ibb failed with error 28 |
2026-10-09 8:17:55 0 [ERROR] InnoDB: Unable to allocate file ./binlog-000003.ibb: "No space left on device". Waiting 60 seconds before trying again... |
2026-10-09 8:18:55 0 [ERROR] InnoDB: preallocating 33554432 bytes for file ./binlog-000003.ibb failed with error 28 |
2026-10-09 8:18:55 0 [ERROR] InnoDB: Unable to allocate file ./binlog-000003.ibb: "No space left on device". Waiting 60 seconds before trying again... |
2026-10-09 8:19:55 0 [ERROR] InnoDB: preallocating 33554432 bytes for file ./binlog-000003.ibb failed with error 28 |
2026-10-09 8:19:55 0 [ERROR] InnoDB: Unable to allocate file ./binlog-000003.ibb: "No space left on device". Waiting 60 seconds before trying again... |
2026-10-09 8:20:55 0 [ERROR] InnoDB: preallocating 33554432 bytes for file ./binlog-000003.ibb failed with error 28 |
2026-10-09 8:20:55 0 [ERROR] InnoDB: Unable to allocate file ./binlog-000003.ibb: "No space left on device". Waiting 60 seconds before trying again... |
2026-10-09 8:21:55 0 [ERROR] InnoDB: preallocating 33554432 bytes for file ./binlog-000003.ibb failed with error 28 |
2026-10-09 8:21:55 0 [ERROR] InnoDB: Unable to allocate file ./binlog-000003.ibb: "No space left on device". Waiting 60 seconds before trying again... |
2026-10-09 8:23:02 90 [Note] Start binlog_dump to slave_server(0), pos(binlog-000004.ibb, 4), using_gtid(1), gtid('0-1-20') |
2026-10-09 8:23:10 96 [Note] Start binlog_dump to slave_server(0), pos(binlog-000004.ibb, 4), using_gtid(1), gtid('0-1-21') |
And can't be interrupted like in
with kill thread or max_statement_time.
thread stall in "killed
To reproduce:
sudo ./t9-after-the-segv.sh setup
|
 |
sudo ./t9-after-the-segv.sh run 5 --debug # with full stacktrace |
or
|
./t9-after-the-segv.sh run 5 |
 |
#5 is the number of iterations. |
 |
options:
|
--debug produces full stacktrace
|
--binlog-cache-size set binlog-cache-size
|
 |
# clean up data
|
sudo ./t9-after-the-segv.sh teardown
|
attached full stacktrace
Claude diagnostic limited to 5 sentences
With binlog_storage_engine=innodb and no space to preallocate the next .ibb file, a transaction crossing an .ibb boundary crashes the server with signal 11 in trx_t::write_serialisation_history (trx0trx.cc:1194) → mtr_t::commit → log_t::append (mtr0mtr.cc:1224) → memcpy, with binlog_ctx unreadable at address 0x0, while innodb_binlog_prealloc_thread sits in fsp_binlog_tablespace_close(file_no=1) → flush_up_to. On restart, after freeing space, crash recovery itself segfaults and the server then aborts with InnoDB: The log was only scanned up to 37882979, while the current LSN at the time of the latest checkpoint 33672574 was 101900964!, so the instance is unrecoverable and the fate of the transaction cannot be determined. The intended behaviour is visible in the runs that survive, which log InnoDB: Unable to allocate file ./binlog-000003.ibb: "No space left on device". Waiting 60 seconds before trying again... and commit normally — the allocation is a background retry, not on the transaction’s critical path. Reproduction rate depends on binlog_cache_size: 0/10 at the 32K default, 1/5 and (in earlier runs of the same shape) 3/4 and 4/5 at 256M; the attached script reproduces it on mariadb:12.3 with a size-limited datadir. Found while investigating MDEV-41194, which concerns the file binlog and a different failure mode. |
Attachments
Issue Links
- is caused by
-
MDEV-34705 Improving performance of binary logging by removing the need of syncing it
-
- Closed
-
- relates to
-
MDEV-4867 Incomplete Binlog writes on disk-full error, also breaking slave replication
-
- Stalled
-
-
MDEV-37949 Implement innodb_log_archive
-
- Closed
-
-
MDEV-41194 Binlog group-commit write-failure: findings summary (probable bug: 11.8+)
-
- Open
-