Uploaded image for project: 'MariaDB Server'
  1. MariaDB Server
  2. MDEV-41442

innodb engine crashed with signal 11 in some case

    XMLWordPrintable

Details

    • Can result in hang or crash
    • Q1/2027 Server Maintenance

    Description

      innodb engine crashed with signal 11 in some case , it looks like a timerace.

      In good cases, the binlog waits until disk space is free again

      2026-10-09  8:17:55 0 [ERROR] InnoDB: preallocating 33554432 bytes for file ./binlog-000003.ibb failed with error 28
      2026-10-09  8:17:55 0 [ERROR] InnoDB: Unable to allocate file ./binlog-000003.ibb: "No space left on device". Waiting 60 seconds before trying again...
      2026-10-09  8:18:55 0 [ERROR] InnoDB: preallocating 33554432 bytes for file ./binlog-000003.ibb failed with error 28
      2026-10-09  8:18:55 0 [ERROR] InnoDB: Unable to allocate file ./binlog-000003.ibb: "No space left on device". Waiting 60 seconds before trying again...
      2026-10-09  8:19:55 0 [ERROR] InnoDB: preallocating 33554432 bytes for file ./binlog-000003.ibb failed with error 28
      2026-10-09  8:19:55 0 [ERROR] InnoDB: Unable to allocate file ./binlog-000003.ibb: "No space left on device". Waiting 60 seconds before trying again...
      2026-10-09  8:20:55 0 [ERROR] InnoDB: preallocating 33554432 bytes for file ./binlog-000003.ibb failed with error 28
      2026-10-09  8:20:55 0 [ERROR] InnoDB: Unable to allocate file ./binlog-000003.ibb: "No space left on device". Waiting 60 seconds before trying again...
      2026-10-09  8:21:55 0 [ERROR] InnoDB: preallocating 33554432 bytes for file ./binlog-000003.ibb failed with error 28
      2026-10-09  8:21:55 0 [ERROR] InnoDB: Unable to allocate file ./binlog-000003.ibb: "No space left on device". Waiting 60 seconds before trying again...
      2026-10-09  8:23:02 90 [Note] Start binlog_dump to slave_server(0), pos(binlog-000004.ibb, 4), using_gtid(1), gtid('0-1-20')
      2026-10-09  8:23:10 96 [Note] Start binlog_dump to slave_server(0), pos(binlog-000004.ibb, 4), using_gtid(1), gtid('0-1-21')
      

      And can't be interrupted like in

      MDEV-41194

      with kill thread or max_statement_time.

      thread stall in "killed

      To reproduce:

      sudo ./t9-after-the-segv.sh setup
       
      sudo ./t9-after-the-segv.sh run 5 --debug   # with full stacktrace
      or 
      ./t9-after-the-segv.sh run 5
       
      #5 is the number of iterations.
       
      options:
      --debug produces full stacktrace
      --binlog-cache-size  set binlog-cache-size 
       
      # clean up data
      sudo ./t9-after-the-segv.sh teardown 
      

      attached full stacktrace

      Claude diagnostic limited to 5 sentences

      With binlog_storage_engine=innodb and no space to preallocate the next .ibb file, a transaction crossing an .ibb boundary crashes the server with signal 11 in trx_t::write_serialisation_history (trx0trx.cc:1194) → mtr_t::commit → log_t::append (mtr0mtr.cc:1224) → memcpy, with binlog_ctx unreadable at address 0x0, while innodb_binlog_prealloc_thread sits in fsp_binlog_tablespace_close(file_no=1) → flush_up_to. On restart, after freeing space, crash recovery itself segfaults and the server then aborts with InnoDB: The log was only scanned up to 37882979, while the current LSN at the time of the latest checkpoint 33672574 was 101900964!, so the instance is unrecoverable and the fate of the transaction cannot be determined. The intended behaviour is visible in the runs that survive, which log InnoDB: Unable to allocate file ./binlog-000003.ibb: "No space left on device". Waiting 60 seconds before trying again... and commit normally — the allocation is a background retry, not on the transaction’s critical path. Reproduction rate depends on binlog_cache_size: 0/10 at the 32K default, 1/5 and (in earlier runs of the same shape) 3/4 and 4/5 at 256M; the attached script reproduces it on mariadb:12.3 with a size-limited datadir. Found while investigating MDEV-41194, which concerns the file binlog and a different failure mode.
      

      Attachments

        Issue Links

          Activity

            People

              knielsen Kristian Nielsen
              Richard Richard Stracke
              Votes:
              1 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Time Tracking

                  Estimated:
                  Original Estimate - 0d
                  0d
                  Remaining:
                  Remaining Estimate - 0d
                  0d
                  Logged:
                  Time Spent - 0.5d
                  0.5d

                  Git Integration

                    Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.