Uploaded image for project: 'MariaDB Server'
  1. MariaDB Server
  2. MDEV-39504

Mariadb crash in innoDB with Signal 11 during startup

    XMLWordPrintable

Details

    Description

      We got a crash in Mariadb in a setup. We noticed there was a mariabackup started around 40mins back and that was going on for that much time :

       
       
       
      2026-04-30  2:21:45 0 [Note] WSREP: async IST sender starting to serve tcp://11.127.5.37:4568 sending 25321-25424, preload starts from 25321
      2026-04-30  2:21:45 0 [Note] WSREP: Donor monitor thread started to monitor
      2026-04-30  2:21:45 0 [Note] WSREP: Running: 'wsrep_sst_mariabackup --role 'donor' --address '11.127.5.37:4444/xtrabackup_sst//1' --local-port 3306 --socket '/var/lib/mysql/mysql.sock' --progress 0 --datadir '/mariadb/data/' --gtid '9d181c43-42e2-11f1-975d-26a608101ef5:25424' --gtid-domain-id 0 --binlog 'mysql-bin' --mysqld-args --wsrep-cluster-address=gcomm://11.127.4.37,11.127.5.37,11.127.6.37 --wsrep-node-address=11.127.6.37'
      2026-04-30  2:21:45 2 [Note] WSREP: sst_donor_thread signaled with 0
      WSREP_SST: [INFO] mariabackup SST started on donor (20260430 02:21:45.468)
      WSREP_SST: [INFO] SSL configuration: CA='', CAPATH='', CERT='', KEY='', MODE='DISABLED', encrypt='0' (20260430 02:21:45.509)
      2026-04-30  2:21:45 0 [Note] WSREP: IST sender 25321 -> 25424
      WSREP_SST: [INFO] Streaming with mbstream (20260430 02:21:45.814)
      WSREP_SST: [INFO] Using socat as streamer (20260430 02:21:45.815)
      2026-04-30  2:21:46 0 [Note] WSREP: (35d7067e-b66f, 'tcp://0.0.0.0:4567') turning message relay requesting off
      WSREP_SST: [INFO] Moving '/mariadb/data/mariabackup.backup.log' to '/mariadb/sst/mariabackup.backup.log.2026.04.30-02.21.45.824567532' (20260430 02:21:46.502)
      WSREP_SST: [INFO] Using '/mariadb/tmp/tmp.afRdQZVXHJ' as mariadb-backup temporary directory (20260430 02:21:46.941)
      WSREP_SST: [INFO] Using '/mariadb/tmp/tmp.eD7XppQWZ4' as mariadb-backup working directory (20260430 02:21:46.943)
      WSREP_SST: [INFO] Streaming GTID file before SST (20260430 02:21:46.945)
      WSREP_SST: [INFO] Evaluating '/usr/bin/mbstream' -c 'mariadb_backup_galera_info' 'donor_galera_info' | socat -u stdio TCP:11.127.5.37:4444; RC=( ${PIPESTATUS[@]} ) (20260430 02:21:46.949)
      WSREP_SST: [INFO] Sleeping before data transfer for SST (20260430 02:21:46.958)
      WSREP_SST: [INFO] Streaming the backup to joiner at 11.127.5.37:4444 (20260430 02:21:57.165)
      WSREP_SST: [INFO] Evaluating /usr/bin/mariadb-backup --backup --no-version-check --databases-exclude='lost+found' --tmpdir='/mariadb/tmp/tmp.afRdQZVXHJ' --innodb-data-home-dir='.' --log-bin='mysql-bin' --user='nrd' --socket='/var/lib/mysql/mysql.sock' --use-memory=2G --galera-info --stream=mbstream --target-dir='/mariadb/tmp/tmp.eD7XppQWZ4' --datadir='/mariadb/data' --mysqld-args '--wsrep-cluster-address=gcomm://11.127.4.37,11.127.5.37,11.127.6.37' '--wsrep-node-address=11.127.6.37' 2> '/mariadb/data/mariabackup.backup.log' | socat -u stdio TCP:11.127.5.37:4444; RC=( ${PIPESTATUS[@]} ) (20260430 02:21:57.167)
      
      

      Then after 40mins, the mariadb sees a crash. Not sure if this is realated to the long running mariabackup process:

       
      ##########################################################
      260430  3:06:18 [ERROR] mariadbd got signal 11 ;
      Sorry, we probably made a mistake, and this is a bug.
       
      Your assistance in bug reporting will enable us to fix this for the next release.
      To report this bug, see https://mariadb.com/kb/en/reporting-bugs about how to report
      a bug on https://jira.mariadb.org/.
       
      Please include the information from the server start above, to the end of the
      information below.
       
      Server version: 11.4.9-MariaDB-log source revision: aa6a2e6bf05d43c3bb44db50566937a5f97a6580
       
      WSREP: Suppressing further logging
      WSREP: Shutting down network communications
       
      The information page at https://mariadb.com/kb/en/how-to-produce-a-full-stack-trace-for-mariadbd/
      contains instructions to obtain a better version of the backtrace below.
      Following these instructions will help MariaDB developers provide a fix quicker.
       
      Attempting backtrace. Include this in the bug report.
      (note: Retrieving this information may fail)
       
      Thread pointer: 0x7f75c0000c58
      stack_bottom = 0x7f77b0380000 thread_stack 0x49000
      2026-04-30  3:06:18 0 [ERROR] [FATAL] InnoDB: fdatasync() returned 107
      Fatal signal 6 while backtracing
      WSREP_SST: [ERROR] mariadb-backup finished with error: 134. Check syslog or '/mariadb/data/mariabackup.backup.log' for details (20260430 03:06:18.686)
      WSREP_SST: [ERROR] Cleanup after exit with status: 22 (20260430 03:06:18.688)
                                                                                        
      
      

      Attachments

        Activity

          People

            Unassigned Unassigned
            Sahai Har Gagan
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

              Created:
              Updated:

              Git Integration

                Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.