Uploaded image for project: 'MariaDB Server'
  1. MariaDB Server
  2. MDEV-39209

10.11.15 and 10.11.16 crashing with SEGV

    XMLWordPrintable

Details

    • Bug
    • Status: In Progress (View Workflow)
    • Major
    • Resolution: Unresolved
    • 10.11.15, 10.11.16
    • 10.11
    • Server
    • None
    • debian 12, baremetal
    • Q2/2026 Server Maintenance

    Description

      We've upgraded to 10.11.16 from 10.11.13 and we've started to see SEGV crashes. We downgraded to 10.11.15 and the crashes are still there. 10.11.13 seems to be the only stable version.

      These crashes aren't producing any traces on mariadb logs or on journalctl either, which is odd. Just:

      Mar 30 10:08:59 clouddb1022 systemd[1]: mariadb@s3.service: Main process exited, code=killed, status=11/SEGV
      

      And

      Mar 30 10:08:59 clouddb1022 systemd[1]: mariadb@s3.service: Main process exited, code=killed, status=11/SEGV
      Mar 30 10:08:59 clouddb1022 systemd[1]: mariadb@s3.service: Failed with result 'signal'.
      Mar 30 10:08:59 clouddb1022 systemd[1]: mariadb@s3.service: Consumed 34min 19.184s CPU time.
      Mar 30 10:09:05 clouddb1022 systemd[1]: mariadb@s3.service: Scheduled restart job, restart counter is at 3.
      Mar 30 10:09:05 clouddb1022 systemd[1]: Stopped mariadb@s3.service - mariadb database server.
      Mar 30 10:09:05 clouddb1022 systemd[1]: mariadb@s3.service: Consumed 34min 19.184s CPU time.
      Mar 30 10:09:05 clouddb1022 systemd[1]: Starting mariadb@s3.service - mariadb database server...
      Mar 30 10:09:05 clouddb1022 mysqld[2586624]: 2026-03-30 10:09:05 0 [Warning] Could not increase number of max_open_files to more than 200001 (request: 800329)
      Mar 30 10:09:05 clouddb1022 mysqld[2586624]: 2026-03-30 10:09:05 0 [Note] Starting MariaDB 10.11.16-MariaDB source revision 3218602d3100db9ce7a875511a591cddc173cc16 server_uid s47KRb1UmTUPdWrEfBCAd7qcmkI= as process 2586624
      Mar 30 10:09:05 clouddb1022 mysqld[2586624]: 2026-03-30 10:09:05 0 [Note] mysqld: Aria engine: starting recovery
      Mar 30 10:09:05 clouddb1022 mysqld[2586624]: recovered pages: 0% 24% 35% 52% 100% (0.0 seconds); tables to flush: 2 1 0
      Mar 30 10:09:05 clouddb1022 mysqld[2586624]:  (0.0 seconds);
      Mar 30 10:09:05 clouddb1022 mysqld[2586624]: 2026-03-30 10:09:05 0 [Note] mysqld: Aria engine: recovery done
      Mar 30 10:09:05 clouddb1022 mysqld[2586624]: 2026-03-30 10:09:05 0 [Note] InnoDB: Compressed tables use zlib 1.3.1
      Mar 30 10:09:05 clouddb1022 mysqld[2586624]: 2026-03-30 10:09:05 0 [Note] InnoDB: Number of transaction pools: 1
      Mar 30 10:09:05 clouddb1022 mysqld[2586624]: 2026-03-30 10:09:05 0 [Note] InnoDB: Using AVX512 instructions
      Mar 30 10:09:05 clouddb1022 mysqld[2586624]: 2026-03-30 10:09:05 0 [Note] InnoDB: Using Linux native AIO
      Mar 30 10:09:05 clouddb1022 mysqld[2586624]: 2026-03-30 10:09:05 0 [Note] InnoDB: innodb_buffer_pool_size_max=102400m, innodb_buffer_pool_size=102400m
      Mar 30 10:09:10 clouddb1022 mysqld[2586624]: 2026-03-30 10:09:10 0 [Note] InnoDB: Completed initialization of buffer pool
      Mar 30 10:09:10 clouddb1022 mysqld[2586624]: 2026-03-30 10:09:10 0 [Note] InnoDB: File system buffers for log disabled (block size=4096 bytes)
      Mar 30 10:09:10 clouddb1022 mysqld[2586624]: 2026-03-30 10:09:10 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=65148004503307
      Mar 30 10:09:19 clouddb1022 mysqld[2586624]: 2026-03-30 10:09:19 0 [Note] InnoDB: End of log at LSN=65149039161580
      Mar 30 10:09:19 clouddb1022 mysqld[2586624]: 2026-03-30 10:09:19 0 [Note] InnoDB: 1 transaction(s) which must be rolled back or cleaned up in total 1 row operations to undo
      Mar 30 10:09:19 clouddb1022 mysqld[2586624]: 2026-03-30 10:09:19 0 [Note] InnoDB: Trx id counter is 165692918816
      Mar 30 10:09:19 clouddb1022 mysqld[2586624]: 2026-03-30 10:09:19 0 [Note] InnoDB: To recover: 676145 pages
      Mar 30 10:09:31 clouddb1022 mysqld[2586624]: 2026-03-30 10:09:31 0 [Note] InnoDB: Last binlog file './db1154-bin.000003', position 590001970
      Mar 30 10:09:31 clouddb1022 mysqld[2586624]: 2026-03-30 10:09:31 0 [Note] InnoDB: 128 rollback segments are active.
      Mar 30 10:09:31 clouddb1022 mysqld[2586624]: 2026-03-30 10:09:31 0 [Note] InnoDB: Starting in background the rollback of recovered transactions
      Mar 30 10:09:31 clouddb1022 mysqld[2586624]: 2026-03-30 10:09:31 0 [Note] InnoDB: Removed temporary tablespace data file: "./ibtmp1"
      Mar 30 10:09:31 clouddb1022 mysqld[2586624]: 2026-03-30 10:09:31 0 [Note] InnoDB: Setting file './ibtmp1' size to 12.000MiB. Physically writing the file full; Please wait ...
      Mar 30 10:09:31 clouddb1022 mysqld[2586624]: 2026-03-30 10:09:31 0 [Note] InnoDB: File './ibtmp1' size is now 12.000MiB.
      Mar 30 10:09:31 clouddb1022 mysqld[2586624]: 2026-03-30 10:09:31 0 [Note] InnoDB: log sequence number 65149039161580; transaction id 165692918817
      Mar 30 10:09:31 clouddb1022 mysqld[2586624]: 2026-03-30 10:09:31 0 [Note] Plugin 'FEEDBACK' is disabled.
      Mar 30 10:09:31 clouddb1022 mysqld[2586624]: 2026-03-30 10:09:31 0 [Note] InnoDB: Loading buffer pool(s) from /srv/sqldata.s3/ib_buffer_pool
      Mar 30 10:09:31 clouddb1022 mysqld[2586624]: 2026-03-30 10:09:31 0 [Note] InnoDB: Rolled back recovered transaction 165692918814
      Mar 30 10:09:31 clouddb1022 mysqld[2586624]: 2026-03-30 10:09:31 0 [Note] InnoDB: Rollback of non-prepared transactions completed
      Mar 30 10:09:31 clouddb1022 mysqld[2586624]: 2026-03-30 10:09:31 0 [Note] Server socket created on IP: '0.0.0.0', port: '3313'.
      Mar 30 10:09:31 clouddb1022 mysqld[2586624]: 2026-03-30 10:09:31 0 [Note] Server socket created on IP: '::', port: '3313'.
      Mar 30 10:09:31 clouddb1022 mysqld[2586624]: 2026-03-30 10:09:31 0 [Note] Server socket created on IP: '0.0.0.0', port: '3333'.
      Mar 30 10:09:31 clouddb1022 mysqld[2586624]: 2026-03-30 10:09:31 0 [Note] Server socket created on IP: '::', port: '3333'.
      Mar 30 10:09:31 clouddb1022 mysqld[2586624]: 2026-03-30 10:09:31 0 [Warning] Neither --relay-log nor --relay-log-index were used; so replication may break when this MariaDB server acts as a replica and has its hostname changed. Please use '--log-basename=#' or '--relay-log=clouddb1022-relay-bin' to avoid this problem.
      Mar 30 10:09:31 clouddb1022 mysqld[2586624]: 2026-03-30 10:09:31 0 [Note] /opt/wmf-mariadb1011/bin/mysqld: ready for connections.
      Mar 30 10:09:31 clouddb1022 mysqld[2586624]: Version: '10.11.16-MariaDB'  socket: '/run/mysqld/mysqld.s3.sock'  port: 3313  MariaDB Server
      Mar 30 10:09:31 clouddb1022 systemd[1]: Started mariadb@s3.service - mariadb database server.
      

      I've attached a gdb and got some traces which I am attaching here.
      This seems to be what is being run right before the crash:

      "SET STATEMENT max_statement_time = 900 FOR\nSELECT * FROM ((\n\n", ' ' <repeats 20 times>, "SELECT\n", ' ' <repeats 24 times>, "'abwiki' AS dbName,\n", ' ' <repeats 24 times>, "revs.rev_id AS id,\n", ' ' <repeats 24 times>, "r"..., m_buf_length = 1077516, m_echo = true, m_echo_saved = false, m_cpp_buf = 0x7fc0c84a5760 "SET STATEMENT max_statement_time = 900 FOR\nSELECT * FROM ((\n\n", ' ' <repeats 20 times>, "SELECT\n", ' ' <repeats 24 times>, "'abwiki' AS dbName,\n", ' ' <repeats 24 times>, "revs.rev_id AS id,\n", ' ' <repeats 24 times>, "r"...
      

      I am attaching a new gdb to try to capture the exact query so we can try to reproduce and investigate.

      Internally being tracked at T420177

      Attachments

        1. bt_crashing_thread.log
          992 kB
        2. mariadb_crash_bt.log
          3.26 MB
        3. query.txt
          1.26 MB

        Issue Links

          Activity

            People

              Gosselin Dave Gosselin
              marostegui Manuel Arostegui
              Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

                Created:
                Updated:

                Git Integration

                  Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.