Uploaded image for project: 'MariaDB Server'
  1. MariaDB Server
  2. MDEV-36871

mariadb-backup incremental segfault querying mariadb_backup_history

    XMLWordPrintable

Details

    • Can result in hang or crash
    • Hide
      Mariabackup since MDEV-30000 (10.11.12, 11.4.6, 11.8.2) has a regression when using --incremental-history-name which incorrectly handled the forcing of the innodb checkpoint causing Mariabackup to crash. Also discovered that in the mysql.mariadb_backup_history the format column when --stream=mbstream was used was recorded as "tar". This has been corrected to "mbstream". Under Windows the full path of mariabackup.exe was used as in the "tool" column of the history table and this is now only recording the executable name without the path. Resilience against user specified --incremental-history-name or --incremental-history-uuid which exceeds the field size also produces an error rather than behaving erroneously.
      Show
      Mariabackup since MDEV-30000 (10.11.12, 11.4.6, 11.8.2) has a regression when using --incremental-history-name which incorrectly handled the forcing of the innodb checkpoint causing Mariabackup to crash. Also discovered that in the mysql.mariadb_backup_history the format column when --stream=mbstream was used was recorded as "tar". This has been corrected to "mbstream". Under Windows the full path of mariabackup.exe was used as in the "tool" column of the history table and this is now only recording the executable name without the path. Resilience against user specified --incremental-history-name or --incremental-history-uuid which exceeds the field size also produces an error rather than behaving erroneously.
    • Q3/2025 Maintenance

    Description

      Hi,

      We in kolla-ansible openstack project are testing mariadb backup procedure, but we see this in our CI

      Here is output of backup :

      "stderr_lines": [
              "2025-05-23 18:45:58.517 INFO Loading config file at /var/lib/kolla/config_files/config.json",
              "2025-05-23 18:45:58.518 INFO Validating config file",
              "2025-05-23 18:45:58.518 INFO Kolla config strategy set to: COPY_ALWAYS",
              "2025-05-23 18:45:58.519 INFO Copying /etc/mysql/my.cnf to /etc/kolla/defaults/etc/mysql/my.cnf",
              "2025-05-23 18:45:58.519 INFO Copying permissions from /etc/mysql/my.cnf onto /etc/kolla/defaults/etc/mysql/my.cnf",
              "2025-05-23 18:45:58.519 INFO Copying service configuration files",
              "2025-05-23 18:45:58.520 INFO Deleting /etc/mysql/my.cnf",
              "2025-05-23 18:45:58.520 INFO Copying /var/lib/kolla/config_files/my.cnf to /etc/mysql/my.cnf",
              "2025-05-23 18:45:58.521 INFO Setting permission for /etc/mysql/my.cnf",
              "2025-05-23 18:45:58.521 INFO Writing out command to execute",
              "2025-05-23 18:45:58.522 INFO Setting permission for /var/log/kolla/mariadb",
              "2025-05-23 18:45:58.522 INFO Setting permission for /backup",
              "[00] 2025-05-23 18:45:58 Connecting to MariaDB server host: 192.0.2.2, user: backup_shard_0, password: set, port: 3306, socket: not set",
              "[00] 2025-05-23 18:45:58 Using server version 10.11.13-MariaDB-deb12-log",
              "mariabackup based on MariaDB server 10.11.13-MariaDB debian-linux-gnu (x86_64)",
              "[00] 2025-05-23 18:45:58 incremental backup from 0 is enabled.",
              "[00] 2025-05-23 18:45:58 uses posix_fadvise().",
              "[00] 2025-05-23 18:45:58 cd to /var/lib/mysql/",
              "[00] 2025-05-23 18:45:58 open files limit requested 0, set to 1048576",
              "[00] 2025-05-23 18:45:58 mariabackup: using the following InnoDB configuration:",
              "[00] 2025-05-23 18:45:58 innodb_data_home_dir = ",
              "[00] 2025-05-23 18:45:58 innodb_data_file_path = ibdata1:12M:autoextend",
              "[00] 2025-05-23 18:45:58 innodb_log_group_home_dir = ./",
              "[00] 2025-05-23 18:45:58 InnoDB: Using liburing",
              "2025-05-23 18:45:58 0 [Note] InnoDB: Number of transaction pools: 1",
              "mariabackup: io_uring_queue_init() failed with EPERM: sysctl kernel.io_uring_disabled has the value 2, or 1 and the user of the process is not a member of sysctl kernel.io_uring_group. (see man 2 io_uring_setup).",
              "2025-05-23 18:45:58 0 [Warning] InnoDB: liburing disabled: falling back to innodb_use_native_aio=OFF",
              "2025-05-23 18:45:58 0 [Note] InnoDB: Memory-mapped log (block size=512 bytes)",
              "250523 18:45:58 [ERROR] mariabackup got signal 11 ;",
              "Sorry, we probably made a mistake, and this is a bug.",
              "",
              "Your assistance in bug reporting will enable us to fix this for the next release.",
              "To report this bug, see https://mariadb.com/kb/en/reporting-bugs about how to report",
              "a bug on https://jira.mariadb.org/.",
              "",
              "Please include the information from the server start above, to the end of the",
              "information below.",
              "",
              "Server version: 10.11.13-MariaDB-deb12 source revision: 8fb09426b98583916ccfd4f8c49741adc115bac3",
              "",
              "The information page at https://mariadb.com/kb/en/how-to-produce-a-full-stack-trace-for-mariadbd/",
              "contains instructions to obtain a better version of the backtrace below.",
              "Following these instructions will help MariaDB developers provide a fix quicker.",
              "",
              "Attempting backtrace. Include this in the bug report.",
              "(note: Retrieving this information may fail)",
              "",
              "Thread pointer: 0x0",
              "stack_bottom = 0x0 thread_stack 0x49000",
              "Printing to addr2line failed",
              "mariabackup(my_print_stacktrace+0x2e)[0x55d3302aa3ae]",
              "mariabackup(handle_fatal_signal+0x229)[0x55d32fdcd6d9]",
              "/lib/x86_64-linux-gnu/libc.so.6(+0x3c050)[0x7f0230668050]",
              "mariabackup(server_mysql_fetch_row+0x14)[0x55d32fa19474]",
              "mariabackup(+0x76ca87)[0x55d32f9eba87]",
              "mariabackup(+0x75f37a)[0x55d32f9de37a]",
              "mariabackup(main+0x163)[0x55d32f983053]",
              "/lib/x86_64-linux-gnu/libc.so.6(+0x2724a)[0x7f023065324a]",
              "/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0x85)[0x7f0230653305]",
              "mariabackup(_start+0x21)[0x55d32f9c8161]",
              "Writing a core file...",
              "Working directory at /var/lib/mysql",
              "Resource Limits (excludes unlimited resources):",
              "Limit Soft Limit Hard Limit Units ",
              "Max stack size 8388608 unlimited bytes ",
              "Max open files 1048576 1048576 files ",
              "Max locked memory 8388608 8388608 bytes ",
              "Max pending signals 30827 30827 signals ",
              "Max msgqueue size 819200 819200 bytes ",
              "Max nice priority 0 0 ",
              "Max realtime priority 0 0 ",
              "Core pattern: core",
              "",
              "Kernel version: Linux version 6.1.0-35-amd64 (debian-kernel@lists.debian.org) (gcc-12 (Debian 12.2.0-14+deb12u1) 12.2.0, GNU ld (GNU Binutils for Debian) 2.40) #1 SMP PREEMPT_DYNAMIC Debian 6.1.137-1 (2025-05-07)",
              "",
              "/usr/local/bin/kolla_mariadb_backup_replica.sh: line 36: 44 Segmentation fault (core dumped) mariabackup --defaults-file=\"${REPLICA_MY_CNF}\" --backup --stream=mbstream --incremental-history-name=\"${LAST_FULL_DATE}\" --history=\"${LAST_FULL_DATE}\"",
              " 45 Done | gzip > \"${BACKUP_DIR}/incremental-$(date +%H)-mysqlbackup-${LAST_FULL_DATE}.qp.mbc.mbs.gz\""
          ],
          "stdout": "Taking an incremental backup\n",
          "stdout_lines": [
              "Taking an incremental backup"
          ]
      }
      

      mariabackup: io_uring_queue_init() failed with EPERM: sysctl kernel.io_uring_disabled has the value 2, or 1 and the user of the process is not a member of sysctl kernel.io_uring_group. (see man 2 io_uring_setup)...........
      .
      .
      

      Sysctl :

      kernel.apparmor_restrict_unprivileged_io_uring = 0
      kernel.io_uring_disabled = 0
      kernel.io_uring_group = -1
      
      

      Server and backup process both running in container, I've also tried to run backup procedure as privileged container, or with seccomp unconfined , nothing helped.

      Mariadb-server and mariadb-backup is 10.11.13, if I make a APT pin and disable installation of 10.11.13 here https://review.opendev.org/c/openstack/kolla/+/950820 tests are working again
      https://review.opendev.org/c/openstack/kolla-ansible/+/950794 / https://6dee196209eb9fb193b3-af1c41ea995a25ff0cf4e07cd0399165.ssl.cf5.rackcdn.com/openstack/8eee012a8cf345f09b2d505071186d11/primary/logs/ansible/test-mariadb .

      Build of image https://6dee196209eb9fb193b3-af1c41ea995a25ff0cf4e07cd0399165.ssl.cf5.rackcdn.com/openstack/8eee012a8cf345f09b2d505071186d11/primary/logs/build/mariadb-server.log

      where it can be seen 10.11.11 is working.

      Thanks for fix

      Attachments

        Issue Links

          Activity

            People

              thiru Thirunarayanan Balathandayuthapani
              kevko Michal Arbet
              Votes:
              0 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Git Integration

                  Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.