Uploaded image for project: 'MariaDB Server'
  1. MariaDB Server
  2. MDEV-33528

Server crash at startup which can't be recovered from

    XMLWordPrintable

Details

    • Bug
    • Status: Closed (View Workflow)
    • Major
    • Resolution: Incomplete
    • 10.3.39
    • N/A
    • None
    • Debian 10 (buster)

    Description

      Hello there,

      We had a crash with MariaDB server 10.3.39 on a Debian 10 server, which we weren't able to recover from. We know this is an old version of MariaDB, so we don't expect this bug to be fixed, but we want to document it in case other person are impacted.

      What happened was: the instance crashed, and we weren't able to start it up again (see the attached partial-backtrace.txt), then we managed to have it start with the additional options "innodb_force_recovery = 3" and "innodb_purge_threads = 0". However, when we remove these options, the server crash during startup again. So the bug is reproducible on our hand,

      The following is the environment of the crashed instance:

      # cat /etc/debian_version
      10.13
      # dpkg -l mariadb-server-10.3
      Desired=Unknown/Install/Remove/Purge/Hold
      | Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
      |/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
      ||/ Name                Version             Architecture Description
      +++-===================-===================-============-=================================
      ii  mariadb-server-10.3 1:10.3.34-0+deb10u1 amd64        MariaDB database server binaries
      # mariadb --version
      mariadb  Ver 15.1 Distrib 10.3.39-MariaDB, for debian-linux-gnu (x86_64) using readline 5.2
      # mysqld --print-defaults or SHOW VARIABLES
      mysqld would have been started with the following arguments:
      --user=mysql --pid-file=/run/mysqld/mysqld.pid --socket=/run/mysqld/mysqld.sock --basedir=/usr --datadir=/var/lib/mysql --tmpdir=/tmp --lc-messages-dir=/usr/share/mysql --bind-address=127.0.0.1 --query_cache_size=16M --log_error=/var/log/mysql/error.log --expire_logs_days=10 --character-set-server=utf8mb4 --collation-server=utf8mb4_general_ci --max_connections=250 --back_log=100 --max_connect_errors=10 --slow_query_log=1 --slow_query_log_file=/var/log/mysql/mysql-slow.log --long_query_time=10 --key_buffer_size=512M --max_allowed_packet=64M --thread_stack=192K --thread_cache_size=1 --max_heap_table_size=64M --table_open_cache=4096 --table_definition_cache=4096 --query_cache_limit=8M --query_cache_size=256M --query_cache_type=1 --max_heap_table_size=128M --tmp_table_size=128M --innodb_file_per_table --innodb_buffer_pool_size=512M --innodb_thread_concurrency=16 --character-set-server=utf8 --collation-server=utf8_general_ci --secure-file-priv= --bind-address=0.0.0.0 --thread_cache_size=1 --innodb_buffer_pool_size=4815M --tmpdir=/home/mysqltmp --innodb_log_file_size=256M --log_bin=/srv/mysql_binlogs/mysql-bin.log --expire_logs_days=5 --max_binlog_size=100M --binlog_format=mixed --sync_binlog=1 --gtid_domain_id=1 --server-id=1 --auto-increment-increment=10 --auto-increment-offset=2 --gtid_strict_mode=On --report-host=10.0.14.154 --report-port=3306 or SHOW VARIABLES
      

      Note, the exact version of the mariadb-server package may not be consistent in the different logs (10.3.34-0+deb10u1 and 10.3.39-MariaDB-0+deb10u2) because I wasn't able to get the debug symbols for the initial version (10.3.39-MariaDB-0+deb10u2) and ended up downgrading the package to get a full back trace. The crash occurred with both package version.

      Note the attached innodb-status.txt if from the instance with innodb_force_recovery set.

      I'll also report this bug downstream on Debian, but I have even less hope for it to be fixed there.

      If I've forgotten to share some information, I'll be happy to provide it.

      Cheers,

      • Brice Waegeneire

      Attachments

        Activity

          People

            Unassigned Unassigned
            bwaegeneire Brice Waegeneire
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Git Integration

                Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.