Uploaded image for project: 'MariaDB Server'
  1. MariaDB Server
  2. MDEV-24423

MariaDB 10.3 crashing and restarting intermittently - segfault at 0

    XMLWordPrintable

Details

    Description

      Hello,

      We are using a regular Debian 10 server with latest MariaDB 10.3.27. It use to work nice for months, but since 1 week, we are facing some regular crashes after a few hours of run. Then applications (zabbix, etc...) loss the DB connections and some transactions are broken.

      System specs : - 4 vCPU - 10G of RAM - Disks are some LUNs on an EMC VNX

      Here is an example of the syslog messages:

      Dec 16 19:44:48 mysqlbddvprd1 kernel: [503847.749484] show_signal_msg: 18 callbacks suppressed
      Dec 16 19:44:48 mysqlbddvprd1 kernel: [503847.749487] mysqld[60145]: segfault at 0 ip 0000557197badfb3 sp 00007f2dbbe2d310 error 6 in mysqld[5571973f0000+80a000]
      Dec 16 19:44:48 mysqlbddvprd1 kernel: [503847.749491] Code: c7 45 00 00 00 00 00 8b 7d cc 4c 89 e2 4c 89 f6 e8 52 2f 84 ff 49 89 c7 49 39 c4 0f 84 06 01 00 00 e8 21 18 00 00 41 8b 4d 00 <89> 08 85 c9 74 37 49 83 ff ff 0f 84 ad 00 00 00 f6 c3 06 75 28 4d
      Dec 16 19:44:48 mysqlbddvprd1 systemd[1]: mariadb.service: Main process exited, code=killed, status=11/SEGV
      Dec 16 19:44:48 mysqlbddvprd1 systemd[1]: mariadb.service: Failed with result 'signal'.
      Dec 16 19:44:53 mysqlbddvprd1 systemd[1]: mariadb.service: Service RestartSec=5s expired, scheduling restart.
      Dec 16 19:44:53 mysqlbddvprd1 systemd[1]: mariadb.service: Scheduled restart job, restart counter is at 1.
      Dec 16 19:44:53 mysqlbddvprd1 systemd[1]: Stopped MariaDB 10.3.27 database server.
       
      Dec 16 19:44:53 mysqlbddvprd1 systemd[1]: Starting MariaDB 10.3.27 database server...
      Dec 16 19:44:53 mysqlbddvprd1 mysqld[43693]: 2020-12-16 19:44:53 0 [Note] /usr/sbin/mysqld (mysqld 10.3.27-MariaDB-0+deb10u1) starting as process 43693 ...
      Dec 16 19:45:00 mysqlbddvprd1 systemd[1]: Started MariaDB 10.3.27 database server.
       
      Dec 16 19:45:00 mysqlbddvprd1 /etc/mysql/debian-start[43750]: Upgrading MySQL tables if necessary.
      Dec 16 19:45:00 mysqlbddvprd1 /etc/mysql/debian-start[43753]: /usr/bin/mysql_upgrade: the '--basedir' option is always ignored
      Dec 16 19:45:00 mysqlbddvprd1 /etc/mysql/debian-start[43753]: Looking for 'mysql' as: /usr/bin/mysql
      Dec 16 19:45:00 mysqlbddvprd1 /etc/mysql/debian-start[43753]: Looking for 'mysqlcheck' as: /usr/bin/mysqlcheck
      Dec 16 19:45:00 mysqlbddvprd1 /etc/mysql/debian-start[43753]: This installation of MySQL is already upgraded to 10.3.27-MariaDB, use --force if you still need to run mysql_upgrade
      Dec 16 19:45:00 mysqlbddvprd1 /etc/mysql/debian-start[43765]: Checking for insecure root accounts.
      Dec 16 19:45:00 mysqlbddvprd1 /etc/mysql/debian-start[43769]: Triggering myisam-recover for all MyISAM tables and aria-recover for all Aria tables
      

      And here is a part of the conf file we use: /etc/mysql/mariadb.conf.d/50-server.cnf

      #
      # * Fine Tuning
      #
      myisam_recover_options  = BACKUP
      max_connections         = 150
       
      #
      # * Fine Tuning for InnoDB
      #
      innodb_buffer_pool_size = 7G            # Go up to 70% to 80% of your available RAM
      innodb_buffer_pool_instances = 4        # Bigger if huge InnoDB Buffer Pool or high concurrency
       
      innodb_file_per_table   = 1             # Is the recommended way nowadays
      innodb_flush_method     = O_DIRECT
      innodb_write_io_threads = 8             # If you have a strong I/O system or SSD
      innodb_read_io_threads  = 8             # If you have a strong I/O system or SSD
      innodb_io_capacity      = 1000          # If you have a strong I/O system or SSD
       
      innodb_flush_log_at_trx_commit = 1      # 1 for durability, 0 or 2 for performance
      innodb_log_buffer_size  = 8M            # Bigger if innodb_flush_log_at_trx_commit = 0
      innodb_log_file_size    = 128M          # Bigger means more write throughput but longer recovery time
       
      #
      # * Query Cache Configuration
      #
      query_cache_type        = 0
      query_cache_size        = 0
      

      Error.log files are linked.
      Any comments are welcome.

      Best regards,

      Attachments

        1. error.log
          3.63 MB
          Stéphane BOCQUET
        2. error.log.1
          3.77 MB
          Stéphane BOCQUET

        Activity

          People

            marko Marko Mäkelä
            sbocquet Stéphane BOCQUET
            Votes:
            3 Vote for this issue
            Watchers:
            8 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Git Integration

                Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.