Uploaded image for project: 'MariaDB Server'
  1. MariaDB Server
  2. MDEV-24423

MariaDB 10.3 crashing and restarting intermittently - segfault at 0

    XMLWordPrintable

Details

    Description

      Hello,

      We are using a regular Debian 10 server with latest MariaDB 10.3.27. It use to work nice for months, but since 1 week, we are facing some regular crashes after a few hours of run. Then applications (zabbix, etc...) loss the DB connections and some transactions are broken.

      System specs : - 4 vCPU - 10G of RAM - Disks are some LUNs on an EMC VNX

      Here is an example of the syslog messages:

      Dec 16 19:44:48 mysqlbddvprd1 kernel: [503847.749484] show_signal_msg: 18 callbacks suppressed
      Dec 16 19:44:48 mysqlbddvprd1 kernel: [503847.749487] mysqld[60145]: segfault at 0 ip 0000557197badfb3 sp 00007f2dbbe2d310 error 6 in mysqld[5571973f0000+80a000]
      Dec 16 19:44:48 mysqlbddvprd1 kernel: [503847.749491] Code: c7 45 00 00 00 00 00 8b 7d cc 4c 89 e2 4c 89 f6 e8 52 2f 84 ff 49 89 c7 49 39 c4 0f 84 06 01 00 00 e8 21 18 00 00 41 8b 4d 00 <89> 08 85 c9 74 37 49 83 ff ff 0f 84 ad 00 00 00 f6 c3 06 75 28 4d
      Dec 16 19:44:48 mysqlbddvprd1 systemd[1]: mariadb.service: Main process exited, code=killed, status=11/SEGV
      Dec 16 19:44:48 mysqlbddvprd1 systemd[1]: mariadb.service: Failed with result 'signal'.
      Dec 16 19:44:53 mysqlbddvprd1 systemd[1]: mariadb.service: Service RestartSec=5s expired, scheduling restart.
      Dec 16 19:44:53 mysqlbddvprd1 systemd[1]: mariadb.service: Scheduled restart job, restart counter is at 1.
      Dec 16 19:44:53 mysqlbddvprd1 systemd[1]: Stopped MariaDB 10.3.27 database server.
       
      Dec 16 19:44:53 mysqlbddvprd1 systemd[1]: Starting MariaDB 10.3.27 database server...
      Dec 16 19:44:53 mysqlbddvprd1 mysqld[43693]: 2020-12-16 19:44:53 0 [Note] /usr/sbin/mysqld (mysqld 10.3.27-MariaDB-0+deb10u1) starting as process 43693 ...
      Dec 16 19:45:00 mysqlbddvprd1 systemd[1]: Started MariaDB 10.3.27 database server.
       
      Dec 16 19:45:00 mysqlbddvprd1 /etc/mysql/debian-start[43750]: Upgrading MySQL tables if necessary.
      Dec 16 19:45:00 mysqlbddvprd1 /etc/mysql/debian-start[43753]: /usr/bin/mysql_upgrade: the '--basedir' option is always ignored
      Dec 16 19:45:00 mysqlbddvprd1 /etc/mysql/debian-start[43753]: Looking for 'mysql' as: /usr/bin/mysql
      Dec 16 19:45:00 mysqlbddvprd1 /etc/mysql/debian-start[43753]: Looking for 'mysqlcheck' as: /usr/bin/mysqlcheck
      Dec 16 19:45:00 mysqlbddvprd1 /etc/mysql/debian-start[43753]: This installation of MySQL is already upgraded to 10.3.27-MariaDB, use --force if you still need to run mysql_upgrade
      Dec 16 19:45:00 mysqlbddvprd1 /etc/mysql/debian-start[43765]: Checking for insecure root accounts.
      Dec 16 19:45:00 mysqlbddvprd1 /etc/mysql/debian-start[43769]: Triggering myisam-recover for all MyISAM tables and aria-recover for all Aria tables
      

      And here is a part of the conf file we use: /etc/mysql/mariadb.conf.d/50-server.cnf

      #
      # * Fine Tuning
      #
      myisam_recover_options  = BACKUP
      max_connections         = 150
       
      #
      # * Fine Tuning for InnoDB
      #
      innodb_buffer_pool_size = 7G            # Go up to 70% to 80% of your available RAM
      innodb_buffer_pool_instances = 4        # Bigger if huge InnoDB Buffer Pool or high concurrency
       
      innodb_file_per_table   = 1             # Is the recommended way nowadays
      innodb_flush_method     = O_DIRECT
      innodb_write_io_threads = 8             # If you have a strong I/O system or SSD
      innodb_read_io_threads  = 8             # If you have a strong I/O system or SSD
      innodb_io_capacity      = 1000          # If you have a strong I/O system or SSD
       
      innodb_flush_log_at_trx_commit = 1      # 1 for durability, 0 or 2 for performance
      innodb_log_buffer_size  = 8M            # Bigger if innodb_flush_log_at_trx_commit = 0
      innodb_log_file_size    = 128M          # Bigger means more write throughput but longer recovery time
       
      #
      # * Query Cache Configuration
      #
      query_cache_type        = 0
      query_cache_size        = 0
      

      Error.log files are linked.
      Any comments are welcome.

      Best regards,

      Attachments

        1. error.log
          3.63 MB
        2. error.log.1
          3.77 MB

        Activity

          People

            marko Marko Mäkelä
            sbocquet Stéphane BOCQUET
            Votes:
            3 Vote for this issue
            Watchers:
            8 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Git Integration

                Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.