Uploaded image for project: 'MariaDB Server'
  1. MariaDB Server
  2. MDEV-24423

MariaDB 10.3 crashing and restarting intermittently - segfault at 0

Details

    Description

      Hello,

      We are using a regular Debian 10 server with latest MariaDB 10.3.27. It use to work nice for months, but since 1 week, we are facing some regular crashes after a few hours of run. Then applications (zabbix, etc...) loss the DB connections and some transactions are broken.

      System specs : - 4 vCPU - 10G of RAM - Disks are some LUNs on an EMC VNX

      Here is an example of the syslog messages:

      Dec 16 19:44:48 mysqlbddvprd1 kernel: [503847.749484] show_signal_msg: 18 callbacks suppressed
      		 
      Dec 16 19:44:48 mysqlbddvprd1 kernel: [503847.749487] mysqld[60145]: segfault at 0 ip 0000557197badfb3 sp 00007f2dbbe2d310 error 6 in mysqld[5571973f0000+80a000]
      		 
      Dec 16 19:44:48 mysqlbddvprd1 kernel: [503847.749491] Code: c7 45 00 00 00 00 00 8b 7d cc 4c 89 e2 4c 89 f6 e8 52 2f 84 ff 49 89 c7 49 39 c4 0f 84 06 01 00 00 e8 21 18 00 00 41 8b 4d 00 <89> 08 85 c9 74 37 49 83 ff ff 0f 84 ad 00 00 00 f6 c3 06 75 28 4d
      		 
      Dec 16 19:44:48 mysqlbddvprd1 systemd[1]: mariadb.service: Main process exited, code=killed, status=11/SEGV
      		 
      Dec 16 19:44:48 mysqlbddvprd1 systemd[1]: mariadb.service: Failed with result 'signal'.
      		 
      Dec 16 19:44:53 mysqlbddvprd1 systemd[1]: mariadb.service: Service RestartSec=5s expired, scheduling restart.
      		 
      Dec 16 19:44:53 mysqlbddvprd1 systemd[1]: mariadb.service: Scheduled restart job, restart counter is at 1.
      		 
      Dec 16 19:44:53 mysqlbddvprd1 systemd[1]: Stopped MariaDB 10.3.27 database server.
      		 
       
      		 
      Dec 16 19:44:53 mysqlbddvprd1 systemd[1]: Starting MariaDB 10.3.27 database server...
      		 
      Dec 16 19:44:53 mysqlbddvprd1 mysqld[43693]: 2020-12-16 19:44:53 0 [Note] /usr/sbin/mysqld (mysqld 10.3.27-MariaDB-0+deb10u1) starting as process 43693 ...
      		 
      Dec 16 19:45:00 mysqlbddvprd1 systemd[1]: Started MariaDB 10.3.27 database server.
      		 
       
      		 
      Dec 16 19:45:00 mysqlbddvprd1 /etc/mysql/debian-start[43750]: Upgrading MySQL tables if necessary.
      		 
      Dec 16 19:45:00 mysqlbddvprd1 /etc/mysql/debian-start[43753]: /usr/bin/mysql_upgrade: the '--basedir' option is always ignored
      		 
      Dec 16 19:45:00 mysqlbddvprd1 /etc/mysql/debian-start[43753]: Looking for 'mysql' as: /usr/bin/mysql
      		 
      Dec 16 19:45:00 mysqlbddvprd1 /etc/mysql/debian-start[43753]: Looking for 'mysqlcheck' as: /usr/bin/mysqlcheck
      		 
      Dec 16 19:45:00 mysqlbddvprd1 /etc/mysql/debian-start[43753]: This installation of MySQL is already upgraded to 10.3.27-MariaDB, use --force if you still need to run mysql_upgrade
      		 
      Dec 16 19:45:00 mysqlbddvprd1 /etc/mysql/debian-start[43765]: Checking for insecure root accounts.
      		 
      Dec 16 19:45:00 mysqlbddvprd1 /etc/mysql/debian-start[43769]: Triggering myisam-recover for all MyISAM tables and aria-recover for all Aria tables

      And here is a part of the conf file we use: /etc/mysql/mariadb.conf.d/50-server.cnf

      #
      		 
      # * Fine Tuning
      		 
      #
      		 
      myisam_recover_options  = BACKUP
      		 
      max_connections         = 150
      		 
       
      		 
      #
      		 
      # * Fine Tuning for InnoDB
      		 
      #
      		 
      innodb_buffer_pool_size = 7G            # Go up to 70% to 80% of your available RAM
      		 
      innodb_buffer_pool_instances = 4        # Bigger if huge InnoDB Buffer Pool or high concurrency
      		 
       
      		 
      innodb_file_per_table   = 1             # Is the recommended way nowadays
      		 
      innodb_flush_method     = O_DIRECT
      		 
      innodb_write_io_threads = 8             # If you have a strong I/O system or SSD
      		 
      innodb_read_io_threads  = 8             # If you have a strong I/O system or SSD
      		 
      innodb_io_capacity      = 1000          # If you have a strong I/O system or SSD
      		 
       
      		 
      innodb_flush_log_at_trx_commit = 1      # 1 for durability, 0 or 2 for performance
      		 
      innodb_log_buffer_size  = 8M            # Bigger if innodb_flush_log_at_trx_commit = 0
      		 
      innodb_log_file_size    = 128M          # Bigger means more write throughput but longer recovery time
      		 
       
      		 
      #
      		 
      # * Query Cache Configuration
      		 
      #
      		 
      query_cache_type        = 0
      		 
      query_cache_size        = 0

      Error.log files are linked.
      Any comments are welcome.

      Best regards,

      Attachments

        1. error.log
          3.63 MB
        2. error.log.1
          3.77 MB

        Activity

          Ascending order - Click to sort in descending order
          sbocquet Stéphane BOCQUET created issue -
          sbocquet Stéphane BOCQUET made changes -
          Field Original Value New Value
          Description Hello,

          We are using a regular Debian 10 server with latest MariaDB 10.3.27. It use to work nice for months, but since 1 week, we are facing some regular crashes after a few hours of run. Then applications (zabbix, etc...) loss the DB connections and some transactions are broken.

          System specs : - 4 vCPU - 10G of RAM - Disks are some LUNs on an EMC VNX

          Here is an example of the syslog messages:

          {code:java}
          Dec 14 16:05:08 mysqlbddvprd1 kernel: [503847.749484] show_signal_msg: 18 callbacks suppressed
          Dec 14 16:05:08 mysqlbddvprd1 kernel: [503847.749487] mysqld[60145]: segfault at 0 ip 0000557197badfb3 sp 00007f2dbbe2d310 error 6 in mysqld[5571973f0000+80a000]
          Dec 14 16:05:08 mysqlbddvprd1 kernel: [503847.749491] Code: c7 45 00 00 00 00 00 8b 7d cc 4c 89 e2 4c 89 f6 e8 52 2f 84 ff 49 89 c7 49 39 c4 0f 84 06 01 00 00 e8 21 18 00 00 41 8b 4d 00 <89> 08 85 c9 74 37 49 83 ff ff 0f 84 ad 00 00 00 f6 c3 06 75 28 4d
          Dec 14 16:05:08 mysqlbddvprd1 systemd[1]: mariadb.service: Main process exited, code=killed, status=11/SEGV
          Dec 14 16:05:08 mysqlbddvprd1 systemd[1]: mariadb.service: Failed with result 'signal'.
          Dec 14 16:05:13 mysqlbddvprd1 systemd[1]: mariadb.service: Service RestartSec=5s expired, scheduling restart.
          Dec 14 16:05:13 mysqlbddvprd1 systemd[1]: mariadb.service: Scheduled restart job, restart counter is at 5.
          Dec 14 16:05:13 mysqlbddvprd1 systemd[1]: Stopped MariaDB 10.3.27 database server.

          Dec 14 16:05:13 mysqlbddvprd1 systemd[1]: Starting MariaDB 10.3.27 database server...
          Dec 14 16:05:14 mysqlbddvprd1 sssd[be[ad.mediapost.fr]]: GSSAPI Error: Unspecified GSS failure. Minor code may provide more information (Server not found in Kerberos database)
          Dec 14 16:05:14 mysqlbddvprd1 mysqld[52124]: 2020-12-14 16:05:14 0 [Note] /usr/sbin/mysqld (mysqld 10.3.27-MariaDB-0+deb10u1) starting as process 52124 ...
          Dec 14 16:05:23 mysqlbddvprd1 systemd[1]: Started MariaDB 10.3.27 database server.

          Dec 14 16:05:23 mysqlbddvprd1 /etc/mysql/debian-start[52181]: Upgrading MySQL tables if necessary.
          Dec 14 16:05:23 mysqlbddvprd1 /etc/mysql/debian-start[52184]: /usr/bin/mysql_upgrade: the '--basedir' option is always ignored
          Dec 14 16:05:23 mysqlbddvprd1 /etc/mysql/debian-start[52184]: Looking for 'mysql' as: /usr/bin/mysql
          Dec 14 16:05:23 mysqlbddvprd1 /etc/mysql/debian-start[52184]: Looking for 'mysqlcheck' as: /usr/bin/mysqlcheck
          Dec 14 16:05:23 mysqlbddvprd1 /etc/mysql/debian-start[52184]: Version check failed. Got the following error when calling the 'mysql' command line client
          Dec 14 16:05:23 mysqlbddvprd1 /etc/mysql/debian-start[52184]: ERROR 1045 (28000): Access denied for user 'root'@'localhost' (using password: NO)
          Dec 14 16:05:23 mysqlbddvprd1 /etc/mysql/debian-start[52184]: FATAL ERROR: Upgrade failed
          Dec 14 16:05:23 mysqlbddvprd1 /etc/mysql/debian-start[52197]: Checking for insecure root accounts.
          Dec 14 16:05:23 mysqlbddvprd1 debian-start[52179]: ERROR 1045 (28000): Access denied for user 'root'@'localhost' (using password: NO)
          Dec 14 16:09:01 mysqlbddvprd1 CRON[52474]: (root) CMD ( [ -x /usr/lib/php/sessionclean ] && if [ ! -d /run/systemd/system ]; then /usr/lib/php/sessionclean; fi)
          Dec 14 16:09:01 mysqlbddvprd1 systemd[1]: Starting Clean php session files...
          Dec 14 16:09:02 mysqlbddvprd1 systemd[1]: phpsessionclean.service: Succeeded.
          Dec 14 16:09:02 mysqlbddvprd1 systemd[1]: Started Clean php session files.
          {code}

          And here is a part of the conf file we use: /etc/mysql/mariadb.conf.d/50-server.cnf

          {code:java}
          #
          # * Fine Tuning
          #
          myisam_recover_options = BACKUP
          max_connections = 150

          #
          # * Fine Tuning for InnoDB
          #
          innodb_buffer_pool_size = 7G # Go up to 70% to 80% of your available RAM
          innodb_buffer_pool_instances = 4 # Bigger if huge InnoDB Buffer Pool or high concurrency

          innodb_file_per_table = 1 # Is the recommended way nowadays
          innodb_flush_method = O_DIRECT
          innodb_write_io_threads = 8 # If you have a strong I/O system or SSD
          innodb_read_io_threads = 8 # If you have a strong I/O system or SSD
          innodb_io_capacity = 1000 # If you have a strong I/O system or SSD

          innodb_flush_log_at_trx_commit = 1 # 1 for durability, 0 or 2 for performance
          innodb_log_buffer_size = 8M # Bigger if innodb_flush_log_at_trx_commit = 0
          innodb_log_file_size = 128M # Bigger means more write throughput but longer recovery time

          #
          # * Query Cache Configuration
          #
          query_cache_type = 0
          query_cache_size = 0
          {code}

          Error.log files are linked.
          Any comments are welcome.

          Best regards,           		Hello,

          We are using a regular Debian 10 server with latest MariaDB 10.3.27. It use to work nice for months, but since 1 week, we are facing some regular crashes after a few hours of run. Then applications (zabbix, etc...) loss the DB connections and some transactions are broken.

          System specs : - 4 vCPU - 10G of RAM - Disks are some LUNs on an EMC VNX

          Here is an example of the syslog messages:

          {code:java}
          Dec 16 19:44:48 mysqlbddvprd1 kernel: [503847.749484] show_signal_msg: 18 callbacks suppressed
          Dec 16 19:44:48 mysqlbddvprd1 kernel: [503847.749487] mysqld[60145]: segfault at 0 ip 0000557197badfb3 sp 00007f2dbbe2d310 error 6 in mysqld[5571973f0000+80a000]
          Dec 16 19:44:48 mysqlbddvprd1 kernel: [503847.749491] Code: c7 45 00 00 00 00 00 8b 7d cc 4c 89 e2 4c 89 f6 e8 52 2f 84 ff 49 89 c7 49 39 c4 0f 84 06 01 00 00 e8 21 18 00 00 41 8b 4d 00 <89> 08 85 c9 74 37 49 83 ff ff 0f 84 ad 00 00 00 f6 c3 06 75 28 4d
          Dec 16 19:44:48 mysqlbddvprd1 systemd[1]: mariadb.service: Main process exited, code=killed, status=11/SEGV
          Dec 16 19:44:48 mysqlbddvprd1 systemd[1]: mariadb.service: Failed with result 'signal'.
          Dec 16 19:44:53 mysqlbddvprd1 systemd[1]: mariadb.service: Service RestartSec=5s expired, scheduling restart.
          Dec 16 19:44:53 mysqlbddvprd1 systemd[1]: mariadb.service: Scheduled restart job, restart counter is at 1.
          Dec 16 19:44:53 mysqlbddvprd1 systemd[1]: Stopped MariaDB 10.3.27 database server.

          Dec 16 19:44:53 mysqlbddvprd1 systemd[1]: Starting MariaDB 10.3.27 database server...
          Dec 16 19:44:53 mysqlbddvprd1 mysqld[43693]: 2020-12-16 19:44:53 0 [Note] /usr/sbin/mysqld (mysqld 10.3.27-MariaDB-0+deb10u1) starting as process 43693 ...
          Dec 16 19:45:00 mysqlbddvprd1 systemd[1]: Started MariaDB 10.3.27 database server.

          Dec 16 19:45:00 mysqlbddvprd1 /etc/mysql/debian-start[43750]: Upgrading MySQL tables if necessary.
          Dec 16 19:45:00 mysqlbddvprd1 /etc/mysql/debian-start[43753]: /usr/bin/mysql_upgrade: the '--basedir' option is always ignored
          Dec 16 19:45:00 mysqlbddvprd1 /etc/mysql/debian-start[43753]: Looking for 'mysql' as: /usr/bin/mysql
          Dec 16 19:45:00 mysqlbddvprd1 /etc/mysql/debian-start[43753]: Looking for 'mysqlcheck' as: /usr/bin/mysqlcheck
          Dec 16 19:45:00 mysqlbddvprd1 /etc/mysql/debian-start[43753]: This installation of MySQL is already upgraded to 10.3.27-MariaDB, use --force if you still need to run mysql_upgrade
          Dec 16 19:45:00 mysqlbddvprd1 /etc/mysql/debian-start[43765]: Checking for insecure root accounts.
          Dec 16 19:45:00 mysqlbddvprd1 /etc/mysql/debian-start[43769]: Triggering myisam-recover for all MyISAM tables and aria-recover for all Aria tables
          {code}

          And here is a part of the conf file we use: /etc/mysql/mariadb.conf.d/50-server.cnf

          {code:java}
          #
          # * Fine Tuning
          #
          myisam_recover_options = BACKUP
          max_connections = 150

          #
          # * Fine Tuning for InnoDB
          #
          innodb_buffer_pool_size = 7G # Go up to 70% to 80% of your available RAM
          innodb_buffer_pool_instances = 4 # Bigger if huge InnoDB Buffer Pool or high concurrency

          innodb_file_per_table = 1 # Is the recommended way nowadays
          innodb_flush_method = O_DIRECT
          innodb_write_io_threads = 8 # If you have a strong I/O system or SSD
          innodb_read_io_threads = 8 # If you have a strong I/O system or SSD
          innodb_io_capacity = 1000 # If you have a strong I/O system or SSD

          innodb_flush_log_at_trx_commit = 1 # 1 for durability, 0 or 2 for performance
          innodb_log_buffer_size = 8M # Bigger if innodb_flush_log_at_trx_commit = 0
          innodb_log_file_size = 128M # Bigger means more write throughput but longer recovery time

          #
          # * Query Cache Configuration
          #
          query_cache_type = 0
          query_cache_size = 0
          {code}

          Error.log files are linked.
          Any comments are welcome.

          Best regards,
          elenst Elena Stepanova made changes -
          Component/s Storage Engine - InnoDB [ 10129 ]
          Component/s Server [ 13907 ]
          Fix Version/s 10.3 [ 22126 ]
          Assignee Marko Mäkelä [ marko ]
          Labels crash innodb crash innodb zabbix
          serg Sergei Golubchik made changes -
          Workflow MariaDB v3 [ 117026 ] MariaDB v4 [ 142439 ]
          marko Marko Mäkelä made changes -
          Status Open [ 1 ] Needs Feedback [ 10501 ]
          elenst Elena Stepanova made changes -
          Fix Version/s N/A [ 14700 ]
          Fix Version/s 10.3 [ 22126 ]
          Resolution Incomplete [ 4 ]
          Status Needs Feedback [ 10501 ] Closed [ 6 ]

          People

            marko Marko Mäkelä
            sbocquet Stéphane BOCQUET
            Votes:
            3 Vote for this issue
            Watchers:
            8 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Git Integration

                Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.