Uploaded image for project: 'MariaDB Server'
  1. MariaDB Server
  2. MDEV-10508

Mariadb crash on out of disk space during dump import

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Closed (View Workflow)
    • Priority: Critical
    • Resolution: Fixed
    • Affects Version/s: 10.1
    • Fix Version/s: 10.1.18
    • Component/s: OTHER
    • Labels:
      None
    • Environment:
      CentOS x86_64 6.8, Amazon AWS c4.4xl instance
    • Sprint:
      10.1.18

      Description

      I have Galera Mariadb cluster, 6 instances, one instance where I've made sql dump import crashed with "mysqld got signal 11". The reason I suppose - /var/log/ partition where all mysqld logs and slow queries logs are set was 100% filled because of slow_query_log enabled during dump import, yes - out of disk space for log files is bad, but definitely should not cause mysqld crash.

      Here is disk sized during MySQL crash:

      Filesystem           Size  Used Avail Use% Mounted on
      /dev/xvda1           9.9G  9.4G     0 100% /         
      tmpfs                 32G     0   32G   0% /dev/shm  
      /dev/mapper/vg0-tmp   52G   11G   41G  20% /tmp      
      /dev/md127           2.0T  974G  975G  50% /var/lib/mysql
      

      Here is mysqld.log output during crash:

      2016-08-06 20:19:31 139895589890816 [Note] InnoDB: Online DDL : Completed
      2016-08-07  7:46:18 139945511483136 [Note] WSREP: (46cc19ec, 'tcp://0.0.0.0:4567') turning message relay requesting on, nonlive peers: tcp://10.10.1.93:456
      7
      2016-08-07  7:46:19 139945511483136 [Note] WSREP: (46cc19ec, 'tcp://0.0.0.0:4567') reconnecting to c3b6c958 (tcp://10.10.1.93:4567), attempt 0
      2016-08-07  7:46:23 139945511483136 [Note] WSREP: (46cc19ec, 'tcp://0.0.0.0:4567') turning message relay requesting off
      2016-08-07  7:46:27 139945511483136 [Note] WSREP: (46cc19ec, 'tcp://0.0.0.0:4567') turning message relay requesting on, nonlive peers: tcp://10.10.1.93:456
      7
      2016-08-07  7:46:28 139945511483136 [Note] WSREP: (46cc19ec, 'tcp://0.0.0.0:4567') reconnecting to c3b6c958 (tcp://10.10.1.93:4567), attempt 0
      2016-08-07  7:46:32 139945511483136 [Note] WSREP: (46cc19ec, 'tcp://0.0.0.0:4567') turning message relay requesting off
      2016-08-07  7:47:01 139945511483136 [Note] WSREP: (46cc19ec, 'tcp://0.0.0.0:4567') turning message relay requesting on, nonlive peers: tcp://10.10.1.92:456
      7
      2016-08-07  7:47:02 139945511483136 [Note] WSREP: (46cc19ec, 'tcp://0.0.0.0:4567') reconnecting to e1ff2a6f (tcp://10.10.1.92:4567), attempt 0
      2016-08-07  7:47:07 139945511483136 [Note] WSREP: (46cc19ec, 'tcp://0.0.0.0:4567') reconnecting to 73a721b1 (tcp://10.10.1.221:4567), attempt 0
      2016-08-07  7:47:22 139945511483136 [Note] WSREP: (46cc19ec, 'tcp://0.0.0.0:4567') turning message relay requesting off
      2016-08-07  7:47:32 139945511483136 [Note] WSREP: (46cc19ec, 'tcp://0.0.0.0:4567') turning message relay requesting on, nonlive peers: tcp://10.10.1.93:456
      7
      2016-08-07  7:47:33 139945511483136 [Note] WSREP: (46cc19ec, 'tcp://0.0.0.0:4567') reconnecting to c3b6c958 (tcp://10.10.1.93:4567), attempt 0
      2016-08-07  7:48:05 139945511483136 [Note] WSREP: (46cc19ec, 'tcp://0.0.0.0:4567') turning message relay requesting off
      2016-08-07  7:48:47 139945511483136 [Note] WSREP: (46cc19ec, 'tcp://0.0.0.0:4567') turning message relay requesting on, nonlive peers: tcp://10.10.1.221:45
      67
      2016-08-07  7:48:48 139945511483136 [Note] WSREP: (46cc19ec, 'tcp://0.0.0.0:4567') reconnecting to 73a721b1 (tcp://10.10.1.221:4567), attempt 0
      2016-08-07  7:48:51 139945511483136 [Note] WSREP: (46cc19ec, 'tcp://0.0.0.0:4567') turning message relay requesting off
      

      2016-08-07  7:49:29 139891662277376 [ERROR] Error writing file '/var/log/mysql/slow-queries.log' (errno: 1 "Operation not permitted")
      160807  7:49:30 [ERROR] mysqld got signal 11 ;
      This could be because you hit a bug. It is also possible that this binary
      or one of the libraries it was linked against is corrupt, improperly built,
      or misconfigured. This error can also be caused by malfunctioning hardware.
       
      To report this bug, see https://mariadb.com/kb/en/reporting-bugs
       
      We will try our best to scrape up some info that will hopefully help
      diagnose the problem, but since we have already160807 07:49:34 mysqld_safe Number of processes running now: 0
      160807 07:49:34 mysqld_safe WSREP: not restarting wsrep node automatically
      160807 07:49:34 mysqld_safe mysqld from pid file /var/lib/mysql/db-east.edoctrina.org.pid ended
      

        Attachments

          Issue Links

            Activity

              People

              Assignee:
              serg Sergei Golubchik
              Reporter:
              kpvmaria Kaidalov Pavel
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved: