Uploaded image for project: 'MariaDB Server'
  1. MariaDB Server
  2. MDEV-24299

mariadb 100% memory consumption crashes mysqld

    XMLWordPrintable

Details

    Description

      Dumps on a SLES-mariadb-slave server use all available Memory+Swap. OS finally kills mysqld:
      Out of memory: Kill process 50603 (mysqld) score 978 or sacrifice child
      [553689.183105] Killed process 50603 (mysqld) total-vm:36304280kB, anon-rss:32185504kB, file-rss:0kB, shmem-rss:8kB

      OS: SLES 12.5 with 32 GB RAM + 2 GB swap

      After restart a dump of a 60 GB database with InnoDb-Tables claims all memory within minutes. It is not freed after the dump has finished. The next dump of a database (500MB) with MyIsam and 20.000 tables crashes mysqld with a good chance.
      It's just a mariadb slave server. no further user interaction.

      Some configuration parameters:

      innodb_buffer_pool_size = 28G
      innodb_log_file_size = 7G
      read_buffer_size = 256k
      read_rnd_buffer_size = 256k
      max_heap_table_size=16M
      tmp_table_size = 16M
      join_buffer_size = 256k
      key_buffer_size = 256M
      max_connect_errors = 1000
      open_files_limit = 32000
      table_definition_cache = 10000
      table_open_cache = 5400
      

      Status data around crash time:
      Dump of database mentioned above (MyIsam, 20.000 tables):

      free -th
                   total       used       free     shared    buffers     cached
      Mem:           31G        31G       245M        81M        47M       394M
      -/+ buffers/cache:        30G       686M
      Swap:         2.0G       2.0G        35M
      Total:         33G        33G       281M
      

      Variable_name   Value
      Memory_used     2439645840
      Memory_used_initial     1213447200
      Qcache_free_memory      536852776
      Variable_name   Value
      Binlog_snapshot_file    mysql-bin.000211
      Com_show_profile        0
      Com_show_profiles       0
      Created_tmp_files       5
      Innodb_num_open_files   2440
      Innodb_scrub_background_page_split_failures_out_of_filespace    0
      Open_files      1351
      Opened_files    892245
      Performance_schema_file_classes_lost    0
      Performance_schema_file_handles_lost    0
      Performance_schema_file_instances_lost  0
      Variable_name   Value
      Acl_table_grants        2
      Com_alter_table 50
      Com_alter_tablespace    0
      Com_create_table        132
      Com_create_temporary_table      42928
      Com_drop_table  6
      Com_drop_temporary_table        42865
      Com_lock_tables 121
      Com_rename_table        57
      Com_show_create_table   71397
      Com_show_open_tables    0
      Com_show_table_status   0
      Com_show_tables 133
      Com_unlock_tables       133
      Created_tmp_disk_tables 133670
      Created_tmp_tables      271943
      Open_table_definitions  10000
      Open_tables     5400
      Opened_table_definitions        142861
      Opened_tables   329535
      Performance_schema_table_handles_lost   0
      Performance_schema_table_instances_lost 0
      Slave_open_temp_tables  1
      Table_locks_immediate   944290
      Table_locks_waited      4
      Table_open_cache_active_instances       1
      Table_open_cache_hits   23994122
      Table_open_cache_misses 286655
      Table_open_cache_overflows      91586
      

      real    1m26.419s
      user    0m0.089s
      sys     0m0.058s
      

      ERROR 2002 (HY000): Can't connect to local MySQL server through socket '/var/lib/mysql/DB/mysql.sock' (111)
      crash here!
      

      Attachments

        1. gstack.out.gz
          2 kB
        2. gstack.out2.gz
          1 kB
        3. perftop.out.gz
          93 kB

        Activity

          People

            Unassigned Unassigned
            thiesmeincke Thies Meincke
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

              Created:
              Updated:

              Git Integration

                Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.