Uploaded image for project: 'MariaDB Server'
  1. MariaDB Server
  2. MDEV-11309

MariaDB server restarts every night (at least, sometimes more) for no apparent reason

Details

    • Bug
    • Status: Closed (View Workflow)
    • Major
    • Resolution: Fixed
    • 10.1.19
    • 10.1.21
    • OTHER
    • None
    • CentOS 6.8 VM 16GB memory.

    Description

      Every night (sometimes even during the day), we get a restart of the MariaDB server (Master in a master/slave replication pair). Only messages in the log are:

      161117 22:46:39 mysqld_safe Number of processes running now: 0
      161117 22:46:39 mysqld_safe mysqld restarted
      2016-11-17 22:46:40 139737201940512 [Note] /usr/sbin/mysqld (mysqld 10.1.19-MariaDB) starting as process 10589 ...
      2016-11-17 22:46:41 139737201940512 [Note] InnoDB: Using mutexes to ref count buffer pool pages
      2016-11-17 22:46:41 139737201940512 [Note] InnoDB: The InnoDB memory heap is disabled
      

      followed by the usual start sequence. Nothing before it that it indicates errors or so

      Attachments

        1. my.cnf
          3 kB
        2. oom.txt.gz
          6.65 MB

        Issue Links

          Activity

            Another clue:

            • I have managed to reduce the frequency of OOM crashes by running every 30 mins "FLUSH TABLES"
            rgallastpsi Rafael Gallastegui added a comment - Another clue: I have managed to reduce the frequency of OOM crashes by running every 30 mins "FLUSH TABLES"

            In this case, maybe you could try reducing table_open_cache which is currently set to a quite big value of 150000.

            elenst Elena Stepanova added a comment - In this case, maybe you could try reducing table_open_cache which is currently set to a quite big value of 150000 .

            I have reduced the table_open_cache to 50000 and have been monitoring the system since yesterday.

            Is there a way to calculate ahead of time how much memory will MariaDB use as a maximum? I mean, it's kind of hard to know what value to use for some parameters, including table_open_cache, just by trial and error, especially in a production environment. table_open_cache could take days to crawl up until the system runs out of memory. I guess I'm concerned that I report to my management that the issue has been solved and find out later on that I get an OOM because another parameter pushed MariaDB to use more memory.

            rgallastpsi Rafael Gallastegui added a comment - I have reduced the table_open_cache to 50000 and have been monitoring the system since yesterday. Is there a way to calculate ahead of time how much memory will MariaDB use as a maximum? I mean, it's kind of hard to know what value to use for some parameters, including table_open_cache, just by trial and error, especially in a production environment. table_open_cache could take days to crawl up until the system runs out of memory. I guess I'm concerned that I report to my management that the issue has been solved and find out later on that I get an OOM because another parameter pushed MariaDB to use more memory.

            By the way, we appreciate the time you've spent helping us!

            rgallastpsi Rafael Gallastegui added a comment - By the way, we appreciate the time you've spent helping us!

            In MDEV-9764 we've now implemented a way to limit MariaDB memory usage per connection. Together with other limits (for global buffers) and a max number of connections limit, this allows you to limit all memory used by MariaDB Server.

            serg Sergei Golubchik added a comment - In MDEV-9764 we've now implemented a way to limit MariaDB memory usage per connection. Together with other limits (for global buffers) and a max number of connections limit, this allows you to limit all memory used by MariaDB Server.

            People

              serg Sergei Golubchik
              rgallastpsi Rafael Gallastegui
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Git Integration

                  Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.