Uploaded image for project: 'MariaDB Server'
  1. MariaDB Server
  2. MDEV-6822

gcache.page files are not deleted

    XMLWordPrintable

Details

    • Bug
    • Status: Closed (View Workflow)
    • Major
    • Resolution: Fixed
    • 10.0.12-galera
    • 10.0.17-galera
    • Galera
    • Server: Dell 720xd, SSD Raid 10, 128 GB RAM, Gigabit link
      OS: Debian 7
      Nodes: Three
      DB Version: MariaDB 10.0.12 Galera Bundle
      wsrep provider: 25.3.5-wheezy(rXXXX)

    Description

      Hello all,

      we are using MariaDB-Galera-Cluster 10.0.12 and we ran (again) into the problem that lots of gcache.page-files are created but not deleted.
      I'am wondering why we have gcache.page-files at all because we set gcache.size to 50G! However, this problem eats up all disk space within a few days.

      Node 1:

      1. ls -lh gcache*
        rw------ 1 mysql mysql 256M Sep 24 08:00 gcache.page.000080

      Node 2:

      1. ls -lh gcache*
        rw------ 1 mysql mysql 256M Sep 21 12:22 gcache.page.000000
        rw------ 1 mysql mysql 256M Sep 21 13:25 gcache.page.000001
        rw------ 1 mysql mysql 256M Sep 21 14:39 gcache.page.000002
        rw------ 1 mysql mysql 256M Sep 21 15:55 gcache.page.000003
        rw------ 1 mysql mysql 256M Sep 21 16:59 gcache.page.000004
        rw------ 1 mysql mysql 256M Sep 21 18:03 gcache.page.000005
        rw------ 1 mysql mysql 256M Sep 21 19:07 gcache.page.000006
        rw------ 1 mysql mysql 256M Sep 21 20:03 gcache.page.000007
        rw------ 1 mysql mysql 256M Sep 21 21:01 gcache.page.000008
        rw------ 1 mysql mysql 256M Sep 21 22:04 gcache.page.000009
        rw------ 1 mysql mysql 256M Sep 21 23:23 gcache.page.000010
        rw------ 1 mysql mysql 256M Sep 22 00:43 gcache.page.000011
        rw------ 1 mysql mysql 256M Sep 22 02:22 gcache.page.000012
        rw------ 1 mysql mysql 256M Sep 22 04:19 gcache.page.000013
        rw------ 1 mysql mysql 256M Sep 22 06:14 gcache.page.000014
        rw------ 1 mysql mysql 256M Sep 22 07:29 gcache.page.000015
        rw------ 1 mysql mysql 256M Sep 22 08:44 gcache.page.000016
        rw------ 1 mysql mysql 256M Sep 22 09:55 gcache.page.000017
        rw------ 1 mysql mysql 256M Sep 22 10:56 gcache.page.000018
        rw------ 1 mysql mysql 256M Sep 22 11:57 gcache.page.000019
        rw------ 1 mysql mysql 256M Sep 22 12:54 gcache.page.000020
        rw------ 1 mysql mysql 256M Sep 22 13:56 gcache.page.000021
        rw------ 1 mysql mysql 256M Sep 22 14:59 gcache.page.000022
        rw------ 1 mysql mysql 256M Sep 22 16:02 gcache.page.000023
        rw------ 1 mysql mysql 256M Sep 22 17:07 gcache.page.000024
        rw------ 1 mysql mysql 256M Sep 22 18:04 gcache.page.000025
        rw------ 1 mysql mysql 256M Sep 22 19:15 gcache.page.000026
        rw------ 1 mysql mysql 256M Sep 22 20:26 gcache.page.000027
        rw------ 1 mysql mysql 256M Sep 22 21:36 gcache.page.000028
        rw------ 1 mysql mysql 256M Sep 22 22:51 gcache.page.000029
        rw------ 1 mysql mysql 256M Sep 23 00:38 gcache.page.000030
        rw------ 1 mysql mysql 256M Sep 23 02:32 gcache.page.000031
        rw------ 1 mysql mysql 256M Sep 23 04:54 gcache.page.000032
        rw------ 1 mysql mysql 256M Sep 23 06:52 gcache.page.000033
        rw------ 1 mysql mysql 256M Sep 23 08:00 gcache.page.000034
        rw------ 1 mysql mysql 256M Sep 23 09:14 gcache.page.000035
        rw------ 1 mysql mysql 256M Sep 23 10:19 gcache.page.000036
        rw------ 1 mysql mysql 256M Sep 23 11:29 gcache.page.000037
        rw------ 1 mysql mysql 256M Sep 23 12:34 gcache.page.000038
        rw------ 1 mysql mysql 256M Sep 23 13:43 gcache.page.000039
        rw------ 1 mysql mysql 256M Sep 23 14:48 gcache.page.000040
        rw------ 1 mysql mysql 256M Sep 23 15:49 gcache.page.000041
        rw------ 1 mysql mysql 256M Sep 23 16:53 gcache.page.000042
        rw------ 1 mysql mysql 256M Sep 23 17:56 gcache.page.000043
        rw------ 1 mysql mysql 256M Sep 23 19:04 gcache.page.000044
        rw------ 1 mysql mysql 256M Sep 23 20:08 gcache.page.000045
        rw------ 1 mysql mysql 256M Sep 23 21:11 gcache.page.000046
        rw------ 1 mysql mysql 256M Sep 23 22:26 gcache.page.000047
        rw------ 1 mysql mysql 256M Sep 23 23:30 gcache.page.000048
        rw------ 1 mysql mysql 256M Sep 24 00:53 gcache.page.000049
        rw------ 1 mysql mysql 256M Sep 24 02:34 gcache.page.000050
        rw------ 1 mysql mysql 256M Sep 24 04:29 gcache.page.000051
        rw------ 1 mysql mysql 256M Sep 24 06:14 gcache.page.000052
        rw------ 1 mysql mysql 256M Sep 24 07:28 gcache.page.000053
        rw------ 1 mysql mysql 256M Sep 24 07:57 gcache.page.000054

      Node 3:

      1. ls -ls gcache*
        262148 rw------ 1 mysql mysql 256M Sep 24 01:29 gcache.page.000000
        262120 rw------ 1 mysql mysql 256M Sep 24 03:22 gcache.page.000001
        262148 rw------ 1 mysql mysql 256M Sep 24 05:14 gcache.page.000002
        262148 rw------ 1 mysql mysql 256M Sep 24 06:49 gcache.page.000003
        262148 rw------ 1 mysql mysql 256M Sep 24 07:53 gcache.page.000004
        262148 rw------ 1 mysql mysql 256M Sep 24 08:01 gcache.page.000005

      So all this started suddenly on Sep 21. We haven't changed anything that day and you can't find something unusual in the error logs. (attached)
      This happend one time before, we "fixed" the problem by stopping the servers successively and deleting the gcache.page-files physically. This worked for about two weeks and now the problem ist back.
      Even more confusing: In contrast to Node 2 and 3, Node 1 creates the gcache.page-files (whatever the reason is) and deletes it again after filling it up..

      All this happens even due to very low load times (at night).

      I also attatched the my.cnf we use.
      I reported this problem at codership too but i haven't received a response until now. (https://groups.google.com/forum/?hl=de#!topic/codership-team/1OKXmpzfmwc)

      I would be very thankful for any help regarding this issue.

      Thank you and best wishes
      Philipp

      Attachments

        1. node1.cnf
          5 kB
          Philipp Kraut
        2. node1.errors.log_8308.20141113_2.txt.gz
          4.51 MB
          Stoykov
        3. node1.log
          295 kB
          Philipp Kraut
        4. node2.cnf
          5 kB
          Philipp Kraut
        5. node2.errors.log_8308.20141113_2.txt.gz
          409 kB
          Stoykov
        6. node2.log
          52 kB
          Philipp Kraut
        7. node3.cnf
          5 kB
          Philipp Kraut
        8. node3.log
          55 kB
          Philipp Kraut

        Issue Links

          Activity

            People

              nirbhay_c Nirbhay Choubey (Inactive)
              p.kraut Philipp Kraut
              Votes:
              2 Vote for this issue
              Watchers:
              8 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Git Integration

                  Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.