Uploaded image for project: 'MariaDB Server'
  1. MariaDB Server
  2. MDEV-14047

"Too many open files"

Details

    Description

      Using default settings, I got the following error, once the DB reached 250 GB:

      2017-10-10  7:31:56 140711626262272 [ERROR] LibRocksDB:[/opt/slapgrid/7c3e140d2be16dd0fc4001874fc875be/parts/mariadb__compile__/mariadb-10.2.9/storage/rocksdb/rocksdb/db/compaction_job.cc:1240] [default] [JOB 13239] OpenCompactionOutputFiles for table #28903 fails at NewWritableFile with status IO error: While open a file for appending: ./.rocksdb/028903.sst: Too many open files
      		 
      2017-10-10  7:31:56 140711626262272 [ERROR] LibRocksDB:[/opt/slapgrid/7c3e140d2be16dd0fc4001874fc875be/parts/mariadb__compile__/mariadb-10.2.9/storage/rocksdb/rocksdb/db/db_impl_compaction_flush.cc:1335] Waiting after background compaction error: IO error: While open a file for appending: ./.rocksdb/028903.sst: Too many open files, Accumulated background error counts: 1
      		 
      mysqld: /opt/slapgrid/7c3e140d2be16dd0fc4001874fc875be/parts/mariadb__compile__/mariadb-10.2.9/storage/rocksdb/rocksdb/db/db_impl.cc:710: void rocksdb::DBImpl::MarkLogsSynced(uint64_t, bool, const rocksdb::Status&): Assertion `log.getting_synced' failed.
      		 
      171010  7:31:57 [ERROR] mysqld got signal 6 ;

      I increased limits so that it can reach 6 TB:

      • rocksdb_max_open_files = 32768
      • rocksdb_merge_buf_size = 256M

      However, RocksDB continues to split the DB in files of 64 MB so I'm not sure I changed the correct variable.

      Anyway, we think that RocksDB should adapt automatically as the DB grows.

      edit: Increasing rocksdb_max_open_files does nothing if open_files_limit is not also increased. Maybe one of them should be calculated automatically from the other.

      Attachments

        Issue Links

          relates to

          Bug - A problem which impairs or prevents the functions of the product. MDEV-14220 (draft) set global rocksdb_pause_background_work=1 freezes

          • Major - Major loss of function.
          • Closed

          Bug - A problem which impairs or prevents the functions of the product. MDEV-13975 [out of disk space assert] I was uploading 100 million records to a rocsdb table and Mariadb crashed

          • Major - Major loss of function.
          • Open

          Activity

            Transition Time In Source Status Execution Times
            Andrii Nikitin (Inactive) made transition -
            Open Confirmed
            		19d 6h 50m 1

            People

              psergei Sergei Petrunia
              jmuchemb Julien Muchembled
              Votes:
              0 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

                Created:
                Updated:

                Git Integration

                  Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.