Andrii Nikitin (Inactive) added a comment - 2017-11-01 15:50 - edited In my understanding no matter how we try to auto-adjust rocksdb_open_files_limit - there will be practical situations where IO attempt will receive errno: 24. And the Engine must survive that error, e.g. just returning user error the same way when Server fails to open .frm file. So I suggest looking at auto-adjustment of rocksdb_open_files_limit in another ticket. AFAIK current formula for adjusting table_open_cache and max_connections is something like : open_files_limit > 9 + 2*table_open_cache + max_connections That comes with assumption that for 100 open MyISAM tables and 50 simultaneous connections server will need at least 2*100+50 file descriptors. (both .MYI and .MYD files are counted). In my understanding RocksDB may be hungry for file descriptors without correlation to neither table_open_cache nor max_connections. (E.g. huge single RocksDB table may be stored in many .sst files ). So I think we should try to investigate something like : long_size_t requested_limit = 9 + 2*configured_table_open_cache + configured_max_connections + configured_rocksdb_open_files_limit ; double koefficient = ( requested_limit < open_files_limit ) ? 1 : requested_limit / open_files_limit ; table_open_cache=configured_table_open_cache / koefficient; max_connections=configured_table_open_cache / koefficient; rocksdb_open_files_limits=configured_table_open_cache / koefficient;

Sergei Petrunia added a comment - 2017-11-02 10:20 In my understanding no matter how we try to auto-adjust rocksdb_open_files_limit - there will be practical situations where IO attempt will receive errno: 24. Well if rocksdb_open_files_limit is sufficiently low and all non-RocksDB file descriptor usage utilizes not more than open_files_limit - rocksdb_open_files_limit descriptors, we should be fine, right? And the Engine must survive that error, e.g. just returning user error the same way when Server fails to open .frm file. It could actually do better - it could try closing a few files it has open and retry. I am not sure about how complex it is implement this, though. In my understanding RocksDB may be hungry for file descriptors without correlation to neither table_open_cache nor max_connections. (E.g. huge single RocksDB table may be stored in many .sst files). Correct. So I think we should try to investigate something like : Does this imply that the values of table_open_cache and max_connections should be "auto-magically" adjusted when MyRocks plugin is loaded ?

Sergei Petrunia added a comment - 2017-11-03 13:05 - edited For the record, I was able to reproduce. I was loading data (concurrently, without bulk load modes) until the datadir was about 64G, and there were 1048 SST files. With that size, the server is not able to start when one is using ulimit -n 512 . It fails with a message: 2017-11-03 9:13:56 140046930872128 [ERROR] LibRocksDB:[/home/ubuntu/mariadb-10.2/storage/ rocksdb/rocksdb/db/version_set.cc:2624] [default] Error in committing version edit to MANIFEST: ... 2017-11-03 9:13:56 140046930872128 [ERROR] RocksDB: Error opening instance, Status Code: 5, Status: IO error: While open a file for appending: ./.rocksdb/019884.dbtmp: Too many open files Then I've started mysqld again under ulimit -n 1200 and continued data loading. Datadir grew to 1108 SST files and 68 G and then the server crashed: 2017/11/03-12:16:50.546730 7f3cf2907700 [/home/ubuntu/mariadb-10.2/storage/rocksdb/rocksdb/db/flush_job.cc:293] [default] [JOB 297] Level-0 flush table #21189: started 2017/11/03-12:16:50.546750 7f3cf2907700 [/home/ubuntu/mariadb-10.2/storage/rocksdb/rocksdb/db/flush_job.cc:324] [default] [JOB 297] Level-0 flush table #21189: 0 bytes IO error: While open a file for appending: ./.rocksdb/021189.sst: Too many open files 2017/11/03-12:16:50.546769 7f3cf2907700 [ERROR] [/home/ubuntu/mariadb-10.2/storage/rocksdb/rocksdb/db/db_impl_compaction_flush.cc:1279] Waiting after background flush error: IO error: While open a file for appending: ./.rocksdb/021189.sst: Too many open filesAccumulated background error counts: 1 Looking at what limits are there by default: ubuntu@ip-172-31-26-163:~$ ulimit -n 1024 ubuntu@ip-172-31-26-163:~$ ulimit -n -H 1048576 Started mysqld and examined its /proc/$PID/limits : Limit Soft Limit Hard Limit Units Max open files 4162 4162 files That number goes higher if one sets higher open_files_limit . I think for the moment we can de-prioritize this bug and just add a note to the tuning guide that those who which to have >200G of data need to increase their open_files_limit . anikitin any objections?

Sergei Petrunia added a comment - 2017-11-04 15:09 Also asked on MyRocks-Dev: https://groups.google.com/forum/#!topic/myrocks-dev/jp40Sk--CY8

Daniel Black added a comment - 2019-01-10 00:49 MyRocks fix https://github.com/facebook/mysql-5.6/commit/aac18139d3cffa7b3b9c19d5f9ad2d6c1d6bbbff