Details
-
Bug
-
Status: Closed (View Workflow)
-
Critical
-
Resolution: Fixed
-
10.5.7, 10.5.8, 10.5.17, 10.11.2
Description
Several operations got much slower starting from 10.5.8 and apparently related to file opening/closing in handler.cc when reaching innodb_open_files limit.
The more tables you have in the instance the more visible, with some tens of thousands of tables it gets problematic.
SHOW TABLE STATUS FROM db gets hundreds of times slower
mysqldump on 600 schemas x 223 empty tables goes from 2:45 minutes to 27 minutes
Increasing innodb_open_files (with all the capping limits open_files_limit, os user limits, system limits) helps but in some case means keeping hundreds of thousands of files open.
The change was introduced in CS 10.5.8
To reproduce you need many thousands of tables(at least higher than innodb_open_files) and execute mysqldump or SHOW TABLE STATUS FROM until the limit is reached, testing with and without increasing innodb_open_files will show the difference.
I have found this if condition in handler.cc getting slower, the body is not even executed:
if (unlikely((error=open(name,mode,test_if_locked)))) |
{
|
ms = duration_cast<milliseconds>(system_clock::now().time_since_epoch()).count();
|
fprintf(stderr,"handler.cc:ha_open 3317 %" PRIu64 "\n", ms); |
if ((error == EACCES || error == EROFS) && mode == O_RDWR && |
(table->db_stat & HA_TRY_READ_ONLY))
|
{
|
table->db_stat|=HA_READ_ONLY;
|
error=open(name,O_RDONLY,test_if_locked);
|
}
|
}
|
That open() call goes to ha_heap.cc calling:
int ha_heap::open(const char *name, int mode, uint test_if_locked) ... |
Attachments
Issue Links
- relates to
-
-
- Closed
-
Activity
Sorry for the delay and confusion - yes the tests were done on the correct version but since it didn't resolve the issue I switched back to 10.6.12 before I provided the output of SHOW GLOBAL VARIABLES. I can test again using the 10.6 branch.
Barry, could you please build latest 10.6(it must contain 0cca8166f3111901019dcd33747a1a1dfd9e66d1 commit) with -DCMAKE_BUILD_TYPE=RelWithDebInfo option, then test it, and, if it still has low performance:
1. attach to the process with gdb, i.e. "gdb -p mariadbd_pid",
2. set gdb logging to file and turn off gdb pagination, i.e.
set logging on
|
set logging file 10.6-MDEV-30775-gdb.log
|
set pagination off
|
3. load gdb space_list.gdb
:
source space_list.gdb
|
4. launch function from the above gdb file:
show_space_list
|
5. quit from gdb(Ctrl+D), and share 10.6-MDEV-30775-gdb.log with us.
Sorry for the delayed reply. I can confirm latest 10.6 (fe89df42686fd41e986dc775e12ad6f3594d5bca) works fine for me. Thanks for all of the help resolving this issue!
Barry, thank you for bringing up the issue well in time for the fix to be included in the quarterly releases.
Barry, could you please confirm you are testing 7c0af5c56994f37c45d40e45fa4e15743acaeb06 commit? BTW, the changes were pushed to 10.6, so you can use the latest 10.6 from repository to test.