[MDEV-28327] InnoDB persistent statistics fail to update after bulk insert Created: 2022-04-15 Updated: 2023-11-13 Resolved: 2022-10-25 |
|
| Status: | Closed |
| Project: | MariaDB Server |
| Component/s: | Storage Engine - InnoDB |
| Affects Version/s: | 10.6.7, 10.6.8, 10.6.9, 10.7.5, 10.8.4, 10.9.2 |
| Fix Version/s: | 10.6.11, 10.7.7, 10.8.6, 10.9.4, 10.10.2, 10.11.1 |
| Type: | Bug | Priority: | Major |
| Reporter: | Brad | Assignee: | Thirunarayanan Balathandayuthapani |
| Resolution: | Fixed | Votes: | 4 |
| Labels: | innodb, optimizer, performance, statistics | ||
| Environment: |
Centos 7 |
||
| Attachments: |
|
||||||||||||||||||||||||||||||||||||
| Issue Links: |
|
||||||||||||||||||||||||||||||||||||
| Description |
|
I have a bunch of CentOS 7 servers that I manage and we recently upgraded MariaDB from 10.5 to 10.6 on those servers. Now I'm seeing some odd problems which manifest as serious performance issues after I import a database. Meaning that SQL queries that were running in a short time suddenly became very slow. I then found that simply running "mysqlcheck -a" totally fixes the problem and I can reproduce this issue every time but I can't figure out what is causing it. In addition to the performance issues that I see, I also see odd stats from information_schema and the cardinality of the indexes. The latter part is I think why the SQL queries become slower. You can see a full test of this issue in the attached screenshot. I can put that into a code box but I thought the annotations made it easier to see the problems here. Anyone seen this before or have any idea what it going on? EDIT: additional info I found that disabling innodb_stats_persistent completely fixes the problem. Of course, that is likely not a real fix. Is there a problem with innodb_stats_persistent in MariaDB 10.6? |
| Comments |
| Comment by Roland [ 2022-04-21 ] | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Same here. We've upgraded a few months ago to 10.6 and since than we have serious problems. Exact same behaviour as you stated, after import, weird table stats shown (mostly it reports zero size). Queries however does work, but extremely slow. We are running those instances on windows 2019 servers. mysqlcheck -o fixes it, but the problem often reappear after an import, not always. We've moved servers recently also, with the way of backup and import sql files. This almost killed the server (hanging queries, etc...) because of this issue. Running mysqlcheck after import on all dbs fixed it. | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Comment by Brad [ 2022-04-21 ] | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
That's very interesting that it's cross platform. Thanks for that Roland! I've been playing with various innodb_stats settings but haven't found anything that helps much. Today I thought, maybe I should try disabling innodb_stats_persistent and when I did, the problem is immediately fixed. I can even restart mysql and switch this setting each time and the problem switches on and off like a light switch. So something must have been broken in regards to innodb_stats_persistent. | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Comment by Gerwin [ 2022-06-08 ] | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
We are seeing strange things also (EL8, !0.6), not specific on imports. But we are filling a temp table with a lot of of data. And then the cardinality on all indexes is 0 (zero). So every action your doing on it: update, delete etc is very slow. Running an analyze on the table is a workaround. I cannot reproduce it as it's not always consistent, but when it happens it starts to happen on all databases and a restart of the server seems to be needed. I have seen this issues 3 times now in a short period and never seen them before to my knowledge. | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Comment by Brad [ 2022-06-08 ] | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Yes, I've seen the exact same behavior. It's not just on imports. I can reproduce it 100% of the time with imports but other actions cause the same issue. I've had several clients that triggered the issue somehow just by adding and removing data in normal data changes. At first, I was doing the analyze table trick when I saw it but after I discovered that it wasn't just on imports I just changed all of my servers to disable persistent innodb stats. I was worried about other performance issues at first but I haven't seen any at all. If there are negative effects it's definitely less than the other issue so I would recommend it. I wish that the devs would take notice of this issue though as it's a HUGE performance problem and I can literally reproduce it 100% of the time. An acknowledgement would at least let us know that someone is looking into it. | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Comment by wareteamflo [ 2022-09-15 ] | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Same problem here. Seriously a bug but seems that persons take the time to see that... | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Comment by Brad [ 2022-09-15 ] | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Thanks @wareteamflo! Yes, I wish that the devs would look at this more. It's definitely a problem and still is from what I can tell. I haven't testing anything outside of the 10.6 branch though. If you are seeing that issue frequently, I would recommend setting "innodb_stats_persistent=0" in your config. It fully solved the problem for me and doesn't seem to cause any secondary issues. | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Comment by wareteamflo [ 2022-09-15 ] | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Strange thing. I think is related to the same issue because it appear when dump sql but it's really really strange. | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Comment by Marko Mäkelä [ 2022-09-16 ] | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
I am not sure if I understood the problem correctly. There was a bug In the attached picture ksnip_20220415-124405.png Even more useful would be if someone could share exact SQL statements for reproducing this problem. Due to the lack of details, I can only present guesses. One more guess might be that the problem is that the automatic recalculation of statistics was triggered too early. There was an intermittent regression test failure | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Comment by wareteamflo [ 2022-09-16 ] | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Hi You can see my last issue for this problem: Seems very very strange. I see another problem related to the import issue. On some database, using mysqlcheck -a "database" solve the problem, and another database need to make this command multiple times to work. And other solution is to import the database in SSD and move it on HDD. Very strange. Same problem on 10.9.2, the latest version so I think is not complety solved. | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Comment by wareteamflo [ 2022-09-16 ] | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Bug appear at exactly 10.3.29 on my situation. | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Comment by wareteamflo [ 2022-09-16 ] | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Apparently, the problem disapear on some dump when disable innodb_stats_persistent like say @Brad | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Comment by Marko Mäkelä [ 2022-09-16 ] | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
wareteamflo, some possibly relevant change between MariaDB Server 10.3.28 and 10.3.29 were It is very hard to do anything without having anything executable that would reproduce this. Can you reproduce this with some dummy data generator, either written by yourself or created by tweaking Sysbench? | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Comment by wareteamflo [ 2022-09-16 ] | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Thank you for your following on this situation. I can reproduce it with some databases (we work on 50+ database with Mariadb and the situation is the same for each). Dumping database from mysql Ver 15.1 Distrib 10.1.48-MariaDB, for debian-linux-gnu (x86_64) using readline 5.2 *to *mysql Ver 15.1 Distrib 10.9.2-MariaDB, for debian-linux-gnu (x86_64) using EditLine wrapper Import using mysql command. What do you want to execute/debug/info log for debug this ? | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Comment by Marko Mäkelä [ 2022-09-16 ] | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
wareteamflo, I am afraid that I would need the dump itself. You could reduce the dump to the minimum (maybe just one table) while keeping the problem reproducible. You could also replace all non-indexed columns with dummy data of the same length. Column and table names can be obfuscated as well. When the InnoDB persistent statistics are not being used, then InnoDB will not read the data from the persistent statistics tables, but it will calculate some less accurate statistics on the fly. I would expect that if you set innodb_stats_auto_recalc=OFF then you will have to run ANALYZE TABLE on the tables manually. That it is what mysqlcheck -a is doing. Possibly the problem is that the automatic recalculation is being triggered in the middle of the logical import, or not properly triggered at all. Do you see any non-zero data in mysql.innodb_index_stats and mysql.innodb_table_stats after the import? | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Comment by wareteamflo [ 2022-09-16 ] | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
This is indexes from one of table :
And yes, I see non zero data in innodb_index_stats Retry an mysqlcheck return also a 0 in cardinality. | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Comment by wareteamflo [ 2022-09-16 ] | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Ok, I confirm that the problem is related this issue. I missing to input innodb_stats_persistent in [mysqld] and not in [mariadb] (which is not working). After this, all request doing fine on all my 40+ database. Issue appear since 10.3.29 with me and others version have the same problem. | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Comment by Sergei Golubchik [ 2022-09-16 ] | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
ok, so you've been able to disable innodb_stats_persistent and it helped, good. | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Comment by wareteamflo [ 2022-09-16 ] | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Yes Sergei, of course is related to this issue. If I enable stats_persistent, querry make 1 second or 2 seconds to take. With 2000querrys, website are not responsive with this lag. | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Comment by Brad [ 2022-09-16 ] | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Hi all, Here is a full test of this under CentOS 7 using MariaDB 10.6.9. I created a new VM, installed MariaDB, did not change any settings, and ran the test. The DB itself is a from a default install of Wordpress that I created. I simply created the WordPress install using default settings and exported the database to get the data. Nothing unual about any of this. DB is attached as well (wptest.sql).
| |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Comment by Brad [ 2022-09-16 ] | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
I ran some more tests to see if this was affecting other versions and I can confirm that this affects 10.7 and 10.8 as well. Looks like it affects everything above 10.5 based on what I'm seeing. I also wanted to explain this more by showing what I believe is the expected behavior and comparing that with what is happening now. This is a LOT of data so let me summarize first. In 10.5, when you import data, the Innodb stats do appear to see some change but aren't 100% accurate (of course, since the stats are not meant to be 100% accurate anyway). So the first query you execute against that data is slow. It needs to first build the stats on that and that's OK for the first query. If you execute just a simple select statement on the data, it populates the stats, which allows the next query to be faster. That is what should happen and does happen in 10.5. In 10.6 and up, there are 2 differences. I just attached 2 files to this issue. If you can review these you will see what I mean. In 10.5, after you run the first select statement, the stats populate. In 10.8, they do not until you manually run analyze tables. Let me know if you need any more information. EDIT: corrected links to attached files | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Comment by Brad [ 2022-09-16 ] | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Just to rule out CentOS 7 as the culprit since all of my other tests have been using it. I ran the same tests using CentOS 9 on 10.9.2 and the problem exists there as well. | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Comment by Marko Mäkelä [ 2022-09-19 ] | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
I can repeat this with the following test and a copy of wptest.sql
For most of the tables, 0 rows would be reported. Those statistics were written at table creation time:
I checked a trace that was generated by ./mtr --rr. Here you can see that the statistics for one table were updated while an INSERT on that table was still in progress:
There clearly is a race condition between the statistics update and the INSERT. If I run the test with - I believe that the simplest fix to this is to make InnoDB automatically update persistent statistics at the end of a Before in MariaDB Server 10.6, you might get STATS_AUTO_RECALC setting the number of rows to a smaller value, while the insert is still running. You might also get a server hang ( Note: If you disable the InnoDB persistent statistics, then so-called "transient statistics" will be used instead of consulting the persistent statistics tables. | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Comment by Marko Mäkelä [ 2022-10-12 ] | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Thank you, this looks mostly OK. |