[MDEV-28969] MariaDB server horribly slow on start Created: 2022-06-28 Updated: 2022-07-23 Resolved: 2022-07-08 |
|
| Status: | Closed |
| Project: | MariaDB Server |
| Component/s: | Galera, Server |
| Affects Version/s: | 10.5.15 |
| Fix Version/s: | N/A |
| Type: | Bug | Priority: | Major |
| Reporter: | COUNOTTE CEDRIC | Assignee: | Unassigned |
| Resolution: | Not a Bug | Votes: | 0 |
| Labels: | None | ||
| Attachments: |
|
| Description |
|
We have a total of 6 servers, running 2 Galera cluster, 1 cluster have 4 nodes, the other have 2 nodes being slave of replication of the first cluster. Not that it actually matters. When any of those servers is restarted, it gets slow like hell for about 30-45 minutes. It seems to load all in memory before it gets ok. It's also slowing down the whole cluster dramatically causing our app to hang completely for several minutes, until I aggressivelly kill query processes started rom recurrent scripts. When pausing for an hour and restarting replication, this problem gets quite apparent because every replicated query (running one at a time) gets stuck for several seconds running (insert/delete or update alike) and then stuck again in closing table for several seconds. Note that replication was originally made to a single MariaDB server and the same symptoms were visible. For nearly an hour, the Seconds_Behind_Master is increasing until this resolves itself. This is creating huge problem when we try to maintain/update servers as it hangs our applications. See attached our current configuration file, everything else being set to default. We started with a blank configuration file and the problem was already there, tweaked many/some settings to no avail. Tried duplicating the settings from our old servers (that were not restarted) didn't make a difference. Already played wiht the followign settings to no avail:
Is there any other settings that affect initial server start and could improve this situation? |
| Comments |
| Comment by Sergei Golubchik [ 2022-07-08 ] |
|
It doesn't look like a bug so far. Please, see https://mariadb.com/kb/en/mailing-lists/ and https://mariadb.com/kb/en/meta/irc-chat-servers-and-zulip-instance/ for various media here you can get help with your MariaDB problem. If you'll find out that it was a bug after all, please, don't hesitate to add a comment here and we'll reopen the issue. |
| Comment by COUNOTTE CEDRIC [ 2022-07-12 ] |
|
I can't imagine that not being a bug because I just restarted a server inside a galera cluster of 4 servers and everything went haywire. Queries taking more than 400 seconds to complete, piling up more than 300 to 400 queries on each server, and that for about 30 minutes to an hour! That server was not even directly used by any front-end or anything else! Also noticed an UPDATE/UGPRADE TABLE FOR UPGRADE process going on upon restart, which is further slowing down the DB, which I have to kill asap otherwise face even more dramatic slow-down. We have about 20 scripts updating DB every 2 or 5 minutes and I have to kill them constantly for the whole time, even though they run on a different server. Long story short it makes the whole DB (and our company service) completely unusable for nearly an hour. And our customers are calling us for an explanation and asking when service will resume. IMO, we cannot maintain/update MariaDB without using a duplicate cluster in another location. |