ATTN danblack
The correct way to adjust systemd settings so they don't get overwritten is to create a directory and file as such:
/etc/systemd/system/mariadb.service.d/timeout.conf
[Service]
|
|
|
TimeoutStartSec=12min
|
created the documentation - https://mariadb.com/kb/en/mariadb/systemd/, still a bit rough.
As I read this bug the major problem is that the configuration of TimeoutStartSec gets overwritten. Hopefully the correct configuration of user systemd settings is now adequately described.
The default value for TimeoutStartSec appears to be 90 seconds. I don't think we should increase this by default. Justifications for an appropriate value welcome.
As an aside 10 minutes for start up seem extraordinarily large. If innodb_buffer_load_at_startup is the cause perhaps MDEV-8923 is a feature to vote on, or perhaps it is an innodb_log_file_size that takes too long. If you think the start-up time can be optimized then create an issue for this.
Nice work on the documentation, should help some administrators (like me) who are quite new to the whole systemd. I guess it should include "systemctl daemon-reload" to make sure the changes are properly taken into effect when manually editing files.
Also our debian install does not honour the TimeoutStartSec setting when invoked using the old init.d commands (e.g. anyone comming from the older debian versions).
The default of 90 seconds should suffice for most users, I guess we shall have to wait patiently for 10.2 as described in MDEV-8923 for the improved restart speeds.
We're all a bit new to systemd I think. Good idea with daemon-reload - added. Debian init scripts seem to respect MYSQLD_STARTUP_TIMEOUT as a mechanism for achieving the same thing.
https://github.com/MariaDB/server/commit/7f19330c595e3183d079fe2c18eecc74740e8f83#diff-f0eed6025f4cb6b214beea14d2d68cd4 seems to have prevented the redirect (which I thought was going to be for 10.0 only, anyway, need to work out what the direction is there).
I can't tell if MDEV-8923 is the cause of your slow start ups, if its not make whatever the cause is that is, it is reported. Watch and vote on the applicable tickets to make sure its evident that those features are wanted.
Please note that for MariaDB Galera cluster, a node startup may require a full state transfer - which can take a considerable amount of time, while all of the databases are copied...
Closing issue according to danblack comment. Fixed by documenting systemd settings. Feel free to reopen if disagree.
We should disable the timeout for server startup. This is especially important in the case of Galera. SST can take a very long time. Killing mysqld because SST is working is completely nonsensical.
or calling
sd_notify(0, "READY=1\n" |
earlier in wsrep_init_startup(true)/wsrep_sst_wait 
danblack, I don't think telling systemd that the service is ready before that's true is a good idea. There are a great many opportunities for SST to fail beyond the default 90s timeout. The only safe thing to do IMO is to remove the timeout entirely, since SST of a large dataset can take hours.
Yeh, I wasn't convinced of the earlier sd_notify fix either.
TimeoutStartSec=0 appears to disable the timeout according to the documentation http://www.freedesktop.org/software/systemd/man/systemd.service.html
I'd recommend using TimeoutSec=0 instead. It can also take >90s to stop the server if it is flushing the buffer pool or completing some other long-running operation.
svoj, this issue is assigned to you. Do you object to setting TimeoutSec=0 in the mariadb.service unit file? This is causing serious problems. The fix is very simple.
No objections from my side, though it's a pity that we can't use this nice systemd feature.
I added this task to next 10.1 sprint backlog.
What "nice systemd feature" do you wish you could use?
start/stop timeouts
What makes you think that start/stop timeouts are such a nice feature? What are the scenarios where MariaDB hasn't started after 1, 2, 5, 10, or 30 minutes and you think the OS should kill the process? SST, InnoDB recovery, etc., can all take a basically undefined amount of time, and timing out after some period of time just seems like it doesn't solve anything. I'm really missing the scenarios where this would be desirable/helpful behavior.
IMHO the fact that MariaDB isn't bug free makes this feature nice. I was able to find at least the following shutdown deadlocks almost instantly:
https://jira.mariadb.org/browse/MDEV-9595
https://jira.mariadb.org/browse/MDEV-4136
https://jira.mariadb.org/browse/MDEV-8476
https://jira.mariadb.org/browse/MDEV-8188
https://jira.mariadb.org/browse/MDEV-7177
https://jira.mariadb.org/browse/MDEV-7634
https://jira.mariadb.org/browse/MDEV-8727
serg, please review fix for this bug.
I'd prefer to keep the default timeout. The default is supposed to be good for most cases. And users should be able to adjust it as needed. See, for example MDEV-8509 and MDEV-5068. With systemd one apparently can adjust the timeout (see above). I think the issue can be closed as is.
Hi ,
If a node deadlock is caused by a bug please kill -9 and open a bug
Using a system feature to hide our own mess can not be a justification , to make most cluster user regular SST feature to failed
Is it possible to revisit this decision now that most old bugs are fixed
/svar
It looks like MDEV-6113 will be merged for at least 10.2, this would alleviate the need for such settings greatly.
But the fact remains that for larger servers the default timeout is on the short side (when a server boots it usually has more things to do than just start the database).
Personally I'd recommend setting it to 3 minutes for 10.2+, the default behaviour appears to be to wait for a full MariaDB boot instead of a hard kill. Given this behaviour we only have to balance a logical timeout before warning the user that something Might be wrong (as the server will continue to boot and exit after it has done so, in a recovery situation a simple restart should suffice, for larger servers the override can be used).
RFE upstream in attempt to get workable solution: https://github.com/systemd/systemd/issues/5868