Document that shutting down the master can break auto_rejoin and lose transactions

MariaDB Monitor's documentation should contain a note about the issues that can occur if a master is shutdown. MariaDB Monitor generally makes failover very easy, so many users assume that it is very safe to shutdown the master without performing a switchover first, but that is generally not the case.

The problem is that when a master server is shutdown and it goes through the normal shutdown process, the master kills client threads in random order. By default, the master also considers its binary log dump threads to be regular client threads. As a consequence, the binary log dump threads can be killed while client threads still exist, and this means that data can be written on the master during a normal shutdown that won't be replicated. This is true even if semi-synchronous replication is being used.

In MariaDB 10.4 and later, this problem can be solved by shutting down the server using either the mysqladmin utility or the SHUTDOWN command, and providing a special option.

For example, this problem can be solved by shutting down the server with the mysqladmin utility and by providing the --wait-for-all-slaves option to the utility and by executing the shutdown command with the utility:

mysqladmin --wait-for-all-slaves shutdown

Or this problem can be solved by shutting down the server with the SHUTDOWN command and by providing the WAIT FOR ALL SLAVES option to the command:

SHUTDOWN WAIT FOR ALL SLAVES;

When one of these special options is provided, the server only kills its binary log dump threads after all client threads have been killed, and it only completes the shutdown after the last binary log has been sent to all connected slaves.

In MariaDB 10.4 and later, it is still not possible to enable this behavior by default. This means that this behavior is currently inaccessible when shutting down the server using tools like systemd or sysVinit.

In MariaDB 10.3 and before, it is recommended to manually switchover slaves to a new master before shutting down the old master.

I think the MariaDB Monitor documentation should contain a note about this.

https://mariadb.com/kb/en/mariadb-maxscale-23-mariadb-monitor/#cluster-manipulation-operations

https://mariadb.com/kb/en/mariadb-maxscale-24-mariadb-monitor/#cluster-manipulation-operations

It may also be a good idea to link to the relevant MariaDB Server documentation:

https://mariadb.com/kb/en/library/replication-threads/#binary-log-dump-threads-and-the-shutdown-process