CPU Scheduler Fix

While the master CPU was overloaded the binlog dump thread was mostly sitting in “Writing to net” status. At the same time the slave was mostly idle, that is from time to time reaching the end of the latest relay-log, and yet the slave lag kept increasing (now measured by Seconds_behind_master). Everytime the slave received something from the master’s binary log it was able to process it rapidly.

The root cause is the Linux scheduler. We don't have insight into what in the Linux Scheduler causes this. MariaDB Engineers overloaded a server with heavy (CPU intensive) read threads and a couple of lighter write threads. Enough to eat up all the CPU by a large margin. This led to the behavior seen in the support ticket.

2.3.2. Proposed Feature
Possible solutions to investigate would be:
Increase thread priority in the Linux Kernel. This might not work as the issue might be related to issues with network resources.
Throttle the threads in MariaDB. This could either be done by:
Using and potentially modifying the thread pooling feature to make sure the binlog dump threads get dedicated CPU time
Throttle connection threads by introducing delays in the query execution path whenever the CPU is overloaded and the binlog dump thread cannot send
Scale down automatically (like innodb_thread_concurrency or max_connections) whenever the CPU is overloaded and the binlog dump thread cannot send