[MDEV-15152] Optimistic parallel slave doesn't cope well with START SLAVE UNTIL Created: 2018-01-31  Updated: 2021-01-07  Resolved: 2020-05-27

Status: Closed
Project: MariaDB Server
Component/s: Replication
Affects Version/s: 10.2.12
Fix Version/s: 10.5.4, 10.1.46, 10.2.33, 10.3.24, 10.4.14

Type: Bug Priority: Critical
Reporter: Hartmut Holzgraefe Assignee: Andrei Elkin
Resolution: Fixed Votes: 3
Labels: parallelslave


 Description   

When giving a parallel optimistic slave a replication stop position with

START SLAVE UNTIL ...file..., ...pos...;

it may actually stop at a position earlier than the given one if the slave worker that performs the transaction that spans over the given stop position has to roll back due to conflicts.

This seems to leave the slave in a state where the SQL master thread still waits for the transaction to complete, but does not hand out any more tasks to the actual worker threads. This seems to include re-execution of failed transactions that start before the stop position.

So replication effectively stops, but the SQL thread still shows as "Slave_SQL_Running: YES" and Exec_master_log_pos showing a log position smaller than the given stop position, whereas UNTIL should actually end with Slave_SQL_Running=No and the Exec position at the beginning of the first transaction (or event?) after the stop position.

Meanwhile the actual slave worker threads seem to have been terminated already, and attempts to stop the slave do not succeed.

 Comments   
Comment by Andrei Elkin [ 2018-02-02 ]

I've seen problems with the Aggressive mode.
It is not flawless yet.
MDEV-12746 - which I have not updated with analysis - a patch is coming to hopefully by night. It's going to refine automatic retry-related logics. Whether/how it might be relevant to the UNTIL one is unclear though now.

Comment by Jonah George [ 2019-01-07 ]

I've also been experiencing sporadic issues that seem related to this (on v10.1.28)- curious if anyone has narrowed down a reproducible test case.

Comment by Jean-François Gagné [ 2019-01-07 ]

I am sure this is fully reproducible. When I did the private support ticket to Hartmut related to that, I was hitting this situation many times. I would expect a tight UPDATE loop on the same row to make the problem very easy to reproduce. I, sadly, do not have much time to work on this now. Let me just add that I am very disappointed that such bug in an important feature is not receiving more attention, which deserves a #MariaDB #BugOfTheDay: https://twitter.com/jfg956/status/1082406523282903041

Comment by Andrei Elkin [ 2020-04-09 ]

I can confirm the following that the until started OPTIMISTIC parallel slave can reach the Until condition and exist having Relay_Master_Log_File:Exec_Master_Log_Pos < Until_Log_File:Until_Log_Pos.
This may happen when optimistically executed transaction range spans more than one binlog file so the actual stop occures in an early file: Relay_Master_Log_File < Until_Log_File.

This is a clear failure and discovered details hint strongly how to fix.

Note too, that there's no traces of the hanging SQL thread now: {{ Slave_SQL_Running: No }}.
(Perhaps removed by MDEV-12746 fixes).

Comment by Andrei Elkin [ 2020-05-15 ]

Sachin, could you please prioritize reviewing the soonest. KristianN has not shown up
for it(, must be just busy).

Comment by Sachin Setiya (Inactive) [ 2020-05-27 ]

Okay to push

Generated at Thu Feb 08 08:19:04 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.