[MDEV-20065] parallel replication for galera slave Created: 2019-07-15  Updated: 2022-09-28

Status: Open
Project: MariaDB Server
Component/s: Galera, Replication
Fix Version/s: None

Type: Task Priority: Major
Reporter: Andrei Elkin Assignee: Andrei Elkin
Resolution: Unresolved Votes: 4
Labels: parallelslave

Issue Links:
Relates
relates to MDEV-6860 Parallel async replication hangs on a... Closed

 Description   

Galera node when acts as the regular replication slave is limited to the single-threaded
slave applier because the pre_commit transaction ordering by Galera may be set up
to violate the BGC one so two transactions from the same binlog group end up in deadlock, like

Gtid_seq_no= 2
 
Thread 34 (Thread 0x7fcd966d2700 (LWP 23891)):
 
#0  0x00007fcda6d56415 in pthread_cond_wait@@GLIBC_2.3.2 () from
 
/usr/lib/libpthread.so.0
 
#1  0x00005569d607d380 in safe_cond_wait (cond=0x7fcd854078e8,
 
mp=0x7fcd85407838, file=0x5569d6240360
 
"/home/sachin/10.1/server/include/mysql/psi/mysql_thread.h",
 
line=1154) at /home/sachin/10.1/server/mysys/thr_mutex.c:493
 
#2  0x00005569d5aec4d0 in inline_mysql_cond_wait (that=0x7fcd854078e8,
 
mutex=0x7fcd85407838, src_file=0x5569d6240cb8
 
"/home/sachin/10.1/server/sql/log.cc", src_line=7387) at
 
/home/sachin/10.1/server/include/mysql/psi/mysql_thread.h:1154
 
#3  0x00005569d5afeee5 in MYSQL_BIN_LOG::queue_for_group_commit
 
(this=0x5569d692d7c0 <mysql_bin_log>, orig_entry=0x7fcd966cf440) at
 
/home/sachin/10.1/server/sql/log.cc:7387
 
#4  0x00005569d5aff5c9 in
 
MYSQL_BIN_LOG::write_transaction_to_binlog_events (this=0x5569d692d7c0
 
<mysql_bin_log>, entry=0x7fcd966cf440) at
 
/home/sachin/10.1/server/sql/log.cc:7607
 
#5  0x00005569d5afecff in MYSQL_BIN_LOG::write_transaction_to_binlog
 
(this=0x5569d692d7c0 <mysql_bin_log>, thd=0x7fcd84c068b0,
 
cache_mngr=0x7fcd84c72c70, end_ev=0x7fcd966cf5e0, all=true,
 
using_stmt_cache=true, using_trx_cache=true) at
 
/home/sachin/10.1/server/sql/log.cc:7290
 
#6  0x00005569d5af0ce6 in binlog_flush_cache (thd=0x7fcd84c068b0,
 
cache_mngr=0x7fcd84c72c70, end_ev=0x7fcd966cf5e0, all=true,
 
using_stmt=true, using_trx=true) at
 
/home/sachin/10.1/server/sql/log.cc:1751
 
#7  0x00005569d5af11bb in binlog_commit_flush_xid_caches
 
(thd=0x7fcd84c068b0, cache_mngr=0x7fcd84c72c70, all=true, xid=2) at
 
/home/sachin/10.1/server/sql/log.cc:1859
 
#8  0x00005569d5b045c8 in MYSQL_BIN_LOG::log_and_order
 
(this=0x5569d692d7c0 <mysql_bin_log>, thd=0x7fcd84c068b0, xid=2,
 
all=true, need_prepare_ordered=false, need_commit_ordered=true) at
 
/home/sachin/10.1/server/sql/log.cc:9575
 
#9  0x00005569d5a1ec0d in ha_commit_trans (thd=0x7fcd84c068b0,
 
all=true) at /home/sachin/10.1/server/sql/handler.cc:1497
 
#10 0x00005569d5925e7e in trans_commit (thd=0x7fcd84c068b0) at
 
/home/sachin/10.1/server/sql/transaction.cc:235
 
#11 0x00005569d5b1b1fa in Xid_log_event::do_apply_event
 
(this=0x7fcd8542a770, rgi=0x7fcd85407800) at
 
/home/sachin/10.1/server/sql/log_event.cc:7720
 
#12 0x00005569d5743fa1 in Log_event::apply_event (this=0x7fcd8542a770,
 
rgi=0x7fcd85407800) at /home/sachin/10.1/server/sql/log_event.h:1343
 
#13 0x00005569d573987e in apply_event_and_update_pos_apply
 
(ev=0x7fcd8542a770, thd=0x7fcd84c068b0, rgi=0x7fcd85407800, reason=0)
 
at /home/sachin/10.1/server/sql/slave.cc:3479
 
#14 0x00005569d5739deb in apply_event_and_update_pos_for_parallel
 
(ev=0x7fcd8542a770, thd=0x7fcd84c068b0, rgi=0x7fcd85407800) at
 
/home/sachin/10.1/server/sql/slave.cc:3623
 
#15 0x00005569d597bfbe in rpt_handle_event (qev=0x7fcd85424770,
 
rpt=0x7fcd85421c88) at /home/sachin/10.1/server/sql/rpl_parallel.cc:50
 
#16 0x00005569d597ed57 in handle_rpl_parallel_thread
 
(arg=0x7fcd85421c88) at
 
/home/sachin/10.1/server/sql/rpl_parallel.cc:1258
 
 
 
Gtid_seq_no= 1
 
Thread 33 (Thread 0x7fcd9671d700 (LWP 23890)):
 
#0  0x00007fcda6d56415 in pthread_cond_wait@@GLIBC_2.3.2 () from
 
/usr/lib/libpthread.so.0
 
#1  0x00007fcd9e7778ab in gu::Lock::wait (this=0x7fcd9671a0c0,
 
cond=...) at galerautils/src/gu_mutex.hpp:40
 
#2  galera::Monitor<galera::ReplicatorSMM::CommitOrder>::enter
 
(this=this@entry=0x7fcda12d5da0, obj=...) at
 
galera/src/monitor.hpp:124
 
#3  0x00007fcd9e771f28 in galera::ReplicatorSMM::pre_commit
 
(this=0x7fcda12d5000, trx=0x7fcd8507e000, meta=<optimized out>) at
 
galera/src/replicator_smm.cpp:796
 
#5  0x00005569d59864d0 in wsrep_run_wsrep_commit (thd=0x7fcd85006a70,
 
all=true) at /home/sachin/10.1/server/sql/wsrep_hton.cc:492
 
#6  0x00005569d5984d6a in wsrep_prepare (hton=0x7fcda583e270,
 
thd=0x7fcd85006a70, all=true) at
 
/home/sachin/10.1/server/sql/wsrep_hton.cc:208
 
#7  0x00005569d5a1e1b0 in prepare_or_error (ht=0x7fcda583e270,
 
thd=0x7fcd85006a70, all=true) at
 
/home/sachin/10.1/server/sql/handler.cc:1196
 
#8  0x00005569d5a1ea1c in ha_commit_trans (thd=0x7fcd85006a70,
 
all=true) at /home/sachin/10.1/server/sql/handler.cc:1475
 
#9  0x00005569d5925e7e in trans_commit (thd=0x7fcd85006a70) at
 
/home/sachin/10.1/server/sql/transaction.cc:235
 
#10 0x00005569d5b1b1fa in Xid_log_event::do_apply_event
 
(this=0x7fcd8542a570, rgi=0x7fcd85407000) at
 
/home/sachin/10.1/server/sql/log_event.cc:7720
 
#11 0x00005569d5743fa1 in Log_event::apply_event (this=0x7fcd8542a570,
 
rgi=0x7fcd85407000) at /home/sachin/10.1/server/sql/log_event.h:1343

In above the 2nd of the two, BGC-ordered earlier though, trx turns into commit after the 1st which is BGC-ordered later. The 1st holds a galera resource while attempting to yield the BGC ordering control to the 2nd which is locked out of the galera resource.

There are few ideas how to open up parallel slave applier for galera slave. In one we consider to introduce an interface to carry out an arbitrary action in specified BGC order. Such action could be wsrep_prepare as well.

 Comments   
Comment by Jan Lindström (Inactive) [ 2022-09-28 ]

seppo Is this even valid request now that we have wsrep_slave_threads parameter ?

Generated at Thu Feb 08 08:56:27 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.