[#MDEV-20398] Slave sql thread blocked by Update_rows_log_event::find

Comment by Andrei Elkin [ 2019-08-21 ]

Sandy, thank you for the report!
Is there a full show processlist log? The description provided on a fragment of it.
The same seems to apply to SHOW ENGINE INNODB STATUS. Could you please provide that too?

I am not sure what the line

{color:#DE350B}878349428	system user		NULL	Connect	0	Update_rows_log_event::find_row(-1)	NULL	0.000{color}

was colored, it must relate to a running slave worker thread, not to the slave IO thread.
Any idea why the title mentions the latter one?

From the first glance I suspect it's the Worker thread that is slow for some reason.
I suggest to try identifying an event it is processing and its and its table properties, presence/lack of unique keys in particular.

Thanks!

Andrei

Comment by Sandy [ 2019-08-22 ]

Andrei，thanks for your response.
Sorry for I given the wrong message, I check the log again ,this message is on the processlist log,and yes when I kill that thread,No.878349428,slave sql thread was stopped.
And the record of No.878349428 thread in engine innodb status log is as follow:

---TRANSACTION 86665517604, ACTIVE (PREPARED) 0 sec
3 lock struct(s), heap size 1184, 1 row lock(s), undo log entries 2
MySQL thread id 878349428, OS thread handle 0x7fb5255fab00, query id 189853061018 Unlocking tables
Trx #rec lock waits 318751 #table lock waits 0
Trx total rec lock wait time 774 SEC
Trx total table lock wait time 0 SEC

---TRANSACTION 86665661686, ACTIVE 0 sec starting index read
mysql tables in use 1, locked 1
1 lock struct(s), heap size 360, 0 row lock(s)
MySQL thread id 878349428, OS thread handle 0x7fb5255fab00, query id 189853425690 Update_rows_log_event::find_row(-1)
Trx #rec lock waits 318754 #table lock waits 0
Trx total rec lock wait time 774 SEC
Trx total table lock wait time 0 SEC

Comment by Sandy [ 2019-08-30 ]

If the config of cocurrency has any related about the question?
The set of this instances is 32.

-------------------------------------------+

Variable_name

Value

-------------------------------------------+

skip_parallel_replication	OFF
slave_domain_parallel_threads	0
slave_parallel_max_queued	131072
slave_parallel_mode	conservative
slave_parallel_threads	8

-------------------------------------------+

------------------------------------------------------------------+

Variable_name

Value

------------------------------------------------------------------+

aria_repair_threads	1
binlog_optimize_thread_scheduling	ON
debug_no_thread_alarm	OFF
innodb_encryption_threads	0
innodb_mtflush_threads	8
innodb_purge_threads	8
innodb_read_io_threads	4
innodb_thread_concurrency	32
innodb_thread_sleep_delay	0
innodb_write_io_threads	8
max_delayed_threads	20
max_insert_delayed_threads	20
myisam_repair_threads	1
performance_schema_max_thread_classes	50
performance_schema_max_thread_instances	2100
pseudo_thread_id	60520634
slave_domain_parallel_threads	0
slave_parallel_threads	8
thread_cache_size	800
thread_concurrency	10
thread_handling	one-thread-per-connection
thread_pool_idle_timeout	60
thread_pool_max_threads	1000
thread_pool_oversubscribe	3
thread_pool_size	12
thread_pool_stall_limit	500
thread_stack	295936
wsrep_slave_threads	1

------------------------------------------------------------------+

Comment by Andrei Elkin [ 2019-09-03 ]

Sandy Update_rows_log_event::find_row(-1) may indicate the full table scan (slow!) if the Update_rows event is on unique-key-less table. I suggested to verify that.

Comment by Miguel P [ 2019-12-20 ]

Happens to me as well. Replication gets stuck in row-based events in either Update_rows_log_event::find_row(-1) or a delete with the same find_row(-1) .

● Happens at updating or deleting any of several tables when tried to restart from different dumps, and very quickly so.
● Tables all have primary keys.
● Tables are all InnoDB.
● None of these tables had more than a couple hundred rows.
● No CPU use; it's not a full table scan taking long. I've left it for 10 days and it's still stuck.
● The SQL thread is just stuck. mysqld doesn't shutdown; must be killed with SIGKILL and master.info deleted in order to recover.
● No other connections to the database. It doesn't seem to be a locking issue.
● Tried with or without GTID replication, same result.
● Tried with MIXED or ROW binlog_format in master, same result.
● I'm running a very simple 1× master 10.1.41-MariaDB-0+deb9u1 → 1× slave 10.1.41-MariaDB-0+deb9u1 replication.

I don't believe I'm doing anything remotely weird here.

Is there a way to make replication work at all?

(Sorry for the bad formatting, Jira is no good for this and preformatted is broken.)

mysql> show full processlist;
---------------------------------------------------------------------------------------------------------

Id	User	Host	db	Command	Time	State	Info	Progress
253	system user		NULL	Connect	441	Waiting for master to send event	NULL	0.000
254	system user		NULL	Connect	441	Update_rows_log_event::find_row(-1)	NULL	0.000
256	root	localhost	sssss	Query	0	init	show full processlist	0.000

---------------------------------------------------------------------------------------------------------

mysql> show slave status\G
Slave_IO_State: Queueing master event to the relay log
Master_Host: SomethingProd
Master_User: somethingslave
Master_Port: 3306
Connect_Retry: 60
Master_Log_File: mysql-bin.060374
Read_Master_Log_Pos: 2381545
Relay_Log_File: relay.000002
Relay_Log_Pos: 19382
Relay_Master_Log_File: mysql-bin.060368
Slave_IO_Running: Yes
Slave_SQL_Running: Yes
Replicate_Do_DB:
Replicate_Ignore_DB:
Replicate_Do_Table:
Replicate_Ignore_Table: sssss.md_session
Replicate_Wild_Do_Table:
Replicate_Wild_Ignore_Table:
Last_Errno: 0
Last_Error:
Skip_Counter: 0
Exec_Master_Log_Pos: 6363905
Relay_Log_Space: 45734876
Until_Condition: None
Until_Log_File:
Until_Log_Pos: 0
Master_SSL_Allowed: No
Master_SSL_CA_File:
Master_SSL_CA_Path:
Master_SSL_Cert:
Master_SSL_Cipher:
Master_SSL_Key:
Seconds_Behind_Master: 7039
Master_SSL_Verify_Server_Cert: No
Last_IO_Errno: 0
Last_IO_Error:
Last_SQL_Errno: 0
Last_SQL_Error:
Replicate_Ignore_Server_Ids:
Master_Server_Id: 1
Master_SSL_Crl:
Master_SSL_Crlpath:
Using_Gtid: No
Gtid_IO_Pos:
Replicate_Do_Domain_Ids:
Replicate_Ignore_Domain_Ids:
Parallel_Mode: conservative

mysql> show engine innodb status\G
Type: InnoDB
Name:
Status:
=====================================
2019-12-20 02:29:24 7f9318e83700 INNODB MONITOR OUTPUT
=====================================
Per second averages calculated from the last 33 seconds
-----------------
BACKGROUND THREAD
-----------------
srv_master_thread loops: 814 srv_active, 0 srv_shutdown, 5358 srv_idle
srv_master_thread log flush and writes: 6171
----------
SEMAPHORES
----------
OS WAIT ARRAY INFO: reservation count 16185
OS WAIT ARRAY INFO: signal count 21147
Mutex spin waits 41138, rounds 406923, OS waits 12368
RW-shared spins 5005, rounds 21969, OS waits 577
RW-excl spins 9383, rounds 119351, OS waits 3209
Spin rounds per wait: 9.89 mutex, 4.39 RW-shared, 12.72 RW-excl
------------
TRANSACTIONS
------------
Trx id counter 13477450
Purge done for trx's n:o < 13477395 undo n:o < 0 state: running but idle
History list length 56
LIST OF TRANSACTIONS FOR EACH SESSION:
---TRANSACTION 0, not started
MySQL thread id 256, OS thread handle 0x7f9318e83700, query id 17514 localhost root init
show engine innodb status
---TRANSACTION 0, not started
mysql tables in use 1, locked 1
MySQL thread id 254, OS thread handle 0x7f9318e1f700, query id 16747 Update_rows_log_event::find_row(-1)
--------
FILE I/O
--------
I/O thread 0 state: waiting for completed aio requests (insert buffer thread)
I/O thread 1 state: waiting for completed aio requests (log thread)
I/O thread 2 state: waiting for completed aio requests (read thread)
I/O thread 3 state: waiting for completed aio requests (read thread)
I/O thread 4 state: waiting for completed aio requests (read thread)
I/O thread 5 state: waiting for completed aio requests (read thread)
I/O thread 6 state: waiting for completed aio requests (write thread)
I/O thread 7 state: waiting for completed aio requests (write thread)
I/O thread 8 state: waiting for completed aio requests (write thread)
I/O thread 9 state: waiting for completed aio requests (write thread)
Pending normal aio reads: 0 [0, 0, 0, 0] , aio writes: 0 [0, 0, 0, 0] ,
ibuf aio reads: 0, log i/o's: 0, sync i/o's: 0
Pending flushes (fsync) log: 0; buffer pool: 0
14500 OS file reads, 873397 OS file writes, 41692 OS fsyncs
0.00 reads/s, 0 avg bytes/read, 0.00 writes/s, 0.00 fsyncs/s
-------------------------------------
INSERT BUFFER AND ADAPTIVE HASH INDEX
-------------------------------------
Ibuf: size 1, free list len 717, seg size 719, 2075 merges
merged operations:
insert 28530, delete mark 0, delete 0
discarded operations:
insert 0, delete mark 0, delete 0
0.00 hash searches/s, 0.00 non-hash searches/s
—
LOG
—
Log sequence number 306283005919
Log flushed up to 306283005919
Pages flushed up to 306283005919
Last checkpoint at 306283005919
Max checkpoint age 32849880
Checkpoint age target 31823322
Modified age 0
Checkpoint age 0
0 pending log writes, 0 pending chkp writes
30638 log i/o's done, 0.00 log i/o's/second
----------------------
BUFFER POOL AND MEMORY
----------------------
Total memory allocated 140574720; in additional pool allocated 0
Total memory allocated by read views 104
Internal hash tables (constant factor + variable factor)
Adaptive hash index 2233952 (2213368 + 20584)
Page hash 139112 (buffer pool 0 only)
Dictionary cache 972528 (554768 + 417760)
File system 933408 (812272 + 121136)
Lock system 333592 (332872 + 720)
Recovery system 0 (0 + 0)
Dictionary memory allocated 417760
Buffer pool size 8191
Buffer pool size, bytes 134201344
Free buffers 1024
Database pages 7166
Old database pages 2625
Modified db pages 0
Percent of dirty pages(LRU & free pages): 0.000
Max dirty pages percent: 75.000
Pending reads 0
Pending writes: LRU 0, flush list 0, single page 0
Pages made young 5889, not young 355905
0.00 youngs/s, 0.00 non-youngs/s
Pages read 14357, created 692551, written 811467
0.00 reads/s, 0.00 creates/s, 0.00 writes/s
No buffer pool page gets since the last printout
Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s
LRU len: 7166, unzip_LRU len: 0
I/O sum[0]:cur[0], unzip sum[0]:cur[0]
--------------
ROW OPERATIONS
--------------
0 queries inside InnoDB, 0 queries in queue
0 read views open inside InnoDB
0 RW transactions active inside InnoDB
0 RO transactions active inside InnoDB
0 out of 1000 descriptors used
Main thread process no. 16927, id 140269215278848, state: sleeping
Number of rows inserted 24597562, updated 0, deleted 0, read 0
0.00 inserts/s, 0.00 updates/s, 0.00 deletes/s, 0.00 reads/s
Number of system rows inserted 0, updated 0, deleted 0, read 0
0.00 inserts/s, 0.00 updates/s, 0.00 deletes/s, 0.00 reads/s
----------------------------
END OF INNODB MONITOR OUTPUT
============================

Comment by Andrei Elkin [ 2019-12-20 ]

mperez, thanks for the detailed report!

It's still unclear what could make the slave applier thread to hang for so long.
Could you please provide two pieces of info:
1. check what was the last executed transaction Relay_Master_Log_File, Exec_Master_Log_Pos
and find what next group of events that follows it, print it out with mysqlbinlog -v for examination by us;
2. sudo gdb -ex "set pagination 0" -ex "thread apply all bt" --batch -p $(pidof mysqld) >/tmp/bt.txt
and send the file to us too.

I hope this will help to resolve.

Cheers,

Andrei.

Comment by Miguel P [ 2019-12-20 ]

Hi @Elkin , thank you so much for looking into this. Here's the Relay_Master_Log_File @ Exec_Master_Log_Pos (I've had to replace app and company names in the -v info by sssss and wwwwwww):

# at 6363905

#191220  0:30:58 server id 1  end_log_pos 6363943   GTID 0-1-139164902 trans

/*!100101 SET @@session.skip_parallel_replication=0*//*!*/;

/*!100001 SET @@session.gtid_domain_id=0*//*!*/;

/*!100001 SET @@session.server_id=1*//*!*/;

/*!100001 SET @@session.gtid_seq_no=139164902*//*!*/;

BEGIN

/*!*/;

# at 6363943

#191220  0:30:58 server id 1  end_log_pos 6364010   Table_map: `sssss`.`ContactCredential` mapped to number 1463683

# at 6364010

#191220  0:30:58 server id 1  end_log_pos 6364089   Update_rows: table id 1463683 flags: STMT_END_F

BINLOG '

Mgj8XRMBAAAAQwAAAGobYQAAAINVFgAAAAEABXNpZ21hABFDb250YWN0Q3JlZGVudGlhbAAFAw/+

/gEGWAD+FP4UDA==

Mgj8XRgBAAAATwAAALkbYQAAAINVFgAAAAEABf//7PawAQAHd2VzdGNvbgDo9rABAAd3ZXN0Y29u

FPgIrIih91mzRiluemd5iwGh0rrmAA==

### UPDATE `sssss`.`ContactCredential`

### WHERE

###   @1=110838

###   @2='wwwwww'

###   @3=NULL

###   @4=NULL

###   @5=0

### SET

###   @1=110838

###   @2='wwwwww'

###   @3='<F8>\x08<AC><88><A1><F7>Y<B3>F)nzgy<8B>\x01<A1>Һ<E6>'

###   @4=NULL

###   @5=0

'/*!*/;

# at 6364089

#191220  0:30:58 server id 1  end_log_pos 6364116   Xid = 204959503

COMMIT/*!*/;

# at 6364116

#191220  0:30:58 server id 1  end_log_pos 6364154   GTID 0-1-139164903

/*!100001 SET @@session.gtid_seq_no=139164903*//*!*/;

BEGIN

/*!*/;

# at 6364154

#191220  0:30:58 server id 1  end_log_pos 6364213   Table_map: `sssss`.`md_session` mapped to number 1463576

# at 6364213

# at 6365244

And here are the backtraces:

bt.txt

Thanks again!

Comment by Andrei Elkin [ 2019-12-20 ]

mperez Thanks. Could you now give us more from the slave incl:
1. show create table `sssss`.`ContactCredential`
2. show global variables ;
3. show global status;

I see the hang occurs near execution of to be ignored Replicate_Ignore_Table: sssss.md_session. But this one is the transaction after the one that is hanging so I doubt MDEV-17446 is relevant.
Just in case could FLUSH TABLES WITH READ LOCK have been run on the slave at about time the beginning of the hang?

Comment by Andrei Elkin [ 2019-12-20 ]

mperez there seems to be a chance of ~~MDEV-20247~~. That case did not limit to the 'preparing' state display, just requires a thread to be of the slave type. And that holds here.

Comment by Miguel P [ 2019-12-21 ]

Hi Andrei Elkin, thanks!

The hang happens right before an event for a Replicate_Ignore_Table in this example, but it also happened in another table before I had this table ignored (I've reconfigured replication from a fresh dump a few times, always hanged at either update or delete with find_rows(-1)). I'm ignoring the `md_session` table because I don't need it and because it's MyISAM, but it didn't help.

We have nothing doing FLUSH TABLES WITH READ LOCK, nor any other threads or clients running when it hangs. (Com_flush is 4 because I did a few FLUSH TABLES during some tests prior to setting up replication.)

Doesn't seem to be an instance of ~~MDEV-20247~~ as there the SQL thread seems to be done ("Slave has read all relay log; waiting for the slave I/O thread to update it") with the IO thread stuck.

Here's the SHOW CREATE TABLE:

CREATE TABLE `ContactCredential` (

  `Contact` int(10) unsigned NOT NULL,

  `Branding` varchar(22) NOT NULL,

  `Password` binary(20) DEFAULT NULL COMMENT '...',

  `OldPassword` binary(20) DEFAULT NULL COMMENT '...',

  `FailedLoginCount` tinyint(4) NOT NULL DEFAULT '0' COMMENT '...',

  PRIMARY KEY (`Contact`,`Branding`),

  CONSTRAINT `ContactCredential_ibfk_1` FOREIGN KEY (`Contact`) REFERENCES `Contact` (`ID`) ON DELETE CASCADE ON UPDATE CASCADE

) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4

SHOW GLOBAL VARIABLES:

Variable_name	Value

aria_block_size	8192

aria_checkpoint_interval	30

aria_checkpoint_log_activity	1048576

aria_encrypt_tables	OFF

aria_force_start_after_recovery_failures	0

aria_group_commit	none

aria_group_commit_interval	0

aria_log_file_size	1073741824

aria_log_purge_type	immediate

aria_max_sort_file_size	9223372036853727232

aria_page_checksum	ON

aria_pagecache_age_threshold	300

aria_pagecache_buffer_size	134217728

aria_pagecache_division_limit	100

aria_pagecache_file_hash_size	512

aria_recover	NORMAL

aria_repair_threads	1

aria_sort_buffer_size	268434432

aria_stats_method	nulls_unequal

aria_sync_log_dir	NEWFILE

aria_used_for_temp_tables	ON

auto_increment_increment	1

auto_increment_offset	1

autocommit	ON

automatic_sp_privileges	ON

back_log	52

basedir	/usr

big_tables	OFF

binlog_annotate_row_events	OFF

binlog_cache_size	65536

binlog_checksum	NONE

binlog_commit_wait_count	0

binlog_commit_wait_usec	100000

binlog_direct_non_transactional_updates	OFF

binlog_format	MIXED

binlog_optimize_thread_scheduling	ON

binlog_row_image	FULL

binlog_stmt_cache_size	32768

bulk_insert_buffer_size	8388608

character_set_client	utf8mb4

character_set_connection	utf8mb4

character_set_database	utf8mb4

character_set_filesystem	binary

character_set_results	utf8mb4

character_set_server	utf8mb4

character_set_system	utf8

character_sets_dir	/usr/share/mysql/charsets/

collation_connection	utf8mb4_general_ci

collation_database	utf8mb4_general_ci

collation_server	utf8mb4_general_ci

completion_type	NO_CHAIN

concurrent_insert	AUTO

connect_timeout	10

core_file	OFF

datadir	/data/mysql/

date_format	%Y-%m-%d

datetime_format	%Y-%m-%d %H:%i:%s

deadlock_search_depth_long	15

deadlock_search_depth_short	4

deadlock_timeout_long	50000000

deadlock_timeout_short	10000

debug_no_thread_alarm	OFF

default_regex_flags

default_storage_engine	InnoDB

default_tmp_storage_engine

default_week_format	0

delay_key_write	ON

delayed_insert_limit	100

delayed_insert_timeout	300

delayed_queue_size	1000

div_precision_increment	4

encrypt_binlog	OFF

encrypt_tmp_disk_tables	OFF

encrypt_tmp_files	OFF

enforce_storage_engine

event_scheduler	DISABLED

expensive_subquery_limit	100

expire_logs_days	15

explicit_defaults_for_timestamp	OFF

extra_max_connections	1

extra_port	0

flush	OFF

flush_time	0

foreign_key_checks	ON

ft_boolean_syntax	+ -><()~*:""&|

ft_max_word_len	20

ft_min_word_len	3

ft_query_expansion_limit	20

ft_stopword_file	(built-in)

general_log	OFF

general_log_file	/data/mysql-log/mysql.log

group_concat_max_len	16384

gtid_binlog_pos

gtid_binlog_state

gtid_current_pos	0-1-139164901

gtid_domain_id	0

gtid_ignore_duplicates	OFF

gtid_slave_pos	0-1-139164901

gtid_strict_mode	OFF

have_compress	YES

have_crypt	YES

have_dynamic_loading	YES

have_geometry	YES

have_openssl	NO

have_profiling	YES

have_query_cache	YES

have_rtree_keys	YES

have_ssl	DISABLED

have_symlink	YES

histogram_size	0

histogram_type	SINGLE_PREC_HB

host_cache_size	138

hostname	SssssSlave

ignore_builtin_innodb	OFF

ignore_db_dirs

init_connect

init_file

init_slave

innodb_adaptive_flushing	ON

innodb_adaptive_flushing_lwm	10.000000

innodb_adaptive_hash_index	ON

innodb_adaptive_hash_index_partitions	1

innodb_adaptive_max_sleep_delay	150000

innodb_additional_mem_pool_size	2097152

innodb_api_bk_commit_interval	5

innodb_api_disable_rowlock	OFF

innodb_api_enable_binlog	OFF

innodb_api_enable_mdl	OFF

innodb_api_trx_level	0

innodb_autoextend_increment	256

innodb_autoinc_lock_mode	1

innodb_background_scrub_data_check_interval	3600

innodb_background_scrub_data_compressed	OFF

innodb_background_scrub_data_interval	604800

innodb_background_scrub_data_uncompressed	OFF

innodb_buf_dump_status_frequency	0

innodb_buffer_pool_dump_at_shutdown	OFF

innodb_buffer_pool_dump_now	OFF

innodb_buffer_pool_dump_pct	100

innodb_buffer_pool_filename	ib_buffer_pool

innodb_buffer_pool_instances	8

innodb_buffer_pool_load_abort	OFF

innodb_buffer_pool_load_at_startup	OFF

innodb_buffer_pool_load_now	OFF

innodb_buffer_pool_populate	OFF

innodb_buffer_pool_size	134217728

innodb_change_buffer_max_size	25

innodb_change_buffering	all

innodb_checksum_algorithm	INNODB

innodb_checksums	ON

innodb_cleaner_lsn_age_factor	HIGH_CHECKPOINT

innodb_cmp_per_index_enabled	OFF

innodb_commit_concurrency	0

innodb_compression_algorithm	zlib

innodb_compression_failure_threshold_pct	5

innodb_compression_level	6

innodb_compression_pad_pct_max	50

innodb_concurrency_tickets	5000

innodb_corrupt_table_action	assert

innodb_data_file_path	ibdata1:4096M:autoextend

innodb_data_home_dir

innodb_default_encryption_key_id	1

innodb_default_row_format	compact

innodb_defragment	OFF

innodb_defragment_fill_factor	0.900000

innodb_defragment_fill_factor_n_recs	20

innodb_defragment_frequency	40

innodb_defragment_n_pages	7

innodb_defragment_stats_accuracy	0

innodb_disable_sort_file_cache	OFF

innodb_disallow_writes	OFF

innodb_doublewrite	ON

innodb_empty_free_list_algorithm	LEGACY

innodb_encrypt_log	OFF

innodb_encrypt_tables	OFF

innodb_encryption_rotate_key_age	1

innodb_encryption_rotation_iops	100

innodb_encryption_threads	0

innodb_fake_changes	OFF

innodb_fast_shutdown	1

innodb_fatal_semaphore_wait_threshold	600

innodb_file_format	Antelope

innodb_file_format_check	ON

innodb_file_format_max	Antelope

innodb_file_per_table	ON

innodb_flush_log_at_timeout	1

innodb_flush_log_at_trx_commit	0

innodb_flush_method

innodb_flush_neighbors	1

innodb_flushing_avg_loops	30

innodb_force_load_corrupted	OFF

innodb_force_primary_key	OFF

innodb_force_recovery	0

innodb_foreground_preflush	EXPONENTIAL_BACKOFF

innodb_ft_aux_table

innodb_ft_cache_size	8000000

innodb_ft_enable_diag_print	OFF

innodb_ft_enable_stopword	ON

innodb_ft_max_token_size	84

innodb_ft_min_token_size	3

innodb_ft_num_word_optimize	2000

innodb_ft_result_cache_limit	2000000000

innodb_ft_server_stopword_table

innodb_ft_sort_pll_degree	2

innodb_ft_total_cache_size	640000000

innodb_ft_user_stopword_table

innodb_idle_flush_pct	100

innodb_immediate_scrub_data_uncompressed	OFF

innodb_instrument_semaphores	OFF

innodb_io_capacity	200

innodb_io_capacity_max	2000

innodb_kill_idle_transaction	0

innodb_large_prefix	OFF

innodb_lock_schedule_algorithm	fcfs

innodb_lock_wait_timeout	50

innodb_locking_fake_changes	ON

innodb_locks_unsafe_for_binlog	OFF

innodb_log_arch_dir	./

innodb_log_arch_expire_sec	0

innodb_log_archive	OFF

innodb_log_block_size	512

innodb_log_buffer_size	1048576

innodb_log_checksum_algorithm	INNODB

innodb_log_compressed_pages	ON

innodb_log_file_size	20971520

innodb_log_files_in_group	2

innodb_log_group_home_dir	./

innodb_lru_scan_depth	1024

innodb_max_bitmap_file_size	104857600

innodb_max_changed_pages	1000000

innodb_max_dirty_pages_pct	75.000000

innodb_max_dirty_pages_pct_lwm	0.001000

innodb_max_purge_lag	0

innodb_max_purge_lag_delay	0

innodb_mirrored_log_groups	1

innodb_monitor_disable

innodb_monitor_enable

innodb_monitor_reset

innodb_monitor_reset_all

innodb_mtflush_threads	8

innodb_old_blocks_pct	37

innodb_old_blocks_time	1000

innodb_online_alter_log_max_size	134217728

innodb_open_files	300

innodb_optimize_fulltext_only	OFF

innodb_page_size	16384

innodb_prefix_index_cluster_optimization	OFF

innodb_print_all_deadlocks	OFF

innodb_print_lock_wait_timeout_info	OFF

innodb_purge_batch_size	300

innodb_purge_threads	1

innodb_random_read_ahead	OFF

innodb_read_ahead_threshold	56

innodb_read_io_threads	4

innodb_read_only	OFF

innodb_replication_delay	0

innodb_rollback_on_timeout	OFF

innodb_rollback_segments	128

innodb_sched_priority_cleaner	19

innodb_scrub_log	OFF

innodb_scrub_log_speed	256

innodb_show_locks_held	10

innodb_show_verbose_locks	0

innodb_simulate_comp_failures	0

innodb_sort_buffer_size	1048576

innodb_spin_wait_delay	6

innodb_stats_auto_recalc	ON

innodb_stats_include_delete_marked	OFF

innodb_stats_method	nulls_equal

innodb_stats_modified_counter	0

innodb_stats_on_metadata	OFF

innodb_stats_persistent	ON

innodb_stats_persistent_sample_pages	20

innodb_stats_sample_pages	8

innodb_stats_traditional	ON

innodb_stats_transient_sample_pages	8

innodb_status_output	OFF

innodb_status_output_locks	OFF

innodb_strict_mode	OFF

innodb_support_xa	ON

innodb_sync_array_size	1

innodb_sync_spin_loops	30

innodb_table_locks	ON

innodb_thread_concurrency	7

innodb_thread_sleep_delay	0

innodb_tmpdir

innodb_track_changed_pages	OFF

innodb_undo_directory	.

innodb_undo_logs	128

innodb_undo_tablespaces	0

innodb_use_atomic_writes	OFF

innodb_use_fallocate	OFF

innodb_use_global_flush_log_at_trx_commit	ON

innodb_use_mtflush	OFF

innodb_use_native_aio	ON

innodb_use_stacktrace	OFF

innodb_use_sys_malloc	ON

innodb_use_trim	OFF

innodb_version	5.6.44-86.0

innodb_write_io_threads	4

interactive_timeout	28800

join_buffer_size	524288

join_buffer_space_limit	2097152

join_cache_level	2

keep_files_on_create	OFF

key_buffer_size	8388608

key_cache_age_threshold	300

key_cache_block_size	1024

key_cache_division_limit	100

key_cache_file_hash_size	512

key_cache_segments	0

large_files_support	ON

large_page_size	0

large_pages	OFF

lc_messages	en_US

lc_messages_dir	/usr/share/mysql

lc_time_names	en_US

license	GPL

local_infile	ON

lock_wait_timeout	31536000

locked_in_memory	OFF

log_bin	OFF

log_bin_basename

log_bin_index

log_bin_trust_function_creators	OFF

log_error	/data/mysql-log/mysql.err

log_output	FILE

log_queries_not_using_indexes	OFF

log_slave_updates	OFF

log_slow_admin_statements	OFF

log_slow_filter	admin,filesort,filesort_on_disk,full_join,full_scan,query_cache,query_cache_miss,tmp_table,tmp_table_on_disk

log_slow_rate_limit	1

log_slow_slave_statements	OFF

log_slow_verbosity

log_tc_size	24576

log_warnings	2

long_query_time	10.000000

low_priority_updates	OFF

lower_case_file_system	OFF

lower_case_table_names	0

master_verify_checksum	OFF

max_allowed_packet	4194304

max_binlog_cache_size	18446744073709547520

max_binlog_size	8388608

max_binlog_stmt_cache_size	18446744073709547520

max_connect_errors	8

max_connections	10

max_delayed_threads	20

max_digest_length	1024

max_error_count	64

max_heap_table_size	33554432

max_insert_delayed_threads	20

max_join_size	18446744073709551615

max_length_for_sort_data	1024

max_long_data_size	4194304

max_prepared_stmt_count	16382

max_relay_log_size	8388608

max_seeks_for_key	4294967295

max_session_mem_used	9223372036854775807

max_sort_length	256

max_sp_recursion_depth	8

max_statement_time	0.000000

max_tmp_tables	32

max_user_connections	19

max_write_lock_count	4294967295

metadata_locks_cache_size	1024

metadata_locks_hash_instances	8

min_examined_row_limit	0

mrr_buffer_size	262144

multi_range_count	256

myisam_block_size	1024

myisam_data_pointer_size	6

myisam_max_sort_file_size	2147483648

myisam_mmap_size	18446744073709551615

myisam_recover_options	BACKUP

myisam_repair_threads	1

myisam_sort_buffer_size	16777216

myisam_stats_method	NULLS_UNEQUAL

myisam_use_mmap	OFF

mysql56_temporal_format	ON

net_buffer_length	16384

net_read_timeout	30

net_retry_count	10

net_write_timeout	60

old	OFF

old_alter_table	OFF

old_mode

old_passwords	OFF

open_files_limit	743

optimizer_prune_level	1

optimizer_search_depth	62

optimizer_selectivity_sampling_limit	100

optimizer_switch	index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=off,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=off

optimizer_use_condition_selectivity	1

performance_schema	OFF

performance_schema_accounts_size	-1

performance_schema_digests_size	-1

performance_schema_events_stages_history_long_size	-1

performance_schema_events_stages_history_size	-1

performance_schema_events_statements_history_long_size	-1

performance_schema_events_statements_history_size	-1

performance_schema_events_waits_history_long_size	-1

performance_schema_events_waits_history_size	-1

performance_schema_hosts_size	-1

performance_schema_max_cond_classes	80

performance_schema_max_cond_instances	-1

performance_schema_max_digest_length	1024

performance_schema_max_file_classes	50

performance_schema_max_file_handles	32768

performance_schema_max_file_instances	-1

performance_schema_max_mutex_classes	200

performance_schema_max_mutex_instances	-1

performance_schema_max_rwlock_classes	40

performance_schema_max_rwlock_instances	-1

performance_schema_max_socket_classes	10

performance_schema_max_socket_instances	-1

performance_schema_max_stage_classes	150

performance_schema_max_statement_classes	178

performance_schema_max_table_handles	-1

performance_schema_max_table_instances	-1

performance_schema_max_thread_classes	50

performance_schema_max_thread_instances	-1

performance_schema_session_connect_attrs_size	-1

performance_schema_setup_actors_size	100

performance_schema_setup_objects_size	100

performance_schema_users_size	-1

pid_file	/var/run/mysqld/mysqld.pid

plugin_dir	/usr/lib/x86_64-linux-gnu/mariadb18/plugin/

plugin_maturity	unknown

port	3306

preload_buffer_size	32768

profiling	OFF

profiling_history_size	15

progress_report_time	5

protocol_version	10

query_alloc_block_size	16384

query_cache_limit	262144

query_cache_min_res_unit	4096

query_cache_size	33554432

query_cache_strip_comments	OFF

query_cache_type	ON

query_cache_wlock_invalidate	ON

query_prealloc_size	24576

range_alloc_block_size	4096

read_buffer_size	786432

read_only	OFF

read_rnd_buffer_size	2097152

relay_log	/data/mysql-log/relay

relay_log_basename	/data/mysql-log/relay

relay_log_index	/data/mysql-log/relay.index

relay_log_info_file	/data/mysql-log/relay.info

relay_log_purge	ON

relay_log_recovery	OFF

relay_log_space_limit	0

replicate_annotate_row_events	OFF

replicate_do_db

replicate_do_table

replicate_events_marked_for_skip	REPLICATE

replicate_ignore_db

replicate_ignore_table	sssss.md_session

replicate_wild_do_table

replicate_wild_ignore_table

report_host

report_password

report_port	3306

report_user

rowid_merge_buff_size	8388608

secure_auth	ON

secure_file_priv

server_id	2

skip_external_locking	ON

skip_name_resolve	ON

skip_networking	OFF

skip_show_database	OFF

slave_compressed_protocol	OFF

slave_ddl_exec_mode	IDEMPOTENT

slave_domain_parallel_threads	0

slave_exec_mode	STRICT

slave_load_tmpdir	/tmp

slave_max_allowed_packet	1073741824

slave_net_timeout	3600

slave_parallel_max_queued	131072

slave_parallel_mode	conservative

slave_parallel_threads	0

slave_run_triggers_for_rbr	NO

slave_skip_errors	OFF

slave_sql_verify_checksum	ON

slave_transaction_retries	10

slave_type_conversions

slow_launch_time	2

slow_query_log	ON

slow_query_log_file	/data/mysql-log/mysql-slow.log

socket	/var/run/mysqld/mysqld.sock

sort_buffer_size	786432

sql_auto_is_null	OFF

sql_big_selects	ON

sql_buffer_result	OFF

sql_log_bin	ON

sql_log_off	OFF

sql_mode	PIPES_AS_CONCAT,STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION

sql_notes	ON

sql_quote_show_create	ON

sql_safe_updates	OFF

sql_select_limit	18446744073709551615

sql_slave_skip_counter	0

sql_warnings	OFF

ssl_ca

ssl_capath

ssl_cert

ssl_cipher

ssl_crl

ssl_crlpath

ssl_key

storage_engine	InnoDB

stored_program_cache	256

strict_password_validation	ON

sync_binlog	0

sync_frm	ON

sync_master_info	10000

sync_relay_log	10000

sync_relay_log_info	10000

system_time_zone	CET

table_definition_cache	400

table_open_cache	350

thread_cache_size	4

thread_concurrency	3

thread_handling	one-thread-per-connection

thread_pool_idle_timeout	60

thread_pool_max_threads	1000

thread_pool_oversubscribe	3

thread_pool_size	2

thread_pool_stall_limit	500

thread_stack	196608

time_format	%H:%i:%s

time_zone	SYSTEM

timed_mutexes	OFF

tmp_table_size	12582912

tmpdir	/tmp

transaction_alloc_block_size	8192

transaction_prealloc_size	4096

tx_isolation	READ-COMMITTED

tx_read_only	OFF

unique_checks	ON

updatable_views_with_limit	YES

use_stat_tables	NEVER

userstat	OFF

version	10.1.41-MariaDB-0+deb9u1

version_comment	Debian 9.9

version_compile_machine	x86_64

version_compile_os	debian-linux-gnu

version_malloc_library	system jemalloc

version_ssl_library	YaSSL 2.4.4

wait_timeout	28800

wsrep_osu_method	TOI

wsrep_auto_increment_control	ON

wsrep_causal_reads	OFF

wsrep_certification_rules	strict

wsrep_certify_nonpk	ON

wsrep_cluster_address

wsrep_cluster_name	my_wsrep_cluster

wsrep_convert_lock_to_trx	OFF

wsrep_data_home_dir	/data/mysql/

wsrep_dbug_option

wsrep_debug	OFF

wsrep_desync	OFF

wsrep_dirty_reads	OFF

wsrep_drupal_282555_workaround	OFF

wsrep_forced_binlog_format	NONE

wsrep_gtid_domain_id	0

wsrep_gtid_mode	OFF

wsrep_load_data_splitting	ON

wsrep_log_conflicts	OFF

wsrep_max_ws_rows	0

wsrep_max_ws_size	2147483647

wsrep_mysql_replication_bundle	0

wsrep_node_address

wsrep_node_incoming_address	AUTO

wsrep_node_name	SssssSlave

wsrep_notify_cmd

wsrep_on	OFF

wsrep_patch_version	wsrep_25.24

wsrep_provider	none

wsrep_provider_options

wsrep_recover	OFF

wsrep_reject_queries	NONE

wsrep_replicate_myisam	OFF

wsrep_restart_slave	OFF

wsrep_retry_autocommit	1

wsrep_slave_fk_checks	ON

wsrep_slave_uk_checks	OFF

wsrep_slave_threads	1

wsrep_sst_auth

wsrep_sst_donor

wsrep_sst_donor_rejects_queries	OFF

wsrep_sst_method	rsync

wsrep_sst_receive_address	AUTO

wsrep_start_position	00000000-0000-0000-0000-000000000000:-1

wsrep_sync_wait	0

SHOW GLOBAL STATUS

Variable_name	Value

Aborted_clients	0

Aborted_connects	0

Access_denied_errors	0

Acl_column_grants	0

Acl_database_grants	10

Acl_function_grants	0

Acl_procedure_grants	0

Acl_proxy_users	1

Acl_role_grants	0

Acl_roles	0

Acl_table_grants	6

Acl_users	11

Aria_pagecache_blocks_not_flushed	0

Aria_pagecache_blocks_unused	15706

Aria_pagecache_blocks_used	0

Aria_pagecache_read_requests	0

Aria_pagecache_reads	0

Aria_pagecache_write_requests	0

Aria_pagecache_writes	0

Aria_transaction_log_syncs	0

Binlog_commits	0

Binlog_group_commits	0

Binlog_group_commit_trigger_count	0

Binlog_group_commit_trigger_lock_wait	0

Binlog_group_commit_trigger_timeout	0

Binlog_snapshot_file

Binlog_snapshot_position	0

Binlog_bytes_written	0

Binlog_cache_disk_use	0

Binlog_cache_use	0

Binlog_stmt_cache_disk_use	0

Binlog_stmt_cache_use	0

Busy_time	0.000000

Bytes_received	10712143359

Bytes_sent	3290678354

Com_admin_commands	2

Com_alter_db	70

Com_alter_db_upgrade	0

Com_alter_event	0

Com_alter_function	0

Com_alter_procedure	0

Com_alter_server	0

Com_alter_table	674

Com_alter_tablespace	0

Com_analyze	0

Com_assign_to_keycache	0

Com_begin	5

Com_binlog	0

Com_call_procedure	0

Com_change_db	2

Com_change_master	2

Com_check	0

Com_checksum	0

Com_commit	341

Com_compound_sql	0

Com_create_db	1

Com_create_event	0

Com_create_function	4

Com_create_index	0

Com_create_procedure	3

Com_create_role	0

Com_create_server	0

Com_create_table	337

Com_create_temporary_table	0

Com_create_trigger	86

Com_create_udf	0

Com_create_user	0

Com_create_view	0

Com_dealloc_sql	0

Com_delete	0

Com_delete_multi	0

Com_do	0

Com_drop_db	1

Com_drop_event	0

Com_drop_function	4

Com_drop_index	0

Com_drop_procedure	3

Com_drop_role	0

Com_drop_server	0

Com_drop_table	337

Com_drop_temporary_table	0

Com_drop_trigger	0

Com_drop_user	0

Com_drop_view	0

Com_empty_query	0

Com_execute_sql	0

Com_flush	4

Com_get_diagnostics	0

Com_grant	0

Com_grant_role	0

Com_ha_close	0

Com_ha_open	0

Com_ha_read	0

Com_help	0

Com_insert	9480

Com_insert_select	0

Com_install_plugin	0

Com_kill	0

Com_load	0

Com_lock_tables	337

Com_optimize	0

Com_preload_keys	0

Com_prepare_sql	0

Com_purge	0

Com_purge_before_date	0

Com_release_savepoint	0

Com_rename_table	0

Com_rename_user	0

Com_repair	0

Com_replace	0

Com_replace_select	0

Com_reset	0

Com_resignal	0

Com_revoke	0

Com_revoke_all	0

Com_revoke_role	0

Com_rollback	0

Com_rollback_to_savepoint	0

Com_savepoint	0

Com_select	274

Com_set_option	2587

Com_show_authors	0

Com_show_binlog_events	0

Com_show_binlogs	0

Com_show_charsets	0

Com_show_collations	0

Com_show_contributors	0

Com_show_create_db	0

Com_show_create_event	0

Com_show_create_func	0

Com_show_create_proc	0

Com_show_create_table	2

Com_show_create_trigger	0

Com_show_databases	12

Com_show_engine_logs	0

Com_show_engine_mutex	0

Com_show_engine_status	2

Com_show_errors	0

Com_show_events	0

Com_show_explain	0

Com_show_fields	3735

Com_show_function_status	0

Com_show_generic	0

Com_show_grants	0

Com_show_keys	0

Com_show_master_status	0

Com_show_open_tables	0

Com_show_plugins	0

Com_show_privileges	0

Com_show_procedure_status	0

Com_show_processlist	66

Com_show_profile	0

Com_show_profiles	0

Com_show_relaylog_events	0

Com_show_slave_hosts	0

Com_show_slave_status	22

Com_show_status	1

Com_show_storage_engines	0

Com_show_table_status	0

Com_show_tables	12

Com_show_triggers	0

Com_show_variables	4

Com_show_warnings	0

Com_shutdown	0

Com_signal	0

Com_start_all_slaves	0

Com_start_slave	1

Com_stmt_close	0

Com_stmt_execute	0

Com_stmt_fetch	0

Com_stmt_prepare	0

Com_stmt_reprepare	0

Com_stmt_reset	0

Com_stmt_send_long_data	0

Com_stop_all_slaves	0

Com_stop_slave	0

Com_truncate	0

Com_uninstall_plugin	0

Com_unlock_tables	337

Com_update	1

Com_update_multi	0

Com_xa_commit	0

Com_xa_end	0

Com_xa_prepare	0

Com_xa_recover	0

Com_xa_rollback	0

Com_xa_start	0

Compression	OFF

Connection_errors_accept	0

Connection_errors_internal	0

Connection_errors_max_connections	0

Connection_errors_peer_address	0

Connection_errors_select	0

Connection_errors_tcpwrap	0

Connections	285

Cpu_time	0.000000

Created_tmp_disk_tables	2

Created_tmp_files	6

Created_tmp_tables	42

Delayed_errors	0

Delayed_insert_threads	0

Delayed_writes	0

Delete_scan	0

Empty_queries	238

Executed_events	0

Executed_triggers	0

Feature_delay_key_write	0

Feature_dynamic_columns	0

Feature_fulltext	0

Feature_gis	0

Feature_locale	0

Feature_subquery	12

Feature_timezone	5

Feature_trigger	210

Feature_xml	0

Flush_commands	1

Handler_commit	10461

Handler_delete	10

Handler_discover	0

Handler_external_lock	0

Handler_icp_attempts	0

Handler_icp_match	0

Handler_mrr_init	0

Handler_mrr_key_refills	0

Handler_mrr_rowid_refills	0

Handler_prepare	0

Handler_read_first	5

Handler_read_key	6579291

Handler_read_last	0

Handler_read_next	7970057

Handler_read_prev	0

Handler_read_retry	0

Handler_read_rnd	0

Handler_read_rnd_deleted	0

Handler_read_rnd_next	807268

Handler_rollback	0

Handler_savepoint	0

Handler_savepoint_rollback	0

Handler_tmp_update	0

Handler_tmp_write	18025

Handler_update	0

Handler_write	24673912

Innodb_available_undo_logs	128

Innodb_background_log_sync	92972

Innodb_buffer_pool_bytes_data	103809024

Innodb_buffer_pool_bytes_dirty	0

Innodb_buffer_pool_dump_status	Dumping buffer pool(s) not yet started

Innodb_buffer_pool_load_status	Loading buffer pool(s) not yet started

Innodb_buffer_pool_pages_data	6336

Innodb_buffer_pool_pages_dirty	0

Innodb_buffer_pool_pages_flushed	811467

Innodb_buffer_pool_pages_free	1024

Innodb_buffer_pool_pages_lru_flushed	0

Innodb_buffer_pool_pages_made_not_young	11840343

Innodb_buffer_pool_pages_made_young	15865

Innodb_buffer_pool_pages_misc	831

Innodb_buffer_pool_pages_old	2318

Innodb_buffer_pool_pages_total	8191

Innodb_buffer_pool_read_ahead	198337

Innodb_buffer_pool_read_ahead_evicted	395

Innodb_buffer_pool_read_ahead_rnd	0

Innodb_buffer_pool_read_requests	228086694

Innodb_buffer_pool_reads	169377

Innodb_buffer_pool_wait_free	0

Innodb_buffer_pool_write_requests	88089440

Innodb_checkpoint_age	0

Innodb_checkpoint_max_age	32849880

Innodb_data_fsyncs	41692

Innodb_data_pending_fsyncs	0

Innodb_data_pending_reads	0

Innodb_data_pending_writes	0

Innodb_data_read	6056645120

Innodb_data_reads	369806

Innodb_data_writes	873397

Innodb_data_written	37874769408

Innodb_dblwr_pages_written	811467

Innodb_dblwr_writes	8730

Innodb_deadlocks	0

Innodb_have_atomic_builtins	ON

Innodb_history_list_length	56

Innodb_ibuf_discarded_delete_marks	0

Innodb_ibuf_discarded_deletes	0

Innodb_ibuf_discarded_inserts	0

Innodb_ibuf_free_list	717

Innodb_ibuf_merged_delete_marks	0

Innodb_ibuf_merged_deletes	0

Innodb_ibuf_merged_inserts	28530

Innodb_ibuf_merges	2075

Innodb_ibuf_segment_size	719

Innodb_ibuf_size	1

Innodb_log_waits	73

Innodb_log_write_requests	25277620

Innodb_log_writes	29807

Innodb_lsn_current	306283005919

Innodb_lsn_flushed	306283005919

Innodb_lsn_last_checkpoint	306283005919

Innodb_master_thread_active_loops	816

Innodb_master_thread_idle_loops	92174

Innodb_max_trx_id	13477470

Innodb_mem_adaptive_hash	15832672

Innodb_mem_dictionary	972528

Innodb_mem_total	140574720

Innodb_mutex_os_waits	12372

Innodb_mutex_spin_rounds	408545

Innodb_mutex_spin_waits	41928

Innodb_oldest_view_low_limit_trx_id	0

Innodb_os_log_fsyncs	7712

Innodb_os_log_pending_fsyncs	0

Innodb_os_log_pending_writes	0

Innodb_os_log_written	11284195840

Innodb_page_size	16384

Innodb_pages_created	692551

Innodb_pages_read	369663

Innodb_pages0_read	132

Innodb_pages_written	811467

Innodb_purge_trx_id	13477395

Innodb_purge_undo_no	0

Innodb_read_views_memory	104

Innodb_row_lock_current_waits	0

Innodb_row_lock_time	0

Innodb_row_lock_time_avg	0

Innodb_row_lock_time_max	0

Innodb_row_lock_waits	0

Innodb_rows_deleted	0

Innodb_rows_inserted	24597562

Innodb_rows_read	9704240

Innodb_rows_updated	0

Innodb_system_rows_deleted	0

Innodb_system_rows_inserted	0

Innodb_system_rows_read	0

Innodb_system_rows_updated	0

Innodb_s_lock_os_waits	658

Innodb_s_lock_spin_rounds	31759

Innodb_s_lock_spin_waits	6659

Innodb_truncated_status_writes	0

Innodb_x_lock_os_waits	3221

Innodb_x_lock_spin_rounds	128098

Innodb_x_lock_spin_waits	10750

Innodb_page_compression_saved	0

Innodb_page_compression_trim_sect512	0

Innodb_page_compression_trim_sect1024	0

Innodb_page_compression_trim_sect2048	0

Innodb_page_compression_trim_sect4096	0

Innodb_page_compression_trim_sect8192	0

Innodb_page_compression_trim_sect16384	0

Innodb_page_compression_trim_sect32768	0

Innodb_num_index_pages_written	622214

Innodb_num_non_index_pages_written	236593

Innodb_num_pages_page_compressed	0

Innodb_num_page_compressed_trim_op	0

Innodb_num_page_compressed_trim_op_saved	0

Innodb_num_pages_page_decompressed	0

Innodb_num_pages_page_compression_error	0

Innodb_num_pages_encrypted	0

Innodb_num_pages_decrypted	0

Innodb_have_lz4	OFF

Innodb_have_lzo	OFF

Innodb_have_lzma	OFF

Innodb_have_bzip2	OFF

Innodb_have_snappy	OFF

Innodb_defragment_compression_failures	0

Innodb_defragment_failures	0

Innodb_defragment_count	0

Innodb_onlineddl_rowlog_rows	0

Innodb_onlineddl_rowlog_pct_used	0

Innodb_onlineddl_pct_progress	0

Innodb_secondary_index_triggered_cluster_reads	2548316

Innodb_secondary_index_triggered_cluster_reads_avoided	0

Innodb_buffered_aio_submitted	198337

Innodb_encryption_rotation_pages_read_from_cache	0

Innodb_encryption_rotation_pages_read_from_disk	0

Innodb_encryption_rotation_pages_modified	0

Innodb_encryption_rotation_pages_flushed	0

Innodb_encryption_rotation_estimated_iops	0

Innodb_encryption_key_rotation_list_length	0

Innodb_encryption_n_merge_blocks_encrypted	0

Innodb_encryption_n_merge_blocks_decrypted	0

Innodb_encryption_n_rowlog_blocks_encrypted	0

Innodb_encryption_n_rowlog_blocks_decrypted	0

Innodb_scrub_background_page_reorganizations	0

Innodb_scrub_background_page_splits	0

Innodb_scrub_background_page_split_failures_underflow	0

Innodb_scrub_background_page_split_failures_out_of_filespace	0

Innodb_scrub_background_page_split_failures_missing_index	0

Innodb_scrub_background_page_split_failures_unknown	0

Innodb_encryption_num_key_requests	0

Key_blocks_not_flushed	0

Key_blocks_unused	6330

Key_blocks_used	1231

Key_blocks_warm	81

Key_read_requests	19180717

Key_reads	366

Key_write_requests	81645

Key_writes	1965

Last_query_cost	0.000000

Master_gtid_wait_count	0

Master_gtid_wait_time	0

Master_gtid_wait_timeouts	0

Max_statement_time_exceeded	0

Max_used_connections	2

Memory_used	243120432

Not_flushed_delayed_rows	0

Open_files	443

Open_streams	0

Open_table_definitions	370

Open_tables	350

Opened_files	5982

Opened_plugin_libraries	0

Opened_table_definitions	1465

Opened_tables	1535

Opened_views	0

Performance_schema_accounts_lost	0

Performance_schema_cond_classes_lost	0

Performance_schema_cond_instances_lost	0

Performance_schema_digest_lost	0

Performance_schema_file_classes_lost	0

Performance_schema_file_handles_lost	0

Performance_schema_file_instances_lost	0

Performance_schema_hosts_lost	0

Performance_schema_locker_lost	0

Performance_schema_mutex_classes_lost	0

Performance_schema_mutex_instances_lost	0

Performance_schema_rwlock_classes_lost	0

Performance_schema_rwlock_instances_lost	0

Performance_schema_session_connect_attrs_lost	0

Performance_schema_socket_classes_lost	0

Performance_schema_socket_instances_lost	0

Performance_schema_stage_classes_lost	0

Performance_schema_statement_classes_lost	0

Performance_schema_table_handles_lost	0

Performance_schema_table_instances_lost	0

Performance_schema_thread_classes_lost	0

Performance_schema_thread_instances_lost	0

Performance_schema_users_lost	0

Prepared_stmt_count	0

Qcache_free_blocks	1

Qcache_free_memory	33536824

Qcache_hits	0

Qcache_inserts	0

Qcache_lowmem_prunes	0

Qcache_not_cached	274

Qcache_queries_in_cache	0

Qcache_total_blocks	1

Queries	19036

Questions	19022

Rows_read	11721162

Rows_sent	3402308

Rows_tmp_read	15272

Rpl_status	AUTH_MASTER

Select_full_join	0

Select_full_range_join	0

Select_range	0

Select_range_check	0

Select_scan	52

Slave_connections	0

Slave_heartbeat_period	1800.000

Slave_open_temp_tables	0

Slave_received_heartbeats	0

Slave_retried_transactions	0

Slave_running	ON

Slave_skipped_errors	0

Slaves_connected	0

Slaves_running	1

Slow_launch_threads	0

Slow_queries	4

Sort_merge_passes	0

Sort_priority_queue_sorts	0

Sort_range	0

Sort_rows	0

Sort_scan	0

Ssl_accept_renegotiates	0

Ssl_accepts	0

Ssl_callback_cache_hits	0

Ssl_cipher

Ssl_cipher_list

Ssl_client_connects	0

Ssl_connect_renegotiates	0

Ssl_ctx_verify_depth	0

Ssl_ctx_verify_mode	0

Ssl_default_timeout	0

Ssl_finished_accepts	0

Ssl_finished_connects	0

Ssl_server_not_after

Ssl_server_not_before

Ssl_session_cache_hits	0

Ssl_session_cache_misses	0

Ssl_session_cache_mode	NONE

Ssl_session_cache_overflows	0

Ssl_session_cache_size	0

Ssl_session_cache_timeouts	0

Ssl_sessions_reused	0

Ssl_used_session_cache_entries	0

Ssl_verify_depth	0

Ssl_verify_mode	0

Ssl_version

Subquery_cache_hit	15158

Subquery_cache_miss	12054

Syncs	3058

Table_locks_immediate	3249

Table_locks_waited	0

Tc_log_max_pages_used	0

Tc_log_page_size	4096

Tc_log_page_waits	0

Threadpool_idle_threads	0

Threadpool_threads	0

Threads_cached	1

Threads_connected	1

Threads_created	2

Threads_running	1

Update_scan	0

Uptime	93089

Uptime_since_flush_status	93089

wsrep_cluster_conf_id	18446744073709551615

wsrep_cluster_size	0

wsrep_cluster_state_uuid

wsrep_cluster_status	Disconnected

wsrep_connected	OFF

wsrep_local_bf_aborts	0

wsrep_local_index	18446744073709551615

wsrep_provider_name

wsrep_provider_vendor

wsrep_provider_version

wsrep_ready	OFF

wsrep_thread_count	0

Comment by Andrei Elkin [ 2019-12-23 ]

mperez: I still think we can't exclude ~~MDEV-20247~~ because there's a path from Rows_log_event::find_row() into ~~MDEV-20247~~'s hanging innobase_srv_conc_enter_innodb:

Thread 29 (Thread 0x7f9318e1f700 (LWP 17465)):
#0 0x00007f93177a7603 in select () from /lib/x86_64-linux-gnu/libc.so.6
#1 0x000055ec5da3f4ff in ?? ()
#2 0x000055ec5d9bf542 in ?? ()
#3 0x000055ec5d9d6ff8 in ?? ()
#4 0x000055ec5d74d082 in handler::ha_rnd_pos(unsigned char*, unsigned char*) ()
#5 0x000055ec5d81bf8c in Rows_log_event::find_row(rpl_group_info*) ()
#6 0x000055ec5d81c604 in Update_rows_log_event::do_exec_row(rpl_group_info*) ()
#7 0x000055ec5d80ef3b in Rows_log_event::do_apply_event(rpl_group_info*) ()
#8 0x000055ec5d52c4ab in ?? ()
#9 0x000055ec5d537553 in handle_slave_sql ()
#10 0x00007f9318b814a4 in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#11 0x00007f93177aed0f in clone () from /lib/x86_64-linux-gnu/libc.so.6

Comment by Sujatha Sivakumar (Inactive) [ 2019-12-23 ]

Hello Miguel P,

Thank you for providing the above details.
Is there a difference in table structure between 'master' and 'slave'.
Can you please share 'SHOW CREATE TABLE ContactCredential;' output on master.
Since you are able to reproduce the issue consistently at your end with few
hundred rows will it be possible to make a simple test case with dummy tables.

Thank you!

Comment by Miguel P [ 2019-12-23 ]

Hi Elkin,

~~MDEV-20247~~ says the IO thread hangs on "preparing", yet this is about the SQL thread hanging on find_rows(-1)... Yet I did notice ~~MDEV-20247~~ says it affected 10.1.41 and a comment said a downgrade (for 10.2) fixed it.

When I was setting this up I had an older package with an older 10.1, and I believe replication worked fine for a day, but I didn't watch it. On the next day I upgraded Debian packages and I noticed this started happening. It could be a bug specific to 10.1.41, I'll try to get 10.1.42+, 10.2.27+, 10.3.18+ or 10.4.8+ as indicated by ~~MDEV-20247~~ . If the upgrade solves it I'll report back here.

Hi sujatha.sivakumar,

The tables have identical SHOW CREATE TABLE, and they have been freshly created with a new mysqldump. I've double checkde it's the same above verbatim including all options.

I'll try updating to the versions I mentioned above, and if this still happens, I'll try to isolate it and share the DB dumps and setups that will reproduce it.

Comment by Miguel P [ 2020-01-24 ]

I can confirm replication works after upgrading to 10.3.18, so it had to be that bug. You can close this, thank you very much for your help.

Not sure if it's worth notifying Debian that replication will be broken for Stretch (Debian 9, oldstable) as it's using 10.1.41 and they should try to upgrade the MariaDB packages to a newer 10.1.

Cheers

[MDEV-20398] Slave sql thread blocked by Update_rows_log_event::find_row(-1) Created: 2019-08-21 Updated: 2020-03-20 Resolved: 2020-02-25
Status:	Closed
Project:	MariaDB Server
Component/s:	Replication
Affects Version/s:	10.1.22
Fix Version/s:	N/A

[MDEV-20398] Slave sql thread blocked by Update_rows_log_event::find_row(-1) Created: 2019-08-21 Updated: 2020-03-20 Resolved: 2020-02-25