[MDEV-24296] Stuck replication - Slave SQL thread is blocked by Update_rows_log_event::find_row(-1) Created: 2020-11-26  Updated: 2020-12-10  Resolved: 2020-12-09

Status: Closed
Project: MariaDB Server
Component/s: Replication
Affects Version/s: 10.4.12
Fix Version/s: N/A

Type: Bug Priority: Major
Reporter: Tomas Kucera Assignee: Alice Sherepa
Resolution: Not a Bug Votes: 0
Labels: hang, replication
Environment:

OS RHEL 7.6, kernel: 3.10.0-957.el7.x86_64 #1 SMP Thu Oct 4 20:48:51 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux



 Description   

Stuck replication - Slave SQL thread is blocked by "Update_rows_log_event::find_row(-1)" state of executed SQL command

Environment parameters of Primary and Replica server
OS: RHEL 7.6
Kernel: 3.10.0-957.el7.x86_64
MariaDB version: MariaDB-server-10.4.12-1.el7
DB size: 6.1TB
Binary log format: mixed

State before issue and issue day:

  • replication was running without problems several weeks
  • replication was stopped for several days due to SW maintenance on Primary side
  • nothing changed on Primary DB only new data were still inserted or modified
  • replication started after several days
  • on Primary DB there were 13 GB changes in binlogs waiting for replication
  • performed replication start (start slave; )
  • replication started with downloading of changes from Primary binlogs to Replica relaylogs
  • first changes performed properly (replication executed several positions from relaylog)
  • but after some time the replication stopped with executing of changes from relaylogs
  • processlist shows that Slave_SQL command is in state: Update_rows_log_event::find_row(-1)
  • stop and start slave did not help to solve issue
  • kill this Slave SQL command and following start slave did not solve this issue and the same state appeared again
  • executing of relaylogs stoped on binlog position 902554004
  • extract from slave status command

    Master_Log_File: bin.000098
    Read_Master_Log_Pos: 575635222
    Exec_Master_Log_Pos: 902554004
    Seconds_Behind_Master: 1911134
    Slave_SQL_Running_State: Update_rows_log_event::find_row(-1)
    Slave_DDL_Groups: 0
    Slave_Non_Transactional_Groups: 0
    Slave_Transactional_Groups: 1
    

  • in relay log position 902554004 is following SQL alter table command

    ...
    #201102 16:50:53 server id 71  end_log_pos 902553870 CRC32 0x8c017e3b   GTID 0-71-7704 ddl
    /*!100001 SET @@session.gtid_seq_no=7704*//*!*/;
    # at 109931178
    #201102 16:50:53 server id 71  end_log_pos 902554004 CRC32 0xa0601914   Query   thread_id=25616 exec_time=1     error_code=0
    SET TIMESTAMP=1604332253/*!*/;
    ALTER TABLE values MODIFY COLUMN node varchar(9) NOT NULL
    /*!*/;
    # at 109931312
    #201102 16:51:45 server id 71  end_log_pos 902554046 CRC32 0xf9ccef43   GTID 0-71-7705 trans
    ...
    

  • size of processed table is 175MB
  • at the same same when this replication was started so another replication between another two DB servers were started also with the same parameters and version and there is not problem, only DB size is 500GB for this another environment
  • I found that similar issue was reported here MDEV-20398 but it should be solved since 10.4.8+ version and I am using 10.4.12
  • no issue found in MariaDB log file
  • I am not aware about using of any unsafe SQL command when mixed binary logging is used instead of safest row binary logging
  • in first two comments below I am sending next info: full slave status, full processlist, full innodb engine status, global status, global variables and show create table info

Do you see any issue in relay log?
Do you have any tip what to check next?
Do you have any tip for the issue reason?
Could the issue be caused by some hiccup that caused issue in binlog or relaylog or in some internal record in DB about processed comand?
Is it possible that issue MDEV-20398 is not repaired in version 10.4.12 or if I met with another reason with the same error?

Thank you



 Comments   
Comment by Tomas Kucera [ 2020-11-26 ]

Additional information

Slave status

MariaDB [(none)]> show slave status \G;
*************************** 1. row ***************************
                Slave_IO_State: Waiting for master to send event
                   Master_Host: xxx
                   Master_User: xxx
                   Master_Port: 3306
                 Connect_Retry: 60
               Master_Log_File: bin.000106
           Read_Master_Log_Pos: 603755911
                Relay_Log_File: xxx-relay-bin.000040
                 Relay_Log_Pos: 109931312
         Relay_Master_Log_File: bin.000016
              Slave_IO_Running: Yes
             Slave_SQL_Running: Yes
               Replicate_Do_DB:
           Replicate_Ignore_DB:
            Replicate_Do_Table:
        Replicate_Ignore_Table:
       Replicate_Wild_Do_Table:
   Replicate_Wild_Ignore_Table:
                    Last_Errno: 0
                    Last_Error:
                  Skip_Counter: 0
           Exec_Master_Log_Pos: 902554004
               Relay_Log_Space: 97769824386
               Until_Condition: None
                Until_Log_File:
                 Until_Log_Pos: 0
            Master_SSL_Allowed: Yes
            Master_SSL_CA_File: xxx
            Master_SSL_CA_Path:
               Master_SSL_Cert: xxx
             Master_SSL_Cipher:
                Master_SSL_Key: xxx
         Seconds_Behind_Master: 2097439
 Master_SSL_Verify_Server_Cert: No
                 Last_IO_Errno: 0
                 Last_IO_Error:
                Last_SQL_Errno: 0
                Last_SQL_Error:
   Replicate_Ignore_Server_Ids:
              Master_Server_Id: 71
                Master_SSL_Crl: xxx
            Master_SSL_Crlpath:
                    Using_Gtid: No
                   Gtid_IO_Pos:
       Replicate_Do_Domain_Ids:
   Replicate_Ignore_Domain_Ids:
                 Parallel_Mode: conservative
                     SQL_Delay: 0
           SQL_Remaining_Delay: NULL
       Slave_SQL_Running_State: Update_rows_log_event::find_row(-1)
              Slave_DDL_Groups: 0
Slave_Non_Transactional_Groups: 0
    Slave_Transactional_Groups: 3
1 row in set (0.000 sec)
 
ERROR: No query specified
 
MariaDB [(none)]>

processlist

MariaDB [(none)]> show processlist;
+------+-------------+-----------+------+-----------+--------+-------------------------------------+------------------+----------+
| Id   | User        | Host      | db   | Command   | Time   | State                               | Info             | Progress |
+------+-------------+-----------+------+-----------+--------+-------------------------------------+------------------+----------+
|    3 | system user |           | NULL | Daemon    |   NULL | InnoDB purge worker                 | NULL             |    0.000 |
|    2 | system user |           | NULL | Daemon    |   NULL | InnoDB purge worker                 | NULL             |    0.000 |
|    1 | system user |           | NULL | Daemon    |   NULL | InnoDB purge coordinator            | NULL             |    0.000 |
|    4 | system user |           | NULL | Daemon    |   NULL | InnoDB purge worker                 | NULL             |    0.000 |
|    5 | system user |           | NULL | Daemon    |   NULL | InnoDB shutdown handler             | NULL             |    0.000 |
| 7970 | system user |           | NULL | Slave_IO  | 139005 | Waiting for master to send event    | NULL             |    0.000 |
| 7971 | system user |           | NULL | Slave_SQL |     20 | Update_rows_log_event::find_row(-1) | NULL             |    0.000 |
| 9446 | root        | localhost | NULL | Query     |      0 | Init                                | show processlist |    0.000 |
+------+-------------+-----------+------+-----------+--------+-------------------------------------+------------------+----------+
8 rows in set (0.000 sec)
 
MariaDB [(none)]>

show innodb status

MariaDB [(none)]> SHOW ENGINE INNODB STATUS \G;
*************************** 1. row ***************************
  Type: InnoDB
  Name:
Status:
=====================================
2020-11-26 23:32:52 0x7f3be80c8700 INNODB MONITOR OUTPUT
=====================================
Per second averages calculated from the last 9 seconds
-----------------
BACKGROUND THREAD
-----------------
srv_master_thread loops: 83765 srv_active, 0 srv_shutdown, 799851 srv_idle
srv_master_thread log flush and writes: 883590
----------
SEMAPHORES
----------
OS WAIT ARRAY INFO: reservation count 611926
OS WAIT ARRAY INFO: signal count 611777
RW-shared spins 0, rounds 1163439, OS waits 565726
RW-excl spins 0, rounds 1097141, OS waits 36520
RW-sx spins 8589, rounds 161143, OS waits 1702
Spin rounds per wait: 1163439.00 RW-shared, 1097141.00 RW-excl, 18.76 RW-sx
------------
TRANSACTIONS
------------
Trx id counter 2820359
Purge done for trx's n:o < 2820359 undo n:o < 0 state: running
History list length 10
LIST OF TRANSACTIONS FOR EACH SESSION:
---TRANSACTION 421370022671000, not started
0 lock struct(s), heap size 1136, 0 row lock(s)
---TRANSACTION 421370022666744, not started
0 lock struct(s), heap size 1136, 0 row lock(s)
---TRANSACTION 2820350, ACTIVE 139106 sec
mysql tables in use 1, locked 1
3291 lock struct(s), heap size 467152, 840999 row lock(s), undo log entries 840999
MySQL thread id 7971, OS thread handle 139895273547520, query id 15969 Update_rows_log_event::find_row(-1)
---TRANSACTION 421370022658232, not started
0 lock struct(s), heap size 1136, 0 row lock(s)
--------
FILE I/O
--------
I/O thread 0 state: waiting for completed aio requests (insert buffer thread)
I/O thread 1 state: waiting for completed aio requests (log thread)
I/O thread 2 state: waiting for completed aio requests (read thread)
I/O thread 3 state: waiting for completed aio requests (read thread)
I/O thread 4 state: waiting for completed aio requests (read thread)
I/O thread 5 state: waiting for completed aio requests (read thread)
I/O thread 6 state: waiting for completed aio requests (write thread)
I/O thread 7 state: waiting for completed aio requests (write thread)
I/O thread 8 state: waiting for completed aio requests (write thread)
I/O thread 9 state: waiting for completed aio requests (write thread)
Pending normal aio reads: [0, 0, 0, 0] , aio writes: [0, 0, 0, 0] ,
 ibuf aio reads:, log i/o's:, sync i/o's:
Pending flushes (fsync) log: 0; buffer pool: 0
440139 OS file reads, 4932468 OS file writes, 4895964 OS fsyncs
0.00 reads/s, 0 avg bytes/read, 7.78 writes/s, 7.78 fsyncs/s
-------------------------------------
INSERT BUFFER AND ADAPTIVE HASH INDEX
-------------------------------------
Ibuf: size 1, free list len 18704, seg size 18706, 0 merges
merged operations:
 insert 0, delete mark 0, delete 0
discarded operations:
 insert 0, delete mark 0, delete 0
Hash table size 4563583, node heap has 0 buffer(s)
Hash table size 4563583, node heap has 2 buffer(s)
Hash table size 4563583, node heap has 11876 buffer(s)
Hash table size 4563583, node heap has 3 buffer(s)
Hash table size 4563583, node heap has 3 buffer(s)
Hash table size 4563583, node heap has 131 buffer(s)
Hash table size 4563583, node heap has 1 buffer(s)
Hash table size 4563583, node heap has 0 buffer(s)
0.00 hash searches/s, 0.11 non-hash searches/s
---
LOG
---
Log sequence number 16427443796127
Log flushed up to   16427443795983
Pages flushed up to 16427443795983
Last checkpoint at  16427443795491
0 pending log flushes, 0 pending chkp writes
2125902 log i/o's done, 3.33 log i/o's/second
----------------------
BUFFER POOL AND MEMORY
----------------------
Total large memory allocated 17884512256
Dictionary memory allocated 659984
Buffer pool size   1057980
Free buffers       600442
Database pages     445494
Old database pages 164326
Modified db pages  2
Percent of dirty pages(LRU & free pages): 0.000
Max dirty pages percent: 75.000
Pending reads 0
Pending writes: LRU 0, flush list 0, single page 0
Pages made young 0, not young 0
0.00 youngs/s, 0.00 non-youngs/s
Pages read 439889, created 5605, written 1472093
0.00 reads/s, 0.00 creates/s, 2.22 writes/s
Buffer pool hit rate 1000 / 1000, young-making rate 0 / 1000 not 0 / 1000
Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s
LRU len: 445494, unzip_LRU len: 0
I/O sum[0]:cur[0], unzip sum[0]:cur[0]
----------------------
INDIVIDUAL BUFFER POOL INFO
----------------------
---BUFFER POOL 0
Buffer pool size   176330
Free buffers       99234
Database pages     75087
Old database pages 27697
Modified db pages  0
Percent of dirty pages(LRU & free pages): 0.000
Max dirty pages percent: 75.000
Pending reads 0
Pending writes: LRU 0, flush list 0, single page 0
Pages made young 0, not young 0
0.00 youngs/s, 0.00 non-youngs/s
Pages read 74057, created 1030, written 256353
0.00 reads/s, 0.00 creates/s, 0.00 writes/s
Buffer pool hit rate 1000 / 1000, young-making rate 0 / 1000 not 0 / 1000
Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s
LRU len: 75087, unzip_LRU len: 0
I/O sum[0]:cur[0], unzip sum[0]:cur[0]
---BUFFER POOL 1
Buffer pool size   176330
Free buffers       101822
Database pages     72500
Old database pages 26742
Modified db pages  1
Percent of dirty pages(LRU & free pages): 0.001
Max dirty pages percent: 75.000
Pending reads 0
Pending writes: LRU 0, flush list 0, single page 0
Pages made young 0, not young 0
0.00 youngs/s, 0.00 non-youngs/s
Pages read 71510, created 990, written 257831
0.00 reads/s, 0.00 creates/s, 1.11 writes/s
Buffer pool hit rate 1000 / 1000, young-making rate 0 / 1000 not 0 / 1000
Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s
LRU len: 72500, unzip_LRU len: 0
I/O sum[0]:cur[0], unzip sum[0]:cur[0]
---BUFFER POOL 2
Buffer pool size   176330
Free buffers       100523
Database pages     73801
Old database pages 27222
Modified db pages  1
Percent of dirty pages(LRU & free pages): 0.001
Max dirty pages percent: 75.000
Pending reads 0
Pending writes: LRU 0, flush list 0, single page 0
Pages made young 0, not young 0
0.00 youngs/s, 0.00 non-youngs/s
Pages read 72916, created 885, written 261691
0.00 reads/s, 0.00 creates/s, 1.11 writes/s
Buffer pool hit rate 1000 / 1000, young-making rate 0 / 1000 not 0 / 1000
Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s
LRU len: 73801, unzip_LRU len: 0
I/O sum[0]:cur[0], unzip sum[0]:cur[0]
---BUFFER POOL 3
Buffer pool size   176330
Free buffers       100491
Database pages     73833
Old database pages 27234
Modified db pages  0
Percent of dirty pages(LRU & free pages): 0.000
Max dirty pages percent: 75.000
Pending reads 0
Pending writes: LRU 0, flush list 0, single page 0
Pages made young 0, not young 0
0.00 youngs/s, 0.00 non-youngs/s
Pages read 72906, created 927, written 241950
0.00 reads/s, 0.00 creates/s, 0.00 writes/s
Buffer pool hit rate 1000 / 1000, young-making rate 0 / 1000 not 0 / 1000
Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s
LRU len: 73833, unzip_LRU len: 0
I/O sum[0]:cur[0], unzip sum[0]:cur[0]
---BUFFER POOL 4
Buffer pool size   176330
Free buffers       99337
Database pages     74986
Old database pages 27660
Modified db pages  0
Percent of dirty pages(LRU & free pages): 0.000
Max dirty pages percent: 75.000
Pending reads 0
Pending writes: LRU 0, flush list 0, single page 0
Pages made young 0, not young 0
0.00 youngs/s, 0.00 non-youngs/s
Pages read 74062, created 924, written 228573
0.00 reads/s, 0.00 creates/s, 0.00 writes/s
Buffer pool hit rate 1000 / 1000, young-making rate 0 / 1000 not 0 / 1000
Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s
LRU len: 74986, unzip_LRU len: 0
I/O sum[0]:cur[0], unzip sum[0]:cur[0]
---BUFFER POOL 5
Buffer pool size   176330
Free buffers       99035
Database pages     75287
Old database pages 27771
Modified db pages  0
Percent of dirty pages(LRU & free pages): 0.000
Max dirty pages percent: 75.000
Pending reads 0
Pending writes: LRU 0, flush list 0, single page 0
Pages made young 0, not young 0
0.00 youngs/s, 0.00 non-youngs/s
Pages read 74438, created 849, written 225695
0.00 reads/s, 0.00 creates/s, 0.00 writes/s
Buffer pool hit rate 1000 / 1000, young-making rate 0 / 1000 not 0 / 1000
Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s
LRU len: 75287, unzip_LRU len: 0
I/O sum[0]:cur[0], unzip sum[0]:cur[0]
--------------
ROW OPERATIONS
--------------
0 queries inside InnoDB, 0 queries in queue
0 read views open inside InnoDB
Process ID=28050, Main thread ID=139876649383680, state: sleeping
Number of rows inserted 0, updated 2799762, deleted 0, read 1623070235949
0.00 inserts/s, 3.11 updates/s, 0.00 deletes/s, 2624384.18 reads/s
Number of system rows inserted 4, updated 0, deleted 2, read 5
0.00 inserts/s, 0.00 updates/s, 0.00 deletes/s, 0.00 reads/s
----------------------------
END OF INNODB MONITOR OUTPUT
============================
 
1 row in set (0.000 sec)
 
ERROR: No query specified
 
MariaDB [(none)]>

Comment by Tomas Kucera [ 2020-11-27 ]

Additional info II

stuck within executing SQL on this table

| values | CREATE TABLE `values` (
  `id` decimal(10,0) NOT NULL,
  `node` varchar(9) NOT NULL,
  `bex` double NOT NULL,
  `computation_type` varchar(25) NOT NULL,
  `hour` int(10) unsigned NOT NULL,
  `type` varchar(25) DEFAULT NULL
) ENGINE=InnoDB DEFAULT CHARSET=utf8 |

global variables

Variable_name   Value
alter_algorithm DEFAULT
analyze_sample_percentage       100.000000
aria_block_size 8192
aria_checkpoint_interval        30
aria_checkpoint_log_activity    1048576
aria_encrypt_tables     OFF
aria_force_start_after_recovery_failures        0
aria_group_commit       none
aria_group_commit_interval      0
aria_log_file_size      1073741824
aria_log_purge_type     immediate
aria_max_sort_file_size 9223372036853727232
aria_page_checksum      ON
aria_pagecache_age_threshold    300
aria_pagecache_buffer_size      134217728
aria_pagecache_division_limit   100
aria_pagecache_file_hash_size   512
aria_recover_options    BACKUP,QUICK
aria_repair_threads     1
aria_sort_buffer_size   268434432
aria_stats_method       nulls_unequal
aria_sync_log_dir       NEWFILE
aria_used_for_temp_tables       ON
auto_increment_increment        1
auto_increment_offset   1
autocommit      ON
automatic_sp_privileges ON
back_log        250
basedir /usr/
big_tables      OFF
bind_address
binlog_annotate_row_events      ON
binlog_cache_size       32768
binlog_checksum CRC32
binlog_commit_wait_count        0
binlog_commit_wait_usec 100000
binlog_direct_non_transactional_updates OFF
binlog_file_cache_size  16384
binlog_format   MIXED
binlog_optimize_thread_scheduling       ON
binlog_row_image        FULL
binlog_stmt_cache_size  32768
bulk_insert_buffer_size 8388608
character_set_client    latin1
character_set_connection        latin1
character_set_database  latin1
character_set_filesystem        binary
character_set_results   latin1
character_set_server    latin1
character_set_system    utf8
character_sets_dir      /usr/share/mysql/charsets/
check_constraint_checks ON
collation_connection    latin1_swedish_ci
collation_database      latin1_swedish_ci
collation_server        latin1_swedish_ci
column_compression_threshold    100
column_compression_zlib_level   6
column_compression_zlib_strategy        DEFAULT_STRATEGY
column_compression_zlib_wrap    OFF
completion_type NO_CHAIN
concurrent_insert       AUTO
connect_timeout 15
core_file       OFF
datadir /opt/mysqldb/data/
date_format     %Y-%m-%d
datetime_format %Y-%m-%d %H:%i:%s
deadlock_search_depth_long      15
deadlock_search_depth_short     4
deadlock_timeout_long   50000000
deadlock_timeout_short  10000
debug_no_thread_alarm   OFF
default_password_lifetime       0
default_regex_flags
default_storage_engine  InnoDB
default_tmp_storage_engine
default_week_format     0
delay_key_write ON
delayed_insert_limit    100
delayed_insert_timeout  300
delayed_queue_size      1000
disconnect_on_expired_password  OFF
div_precision_increment 4
encrypt_binlog  OFF
encrypt_tmp_disk_tables OFF
encrypt_tmp_files       OFF
enforce_storage_engine
eq_range_index_dive_limit       200
event_scheduler OFF
expensive_subquery_limit        100
expire_logs_days        20
explicit_defaults_for_timestamp OFF
extra_max_connections   1
extra_port      0
flush   OFF
flush_time      0
foreign_key_checks      ON
ft_boolean_syntax       + -><()~*:""&|
ft_max_word_len 84
ft_min_word_len 4
ft_query_expansion_limit        20
ft_stopword_file        (built-in)
general_log     OFF
general_log_file        p2fbadb01.log
group_concat_max_len    1048576
gtid_binlog_pos 0-72-4115
gtid_binlog_state       0-72-4115
gtid_cleanup_batch_size 64
gtid_current_pos        0-71-10767
gtid_domain_id  0
gtid_ignore_duplicates  OFF
gtid_pos_auto_engines
gtid_slave_pos  0-71-10767
gtid_strict_mode        OFF
have_compress   YES
have_crypt      YES
have_dynamic_loading    YES
have_geometry   YES
have_openssl    YES
have_profiling  YES
have_query_cache        YES
have_rtree_keys YES
have_ssl        YES
have_symlink    YES
histogram_size  254
histogram_type  DOUBLE_PREC_HB
host_cache_size 653
hostname        p2fbadb01
idle_readonly_transaction_timeout       0
idle_transaction_timeout        0
idle_write_transaction_timeout  0
ignore_builtin_innodb   OFF
ignore_db_dirs
in_predicate_conversion_threshold       1000
init_connect
init_file
init_slave
innodb_adaptive_flushing        ON
innodb_adaptive_flushing_lwm    10.000000
innodb_adaptive_hash_index      ON
innodb_adaptive_hash_index_parts        8
innodb_adaptive_max_sleep_delay 150000
innodb_autoextend_increment     64
innodb_autoinc_lock_mode        1
innodb_background_scrub_data_check_interval     3600
innodb_background_scrub_data_compressed OFF
innodb_background_scrub_data_interval   604800
innodb_background_scrub_data_uncompressed       OFF
innodb_buf_dump_status_frequency        0
innodb_buffer_pool_chunk_size   134217728
innodb_buffer_pool_dump_at_shutdown     ON
innodb_buffer_pool_dump_now     OFF
innodb_buffer_pool_dump_pct     25
innodb_buffer_pool_filename     ib_buffer_pool
innodb_buffer_pool_instances    6
innodb_buffer_pool_load_abort   OFF
innodb_buffer_pool_load_at_startup      ON
innodb_buffer_pool_load_now     OFF
innodb_buffer_pool_size 17716740096
innodb_change_buffer_max_size   25
innodb_change_buffering all
innodb_checksum_algorithm       crc32
innodb_checksums        ON
innodb_cmp_per_index_enabled    OFF
innodb_commit_concurrency       0
innodb_compression_algorithm    zlib
innodb_compression_default      OFF
innodb_compression_failure_threshold_pct        5
innodb_compression_level        6
innodb_compression_pad_pct_max  50
innodb_concurrency_tickets      5000
innodb_data_file_path   ibdata1:12M:autoextend
innodb_data_home_dir
innodb_deadlock_detect  ON
innodb_default_encryption_key_id        1
innodb_default_row_format       dynamic
innodb_defragment       OFF
innodb_defragment_fill_factor   0.900000
innodb_defragment_fill_factor_n_recs    20
innodb_defragment_frequency     40
innodb_defragment_n_pages       7
innodb_defragment_stats_accuracy        0
innodb_disable_sort_file_cache  OFF
innodb_disallow_writes  OFF
innodb_doublewrite      ON
innodb_encrypt_log      OFF
innodb_encrypt_tables   OFF
innodb_encrypt_temporary_tables OFF
innodb_encryption_rotate_key_age        1
innodb_encryption_rotation_iops 100
innodb_encryption_threads       0
innodb_fast_shutdown    0
innodb_fatal_semaphore_wait_threshold   600
innodb_file_format
innodb_file_per_table   ON
innodb_fill_factor      100
innodb_flush_log_at_timeout     1
innodb_flush_log_at_trx_commit  1
innodb_flush_method     fsync
innodb_flush_neighbors  1
innodb_flush_sync       ON
innodb_flushing_avg_loops       30
innodb_force_load_corrupted     OFF
innodb_force_primary_key        OFF
innodb_force_recovery   0
innodb_ft_aux_table
innodb_ft_cache_size    8000000
innodb_ft_enable_diag_print     OFF
innodb_ft_enable_stopword       ON
innodb_ft_max_token_size        84
innodb_ft_min_token_size        3
innodb_ft_num_word_optimize     2000
innodb_ft_result_cache_limit    2000000000
innodb_ft_server_stopword_table
innodb_ft_sort_pll_degree       2
innodb_ft_total_cache_size      640000000
innodb_ft_user_stopword_table
innodb_idle_flush_pct   100
innodb_immediate_scrub_data_uncompressed        OFF
innodb_io_capacity      200
innodb_io_capacity_max  2000
innodb_large_prefix
innodb_lock_schedule_algorithm  fcfs
innodb_lock_wait_timeout        10000
innodb_locks_unsafe_for_binlog  OFF
innodb_log_buffer_size  83886080
innodb_log_checksums    ON
innodb_log_compressed_pages     ON
innodb_log_file_size    536870912
innodb_log_files_in_group       2
innodb_log_group_home_dir       ./
innodb_log_optimize_ddl ON
innodb_log_write_ahead_size     8192
innodb_lru_scan_depth   1024
innodb_max_dirty_pages_pct      75.000000
innodb_max_dirty_pages_pct_lwm  0.000000
innodb_max_purge_lag    0
innodb_max_purge_lag_delay      0
innodb_max_undo_log_size        10485760
innodb_monitor_disable
innodb_monitor_enable
innodb_monitor_reset
innodb_monitor_reset_all
innodb_old_blocks_pct   37
innodb_old_blocks_time  1000
innodb_online_alter_log_max_size        134217728
innodb_open_files       4096
innodb_optimize_fulltext_only   OFF
innodb_page_cleaners    4
innodb_page_size        16384
innodb_prefix_index_cluster_optimization        OFF
innodb_print_all_deadlocks      OFF
innodb_purge_batch_size 300
innodb_purge_rseg_truncate_frequency    128
innodb_purge_threads    4
innodb_random_read_ahead        OFF
innodb_read_ahead_threshold     56
innodb_read_io_threads  4
innodb_read_only        OFF
innodb_replication_delay        0
innodb_rollback_on_timeout      OFF
innodb_rollback_segments        128
innodb_scrub_log        OFF
innodb_scrub_log_speed  256
innodb_sort_buffer_size 1048576
innodb_spin_wait_delay  4
innodb_stats_auto_recalc        ON
innodb_stats_include_delete_marked      OFF
innodb_stats_method     nulls_equal
innodb_stats_modified_counter   0
innodb_stats_on_metadata        OFF
innodb_stats_persistent ON
innodb_stats_persistent_sample_pages    20
innodb_stats_sample_pages       8
innodb_stats_traditional        ON
innodb_stats_transient_sample_pages     8
innodb_status_output    OFF
innodb_status_output_locks      OFF
innodb_strict_mode      ON
innodb_sync_array_size  1
innodb_sync_spin_loops  30
innodb_table_locks      ON
innodb_temp_data_file_path      ibtmp1:12M:autoextend
innodb_thread_concurrency       0
innodb_thread_sleep_delay       10000
innodb_tmpdir
innodb_undo_directory   ./
innodb_undo_log_truncate        OFF
innodb_undo_logs        128
innodb_undo_tablespaces 0
innodb_use_atomic_writes        ON
innodb_use_native_aio   ON
innodb_version  10.4.12
innodb_write_io_threads 4
interactive_timeout     28800
join_buffer_size        262144
join_buffer_space_limit 2097152
join_cache_level        2
keep_files_on_create    OFF
key_buffer_size 67108864
key_cache_age_threshold 300
key_cache_block_size    1024
key_cache_division_limit        100
key_cache_file_hash_size        512
key_cache_segments      0
large_files_support     ON
large_page_size 0
large_pages     OFF
lc_messages     en_US
lc_messages_dir
lc_time_names   en_US
license GPL
local_infile    ON
lock_wait_timeout       86400
locked_in_memory        OFF
log_bin ON
log_bin_basename        /opt/mysqldb/binlogs/bin
log_bin_compress        OFF
log_bin_compress_min_len        256
log_bin_index   /opt/mysqldb/binlogs/bin.index
log_bin_trust_function_creators OFF
log_disabled_statements sp
log_error       /opt/logs/mysql/mysqld.log
log_output      FILE
log_queries_not_using_indexes   OFF
log_slave_updates       OFF
log_slow_admin_statements       ON
log_slow_disabled_statements    sp
log_slow_filter admin,filesort,filesort_on_disk,filesort_priority_queue,full_join,full_scan,query_cache,query_cache_miss,tmp_table,tmp_table_on_disk
log_slow_rate_limit     1
log_slow_slave_statements       ON
log_slow_verbosity
log_tc_size     24576
log_warnings    2
long_query_time 10.000000
low_priority_updates    OFF
lower_case_file_system  OFF
lower_case_table_names  1
master_verify_checksum  OFF
max_allowed_packet      1073741824
max_binlog_cache_size   18446744073709547520
max_binlog_size 1073741824
max_binlog_stmt_cache_size      18446744073709547520
max_connect_errors      10000
max_connections 1000
max_delayed_threads     20
max_digest_length       1024
max_error_count 64
max_heap_table_size     67108864
max_insert_delayed_threads      20
max_join_size   18446744073709551615
max_length_for_sort_data        1024
max_long_data_size      1073741824
max_password_errors     4294967295
max_prepared_stmt_count 16382
max_recursive_iterations        4294967295
max_relay_log_size      1073741824
max_rowid_filter_size   131072
max_seeks_for_key       4294967295
max_session_mem_used    9223372036854775807
max_sort_length 1024
max_sp_recursion_depth  0
max_statement_time      0.000000
max_tmp_tables  32
max_user_connections    0
max_write_lock_count    4294967295
metadata_locks_cache_size       1024
metadata_locks_hash_instances   8
min_examined_row_limit  0
mrr_buffer_size 262144
multi_range_count       256
myisam_block_size       1024
myisam_data_pointer_size        6
myisam_max_sort_file_size       9223372036853727232
myisam_mmap_size        18446744073709551615
myisam_recover_options  BACKUP,FORCE
myisam_repair_threads   1
myisam_sort_buffer_size 134216704
myisam_stats_method     NULLS_UNEQUAL
myisam_use_mmap OFF
mysql56_temporal_format ON
net_buffer_length       16384
net_read_timeout        30
net_retry_count 10
net_write_timeout       60
old     OFF
old_alter_table DEFAULT
old_mode
old_passwords   OFF
open_files_limit        65535
optimizer_prune_level   1
optimizer_search_depth  62
optimizer_selectivity_sampling_limit    100
optimizer_switch        index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=on,condition_pushdown_from_having=on
optimizer_trace enabled=off
optimizer_trace_max_mem_size    1048576
optimizer_use_condition_selectivity     4
performance_schema      OFF
performance_schema_accounts_size        -1
performance_schema_digests_size -1
performance_schema_events_stages_history_long_size      -1
performance_schema_events_stages_history_size   -1
performance_schema_events_statements_history_long_size  -1
performance_schema_events_statements_history_size       -1
performance_schema_events_waits_history_long_size       -1
performance_schema_events_waits_history_size    -1
performance_schema_hosts_size   -1
performance_schema_max_cond_classes     90
performance_schema_max_cond_instances   -1
performance_schema_max_digest_length    1024
performance_schema_max_file_classes     50
performance_schema_max_file_handles     32768
performance_schema_max_file_instances   -1
performance_schema_max_mutex_classes    200
performance_schema_max_mutex_instances  -1
performance_schema_max_rwlock_classes   40
performance_schema_max_rwlock_instances -1
performance_schema_max_socket_classes   10
performance_schema_max_socket_instances -1
performance_schema_max_stage_classes    160
performance_schema_max_statement_classes        202
performance_schema_max_table_handles    -1
performance_schema_max_table_instances  -1
performance_schema_max_thread_classes   50
performance_schema_max_thread_instances -1
performance_schema_session_connect_attrs_size   -1
performance_schema_setup_actors_size    100
performance_schema_setup_objects_size   100
performance_schema_users_size   -1
pid_file        /opt/mysqldb/data/p2fbadb01.pid
plugin_dir      /usr/lib64/mysql/plugin/
plugin_maturity gamma
port    3306
preload_buffer_size     32768
profiling       OFF
profiling_history_size  15
progress_report_time    5
protocol_version        10
proxy_protocol_networks
query_alloc_block_size  16384
query_cache_limit       1048576
query_cache_min_res_unit        4096
query_cache_size        536870912
query_cache_strip_comments      OFF
query_cache_type        DEMAND
query_cache_wlock_invalidate    OFF
query_prealloc_size     24576
range_alloc_block_size  4096
read_binlog_speed_limit 0
read_buffer_size        131072
read_only       OFF
read_rnd_buffer_size    262144
relay_log
relay_log_basename
relay_log_index
relay_log_info_file     relay-log.info
relay_log_purge ON
relay_log_recovery      OFF
relay_log_space_limit   0
replicate_annotate_row_events   ON
replicate_do_db
replicate_do_table
replicate_events_marked_for_skip        REPLICATE
replicate_ignore_db
replicate_ignore_table
replicate_wild_do_table
replicate_wild_ignore_table
report_host
report_password
report_port     3306
report_user
rowid_merge_buff_size   8388608
rpl_semi_sync_master_enabled    OFF
rpl_semi_sync_master_timeout    10000
rpl_semi_sync_master_trace_level        32
rpl_semi_sync_master_wait_no_slave      ON
rpl_semi_sync_master_wait_point AFTER_COMMIT
rpl_semi_sync_slave_delay_master        OFF
rpl_semi_sync_slave_enabled     OFF
rpl_semi_sync_slave_kill_conn_timeout   5
rpl_semi_sync_slave_trace_level 32
secure_auth     ON
secure_file_priv
secure_timestamp        NO
server_id       72
session_track_schema    ON
session_track_state_change      OFF
session_track_system_variables  autocommit,character_set_client,character_set_connection,character_set_results,time_zone
session_track_transaction_info  OFF
skip_external_locking   ON
skip_name_resolve       ON
skip_networking OFF
skip_show_database      OFF
slave_compressed_protocol       OFF
slave_ddl_exec_mode     IDEMPOTENT
slave_domain_parallel_threads   0
slave_exec_mode STRICT
slave_load_tmpdir       /opt/mysqldb/temp
slave_max_allowed_packet        1073741824
slave_net_timeout       60
slave_parallel_max_queued       131072
slave_parallel_mode     conservative
slave_parallel_threads  0
slave_parallel_workers  0
slave_run_triggers_for_rbr      NO
slave_skip_errors       OFF
slave_sql_verify_checksum       ON
slave_transaction_retries       10
slave_transaction_retry_errors  1158,1159,1160,1161,1205,1213,1429,2013,12701
slave_transaction_retry_interval        0
slave_type_conversions
slow_launch_time        2
slow_query_log  OFF
slow_query_log_file     p2fbadb01-slow.log
socket  /opt/mysqldb/data/mysql.sock
sort_buffer_size        2097152
sql_auto_is_null        OFF
sql_big_selects ON
sql_buffer_result       OFF
sql_log_bin     ON
sql_log_off     OFF
sql_mode        STRICT_TRANS_TABLES,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION
sql_notes       ON
sql_quote_show_create   ON
sql_safe_updates        OFF
sql_select_limit        18446744073709551615
sql_slave_skip_counter  0
sql_warnings    OFF
ssl_ca  /etc/mysql/ca-crt.pem
ssl_capath
ssl_cert        /etc/mysql/server-crt.pem
ssl_cipher
ssl_crl
ssl_crlpath
ssl_key /etc/mysql/server-key.pem
standard_compliant_cte  ON
storage_engine  InnoDB
stored_program_cache    256
strict_password_validation      ON
sync_binlog     0
sync_frm        ON
sync_master_info        10000
sync_relay_log  10000
sync_relay_log_info     10000
system_time_zone        CET
system_versioning_alter_history ERROR
system_versioning_asof  DEFAULT
table_definition_cache  2048
table_open_cache        4096
table_open_cache_instances      8
tcp_keepalive_interval  0
tcp_keepalive_probes    0
tcp_keepalive_time      0
tcp_nodelay     ON
thread_cache_size       100
thread_concurrency      10
thread_handling one-thread-per-connection
thread_pool_idle_timeout        60
thread_pool_max_threads 65536
thread_pool_oversubscribe       3
thread_pool_prio_kickup_timer   1000
thread_pool_priority    auto
thread_pool_size        8
thread_pool_stall_limit 500
thread_stack    299008
time_format     %H:%i:%s
time_zone       +00:00
timed_mutexes   OFF
tls_version     TLSv1.1,TLSv1.2,TLSv1.3
tmp_disk_table_size     18446744073709551615
tmp_memory_table_size   1073741824
tmp_table_size  1073741824
tmpdir  /opt/mysqldb/temp
transaction_alloc_block_size    8192
transaction_prealloc_size       4096
tx_isolation    READ-COMMITTED
tx_read_only    OFF
unique_checks   ON
updatable_views_with_limit      YES
use_stat_tables PREFERABLY_FOR_QUERIES
userstat        OFF
version 10.4.12-MariaDB-log
version_comment MariaDB Server
version_compile_machine x86_64
version_compile_os      Linux
version_malloc_library  system
version_source_revision ba6bfc402c352372cc1a9ec20b5dc50b2204549f
version_ssl_library     OpenSSL 1.0.2k-fips  26 Jan 2017
wait_timeout    28800
wsrep_osu_method        TOI
wsrep_sr_store  table
wsrep_auto_increment_control    ON
wsrep_causal_reads      OFF
wsrep_certification_rules       strict
wsrep_certify_nonpk     ON
wsrep_cluster_address
wsrep_cluster_name      my_wsrep_cluster
wsrep_convert_lock_to_trx       OFF
wsrep_data_home_dir     /opt/mysqldb/data/
wsrep_dbug_option
wsrep_debug     NONE
wsrep_desync    OFF
wsrep_dirty_reads       OFF
wsrep_drupal_282555_workaround  OFF
wsrep_forced_binlog_format      NONE
wsrep_gtid_domain_id    0
wsrep_gtid_mode OFF
wsrep_ignore_apply_errors       7
wsrep_load_data_splitting       OFF
wsrep_log_conflicts     OFF
wsrep_max_ws_rows       0
wsrep_max_ws_size       2147483647
wsrep_mysql_replication_bundle  0
wsrep_node_address
wsrep_node_incoming_address     AUTO
wsrep_node_name p2fbadb01
wsrep_notify_cmd
wsrep_on        OFF
wsrep_patch_version     wsrep_26.22
wsrep_provider  none
wsrep_provider_options
wsrep_recover   OFF
wsrep_reject_queries    NONE
wsrep_replicate_myisam  OFF
wsrep_restart_slave     OFF
wsrep_retry_autocommit  1
wsrep_slave_fk_checks   ON
wsrep_slave_uk_checks   OFF
wsrep_slave_threads     1
wsrep_sst_auth
wsrep_sst_donor
wsrep_sst_donor_rejects_queries OFF
wsrep_sst_method        rsync
wsrep_sst_receive_address       AUTO
wsrep_start_position    00000000-0000-0000-0000-000000000000:-1
wsrep_sync_wait 0
wsrep_trx_fragment_size 0
wsrep_trx_fragment_unit bytes

global status

Variable_name   Value
Aborted_clients 2
Aborted_connects        0
Aborted_connects_preauth        0
Access_denied_errors    0
Acl_column_grants       0
Acl_database_grants     103
Acl_function_grants     0
Acl_procedure_grants    0
Acl_package_spec_grants 0
Acl_package_body_grants 0
Acl_proxy_users 0
Acl_role_grants 0
Acl_roles       0
Acl_table_grants        0
Acl_users       80
Aria_pagecache_blocks_not_flushed       0
Aria_pagecache_blocks_unused    15694
Aria_pagecache_blocks_used      13
Aria_pagecache_read_requests    16
Aria_pagecache_reads    16
Aria_pagecache_write_requests   0
Aria_pagecache_writes   0
Aria_transaction_log_syncs      0
Binlog_commits  0
Binlog_group_commits    0
Binlog_group_commit_trigger_count       0
Binlog_group_commit_trigger_lock_wait   0
Binlog_group_commit_trigger_timeout     0
Binlog_snapshot_file    bin.000079
Binlog_snapshot_position        336
Binlog_bytes_written    0
Binlog_cache_disk_use   0
Binlog_cache_use        0
Binlog_stmt_cache_disk_use      0
Binlog_stmt_cache_use   0
Busy_time       0.000000
Bytes_received  44794471959
Bytes_sent      26289467
Column_compressions     0
Column_decompressions   0
Com_admin_commands      0
Com_alter_db    0
Com_alter_db_upgrade    0
Com_alter_event 0
Com_alter_function      0
Com_alter_procedure     0
Com_alter_server        0
Com_alter_sequence      0
Com_alter_table 0
Com_alter_tablespace    0
Com_alter_user  0
Com_analyze     0
Com_assign_to_keycache  0
Com_backup      0
Com_backup_lock 0
Com_begin       3
Com_binlog      0
Com_call_procedure      0
Com_change_db   1
Com_change_master       0
Com_check       0
Com_checksum    0
Com_commit      0
Com_compound_sql        0
Com_create_db   0
Com_create_event        0
Com_create_function     0
Com_create_index        0
Com_create_package      0
Com_create_package_body 0
Com_create_procedure    0
Com_create_role 0
Com_create_sequence     0
Com_create_server       0
Com_create_table        0
Com_create_temporary_table      0
Com_create_trigger      0
Com_create_udf  0
Com_create_user 0
Com_create_view 0
Com_dealloc_sql 0
Com_delete      0
Com_delete_multi        0
Com_do  0
Com_drop_db     0
Com_drop_event  0
Com_drop_function       0
Com_drop_index  0
Com_drop_procedure      0
Com_drop_package        0
Com_drop_package_body   0
Com_drop_role   0
Com_drop_server 0
Com_drop_sequence       0
Com_drop_table  0
Com_drop_temporary_table        0
Com_drop_trigger        0
Com_drop_user   0
Com_drop_view   0
Com_empty_query 0
Com_execute_immediate   0
Com_execute_sql 0
Com_flush       0
Com_get_diagnostics     0
Com_grant       0
Com_grant_role  0
Com_ha_close    0
Com_ha_open     0
Com_ha_read     0
Com_help        0
Com_insert      0
Com_insert_select       0
Com_install_plugin      0
Com_kill        2
Com_load        0
Com_lock_tables 0
Com_multi       0
Com_optimize    0
Com_preload_keys        0
Com_prepare_sql 0
Com_purge       0
Com_purge_before_date   0
Com_release_savepoint   0
Com_rename_table        0
Com_rename_user 0
Com_repair      0
Com_replace     0
Com_replace_select      0
Com_reset       0
Com_resignal    0
Com_revoke      0
Com_revoke_all  0
Com_revoke_role 0
Com_rollback    0
Com_rollback_to_savepoint       0
Com_savepoint   0
Com_select      60
Com_set_option  6
Com_show_authors        0
Com_show_binlog_events  0
Com_show_binlogs        0
Com_show_charsets       1
Com_show_collations     1
Com_show_contributors   0
Com_show_create_db      0
Com_show_create_event   0
Com_show_create_func    0
Com_show_create_package 0
Com_show_create_package_body    0
Com_show_create_proc    0
Com_show_create_table   0
Com_show_create_trigger 0
Com_show_create_user    0
Com_show_databases      1
Com_show_engine_logs    0
Com_show_engine_mutex   0
Com_show_engine_status  2
Com_show_errors 0
Com_show_events 0
Com_show_explain        0
Com_show_fields 0
Com_show_function_status        0
Com_show_generic        0
Com_show_grants 0
Com_show_keys   0
Com_show_master_status  0
Com_show_open_tables    0
Com_show_package_status 0
Com_show_package_body_status    0
Com_show_plugins        0
Com_show_privileges     0
Com_show_procedure_status       0
Com_show_processlist    10
Com_show_profile        0
Com_show_profiles       0
Com_show_relaylog_events        0
Com_show_slave_hosts    0
Com_show_slave_status   9478
Com_show_status 1
Com_show_storage_engines        1
Com_show_table_status   0
Com_show_tables 0
Com_show_triggers       0
Com_show_variables      5
Com_show_warnings       0
Com_shutdown    0
Com_signal      0
Com_start_all_slaves    0
Com_start_slave 3
Com_stmt_close  0
Com_stmt_execute        0
Com_stmt_fetch  0
Com_stmt_prepare        0
Com_stmt_reprepare      0
Com_stmt_reset  0
Com_stmt_send_long_data 0
Com_stop_all_slaves     0
Com_stop_slave  2
Com_truncate    0
Com_uninstall_plugin    0
Com_unlock_tables       0
Com_update      0
Com_update_multi        0
Com_xa_commit   0
Com_xa_end      0
Com_xa_prepare  0
Com_xa_recover  0
Com_xa_rollback 0
Com_xa_start    0
Compression     OFF
Connection_errors_accept        0
Connection_errors_internal      0
Connection_errors_max_connections       0
Connection_errors_peer_address  0
Connection_errors_select        0
Connection_errors_tcpwrap       0
Connections     9503
Cpu_time        0.000000
Created_tmp_disk_tables 134
Created_tmp_files       5
Created_tmp_tables      878
Delayed_errors  0
Delayed_insert_threads  0
Delayed_writes  0
Delete_scan     0
Empty_queries   2
Executed_events 0
Executed_triggers       0
Feature_application_time_periods        0
Feature_check_constraint        1
Feature_custom_aggregate_functions      0
Feature_delay_key_write 0
Feature_dynamic_columns 0
Feature_fulltext        0
Feature_gis     0
Feature_invisible_columns       0
Feature_json    496
Feature_locale  0
Feature_subquery        0
Feature_system_versioning       0
Feature_timezone        1
Feature_trigger 0
Feature_window_functions        0
Feature_xml     0
Flush_commands  1
Handler_commit  22
Handler_delete  0
Handler_discover        52
Handler_external_lock   0
Handler_icp_attempts    0
Handler_icp_match       0
Handler_mrr_init        0
Handler_mrr_key_refills 0
Handler_mrr_rowid_refills       0
Handler_prepare 0
Handler_read_first      4
Handler_read_key        3222590
Handler_read_last       2
Handler_read_next       27311952
Handler_read_prev       0
Handler_read_retry      0
Handler_read_rnd        106
Handler_read_rnd_deleted        0
Handler_read_rnd_next   1635247104397
Handler_rollback        5
Handler_savepoint       0
Handler_savepoint_rollback      0
Handler_tmp_delete      0
Handler_tmp_update      433021
Handler_tmp_write       7708
Handler_update  2814153
Handler_write   4
Innodb_buffer_pool_dump_status
Innodb_buffer_pool_load_status  Buffer pool(s) load completed at 201116 16:49:49
Innodb_buffer_pool_resize_status
Innodb_buffer_pool_load_incomplete      OFF
Innodb_buffer_pool_pages_data   445527
Innodb_buffer_pool_bytes_data   7299514368
Innodb_buffer_pool_pages_dirty  2
Innodb_buffer_pool_bytes_dirty  32768
Innodb_buffer_pool_pages_flushed        1482901
Innodb_buffer_pool_pages_free   600410
Innodb_buffer_pool_pages_misc   12043
Innodb_buffer_pool_pages_total  1057980
Innodb_buffer_pool_read_ahead_rnd       0
Innodb_buffer_pool_read_ahead   109974
Innodb_buffer_pool_read_ahead_evicted   0
Innodb_buffer_pool_read_requests        1641715578454
Innodb_buffer_pool_reads        329916
Innodb_buffer_pool_wait_free    0
Innodb_buffer_pool_write_requests       7712099
Innodb_data_fsyncs      4927011
Innodb_data_pending_fsyncs      0
Innodb_data_pending_reads       0
Innodb_data_pending_writes      0
Innodb_data_read        7207225856
Innodb_data_reads       440139
Innodb_data_writes      4963685
Innodb_data_written     50803233280
Innodb_dblwr_pages_written      1482770
Innodb_dblwr_writes     1340850
Innodb_log_waits        0
Innodb_log_write_requests       1385861
Innodb_log_writes       1574793
Innodb_os_log_fsyncs    2140102
Innodb_os_log_pending_fsyncs    0
Innodb_os_log_pending_writes    0
Innodb_os_log_written   1925289472
Innodb_page_size        16384
Innodb_pages_created    5638
Innodb_pages_read       439889
Innodb_pages_written    1482901
Innodb_row_lock_current_waits   0
Innodb_row_lock_time    0
Innodb_row_lock_time_avg        0
Innodb_row_lock_time_max        0
Innodb_row_lock_waits   0
Innodb_rows_deleted     0
Innodb_rows_inserted    0
Innodb_rows_read        1635277209135
Innodb_rows_updated     2814153
Innodb_system_rows_deleted      2
Innodb_system_rows_inserted     4
Innodb_system_rows_read 5
Innodb_system_rows_updated      0
Innodb_num_open_files   242
Innodb_truncated_status_writes  0
Innodb_available_undo_logs      128
Innodb_undo_truncations 0
Innodb_page_compression_saved   0
Innodb_num_index_pages_written  0
Innodb_num_non_index_pages_written      0
Innodb_num_pages_page_compressed        0
Innodb_num_page_compressed_trim_op      0
Innodb_num_pages_page_decompressed      0
Innodb_num_pages_page_compression_error 0
Innodb_num_pages_encrypted      0
Innodb_num_pages_decrypted      0
Innodb_have_lz4 OFF
Innodb_have_lzo OFF
Innodb_have_lzma        ON
Innodb_have_bzip2       OFF
Innodb_have_snappy      OFF
Innodb_have_punch_hole  ON
Innodb_defragment_compression_failures  0
Innodb_defragment_failures      0
Innodb_defragment_count 0
Innodb_instant_alter_column     0
Innodb_onlineddl_rowlog_rows    0
Innodb_onlineddl_rowlog_pct_used        0
Innodb_onlineddl_pct_progress   0
Innodb_secondary_index_triggered_cluster_reads  27311924
Innodb_secondary_index_triggered_cluster_reads_avoided  0
Innodb_encryption_rotation_pages_read_from_cache        0
Innodb_encryption_rotation_pages_read_from_disk 0
Innodb_encryption_rotation_pages_modified       0
Innodb_encryption_rotation_pages_flushed        0
Innodb_encryption_rotation_estimated_iops       0
Innodb_encryption_key_rotation_list_length      0
Innodb_encryption_n_merge_blocks_encrypted      0
Innodb_encryption_n_merge_blocks_decrypted      0
Innodb_encryption_n_rowlog_blocks_encrypted     0
Innodb_encryption_n_rowlog_blocks_decrypted     0
Innodb_encryption_n_temp_blocks_encrypted       0
Innodb_encryption_n_temp_blocks_decrypted       0
Innodb_scrub_background_page_reorganizations    0
Innodb_scrub_background_page_splits     0
Innodb_scrub_background_page_split_failures_underflow   0
Innodb_scrub_background_page_split_failures_out_of_filespace    0
Innodb_scrub_background_page_split_failures_missing_index       0
Innodb_scrub_background_page_split_failures_unknown     0
Innodb_scrub_log        0
Innodb_encryption_num_key_requests      0
Key_blocks_not_flushed  0
Key_blocks_unused       53578
Key_blocks_used 0
Key_blocks_warm 0
Key_read_requests       0
Key_reads       0
Key_write_requests      0
Key_writes      0
Last_query_cost 0.000000
Master_gtid_wait_count  0
Master_gtid_wait_time   0
Master_gtid_wait_timeouts       0
Max_statement_time_exceeded     0
Max_used_connections    3
Memory_used     752076240
Memory_used_initial     743757776
Not_flushed_delayed_rows        0
Open_files      63
Open_streams    0
Open_table_definitions  348
Open_tables     326
Opened_files    1456
Opened_plugin_libraries 0
Opened_table_definitions        315
Opened_tables   332
Opened_views    473
Performance_schema_accounts_lost        0
Performance_schema_cond_classes_lost    0
Performance_schema_cond_instances_lost  0
Performance_schema_digest_lost  0
Performance_schema_file_classes_lost    0
Performance_schema_file_handles_lost    0
Performance_schema_file_instances_lost  0
Performance_schema_hosts_lost   0
Performance_schema_locker_lost  0
Performance_schema_mutex_classes_lost   0
Performance_schema_mutex_instances_lost 0
Performance_schema_rwlock_classes_lost  0
Performance_schema_rwlock_instances_lost        0
Performance_schema_session_connect_attrs_lost   0
Performance_schema_socket_classes_lost  0
Performance_schema_socket_instances_lost        0
Performance_schema_stage_classes_lost   0
Performance_schema_statement_classes_lost       0
Performance_schema_table_handles_lost   0
Performance_schema_table_instances_lost 0
Performance_schema_thread_classes_lost  0
Performance_schema_thread_instances_lost        0
Performance_schema_users_lost   0
Prepared_stmt_count     0
Qcache_free_blocks      1
Qcache_free_memory      536852776
Qcache_hits     0
Qcache_inserts  0
Qcache_lowmem_prunes    0
Qcache_not_cached       60
Qcache_queries_in_cache 0
Qcache_total_blocks     1
Queries 19066
Questions       19060
Rows_read       1269429975533
Rows_sent       3083
Rows_tmp_read   440835
Rpl_semi_sync_master_clients    0
Rpl_semi_sync_master_get_ack    0
Rpl_semi_sync_master_net_avg_wait_time  0
Rpl_semi_sync_master_net_wait_time      0
Rpl_semi_sync_master_net_waits  0
Rpl_semi_sync_master_no_times   0
Rpl_semi_sync_master_no_tx      0
Rpl_semi_sync_master_request_ack        0
Rpl_semi_sync_master_status     OFF
Rpl_semi_sync_master_timefunc_failures  0
Rpl_semi_sync_master_tx_avg_wait_time   0
Rpl_semi_sync_master_tx_wait_time       0
Rpl_semi_sync_master_tx_waits   0
Rpl_semi_sync_master_wait_pos_backtraverse      0
Rpl_semi_sync_master_wait_sessions      0
Rpl_semi_sync_master_yes_tx     0
Rpl_semi_sync_slave_send_ack    0
Rpl_semi_sync_slave_status      OFF
Rpl_status      AUTH_MASTER
Rpl_transactions_multi_engine   0
Select_full_join        0
Select_full_range_join  0
Select_range    0
Select_range_check      0
Select_scan     34
Slave_connections       0
Slave_heartbeat_period  30.000
Slave_open_temp_tables  0
Slave_received_heartbeats       20865
Slave_retried_transactions      0
Slave_running   ON
Slave_skipped_errors    0
Slaves_connected        0
Slaves_running  1
Slow_launch_threads     0
Slow_queries    4
Sort_merge_passes       0
Sort_priority_queue_sorts       0
Sort_range      0
Sort_rows       106
Sort_scan       8
Ssl_accept_renegotiates 0
Ssl_accepts     9486
Ssl_callback_cache_hits 0
Ssl_cipher      DHE-RSA-AES256-GCM-SHA384
Ssl_cipher_list ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA:ECDHE-ECDSA-AES256-SHA:DH-DSS-AES256-GCM-SHA384:DHE-DSS-AES256-GCM-SHA384:DH-RSA-AES256-GCM-SHA384:DHE-RSA-AES256-GCM-SHA384:DHE-RSA-AES256-SHA256:DHE-DSS-AES256-SHA256:DH-RSA-AES256-SHA256:DH-DSS-AES256-SHA256:DHE-RSA-AES256-SHA:DHE-DSS-AES256-SHA:DH-RSA-AES256-SHA:DH-DSS-AES256-SHA:DHE-RSA-CAMELLIA256-SHA:DHE-DSS-CAMELLIA256-SHA:DH-RSA-CAMELLIA256-SHA:DH-DSS-CAMELLIA256-SHA:ECDH-RSA-AES256-GCM-SHA384:ECDH-ECDSA-AES256-GCM-SHA384:ECDH-RSA-AES256-SHA384:ECDH-ECDSA-AES256-SHA384:ECDH-RSA-AES256-SHA:ECDH-ECDSA-AES256-SHA:AES256-GCM-SHA384:AES256-SHA256:AES256-SHA:CAMELLIA256-SHA:PSK-AES256-CBC-SHA:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES128-SHA:DH-DSS-AES128-GCM-SHA256:DHE-DSS-AES128-GCM-SHA256:DH-RSA-AES128-GCM-SHA256:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES128-SHA256:DHE-DSS-AES128-SHA256:DH-RSA-AES128-SHA256:DH-DSS-AES128-SHA256:DHE-RSA-AES128-SHA:DHE-DSS-AES128-SHA:DH-RSA-AES128-SHA:DH-DSS-AES128-SHA:DHE-RSA-SEED-SHA:DHE-DSS-SEED-SHA:DH-RSA-SEED-SHA:DH-DSS-SEED-SHA:DHE-RSA-CAMELLIA128-SHA:DHE-DSS-CAMELLIA128-SHA:DH-RSA-CAMELLIA128-SHA:DH-DSS-CAMELLIA128-SHA:ECDH-RSA-AES128-GCM-SHA256:ECDH-ECDSA-AES128-GCM-SHA256:ECDH-RSA-AES128-SHA256:ECDH-ECDSA-AES128-SHA256:ECDH-RSA-AES128-SHA:ECDH-ECDSA-AES128-SHA:AES128-GCM-SHA256:AES128-SHA256:AES128-SHA:SEED-SHA:CAMELLIA128-SHA:PSK-AES128-CBC-SHA:ECDHE-RSA-DES-CBC3-SHA:ECDHE-ECDSA-DES-CBC3-SHA:EDH-RSA-DES-CBC3-SHA:EDH-DSS-DES-CBC3-SHA:DH-RSA-DES-CBC3-SHA:DH-DSS-DES-CBC3-SHA:ECDH-RSA-DES-CBC3-SHA:ECDH-ECDSA-DES-CBC3-SHA:DES-CBC3-SHA:IDEA-CBC-SHA:PSK-3DES-EDE-CBC-SHA:KRB5-IDEA-CBC-SHA:KRB5-DES-CBC3-SHA:KRB5-IDEA-CBC-MD5:KRB5-DES-CBC3-MD5:ECDHE-RSA-RC4-SHA:ECDHE-ECDSA-RC4-SHA:ECDH-RSA-RC4-SHA:ECDH-ECDSA-RC4-SHA:RC4-SHA:RC4-MD5:PSK-RC4-SHA:KRB5-RC4-SHA:KRB5-RC4-MD5
Ssl_client_connects     0
Ssl_connect_renegotiates        0
Ssl_ctx_verify_depth    18446744073709551615
Ssl_ctx_verify_mode     5
Ssl_default_timeout     7200
Ssl_finished_accepts    9486
Ssl_finished_connects   0
Ssl_server_not_after    May 22 09:11:36 2032 GMT
Ssl_server_not_before   May 27 09:11:36 2012 GMT
Ssl_session_cache_hits  0
Ssl_session_cache_misses        0
Ssl_session_cache_mode  SERVER
Ssl_session_cache_overflows     0
Ssl_session_cache_size  128
Ssl_session_cache_timeouts      0
Ssl_sessions_reused     0
Ssl_used_session_cache_entries  0
Ssl_verify_depth        18446744073709551615
Ssl_verify_mode 5
Ssl_version     TLSv1.2
Subquery_cache_hit      0
Subquery_cache_miss     0
Syncs   12248
Table_locks_immediate   23
Table_locks_waited      0
Table_open_cache_active_instances       1
Table_open_cache_hits   3907
Table_open_cache_misses 805
Table_open_cache_overflows      0
Tc_log_max_pages_used   0
Tc_log_page_size        0
Tc_log_page_waits       0
Threadpool_idle_threads 0
Threadpool_threads      0
Threads_cached  0
Threads_connected       1
Threads_created 1065
Threads_running 8
Transactions_gtid_foreign_engine        0
Transactions_multi_engine       0
Update_scan     0
Uptime  893055
Uptime_since_flush_status       893055
wsrep   58502558202168
wsrep_applier_thread_count      0
wsrep_cluster_capabilities
wsrep_cluster_conf_id   18446744073709551615
wsrep_cluster_size      0
wsrep_cluster_state_uuid
wsrep_cluster_status    Disconnected
wsrep_connected OFF
wsrep_local_bf_aborts   0
wsrep_local_index       18446744073709551615
wsrep_provider_capabilities
wsrep_provider_name
wsrep_provider_vendor
wsrep_provider_version
wsrep_ready     OFF
wsrep_rollbacker_thread_count   0
wsrep_thread_count      0

Comment by Tomas Kucera [ 2020-11-27 ]

Additional info III

GNU Debugger output

gdb -ex "set pagination 0" -ex "thread apply all bt" --batch -p $(pidof mysqld)

[New LWP 25833]
[New LWP 25832]
[New LWP 28091]
[New LWP 28090]
[New LWP 28085]
[New LWP 28084]
[New LWP 28083]
[New LWP 28082]
[New LWP 28081]
[New LWP 28080]
[New LWP 28079]
[New LWP 28078]
[New LWP 28077]
[New LWP 28076]
[New LWP 28075]
[New LWP 28074]
[New LWP 28073]
[New LWP 28072]
[New LWP 28071]
[New LWP 28068]
[New LWP 28067]
[New LWP 28066]
[New LWP 28064]
[New LWP 28063]
[New LWP 28062]
[New LWP 28061]
[New LWP 28060]
[New LWP 28059]
[New LWP 28058]
[New LWP 28057]
[New LWP 28056]
[New LWP 28055]
[New LWP 28054]
[New LWP 28053]
[New LWP 28052]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
0x00007f3c0eec620d in poll () from /lib64/libc.so.6
 
Thread 36 (Thread 0x7f3bec07c700 (LWP 28052)):
#0  0x00007f3c10b31d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x000055b6f5bcc027 in timer_handler ()
#2  0x00007f3c10b2ddd5 in start_thread () from /lib64/libpthread.so.0
#3  0x00007f3c0eed0ead in clone () from /lib64/libc.so.6
 
Thread 35 (Thread 0x7f3bdb450700 (LWP 28053)):
#0  0x00007f3c10b31d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x000055b6f5aa6706 in my_service_thread_sleep ()
#2  0x000055b6f5a9e05e in ma_checkpoint_background ()
#3  0x00007f3c10b2ddd5 in start_thread () from /lib64/libpthread.so.0
#4  0x00007f3c0eed0ead in clone () from /lib64/libc.so.6
 
Thread 34 (Thread 0x7f377f57f700 (LWP 28054)):
#0  0x00007f3c106fe644 in __io_getevents_0_4 () from /lib64/libaio.so.1
#1  0x000055b6f5876bca in LinuxAIOHandler::collect() ()
#2  0x000055b6f58771f2 in LinuxAIOHandler::poll(fil_node_t**, void**, IORequest*) ()
#3  0x000055b6f587ed79 in os_aio_handler(unsigned long, fil_node_t**, void**, IORequest*) ()
#4  0x000055b6f59fd661 in fil_aio_wait(unsigned long) ()
#5  0x000055b6f590bdd8 in io_handler_thread ()
#6  0x00007f3c10b2ddd5 in start_thread () from /lib64/libpthread.so.0
#7  0x00007f3c0eed0ead in clone () from /lib64/libc.so.6
 
Thread 33 (Thread 0x7f377ed7e700 (LWP 28055)):
#0  0x00007f3c106fe644 in __io_getevents_0_4 () from /lib64/libaio.so.1
#1  0x000055b6f5876bca in LinuxAIOHandler::collect() ()
#2  0x000055b6f58771f2 in LinuxAIOHandler::poll(fil_node_t**, void**, IORequest*) ()
#3  0x000055b6f587ed79 in os_aio_handler(unsigned long, fil_node_t**, void**, IORequest*) ()
#4  0x000055b6f59fd661 in fil_aio_wait(unsigned long) ()
#5  0x000055b6f590bdd8 in io_handler_thread ()
#6  0x00007f3c10b2ddd5 in start_thread () from /lib64/libpthread.so.0
#7  0x00007f3c0eed0ead in clone () from /lib64/libc.so.6
 
Thread 32 (Thread 0x7f377e57d700 (LWP 28056)):
#0  0x00007f3c106fe644 in __io_getevents_0_4 () from /lib64/libaio.so.1
#1  0x000055b6f5876bca in LinuxAIOHandler::collect() ()
#2  0x000055b6f58771f2 in LinuxAIOHandler::poll(fil_node_t**, void**, IORequest*) ()
#3  0x000055b6f587ed79 in os_aio_handler(unsigned long, fil_node_t**, void**, IORequest*) ()
#4  0x000055b6f59fd661 in fil_aio_wait(unsigned long) ()
#5  0x000055b6f590bdd8 in io_handler_thread ()
#6  0x00007f3c10b2ddd5 in start_thread () from /lib64/libpthread.so.0
#7  0x00007f3c0eed0ead in clone () from /lib64/libc.so.6
 
Thread 31 (Thread 0x7f377dd7c700 (LWP 28057)):
#0  0x00007f3c106fe644 in __io_getevents_0_4 () from /lib64/libaio.so.1
#1  0x000055b6f5876bca in LinuxAIOHandler::collect() ()
#2  0x000055b6f58771f2 in LinuxAIOHandler::poll(fil_node_t**, void**, IORequest*) ()
#3  0x000055b6f587ed79 in os_aio_handler(unsigned long, fil_node_t**, void**, IORequest*) ()
#4  0x000055b6f59fd661 in fil_aio_wait(unsigned long) ()
#5  0x000055b6f590bdd8 in io_handler_thread ()
#6  0x00007f3c10b2ddd5 in start_thread () from /lib64/libpthread.so.0
#7  0x00007f3c0eed0ead in clone () from /lib64/libc.so.6
 
Thread 30 (Thread 0x7f377d57b700 (LWP 28058)):
#0  0x00007f3c106fe644 in __io_getevents_0_4 () from /lib64/libaio.so.1
#1  0x000055b6f5876bca in LinuxAIOHandler::collect() ()
#2  0x000055b6f58771f2 in LinuxAIOHandler::poll(fil_node_t**, void**, IORequest*) ()
#3  0x000055b6f587ed79 in os_aio_handler(unsigned long, fil_node_t**, void**, IORequest*) ()
#4  0x000055b6f59fd661 in fil_aio_wait(unsigned long) ()
#5  0x000055b6f590bdd8 in io_handler_thread ()
#6  0x00007f3c10b2ddd5 in start_thread () from /lib64/libpthread.so.0
#7  0x00007f3c0eed0ead in clone () from /lib64/libc.so.6
 
Thread 29 (Thread 0x7f377cd7a700 (LWP 28059)):
#0  0x00007f3c106fe644 in __io_getevents_0_4 () from /lib64/libaio.so.1
#1  0x000055b6f5876bca in LinuxAIOHandler::collect() ()
#2  0x000055b6f58771f2 in LinuxAIOHandler::poll(fil_node_t**, void**, IORequest*) ()
#3  0x000055b6f587ed79 in os_aio_handler(unsigned long, fil_node_t**, void**, IORequest*) ()
#4  0x000055b6f59fd661 in fil_aio_wait(unsigned long) ()
#5  0x000055b6f590bdd8 in io_handler_thread ()
#6  0x00007f3c10b2ddd5 in start_thread () from /lib64/libpthread.so.0
#7  0x00007f3c0eed0ead in clone () from /lib64/libc.so.6
 
Thread 28 (Thread 0x7f377c579700 (LWP 28060)):
#0  0x00007f3c106fe644 in __io_getevents_0_4 () from /lib64/libaio.so.1
#1  0x000055b6f5876bca in LinuxAIOHandler::collect() ()
#2  0x000055b6f58771f2 in LinuxAIOHandler::poll(fil_node_t**, void**, IORequest*) ()
#3  0x000055b6f587ed79 in os_aio_handler(unsigned long, fil_node_t**, void**, IORequest*) ()
#4  0x000055b6f59fd661 in fil_aio_wait(unsigned long) ()
#5  0x000055b6f590bdd8 in io_handler_thread ()
#6  0x00007f3c10b2ddd5 in start_thread () from /lib64/libpthread.so.0
#7  0x00007f3c0eed0ead in clone () from /lib64/libc.so.6
 
Thread 27 (Thread 0x7f377bd78700 (LWP 28061)):
#0  0x00007f3c106fe644 in __io_getevents_0_4 () from /lib64/libaio.so.1
#1  0x000055b6f5876bca in LinuxAIOHandler::collect() ()
#2  0x000055b6f58771f2 in LinuxAIOHandler::poll(fil_node_t**, void**, IORequest*) ()
#3  0x000055b6f587ed79 in os_aio_handler(unsigned long, fil_node_t**, void**, IORequest*) ()
#4  0x000055b6f59fd661 in fil_aio_wait(unsigned long) ()
#5  0x000055b6f590bdd8 in io_handler_thread ()
#6  0x00007f3c10b2ddd5 in start_thread () from /lib64/libpthread.so.0
#7  0x00007f3c0eed0ead in clone () from /lib64/libc.so.6
 
Thread 26 (Thread 0x7f377b577700 (LWP 28062)):
#0  0x00007f3c106fe644 in __io_getevents_0_4 () from /lib64/libaio.so.1
#1  0x000055b6f5876bca in LinuxAIOHandler::collect() ()
#2  0x000055b6f58771f2 in LinuxAIOHandler::poll(fil_node_t**, void**, IORequest*) ()
#3  0x000055b6f587ed79 in os_aio_handler(unsigned long, fil_node_t**, void**, IORequest*) ()
#4  0x000055b6f59fd661 in fil_aio_wait(unsigned long) ()
#5  0x000055b6f590bdd8 in io_handler_thread ()
#6  0x00007f3c10b2ddd5 in start_thread () from /lib64/libpthread.so.0
#7  0x00007f3c0eed0ead in clone () from /lib64/libc.so.6
 
Thread 25 (Thread 0x7f377ad76700 (LWP 28063)):
#0  0x00007f3c106fe644 in __io_getevents_0_4 () from /lib64/libaio.so.1
#1  0x000055b6f5876bca in LinuxAIOHandler::collect() ()
#2  0x000055b6f58771f2 in LinuxAIOHandler::poll(fil_node_t**, void**, IORequest*) ()
#3  0x000055b6f587ed79 in os_aio_handler(unsigned long, fil_node_t**, void**, IORequest*) ()
#4  0x000055b6f59fd661 in fil_aio_wait(unsigned long) ()
#5  0x000055b6f590bdd8 in io_handler_thread ()
#6  0x00007f3c10b2ddd5 in start_thread () from /lib64/libpthread.so.0
#7  0x00007f3c0eed0ead in clone () from /lib64/libc.so.6
 
Thread 24 (Thread 0x7f377a575700 (LWP 28064)):
#0  0x00007f3c10b31d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x000055b6f587fda3 in os_event::timed_wait(timespec const*) ()
#2  0x000055b6f587ffb1 in os_event::wait_time_low(unsigned long, long) ()
#3  0x000055b6f59a698d in buf_flush_page_cleaner_coordinator ()
#4  0x00007f3c10b2ddd5 in start_thread () from /lib64/libpthread.so.0
#5  0x00007f3c0eed0ead in clone () from /lib64/libc.so.6
 
Thread 23 (Thread 0x7f3771573700 (LWP 28066)):
#0  0x00007f3c10b31965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x000055b6f58802b1 in os_event_wait_low(os_event*, long) ()
#2  0x000055b6f59a5713 in buf_flush_page_cleaner_worker ()
#3  0x00007f3c10b2ddd5 in start_thread () from /lib64/libpthread.so.0
#4  0x00007f3c0eed0ead in clone () from /lib64/libc.so.6
 
Thread 22 (Thread 0x7f3779573700 (LWP 28067)):
#0  0x00007f3c10b31965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x000055b6f58802b1 in os_event_wait_low(os_event*, long) ()
#2  0x000055b6f59a5713 in buf_flush_page_cleaner_worker ()
#3  0x00007f3c10b2ddd5 in start_thread () from /lib64/libpthread.so.0
#4  0x00007f3c0eed0ead in clone () from /lib64/libc.so.6
 
Thread 21 (Thread 0x7f3778d72700 (LWP 28068)):
#0  0x00007f3c10b31965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x000055b6f58802b1 in os_event_wait_low(os_event*, long) ()
#2  0x000055b6f59a5713 in buf_flush_page_cleaner_worker ()
#3  0x00007f3c10b2ddd5 in start_thread () from /lib64/libpthread.so.0
#4  0x00007f3c0eed0ead in clone () from /lib64/libc.so.6
 
Thread 20 (Thread 0x7f3793fff700 (LWP 28071)):
#0  0x00007f3c10b31d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x000055b6f587fda3 in os_event::timed_wait(timespec const*) ()
#2  0x000055b6f587ffb1 in os_event::wait_time_low(unsigned long, long) ()
#3  0x000055b6f5855ccf in lock_wait_timeout_thread ()
#4  0x00007f3c10b2ddd5 in start_thread () from /lib64/libpthread.so.0
#5  0x00007f3c0eed0ead in clone () from /lib64/libc.so.6
 
Thread 19 (Thread 0x7f379896f700 (LWP 28072)):
#0  0x00007f3c10b31d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x000055b6f587fda3 in os_event::timed_wait(timespec const*) ()
#2  0x000055b6f587ffb1 in os_event::wait_time_low(unsigned long, long) ()
#3  0x000055b6f5908d5c in srv_error_monitor_thread ()
#4  0x00007f3c10b2ddd5 in start_thread () from /lib64/libpthread.so.0
#5  0x00007f3c0eed0ead in clone () from /lib64/libc.so.6
 
Thread 18 (Thread 0x7f37937fe700 (LWP 28073)):
#0  0x00007f3c10b31d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x000055b6f587fda3 in os_event::timed_wait(timespec const*) ()
#2  0x000055b6f587ffb1 in os_event::wait_time_low(unsigned long, long) ()
#3  0x000055b6f5908650 in srv_monitor_thread ()
#4  0x00007f3c10b2ddd5 in start_thread () from /lib64/libpthread.so.0
#5  0x00007f3c0eed0ead in clone () from /lib64/libc.so.6
 
Thread 17 (Thread 0x7f3792ffd700 (LWP 28074)):
#0  0x00007f3c10b31d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x000055b6f587fda3 in os_event::timed_wait(timespec const*) ()
#2  0x000055b6f587ffb1 in os_event::wait_time_low(unsigned long, long) ()
#3  0x000055b6f59ece27 in dict_stats_thread ()
#4  0x00007f3c10b2ddd5 in start_thread () from /lib64/libpthread.so.0
#5  0x00007f3c0eed0ead in clone () from /lib64/libc.so.6
 
Thread 16 (Thread 0x7f37927fc700 (LWP 28075)):
#0  0x00007f3c10b31d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x000055b6f587fda3 in os_event::timed_wait(timespec const*) ()
#2  0x000055b6f587ffb1 in os_event::wait_time_low(unsigned long, long) ()
#3  0x000055b6f5951d2e in ib_wqueue_timedwait(ib_wqueue_t*, unsigned long) ()
#4  0x000055b6f5a35c32 in fts_optimize_thread(void*) ()
#5  0x00007f3c10b2ddd5 in start_thread () from /lib64/libpthread.so.0
#6  0x00007f3c0eed0ead in clone () from /lib64/libc.so.6
 
Thread 15 (Thread 0x7f3791ffb700 (LWP 28076)):
#0  0x00007f3c10b34e3d in nanosleep () from /lib64/libpthread.so.0
#1  0x000055b6f58804f0 in os_thread_sleep(unsigned long) ()
#2  0x000055b6f5909789 in srv_master_thread ()
#3  0x00007f3c10b2ddd5 in start_thread () from /lib64/libpthread.so.0
#4  0x00007f3c0eed0ead in clone () from /lib64/libc.so.6
 
Thread 14 (Thread 0x7f37917fa700 (LWP 28077)):
#0  0x00007f3c10b31965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x000055b6f58802b1 in os_event_wait_low(os_event*, long) ()
#2  0x000055b6f590ae1a in srv_purge_coordinator_thread ()
#3  0x00007f3c10b2ddd5 in start_thread () from /lib64/libpthread.so.0
#4  0x00007f3c0eed0ead in clone () from /lib64/libc.so.6
 
Thread 13 (Thread 0x7f3790ff9700 (LWP 28078)):
#0  0x00007f3c10b31965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x000055b6f58802b1 in os_event_wait_low(os_event*, long) ()
#2  0x000055b6f59045db in srv_resume_thread(srv_slot_t*, long, bool, unsigned long) [clone .constprop.71] ()
#3  0x000055b6f590a137 in srv_worker_thread ()
#4  0x00007f3c10b2ddd5 in start_thread () from /lib64/libpthread.so.0
#5  0x00007f3c0eed0ead in clone () from /lib64/libc.so.6
 
Thread 12 (Thread 0x7f378bfff700 (LWP 28079)):
#0  0x00007f3c10b31965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x000055b6f58802b1 in os_event_wait_low(os_event*, long) ()
#2  0x000055b6f59045db in srv_resume_thread(srv_slot_t*, long, bool, unsigned long) [clone .constprop.71] ()
#3  0x000055b6f590a137 in srv_worker_thread ()
#4  0x00007f3c10b2ddd5 in start_thread () from /lib64/libpthread.so.0
#5  0x00007f3c0eed0ead in clone () from /lib64/libc.so.6
 
Thread 11 (Thread 0x7f378b7fe700 (LWP 28080)):
#0  0x00007f3c10b31965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x000055b6f58802b1 in os_event_wait_low(os_event*, long) ()
#2  0x000055b6f59045db in srv_resume_thread(srv_slot_t*, long, bool, unsigned long) [clone .constprop.71] ()
#3  0x000055b6f590a137 in srv_worker_thread ()
#4  0x00007f3c10b2ddd5 in start_thread () from /lib64/libpthread.so.0
#5  0x00007f3c0eed0ead in clone () from /lib64/libc.so.6
 
Thread 10 (Thread 0x7f378affd700 (LWP 28081)):
#0  0x00007f3c10b31965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x000055b6f58802b1 in os_event_wait_low(os_event*, long) ()
#2  0x000055b6f59a0ef7 in buf_dump_thread ()
#3  0x00007f3c10b2ddd5 in start_thread () from /lib64/libpthread.so.0
#4  0x00007f3c0eed0ead in clone () from /lib64/libc.so.6
 
Thread 9 (Thread 0x7f378a7fc700 (LWP 28082)):
#0  0x00007f3c10b34e3d in nanosleep () from /lib64/libpthread.so.0
#1  0x000055b6f58804f0 in os_thread_sleep(unsigned long) ()
#2  0x000055b6f598978a in btr_defragment_thread ()
#3  0x00007f3c10b2ddd5 in start_thread () from /lib64/libpthread.so.0
#4  0x00007f3c0eed0ead in clone () from /lib64/libc.so.6
 
Thread 8 (Thread 0x7f3789ffb700 (LWP 28083)):
#0  0x00007f3c10b31965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x000055b6f58802b1 in os_event_wait_low(os_event*, long) ()
#2  0x000055b6f5999112 in buf_resize_thread ()
#3  0x00007f3c10b2ddd5 in start_thread () from /lib64/libpthread.so.0
#4  0x00007f3c0eed0ead in clone () from /lib64/libc.so.6
 
Thread 7 (Thread 0x7f37897fa700 (LWP 28084)):
#0  0x00007f3c10b31965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x000055b6f57fc363 in thd_destructor_proxy ()
#2  0x00007f3c10b2ddd5 in start_thread () from /lib64/libpthread.so.0
#3  0x00007f3c0eed0ead in clone () from /lib64/libc.so.6
 
Thread 6 (Thread 0x7f3c10dae700 (LWP 28085)):
#0  0x00007f3c10b31965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x000055b6f574744f in binlog_background_thread ()
#2  0x00007f3c10b2ddd5 in start_thread () from /lib64/libpthread.so.0
#3  0x00007f3c0eed0ead in clone () from /lib64/libc.so.6
 
Thread 5 (Thread 0x7f3be823f700 (LWP 28090)):
#0  0x00007f3c0ee0a40a in sigwaitinfo () from /lib64/libc.so.6
#1  0x000055b6f5380be3 in signal_hand ()
#2  0x00007f3c10b2ddd5 in start_thread () from /lib64/libpthread.so.0
#3  0x00007f3c0eed0ead in clone () from /lib64/libc.so.6
 
Thread 4 (Thread 0x7f3be81f4700 (LWP 28091)):
#0  0x00007f3c10b31965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x000055b6f539d9ab in handle_slave_background ()
#2  0x00007f3c10b2ddd5 in start_thread () from /lib64/libpthread.so.0
#3  0x00007f3c0eed0ead in clone () from /lib64/libc.so.6
 
Thread 3 (Thread 0x7f3be8113700 (LWP 25832)):
#0  0x00007f3c0eec620d in poll () from /lib64/libc.so.6
#1  0x000055b6f5c2725e in vio_io_wait ()
#2  0x000055b6f5c27358 in vio_socket_io_wait ()
#3  0x000055b6f5c27d6a in vio_ssl_read ()
#4  0x000055b6f576e6db in my_real_read(st_net*, unsigned long*, char) ()
#5  0x000055b6f576f548 in my_net_read_packet_reallen ()
#6  0x000055b6f562f737 in cli_safe_read_reallen ()
#7  0x000055b6f53aa440 in handle_slave_io ()
#8  0x00007f3c10b2ddd5 in start_thread () from /lib64/libpthread.so.0
#9  0x00007f3c0eed0ead in clone () from /lib64/libc.so.6
 
Thread 2 (Thread 0x7f3be815e700 (LWP 25833)):
#0  0x000055b6f5993180 in buf_page_optimistic_get(unsigned long, buf_block_t*, unsigned long, char const*, unsigned int, mtr_t*) ()
#1  0x000055b6f5969d35 in btr_cur_optimistic_latch_leaves(buf_block_t*, unsigned long, unsigned long*, btr_cur_t*, char const*, unsigned int, mtr_t*) ()
#2  0x000055b6f5979633 in btr_pcur_restore_position_func(unsigned long, btr_pcur_t*, char const*, unsigned int, mtr_t*) ()
#3  0x000055b6f534db38 in sel_restore_position_for_mysql(unsigned long*, unsigned long, btr_pcur_t*, unsigned long, mtr_t*) [clone .constprop.81] ()
#4  0x000055b6f58ed1a7 in row_search_mvcc(unsigned char*, page_cur_mode_t, row_prebuilt_t*, unsigned long, unsigned long) ()
#5  0x000055b6f580ad80 in ha_innobase::general_fetch(unsigned char*, unsigned int, unsigned int) ()
#6  0x000055b6f566088f in handler::ha_rnd_next(unsigned char*) ()
#7  0x000055b6f57655be in Rows_log_event::find_row(rpl_group_info*) ()
#8  0x000055b6f5765a54 in Update_rows_log_event::do_exec_row(rpl_group_info*) ()
#9  0x000055b6f5757cb5 in Rows_log_event::do_apply_event(rpl_group_info*) ()
#10 0x000055b6f539c560 in apply_event_and_update_pos_apply(Log_event*, THD*, rpl_group_info*, int) ()
#11 0x000055b6f53a6152 in handle_slave_sql ()
#12 0x00007f3c10b2ddd5 in start_thread () from /lib64/libpthread.so.0
#13 0x00007f3c0eed0ead in clone () from /lib64/libc.so.6
 
Thread 1 (Thread 0x7f3c10f4a8c0 (LWP 28050)):
#0  0x00007f3c0eec620d in poll () from /lib64/libc.so.6
#1  0x000055b6f5389d9f in handle_connections_sockets() ()
#2  0x000055b6f538b024 in mysqld_main(int, char**) ()
#3  0x00007f3c0edf53d5 in __libc_start_main () from /lib64/libc.so.6
#4  0x000055b6f537e3f4 in _start ()
[Inferior 1 (process 28050) detached]

Comment by Alice Sherepa [ 2020-12-01 ]


Is it possible that that table was corrupted? Maybe you could send us a dump of this table (from the slave - or maybe from the backup of the slave) - then I will try to repeat the issue on my side

Comment by Tomas Kucera [ 2020-12-02 ]

Dear Alice,
Thank you for the reaction and your time. Unfortunately the slave DB was re-imported by the export from master DB due to the replication repair.
Before the occured issue and before the stopping of the DB replication the data was being inserted to the discussed table properly so I suppose that the tables was working ok. I did not create the backup of the slave database and I did not try to perform the mysqlcheck command. If the mysqlcheck command would be finished ok, I suppose that the table corruption reason can be excluded?
There is planned the maintenance on the master environment soon so I will stop the slave DB for several hours again. I will inform if everything is ok after the replication re-start or if the issue is repeated.
Did you see any issue in the provided information relating the DB slave state?
Best regards
Tomas

Comment by Tomas Kucera [ 2020-12-04 ]

Dear Alice,
The small note. There was a maintenance yesterday and the DB replication was stopped for several hours and then started and everything is ok now and the DB replication works properly. It means that the issue was not reproduced by new stop and start of the replication.
Best regards
Tomas

Comment by Alice Sherepa [ 2020-12-04 ]

Dear Tomas,
Excellent news!
I tried to come up with theories of what might have been wrong and glad that replication works now.
Also with rbr it is important to have PK/unique indexes on every table, otherwise, it could be very slow.
Best regards,
Alice

Comment by Tomas Kucera [ 2020-12-10 ]

Dear Alice,
We will see if during the future structure changes of the DB tables on the master side, everthing will stay ok. The changes are planned after 3 months.
Regarding the PK/unique indexes so the MariaDB service recommends in the log file to add the PK/unique indexes for two tables due to the performance improving. One of this table is the table where the replication process was stucked.
I will inform here if the issue is appeared after the PK/unique indexes creation and after the scheduled master DB structure changes.
Best regards
Tomas

Generated at Thu Feb 08 09:28:54 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.