Server hang caused by InnoDB change buffer

The problem was found on

   origin/10.6-MDEV-33508 208f3ee34381f7c8a79d114f610c42bb53a5f394 2024-02-21T11:35:06+02:00

Per Marko the official 10.6 would suffer from that too.

Scenario:

1. Start the server and generate some initial data.

2. 33 "worker" run concurrent some DML mix

3. Some RQG component checks periodic the processlist.

pluto:/data/results/1708610110/HANG1/rqg.log

Line 1322 (2024-02-22T16:08:28)  9 workers active, the other have disconnected

       They all run some INSERT IGNORE INTO ... (one row only).

Line 1797 (2024-02-22T16:12:39) 9 workers active

       They all run the same statement like above.

       Only the time was going up by ~ 250s.

State  somewhere between 16:12:39 and 16:16 

pluto:/data/results/1708610110/HANG1$ gdb -c ./1/data/gcore.2353714 /data/Server_bin/10.6-MDEV-33508_debug_Og/bin/mariadbd

State  somewhere between 16:16 and 16:19

pluto:/data/results/1708610110/HANG1$ gdb -c ./1/data/core.2353714 /data/Server_bin/10.6-MDEV-33508_debug_Og/bin/mariadbd

RQG

===

# git clone https://github.com/mleich1/rqg --branch <pick the right branch> RQG

#

# GIT_SHOW: HEAD -> master, origin/master, origin/HEAD 24efdf20d844da7e26bf02b494515a503ef8e474 2024-01-18T19:39:19+01:00

# rqg.pl  : Version 4.4.0 (2023-08)

#

# $RQG_HOME/rqg.pl \

# --gendata=conf/mariadb/oltp.zz \

# --max_gd_duration=900 \

# --grammar=conf/mariadb/oltp.yy \

# --mysqld=--transaction-isolation=REPEATABLE-READ \

# --validator=SelectStability \

# --mysqld=--loose-innodb_lock_schedule_algorithm=fcfs \

# --mysqld=--loose-idle_write_transaction_timeout=0 \

# --mysqld=--loose-idle_transaction_timeout=0 \

# --mysqld=--loose-idle_readonly_transaction_timeout=0 \

# --mysqld=--connect_timeout=60 \

# --mysqld=--interactive_timeout=28800 \

# --mysqld=--slave_net_timeout=60 \

# --mysqld=--net_read_timeout=30 \

# --mysqld=--net_write_timeout=60 \

# --mysqld=--loose-table_lock_wait_timeout=50 \

# --mysqld=--wait_timeout=28800 \

# --no_mask \

# --queries=10000000 \

# --seed=random \

# --reporters=None \

# --reporters=ErrorLog \

# --reporters=Deadlock \

# --validators=None \

# --mysqld=--log_output=none \

# --mysqld=--log_bin_trust_function_creators=1 \

# --mysqld=--loose-debug_assert_on_not_freed_memory=0 \

# --engine=InnoDB \

# --restart_timeout=240 \

# --mysqld=--plugin-load-add=file_key_management.so \

# --mysqld=--loose-file-key-management-filename=$RQG_HOME/conf/mariadb/encryption_keys.txt \

# --mysqld=--plugin-load-add=provider_lzo.so \

# --mysqld=--plugin-load-add=provider_bzip2.so \

# --mysqld=--plugin-load-add=provider_lzma.so \

# --mysqld=--plugin-load-add=provider_snappy.so \

# --mysqld=--plugin-load-add=provider_lz4.so \

# --mysqld=--loose-innodb_compression_level=1 \

# --duration=300 \

# --mysqld=--loose-innodb_fatal_semaphore_wait_threshold=300 \

# --mysqld=--lock-wait-timeout=86400 \

# --mysqld=--innodb-lock-wait-timeout=50 \    

# --mysqld=--sql_mode=STRICT_TRANS_TABLES,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION \

# --mysqld=--loose-innodb_read_only_compressed=OFF \

# --mysqld=--innodb_stats_persistent=off \

# --mysqld=--innodb_adaptive_hash_index=off \

# --mysqld=--innodb_random_read_ahead=OFF \

# --mysqld=--log-bin \

# --mysqld=--sync-binlog=1 \

# --mysqld=--loose-innodb_evict_tables_on_commit_debug=on \

# --threads=33 \

# --mysqld=--innodb_use_native_aio=1 \

# --mysqld=--innodb_undo_log_truncate=OFF \

# --mysqld=--loose_innodb_change_buffering=all \     # <============

# --mysqld=--innodb_rollback_on_timeout=OFF \

# --mysqld=--innodb_page_size=16K \

# --mysqld=--innodb-buffer-pool-size=10M \

# --vardir_type=slow \

# <local settings>