[MDEV-30382] Replication CPU usage spikes Created: 2023-01-11  Updated: 2023-02-27

Status: Open
Project: MariaDB Server
Component/s: Replication, Server, Storage Engine - InnoDB
Affects Version/s: 10.10.0
Fix Version/s: None

Type: Bug Priority: Minor
Reporter: Denis Chernyaev Assignee: Unassigned
Resolution: Unresolved Votes: 0
Labels: None
Environment:

Master-master replication between Windows Server 2019 Standard and CentOS 8. CentOS instance is currently handling all the connections and Windows instance only replicates.



 Description   

Since we have updated to 10.10, there is quite a strange bug when the replication is active. When it occurs, the CentOS instance starts getting a CPU usage spike roughly every 60 seconds, and these spikes grow over time, and the instance on Windows starts using consistent 8% CPU (on a 6-core CPU with HT).

Over time, the replication on Windows instance falls farther and farther behind, reaching nearly a day long gap in "Seconds behind master". CentOS instance consumes more and more CPU.

During this time, the processlist on Windows instance consistently shows the same queries running slow:

UPDATE a SET am = NOW(), an = NOW(), bc = 1, bd = 'string' WHERE a IN(<up to 100 integer values>)

*Once I `REPAIR` this table, the bug goes away but reappears again after some time.
Here is this table's structure:*

CREATE TABLE `a` (
  `a` mediumint(8) unsigned NOT NULL AUTO_INCREMENT,
  `b` varchar(255) CHARACTER SET cp1251 COLLATE cp1251_general_cs DEFAULT NULL,
  `c` varchar(128) DEFAULT NULL,
  `d` varchar(255) DEFAULT NULL,
  `e` varchar(128) DEFAULT NULL,
  `f` varchar(128) DEFAULT NULL,
  `g` varchar(255) DEFAULT NULL,
  `h` smallint(1) unsigned NOT NULL DEFAULT 0,
  `i` date DEFAULT NULL,
  `j` tinyint(4) unsigned DEFAULT NULL,
  `k` tinyint(4) unsigned DEFAULT NULL,
  `l` tinyint(4) unsigned DEFAULT NULL,
  `m` tinyint(4) unsigned DEFAULT NULL,
  `n` tinyint(4) unsigned DEFAULT NULL,
  `o` tinyint(4) unsigned DEFAULT NULL,
  `p` tinyint(4) unsigned DEFAULT NULL,
  `q` tinyint(4) unsigned DEFAULT NULL,
  `r` tinyint(4) unsigned DEFAULT NULL,
  `s` tinyint(4) unsigned DEFAULT NULL,
  `t` tinyint(4) unsigned DEFAULT NULL,
  `u` tinyint(4) unsigned DEFAULT NULL,
  `v` tinyint(4) unsigned DEFAULT NULL,
  `w` tinyint(4) unsigned DEFAULT NULL,
  `x` tinyint(4) unsigned DEFAULT NULL,
  `y` timestamp NULL DEFAULT NULL,
  `z` varchar(50) DEFAULT NULL,
  `aa` tinyint(1) NOT NULL DEFAULT 0,
  `ab` timestamp NULL DEFAULT '0000-00-00 00:00:00',
  `ac` timestamp NULL DEFAULT '0000-00-00 00:00:00',
  `ad` varchar(20) DEFAULT NULL,
  `ae` varchar(11) DEFAULT NULL,
  `af` int(11) DEFAULT 0,
  `ag` tinyint(3) unsigned DEFAULT 0,
  `ah` char(1) NOT NULL DEFAULT 'A',
  `ai` tinyint(1) NOT NULL DEFAULT 0,
  `aj` tinyint(1) DEFAULT 0,
  `ak` tinyint(1) unsigned NOT NULL DEFAULT 0,
  `al` tinyint(1) NOT NULL DEFAULT 0,
  `am` timestamp NULL DEFAULT NULL,
  `an` timestamp NULL DEFAULT NULL,
  `ao` varchar(255) DEFAULT NULL,
  `ap` varchar(255) DEFAULT NULL,
  `aq` varchar(255) DEFAULT NULL,
  `ar` varchar(255) DEFAULT NULL,
  `as` tinyint(1) DEFAULT 0,
  `at` tinyint(1) DEFAULT 0,
  `au` tinyint(1) DEFAULT 1,
  `av` varchar(100) DEFAULT 'undefined',
  `aw` tinyint(1) unsigned DEFAULT 1,
  `ax` tinyint(1) unsigned DEFAULT 0,
  `ay` tinyint(1) unsigned DEFAULT 0,
  `az` tinyint(1) unsigned DEFAULT 0,
  `ba` timestamp NULL DEFAULT NULL,
  `bb` timestamp NULL DEFAULT NULL,
  `bc` tinyint(3) unsigned DEFAULT 0,
  `bd` varchar(255) DEFAULT NULL,
  `be` tinyint(3) unsigned DEFAULT 0,
  `bf` varchar(255) NOT NULL DEFAULT '',
  `bg` varchar(32) DEFAULT NULL,
  `bh` timestamp NULL DEFAULT NULL,
  `bi` tinyint(4) unsigned DEFAULT 0,
  `bj` tinyint(4) unsigned DEFAULT 0,
  `bk` tinyint(3) unsigned DEFAULT 0,
  `bl` tinyint(3) unsigned DEFAULT 0,
  `bm` tinyint(3) unsigned DEFAULT 0,
  `bn` tinyint(3) unsigned DEFAULT 0,
  `bo` tinyint(3) unsigned DEFAULT 0,
  `bp` text CHARACTER SET cp1251 COLLATE cp1251_general_cs DEFAULT NULL,
  PRIMARY KEY (`a`,`aa`),
  KEY `idx_ae` (`ae`) KEY_BLOCK_SIZE=8192,
  KEY `idx_ab` (`ab`) KEY_BLOCK_SIZE=8192,
  KEY `av` (`av`) KEY_BLOCK_SIZE=8192,
  KEY `am` (`am`) KEY_BLOCK_SIZE=8192,
  KEY `an` (`an`) KEY_BLOCK_SIZE=8192,
  KEY `ac` (`ac`),
  KEY `bn` (`bn`),
  KEY `bo` (`bo`)
) ENGINE=InnoDB AUTO_INCREMENT=43300 DEFAULT CHARSET=cp1251 COLLATE=cp1251_general_ci ROW_FORMAT=DYNAMIC `page_compressed`=0
 PARTITION BY LIST (`aa`)
(PARTITION `a0` VALUES IN (0) ENGINE = InnoDB,
 PARTITION `a1` VALUES IN (1) ENGINE = InnoDB)



 Comments   
Comment by Denis Chernyaev [ 2023-02-27 ]

The issue no longer takes place after upgrading to 10.11.2

Generated at Thu Feb 08 10:15:49 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.