So after recloning the host and after waiting for all the 8 replication threads to be totally synced with the master, and the host showing 0 issues, I decided to restart it with
innodb_purge_threads = 10
|
and the host started to show errors and crashed
Apr 28 05:41:34 labsdb1011 mysqld[31450]: 0: len 43; hex 50686f746f6772617068735f6f665f686162697461745f6f665f696e73656374655f62795f477
|
Apr 28 05:41:34 labsdb1011 mysqld[31450]: 1: len 4; hex 5ea7c180; asc ^ ;;
|
Apr 28 05:41:34 labsdb1011 mysqld[31450]: 2: len 4; hex 0312d9e9; asc ;;
|
Apr 28 05:41:34 labsdb1011 mysqld[31450]: InnoDB: record PHYSICAL RECORD: n_fields 3; compact format; info bits 0
|
Apr 28 05:41:34 labsdb1011 mysqld[31450]: 0: len 30; hex 50686f746f6772617068735f6f665f686162697461745f6f665f6661756e; asc Photographs
|
Apr 28 05:41:34 labsdb1011 mysqld[31450]: 1: len 4; hex 5ea7c1b6; asc ^ ;;
|
Apr 28 05:41:34 labsdb1011 mysqld[31450]: 2: len 4; hex 031a3825; asc 8%;;
|
Apr 28 05:41:34 labsdb1011 mysqld[31450]: 2020-04-28 5:41:34 0 [ERROR] InnoDB: page [page id: space=274393, page number=3535947] (166
|
Apr 28 05:41:34 labsdb1011 mysqld[31450]: 2020-04-28 5:41:34 0 [ERROR] InnoDB: Submit a detailed bug report to https://jira.mariadb.or
|
Apr 28 05:41:34 labsdb1011 mysqld[31450]: 2020-04-28 5:41:34 0 [ERROR] InnoDB: Unable to find a record to delete-mark
|
Apr 28 05:41:34 labsdb1011 mysqld[31450]: InnoDB: tuple DATA TUPLE: 3 fields;
|
Apr 28 05:41:34 labsdb1011 mysqld[31450]: 0: len 43; hex 50686f746f6772617068735f6f665f686162697461745f6f665f696e73656374655f62795f477
|
Apr 28 05:41:34 labsdb1011 mysqld[31450]: 1: len 4; hex 5ea7c187; asc ^ ;;
|
Apr 28 05:41:34 labsdb1011 mysqld[31450]: 2: len 4; hex 0391d80d; asc ;;
|
Apr 28 05:41:34 labsdb1011 mysqld[31450]: InnoDB: record PHYSICAL RECORD: n_fields 3; compact format; info bits 0
|
Apr 28 05:41:34 labsdb1011 mysqld[31450]: 0: len 30; hex 50686f746f6772617068735f6f665f686162697461745f6f665f6661756e; asc Photographs
|
Apr 28 05:41:34 labsdb1011 mysqld[31450]: 1: len 4; hex 5ea7c1b6; asc ^ ;;
|
Apr 28 05:41:34 labsdb1011 mysqld[31450]: 2: len 4; hex 031a3825; asc 8%;;
|
Apr 28 05:41:34 labsdb1011 mysqld[31450]: 2020-04-28 5:41:34 0 [ERROR] InnoDB: page [page id: space=274393, page number=3535947] (167
|
Apr 28 05:41:34 labsdb1011 mysqld[31450]: 2020-04-28 5:41:34 0 [ERROR] InnoDB: Submit a detailed bug report to https://jira.mariadb.or
|
lines 1048-1067/1067 (END)
|
Apr 28 05:41:34 labsdb1011 mysqld[31450]: 0: len 43; hex 50686f746f6772617068735f6f665f686162697461745f6f665f696e73656374655f62795f477a656e3932; asc Photographs_of_habitat_of_insecte_by_Gzen92;;
|
Apr 28 05:41:34 labsdb1011 mysqld[31450]: 1: len 4; hex 5ea7c180; asc ^ ;;
|
Apr 28 05:41:34 labsdb1011 mysqld[31450]: 2: len 4; hex 0312d9e9; asc ;;
|
Apr 28 05:41:34 labsdb1011 mysqld[31450]: InnoDB: record PHYSICAL RECORD: n_fields 3; compact format; info bits 0
|
Apr 28 05:41:34 labsdb1011 mysqld[31450]: 0: len 30; hex 50686f746f6772617068735f6f665f686162697461745f6f665f6661756e; asc Photographs_of_habitat_of_faun; (total 41 bytes);
|
Apr 28 05:41:34 labsdb1011 mysqld[31450]: 1: len 4; hex 5ea7c1b6; asc ^ ;;
|
Apr 28 05:41:34 labsdb1011 mysqld[31450]: 2: len 4; hex 031a3825; asc 8%;;
|
Apr 28 05:41:34 labsdb1011 mysqld[31450]: 2020-04-28 5:41:34 0 [ERROR] InnoDB: page [page id: space=274393, page number=3535947] (166 records, index id 914554).
|
Apr 28 05:41:34 labsdb1011 mysqld[31450]: 2020-04-28 5:41:34 0 [ERROR] InnoDB: Submit a detailed bug report to https://jira.mariadb.org/
|
Apr 28 05:41:34 labsdb1011 mysqld[31450]: 2020-04-28 5:41:34 0 [ERROR] InnoDB: Unable to find a record to delete-mark
|
Apr 28 05:41:34 labsdb1011 mysqld[31450]: InnoDB: tuple DATA TUPLE: 3 fields;
|
Apr 28 05:41:34 labsdb1011 mysqld[31450]: 0: len 43; hex 50686f746f6772617068735f6f665f686162697461745f6f665f696e73656374655f62795f477a656e3932; asc Photographs_of_habitat_of_insecte_by_Gzen92;;
|
Apr 28 05:41:34 labsdb1011 mysqld[31450]: 1: len 4; hex 5ea7c187; asc ^ ;;
|
Apr 28 05:41:34 labsdb1011 mysqld[31450]: 2: len 4; hex 0391d80d; asc ;;
|
Apr 28 05:41:34 labsdb1011 mysqld[31450]: InnoDB: record PHYSICAL RECORD: n_fields 3; compact format; info bits 0
|
Apr 28 05:41:34 labsdb1011 mysqld[31450]: 0: len 30; hex 50686f746f6772617068735f6f665f686162697461745f6f665f6661756e; asc Photographs_of_habitat_of_faun; (total 41 bytes);
|
Apr 28 05:41:34 labsdb1011 mysqld[31450]: 1: len 4; hex 5ea7c1b6; asc ^ ;;
|
Apr 28 05:41:34 labsdb1011 mysqld[31450]: 2: len 4; hex 031a3825; asc 8%;;
|
Apr 28 05:41:34 labsdb1011 mysqld[31450]: 2020-04-28 5:41:34 0 [ERROR] InnoDB: page [page id: space=274393, page number=3535947] (167 records, index id 914554).
|
Apr 28 05:41:34 labsdb1011 mysqld[31450]: 2020-04-28 5:41:34 0 [ERROR] InnoDB: Submit a detailed bug report to https://jira.mariadb.org/
|
|
|
Apr 28 05:43:21 labsdb1011 mysqld[31450]: 2020-04-28 5:43:21 29 [ERROR] Master 's2': InnoDB: Record in index `cl_sortkey` of table `zhwiki`.`categorylinks` was not found on update: TUPLE (info_bits=0, 4 fields): {[27] (0xE6A183E59C92E5B882E9AB98E7B49AE4B8ADE7AD89E5ADB8E6A0A1),[1] (0x01),[39] (0xE6A183E59C92E5B882E68890E58A9FE9AB98E7B49AE5B7A5E59586E881B7E6A5ADE5ADB8E6A0A1),[4] ~k(0x00097E6B)} at: COMPACT RECORD(info_bits=0, 4 fields): {[27] (0xE6A183E59C92E5B882E9AB98E7B49AE4B8ADE7AD89E5ADB8E6A0A1),[1] (0x01),[39] (0xE6A183E59C92E5B882E68890E58A9FE5B7A5E59586E9AB98E7B49AE4B8ADE7AD89E5ADB8E6A0A1),[4] ~k(0x00097E6B)}
|
Apr 28 05:43:21 labsdb1011 mysqld[31450]: 2020-04-28 5:43:21 29 [ERROR] Master 's2': InnoDB: Record in index `cl_sortkey` of table `zhwiki`.`categorylinks` was not found on update: TUPLE (info_bits=0, 4 fields): {[30] (0xE5889DE7BAA7E4BD8EE9878DE8A681E5BAA6E887BAE781A3E69DA1E79BAE),[1] (0x01),[39] (0xE6A183E59C92E5B882E68890E58A9FE9AB98E7B49AE5B7A5E59586E881B7E6A5ADE5ADB8E6A0A1),[4] `(0x000AB660)} at: COMPACT RECORD(info_bits=0, 4 fields): {[30] (0xE5889DE7BAA7E4BD8EE9878DE8A681E5BAA6E887BAE781A3E69DA1E79BAE),[1] (0x01),[39]
|
Providing some of the table schemas that were mentioned on the crashes:
CREATE TABLE `categorylinks` (
|
`cl_from` int(8) unsigned NOT NULL DEFAULT 0,
|
`cl_to` varbinary(255) NOT NULL DEFAULT '',
|
`cl_sortkey` varbinary(230) NOT NULL DEFAULT '',
|
`cl_timestamp` timestamp NOT NULL DEFAULT current_timestamp() ON UPDATE current_timestamp(),
|
`cl_sortkey_prefix` varbinary(255) NOT NULL DEFAULT '',
|
`cl_collation` varbinary(32) NOT NULL DEFAULT '',
|
`cl_type` enum('page','subcat','file') NOT NULL DEFAULT 'page',
|
PRIMARY KEY (`cl_from`,`cl_to`),
|
KEY `cl_timestamp` (`cl_to`,`cl_timestamp`),
|
KEY `cl_sortkey` (`cl_to`,`cl_type`,`cl_sortkey`,`cl_from`),
|
KEY `cl_collation_ext` (`cl_collation`,`cl_to`,`cl_type`,`cl_from`)
|
) ENGINE=InnoDB DEFAULT CHARSET=binary ROW_FORMAT=COMPRESSED KEY_BLOCK_SIZE=8
|
|
|
CREATE TABLE `revision` (
|
`rev_id` int(8) unsigned NOT NULL AUTO_INCREMENT,
|
`rev_page` int(8) unsigned NOT NULL DEFAULT 0,
|
`rev_text_id` int(8) unsigned NOT NULL DEFAULT 0,
|
`rev_comment` varbinary(255) NOT NULL DEFAULT '',
|
`rev_user` int(5) unsigned NOT NULL DEFAULT 0,
|
`rev_user_text` varbinary(255) NOT NULL DEFAULT '',
|
`rev_timestamp` varbinary(14) NOT NULL DEFAULT '',
|
`rev_minor_edit` tinyint(1) unsigned NOT NULL DEFAULT 0,
|
`rev_deleted` tinyint(1) unsigned NOT NULL DEFAULT 0,
|
`rev_len` int(8) unsigned DEFAULT NULL,
|
`rev_parent_id` int(8) unsigned DEFAULT NULL,
|
`rev_sha1` varbinary(32) NOT NULL DEFAULT '',
|
`rev_content_model` varbinary(32) DEFAULT NULL,
|
`rev_content_format` varbinary(64) DEFAULT NULL,
|
PRIMARY KEY (`rev_id`),
|
KEY `rev_timestamp` (`rev_timestamp`),
|
KEY `page_timestamp` (`rev_page`,`rev_timestamp`),
|
KEY `user_timestamp` (`rev_user`,`rev_timestamp`),
|
KEY `usertext_timestamp` (`rev_user_text`,`rev_timestamp`),
|
KEY `page_user_timestamp` (`rev_page`,`rev_user`,`rev_timestamp`),
|
KEY `rev_page_id` (`rev_page`,`rev_id`)
|
) ENGINE=InnoDB AUTO_INCREMENT=22225197 DEFAULT CHARSET=binary ROW_FORMAT=COMPRESSED KEY_BLOCK_SIZE=8
|
|
|
| flaggedrevs_tracking | CREATE TABLE `flaggedrevs_tracking` (
|
`ftr_from` int(10) unsigned NOT NULL DEFAULT 0,
|
`ftr_namespace` int(11) NOT NULL DEFAULT 0,
|
`ftr_title` varbinary(255) NOT NULL DEFAULT '',
|
UNIQUE KEY `from_namespace_title` (`ftr_from`,`ftr_namespace`,`ftr_title`),
|
KEY `namespace_title_from` (`ftr_namespace`,`ftr_title`,`ftr_from`)
|
) ENGINE=InnoDB DEFAULT CHARSET=binary ROW_FORMAT=COMPRESSED KEY_BLOCK_SIZE=8
|
|
|
CREATE TABLE `templatelinks` (
|
`tl_from` int(8) unsigned NOT NULL DEFAULT 0,
|
`tl_namespace` int(11) NOT NULL DEFAULT 0,
|
`tl_title` varbinary(255) NOT NULL DEFAULT '',
|
`tl_from_namespace` int(11) NOT NULL DEFAULT 0,
|
PRIMARY KEY (`tl_from`,`tl_namespace`,`tl_title`),
|
KEY `tl_namespace` (`tl_namespace`,`tl_title`,`tl_from`),
|
KEY `tl_backlinks_namespace` (`tl_from_namespace`,`tl_namespace`,`tl_title`,`tl_from`)
|
) ENGINE=InnoDB DEFAULT CHARSET=binary ROW_FORMAT=COMPRESSED KEY_BLOCK_SIZE=8
|
|
|
CREATE TABLE `iwlinks` (
|
`iwl_from` int(10) unsigned NOT NULL DEFAULT 0,
|
`iwl_prefix` varbinary(20) NOT NULL DEFAULT '',
|
`iwl_title` varbinary(255) NOT NULL DEFAULT '',
|
PRIMARY KEY (`iwl_from`,`iwl_prefix`,`iwl_title`),
|
KEY `iwl_prefix_title_from` (`iwl_prefix`,`iwl_title`,`iwl_from`),
|
KEY `iwl_prefix_from_title` (`iwl_prefix`,`iwl_from`,`iwl_title`)
|
) ENGINE=InnoDB DEFAULT CHARSET=binary ROW_FORMAT=COMPRESSED KEY_BLOCK_SIZE=8
|
|
|
CREATE TABLE `recentchanges` (
|
`rc_id` int(8) NOT NULL AUTO_INCREMENT,
|
`rc_timestamp` varbinary(14) NOT NULL DEFAULT '',
|
`rc_actor` bigint(20) unsigned NOT NULL,
|
`rc_namespace` int(11) NOT NULL DEFAULT 0,
|
`rc_title` varbinary(255) NOT NULL DEFAULT '',
|
`rc_comment_id` bigint(20) unsigned NOT NULL,
|
`rc_minor` tinyint(3) unsigned NOT NULL DEFAULT 0,
|
`rc_bot` tinyint(3) unsigned NOT NULL DEFAULT 0,
|
`rc_new` tinyint(3) unsigned NOT NULL DEFAULT 0,
|
`rc_cur_id` int(10) unsigned NOT NULL DEFAULT 0,
|
`rc_this_oldid` int(10) unsigned NOT NULL DEFAULT 0,
|
`rc_last_oldid` int(10) unsigned NOT NULL DEFAULT 0,
|
`rc_type` tinyint(3) unsigned NOT NULL DEFAULT 0,
|
`rc_source` varbinary(16) NOT NULL DEFAULT '',
|
`rc_patrolled` tinyint(3) unsigned NOT NULL DEFAULT 0,
|
`rc_ip` varbinary(40) NOT NULL DEFAULT '',
|
`rc_old_len` int(10) DEFAULT NULL,
|
`rc_new_len` int(10) DEFAULT NULL,
|
`rc_deleted` tinyint(1) unsigned NOT NULL DEFAULT 0,
|
`rc_logid` int(10) unsigned NOT NULL DEFAULT 0,
|
`rc_log_type` varbinary(255) DEFAULT NULL,
|
`rc_log_action` varbinary(255) DEFAULT NULL,
|
`rc_params` blob NOT NULL,
|
PRIMARY KEY (`rc_id`),
|
KEY `rc_timestamp` (`rc_timestamp`),
|
KEY `rc_cur_id` (`rc_cur_id`),
|
KEY `new_name_timestamp` (`rc_new`,`rc_namespace`,`rc_timestamp`),
|
KEY `rc_ip` (`rc_ip`),
|
KEY `rc_name_type_patrolled_timestamp` (`rc_namespace`,`rc_type`,`rc_patrolled`,`rc_timestamp`),
|
KEY `rc_comment_deleted` (`rc_comment_id`,`rc_deleted`),
|
KEY `rc_ns_actor` (`rc_namespace`,`rc_actor`),
|
KEY `rc_actor` (`rc_actor`,`rc_timestamp`),
|
KEY `rc_namespace_title_timestamp` (`rc_namespace`,`rc_title`,`rc_timestamp`),
|
KEY `rc_actor_deleted` (`rc_actor`,`rc_deleted`),
|
KEY `rc_this_oldid` (`rc_this_oldid`)
|
) ENGINE=InnoDB AUTO_INCREMENT=38170865 DEFAULT CHARSET=binary ROW_FORMAT=COMPRESSED KEY_BLOCK_SIZE=8
|
|
|
CREATE TABLE `revision_comment_temp` (
|
`revcomment_rev` int(10) unsigned NOT NULL,
|
`revcomment_comment_id` bigint(20) unsigned NOT NULL,
|
PRIMARY KEY (`revcomment_rev`,`revcomment_comment_id`),
|
UNIQUE KEY `revcomment_rev` (`revcomment_rev`),
|
KEY `revcomment_comment_id` (`revcomment_comment_id`)
|
) ENGINE=InnoDB DEFAULT CHARSET=binary ROW_FORMAT=COMPRESSED KEY_BLOCK_SIZE=8
|
Worth mentioning that these crashes are also happening without
innodb_purge_threads = 10
|
The server was recloned from a healthy server running 10.1
Once the data was copied to the one running 10.4.12, we ran
mysql_upgrade
|
and after a while it started to show InnoDB errors again:
Apr 28 13:41:08 labsdb1011 mysqld[21278]: 2020-04-28 13:41:08 1 [ERROR] InnoDB: page [page id: space=337413, page number=1228322] (219 records, index id 1245829).
|
Apr 28 13:41:08 labsdb1011 mysqld[21278]: 2020-04-28 13:41:08 1 [ERROR] InnoDB: Submit a detailed bug report to https://jira.mariadb.org/
|
Apr 28 13:41:08 labsdb1011 mysqld[21278]: 2020-04-28 13:41:08 4 [ERROR] InnoDB: Unable to find a record to delete-mark
|
Apr 28 13:41:08 labsdb1011 mysqld[21278]: InnoDB: tuple DATA TUPLE: 3 fields;
|
Apr 28 13:41:08 labsdb1011 mysqld[21278]: 0: len 4; hex 80000000; asc ;;
|
Apr 28 13:41:08 labsdb1011 mysqld[21278]: 1: len 19; hex 4465746c65665f4dc3bc6c6c65722d4d61686e; asc Detlef_M ller-Mahn;;
|
Apr 28 13:41:08 labsdb1011 mysqld[21278]: 2: len 4; hex 00a7561c; asc V ;;
|
Apr 28 13:41:08 labsdb1011 mysqld[21278]: InnoDB: record PHYSICAL RECORD: n_fields 3; compact format; info bits 0
|
Apr 28 13:41:08 labsdb1011 mysqld[21278]: 0: len 4; hex 80000000; asc ;;
|
Apr 28 13:41:08 labsdb1011 mysqld[21278]: 1: len 19; hex 4465746c65665f4dc3bc6c6c65722d4d61686e; asc Detlef_M ller-Mahn;;
|
Apr 28 13:41:08 labsdb1011 mysqld[21278]: 2: len 4; hex 009eb61d; asc ;;
|
Apr 28 13:41:08 labsdb1011 mysqld[21278]: 2020-04-28 13:41:08 4 [ERROR] InnoDB: page [page id: space=337413, page number=1625870] (252 records, index id 1245828).
|
Apr 28 13:41:08 labsdb1011 mysqld[21278]: 2020-04-28 13:41:08 4 [ERROR] InnoDB: Submit a detailed bug report to https://jira.mariadb.org/
|
Apr 28 13:41:08 labsdb1011 mysqld[21278]: 2020-04-28 13:41:08 352 [ERROR] Master 's2': InnoDB: Record in index `tl_namespace` of table `zhwiki`.`templatelinks` was not found on update: TUPLE (info_bits=0, 3 fields): {[4] (0x8000000A),[7]Str_len(0x5374725F6C656E),[4] C
|
(0x004395E6)} at: COMPACT RECORD(info_bits=0, 3 fields): {[4] (0x8000000A),[7]Str_len(0x5374725F6C656E),[4] C (0x004395E5)}
|
Apr 28 13:41:08 labsdb1011 mysqld[21278]: 2020-04-28 13:41:08 352 [ERROR] Master 's2': InnoDB: Record in index `tl_namespace` of table `zhwiki`.`templatelinks` was not found on update: TUPLE (info_bits=0, 3 fields): {[4] <(0x8000033C),[21]Namespace_detect/data(0x4E616D6
|
573706163655F6465746563742F64617461),[4] C (0x004395E6)} at: COMPACT RECORD(info_bits=0, 3 fields): {[4] <(0x8000033C),[21]Namespace_detect/data(0x4E616D6573706163655F6465746563742F64617461),[4] C (0x004395E5)}
|
Apr 28 13:41:08 labsdb1011 mysqld[21278]: 2020-04-28 13:41:08 4 [ERROR] InnoDB: Unable to find a record to delete-mark
|
Apr 28 13:41:08 labsdb1011 mysqld[21278]: 2: len 6; hex 537472696e67; asc String;;
|
Apr 28 13:41:08 labsdb1011 mysqld[21278]: 3: len 4; hex 000c3e9d; asc > ;;
|
Apr 28 13:41:08 labsdb1011 mysqld[21278]: 2020-04-28 13:41:08 4 [ERROR] InnoDB: page [page id: space=360989, page number=796694] (471 records, index id 1332346).
|
Apr 28 13:41:08 labsdb1011 mysqld[21278]: 2020-04-28 13:41:08 4 [ERROR] InnoDB: Submit a detailed bug report to https://jira.mariadb.org/
|
Apr 28 13:41:08 labsdb1011 mysqld[21278]: 2020-04-28 13:41:08 3 [ERROR] InnoDB: Unable to find a record to delete-mark
|
Apr 28 13:41:08 labsdb1011 mysqld[21278]: InnoDB: tuple DATA TUPLE: 3 fields;
|
Apr 28 13:41:08 labsdb1011 mysqld[21278]: 0: len 4; hex 8000033c; asc <;;
|
Apr 28 13:41:08 labsdb1011 mysqld[21278]: 1: len 23; hex 43617465676f72795f68616e646c65722f636f6e666967; asc Category_handler/config;;
|
Apr 28 13:41:08 labsdb1011 mysqld[21278]: 2: len 4; hex 000c3eb7; asc > ;;
|
Apr 28 13:41:08 labsdb1011 mysqld[21278]: InnoDB: record PHYSICAL RECORD: n_fields 3; compact format; info bits 0
|
Apr 28 13:41:08 labsdb1011 mysqld[21278]: 0: len 4; hex 8000033c; asc <;;
|
Apr 28 13:41:08 labsdb1011 mysqld[21278]: 1: len 23; hex 43617465676f72795f68616e646c65722f636f6e666967; asc Category_handler/config;;
|
Apr 28 13:41:08 labsdb1011 mysqld[21278]: 2: len 4; hex 000c3ea6; asc > ;;
|
Apr 28 13:41:08 labsdb1011 mysqld[21278]: 2020-04-28 13:41:08 3 [ERROR] InnoDB: page [page id: space=360989, page number=447482] (227 records, index id 1332345).
|
Apr 28 13:41:08 labsdb1011 mysqld[21278]: 2020-04-28 13:41:08 3 [ERROR] InnoDB: Submit a detailed bug report to https://jira.mariadb.org/
|
Apr 28 13:41:08 labsdb1011 mysqld[21278]: 2020-04-28 13:41:08 4 [ERROR] InnoDB: Unable to find a record to delete-mark
|
Apr 28 13:41:08 labsdb1011 mysqld[21278]: InnoDB: tuple DATA TUPLE: 3 fields;
|
Apr 28 13:41:08 labsdb1011 mysqld[21278]: 0: len 4; hex 8000033c; asc <;;
|
Apr 28 13:41:08 labsdb1011 mysqld[21278]: 1: len 11; hex 4d6573736167655f626f78; asc Message_box;;
|
Apr 28 13:41:08 labsdb1011 mysqld[21278]: 2: len 4; hex 000c3eb7; asc > ;;
|
Apr 28 13:41:08 labsdb1011 mysqld[21278]: InnoDB: record PHYSICAL RECORD: n_fields 3; compact format; info bits 0
|
Apr 28 13:41:08 labsdb1011 mysqld[21278]: 0: len 4; hex 8000033c; asc <;;
|
Apr 28 13:41:08 labsdb1011 mysqld[21278]: 1: len 11; hex 4d6573736167655f626f78; asc Message_box;;
|
Apr 28 13:41:08 labsdb1011 mysqld[21278]: 2: len 4; hex 000c3ea6; asc > ;;
|
Apr 28 13:41:08 labsdb1011 mysqld[21278]: 2020-04-28 13:41:08 4 [ERROR] InnoDB: page [page id: space=360989, page number=468334] (321 records, index id 1332345).
|
Apr 28 13:41:08 labsdb1011 mysqld[21278]: 2020-04-28 13:41:08 4 [ERROR] InnoDB: Submit a detailed bug report to https://jira.mariadb.org/
|
Apr 28 13:41:08 labsdb1011 mysqld[21278]: 2020-04-28 13:41:08 4 [ERROR] InnoDB: Unable to find a record to delete-mark
|
Apr 28 13:41:08 labsdb1011 mysqld[21278]: InnoDB: tuple DATA TUPLE: 3 fields;
|
Apr 28 13:41:08 labsdb1011 mysqld[21278]: 0: len 4; hex 8000033c; asc <;;
|
Apr 28 13:41:08 labsdb1011 mysqld[21278]: 1: len 26; hex 43617465676f72795f68616e646c65722f626c61636b6c697374; asc Category_handler/blacklist;;
|
|
|
Apr 28 15:11:03 labsdb1011 mysqld[21278]: 2020-04-28 15:11:03 358 [ERROR] Master 's8': InnoDB: Record in index `pl_backlinks_namespace` of table `wikidatawiki`.`pagelinks` was not found on update: TUPLE (info_bits=0, 4 fields): {[4] (0x80000004),[4] (0x80000000),[6]
|
Q12131(0x513132313331),[4] (0x038AEE87)} at: COMPACT RECORD(info_bits=0, 4 fields): {[4] (0x80000004),[4] (0x80000000),[6]Q12131(0x513132313331),[4] (0x03888E20)}
|
Apr 28 15:11:03 labsdb1011 mysqld[21278]: 2020-04-28 15:11:03 358 [ERROR] Master 's8': InnoDB: Record in index `pl_backlinks_namespace` of table `wikidatawiki`.`pagelinks` was not found on update: TUPLE (info_bits=0, 4 fields): {[4] (0x80000004),[4] (0x80000000),[9]
|
Q39209771(0x513339323039373731),[4] (0x038AEE87)} at: COMPACT RECORD(info_bits=0, 4 fields): {[4] (0x80000004),[4] (0x80000000),[8]Q3920970(0x5133393230393730),[4] (0x010C8AF7)}
|
Apr 28 15:11:03 labsdb1011 mysqld[21278]: 2020-04-28 15:11:03 358 [ERROR] Master 's8': InnoDB: Record in index `pl_backlinks_namespace` of table `wikidatawiki`.`pagelinks` was not found on update: TUPLE (info_bits=0, 4 fields): {[4] (0x80000004),[4] (0x80000000),[9]
|
Q56926409(0x513536393236343039),[4] (0x038AEE87)} at: COMPACT RECORD(info_bits=0, 4 fields): {[4] (0x80000004),[4] (0x80000000),[7]Q569264(0x51353639323634),[4] 3 (0x013320F2)}
|
Apr 28 15:11:03 labsdb1011 mysqld[21278]: 2020-04-28 15:11:03 358 [ERROR] Master 's8': InnoDB: Record in index `pl_backlinks_namespace` of table `wikidatawiki`.`pagelinks` was not found on update: TUPLE (info_bits=0, 4 fields): {[4] (0x80000004),[4] (0x80000000),[9]
|
Q56926417(0x513536393236343137),[4] (0x038AEE87)} at: COMPACT RECORD(info_bits=0, 4 fields): {[4] (0x80000004),[4] (0x80000000),[7]Q569264(0x51353639323634),[4] 3 (0x013320F2)}
|
Apr 28 15:11:03 labsdb1011 mysqld[21278]: 2020-04-28 15:11:03 358 [ERROR] Master 's8': InnoDB: Record in index `pl_backlinks_namespace` of table `wikidatawiki`.`pagelinks` was not found on update: TUPLE (info_bits=0, 4 fields): {[4] (0x80000004),[4] (0x80000000),[9]
|
Q56926423(0x513536393236343233),[4] (0x038AEE87)} at: COMPACT RECORD(info_bits=0, 4 fields): {[4] (0x80000004),[4] (0x80000000),[7]Q569264(0x51353639323634),[4] 3 (0x013320F2)}
|
Apr 28 15:11:03 labsdb1011 mysqld[21278]: 2020-04-28 15:11:03 358 [ERROR] Master 's8': InnoDB: Record in index `pl_backlinks_namespace` of table `wikidatawiki`.`pagelinks` was not found on update: TUPLE (info_bits=0, 4 fields): {[4] (0x80000004),[4] (0x80000000),[9]
|
Q56926427(0x513536393236343237),[4] (0x038AEE87)} at: COMPACT RECORD(info_bits=0, 4 fields): {[4] (0x80000004),[4] (0x80000000),[7]Q569264(0x51353639323634),[4] 3 (0x013320F2)}
|
Apr 28 15:11:03 labsdb1011 mysqld[21278]: 2020-04-28 15:11:03 358 [ERROR] Master 's8': InnoDB: Record in index `pl_backlinks_namespace` of table `wikidatawiki`.`pagelinks` was not found on update: TUPLE (info_bits=0, 4 fields): {[4] (0x80000004),[4] (0x80000000),[9]
|
Q56926434(0x513536393236343334),[4] (0x038AEE87)} at: COMPACT RECORD(info_bits=0, 4 fields): {[4] (0x80000004),[4] (0x80000000),[7]Q569264(0x51353639323634),[4] 3 (0x013320F2)}
|
Apr 28 15:11:05 labsdb1011 mysqld[21278]: 2020-04-28 15:11:05 1 [ERROR] InnoDB: Unable to find a record to delete-mark
|
Apr 28 15:11:05 labsdb1011 mysqld[21278]: InnoDB: tuple DATA TUPLE: 3 fields;
|
Apr 28 15:11:05 labsdb1011 mysqld[21278]: 0: len 8; hex 00000000090be5ea; asc ;;
|
Apr 28 15:11:05 labsdb1011 mysqld[21278]: 1: len 1; hex 00; asc ;;
|
Apr 28 15:11:05 labsdb1011 mysqld[21278]: 2: len 4; hex d2169fd7; asc ;;
|
Apr 28 15:11:05 labsdb1011 mysqld[21278]: InnoDB: record PHYSICAL RECORD: n_fields 3; compact format; info bits 32
|
Apr 28 15:11:05 labsdb1011 mysqld[21278]: 0: len 8; hex 00000000090be5ea; asc ;;
|
Apr 28 15:11:05 labsdb1011 mysqld[21278]: 1: len 1; hex 00; asc ;;
|
Apr 28 15:11:05 labsdb1011 mysqld[21278]: 2: len 4; hex d210b539; asc 9;;
|
Apr 28 15:11:05 labsdb1011 mysqld[21278]: 2020-04-28 15:11:05 1 [ERROR] InnoDB: page [page id: space=399231, page number=67490] (481 records, index id 1549094).
|
Apr 28 15:11:05 labsdb1011 mysqld[21278]: 2020-04-28 15:11:05 1 [ERROR] InnoDB: Submit a detailed bug report to https://jira.mariadb.org/
|
Apr 28 15:11:05 labsdb1011 mysqld[21278]: 2020-04-28 15:11:05 1 [ERROR] InnoDB: Unable to find a record to delete-mark
|
Apr 28 15:11:05 labsdb1011 mysqld[21278]: InnoDB: tuple DATA TUPLE: 4 fields;
|
Apr 28 15:11:05 labsdb1011 mysqld[21278]: 0: len 4; hex 8000000a; asc ;;
|
Apr 28 15:11:05 labsdb1011 mysqld[21278]: 1: len 4; hex 80000000; asc ;;
|
Apr 28 15:11:05 labsdb1011 mysqld[21278]: 2: len 13; hex 416d7061726f5f4d75c3b16f7a; asc Amparo_Mu oz;;
|
Apr 28 15:11:05 labsdb1011 mysqld[21278]: 3: len 4; hex 02f8b133; asc 3;;
|
Apr 28 15:11:05 labsdb1011 mysqld[21278]: InnoDB: record PHYSICAL RECORD: n_fields 4; compact format; info bits 0
|
Apr 28 15:11:05 labsdb1011 mysqld[21278]: 0: len 4; hex 8000000a; asc ;;
|
Apr 28 15:11:05 labsdb1011 mysqld[21278]: 1: len 4; hex 80000000; asc ;;
|
Apr 28 15:11:05 labsdb1011 mysqld[21278]: 2: len 13; hex 416d7061726f5f4d75c3b16f7a; asc Amparo_Mu oz;;
|
Apr 28 15:11:05 labsdb1011 mysqld[21278]: 3: len 4; hex 02dc7196; asc q ;;
|
Apr 28 15:11:05 labsdb1011 mysqld[21278]: 2020-04-28 15:11:05 1 [ERROR] InnoDB: page [page id: space=339140, page number=9549942] (255 records, index id 1252152).
|
Apr 28 15:11:05 labsdb1011 mysqld[21278]: 2020-04-28 15:11:05 1 [ERROR] InnoDB: Submit a detailed bug report to https://jira.mariadb.org/
|
Apr 28 15:11:07 labsdb1011 mysqld[21278]: 2020-04-28 15:11:07 0x7f6f181fc700 InnoDB: Assertion failure in file /root/mariadb-10.4.12/storage/innobase/row/row0ins.cc line 263
|
Apr 28 15:11:07 labsdb1011 mysqld[21278]: InnoDB: Failing assertion: !cursor->index->is_committed()
|
Apr 28 15:11:07 labsdb1011 mysqld[21278]: InnoDB: We intentionally generate a memory trap.
|
Apr 28 15:11:07 labsdb1011 mysqld[21278]: InnoDB: Submit a detailed bug report to https://jira.mariadb.org/
|
Apr 28 15:11:07 labsdb1011 mysqld[21278]: InnoDB: If you get repeated assertion failures or crashes, even
|
Apr 28 15:11:07 labsdb1011 mysqld[21278]: InnoDB: immediately after the mysqld startup, there may be
|
Apr 28 15:11:07 labsdb1011 mysqld[21278]: InnoDB: corruption in the InnoDB tablespace. Please refer to
|
Apr 28 15:11:07 labsdb1011 mysqld[21278]: InnoDB: https://mariadb.com/kb/en/library/innodb-recovery-modes/
|
Apr 28 15:11:07 labsdb1011 mysqld[21278]: InnoDB: about forcing recovery.
|
Apr 28 15:11:07 labsdb1011 mysqld[21278]: 200428 15:11:07 [ERROR] mysqld got signal 6 ;
|
Apr 28 15:11:07 labsdb1011 mysqld[21278]: This could be because you hit a bug. It is also possible that this binary
|
Apr 28 15:11:07 labsdb1011 mysqld[21278]: or one of the libraries it was linked against is corrupt, improperly built,
|
Apr 28 15:11:07 labsdb1011 mysqld[21278]: or misconfigured. This error can also be caused by malfunctioning hardware.
|
Apr 28 15:11:07 labsdb1011 mysqld[21278]: To report this bug, see https://mariadb.com/kb/en/reporting-bugs
|
Apr 28 15:11:07 labsdb1011 mysqld[21278]: We will try our best to scrape up some info that will hopefully help
|
Apr 28 15:11:07 labsdb1011 mysqld[21278]: diagnose the problem, but since we have already crashed,
|
Apr 28 15:11:07 labsdb1011 mysqld[21278]: something is definitely wrong and this may fail.
|
Apr 28 15:11:07 labsdb1011 mysqld[21278]: Server version: 10.4.12-MariaDB
|
Apr 28 15:11:07 labsdb1011 mysqld[21278]: key_buffer_size=134217728
|
Apr 28 15:11:07 labsdb1011 mysqld[21278]: read_buffer_size=131072
|
Apr 28 15:11:07 labsdb1011 mysqld[21278]: max_used_connections=13
|
Apr 28 15:11:07 labsdb1011 mysqld[21278]: max_threads=1026
|
Apr 28 15:11:07 labsdb1011 mysqld[21278]: thread_count=36
|
Apr 28 15:11:07 labsdb1011 mysqld[21278]: It is possible that mysqld could use up to
|
Apr 28 15:11:07 labsdb1011 mysqld[21278]: key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 2388976 K bytes of memory
|
Apr 28 15:11:07 labsdb1011 mysqld[21278]: Hope that's ok; if not, decrease some variables in the equation.
|
Apr 28 15:11:07 labsdb1011 mysqld[21278]: Thread pointer: 0x7f6e500014f8
|
Apr 28 15:11:07 labsdb1011 mysqld[21278]: Attempting backtrace. You can use the following information to find out
|
Apr 28 15:11:07 labsdb1011 mysqld[21278]: where mysqld died. If you see no messages after this, something went
|
Apr 28 15:11:07 labsdb1011 mysqld[21278]: terribly wrong...
|
Apr 28 15:11:07 labsdb1011 mysqld[21278]: stack_bottom = 0x7f6f181fae98 thread_stack 0x30000
|
Apr 28 15:11:08 labsdb1011 mysqld[21278]: /opt/wmf-mariadb104/bin/mysqld(my_print_stacktrace+0x2e)[0x55df664045ee]
|
Apr 28 15:11:09 labsdb1011 mysqld[21278]: /opt/wmf-mariadb104/bin/mysqld(handle_fatal_signal+0x54d)[0x55df65efa13d]
|
Apr 28 15:11:10 labsdb1011 mysqld[21278]: /lib/x86_64-linux-gnu/libpthread.so.0(+0x12730)[0x7fd5cd64d730]
|
Apr 28 15:11:11 labsdb1011 mysqld[21278]: /lib/x86_64-linux-gnu/libc.so.6(gsignal+0x10b)[0x7fd5cccb57bb]
|
Apr 28 15:11:12 labsdb1011 mysqld[21278]: /lib/x86_64-linux-gnu/libc.so.6(abort+0x121)[0x7fd5ccca0535]
|
Apr 28 15:11:13 labsdb1011 mysqld[21278]: /opt/wmf-mariadb104/bin/mysqld(+0x5a10e5)[0x55df65bef0e5]
|
Apr 28 15:11:15 labsdb1011 mysqld[21278]: /opt/wmf-mariadb104/bin/mysqld(+0x58fe9b)[0x55df65bdde9b]
|
Apr 28 15:11:15 labsdb1011 mysqld[21278]: /opt/wmf-mariadb104/bin/mysqld(+0xaf46e0)[0x55df661426e0]
|
Apr 28 15:11:16 labsdb1011 mysqld[21278]: /opt/wmf-mariadb104/bin/mysqld(+0xaf4d87)[0x55df66142d87]
|
Apr 28 15:11:17 labsdb1011 mysqld[21278]: /opt/wmf-mariadb104/bin/mysqld(+0xb0455a)[0x55df6615255a]
|
Apr 28 15:11:18 labsdb1011 mysqld[21278]: /opt/wmf-mariadb104/bin/mysqld(+0xa53b53)[0x55df660a1b53]
|
Apr 28 15:11:19 labsdb1011 mysqld[21278]: /opt/wmf-mariadb104/bin/mysqld(_ZN7handler12ha_write_rowEPKh+0x14d)[0x55df65f0666d]
|
Apr 28 15:11:20 labsdb1011 mysqld[21278]: /opt/wmf-mariadb104/bin/mysqld(_ZN14Rows_log_event9write_rowEP14rpl_group_infob+0x174)[0x55df65ff8cf4]
|
Apr 28 15:11:21 labsdb1011 mysqld[21278]: /opt/wmf-mariadb104/bin/mysqld(_ZN20Write_rows_log_event11do_exec_rowEP14rpl_group_info+0x7d)[0x55df65ff928d]
|
Apr 28 15:11:22 labsdb1011 mysqld[21278]: /opt/wmf-mariadb104/bin/mysqld(_ZN14Rows_log_event14do_apply_eventEP14rpl_group_info+0x25d)[0x55df65fed47d]
|
Apr 28 15:11:23 labsdb1011 mysqld[21278]: /opt/wmf-mariadb104/bin/mysqld(+0x5faac8)[0x55df65c48ac8]
|
Apr 28 15:11:24 labsdb1011 mysqld[21278]: /opt/wmf-mariadb104/bin/mysqld(handle_slave_sql+0x169a)[0x55df65c51c6a]
|
Apr 28 15:11:25 labsdb1011 mysqld[21278]: /opt/wmf-mariadb104/bin/mysqld(+0xd66f8b)[0x55df663b4f8b]
|
Apr 28 15:11:26 labsdb1011 mysqld[21278]: /lib/x86_64-linux-gnu/libpthread.so.0(+0x7fa3)[0x7fd5cd642fa3]
|
Apr 28 15:11:27 labsdb1011 mysqld[21278]: /lib/x86_64-linux-gnu/libc.so.6(clone+0x3f)[0x7fd5ccd774cf]
|
Apr 28 15:11:27 labsdb1011 mysqld[21278]: Trying to get some variables.
|
Apr 28 15:11:27 labsdb1011 mysqld[21278]: Some pointers may be invalid and cause the dump to abort.
|
Apr 28 15:11:27 labsdb1011 mysqld[21278]: Query (0x0):
|
Apr 28 15:11:27 labsdb1011 mysqld[21278]: Connection ID (thread ID): 362
|
Apr 28 15:11:27 labsdb1011 mysqld[21278]: Status: NOT_KILLED
|
Apr 28 15:11:27 labsdb1011 mysqld[21278]: Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=on,condition_pushdown_from_having=on
|
Apr 28 15:11:27 labsdb1011 mysqld[21278]: The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
|
Apr 28 15:11:27 labsdb1011 mysqld[21278]: information that should help you find out what is causing the crash.
|
Apr 28 15:11:27 labsdb1011 mysqld[21278]: Writing a core file...
|
Apr 28 15:11:27 labsdb1011 mysqld[21278]: Working directory at /srv/sqldata
|
Apr 28 15:11:27 labsdb1011 mysqld[21278]: Resource Limits:
|
Apr 28 15:11:27 labsdb1011 mysqld[21278]: Limit Soft Limit Hard Limit Units
|
Apr 28 15:11:27 labsdb1011 mysqld[21278]: Max cpu time unlimited unlimited seconds
|
Apr 28 15:11:27 labsdb1011 mysqld[21278]: Max file size unlimited unlimited bytes
|
Apr 28 15:11:27 labsdb1011 mysqld[21278]: Max data size unlimited unlimited bytes
|
Apr 28 15:11:27 labsdb1011 mysqld[21278]: Max stack size 8388608 unlimited bytes
|
Apr 28 15:11:27 labsdb1011 mysqld[21278]: Max core file size 0 0 bytes
|
Apr 28 15:11:27 labsdb1011 mysqld[21278]: Max resident set unlimited unlimited bytes
|
Apr 28 15:11:27 labsdb1011 mysqld[21278]: Max processes 2063395 2063395 processes
|
Apr 28 15:11:27 labsdb1011 mysqld[21278]: Max open files 200001 200001 files
|
Apr 28 15:11:27 labsdb1011 mysqld[21278]: Max locked memory 65536 65536 bytes
|
Apr 28 15:11:27 labsdb1011 mysqld[21278]: Max address space unlimited unlimited bytes
|
Apr 28 15:11:27 labsdb1011 mysqld[21278]: Max file locks unlimited unlimited locks
|
Apr 28 15:11:27 labsdb1011 mysqld[21278]: Max pending signals 2063395 2063395 signals
|
Apr 28 15:11:27 labsdb1011 mysqld[21278]: Max msgqueue size 819200 819200 bytes
|
Apr 28 15:11:27 labsdb1011 mysqld[21278]: Max nice priority 0 0
|
Apr 28 15:11:27 labsdb1011 mysqld[21278]: Max realtime priority 0 0
|
Apr 28 15:11:27 labsdb1011 mysqld[21278]: Max realtime timeout unlimited unlimited us
|
Apr 28 15:11:27 labsdb1011 mysqld[21278]: Core pattern: /var/tmp/core/core.%h.%e.%p.%t
|
This also happens after importing a mydumper logical copy from a 10.1 host into the 10.4 one.
We have seen more things:
- After a logical reimport, if we configure replication, start it and DO NOT restart mysql daemon, this doesn't happen.
We left the host running for a week and nothing showed up. The 8 replication threads worked fine with no errors.
As soon as we restarted the mysql daemon, and started all replication threads we started seeing:
May 26 04:20:39 labsdb1011 mysqld[19806]: 2020-05-26 4:20:39 31 [Note] Master 's6': Slave I/O thread: Start asynchronous replication to master 'repl@db1125.eqiad.wmnet:3316' in log 'db1125-bin.001516' at position 150259934
|
May 26 04:20:39 labsdb1011 mysqld[19806]: 2020-05-26 4:20:39 33 [Note] Master 's2': Slave I/O thread: Start asynchronous replication to master 'repl@db1125.eqiad.wmnet:3312' in log 'db1125-bin.002403' at position 440364672
|
May 26 04:20:39 labsdb1011 mysqld[19806]: 2020-05-26 4:20:39 35 [Note] Master 's3': Slave I/O thread: Start asynchronous replication to master 'repl@db1124.eqiad.wmnet:3313' in log 'db1124-bin.002091' at position 844723555
|
May 26 04:20:39 labsdb1011 mysqld[19806]: 2020-05-26 4:20:39 37 [Note] Master 's4': Slave I/O thread: Start asynchronous replication to master 'repl@db1125.eqiad.wmnet:3314' in log 'db1125-bin.003581' at position 680368537
|
May 26 04:20:39 labsdb1011 mysqld[19806]: 2020-05-26 4:20:39 39 [Note] Master 's8': Slave I/O thread: Start asynchronous replication to master 'repl@db1124.eqiad.wmnet:3318' in log 'db1124-bin.003703' at position 26857432
|
May 26 04:20:39 labsdb1011 mysqld[19806]: 2020-05-26 4:20:39 41 [Note] Master 's5': Slave I/O thread: Start asynchronous replication to master 'repl@db1124.eqiad.wmnet:3315' in log 'db1124-bin.001226' at position 889729958
|
May 26 04:20:39 labsdb1011 mysqld[19806]: 2020-05-26 4:20:39 43 [Note] Master 's7': Slave I/O thread: Start asynchronous replication to master 'repl@db1125.eqiad.wmnet:3317' in log 'db1125-bin.002189' at position 486105063
|
May 26 04:20:39 labsdb1011 mysqld[19806]: 2020-05-26 4:20:39 45 [Note] Master 's1': Slave I/O thread: Start asynchronous replication to master 'repl@db1124.eqiad.wmnet:3311' in log 'db1124-bin.002696' at position 813770887
|
May 26 04:20:39 labsdb1011 mysqld[19806]: 2020-05-26 4:20:39 37 [Note] Master 's4': Slave I/O thread: connected to master 'repl@db1125.eqiad.wmnet:3314',replication started in log 'db1125-bin.003581' at position 680368537
|
May 26 04:20:39 labsdb1011 mysqld[19806]: 2020-05-26 4:20:39 35 [Note] Master 's3': Slave I/O thread: connected to master 'repl@db1124.eqiad.wmnet:3313',replication started in log 'db1124-bin.002091' at position 844723555
|
May 26 04:20:39 labsdb1011 mysqld[19806]: 2020-05-26 4:20:39 43 [Note] Master 's7': Slave I/O thread: connected to master 'repl@db1125.eqiad.wmnet:3317',replication started in log 'db1125-bin.002189' at position 486105063
|
May 26 04:20:39 labsdb1011 mysqld[19806]: 2020-05-26 4:20:39 45 [Note] Master 's1': Slave I/O thread: connected to master 'repl@db1124.eqiad.wmnet:3311',replication started in log 'db1124-bin.002696' at position 813770887
|
May 26 04:20:39 labsdb1011 mysqld[19806]: 2020-05-26 4:20:39 33 [Note] Master 's2': Slave I/O thread: connected to master 'repl@db1125.eqiad.wmnet:3312',replication started in log 'db1125-bin.002403' at position 440364672
|
May 26 04:20:39 labsdb1011 mysqld[19806]: 2020-05-26 4:20:39 31 [Note] Master 's6': Slave I/O thread: connected to master 'repl@db1125.eqiad.wmnet:3316',replication started in log 'db1125-bin.001516' at position 150259934
|
May 26 04:20:39 labsdb1011 mysqld[19806]: 2020-05-26 4:20:39 41 [Note] Master 's5': Slave I/O thread: connected to master 'repl@db1124.eqiad.wmnet:3315',replication started in log 'db1124-bin.001226' at position 889729958
|
May 26 04:20:39 labsdb1011 mysqld[19806]: 2020-05-26 4:20:39 39 [Note] Master 's8': Slave I/O thread: connected to master 'repl@db1124.eqiad.wmnet:3318',replication started in log 'db1124-bin.003703' at position 26857432
|
May 26 04:20:44 labsdb1011 mysqld[19806]: 2020-05-26 4:20:44 4 [ERROR] InnoDB: Unable to find a record to delete-mark
|
May 26 04:20:44 labsdb1011 mysqld[19806]: InnoDB: tuple DATA TUPLE: 4 fields;
|
May 26 04:20:44 labsdb1011 mysqld[19806]: 0: len 4; hex 80000004; asc ;;
|
May 26 04:20:44 labsdb1011 mysqld[19806]: 1: len 4; hex 80000000; asc ;;
|
May 26 04:20:44 labsdb1011 mysqld[19806]: 2: len 22; hex 5468655f47797073795f616e645f7468655f4b696e67; asc The_Gypsy_and_the_King;;
|
May 26 04:20:44 labsdb1011 mysqld[19806]: 3: len 4; hex 00109a35; asc 5;;
|
May 26 04:20:44 labsdb1011 mysqld[19806]: InnoDB: record PHYSICAL RECORD: n_fields 4; compact format; info bits 0
|
May 26 04:20:44 labsdb1011 mysqld[19806]: 0: len 4; hex 80000004; asc ;;
|
May 26 04:20:44 labsdb1011 mysqld[19806]: 1: len 4; hex 80000000; asc ;;
|
May 26 04:20:44 labsdb1011 mysqld[19806]: 2: len 15; hex 5468655f47797073795f547261696c; asc The_Gypsy_Trail;;
|
May 26 04:20:44 labsdb1011 mysqld[19806]: 3: len 4; hex 02e4fc4d; asc M;;
|
May 26 04:20:44 labsdb1011 mysqld[19806]: 2020-05-26 4:20:44 4 [ERROR] InnoDB: page [page id: space=94970, page number=4807528] (305 records, index id 334204).
|
May 26 04:20:44 labsdb1011 mysqld[19806]: 2020-05-26 4:20:44 4 [ERROR] InnoDB: Submit a detailed bug report to https://jira.mariadb.org/
|
May 26 04:20:45 labsdb1011 mysqld[19806]: 2020-05-26 4:20:45 4 [ERROR] InnoDB: Unable to find a record to delete-mark
|
May 26 04:20:45 labsdb1011 mysqld[19806]: InnoDB: tuple DATA TUPLE: 3 fields;
|
May 26 04:20:45 labsdb1011 mysqld[19806]: 0: len 4; hex 80000000; asc ;;
|
May 26 04:20:45 labsdb1011 mysqld[19806]: 1: len 19; hex 484d535f5265736f6c7574655f283138353029; asc HMS_Resolute_(1850);;
|
May 26 04:20:45 labsdb1011 mysqld[19806]: 2: len 4; hex 02ea5d4b; asc ]K;;
|
May 26 04:20:45 labsdb1011 mysqld[19806]: InnoDB: record PHYSICAL RECORD: n_fields 3; compact format; info bits 0
|
May 26 04:20:45 labsdb1011 mysqld[19806]: 0: len 4; hex 80000000; asc ;;
|
May 26 04:20:45 labsdb1011 mysqld[19806]: 1: len 19; hex 484d535f5265736f6c7574655f283138353029; asc HMS_Resolute_(1850);;
|
May 26 04:20:45 labsdb1011 mysqld[19806]: 2: len 4; hex 02e860c8; asc ` ;;
|
May 26 04:20:45 labsdb1011 mysqld[19806]: 2020-05-26 4:20:45 4 [ERROR] InnoDB: page [page id: space=94970, page number=1579497] (421 records, index id 334203).
|
May 26 04:20:45 labsdb1011 mysqld[19806]: 2020-05-26 4:20:45 4 [ERROR] InnoDB: Submit a detailed bug report to https://jira.mariadb.org/
|
May 26 04:20:47 labsdb1011 mysqld[19806]: 2020-05-26 4:20:47 46 [ERROR] Master 's1': InnoDB: Record in index `pl_backlinks_namespace` of table `enwiki`.`pagelinks` was not found on update: TUPLE (info_bits=0, 4 fields): {[4] (0x80000004),[4] (0x80000000),[22]The_Gypsy_and_the_King(0x5468655F47797073795F616E645F7468655F4B696E67),[4] e (0x000665A4)} at: COMPACT RECORD(info_bits=0, 4 fields): {[4] (0x80000004),[4] (0x80000000),[15]The_Gypsy_Trail(0x5468655F47797073795F547261696C),[4] M(0x02E4FC4D)}
|
May 26 04:20:53 labsdb1011 mysqld[19806]: 2020-05-26 4:20:53 3 [ERROR] InnoDB: Unable to find a record to delete-mark
|
May 26 04:20:53 labsdb1011 mysqld[19806]: InnoDB: tuple DATA TUPLE: 3 fields;
|
|
|
May 26 04:21:51 labsdb1011 mysqld[19806]: InnoDB: record PHYSICAL RECORD: n_fields 4; compact format; info bits 0
|
May 26 04:21:51 labsdb1011 mysqld[19806]: 0: len 4; hex 80000002; asc ;;
|
May 26 04:21:51 labsdb1011 mysqld[19806]: 1: len 4; hex 80000000; asc ;;
|
May 26 04:21:51 labsdb1011 mysqld[19806]: 2: len 10; hex 43616c696d6572697573; asc Calimerius;;
|
May 26 04:21:51 labsdb1011 mysqld[19806]: 3: len 4; hex 00dac0fe; asc ;;
|
May 26 04:21:51 labsdb1011 mysqld[19806]: 2020-05-26 4:21:51 4 [ERROR] InnoDB: page [page id: space=94970, page number=10530772] (382 records, index id 334204).
|
May 26 04:21:51 labsdb1011 mysqld[19806]: 2020-05-26 4:21:51 4 [ERROR] InnoDB: Submit a detailed bug report to https://jira.mariadb.org/
|
May 26 04:22:01 labsdb1011 mysqld[19806]: 2020-05-26 4:22:01 46 [ERROR] Master 's1': InnoDB: Record in index `pl_backlinks_namespace` of table `enwiki`.`pagelinks` was not found on update: TUPLE (info_bits=0, 4 fields): {[4] (0x80000004),[4] (0x80000000),[22]The_Gypsy_and_the_King(0x5468655F47797073795F616E645F7468655F4B696E67),[4] (0x00100CFD)} at: COMPACT RECORD(info_bits=0, 4 fields): {[4] (0x80000004),[4] (0x80000000),[15]The_Gypsy_Trail(0x5468655F47797073795F547261696C),[4] M(0x02E4FC4D)}
|
Global variables:
root@labsdb1011:~# mysql -e "show global variables" -BN
|
alter_algorithm DEFAULT
|
analyze_sample_percentage 100.000000
|
aria_block_size 8192
|
aria_checkpoint_interval 30
|
aria_checkpoint_log_activity 1048576
|
aria_encrypt_tables OFF
|
aria_force_start_after_recovery_failures 0
|
aria_group_commit none
|
aria_group_commit_interval 0
|
aria_log_file_size 1073741824
|
aria_log_purge_type immediate
|
aria_max_sort_file_size 9223372036853727232
|
aria_page_checksum ON
|
aria_pagecache_age_threshold 300
|
aria_pagecache_buffer_size 134217728
|
aria_pagecache_division_limit 100
|
aria_pagecache_file_hash_size 512
|
aria_recover_options BACKUP,QUICK
|
aria_repair_threads 1
|
aria_sort_buffer_size 268434432
|
aria_stats_method nulls_unequal
|
aria_sync_log_dir NEWFILE
|
aria_used_for_temp_tables ON
|
auto_increment_increment 1
|
auto_increment_offset 1
|
autocommit ON
|
automatic_sp_privileges ON
|
back_log 254
|
basedir /opt/wmf-mariadb104/
|
big_tables OFF
|
bind_address
|
binlog_annotate_row_events ON
|
binlog_cache_size 32768
|
binlog_checksum CRC32
|
binlog_commit_wait_count 0
|
binlog_commit_wait_usec 100000
|
binlog_direct_non_transactional_updates OFF
|
binlog_file_cache_size 16384
|
binlog_format MIXED
|
binlog_optimize_thread_scheduling ON
|
binlog_row_image FULL
|
binlog_stmt_cache_size 32768
|
bulk_insert_buffer_size 8388608
|
character_set_client binary
|
character_set_connection binary
|
character_set_database binary
|
character_set_filesystem binary
|
character_set_results binary
|
character_set_server binary
|
character_set_system utf8
|
character_sets_dir /opt/wmf-mariadb104/share/charsets/
|
check_constraint_checks ON
|
collation_connection binary
|
collation_database binary
|
collation_server binary
|
column_compression_threshold 100
|
column_compression_zlib_level 6
|
column_compression_zlib_strategy DEFAULT_STRATEGY
|
column_compression_zlib_wrap OFF
|
completion_type NO_CHAIN
|
concurrent_insert AUTO
|
connect_timeout 5
|
core_file OFF
|
datadir /srv/sqldata/
|
date_format %Y-%m-%d
|
datetime_format %Y-%m-%d %H:%i:%s
|
deadlock_search_depth_long 15
|
deadlock_search_depth_short 4
|
deadlock_timeout_long 50000000
|
deadlock_timeout_short 10000
|
debug_no_thread_alarm OFF
|
default_password_lifetime 0
|
default_regex_flags
|
default_storage_engine InnoDB
|
default_tmp_storage_engine
|
default_week_format 0
|
delay_key_write ON
|
delayed_insert_limit 100
|
delayed_insert_timeout 300
|
delayed_queue_size 1000
|
disconnect_on_expired_password OFF
|
div_precision_increment 4
|
encrypt_binlog OFF
|
encrypt_tmp_disk_tables OFF
|
encrypt_tmp_files OFF
|
enforce_storage_engine
|
eq_range_index_dive_limit 200
|
event_scheduler ON
|
expensive_subquery_limit 100
|
expire_logs_days 0
|
explicit_defaults_for_timestamp OFF
|
extra_max_connections 1
|
extra_port 0
|
flush OFF
|
flush_time 0
|
foreign_key_checks ON
|
ft_boolean_syntax + -><()~*:""&|
|
ft_max_word_len 84
|
ft_min_word_len 4
|
ft_query_expansion_limit 20
|
ft_stopword_file (built-in)
|
general_log OFF
|
general_log_file labsdb1011.log
|
group_concat_max_len 1048576
|
gtid_binlog_pos
|
gtid_binlog_state
|
gtid_cleanup_batch_size 64
|
gtid_current_pos 0-171970637-5484646134,171966555-171966555-1275,171966557-171966557-578966402,171966558-171966558-189,171966574-171966574-2221092918,171966668-171966668-2920,171966669-171966669-4196523483,171966670-171966670-2410812544,171970567-171970567-19776,171970577-171970577-357260,171970589-171970589-201132050,171970593-171970593-3479,171970594-171970594-839408486,171970599-171970599-24483,171970637-171970637-2116621969,171970645-171970645-288070551,171970661-171970661-1415167773,171970663-171970663-274,171970664-171970664-1099438288,171970751-171970751-58940,171974668-171974668-1523,171974686-171974686-867,171974720-171974720-2572451842,171974769-171974769-2185928,171974784-171974784-43861836,171974792-171974792-378345284,171974853-171974853-690385053,171974883-171974883-1921892293,171974884-171974884-1473084269,171978764-171978764-57,171978765-171978765-106,171978766-171978766-117878,171978767-171978767-4484858466,171978771-171978771-30,171978772-171978772-1562,171978775-171978775-4822899280,171978777-171978777-2596176831,171978778-171978778-3298185533,171978786-171978786-1568165471,171978787-171978787-1393868950,171978876-171978876-1953943060,171978924-171978924-2600311186,180355078-180355078-26706647,180355111-180355111-131673159,180355171-180355171-148310907,180355173-180355173-62400,180359172-180359172-49702202,180359174-180359174-94123432,180359175-180359175-43143522,180359190-180359190-192195477,180359207-180359207-2,180359241-180359241-121693516,180359242-180359242-170963125,180359271-180359271-6045596,180363367-180363367-133158799,180367364-180367364-74755871,180367474-180367474-91976046
|
gtid_domain_id 171975960
|
gtid_ignore_duplicates OFF
|
gtid_pos_auto_engines
|
gtid_slave_pos 0-171970637-5484646134,171966555-171966555-1275,171966557-171966557-578966402,171966558-171966558-189,171966574-171966574-2221092918,171966668-171966668-2920,171966669-171966669-4196523483,171966670-171966670-2410812544,171970567-171970567-19776,171970577-171970577-357260,171970589-171970589-201132050,171970593-171970593-3479,171970594-171970594-839408486,171970599-171970599-24483,171970637-171970637-2116621969,171970645-171970645-288070551,171970661-171970661-1415167774,171970663-171970663-274,171970664-171970664-1099438288,171970751-171970751-58940,171974668-171974668-1523,171974686-171974686-867,171974720-171974720-2572451842,171974769-171974769-2185928,171974784-171974784-43861836,171974792-171974792-378345284,171974853-171974853-690385053,171974883-171974883-1921892293,171974884-171974884-1473084269,171978764-171978764-57,171978765-171978765-106,171978766-171978766-117878,171978767-171978767-4484858466,171978771-171978771-30,171978772-171978772-1562,171978775-171978775-4822899280,171978777-171978777-2596176831,171978778-171978778-3298185533,171978786-171978786-1568165471,171978787-171978787-1393868950,171978876-171978876-1953943060,171978924-171978924-2600311186,180355078-180355078-26706647,180355111-180355111-131673159,180355171-180355171-148310907,180355173-180355173-62400,180359172-180359172-49702202,180359174-180359174-94123432,180359175-180359175-43143522,180359190-180359190-192195477,180359207-180359207-2,180359241-180359241-121693516,180359242-180359242-170963125,180359271-180359271-6045596,180363367-180363367-133158799,180367364-180367364-74755871,180367474-180367474-91976046
|
gtid_strict_mode OFF
|
have_compress YES
|
have_crypt YES
|
have_dynamic_loading YES
|
have_geometry YES
|
have_openssl YES
|
have_profiling YES
|
have_query_cache YES
|
have_rtree_keys YES
|
have_ssl YES
|
have_symlink YES
|
histogram_size 254
|
histogram_type DOUBLE_PREC_HB
|
host_cache_size 654
|
hostname labsdb1011
|
idle_readonly_transaction_timeout 0
|
idle_transaction_timeout 0
|
idle_write_transaction_timeout 0
|
ignore_builtin_innodb OFF
|
ignore_db_dirs
|
in_predicate_conversion_threshold 1000
|
init_connect
|
init_file
|
init_slave
|
innodb_adaptive_flushing ON
|
innodb_adaptive_flushing_lwm 10.000000
|
innodb_adaptive_hash_index ON
|
innodb_adaptive_hash_index_parts 8
|
innodb_adaptive_max_sleep_delay 150000
|
innodb_autoextend_increment 64
|
innodb_autoinc_lock_mode 1
|
innodb_background_scrub_data_check_interval 3600
|
innodb_background_scrub_data_compressed OFF
|
innodb_background_scrub_data_interval 604800
|
innodb_background_scrub_data_uncompressed OFF
|
innodb_buf_dump_status_frequency 0
|
innodb_buffer_pool_chunk_size 134217728
|
innodb_buffer_pool_dump_at_shutdown ON
|
innodb_buffer_pool_dump_now OFF
|
innodb_buffer_pool_dump_pct 25
|
innodb_buffer_pool_filename ib_buffer_pool
|
innodb_buffer_pool_instances 1
|
innodb_buffer_pool_load_abort OFF
|
innodb_buffer_pool_load_at_startup ON
|
innodb_buffer_pool_load_now OFF
|
innodb_buffer_pool_size 432717955072
|
innodb_change_buffer_max_size 25
|
innodb_change_buffering all
|
innodb_checksum_algorithm crc32
|
innodb_checksums ON
|
innodb_cmp_per_index_enabled OFF
|
innodb_commit_concurrency 0
|
innodb_compression_algorithm zlib
|
innodb_compression_default OFF
|
innodb_compression_failure_threshold_pct 5
|
innodb_compression_level 6
|
innodb_compression_pad_pct_max 50
|
innodb_concurrency_tickets 5000
|
innodb_data_file_path ibdata1:12M:autoextend
|
innodb_data_home_dir
|
innodb_deadlock_detect ON
|
innodb_default_encryption_key_id 1
|
innodb_default_row_format dynamic
|
innodb_defragment OFF
|
innodb_defragment_fill_factor 0.900000
|
innodb_defragment_fill_factor_n_recs 20
|
innodb_defragment_frequency 40
|
innodb_defragment_n_pages 7
|
innodb_defragment_stats_accuracy 0
|
innodb_disable_sort_file_cache OFF
|
innodb_disallow_writes OFF
|
innodb_doublewrite ON
|
innodb_encrypt_log OFF
|
innodb_encrypt_tables OFF
|
innodb_encrypt_temporary_tables OFF
|
innodb_encryption_rotate_key_age 1
|
innodb_encryption_rotation_iops 100
|
innodb_encryption_threads 0
|
innodb_fast_shutdown 1
|
innodb_fatal_semaphore_wait_threshold 600
|
innodb_file_format barracuda
|
innodb_file_per_table ON
|
innodb_fill_factor 100
|
innodb_flush_log_at_timeout 1
|
innodb_flush_log_at_trx_commit 0
|
innodb_flush_method fsync
|
innodb_flush_neighbors 1
|
innodb_flush_sync ON
|
innodb_flushing_avg_loops 30
|
innodb_force_load_corrupted OFF
|
innodb_force_primary_key OFF
|
innodb_force_recovery 0
|
innodb_ft_aux_table
|
innodb_ft_cache_size 8000000
|
innodb_ft_enable_diag_print OFF
|
innodb_ft_enable_stopword ON
|
innodb_ft_max_token_size 84
|
innodb_ft_min_token_size 3
|
innodb_ft_num_word_optimize 2000
|
innodb_ft_result_cache_limit 2000000000
|
innodb_ft_server_stopword_table
|
innodb_ft_sort_pll_degree 2
|
innodb_ft_total_cache_size 640000000
|
innodb_ft_user_stopword_table
|
innodb_idle_flush_pct 100
|
innodb_immediate_scrub_data_uncompressed OFF
|
innodb_instant_alter_column_allowed add_drop_reorder
|
innodb_io_capacity 5000
|
innodb_io_capacity_max 10000
|
innodb_large_prefix
|
innodb_lock_schedule_algorithm fcfs
|
innodb_lock_wait_timeout 50
|
innodb_locks_unsafe_for_binlog OFF
|
innodb_log_buffer_size 16777216
|
innodb_log_checksums ON
|
innodb_log_compressed_pages ON
|
innodb_log_file_size 2147483648
|
innodb_log_files_in_group 2
|
innodb_log_group_home_dir ./
|
innodb_log_optimize_ddl ON
|
innodb_log_write_ahead_size 8192
|
innodb_lru_scan_depth 1024
|
innodb_max_dirty_pages_pct 75.000000
|
innodb_max_dirty_pages_pct_lwm 0.000000
|
innodb_max_purge_lag 0
|
innodb_max_purge_lag_delay 0
|
innodb_max_undo_log_size 10485760
|
innodb_monitor_disable
|
innodb_monitor_enable
|
innodb_monitor_reset
|
innodb_monitor_reset_all
|
innodb_old_blocks_pct 37
|
innodb_old_blocks_time 1000
|
innodb_online_alter_log_max_size 134217728
|
innodb_open_files 5000
|
innodb_optimize_fulltext_only OFF
|
innodb_page_cleaners 1
|
innodb_page_size 16384
|
innodb_prefix_index_cluster_optimization OFF
|
innodb_print_all_deadlocks OFF
|
innodb_purge_batch_size 300
|
innodb_purge_rseg_truncate_frequency 128
|
innodb_purge_threads 4
|
innodb_random_read_ahead OFF
|
innodb_read_ahead_threshold 56
|
innodb_read_io_threads 4
|
innodb_read_only OFF
|
innodb_replication_delay 0
|
innodb_rollback_on_timeout OFF
|
innodb_rollback_segments 128
|
innodb_scrub_log OFF
|
innodb_scrub_log_speed 256
|
innodb_sort_buffer_size 1048576
|
innodb_spin_wait_delay 4
|
innodb_stats_auto_recalc ON
|
innodb_stats_include_delete_marked OFF
|
innodb_stats_method nulls_unequal
|
innodb_stats_modified_counter 0
|
innodb_stats_on_metadata OFF
|
innodb_stats_persistent ON
|
innodb_stats_persistent_sample_pages 20
|
innodb_stats_sample_pages 16
|
innodb_stats_traditional ON
|
innodb_stats_transient_sample_pages 16
|
innodb_status_output OFF
|
innodb_status_output_locks OFF
|
innodb_strict_mode ON
|
innodb_sync_array_size 1
|
innodb_sync_spin_loops 30
|
innodb_table_locks ON
|
innodb_temp_data_file_path ibtmp1:12M:autoextend
|
innodb_thread_concurrency 0
|
innodb_thread_sleep_delay 10000
|
innodb_tmpdir
|
innodb_undo_directory ./
|
innodb_undo_log_truncate OFF
|
innodb_undo_logs 128
|
innodb_undo_tablespaces 0
|
innodb_use_atomic_writes ON
|
innodb_use_native_aio ON
|
innodb_version 10.4.13
|
innodb_write_io_threads 4
|
interactive_timeout 600
|
join_buffer_size 262144
|
join_buffer_space_limit 2097152
|
join_cache_level 2
|
keep_files_on_create OFF
|
key_buffer_size 134217728
|
key_cache_age_threshold 300
|
key_cache_block_size 1024
|
key_cache_division_limit 100
|
key_cache_file_hash_size 512
|
key_cache_segments 0
|
large_files_support ON
|
large_page_size 0
|
large_pages OFF
|
lc_messages en_US
|
lc_messages_dir
|
lc_time_names en_US
|
license GPL
|
local_infile ON
|
lock_wait_timeout 86400
|
locked_in_memory OFF
|
log_bin OFF
|
log_bin_basename
|
log_bin_compress OFF
|
log_bin_compress_min_len 256
|
log_bin_index
|
log_bin_trust_function_creators OFF
|
log_disabled_statements sp
|
log_error
|
log_output FILE
|
log_queries_not_using_indexes OFF
|
log_slave_updates OFF
|
log_slow_admin_statements ON
|
log_slow_disabled_statements sp
|
log_slow_filter admin,filesort,filesort_on_disk,filesort_priority_queue,full_join,full_scan,query_cache,query_cache_miss,tmp_table,tmp_table_on_disk
|
log_slow_rate_limit 1
|
log_slow_slave_statements ON
|
log_slow_verbosity
|
log_tc_size 24576
|
log_warnings 0
|
long_query_time 10.000000
|
low_priority_updates OFF
|
lower_case_file_system OFF
|
lower_case_table_names 0
|
master_verify_checksum OFF
|
max_allowed_packet 33554432
|
max_binlog_cache_size 18446744073709547520
|
max_binlog_size 1073741824
|
max_binlog_stmt_cache_size 18446744073709547520
|
max_connect_errors 1000000000
|
max_connections 1024
|
max_delayed_threads 20
|
max_digest_length 1024
|
max_error_count 64
|
max_heap_table_size 16777216
|
max_insert_delayed_threads 20
|
max_join_size 18446744073709551615
|
max_length_for_sort_data 1024
|
max_long_data_size 33554432
|
max_password_errors 4294967295
|
max_prepared_stmt_count 16382
|
max_recursive_iterations 4294967295
|
max_relay_log_size 1073741824
|
max_rowid_filter_size 131072
|
max_seeks_for_key 4294967295
|
max_session_mem_used 9223372036854775807
|
max_sort_length 1024
|
max_sp_recursion_depth 0
|
max_statement_time 0.000000
|
max_tmp_tables 32
|
max_user_connections 0
|
max_write_lock_count 4294967295
|
metadata_locks_cache_size 1024
|
metadata_locks_hash_instances 8
|
min_examined_row_limit 0
|
mrr_buffer_size 262144
|
multi_range_count 256
|
myisam_block_size 1024
|
myisam_data_pointer_size 6
|
myisam_max_sort_file_size 9223372036853727232
|
myisam_mmap_size 18446744073709551615
|
myisam_recover_options BACKUP,QUICK
|
myisam_repair_threads 1
|
myisam_sort_buffer_size 134216704
|
myisam_stats_method NULLS_UNEQUAL
|
myisam_use_mmap OFF
|
mysql56_temporal_format ON
|
net_buffer_length 16384
|
net_read_timeout 30
|
net_retry_count 10
|
net_write_timeout 60
|
old OFF
|
old_alter_table DEFAULT
|
old_mode
|
old_passwords OFF
|
open_files_limit 81071
|
optimizer_prune_level 1
|
optimizer_search_depth 62
|
optimizer_selectivity_sampling_limit 100
|
optimizer_switch index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=on,condition_pushdown_from_having=on
|
optimizer_trace enabled=off
|
optimizer_trace_max_mem_size 1048576
|
optimizer_use_condition_selectivity 4
|
performance_schema ON
|
performance_schema_accounts_size 300
|
performance_schema_digests_size 10000
|
performance_schema_events_stages_history_long_size 1000
|
performance_schema_events_stages_history_size 10
|
performance_schema_events_statements_history_long_size 1000
|
performance_schema_events_statements_history_size 10
|
performance_schema_events_waits_history_long_size 1000
|
performance_schema_events_waits_history_size 10
|
performance_schema_hosts_size 300
|
performance_schema_max_cond_classes 90
|
performance_schema_max_cond_instances 1000
|
performance_schema_max_digest_length 1024
|
performance_schema_max_file_classes 50
|
performance_schema_max_file_handles 32768
|
performance_schema_max_file_instances 3077
|
performance_schema_max_mutex_classes 200
|
performance_schema_max_mutex_instances 5000
|
performance_schema_max_rwlock_classes 40
|
performance_schema_max_rwlock_instances 2000
|
performance_schema_max_socket_classes 10
|
performance_schema_max_socket_instances 500
|
performance_schema_max_stage_classes 160
|
performance_schema_max_statement_classes 202
|
performance_schema_max_table_handles 4000
|
performance_schema_max_table_instances 1000
|
performance_schema_max_thread_classes 50
|
performance_schema_max_thread_instances 500
|
performance_schema_session_connect_attrs_size 512
|
performance_schema_setup_actors_size 100
|
performance_schema_setup_objects_size 100
|
performance_schema_users_size 100
|
pid_file /srv/sqldata/labsdb1011.pid
|
plugin_dir /opt/wmf-mariadb104/lib/plugin/
|
plugin_maturity gamma
|
port 3306
|
preload_buffer_size 32768
|
profiling OFF
|
profiling_history_size 15
|
progress_report_time 5
|
protocol_version 10
|
proxy_protocol_networks
|
query_alloc_block_size 16384
|
query_cache_limit 1048576
|
query_cache_min_res_unit 4096
|
query_cache_size 0
|
query_cache_strip_comments OFF
|
query_cache_type OFF
|
query_cache_wlock_invalidate OFF
|
query_prealloc_size 24576
|
range_alloc_block_size 4096
|
read_binlog_speed_limit 0
|
read_buffer_size 131072
|
read_only ON
|
read_rnd_buffer_size 262144
|
relay_log
|
relay_log_basename
|
relay_log_index
|
relay_log_info_file relay-log.info
|
relay_log_purge ON
|
relay_log_recovery OFF
|
relay_log_space_limit 0
|
replicate_annotate_row_events ON
|
replicate_do_db
|
replicate_do_table
|
replicate_events_marked_for_skip REPLICATE
|
replicate_ignore_db
|
replicate_ignore_table
|
replicate_wild_do_table
|
replicate_wild_ignore_table
|
report_host
|
report_password
|
report_port 3306
|
report_user
|
rowid_merge_buff_size 8388608
|
rpl_semi_sync_master_enabled OFF
|
rpl_semi_sync_master_timeout 10000
|
rpl_semi_sync_master_trace_level 32
|
rpl_semi_sync_master_wait_no_slave ON
|
rpl_semi_sync_master_wait_point AFTER_COMMIT
|
rpl_semi_sync_slave_delay_master OFF
|
rpl_semi_sync_slave_enabled OFF
|
rpl_semi_sync_slave_kill_conn_timeout 5
|
rpl_semi_sync_slave_trace_level 32
|
secure_auth ON
|
secure_file_priv /dev/null/
|
secure_timestamp NO
|
server_id 171975960
|
session_track_schema ON
|
session_track_state_change OFF
|
session_track_system_variables autocommit,character_set_client,character_set_connection,character_set_results,time_zone
|
session_track_transaction_info OFF
|
skip_external_locking ON
|
skip_name_resolve ON
|
skip_networking OFF
|
skip_show_database OFF
|
slave_compressed_protocol OFF
|
slave_ddl_exec_mode IDEMPOTENT
|
slave_domain_parallel_threads 0
|
slave_exec_mode STRICT
|
slave_load_tmpdir /srv/tmp
|
slave_max_allowed_packet 1073741824
|
slave_net_timeout 60
|
slave_parallel_max_queued 16777216
|
slave_parallel_mode none
|
slave_parallel_threads 0
|
slave_parallel_workers 0
|
slave_run_triggers_for_rbr NO
|
slave_skip_errors OFF
|
slave_sql_verify_checksum ON
|
slave_transaction_retries 4294967295
|
slave_transaction_retry_errors 1158,1159,1160,1161,1205,1213,1429,2013,12701
|
slave_transaction_retry_interval 0
|
slave_type_conversions ALL_NON_LOSSY
|
slow_launch_time 2
|
slow_query_log OFF
|
slow_query_log_file labsdb1011-slow.log
|
socket /run/mysqld/mysqld.sock
|
sort_buffer_size 2097152
|
sql_auto_is_null OFF
|
sql_big_selects ON
|
sql_buffer_result OFF
|
sql_log_bin ON
|
sql_log_off OFF
|
sql_mode STRICT_TRANS_TABLES,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION
|
sql_notes ON
|
sql_quote_show_create ON
|
sql_safe_updates OFF
|
sql_select_limit 18446744073709551615
|
sql_slave_skip_counter 0
|
sql_warnings OFF
|
ssl_ca /etc/ssl/certs/Puppet_Internal_CA.pem
|
ssl_capath
|
ssl_cert /etc/mysql/ssl/cert.pem
|
ssl_cipher TLSv1.2
|
ssl_crl
|
ssl_crlpath
|
ssl_key /etc/mysql/ssl/server.key
|
standard_compliant_cte ON
|
storage_engine InnoDB
|
stored_program_cache 256
|
strict_password_validation ON
|
sync_binlog 0
|
sync_frm ON
|
sync_master_info 10000
|
sync_relay_log 10000
|
sync_relay_log_info 10000
|
system_time_zone UTC
|
system_versioning_alter_history ERROR
|
system_versioning_asof DEFAULT
|
table_definition_cache 50000
|
table_open_cache 5000
|
table_open_cache_instances 8
|
tcp_keepalive_interval 0
|
tcp_keepalive_probes 0
|
tcp_keepalive_time 0
|
tcp_nodelay ON
|
thread_cache_size 128
|
thread_concurrency 10
|
thread_handling one-thread-per-connection
|
thread_pool_idle_timeout 60
|
thread_pool_max_threads 65536
|
thread_pool_oversubscribe 3
|
thread_pool_prio_kickup_timer 1000
|
thread_pool_priority auto
|
thread_pool_size 16
|
thread_pool_stall_limit 500
|
thread_stack 196608
|
time_format %H:%i:%s
|
time_zone SYSTEM
|
timed_mutexes OFF
|
tls_version TLSv1.1,TLSv1.2,TLSv1.3
|
tmp_disk_table_size 18446744073709551615
|
tmp_memory_table_size 16777216
|
tmp_table_size 16777216
|
tmpdir /srv/tmp
|
transaction_alloc_block_size 8192
|
transaction_prealloc_size 4096
|
tx_isolation REPEATABLE-READ
|
tx_read_only OFF
|
unique_checks ON
|
updatable_views_with_limit YES
|
use_stat_tables PREFERABLY_FOR_QUERIES
|
userstat OFF
|
version 10.4.13-MariaDB
|
version_comment MariaDB Server
|
version_compile_machine x86_64
|
version_compile_os Linux
|
version_malloc_library system
|
version_source_revision 1b18cddaa23711776537ee98f16529a74ff861c2
|
version_ssl_library OpenSSL 1.1.1d 10 Sep 2019
|
wait_timeout 300
|
wsrep_osu_method TOI
|
wsrep_sr_store table
|
wsrep_auto_increment_control ON
|
wsrep_causal_reads OFF
|
wsrep_certification_rules strict
|
wsrep_certify_nonpk ON
|
wsrep_cluster_address
|
wsrep_cluster_name my_wsrep_cluster
|
wsrep_convert_lock_to_trx OFF
|
wsrep_data_home_dir /srv/sqldata/
|
wsrep_dbug_option
|
wsrep_debug NONE
|
wsrep_desync OFF
|
wsrep_dirty_reads OFF
|
wsrep_drupal_282555_workaround OFF
|
wsrep_forced_binlog_format NONE
|
wsrep_gtid_domain_id 0
|
wsrep_gtid_mode OFF
|
wsrep_ignore_apply_errors 7
|
wsrep_load_data_splitting OFF
|
wsrep_log_conflicts OFF
|
wsrep_max_ws_rows 0
|
wsrep_max_ws_size 2147483647
|
wsrep_mysql_replication_bundle 0
|
wsrep_node_address
|
wsrep_node_incoming_address AUTO
|
wsrep_node_name labsdb1011
|
wsrep_notify_cmd
|
wsrep_on OFF
|
wsrep_patch_version wsrep_26.22
|
wsrep_provider none
|
wsrep_provider_options
|
wsrep_recover OFF
|
wsrep_reject_queries NONE
|
wsrep_replicate_myisam OFF
|
wsrep_restart_slave OFF
|
wsrep_retry_autocommit 1
|
wsrep_slave_fk_checks ON
|
wsrep_slave_uk_checks OFF
|
wsrep_slave_threads 1
|
wsrep_sst_auth
|
wsrep_sst_donor
|
wsrep_sst_donor_rejects_queries OFF
|
wsrep_sst_method rsync
|
wsrep_sst_receive_address AUTO
|
wsrep_start_position 00000000-0000-0000-0000-000000000000:-1
|
wsrep_sync_wait 0
|
wsrep_trx_fragment_size 0
|
wsrep_trx_fragment_unit bytes
|
And this ends up crashing the server
May 27 02:11:02 labsdb1011 mysqld[23527]: 2020-05-27 02:11:02 0x7fc65c207700 InnoDB: Assertion failure in file /root/mariadb-10.4.13/storage/innobase/row/row0ins.cc line 231
|
May 27 02:11:02 labsdb1011 mysqld[23527]: InnoDB: Failing assertion: !cursor->index->is_committed()
|
May 27 02:11:02 labsdb1011 mysqld[23527]: InnoDB: We intentionally generate a memory trap.
|
May 27 02:11:02 labsdb1011 mysqld[23527]: InnoDB: Submit a detailed bug report to https://jira.mariadb.org/
|
May 27 02:11:02 labsdb1011 mysqld[23527]: InnoDB: If you get repeated assertion failures or crashes, even
|
May 27 02:11:02 labsdb1011 mysqld[23527]: InnoDB: immediately after the mysqld startup, there may be
|
May 27 02:11:02 labsdb1011 mysqld[23527]: InnoDB: corruption in the InnoDB tablespace. Please refer to
|
May 27 02:11:02 labsdb1011 mysqld[23527]: InnoDB: https://mariadb.com/kb/en/library/innodb-recovery-modes/
|
May 27 02:11:02 labsdb1011 mysqld[23527]: InnoDB: about forcing recovery.
|
May 27 02:11:02 labsdb1011 mysqld[23527]: 200527 2:11:02 [ERROR] mysqld got signal 6 ;
|
May 27 02:11:02 labsdb1011 mysqld[23527]: This could be because you hit a bug. It is also possible that this binary
|
May 27 02:11:02 labsdb1011 mysqld[23527]: or one of the libraries it was linked against is corrupt, improperly built,
|
May 27 02:11:02 labsdb1011 mysqld[23527]: or misconfigured. This error can also be caused by malfunctioning hardware.
|
May 27 02:11:02 labsdb1011 mysqld[23527]: To report this bug, see https://mariadb.com/kb/en/reporting-bugs
|
May 27 02:11:02 labsdb1011 mysqld[23527]: We will try our best to scrape up some info that will hopefully help
|
May 27 02:11:02 labsdb1011 mysqld[23527]: diagnose the problem, but since we have already crashed,
|
May 27 02:11:02 labsdb1011 mysqld[23527]: something is definitely wrong and this may fail.
|
May 27 02:11:02 labsdb1011 mysqld[23527]: Server version: 10.4.13-MariaDB
|
May 27 02:11:02 labsdb1011 mysqld[23527]: key_buffer_size=134217728
|
May 27 02:11:02 labsdb1011 mysqld[23527]: read_buffer_size=131072
|
May 27 02:11:02 labsdb1011 mysqld[23527]: max_used_connections=125
|
May 27 02:11:02 labsdb1011 mysqld[23527]: max_threads=1026
|
May 27 02:11:02 labsdb1011 mysqld[23527]: thread_count=148
|
May 27 02:11:02 labsdb1011 mysqld[23527]: It is possible that mysqld could use up to
|
May 27 02:11:02 labsdb1011 mysqld[23527]: key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 2388976 K bytes of memory
|
May 27 02:11:02 labsdb1011 mysqld[23527]: Hope that's ok; if not, decrease some variables in the equation.
|
May 27 02:11:02 labsdb1011 mysqld[23527]: Thread pointer: 0x7f5f1c0014f8
|
May 27 02:11:02 labsdb1011 mysqld[23527]: Attempting backtrace. You can use the following information to find out
|
May 27 02:11:02 labsdb1011 mysqld[23527]: where mysqld died. If you see no messages after this, something went
|
May 27 02:11:02 labsdb1011 mysqld[23527]: terribly wrong...
|
May 27 02:11:02 labsdb1011 mysqld[23527]: stack_bottom = 0x7fc65c206698 thread_stack 0x30000
|
May 27 02:11:04 labsdb1011 mysqld[23527]: /opt/wmf-mariadb104/bin/mysqld(my_print_stacktrace+0x2e)[0x556d7c07f7de]
|
May 27 02:11:06 labsdb1011 mysqld[23527]: /opt/wmf-mariadb104/bin/mysqld(handle_fatal_signal+0x54d)[0x556d7bb77b4d]
|
May 27 02:11:08 labsdb1011 mysqld[23527]: /lib/x86_64-linux-gnu/libpthread.so.0(+0x12730)[0x7fc67eb8e730]
|
May 27 02:11:10 labsdb1011 mysqld[23527]: /lib/x86_64-linux-gnu/libc.so.6(gsignal+0x10b)[0x7fc67e1f67bb]
|
May 27 02:11:11 labsdb1011 mysqld[23527]: /lib/x86_64-linux-gnu/libc.so.6(abort+0x121)[0x7fc67e1e1535]
|
May 27 02:11:13 labsdb1011 mysqld[23527]: /opt/wmf-mariadb104/bin/mysqld(+0x5a19d5)[0x556d7b8739d5]
|
May 27 02:11:15 labsdb1011 mysqld[23527]: /opt/wmf-mariadb104/bin/mysqld(+0x5907c1)[0x556d7b8627c1]
|
May 27 02:11:17 labsdb1011 mysqld[23527]: /opt/wmf-mariadb104/bin/mysqld(+0xaeee0e)[0x556d7bdc0e0e]
|
May 27 02:11:19 labsdb1011 mysqld[23527]: /opt/wmf-mariadb104/bin/mysqld(+0xaef4bd)[0x556d7bdc14bd]
|
May 27 02:11:21 labsdb1011 mysqld[23527]: /opt/wmf-mariadb104/bin/mysqld(+0xaff23a)[0x556d7bdd123a]
|
May 27 02:11:22 labsdb1011 mysqld[23527]: /opt/wmf-mariadb104/bin/mysqld(+0xa4dbd5)[0x556d7bd1fbd5]
|
May 27 02:11:24 labsdb1011 mysqld[23527]: /opt/wmf-mariadb104/bin/mysqld(_ZN7handler12ha_write_rowEPKh+0x14d)[0x556d7bb837ed]
|
May 27 02:11:26 labsdb1011 mysqld[23527]: /opt/wmf-mariadb104/bin/mysqld(_ZN14Rows_log_event9write_rowEP14rpl_group_infob+0x174)[0x556d7bc755d4]
|
May 27 02:11:28 labsdb1011 mysqld[23527]: /opt/wmf-mariadb104/bin/mysqld(_ZN20Write_rows_log_event11do_exec_rowEP14rpl_group_info+0x7d)[0x556d7bc75b6d]
|
May 27 02:11:30 labsdb1011 mysqld[23527]: /opt/wmf-mariadb104/bin/mysqld(_ZN14Rows_log_event14do_apply_eventEP14rpl_group_info+0x23c)[0x556d7bc69f8c]
|
May 27 02:11:31 labsdb1011 mysqld[23527]: /opt/wmf-mariadb104/bin/mysqld(+0x5faf02)[0x556d7b8ccf02]
|
May 27 02:11:33 labsdb1011 mysqld[23527]: /opt/wmf-mariadb104/bin/mysqld(handle_slave_sql+0x12e2)[0x556d7b8d5ef2]
|
May 27 02:11:35 labsdb1011 mysqld[23527]: /opt/wmf-mariadb104/bin/mysqld(+0xd5e28b)[0x556d7c03028b]
|
May 27 02:11:37 labsdb1011 mysqld[23527]: /lib/x86_64-linux-gnu/libpthread.so.0(+0x7fa3)[0x7fc67eb83fa3]
|
May 27 02:11:39 labsdb1011 mysqld[23527]: /lib/x86_64-linux-gnu/libc.so.6(clone+0x3f)[0x7fc67e2b84cf]
|
May 27 02:11:39 labsdb1011 mysqld[23527]: Trying to get some variables.
|
May 27 02:11:39 labsdb1011 mysqld[23527]: Some pointers may be invalid and cause the dump to abort.
|
May 27 02:11:39 labsdb1011 mysqld[23527]: Query (0x0):
|
May 27 02:11:39 labsdb1011 mysqld[23527]: Connection ID (thread ID): 88
|
May 27 02:11:39 labsdb1011 mysqld[23527]: Status: NOT_KILLED
|
May 27 02:11:39 labsdb1011 mysqld[23527]: Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=on,condition_pushdown_from_having=on
|
May 27 02:11:39 labsdb1011 mysqld[23527]: The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
|
May 27 02:11:39 labsdb1011 mysqld[23527]: information that should help you find out what is causing the crash.
|
May 27 02:11:39 labsdb1011 mysqld[23527]: Writing a core file...
|
May 27 02:11:39 labsdb1011 mysqld[23527]: Working directory at /srv/sqldata
|
May 27 02:11:39 labsdb1011 mysqld[23527]: Resource Limits:
|
May 27 02:11:39 labsdb1011 mysqld[23527]: Limit Soft Limit Hard Limit Units
|
May 27 02:11:39 labsdb1011 mysqld[23527]: Max cpu time unlimited unlimited seconds
|
May 27 02:11:39 labsdb1011 mysqld[23527]: Max file size unlimited unlimited bytes
|
May 27 02:11:39 labsdb1011 mysqld[23527]: Max data size unlimited unlimited bytes
|
May 27 02:11:39 labsdb1011 mysqld[23527]: Max stack size 8388608 unlimited bytes
|
May 27 02:11:39 labsdb1011 mysqld[23527]: Max core file size 0 0 bytes
|
May 27 02:11:39 labsdb1011 mysqld[23527]: Max resident set unlimited unlimited bytes
|
May 27 02:11:39 labsdb1011 mysqld[23527]: Max processes 2063395 2063395 processes
|
May 27 02:11:39 labsdb1011 mysqld[23527]: Max open files 200001 200001 files
|
May 27 02:11:39 labsdb1011 mysqld[23527]: Max locked memory 65536 65536 bytes
|
May 27 02:11:39 labsdb1011 mysqld[23527]: Max address space unlimited unlimited bytes
|
May 27 02:11:39 labsdb1011 mysqld[23527]: Max file locks unlimited unlimited locks
|
May 27 02:11:39 labsdb1011 mysqld[23527]: Max pending signals 2063395 2063395 signals
|
May 27 02:11:39 labsdb1011 mysqld[23527]: Max msgqueue size 819200 819200 bytes
|
May 27 02:11:39 labsdb1011 mysqld[23527]: Max nice priority 0 0
|
May 27 02:11:39 labsdb1011 mysqld[23527]: Max realtime priority 0 0
|
May 27 02:11:39 labsdb1011 mysqld[23527]: Max realtime timeout unlimited unlimited us
|
May 27 02:11:39 labsdb1011 mysqld[23527]: Core pattern: /var/tmp/core/core.%h.%e.%p....
|
May 27 02:12:13 labsdb1011 systemd[1]: mariadb.service: Main process exited, code=killed, status=6/ABRT
|
Happened in a different host (so we can rule out HW specific issues) and with data coming from a logical dump from a different server (so we can rule out data specific corruption issues).
This would block us to be able to migrate to 10.4 entirely.
We have seen another crash with similar output on a different host that doesn't run multi-source.
Jul 02 11:00:33 dbstore1005 mysqld[4259]: 2020-07-02 11:00:33 32 [ERROR] InnoDB: Record in index `page_redirect_namespace_len` of table `wikidatawiki`.`page` was not found on update: TUPLE (info_bits=0, 4 fields): {[1] (0x00),[4] (0x80
|
000000),[4] ' (0x000027CB),[4] } (0x01017DA1)} at: COMPACT RECORD(info_bits=0, 4 fields): {[1] (0x00),[4] (0x80000000),[4] ' (0x000027CB),[4] (0x0100C39D)}
|
Jul 02 11:00:33 dbstore1005 mysqld[4259]: 2020-07-02 11:00:33 32 [ERROR] InnoDB: Record in index `pl_backlinks_namespace` of table `wikidatawiki`.`pagelinks` was not found on update: TUPLE (info_bits=0, 4 fields): {[4] (0x80000000),[4]
|
x(0x80000078),[5]P4656(0x5034363536),[4] } (0x01017DA1)} at: COMPACT RECORD(info_bits=0, 4 fields): {[4] (0x80000000),[4] x(0x80000078),[5]P4656(0x5034363536),[4] } (0x01017D97)}
|
Jul 02 11:00:33 dbstore1005 mysqld[4259]: 2020-07-02 11:00:33 32 [ERROR] InnoDB: Record in index `pl_backlinks_namespace` of table `wikidatawiki`.`pagelinks` was not found on update: TUPLE (info_bits=0, 4 fields): {[4] (0x80000000),[4]
|
x(0x80000078),[4]P571(0x50353731),[4] } (0x01017DA1)} at: COMPACT RECORD(info_bits=0, 4 fields): {[4] (0x80000000),[4] x(0x80000078),[4]P571(0x50353731),[4] } (0x01017D97)}
|
Jul 02 11:00:35 dbstore1005 mysqld[4259]: 2020-07-02 11:00:35 32 [ERROR] InnoDB: Record in index `page_len` of table `wikidatawiki`.`page` was not found on update: TUPLE (info_bits=0, 2 fields): {[4] ) (0x000029EE),[4] HT(0x01014854)} a
|
t: COMPACT RECORD(info_bits=0, 2 fields): {[4] ) (0x000029EE),[4] (0x010115D8)}
|
Jul 02 11:00:35 dbstore1005 mysqld[4259]: 2020-07-02 11:00:35 32 [ERROR] InnoDB: Record in index `page_redirect_namespace_len` of table `wikidatawiki`.`page` was not found on update: TUPLE (info_bits=0, 4 fields): {[1] (0x00),[4] (0x80
|
000000),[4] ) (0x000029EE),[4] HT(0x01014854)} at: COMPACT RECORD(info_bits=0, 4 fields): {[1] (0x00),[4] (0x80000000),[4] ) (0x000029EE),[4] (0x010115D8)}
|
Jul 02 11:00:35 dbstore1005 mysqld[4259]: 2020-07-02 11:00:35 32 [ERROR] InnoDB: Record in index `pl_backlinks_namespace` of table `wikidatawiki`.`pagelinks` was not found on update: TUPLE (info_bits=0, 4 fields): {[4] (0x80000000),[4]
|
x(0x80000078),[5]P4656(0x5034363536),[4] HT(0x01014854)} at: COMPACT RECORD(info_bits=0, 4 fields): {[4] (0x80000000),[4] x(0x80000078),[5]P4656(0x5034363536),[4] H2(0x01014832)}
|
Jul 02 11:00:35 dbstore1005 mysqld[4259]: 2020-07-02 11:00:35 32 [ERROR] InnoDB: Record in index `pl_backlinks_namespace` of table `wikidatawiki`.`pagelinks` was not found on update: TUPLE (info_bits=0, 4 fields): {[4] (0x80000000),[4]
|
x(0x80000078),[4]P571(0x50353731),[4] HT(0x01014854)} at: COMPACT RECORD(info_bits=0, 4 fields): {[4] (0x80000000),[4] x(0x80000078),[4]P571(0x50353731),[4] H (0x01014818)}
|
Jul 02 11:00:37 dbstore1005 mysqld[4259]: 2020-07-02 11:00:37 1 [ERROR] InnoDB: Unable to find a record to delete-mark
|
Jul 02 11:00:37 dbstore1005 mysqld[4259]: InnoDB: tuple DATA TUPLE: 4 fields;
|
Jul 02 11:00:37 dbstore1005 mysqld[4259]: 0: len 4; hex 80000000; asc ;;
|
Jul 02 11:00:37 dbstore1005 mysqld[4259]: 1: len 4; hex 80000078; asc x;;
|
Jul 02 11:00:37 dbstore1005 mysqld[4259]: 2: len 4; hex 50353731; asc P571;;
|
Jul 02 11:00:37 dbstore1005 mysqld[4259]: 3: len 4; hex 00ff1b9a; asc ;;
|
Jul 02 11:00:37 dbstore1005 mysqld[4259]: InnoDB: record PHYSICAL RECORD: n_fields 4; compact format; info bits 0
|
Jul 02 11:00:37 dbstore1005 mysqld[4259]: 0: len 4; hex 80000000; asc ;;
|
Jul 02 11:00:37 dbstore1005 mysqld[4259]: 1: len 4; hex 80000078; asc x;;
|
Jul 02 11:00:37 dbstore1005 mysqld[4259]: 2: len 4; hex 50353731; asc P571;;
|
Jul 02 11:00:37 dbstore1005 mysqld[4259]: 3: len 4; hex 00ff1b99; asc ;;
|
Jul 02 11:00:37 dbstore1005 mysqld[4259]: 2020-07-02 11:00:37 1 [ERROR] InnoDB: page [page id: space=1020, page number=6276571] (429 records, index id 3321).
|
Jul 02 11:00:37 dbstore1005 mysqld[4259]: 2020-07-02 11:00:37 1 [ERROR] InnoDB: Submit a detailed bug report to https://jira.mariadb.org/
|
Jul 02 11:00:37 dbstore1005 mysqld[4259]: 2020-07-02 11:00:37 1 [ERROR] InnoDB: Unable to find a record to delete-mark
|
Jul 02 11:00:37 dbstore1005 mysqld[4259]: InnoDB: tuple DATA TUPLE: 4 fields;
|
Jul 02 11:00:37 dbstore1005 mysqld[4259]: 0: len 4; hex 80000000; asc ;;
|
Jul 02 11:00:37 dbstore1005 mysqld[4259]: 1: len 4; hex 80000078; asc x;;
|
Jul 02 11:00:37 dbstore1005 mysqld[4259]: 2: len 5; hex 5034363536; asc P4656;;
|
Jul 02 11:00:37 dbstore1005 mysqld[4259]: 3: len 4; hex 00ff1b9a; asc ;;
|
Jul 02 11:00:37 dbstore1005 mysqld[4259]: InnoDB: record PHYSICAL RECORD: n_fields 4; compact format; info bits 0
|
Jul 02 11:00:37 dbstore1005 mysqld[4259]: 0: len 4; hex 80000000; asc ;;
|
Jul 02 11:00:37 dbstore1005 mysqld[4259]: 1: len 4; hex 80000078; asc x;;
|
Jul 02 11:00:37 dbstore1005 mysqld[4259]: 2: len 5; hex 5034363536; asc P4656;;
|
Jul 02 11:00:37 dbstore1005 mysqld[4259]: 3: len 4; hex 00ff1b99; asc ;;
|
Jul 02 11:00:37 dbstore1005 mysqld[4259]: 2020-07-02 11:00:37 1 [ERROR] InnoDB: page [page id: space=1020, page number=9314226] (519 records, index id 3321).
|
Jul 02 11:00:37 dbstore1005 mysqld[4259]: 2020-07-02 11:00:37 1 [ERROR] InnoDB: Submit a detailed bug report to https://jira.mariadb.org/
|
Jul 02 11:00:39 dbstore1005 mysqld[4259]: 2020-07-02 11:00:39 32 [ERROR] InnoDB: Record in index `page_redirect_namespace_len` of table `wikidatawiki`.`page` was not found on update: TUPLE (info_bits=0, 4 fields): {[1] (0x00),[4] (0x80
|
000000),[4] (0x0000167F),[4] W (0x010157D3)} at: COMPACT RECORD(info_bits=0, 4 fields): {[1] (0x00),[4] (0x80000000),[4] (0x0000167F),[4] ;_(0x00FF3B5F)}
|
Jul 02 11:00:39 dbstore1005 mysqld[4259]: 2020-07-02 11:00:39 32 [ERROR] InnoDB: Record in index `pl_backlinks_namespace` of table `wikidatawiki`.`pagelinks` was not found on update: TUPLE (info_bits=0, 4 fields): {[4] (0x80000000),[4]
|
x(0x80000078),[4]P571(0x50353731),[4] W (0x010157D3)} at: COMPACT RECORD(info_bits=0, 4 fields): {[4] (0x80000000),[4] x(0x80000078),[4]P571(0x50353731),[4] W (0x010157D1)}
|
Jul 02 11:00:39 dbstore1005 mysqld[4259]: 2020-07-02 11:00:39 1 [ERROR] InnoDB: Unable to find a record to delete-mark
|
Jul 02 11:00:39 dbstore1005 mysqld[4259]: InnoDB: tuple DATA TUPLE: 4 fields;
|
Jul 02 11:00:39 dbstore1005 mysqld[4259]: 0: len 4; hex 80000000; asc ;;
|
Jul 02 11:00:39 dbstore1005 mysqld[4259]: 1: len 4; hex 80000078; asc x;;
|
Jul 02 11:00:39 dbstore1005 mysqld[4259]: 2: len 5; hex 5034363536; asc P4656;;
|
Jul 02 11:00:39 dbstore1005 mysqld[4259]: 3: len 4; hex 010157d3; asc W ;;
|
Jul 02 11:00:39 dbstore1005 mysqld[4259]: InnoDB: record PHYSICAL RECORD: n_fields 4; compact format; info bits 0
|
Jul 02 11:00:39 dbstore1005 mysqld[4259]: 0: len 4; hex 80000000; asc ;;
|
Jul 02 11:00:39 dbstore1005 mysqld[4259]: 1: len 4; hex 80000078; asc x;;
|
Jul 02 11:00:39 dbstore1005 mysqld[4259]: 2: len 5; hex 5034363536; asc P4656;;
|
Jul 02 11:00:39 dbstore1005 mysqld[4259]: 3: len 4; hex 010157b6; asc W ;;
|
Jul 02 11:00:39 dbstore1005 mysqld[4259]: 2020-07-02 11:00:39 1 [ERROR] InnoDB: page [page id: space=1020, page number=4847974] (364 records, index id 3321).
|
Jul 02 11:00:39 dbstore1005 mysqld[4259]: 2020-07-02 11:00:39 1 [ERROR] InnoDB: Submit a detailed bug report to https://jira.mariadb.org/
|
Jul 02 11:00:41 dbstore1005 mysqld[4259]: 2020-07-02 11:00:41 32 [ERROR] InnoDB: Record in index `page_len` of table `wikidatawiki`.`page` was not found on update: TUPLE (info_bits=0, 2 fields): {[4] W(0x00001457),[4] !R(0x01022152)} at: COMPACT RECORD(info_bits=0, 2 fields): {[4] W(0x00001457),[4] SM(0x0101534D)}
|
Jul 02 11:00:41 dbstore1005 mysqld[4259]: 2020-07-02 11:00:41 32 [ERROR] InnoDB: Record in index `page_redirect_namespace_len` of table `wikidatawiki`.`page` was not found on update: TUPLE (info_bits=0, 4 fields): {[1] (0x00),[4] (0x80000000),[4] W(0x00001457),[4] !R(0x01022152)} at: COMPACT RECORD(info_bits=0, 4 fields): {[1] (0x00),[4] (0x80000000),[4] W(0x00001457),[4] SM(0x0101534D)}
|
Jul 02 11:00:41 dbstore1005 mysqld[4259]: 2020-07-02 11:00:41 32 [ERROR] InnoDB: Record in index `pl_backlinks_namespace` of table `wikidatawiki`.`pagelinks` was not found on update: TUPLE (info_bits=0, 4 fields): {[4] (0x80000000),[4] x(0x80000078),[5]P4656(0x5034363536),[4] !R(0x01022152)} at: COMPACT RECORD(info_bits=0, 4 fields): {[4] (0x80000000),[4] x(0x80000078),[5]P4656(0x5034363536),[4] !O(0x0102214F)}
|
Jul 02 11:00:41 dbstore1005 mysqld[4259]: 2020-07-02 11:00:41 32 [ERROR] InnoDB: Record in index `pl_backlinks_namespace` of table `wikidatawiki`.`pagelinks` was not found on update: TUPLE (info_bits=0, 4 fields): {[4] (0x80000000),[4] x(0x80000078),[4]P571(0x50353731),[4] !R(0x01022152)} at: COMPACT RECORD(info_bits=0, 4 fields): {[4] (0x80000000),[4] x(0x80000078),[4]P571(0x50353731),[4] !M(0x0102214D)}
|
Jul 02 11:00:42 dbstore1005 mysqld[4259]: 2020-07-02 11:00:42 32 [ERROR] InnoDB: Record in index `page_len` of table `wikidatawiki`.`page` was not found on update: TUPLE (info_bits=0, 2 fields): {[4] S(0x00001553),[4] O (0x011A4FCB)} at: COMPACT RECORD(info_bits=0, 2 fields): {[4] S(0x00001553),[4] 1 (0x011A31EB)}
|
Jul 02 11:00:42 dbstore1005 mysqld[4259]: 2020-07-02 11:00:42 32 [ERROR] InnoDB: Record in index `pl_backlinks_namespace` of table `wikidatawiki`.`pagelinks` was not found on update: TUPLE (info_bits=0, 4 fields): {[4] (0x80000000),[4] x(0x80000078),[5]P4656(0x5034363536),[4] O (0x011A4FCB)} at: COMPACT RECORD(info_bits=0, 4 fields): {[4] (0x80000000),[4] x(0x80000078),[5]P4656(0x5034363536),[4] O (0x011A4FC9)}
|
Jul 02 11:00:44 dbstore1005 mysqld[4259]: 2020-07-02 11:00:44 32 [ERROR] InnoDB: Record in index `pl_backlinks_namespace` of table `wikidatawiki`.`pagelinks` was not found on update: TUPLE (info_bits=0, 4 fields): {[4] (0x80000000),[4] x(0x80000078),[5]P4656(0x5034363536),[4] ](0x0101905D)} at: COMPACT RECORD(info_bits=0, 4 fields): {[4] (0x80000000),[4] x(0x80000078),[5]P4656(0x5034363536),[4] [(0x0101905B)}
|
Jul 02 11:00:44 dbstore1005 mysqld[4259]: 2020-07-02 11:00:44 32 [ERROR] InnoDB: Record in index `pl_backlinks_namespace` of table `wikidatawiki`.`pagelinks` was not found on update: TUPLE (info_bits=0, 4 fields): {[4] (0x80000000),[4] x(0x80000078),[4]P571(0x50353731),[4] ](0x0101905D)} at: COMPACT RECORD(info_bits=0, 4 fields): {[4] (0x80000000),[4] x(0x80000078),[4]P571(0x50353731),[4] [(0x0101905B)}
|
Jul 02 11:00:46 dbstore1005 mysqld[4259]: 2020-07-02 11:00:46 32 [ERROR] InnoDB: Record in index `pl_backlinks_namespace` of table `wikidatawiki`.`pagelinks` was not found on update: TUPLE (info_bits=0, 4 fields): {[4] (0x80000000),[4] x(0x80000078),[4]P571(0x50353731),[4] "(0x01020F22)} at: COMPACT RECORD(info_bits=0, 4 fields): {[4] (0x80000000),[4] x(0x80000078),[4]P571(0x50353731),[4] (0x01020F0D)}
|
Jul 02 11:00:47 dbstore1005 mysqld[4259]: 2020-07-02 11:00:47 32 [ERROR] InnoDB: Record in index `page_redirect_namespace_len` of table `wikidatawiki`.`page` was not found on update: TUPLE (info_bits=0, 4 fields): {[1] (0x00),[4] (0x80000000),[4] ) (0x000029B3),[4] B!(0x01024221)} at: COMPACT RECORD(info_bits=0, 4 fields): {[1] (0x00),[4] (0x80000000),[4] ) (0x000029B3),[4] )1(0x01022931)}
|
Jul 02 11:00:47 dbstore1005 mysqld[4259]: 2020-07-02 11:00:47 32 [ERROR] InnoDB: Record in index `pl_backlinks_namespace` of table `wikidatawiki`.`pagelinks` was not found on update: TUPLE (info_bits=0, 4 fields): {[4] (0x80000000),[4] x(0x80000078),[4]P571(0x50353731),[4] B!(0x01024221)} at: COMPACT RECORD(info_bits=0, 4 fields): {[4] (0x80000000),[4] x(0x80000078),[4]P571(0x50353731),[4] B (0x01024211)}
|
Jul 02 11:00:49 dbstore1005 mysqld[4259]: 2020-07-02 11:00:49 32 [ERROR] InnoDB: Record in index `page_len` of table `wikidatawiki`.`page` was not found on update: TUPLE (info_bits=0, 2 fields): {[4] I(0x00001549),[4] M (0x011A4D88)} at: COMPACT RECORD(info_bits=0, 2 fields): {[4] I(0x00001549),[4] (0x011A160D)}
|
Jul 02 11:00:49 dbstore1005 mysqld[4259]: 2020-07-02 11:00:49 32 [ERROR] InnoDB: Record in index `el_to` of table `wikidatawiki`.`externallinks` was not found on update: TUPLE (info_bits=0, 3 fields): {[60]https://en.wikipedia.org/w/index.php?title=Sunnyside_(Newsom(0x68747470733A2F2F656E2E77696B6970656469612E6F72672F772F696E6465782E7068703F7469746C653D53756E6E79736964655F284E6577736F6D),[4] M (0x011A4D88),[4] (0x04A3A4DC)} at: COMPACT RECORD(info_bits=0, 3 fields): {[60]https://en.wikipedia.org/w/index.php?title=Sunnyside_(Heaths(0x68747470733A2F2F656E2E77696B6970656469612E6F72672F772F696E6465782E7068703F7469746C653D53756E6E79736964655F28486561746873),[4] (0x00F9F111),[4] (0x0205DCF7)}
|
Jul 02 11:00:49 dbstore1005 mysqld[4259]: 2020-07-02 11:00:49 32 [ERROR] InnoDB: Record in index `el_index` of table `wikidatawiki`.`externallinks` was not found on update: TUPLE (info_bits=0, 2 fields): {[60]https://org.wikipedia.en./w/index.php?title=Sunnyside_(Newso(0x68747470733A2F2F6F72672E77696B6970656469612E656E2E2F772F696E6465782E7068703F7469746C653D53756E6E79736964655F284E6577736F),[4] (0x04A3A4DC)} at: COMPACT RECORD(info_bits=0, 2 fields): {[60]https://org.wikipedia.en./w/index.php?title=Sunnyside_(Heath(0x68747470733A2F2F6F72672E77696B6970656469612E656E2E2F772F696E6465782E7068703F7469746C653D53756E6E79736964655F284865617468),[4] (0x0205DCF7)}
|
Jul 02 11:00:51 dbstore1005 mysqld[4259]: 2020-07-02 11:00:51 32 [ERROR] InnoDB: Record in index `page_redirect_namespace_len` of table `wikidatawiki`.`page` was not found on update: TUPLE (info_bits=0, 4 fields): {[1] (0x00),[4] (0x80000000),[4] (0x0000170A),[4] (0x0101F4A2)} at: COMPACT RECORD(info_bits=0, 4 fields): {[1] (0x00),[4] (0x80000000),[4] (0x0000170A),[4] (0x0101F317)}
|
Jul 02 11:00:51 dbstore1005 mysqld[4259]: 2020-07-02 11:00:51 32 [ERROR] InnoDB: Record in index `pl_backlinks_namespace` of table `wikidatawiki`.`pagelinks` was not found on update: TUPLE (info_bits=0, 4 fields): {[4] (0x80000000),[4] x(0x80000078),[5]P4656(0x5034363536),[4] (0x0101F4A2)} at: COMPACT RECORD(info_bits=0, 4 fields): {[4] (0x80000000),[4] x(0x80000078),[5]P4656(0x5034363536),[4] w(0x0101F477)}
|
Jul 02 11:00:51 dbstore1005 mysqld[4259]: 2020-07-02 11:00:51 32 [ERROR] InnoDB: Record in index `pl_backlinks_namespace` of table `wikidatawiki`.`pagelinks` was not found on update: TUPLE (info_bits=0, 4 fields): {[4] (0x80000000),[4] x(0x80000078),[4]P571(0x50353731),[4] (0x0101F4A2)} at: COMPACT RECORD(info_bits=0, 4 fields): {[4] (0x80000000),[4] x(0x80000078),[4]P571(0x50353731),[4] b(0x0101F462)}
|
Jul 02 11:00:53 dbstore1005 mysqld[4259]: 2020-07-02 11:00:53 32 [ERROR] InnoDB: Record in index `page_len` of table `wikidatawiki`.`page` was not found on update: TUPLE (info_bits=0, 2 fields): {[4] %#(0x00002523),[4] 1 (0x010231C9)} at: COMPACT RECORD(info_bits=0, 2 fields): {[4] %#(0x00002523),[4] (0x010014A8)}
|
Jul 02 11:00:53 dbstore1005 mysqld[4259]: 2020-07-02 11:00:53 0x7fdf103d3700 InnoDB: Assertion failure in file /root/mariadb-10.4.13/storage/innobase/row/row0ins.cc line 231
|
Jul 02 11:00:53 dbstore1005 mysqld[4259]: InnoDB: Failing assertion: !cursor->index->is_committed()
|
Jul 02 11:00:53 dbstore1005 mysqld[4259]: InnoDB: We intentionally generate a memory trap.
|
Jul 02 11:00:53 dbstore1005 mysqld[4259]: InnoDB: Submit a detailed bug report to https://jira.mariadb.org/
|
Jul 02 11:00:53 dbstore1005 mysqld[4259]: InnoDB: If you get repeated assertion failures or crashes, even
|
Jul 02 11:00:53 dbstore1005 mysqld[4259]: InnoDB: immediately after the mysqld startup, there may be
|
Jul 02 11:00:53 dbstore1005 mysqld[4259]: InnoDB: corruption in the InnoDB tablespace. Please refer to
|
Jul 02 11:00:53 dbstore1005 mysqld[4259]: InnoDB: https://mariadb.com/kb/en/library/innodb-recovery-modes/
|
Jul 02 11:00:53 dbstore1005 mysqld[4259]: InnoDB: about forcing recovery.
|
Jul 02 11:00:53 dbstore1005 mysqld[4259]: 200702 11:00:53 [ERROR] mysqld got signal 6 ;
|
Jul 02 11:00:53 dbstore1005 mysqld[4259]: This could be because you hit a bug. It is also possible that this binary
|
Jul 02 11:00:53 dbstore1005 mysqld[4259]: or one of the libraries it was linked against is corrupt, improperly built,
|
Jul 02 11:00:53 dbstore1005 mysqld[4259]: or misconfigured. This error can also be caused by malfunctioning hardware.
|
Jul 02 11:00:53 dbstore1005 mysqld[4259]: To report this bug, see https://mariadb.com/kb/en/reporting-bugs
|
Jul 02 11:00:53 dbstore1005 mysqld[4259]: We will try our best to scrape up some info that will hopefully help
|
Jul 02 11:00:53 dbstore1005 mysqld[4259]: diagnose the problem, but since we have already crashed,
|
Jul 02 11:00:53 dbstore1005 mysqld[4259]: something is definitely wrong and this may fail.
|
Jul 02 11:00:53 dbstore1005 mysqld[4259]: Server version: 10.4.13-MariaDB
|
Jul 02 11:00:53 dbstore1005 mysqld[4259]: key_buffer_size=1048576
|
Jul 02 11:00:53 dbstore1005 mysqld[4259]: read_buffer_size=131072
|
Jul 02 11:00:53 dbstore1005 mysqld[4259]: max_used_connections=77
|
Jul 02 11:00:53 dbstore1005 mysqld[4259]: max_threads=252
|
Jul 02 11:00:53 dbstore1005 mysqld[4259]: thread_count=83
|
Jul 02 11:00:53 dbstore1005 mysqld[4259]: It is possible that mysqld could use up to
|
Jul 02 11:00:53 dbstore1005 mysqld[4259]: key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 555578 K bytes of memory
|
Jul 02 11:00:53 dbstore1005 mysqld[4259]: Hope that's ok; if not, decrease some variables in the equation.
|
Jul 02 11:00:53 dbstore1005 mysqld[4259]: Thread pointer: 0x7fa6940014f8
|
Jul 02 11:00:53 dbstore1005 mysqld[4259]: Attempting backtrace. You can use the following information to find out
|
Jul 02 11:00:53 dbstore1005 mysqld[4259]: where mysqld died. If you see no messages after this, something went
|
Jul 02 11:00:53 dbstore1005 mysqld[4259]: terribly wrong...
|
Jul 02 11:00:53 dbstore1005 mysqld[4259]: stack_bottom = 0x7fdf103d2698 thread_stack 0x49000
|
Jul 02 11:00:53 dbstore1005 mysqld[4259]: /opt/wmf-mariadb104/bin/mysqld(my_print_stacktrace+0x2e)[0x558719c157de]
|
Jul 02 11:00:53 dbstore1005 mysqld[4259]: /opt/wmf-mariadb104/bin/mysqld(handle_fatal_signal+0x54d)[0x55871970db4d]
|
Jul 02 11:00:54 dbstore1005 mysqld[4259]: /lib/x86_64-linux-gnu/libpthread.so.0(+0x12730)[0x7fdf2a0ed730]
|
Jul 02 11:00:54 dbstore1005 mysqld[4259]: /lib/x86_64-linux-gnu/libc.so.6(gsignal+0x10b)[0x7fdf297557bb]
|
Jul 02 11:00:54 dbstore1005 mysqld[4259]: /lib/x86_64-linux-gnu/libc.so.6(abort+0x121)[0x7fdf29740535]
|
Jul 02 11:00:54 dbstore1005 mysqld[4259]: /opt/wmf-mariadb104/bin/mysqld(+0x5a19d5)[0x5587194099d5]
|
Jul 02 11:00:54 dbstore1005 mysqld[4259]: /opt/wmf-mariadb104/bin/mysqld(+0x5907c1)[0x5587193f87c1]
|
Jul 02 11:00:54 dbstore1005 mysqld[4259]: /opt/wmf-mariadb104/bin/mysqld(+0xaeee0e)[0x558719956e0e]
|
Jul 02 11:00:54 dbstore1005 mysqld[4259]: /opt/wmf-mariadb104/bin/mysqld(+0xb22478)[0x55871998a478]
|
Jul 02 11:00:54 dbstore1005 mysqld[4259]: /opt/wmf-mariadb104/bin/mysqld(+0xb27e0f)[0x55871998fe0f]
|
Jul 02 11:00:54 dbstore1005 mysqld[4259]: /opt/wmf-mariadb104/bin/mysqld(+0xb0023c)[0x55871996823c]
|
Jul 02 11:00:54 dbstore1005 mysqld[4259]: /opt/wmf-mariadb104/bin/mysqld(+0xa5043f)[0x5587198b843f]
|
Jul 02 11:00:55 dbstore1005 mysqld[4259]: /opt/wmf-mariadb104/bin/mysqld(_ZN7handler13ha_update_rowEPKhS1_+0xbb)[0x558719719a0b]
|
Jul 02 11:00:55 dbstore1005 mysqld[4259]: /opt/wmf-mariadb104/bin/mysqld(_Z12mysql_updateP3THDP10TABLE_LISTR4ListI4ItemES6_PS4_jP8st_orderybPySA_+0x1b27)[0x5587195b5617]
|
Jul 02 11:00:55 dbstore1005 mysqld[4259]: /opt/wmf-mariadb104/bin/mysqld(_Z21mysql_execute_commandP3THD+0x26ca)[0x5587195084fa]
|
Jul 02 11:00:55 dbstore1005 mysqld[4259]: /opt/wmf-mariadb104/bin/mysqld(_Z11mysql_parseP3THDPcjP12Parser_statebb+0x1c9)[0x55871950e179]
|
Jul 02 11:00:55 dbstore1005 mysqld[4259]: /opt/wmf-mariadb104/bin/mysqld(_ZN15Query_log_event14do_apply_eventEP14rpl_group_infoPKcj+0x682)[0x558719804e52]
|
Jul 02 11:00:55 dbstore1005 mysqld[4259]: /opt/wmf-mariadb104/bin/mysqld(+0x5faf02)[0x558719462f02]
|
Jul 02 11:00:55 dbstore1005 mysqld[4259]: /opt/wmf-mariadb104/bin/mysqld(handle_slave_sql+0x12e2)[0x55871946bef2]
|
Jul 02 11:00:55 dbstore1005 mysqld[4259]: /opt/wmf-mariadb104/bin/mysqld(+0xd5e28b)[0x558719bc628b]
|
Jul 02 11:00:55 dbstore1005 mysqld[4259]: /lib/x86_64-linux-gnu/libpthread.so.0(+0x7fa3)[0x7fdf2a0e2fa3]
|
Jul 02 11:00:55 dbstore1005 mysqld[4259]: /lib/x86_64-linux-gnu/libc.so.6(clone+0x3f)[0x7fdf298174cf]
|
Jul 02 11:00:55 dbstore1005 mysqld[4259]: Trying to get some variables.
|
Jul 02 11:00:55 dbstore1005 mysqld[4259]: Some pointers may be invalid and cause the dump to abort.
|
Jul 02 11:00:55 dbstore1005 mysqld[4259]: Query (0x7fa6943bd91b): UPDATE /* WikiPage::updateRevisionOn */ `page` SET page_latest = 1222570180,page_touched = '20200702110053',page_is_new = 0,page_is_redirect = 0,page_len = 8230,page_content_model = 'wikibase-item' WHERE page_id = 16921033
|
Jul 02 11:00:55 dbstore1005 mysqld[4259]: Connection ID (thread ID): 32
|
Jul 02 11:00:55 dbstore1005 mysqld[4259]: Status: NOT_KILLED
|
Jul 02 11:00:55 dbstore1005 mysqld[4259]: Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=on,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=on,condition_pushdown_from_having=on
|
Jul 02 11:00:55 dbstore1005 mysqld[4259]: The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
|
Jul 02 11:00:55 dbstore1005 mysqld[4259]: information that should help you find out what is causing the crash.
|
Jul 02 11:00:55 dbstore1005 mysqld[4259]: Writing a core file...
|
Jul 02 11:00:55 dbstore1005 mysqld[4259]: Working directory at /srv/sqldata.s8
|
Jul 02 11:00:55 dbstore1005 mysqld[4259]: Resource Limits:
|
Jul 02 11:00:55 dbstore1005 mysqld[4259]: Limit Soft Limit Hard Limit Units
|
Jul 02 11:00:55 dbstore1005 mysqld[4259]: Max cpu time unlimited unlimited seconds
|
Jul 02 11:00:55 dbstore1005 mysqld[4259]: Max file size unlimited unlimited bytes
|
Jul 02 11:00:55 dbstore1005 mysqld[4259]: Max data size unlimited unlimited bytes
|
Jul 02 11:00:55 dbstore1005 mysqld[4259]: Max stack size 8388608 unlimited bytes
|
Jul 02 11:00:55 dbstore1005 mysqld[4259]: Max core file size 0 0 bytes
|
Jul 02 11:00:55 dbstore1005 mysqld[4259]: Max resident set unlimited unlimited bytes
|
Jul 02 11:00:55 dbstore1005 mysqld[4259]: Max processes 2058623 2058623 processes
|
Jul 02 11:00:55 dbstore1005 mysqld[4259]: Max open files 200001 200001 files
|
Jul 02 11:00:55 dbstore1005 mysqld[4259]: Max locked memory 65536 65536 bytes
|
Jul 02 11:00:55 dbstore1005 mysqld[4259]: Max address space unlimited unlimited bytes
|
Jul 02 11:00:55 dbstore1005 mysqld[4259]: Max file locks unlimited unlimited locks
|
Jul 02 11:00:55 dbstore1005 mysqld[4259]: Max pending signals 2058623 2058623 signals
|
Jul 02 11:00:55 dbstore1005 mysqld[4259]: Max msgqueue size 819200 819200 bytes
|
Jul 02 11:00:55 dbstore1005 mysqld[4259]: Max nice priority 0 0
|
Jul 02 11:00:55 dbstore1005 mysqld[4259]: Max realtime priority 0 0
|
Jul 02 11:00:55 dbstore1005 mysqld[4259]: Max realtime timeout unlimited unlimited us
|
Jul 02 11:00:55 dbstore1005 mysqld[4259]: Core pattern: /var/tmp/core/core.%h.%e.%p....
|
Jul 02 11:00:56 dbstore1005 systemd[1]: mariadb@s8.service: Main process exited, code=killed, status=6/ABRT
|
Any thoughts?
Thanks
marostegui, this is basically a corrupted secondary index. The corruption may have been introduced much earlier.
We have not been able to reproduce bugs in the MDEV-9663 family internally, unless indexed virtual columns (MDEV-5800) were involved. For those there are plenty of known problems.
MDEV-13542 should eventually introduce proper error propagation and avoid intentional crashes in cases like this.
If you can reproduce this crash with SQL (starting from an empty database), we can fix it, especially if the problem is repeatable under https://rr-project.org/. If not, unfortunately there is not much that we can do about it. You can work around the bug by dropping the corrupted indexes and creating them again.
Thanks for the answer Marko.
We did try to drop and recreate all the reported indexes, as we also thought it could be a real corruption, but that didn't really fix it.
Then it started to crash again and different indexes started to show up on the error logs. (https://phabricator.wikimedia.org/T249188#6147814)
We also dropped everything on the host and built it back form a mysqldump from another host, as we thought that would rule out the possible index corruption, but it also didn't make any difference.
marostegui, if you can reproduce this by replaying mysqldump on an empty tablespace and then invoking some SQL statements, then that would be a very promising starting point for us (unless the indexes in question include virtual columns, which is a separate class of bugs, such as MDEV-20484).
Maybe you could obfuscate enough of the data (anything non-indexed could be replaced with arbitrary values) so that the problem still repeats, or you could just upload it to a private file server, and we promise to keep it private and discard it afterwards?
That's logistically very hard. The total database is around 9TB, the data isn't a problem as it is public already. The problem is that we don't really know what triggers it. We have discarded HW as we've tried different servers.
We have also had the issue with data imported via mysqldump (mydumper really) from one host and with just a binary copy from another host (which could explain if the host we were copy the data from is corrupted).
Those host have a bunch of views which runs multisource replication and replicates from 8 masters.
This host receives hard queries 24x7, and usually gets delay.
I wouldn't be able to know which queries make them crash, however, we did see the host crashing just after a restart.
As in: we reimport the massive dump, start replication.......all good. Issue a stop all slaves (or even stopping its master to make sure there was no lag) restart mysql and then... crashes.
marostegui, I see. I have previously suspected that bugs in the InnoDB adaptive hash index might corrupt the data, but that is only a guess, and I have no firm idea what the exact mechanism could be. That was the main motivation why I disabled the feature by default in the 10.5 release (MDEV-20487). Since that decision, we have introduced https://rr-project.org/ to our internal testing workflow, and based on rr replay we have fixed some bugs related to the adaptive hash index, but probably not all of them yet. MDEV-20203 remains a mystery, for example. Today, we are testing a fix for MDEV-23233, which is a recent regression caused by MDEV-22456.
I also suspect that innodb_change_buffering=inserts or innodb_change_buffering=none might prevent corruption. We recently fixed MDEV-22497, but there could be further bugs in that area.
Maybe you could experiment with these two parameters and find some correlation?
Also, we have recently found and fixed bugs in the InnoDB buffer pool resizing feature, such as MDEV-22646. Are you using that?
After loading the dump, are you executing any DDL? Or is it DML-only workload?
Note: for MDEV-14643, an rr replay trace was generated last May, but unfortunately I was too busy with other things to analyze it before the trace was accidentally lost in June. I am confident that over time, we should obtain a trace again from internal testing. But we cannot predict when.
marostegui, we now have an rr replay trace for MDEV-22924 on MariaDB 10.5 where a small table (with only 7 rows) is corrupted.
Edit: That failure trace is related to the explicit use of XA transactions (basically a duplicate of MDEV-15532). A COMMIT statement would fail because it is not allowed after XA START, but yet we would wrongly release the meta-data locks while the transaction remains active inside InnoDB. This would wrongly allow an ALTER TABLE to start executing even when there exist active transactions on the table, and lead to corruption.
You did not reply whether you are issuing any DDL statements during the workload. If your workload only consists of DML, the corruption should be due to something else.
Let me try to address your previous comments
- We do not use innodb resizing on those hosts
- Those hosts replicate from 8 masters, which is quite intensive, they do occasionally receive alter tables from upstream (not often), but never out of band as those hosts are set to read-only.
- The majority of their heavy traffic consist on heavy and intensive massive SELECT queries, with lots of JOINs and table scans.
- After loading the dumps, we'd just enable replication on the 8 threads and the hosts replicated fine for days. However, after restarting mysql we did see InnoDB errors on the log and not long after that, the crashes.
marostegui, you mentioned this bug in MDEV-22497, whose fixVersion was wrong: the fix is included in 10.4.14, not 10.4.13. I am not convinced that MDEV-22497 would address this problem, though.
We now have a reasonably small test case for reproducing MDEV-22924, and I am working on analyzing it. Currently, it looks like that corruption does would involve the change buffer at all, because there are relatively few pages.
Thanks for the heads up Marko!
Marko, we have seen a similar crash on a host running 10.4.14 but this one ended with a buffer overflow.
After restarting mariadb and as soon as replication started to flow:
Sep 17 13:53:07 db2125 mysqld[3189]: Version: '10.4.14-MariaDB-log' socket: '/run/mysqld/mysqld.sock' port: 3306 MariaDB Server
|
Sep 17 13:53:07 db2125 systemd[1]: Started mariadb database server.
|
-- Subject: A start job for unit mariadb.service has finished successfully
|
-- Defined-By: systemd
|
-- Support: https://www.debian.org/support
|
--
|
-- A start job for unit mariadb.service has finished successfully.
|
--
|
-- The job identifier is 710.
|
Sep 17 13:53:07 db2125 mysqld[3189]: 2020-09-17 13:53:07 8 [Note] Slave I/O thread: connected to master 'repl@db2107.codfw.wmnet:3306',replication starts at GTID position '171970567-171970567-390719905,0-180359173-4880477695,180359173-180359173-70825087,171978786-17197878
|
Sep 17 13:54:16 db2125 mysqld[3189]: 2020-09-17 13:54:16 0 [ERROR] InnoDB: Unable to purge a record
|
Sep 17 13:54:16 db2125 mysqld[3189]: InnoDB: tuple DATA TUPLE: 2 fields;
|
Sep 17 13:54:16 db2125 mysqld[3189]: 0: len 4; hex 005c20d2; asc \ ;;
|
Sep 17 13:54:16 db2125 mysqld[3189]: 1: len 4; hex 0341d33a; asc A :;;
|
Sep 17 13:54:16 db2125 mysqld[3189]: InnoDB: record PHYSICAL RECORD: n_fields 2; compact format; info bits 0
|
Sep 17 13:54:16 db2125 mysqld[3189]: 0: len 4; hex 005c20d2; asc \ ;;
|
Sep 17 13:54:16 db2125 mysqld[3189]: 1: len 4; hex 0341d33a; asc A :;;
|
Sep 17 13:54:16 db2125 mysqld[3189]: space 7137 offset 4792 (1103 records, index id 25623)
|
Sep 17 13:54:16 db2125 mysqld[3189]: InnoDB: Submit a detailed bug report to https://jira.mariadb.org/
|
Sep 17 13:56:27 db2125 mysqld[3189]: 2020-09-17 13:56:27 0 [ERROR] InnoDB: Unable to purge a record
|
Sep 17 13:56:27 db2125 mysqld[3189]: InnoDB: tuple DATA TUPLE: 2 fields;
|
Sep 17 13:56:27 db2125 mysqld[3189]: 0: len 4; hex 002b95d4; asc + ;;
|
Sep 17 13:56:27 db2125 mysqld[3189]: 1: len 4; hex 0061c475; asc a u;;
|
Sep 17 13:56:27 db2125 mysqld[3189]: InnoDB: record PHYSICAL RECORD: n_fields 2; compact format; info bits 0
|
Sep 17 13:56:27 db2125 mysqld[3189]: 0: len 4; hex 002b95d4; asc + ;;
|
Sep 17 13:56:27 db2125 mysqld[3189]: 1: len 4; hex 0061c475; asc a u;;
|
Sep 17 13:56:27 db2125 mysqld[3189]: space 7251 offset 8988 (919 records, index id 26350)
|
Sep 17 13:56:27 db2125 mysqld[3189]: InnoDB: Submit a detailed bug report to https://jira.mariadb.org/
|
Sep 17 14:13:21 db2125 mysqld[3189]: 2020-09-17 14:13:21 0 [Note] InnoDB: Buffer pool(s) load completed at 200917 14:13:21
|
Sep 17 14:33:45 db2125 mysqld[3189]: 2020-09-17 14:33:45 9 [ERROR] InnoDB: Record in index `el_index_60` of table `zhwiki`.`externallinks` was not found on update: TUPLE (info_bits=0, 2 fields): {[60]https://com.kalashnikov./en/product/special/spec/krasnopol.h(0x687474707
|
Sep 17 14:58:13 db2125 mysqld[3189]: 2020-09-17 14:58:13 8 [ERROR] Error reading packet from server: Lost connection to MySQL server during query (server_errno=2013)
|
Sep 17 14:58:13 db2125 mysqld[3189]: 2020-09-17 14:58:13 8 [Note] Slave I/O thread: Failed reading log event, reconnecting to retry, log 'db2107-bin.001694' at position 198868170; GTID position '171970567-171970567-390719905,0-180359173-4880477695,180359173-180359173-7082
|
Sep 17 16:13:17 db2125 mysqld[3189]: 2020-09-17 16:13:17 8 [Note] Slave: received end packet from server, apparent master shutdown:
|
Sep 17 16:13:17 db2125 mysqld[3189]: 2020-09-17 16:13:17 8 [Note] Slave I/O thread: Failed reading log event, reconnecting to retry, log 'db2107-bin.001694' at position 338352002; GTID position '171970567-171970567-390719905,0-180359173-4880477695,180359173-180359173-7082
|
Sep 17 16:20:03 db2125 mysqld[3189]: 2020-09-17 16:20:03 9 [ERROR] InnoDB: Record in index `tl_namespace` of table `enwiktionary`.`templatelinks` was not found on update: TUPLE (info_bits=0, 3 fields): {[4] (0x8000000A),[11]catlangname(0x6361746C616E676E616D65),[4] 7?
|
|
Mariadb fully crashed ending with a buffer overflow:
Sep 17 17:58:53 db2125 mysqld[3189]: 2020-09-17 17:58:53 9 [ERROR] InnoDB: Record in index `pl_namespace` of table `idwiki`.`pagelinks` was not found on update: TUPLE (info_bits=0, 3 fields): {[4] (0x80000000),[11]Locker_Room(0x4C6F636B65725F526F6F6D),[4] . (0x00172E16)} at: COMPACT RECORD(info_bits=0, 3 fields): {[4] (0x80000000),[11]Locker_Room(0x4C6F636B65725F526F6F6D),[4] ! (0x00092116)}
|
Sep 17 18:30:43 db2125 mysqld[3189]: 2020-09-17 18:30:43 9 [ERROR] InnoDB: Record in index `iwl_prefix_title_from` of table `nlwiki`.`iwlinks` was not found on update: TUPLE (info_bits=0, 3 fields): {[1]d(0x64),[28]Wikidata:Status_updates/Next(0x57696B69646174613A5374617475735F757064617465732F4E657874),[4] x(0x0003B678)} at: COMPACT RECORD(info_bits=0, 3 fields): {[1]d(0x64),[46]Wikidata:Status_updates/Feedback_300th_edition(0x57696B69646174613A5374617475735F757064617465732F466565646261636B5F33303074685F65646974696F6E),[4] MN (0x004D4EBB)}
|
Sep 17 18:30:43 db2125 mysqld[3189]: 2020-09-17 18:30:43 9 [ERROR] InnoDB: Record in index `iwl_prefix_from_title` of table `nlwiki`.`iwlinks` was not found on update: TUPLE (info_bits=0, 3 fields): {[1]m(0x6D),[4] x(0x0003B678),[48]Global_message_delivery/Targets/Tech_ambassadors(0x476C6F62616C5F6D6573736167655F64656C69766572792F546172676574732F546563685F616D6261737361646F7273)} at: COMPACT RECORD(info_bits=0, 3 fields): {[1]m(0x6D),[4] (0x0003B5D5),[13]User:Rodejong(0x557365723A526F64656A6F6E67)}
|
Sep 17 18:30:43 db2125 mysqld[3189]: 2020-09-17 18:30:43 9 [ERROR] InnoDB: Record in index `iwl_prefix_from_title` of table `nlwiki`.`iwlinks` was not found on update: TUPLE (info_bits=0, 3 fields): {[1]m(0x6D),[4] x(0x0003B678),[40]Global_message_delivery/Targets/Wikidata(0x476C6F62616C5F6D6573736167655F64656C69766572792F546172676574732F57696B6964617461)} at: COMPACT RECORD(info_bits=0, 3 fields): {[1]m(0x6D),[4] (0x0003B5D5),[13]User:Rodejong(0x557365723A526F64656A6F6E67)}
|
Sep 17 18:30:43 db2125 mysqld[3189]: 2020-09-17 18:30:43 9 [ERROR] InnoDB: Record in index `iwl_prefix_from_title` of table `nlwiki`.`iwlinks` was not found on update: TUPLE (info_bits=0, 3 fields): {[1]m(0x6D),[4] x(0x0003B678),[28]Special:MyLanguage/Tech/News(0x5370656369616C3A4D794C616E67756167652F546563682F4E657773)} at: COMPACT RECORD(info_bits=0, 3 fields): {[1]m(0x6D),[4] (0x0003B5D5),[13]User:Rodejong(0x557365723A526F64656A6F6E67)}
|
Sep 17 18:30:43 db2125 mysqld[3189]: 2020-09-17 18:30:43 9 [ERROR] InnoDB: Record in index `iwl_prefix_from_title` of table `nlwiki`.`iwlinks` was not found on update: TUPLE (info_bits=0, 3 fields): {[1]m(0x6D),[4] x(0x0003B678),[36]Special:MyLanguage/Tech/News/2020/38(0x5370656369616C3A4D794C616E67756167652F546563682F4E6577732F323032302F3338)} at: COMPACT RECORD(info_bits=0, 3 fields): {[1]m(0x6D),[4] (0x0003B5D5),[13]User:Rodejong(0x557365723A526F64656A6F6E67)}
|
Sep 17 18:30:43 db2125 mysqld[3189]: 2020-09-17 18:30:43 9 [ERROR] InnoDB: Record in index `iwl_prefix_from_title` of table `nlwiki`.`iwlinks` was not found on update: TUPLE (info_bits=0, 3 fields): {[1]m(0x6D),[4] x(0x0003B678),[36]Special:MyLanguage/Tech/News/Writers(0x5370656369616C3A4D794C616E67756167652F546563682F4E6577732F57726974657273)} at: COMPACT RECORD(info_bits=0, 3 fields): {[1]m(0x6D),[4] (0x0003B5D5),[13]User:Rodejong(0x557365723A526F64656A6F6E67)}
|
Sep 17 18:30:43 db2125 mysqld[3189]: 2020-09-17 18:30:43 9 [ERROR] InnoDB: Record in index `iwl_prefix_from_title` of table `nlwiki`.`iwlinks` was not found on update: TUPLE (info_bits=0, 3 fields): {[1]m(0x6D),[4] x(0x0003B678),[50]Special:MyLanguage/User:MediaWiki_message_delivery(0x5370656369616C3A4D794C616E67756167652F557365723A4D6564696157696B695F6D6573736167655F64656C6976657279)} at: COMPACT RECORD(info_bits=0, 3 fields): {[1]m(0x6D),[4] (0x0003B5D5),[13]User:Rodejong(0x557365723A526F64656A6F6E67)}
|
Sep 17 18:30:43 db2125 mysqld[3189]: 2020-09-17 18:30:43 9 [ERROR] InnoDB: Record in index `iwl_prefix_from_title` of table `nlwiki`.`iwlinks` was not found on update: TUPLE (info_bits=0, 3 fields): {[1]m(0x6D),[4] x(0x0003B678),[14]Talk:Tech/News(0x54616C6B3A546563682F4E657773)} at: COMPACT RECORD(info_bits=0, 3 fields): {[1]m(0x6D),[4] (0x0003B5D5),[13]User:Rodejong(0x557365723A526F64656A6F6E67)}
|
Sep 17 18:30:43 db2125 mysqld[3189]: 2020-09-17 18:30:43 9 [ERROR] InnoDB: Record in index `iwl_prefix_from_title` of table `nlwiki`.`iwlinks` was not found on update: TUPLE (info_bits=0, 3 fields): {[1]m(0x6D),[4] x(0x0003B678),[4]Tech(0x54656368)} at: COMPACT RECORD(info_bits=0, 3 fields): {[1]m(0x6D),[4] (0x0003B5D5),[13]User:Rodejong(0x557365723A526F64656A6F6E67)}
|
Sep 17 18:32:42 db2125 mysqld[3189]: 2020-09-17 18:32:42 9 [ERROR] InnoDB: Record in index `iwl_prefix_title_from` of table `nlwiki`.`iwlinks` was not found on update: TUPLE (info_bits=0, 3 fields): {[1]d(0x64),[28]Wikidata:Status_updates/Next(0x57696B69646174613A5374617475735F757064617465732F4E657874),[4] Kut(0x004B7574)} at: COMPACT RECORD(info_bits=0, 3 fields): {[1]d(0x64),[28]Wikidata:Status_updates/Next(0x57696B69646174613A5374617475735F757064617465732F4E657874),[4] KG (0x004B4712)}
|
Sep 17 19:13:08 db2125 mysqld[3189]: 2020-09-17 19:13:08 8 [Note] Slave: received end packet from server, apparent master shutdown:
|
Sep 17 19:13:08 db2125 mysqld[3189]: 2020-09-17 19:13:08 8 [Note] Slave I/O thread: Failed reading log event, reconnecting to retry, log 'db2107-bin.001694' at position 691042354; GTID position '171970567-171970567-390719905,0-180359173-4880477695,180359173-180359173-70825087,171978786-171978786-2108483244,180359271-180
|
Sep 17 19:17:35 db2125 mysqld[3189]: 2020-09-17 19:17:35 14678274 [ERROR] Event Scheduler: [root@localhost][ops.wmf_slave_wikiuser_sleep] Unknown thread id: 14621591
|
Sep 17 19:17:35 db2125 mysqld[3189]: 2020-09-17 19:17:35 14678274 [Note] Event Scheduler: [root@localhost][ops.wmf_slave_wikiuser_sleep] At line 32 in ops.wmf_slave_wikiuser_sleep
|
Sep 17 19:17:35 db2125 mysqld[3189]: 2020-09-17 19:17:35 14678274 [Note] Event Scheduler: [root@localhost].[ops.wmf_slave_wikiuser_sleep] event execution failed.
|
Sep 17 19:25:13 db2125 mysqld[3189]: 2020-09-17 19:25:13 9 [ERROR] InnoDB: Record in index `cl_timestamp` of table `nlwiki`.`categorylinks` was not found on update: TUPLE (info_bits=0, 3 fields): {[20]Muziekalbum_uit_2016(0x4D757A69656B616C62756D5F7569745F32303136),[4]_c_ (0x5F635F98),[4] R k(0x0052C56B)} at: COMPACT RECORD(info_bits=0, 3 fields): {[20]Muziekalbum_uit_2016(0x4D757A69656B616C62756D5F7569745F32303136),[4]_! S(0x5F21D453),[4] R [(0x00527F5B)}
|
Sep 17 19:45:38 db2125 mysqld[3189]: 2020-09-17 19:45:38 9 [ERROR] InnoDB: Record in index `pl_backlinks_namespace` of table `nowiki`.`pagelinks` was not found on update: TUPLE (info_bits=0, 4 fields): {[4] (0x80000000),[4] (0x80000000),[10]Josh_Allen(0x4A6F73685F416C6C656E),[4] (0x001AD6D3)} at: COMPACT RECORD(info_bits=0, 4 fields): {[4] (0x80000000),[4] (0x80000000),[10]Josh_Allen(0x4A6F73685F416C6C656E),[4] (0x001992D2)}
|
Sep 17 19:45:51 db2125 mysqld[3189]: 2020-09-17 19:45:51 9 [ERROR] InnoDB: Record in index `pl_backlinks_namespace` of table `nowiki`.`pagelinks` was not found on update: TUPLE (info_bits=0, 4 fields): {[4] (0x80000000),[4] (0x80000000),[10]Josh_Allen(0x4A6F73685F416C6C656E),[4] (0x001AD617)} at: COMPACT RECORD(info_bits=0, 4 fields): {[4] (0x80000000),[4] (0x80000000),[10]Josh_Allen(0x4A6F73685F416C6C656E),[4] (0x001992D2)}
|
Sep 17 20:08:44 db2125 mysqld[3189]: 2020-09-17 20:08:44 9 [ERROR] InnoDB: Record in index `pl_namespace` of table `ptwiki`.`pagelinks` was not found on update: TUPLE (info_bits=0, 3 fields): {[4] (0x80000000),[7]Jamaica(0x4A616D61696361),[4] 7?R(0x00373F52)} at: COMPACT RECORD(info_bits=0, 3 fields): {[4] (0x80000000),[7]Jamaica(0x4A616D61696361),[4] 7.5(0x00372E35)}
|
Sep 17 20:08:45 db2125 mysqld[3189]: 2020-09-17 20:08:45 9 [ERROR] InnoDB: Record in index `pl_namespace` of table `ptwiki`.`pagelinks` was not found on update: TUPLE (info_bits=0, 3 fields): {[4] (0x80000000),[7]Jamaica(0x4A616D61696361),[4] 4 h(0x0034C868)} at: COMPACT RECORD(info_bits=0, 3 fields): {[4] (0x80000000),[7]Jamaica(0x4A616D61696361),[4] 4 (0x0034B01F)}
|
Sep 17 20:08:45 db2125 mysqld[3189]: 2020-09-17 20:08:45 9 [ERROR] InnoDB: Record in index `pl_namespace` of table `ptwiki`.`pagelinks` was not found on update: TUPLE (info_bits=0, 3 fields): {[4] (0x80000000),[7]Jamaica(0x4A616D61696361),[4] ; (0x003BA4E3)} at: COMPACT RECORD(info_bits=0, 3 fields): {[4] (0x80000000),[7]Jamaica(0x4A616D61696361),[4] ; g(0x003BA367)}
|
Sep 17 20:08:45 db2125 mysqld[3189]: 2020-09-17 20:08:45 9 [ERROR] InnoDB: Record in index `pl_namespace` of table `ptwiki`.`pagelinks` was not found on update: TUPLE (info_bits=0, 3 fields): {[4] (0x80000000),[7]Jamaica(0x4A616D61696361),[4] =C (0x003D4384)} at: COMPACT RECORD(info_bits=0, 3 fields): {[4] (0x80000000),[7]Jamaica(0x4A616D61696361),[4] =@H(0x003D4048)}
|
Sep 17 20:08:45 db2125 mysqld[3189]: 2020-09-17 20:08:45 9 [ERROR] InnoDB: Record in index `pl_namespace` of table `ptwiki`.`pagelinks` was not found on update: TUPLE (info_bits=0, 3 fields): {[4] (0x80000000),[7]Jamaica(0x4A616D61696361),[4] >' (0x003E27D3)} at: COMPACT RECORD(info_bits=0, 3 fields): {[4] (0x80000000),[7]Jamaica(0x4A616D61696361),[4] > z(0x003E0D7A)}
|
Sep 17 20:08:46 db2125 mysqld[3189]: 2020-09-17 20:08:46 9 [ERROR] InnoDB: Record in index `pl_namespace` of table `ptwiki`.`pagelinks` was not found on update: TUPLE (info_bits=0, 3 fields): {[4] (0x80000000),[7]Jamaica(0x4A616D61696361),[4] 4 (0x0034F61A)} at: COMPACT RECORD(info_bits=0, 3 fields): {[4] (0x80000000),[7]Jamaica(0x4A616D61696361),[4] 4 (0x0034CBED)}
|
Sep 17 20:18:36 db2125 mysqld[3189]: 2020-09-17 20:18:36 8 [ERROR] Error reading packet from server: Lost connection to MySQL server during query (server_errno=2013)
|
Sep 17 20:18:36 db2125 mysqld[3189]: 2020-09-17 20:18:36 8 [Note] Slave I/O thread: Failed reading log event, reconnecting to retry, log 'db2107-bin.001694' at position 826990955; GTID position '171970567-171970567-390719905,0-180359173-4880477695,180359173-180359173-70825087,171978786-171978786-2108483244,180359271-180
|
Sep 17 21:27:49 db2125 mysqld[3189]: 2020-09-17 21:27:49 8 [ERROR] Error reading packet from server: Lost connection to MySQL server during query (server_errno=2013)
|
Sep 17 21:27:49 db2125 mysqld[3189]: 2020-09-17 21:27:49 8 [Note] Slave I/O thread: Failed reading log event, reconnecting to retry, log 'db2107-bin.001694' at position 958867421; GTID position '171970567-171970567-390719905,0-180359173-4880477695,180359173-180359173-70825087,171978786-171978786-2108483244,180359271-180
|
Sep 17 22:48:59 db2125 mysqld[3189]: 2020-09-17 22:48:59 8 [ERROR] Error reading packet from server: Lost connection to MySQL server during query (server_errno=2013)
|
Sep 17 22:48:59 db2125 mysqld[3189]: 2020-09-17 22:48:59 8 [Note] Slave I/O thread: Failed reading log event, reconnecting to retry, log 'db2107-bin.001695' at position 42660168; GTID position '171970567-171970567-390719905,0-180359173-4880477695,180359173-180359173-70825087,171978786-171978786-2108483244,180359271-1803
|
Sep 17 23:32:55 db2125 mysqld[3189]: 2020-09-17 23:32:55 9 [ERROR] InnoDB: Record in index `pl_backlinks_namespace` of table `enwiktionary`.`pagelinks` was not found on update: TUPLE (info_bits=0, 4 fields): {[4] (0x80000003),[4] (0x80000002),[6]Eirikr(0x456972696B72),[4] FU>(0x0046553E)} at: COMPACT RECORD(info_bits=0, 4 fields): {[4] (0x80000003),[4] (0x80000002),[6]Eirikr(0x456972696B72),[4] FK (0x00464BED)}
|
Sep 18 00:13:35 db2125 mysqld[3189]: 2020-09-18 0:13:35 27595501 [ERROR] Event Scheduler: [root@localhost][ops.wmf_slave_wikiuser_sleep] Unknown thread id: 27556115
|
Sep 18 00:13:35 db2125 mysqld[3189]: 2020-09-18 0:13:35 27595501 [Note] Event Scheduler: [root@localhost][ops.wmf_slave_wikiuser_sleep] At line 32 in ops.wmf_slave_wikiuser_sleep
|
Sep 18 00:13:35 db2125 mysqld[3189]: 2020-09-18 0:13:35 27595501 [Note] Event Scheduler: [root@localhost].[ops.wmf_slave_wikiuser_sleep] event execution failed.
|
Sep 18 00:44:43 db2125 mysqld[3189]: 2020-09-18 0:44:43 8 [Note] Slave: received end packet from server, apparent master shutdown:
|
Sep 18 00:44:43 db2125 mysqld[3189]: 2020-09-18 0:44:43 8 [Note] Slave I/O thread: Failed reading log event, reconnecting to retry, log 'db2107-bin.001695' at position 211812871; GTID position '171970567-171970567-390719905,0-180359173-4880477695,180359173-180359173-70825087,171978786-171978786-2108483244,180359271-180
|
Sep 18 01:58:58 db2125 mysqld[3189]: 2020-09-18 1:58:58 8 [ERROR] Error reading packet from server: Lost connection to MySQL server during query (server_errno=2013)
|
Sep 18 01:58:58 db2125 mysqld[3189]: 2020-09-18 1:58:58 8 [Note] Slave I/O thread: Failed reading log event, reconnecting to retry, log 'db2107-bin.001695' at position 312491366; GTID position '171970567-171970567-390719905,0-180359173-4880477695,180359173-180359173-70825087,171978786-171978786-2108483244,180359271-180
|
Sep 18 02:16:05 db2125 mysqld[3189]: 2020-09-18 2:16:05 32817518 [ERROR] Event Scheduler: [root@localhost][ops.wmf_slave_wikiuser_sleep] Unknown thread id: 32767149
|
Sep 18 02:16:05 db2125 mysqld[3189]: 2020-09-18 2:16:05 32817518 [Note] Event Scheduler: [root@localhost][ops.wmf_slave_wikiuser_sleep] At line 32 in ops.wmf_slave_wikiuser_sleep
|
Sep 18 02:16:05 db2125 mysqld[3189]: 2020-09-18 2:16:05 32817518 [Note] Event Scheduler: [root@localhost].[ops.wmf_slave_wikiuser_sleep] event execution failed.
|
Sep 18 02:22:35 db2125 mysqld[3189]: 2020-09-18 2:22:35 33108367 [ERROR] Event Scheduler: [root@localhost][ops.wmf_slave_wikiuser_sleep] Unknown thread id: 33049722
|
Sep 18 02:22:35 db2125 mysqld[3189]: 2020-09-18 2:22:35 33108367 [Note] Event Scheduler: [root@localhost][ops.wmf_slave_wikiuser_sleep] At line 32 in ops.wmf_slave_wikiuser_sleep
|
Sep 18 02:22:35 db2125 mysqld[3189]: 2020-09-18 2:22:35 33108367 [Note] Event Scheduler: [root@localhost].[ops.wmf_slave_wikiuser_sleep] event execution failed.
|
Sep 18 04:05:41 db2125 mysqld[3189]: 2020-09-18 4:05:41 37696350 [ERROR] InnoDB: Clustered record for sec rec not found index `eu_page_id` of table `zhwiki`.`wbc_entity_usage`
|
Sep 18 04:05:41 db2125 mysqld[3189]: InnoDB: sec index record PHYSICAL RECORD: n_fields 3; compact format; info bits 0
|
Sep 18 04:05:41 db2125 mysqld[3189]: 0: len 4; hex 802e651a; asc .e ;;
|
Sep 18 04:05:41 db2125 mysqld[3189]: 1: len 8; hex 5132323230393037; asc Q2220907;;
|
Sep 18 04:05:41 db2125 mysqld[3189]: 2: len 8; hex 80000000028839df; asc 9 ;;
|
Sep 18 04:05:41 db2125 mysqld[3189]: InnoDB: clust index record PHYSICAL RECORD: n_fields 6; compact format; info bits 0
|
Sep 18 04:05:41 db2125 mysqld[3189]: 0: len 8; hex 80000000028839dd; asc 9 ;;
|
Sep 18 04:05:41 db2125 mysqld[3189]: 1: len 6; hex 0007821d1bff; asc ;;
|
Sep 18 04:05:41 db2125 mysqld[3189]: 2: len 7; hex 0c0003c02f1913; asc / ;;
|
Sep 18 04:05:41 db2125 mysqld[3189]: 3: len 6; hex 513739393335; asc Q79935;;
|
Sep 18 04:05:41 db2125 mysqld[3189]: 4: len 1; hex 53; asc S;;
|
Sep 18 04:05:41 db2125 mysqld[3189]: 5: len 4; hex 8002db18; asc ;;
|
Sep 18 04:05:41 db2125 mysqld[3189]: TRANSACTION 422056457627896, ACTIVE 1 sec starting index read
|
Sep 18 04:05:41 db2125 mysqld[3189]: mysql tables in use 1, locked 0
|
Sep 18 04:05:41 db2125 mysqld[3189]: 0 lock struct(s), heap size 1128, 0 row lock(s)
|
Sep 18 04:05:41 db2125 mysqld[3189]: MySQL thread id 37696350, OS thread handle 140168401090304, query id 560925894 10.192.32.117 wikiuser Sending data
|
Sep 18 04:05:41 db2125 mysqld[3189]: SELECT /* Wikibase\Client\Usage\Sql\EntityUsageTable::queryUsages */ eu_aspect,eu_entity_id FROM `wbc_entity_usage` WHERE eu_page_id = xxxx
|
Sep 18 04:05:41 db2125 mysqld[3189]: InnoDB: Submit a detailed bug report to https://jira.mariadb.org/
|
Sep 18 04:05:41 db2125 mysqld[3189]: 2020-09-18 4:05:41 37696350 [ERROR] InnoDB: Clustered record for sec rec not found index `eu_page_id` of table `zhwiki`.`wbc_entity_usage`
|
Sep 18 04:05:41 db2125 mysqld[3189]: InnoDB: sec index record PHYSICAL RECORD: n_fields 3; compact format; info bits 0
|
Sep 18 04:05:41 db2125 mysqld[3189]: 0: len 4; hex 802e651a; asc .e ;;
|
Sep 18 04:05:41 db2125 mysqld[3189]: 1: len 8; hex 5132323230393037; asc Q2220907;;
|
Sep 18 04:05:41 db2125 mysqld[3189]: 2: len 8; hex 800000000633fb04; asc 3 ;;
|
Sep 18 04:05:41 db2125 mysqld[3189]: InnoDB: clust index record PHYSICAL RECORD: n_fields 6; compact format; info bits 0
|
Sep 18 04:05:41 db2125 mysqld[3189]: 0: len 8; hex 800000000633fb03; asc 3 ;;
|
Sep 18 04:05:41 db2125 mysqld[3189]: 1: len 6; hex 0009b81afd6c; asc l;;
|
Sep 18 04:05:41 db2125 mysqld[3189]: 2: len 7; hex 830003800c0110; asc ;;
|
Sep 18 04:05:41 db2125 mysqld[3189]: 3: len 8; hex 5132363037323632; asc Q2607262;;
|
Sep 18 04:05:41 db2125 mysqld[3189]: 4: len 6; hex 432e50363235; asc C.P625;;
|
Sep 18 04:05:41 db2125 mysqld[3189]: 5: len 4; hex 805776a1; asc Wv ;;
|
Sep 18 04:05:41 db2125 mysqld[3189]: TRANSACTION 422056457627896, ACTIVE 1 sec starting index read
|
Sep 18 04:05:41 db2125 mysqld[3189]: mysql tables in use 1, locked 0
|
Sep 18 04:05:41 db2125 mysqld[3189]: 0 lock struct(s), heap size 1128, 0 row lock(s)
|
Sep 18 04:05:41 db2125 mysqld[3189]: MySQL thread id 37696350, OS thread handle 140168401090304, query id 560925894 10.192.32.117 wikiuser Sending data
|
Sep 18 04:05:41 db2125 mysqld[3189]: SELECT /* Wikibase\Client\Usage\Sql\EntityUsageTable::queryUsages */ eu_aspect,eu_entity_id FROM `wbc_entity_usage` WHERE eu_page_id = xxx
|
Sep 18 04:05:41 db2125 mysqld[3189]: InnoDB: Submit a detailed bug report to https://jira.mariadb.org/
|
Sep 18 04:15:05 db2125 mysqld[3189]: 2020-09-18 4:15:05 38072789 [ERROR] Event Scheduler: [root@localhost][ops.wmf_slave_wikiuser_sleep] Unknown thread id: 38025135
|
Sep 18 04:15:05 db2125 mysqld[3189]: 2020-09-18 4:15:05 38072789 [Note] Event Scheduler: [root@localhost][ops.wmf_slave_wikiuser_sleep] At line 32 in ops.wmf_slave_wikiuser_sleep
|
Sep 18 04:15:05 db2125 mysqld[3189]: 2020-09-18 4:15:05 38072789 [Note] Event Scheduler: [root@localhost].[ops.wmf_slave_wikiuser_sleep] event execution failed.
|
Sep 18 05:01:50 db2125 mysqld[3189]: 2020-09-18 5:01:50 8 [ERROR] Error reading packet from server: Lost connection to MySQL server during query (server_errno=2013)
|
Sep 18 05:01:50 db2125 mysqld[3189]: 2020-09-18 5:01:50 8 [Note] Slave I/O thread: Failed reading log event, reconnecting to retry, log 'db2107-bin.001695' at position 536738264; GTID position '171970567-171970567-390719905,0-180359173-4880477695,180359173-180359173-70825087,171978786-171978786-2108483244,180359271-180
|
Sep 18 06:49:25 db2125 mysqld[3189]: 2020-09-18 6:49:25 9 [ERROR] InnoDB: Record in index `linter_page` of table `idwiki`.`linter` was not found on update: TUPLE (info_bits=0, 2 fields): {[4] (0x0020E208),[4] r, (0x00722CFA)} at: COMPACT RECORD(info_bits=0, 2 fields): {[4] (0x0020E208),[4] r s(0x00721E73)}
|
Sep 18 06:54:28 db2125 mysqld[3189]: 2020-09-18 6:54:28 9 [ERROR] InnoDB: Record in index `pl_namespace` of table `idwiki`.`pagelinks` was not found on update: TUPLE (info_bits=0, 3 fields): {[4] (0x80000000),[11]Locker_Room(0x4C6F636B65725F526F6F6D),[4] - (0x00172DC8)} at: COMPACT RECORD(info_bits=0, 3 fields): {[4] (0x80000000),[11]Locker_Room(0x4C6F636B65725F526F6F6D),[4] ! (0x00092116)}
|
Sep 18 07:14:40 db2125 mysqld[3189]: 2020-09-18 7:14:40 8 [ERROR] Error reading packet from server: Lost connection to MySQL server during query (server_errno=2013)
|
Sep 18 07:14:40 db2125 mysqld[3189]: 2020-09-18 7:14:40 8 [Note] Slave I/O thread: Failed reading log event, reconnecting to retry, log 'db2107-bin.001695' at position 717494783; GTID position '171970567-171970567-390719905,0-180359173-4880477695,180359173-180359173-70825087,171978786-171978786-2108483244,180359271-180
|
Sep 18 08:39:44 db2125 mysqld[3189]: 2020-09-18 8:39:44 8 [ERROR] Error reading packet from server: Lost connection to MySQL server during query (server_errno=2013)
|
Sep 18 08:39:44 db2125 mysqld[3189]: 2020-09-18 8:39:44 8 [Note] Slave I/O thread: Failed reading log event, reconnecting to retry, log 'db2107-bin.001695' at position 832320892; GTID position '171970567-171970567-390719905,0-180359173-4880477695,180359173-180359173-70825087,171978786-171978786-2108483244,180359271-180
|
Sep 18 09:35:40 db2125 mysqld[3189]: 2020-09-18 9:35:40 0 [ERROR] InnoDB: Unable to find a record to delete-mark
|
Sep 18 09:35:40 db2125 mysqld[3189]: InnoDB: tuple DATA TUPLE: 4 fields;
|
Sep 18 09:35:40 db2125 mysqld[3189]: 0: len 4; hex 80000004; asc ;;
|
Sep 18 09:35:40 db2125 mysqld[3189]: 1: len 4; hex 80000002; asc ;;
|
Sep 18 09:35:40 db2125 mysqld[3189]: 2: len 6; hex 4861726f6c64; asc Harold;;
|
Sep 18 09:35:40 db2125 mysqld[3189]: 3: len 4; hex 000529e7; asc ) ;;
|
Sep 18 09:35:40 db2125 mysqld[3189]: InnoDB: record PHYSICAL RECORD: n_fields 4; compact format; info bits 0
|
Sep 18 09:35:40 db2125 mysqld[3189]: 0: len 4; hex 80000004; asc ;;
|
Sep 18 09:35:40 db2125 mysqld[3189]: 1: len 4; hex 80000002; asc ;;
|
Sep 18 09:35:40 db2125 mysqld[3189]: 2: len 6; hex 4861726f6c64; asc Harold;;
|
Sep 18 09:35:40 db2125 mysqld[3189]: 3: len 4; hex 00052839; asc (9;;
|
Sep 18 09:35:40 db2125 mysqld[3189]: 2020-09-18 9:35:40 0 [ERROR] InnoDB: page [page id: space=7300, page number=23852] (477 records, index id 26549).
|
Sep 18 09:35:40 db2125 mysqld[3189]: 2020-09-18 9:35:40 0 [ERROR] InnoDB: Submit a detailed bug report to https://jira.mariadb.org/
|
Sep 18 09:47:10 db2125 mysqld[3189]: 2020-09-18 9:47:10 9 [ERROR] InnoDB: Record in index `pl_namespace` of table `idwiki`.`pagelinks` was not found on update: TUPLE (info_bits=0, 3 fields): {[4] (0x80000000),[11]Locker_Room(0x4C6F636B65725F526F6F6D),[4] - (0x00172DF1)} at: COMPACT RECORD(info_bits=0, 3 fields): {[4] (0x80000000),[11]Locker_Room(0x4C6F636B65725F526F6F6D),[4] ! (0x00092116)}
|
Sep 18 10:42:20 db2125 mysqld[3189]: 2020-09-18 10:42:20 8 [Note] Slave: received end packet from server, apparent master shutdown:
|
Sep 18 10:42:20 db2125 mysqld[3189]: 2020-09-18 10:42:20 8 [Note] Slave I/O thread: Failed reading log event, reconnecting to retry, log 'db2107-bin.001695' at position 1024661670; GTID position '171970567-171970567-390719905,0-180359173-4880477695,180359173-180359173-70825087,171978786-171978786-2108483244,180359271-18
|
Sep 18 10:58:35 db2125 mysqld[3189]: 2020-09-18 10:58:35 57025234 [ERROR] Event Scheduler: [root@localhost][ops.wmf_slave_wikiuser_sleep] Unknown thread id: 56914946
|
Sep 18 10:58:35 db2125 mysqld[3189]: 2020-09-18 10:58:35 57025234 [Note] Event Scheduler: [root@localhost][ops.wmf_slave_wikiuser_sleep] At line 32 in ops.wmf_slave_wikiuser_sleep
|
Sep 18 10:58:35 db2125 mysqld[3189]: 2020-09-18 10:58:35 57025234 [Note] Event Scheduler: [root@localhost].[ops.wmf_slave_wikiuser_sleep] event execution failed.
|
Sep 18 10:59:35 db2125 mysqld[3189]: 2020-09-18 10:59:35 57078930 [ERROR] Event Scheduler: [root@localhost][ops.wmf_slave_wikiuser_sleep] Unknown thread id: 56870567
|
Sep 18 10:59:35 db2125 mysqld[3189]: 2020-09-18 10:59:35 57078930 [Note] Event Scheduler: [root@localhost][ops.wmf_slave_wikiuser_sleep] At line 32 in ops.wmf_slave_wikiuser_sleep
|
Sep 18 10:59:35 db2125 mysqld[3189]: 2020-09-18 10:59:35 57078930 [Note] Event Scheduler: [root@localhost].[ops.wmf_slave_wikiuser_sleep] event execution failed.
|
Sep 18 11:10:32 db2125 mysqld[3189]: 2020-09-18 11:10:32 0x7fdbf8187700 InnoDB: Assertion failure in file /root/mariadb-10.4.14/storage/innobase/row/row0ins.cc line 218
|
Sep 18 11:10:32 db2125 mysqld[3189]: InnoDB: Failing assertion: !cursor->index->is_committed()
|
Sep 18 11:10:32 db2125 mysqld[3189]: InnoDB: We intentionally generate a memory trap.
|
Sep 18 11:10:32 db2125 mysqld[3189]: InnoDB: Submit a detailed bug report to https://jira.mariadb.org/
|
Sep 18 11:10:32 db2125 mysqld[3189]: InnoDB: If you get repeated assertion failures or crashes, even
|
Sep 18 11:10:32 db2125 mysqld[3189]: InnoDB: immediately after the mysqld startup, there may be
|
Sep 18 11:10:32 db2125 mysqld[3189]: InnoDB: corruption in the InnoDB tablespace. Please refer to
|
Sep 18 11:10:32 db2125 mysqld[3189]: InnoDB: https://mariadb.com/kb/en/library/innodb-recovery-modes/
|
Sep 18 11:10:32 db2125 mysqld[3189]: InnoDB: about forcing recovery.
|
Sep 18 11:10:32 db2125 mysqld[3189]: 200918 11:10:32 [ERROR] mysqld got signal 6 ;
|
Sep 18 11:10:32 db2125 mysqld[3189]: This could be because you hit a bug. It is also possible that this binary
|
Sep 18 11:10:32 db2125 mysqld[3189]: or one of the libraries it was linked against is corrupt, improperly built,
|
Sep 18 11:10:32 db2125 mysqld[3189]: or misconfigured. This error can also be caused by malfunctioning hardware.
|
Sep 18 11:10:32 db2125 mysqld[3189]: To report this bug, see https://mariadb.com/kb/en/reporting-bugs
|
Sep 18 11:10:32 db2125 mysqld[3189]: We will try our best to scrape up some info that will hopefully help
|
Sep 18 11:10:32 db2125 mysqld[3189]: diagnose the problem, but since we have already crashed,
|
Sep 18 11:10:32 db2125 mysqld[3189]: something is definitely wrong and this may fail.
|
Sep 18 11:10:32 db2125 mysqld[3189]: Server version: 10.4.14-MariaDB-log
|
Sep 18 11:10:32 db2125 mysqld[3189]: key_buffer_size=134217728
|
Sep 18 11:10:32 db2125 mysqld[3189]: read_buffer_size=131072
|
Sep 18 11:10:32 db2125 mysqld[3189]: max_used_connections=581
|
Sep 18 11:10:32 db2125 mysqld[3189]: max_threads=2010
|
Sep 18 11:10:32 db2125 mysqld[3189]: thread_count=152
|
Sep 18 11:10:32 db2125 mysqld[3189]: It is possible that mysqld could use up to
|
Sep 18 11:10:32 db2125 mysqld[3189]: key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 4752266 K bytes of memory
|
Sep 18 11:10:32 db2125 mysqld[3189]: Hope that's ok; if not, decrease some variables in the equation.
|
Sep 18 11:10:32 db2125 mysqld[3189]: Thread pointer: 0x7f7b840014f8
|
Sep 18 11:10:32 db2125 mysqld[3189]: Attempting backtrace. You can use the following information to find out
|
Sep 18 11:10:32 db2125 mysqld[3189]: where mysqld died. If you see no messages after this, something went
|
Sep 18 11:10:32 db2125 mysqld[3189]: terribly wrong...
|
Sep 18 11:10:32 db2125 mysqld[3189]: stack_bottom = 0x7fdbf8186698 thread_stack 0x30000
|
Sep 18 11:10:35 db2125 mysqld[3189]: *** buffer overflow detected ***: /opt/wmf-mariadb104/bin/mysqld terminated
|
Sep 18 11:10:44 db2125 systemd[1]: mariadb.service: Main process exited, code=killed, status=6/ABRT
|
For the record:
mysql:root@localhost [(none)]> show global variables like '%change%';
|
+-------------------------------+-------+
|
| Variable_name | Value |
|
+-------------------------------+-------+
|
| innodb_change_buffer_max_size | 25 |
|
| innodb_change_buffering | all |
|
| session_track_state_change | OFF |
|
+-------------------------------+-------+
|
I don't have any suggestions on how to reproduce this really (way too many variables to be coherent) but I think I likely encountered this problem in 10.3.24. After parsing out the affected tables and rebuilding each with ALTER TABLE t1 ENGINE=INNODB the problem I don't think is manifesting further.
This is how I got a list of tables before deduping:
grep '^space.*offset.*records.*index id' /var/log/mysqld.log | awk '{print $2}' | sort | uniq |
grep -h 'space=' /var/log/mysqld.log | awk '$9 ~ /space=/ {print $9}' | sed 's/space=//g;s/,$//g' | sort | uniq | sort -nk1 |
Then generate a list to rebuild, eg:
select concat('alter table `',replace(name, '/', '`.`'), '` engine=innodb;') from information_schema.innodb_sys_tables where space in (82125909); |
Finally, for good measure I did a shutdown with innodb_fast_shutdown=0, deleted /data/ib_buffer_pool, and started back up.
Replication never seemed to really have any trouble, presumably because binlog_format=row was in use so tables were looked up by non-secondary index (via PK)
This comment is not likely useful to devs (besides my version being 10.3 series) but perhaps it would be useful to users that run in to a similar problem. I do wonder if this is a variation of an old upstream bug 73767
(edit formatting)
Had a couple different ones pop up with a new error, this helped:
awk '/InnoDB: Record in index/ {print $12}' /var/log/mysqld.log | sort | uniq | sed 's/^/alter table /g;s/$/ engine=innodb;/g' |
Error was like:
[ERROR] InnoDB: Record in index `IX_foo` of table `db1`.`t2` was not found on update: TUPLE (info_bits=0, 2 fields): {[4] (0x80000000),[4] ... |
I did hit an assert but I checked the Table_map on the next GTID position from the primary's binary log, "repaired" that table. I've not see a table show up again after it was itself fixed. For now, I have change buffer off as an additional attempt to salvage this slave, which I can't easily repair right now. Again, probably overkill, but after letting replication run for some time and seeing new tables with the error, I stop replication, fix those, do a slow shutdown. Probably good to have skip-slave-start if you are in this situation for a while.
marostegui, I just noticed your following claim:
After a logical reimport, if we configure replication, start it and DO NOT restart mysql daemon, this doesn't happen.
This would suggest some problem in InnoDB shutdown or startup. How would you restart the server? What is the value of innodb_fast_shutdown?
I would request you to narrow this down by trying the following parameters, which may significantly reduce performance:
- Restart with innodb_purge_threads=1. The default was changed to 4 in MySQL 5.7 and MariaDB 10.2.
- SET GLOBAL innodb_change_buffering=changes; or SET GLOBAL innodb_change_buffering=none;. I think that this is a somewhat unlikely cause, because
MDEV-21790claims that 5.5 worked just fine, and the change buffering was introduced in that version. - SET GLOBAL innodb_adaptive_hash_index=0; for reasons given in
MDEV-20487. (It might be that MDEV-20203 has been fixed byMDEV-22901and related work.)
My primary suspect is the multi-threaded purge. Our internal testing is conducted primarily on debug builds and rather small amounts of data, so it might not catch this.
For changing any of these parameters, I think that it is safest to start from the scratch (initialize from a logical dump), to eliminate the possibility that some dormant corruption might already be present in the data files.
Thanks Marko
Our default is:
+----------------------+-------+
|
| Variable_name | Value |
|
+----------------------+-------+
|
| innodb_fast_shutdown | 1 |
|
+----------------------+-------+
|
We are already experimenting with innodb_change_buffering=inserts, first checking that it doesn't impact any other thing. If it goes well, we might deploy it across the infrastructure.
Our default for purge_threads is already=1 everywhere.
We have a host that is crashing often with the same error, but it comes from a HW crash, it is probably not useful to troubleshoot things anymore, as the data is most likely corrupted.
Just to test, on that host, I set innodb_change_buffering=inserts to see if it would stop the following errors to happen:
ct 02 03:42:02 db2125 mysqld[20253]: 2020-10-02 3:42:02 1 [ERROR] InnoDB: Unable to find a record to delete-mark
|
Oct 02 03:42:02 db2125 mysqld[20253]: InnoDB: tuple DATA TUPLE: 3 fields;
|
Oct 02 03:42:02 db2125 mysqld[20253]: 0: len 4; hex 000252fe; asc R ;;
|
Oct 02 03:42:02 db2125 mysqld[20253]: 1: SQL NULL;
|
Oct 02 03:42:02 db2125 mysqld[20253]: 2: len 4; hex 00b022d6; asc " ;;
|
Oct 02 03:42:02 db2125 mysqld[20253]: InnoDB: record PHYSICAL RECORD: n_fields 3; compact format; info bits 0
|
Oct 02 03:42:02 db2125 mysqld[20253]: 0: len 4; hex 000252fe; asc R ;;
|
Oct 02 03:42:02 db2125 mysqld[20253]: 1: SQL NULL;
|
Oct 02 03:42:02 db2125 mysqld[20253]: 2: len 4; hex 00b01e65; asc e;;
|
Those haven't stopped by doing innodb_change_buffering=inserts . But as I said, the data is most likely already corrupted after the HW crash. But as it will take a few days until we can replace the malfuctioning hardware and the host is out of rotation, I am happy to try things there. Data corruption won't be fixed, but maybe we can try to debug things there and see if we can find what's the source of those crashes.
I just set there the following:
mysql:root@localhost [(none)]> SET GLOBAL innodb_change_buffering=changes;
|
Query OK, 0 rows affected (0.000 sec)
|
Let's see if the host keeps crashing the MySQL server.
Any idea also why mariadb shows a buffer overflow after all these crashes?
Oct 01 17:49:44 db2125 mysqld[23951]: 2020-10-01 17:49:44 0x7f91dc1af700 InnoDB: Assertion failure in file /root/mariadb-10.4.14/storage/innobase/row/row0ins.cc line 218
|
Oct 01 17:49:44 db2125 mysqld[23951]: InnoDB: Failing assertion: !cursor->index->is_committed()
|
Oct 01 17:49:44 db2125 mysqld[23951]: InnoDB: We intentionally generate a memory trap.
|
Oct 01 17:49:44 db2125 mysqld[23951]: InnoDB: Submit a detailed bug report to https://jira.mariadb.org/
|
Oct 01 17:49:44 db2125 mysqld[23951]: InnoDB: If you get repeated assertion failures or crashes, even
|
Oct 01 17:49:44 db2125 mysqld[23951]: InnoDB: immediately after the mysqld startup, there may be
|
Oct 01 17:49:44 db2125 mysqld[23951]: InnoDB: corruption in the InnoDB tablespace. Please refer to
|
Oct 01 17:49:44 db2125 mysqld[23951]: InnoDB: https://mariadb.com/kb/en/library/innodb-recovery-modes/
|
Oct 01 17:49:44 db2125 mysqld[23951]: InnoDB: about forcing recovery.
|
Oct 01 17:49:44 db2125 mysqld[23951]: 201001 17:49:44 [ERROR] mysqld got signal 6 ;
|
Oct 01 17:49:44 db2125 mysqld[23951]: This could be because you hit a bug. It is also possible that this binary
|
Oct 01 17:49:44 db2125 mysqld[23951]: or one of the libraries it was linked against is corrupt, improperly built,
|
Oct 01 17:49:44 db2125 mysqld[23951]: or misconfigured. This error can also be caused by malfunctioning hardware.
|
Oct 01 17:49:44 db2125 mysqld[23951]: To report this bug, see https://mariadb.com/kb/en/reporting-bugs
|
Oct 01 17:49:44 db2125 mysqld[23951]: We will try our best to scrape up some info that will hopefully help
|
Oct 01 17:49:44 db2125 mysqld[23951]: diagnose the problem, but since we have already crashed,
|
Oct 01 17:49:44 db2125 mysqld[23951]: something is definitely wrong and this may fail.
|
Oct 01 17:49:44 db2125 mysqld[23951]: Server version: 10.4.14-MariaDB-log
|
Oct 01 17:49:44 db2125 mysqld[23951]: key_buffer_size=134217728
|
Oct 01 17:49:44 db2125 mysqld[23951]: read_buffer_size=131072
|
Oct 01 17:49:44 db2125 mysqld[23951]: max_used_connections=15
|
Oct 01 17:49:44 db2125 mysqld[23951]: max_threads=2010
|
Oct 01 17:49:44 db2125 mysqld[23951]: thread_count=17
|
Oct 01 17:49:44 db2125 mysqld[23951]: It is possible that mysqld could use up to
|
Oct 01 17:49:44 db2125 mysqld[23951]: key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 4752266 K bytes of memory
|
Oct 01 17:49:44 db2125 mysqld[23951]: Hope that's ok; if not, decrease some variables in the equation.
|
Oct 01 17:49:44 db2125 mysqld[23951]: Thread pointer: 0x7f314c0014f8
|
Oct 01 17:49:44 db2125 mysqld[23951]: Attempting backtrace. You can use the following information to find out
|
Oct 01 17:49:44 db2125 mysqld[23951]: where mysqld died. If you see no messages after this, something went
|
Oct 01 17:49:44 db2125 mysqld[23951]: terribly wrong...
|
Oct 01 17:49:44 db2125 mysqld[23951]: stack_bottom = 0x7f91dc1ae698 thread_stack 0x30000
|
Oct 01 17:49:45 db2125 mysqld[23951]: *** buffer overflow detected ***: /opt/wmf-mariadb104/bin/mysqld terminated
|
I was noticing the buffer overflow in a few crashes - looking at MDEV-21329 it appears to be in the address resolution function my_addr_resolve
I have seen our stack trace reporter crashing as well. I think that it was a bad idea to introduce the fork() and exec of addr2line to our stack trace reporter. Yes, for some reason, the symbols for InnoDB functions were often not resolved with the old code. Someone should figure out why that is the case. In my opinion, the stack trace of only the current thread is often rather useless for InnoDB bugs. We would very often need stack traces of all threads.
marostegui, it is interesting to see that you were already running with innodb_purge_threads=1. Also, innodb_change_buffering=inserts is a more restrictive value than the changes that I suggested. The default all would allow all 3 types of operations to be buffered in secondary indexes: inserts, delete-marks, and purge of delete-marked records. You should keep the inserts.
Any changes to configuration can be useless if the data files are already corrupted in some way. We can only truly narrow this down by starting with a specific configuration and a logical data dump. How often do you run CHECK TABLE? It should detect this corruption (and with the MDEV-22924 fix, hopefully not produce false alarms).
Another crash dump that I incorrectly posted on another bug report (MDEV-23653) so posting it here for the record:
-- The job identifier is 14462.
|
Sep 30 12:28:39 db2125 mysqld[30220]: 2020-09-30 12:28:39 8 [Note] Slave I/O thread: connected to master 'repl@db2107.codfw.wmnet:3306',replication starts at GTID position '171970567-171970567-390719905,0-180359173-4880477695,180359173-180359173-70825087,171978786-171978786-2108483244,180359271-180359271-191513097,171966574-171966574-2221092918,180359241-180359241-146491165,171966670-171966670-2410812544,180363270-180363270-170'
|
Sep 30 12:43:10 db2125 mysqld[30220]: 2020-09-30 12:43:10 9 [ERROR] InnoDB: Record in index `namespace_title` of table `zhwiki`.`watchlist` was not found on update: TUPLE (info_bits=0, 3 fields): {[4] (0x8000000B),[21]Taxonomy/Acmaeopleura(0x5461786F6E6F6D792F41636D61656F706C65757261),[4] &" (0x012622B1)} at: COMPACT RECORD(info_bits=0, 3 fields): {[4] (0x8000000B),[18]Taxonomy/Acmaeidae(0x5461786F6E6F6D792F41636D616569646165),[4] (0x00B7838E)}
|
Sep 30 12:43:10 db2125 mysqld[30220]: 2020-09-30 12:43:10 9 [ERROR] InnoDB: Record in index `namespace_title` of table `zhwiki`.`watchlist` was not found on update: TUPLE (info_bits=0, 3 fields): {[4] (0x8000000B),[16]Taxonomy/Acartia(0x5461786F6E6F6D792F41636172746961),[4] &# (0x01262385)} at: COMPACT RECORD(info_bits=0, 3 fields): {[4] (0x8000000B),[20]Taxonomy/Acariformes(0x5461786F6E6F6D792F4163617269666F726D6573),[4] (0x00D6A3C3)}
|
Sep 30 12:43:10 db2125 mysqld[30220]: 2020-09-30 12:43:10 9 [ERROR] InnoDB: Record in index `namespace_title` of table `zhwiki`.`watchlist` was not found on update: TUPLE (info_bits=0, 3 fields): {[4] (0x8000000B),[19]Taxonomy/Acartiidae(0x5461786F6E6F6D792F41636172746969646165),[4] &# (0x01262389)} at: COMPACT RECORD(info_bits=0, 3 fields): {[4] (0x8000000B),[20]Taxonomy/Acariformes(0x5461786F6E6F6D792F4163617269666F726D6573),[4] (0x00D6A3C3)}
|
Sep 30 12:43:10 db2125 mysqld[30220]: 2020-09-30 12:43:10 9 [ERROR] InnoDB: Record in index `namespace_title` of table `zhwiki`.`watchlist` was not found on update: TUPLE (info_bits=0, 3 fields): {[4] (0x8000000B),[19]Taxonomy/Acartiella(0x5461786F6E6F6D792F416361727469656C6C61),[4] &$ (0x012624E5)} at: COMPACT RECORD(info_bits=0, 3 fields): {[4] (0x8000000B),[20]Taxonomy/Acariformes(0x5461786F6E6F6D792F4163617269666F726D6573),[4] (0x00D6A3C3)}
|
Sep 30 12:55:26 db2125 mysqld[30220]: 2020-09-30 12:55:26 0 [Note] InnoDB: Buffer pool(s) load completed at 200930 12:55:26
|
Sep 30 13:44:02 db2125 mysqld[30220]: 2020-09-30 13:44:02 9 [ERROR] InnoDB: Record in index `linter_page` of table `plwiki`.`linter` was not found on update: TUPLE (info_bits=0, 2 fields): {[4] Sh(0x001B5368),[4] (0x00A5BDD7)} at: COMPACT RECORD(info_bits=0, 2 fields): {[4] Sh(0x001B5368),[4] D(0x00A5BB44)}
|
Sep 30 13:46:15 db2125 mysqld[30220]: 2020-09-30 13:46:15 8 [ERROR] Error reading packet from server: Lost connection to MySQL server during query (server_errno=2013)
|
Sep 30 13:46:15 db2125 mysqld[30220]: 2020-09-30 13:46:15 8 [Note] Slave I/O thread: Failed reading log event, reconnecting to retry, log 'db2107-bin.001729' at position 103635891; GTID position '171970567-171970567-390719905,0-180359173-4880477695,180359173-180359173-70825087,171978786-171978786-2108483244,180359271-180
|
Sep 30 14:09:26 db2125 mysqld[30220]: 2020-09-30 14:09:26 9 [ERROR] InnoDB: Record in index `cat_pages` of table `idwiki`.`category` was not found on update: TUPLE (info_bits=0, 2 fields): {[4] (0x80000620),[4] d (0x00886419)} at: COMPACT RECORD(info_bits=32, 2 fields): {[4] (0x80000620),[4] < (0x00873CC5)}
|
Sep 30 14:39:21 db2125 mysqld[30220]: 2020-09-30 14:39:21 9 [ERROR] InnoDB: Record in index `wl_user_notificationtimestamp` of table `fiwiki`.`watchlist` was not found on update: TUPLE (info_bits=0, 3 fields): {[4] (0x0003A1EF),NULL,[4] 3 (0x00339DCE)} at: COMPACT RECORD(info_bits=0, 3 fields): {[4] (0x0003A1EF),NULL,[4] 3 (0x00339C13)}
|
Sep 30 14:59:47 db2125 mysqld[30220]: 2020-09-30 14:59:47 9 [ERROR] InnoDB: Record in index `cat_pages` of table `nowiki`.`category` was not found on update: TUPLE (info_bits=0, 2 fields): {[4] 4m(0x8000346D),[4] lBa(0x006C4261)} at: COMPACT RECORD(info_bits=32, 2 fields): {[4] 4m(0x8000346D),[4] [Iw(0x005B4977)}
|
Sep 30 15:23:29 db2125 mysqld[30220]: 2020-09-30 15:23:29 8 [ERROR] Error reading packet from server: Lost connection to MySQL server during query (server_errno=2013)
|
Sep 30 15:23:29 db2125 mysqld[30220]: 2020-09-30 15:23:29 8 [Note] Slave I/O thread: Failed reading log event, reconnecting to retry, log 'db2107-bin.001729' at position 289795744; GTID position '171970567-171970567-390719905,0-180359173-4880477695,180359173-180359173-70825087,171978786-171978786-2108483244,180359271-180
|
Sep 30 15:35:30 db2125 mysqld[30220]: 2020-09-30 15:35:30 9 [ERROR] InnoDB: Record in index `rd_ns_title` of table `itwiki`.`redirect` was not found on update: TUPLE (info_bits=0, 3 fields): {[4] (0x80000001),[26]Castelli_della_val_di_Susa(0x43617374656C6C695F64656C6C615F76616C5F64695F53757361),[4] (z(0x0084287A)} at: COMPACT RECORD(info_bits=0, 3 fields): {[4] (0x80000001),[33]Castel_di_Sangro_Calcio_1996-1997(0x43617374656C5F64695F53616E67726F5F43616C63696F5F313939362D31393937),[4] ^mT(0x005E6D54)}
|
Sep 30 16:25:00 db2125 mysqld[30220]: 2020-09-30 16:25:00 8 [ERROR] Error reading packet from server: Lost connection to MySQL server during query (server_errno=2013)
|
Sep 30 16:25:00 db2125 mysqld[30220]: 2020-09-30 16:25:00 8 [Note] Slave I/O thread: Failed reading log event, reconnecting to retry, log 'db2107-bin.001729' at position 427327501; GTID position '171970567-171970567-390719905,0-180359173-4880477695,180359173-180359173-70825087,171978786-171978786-2108483244,180359271-180
|
Sep 30 17:05:32 db2125 mysqld[30220]: 2020-09-30 17:05:32 9 [ERROR] InnoDB: Record in index `wl_user_notificationtimestamp` of table `nlwiki`.`watchlist` was not found on update: TUPLE (info_bits=0, 3 fields): {[4] (0x0002C2EA),NULL,[4] " (0x00B022E6)} at: COMPACT RECORD(info_bits=0, 3 fields): {[4] (0x0002C2EA),NULL,[4] (0x00B01D9F)}
|
Sep 30 17:05:32 db2125 mysqld[30220]: 2020-09-30 17:05:32 9 [ERROR] InnoDB: Record in index `namespace_title` of table `nlwiki`.`watchlist` was not found on update: TUPLE (info_bits=0, 3 fields): {[4] (0x80000001),[34]Vrouwen_Electriciteits_Vereeniging(0x56726F7577656E5F456C6563747269636974656974735F56657265656E6967696E67),[4] " (0x00B022E7)} at: COMPACT RECORD(info_bits=0, 3 fields): {[4] (0x80000001),[34]Vrouwen_Electriciteits_Vereeniging(0x56726F7577656E5F456C6563747269636974656974735F56657265656E6967696E67),[4] D(0x0098EE44)}
|
Sep 30 17:05:32 db2125 mysqld[30220]: 2020-09-30 17:05:32 9 [ERROR] InnoDB: Record in index `wl_user_notificationtimestamp` of table `nlwiki`.`watchlist` was not found on update: TUPLE (info_bits=0, 3 fields): {[4] (0x0002C2EA),NULL,[4] " (0x00B022E7)} at: COMPACT RECORD(info_bits=0, 3 fields): {[4] (0x0002C2EA),NULL,[4] (0x00B01D9F)}
|
Sep 30 17:29:09 db2125 mysqld[30220]: 2020-09-30 17:29:09 8 [ERROR] Error reading packet from server: Lost connection to MySQL server during query (server_errno=2013)
|
Sep 30 17:29:09 db2125 mysqld[30220]: 2020-09-30 17:29:09 8 [Note] Slave I/O thread: Failed reading log event, reconnecting to retry, log 'db2107-bin.001729' at position 564603880; GTID position '171970567-171970567-390719905,0-180359173-4880477695,180359173-180359173-70825087,171978786-171978786-2108483244,180359271-180
|
Sep 30 18:42:11 db2125 mysqld[30220]: 2020-09-30 18:42:11 8 [Note] Slave: received end packet from server, apparent master shutdown:
|
Sep 30 18:42:11 db2125 mysqld[30220]: 2020-09-30 18:42:11 8 [Note] Slave I/O thread: Failed reading log event, reconnecting to retry, log 'db2107-bin.001729' at position 732916829; GTID position '171970567-171970567-390719905,0-180359173-4880477695,180359173-180359173-70825087,171978786-171978786-2108483244,180359271-180
|
Sep 30 18:45:04 db2125 mysqld[30220]: 2020-09-30 18:45:04 9 [ERROR] InnoDB: Record in index `namespace_title` of table `zhwiki`.`watchlist` was not found on update: TUPLE (info_bits=0, 3 fields): {[4] (0x8000000A),[21]Taxonomy/Chinapotamon(0x5461786F6E6F6D792F4368696E61706F74616D6F6E),[4] &#r(0x01262372)} at: COMPACT RECORD(info_bits=0, 3 fields): {[4] (0x8000000A),[21]Taxonomy/Chimonanthus(0x5461786F6E6F6D792F4368696D6F6E616E74687573),[4] ! (0x01162116)}
|
Sep 30 18:45:04 db2125 mysqld[30220]: 2020-09-30 18:45:04 9 [ERROR] InnoDB: Record in index `namespace_title` of table `zhwiki`.`watchlist` was not found on update: TUPLE (info_bits=0, 3 fields): {[4] (0x8000000A),[24]Taxonomy/Canthocamptidae(0x5461786F6E6F6D792F43616E74686F63616D707469646165),[4] &%$(0x01262524)} at: COMPACT RECORD(info_bits=0, 3 fields): {[4] (0x8000000A),[20]Taxonomy/Cantharinae(0x5461786F6E6F6D792F43616E74686172696E6165),[4] m(0x00BDBA6D)}
|
Sep 30 19:21:48 db2125 mysqld[30220]: 2020-09-30 19:21:48 9 [ERROR] InnoDB: Record in index `wl_user_notificationtimestamp` of table `nowiki`.`watchlist` was not found on update: TUPLE (info_bits=0, 3 fields): {[4] (0x0000FB93),NULL,[4] MS (0x004D53E6)} at: COMPACT RECORD(info_bits=0, 3 fields): {[4] (0x0000FB93),NULL,[4] MJ (0x004D4AD9)}
|
Sep 30 19:46:59 db2125 mysqld[30220]: 2020-09-30 19:46:59 9 [ERROR] InnoDB: Record in index `wl_user_notificationtimestamp` of table `nlwiki`.`watchlist` was not found on update: TUPLE (info_bits=0, 3 fields): {[4] HI(0x00014849),NULL,[4] 4 (0x00347F1F)} at: COMPACT RECORD(info_bits=0, 3 fields): {[4] HI(0x00014849),NULL,[4] 4 (0x00347F1E)}
|
Sep 30 20:00:28 db2125 mysqld[30220]: 2020-09-30 20:00:28 8 [ERROR] Error reading packet from server: Lost connection to MySQL server during query (server_errno=2013)
|
Sep 30 20:00:28 db2125 mysqld[30220]: 2020-09-30 20:00:28 8 [Note] Slave I/O thread: Failed reading log event, reconnecting to retry, log 'db2107-bin.001729' at position 902518257; GTID position '171970567-171970567-390719905,0-180359173-4880477695,180359173-180359173-70825087,171978786-171978786-2108483244,180359271-180
|
Sep 30 21:02:29 db2125 mysqld[30220]: 2020-09-30 21:02:29 8 [ERROR] Error reading packet from server: Lost connection to MySQL server during query (server_errno=2013)
|
Sep 30 21:02:29 db2125 mysqld[30220]: 2020-09-30 21:02:29 8 [Note] Slave I/O thread: Failed reading log event, reconnecting to retry, log 'db2107-bin.001729' at position 1010128690; GTID position '171970567-171970567-390719905,0-180359173-4880477695,180359173-180359173-70825087,171978786-171978786-2108483244,180359271-18
|
Sep 30 21:31:08 db2125 mysqld[30220]: 2020-09-30 21:31:08 9 [ERROR] InnoDB: Record in index `tl_namespace` of table `itwiki`.`templatelinks` was not found on update: TUPLE (info_bits=0, 3 fields): {[4] <(0x8000033C),[9]Linguaggi(0x4C696E677561676769),[4] (0x0007F500)} at: COMPACT RECORD(info_bits=0, 3 fields): {[4] <(0x8000033C),[9]Linguaggi(0x4C696E677561676769),[4] (0x0007F4F9)}
|
Sep 30 21:36:01 db2125 mysqld[30220]: 2020-09-30 21:36:01 9 [ERROR] InnoDB: Record in index `tl_namespace` of table `itwiki`.`templatelinks` was not found on update: TUPLE (info_bits=0, 3 fields): {[4] <(0x8000033C),[9]Arguments(0x417267756D656E7473),[4] KZ (0x004B5A1B)} at: COMPACT RECORD(info_bits=0, 3 fields): {[4] <(0x8000033C),[9]Arguments(0x417267756D656E7473),[4] KZ (0x004B5A1A)}
|
Sep 30 22:21:32 db2125 mysqld[30220]: 2020-09-30 22:21:32 8 [ERROR] Error reading packet from server: Lost connection to MySQL server during query (server_errno=2013)
|
Sep 30 22:21:32 db2125 mysqld[30220]: 2020-09-30 22:21:32 8 [Note] Slave I/O thread: Failed reading log event, reconnecting to retry, log 'db2107-bin.001730' at position 87230375; GTID position '171970567-171970567-390719905,0-180359173-4880477695,180359173-180359173-70825087,171978786-171978786-2108483244,180359271-1803
|
Sep 30 22:51:48 db2125 mysqld[30220]: 2020-09-30 22:51:48 9 [ERROR] InnoDB: Record in index `rd_ns_title` of table `thwiki`.`redirect` was not found on update: TUPLE (info_bits=0, 3 fields): {[4] (0x80000000),[45] (0xE0B8A7E0B8B1E0B894E0B983E0B8ABE0B8A1E0B988E0B89EE0B8B4E0B980E0B8A3E0B899E0B897E0B8A3E0B98C),[4] , (0x00112CC8)} at: COMPACT RECORD(info_bits=0, 3 fields): {[4] (0x80000000),[45] (0xE0B8A7E0B8B1E0B894E0B983E0B8ABE0B88DE0B988E0B8ADE0B8B4E0B899E0B897E0B8B2E0B8A3E0B8B2E0B8A1),[4] (0x000BF88C)}
|
Sep 30 22:52:02 db2125 mysqld[30220]: 2020-09-30 22:52:02 9 [ERROR] InnoDB: Record in index `linter_page` of table `thwiki`.`linter` was not found on update: TUPLE (info_bits=0, 2 fields): {[4] (0x000AB8FB),[4] ;= (0x003B3DE1)} at: COMPACT RECORD(info_bits=0, 2 fields): {[4] (0x000AB8FB),[4] : (0x003AB796)}
|
Sep 30 22:54:16 db2125 mysqld[30220]: 2020-09-30 22:54:16 9 [ERROR] InnoDB: Clustered record for sec rec not found index `linter_page` of table `thwiki`.`linter`
|
Sep 30 22:54:16 db2125 mysqld[30220]: InnoDB: sec index record PHYSICAL RECORD: n_fields 2; compact format; info bits 0
|
Sep 30 22:54:16 db2125 mysqld[30220]: 0: len 4; hex 000ab8fb; asc ;;
|
Sep 30 22:54:16 db2125 mysqld[30220]: 1: len 4; hex 003ab796; asc : ;;
|
Sep 30 22:54:16 db2125 mysqld[30220]: InnoDB: clust index record PHYSICAL RECORD: n_fields 8; compact format; info bits 0
|
Sep 30 22:54:16 db2125 mysqld[30220]: 0: len 4; hex 003ab795; asc : ;;
|
Sep 30 22:54:16 db2125 mysqld[30220]: 1: len 6; hex 000c1bf83559; asc 5Y;;
|
Sep 30 22:54:16 db2125 mysqld[30220]: 2: len 7; hex d300018fc50110; asc ;;
|
Sep 30 22:54:16 db2125 mysqld[30220]: 3: len 4; hex 000430cc; asc 0 ;;
|
Sep 30 22:54:16 db2125 mysqld[30220]: 4: len 4; hex 00000002; asc ;;
|
Sep 30 22:54:16 db2125 mysqld[30220]: 5: len 4; hex 00000d81; asc ;;
|
Sep 30 22:54:16 db2125 mysqld[30220]: 6: len 4; hex 00000dd8; asc ;;
|
Sep 30 22:54:16 db2125 mysqld[30220]: 7: len 30; hex 7b226e616d65223a2263656e746572222c2274656d706c617465496e666f; asc {"name":"center","templateInfo; (total 73 bytes);
|
Sep 30 22:54:16 db2125 mysqld[30220]: TRANSACTION 52208853465, ACTIVE 0 sec starting index read
|
Sep 30 22:54:16 db2125 mysqld[30220]: mysql tables in use 1, locked 1
|
Sep 30 22:54:16 db2125 mysqld[30220]: 2 lock struct(s), heap size 1128, 1 row lock(s)
|
Sep 30 22:54:16 db2125 mysqld[30220]: MySQL thread id 9, OS thread handle 140403288561408, query id 6496648 Updating
|
Sep 30 22:54:16 db2125 mysqld[30220]: DELETE /* MediaWiki\Linter\Database::setForPage */ FROM `linter` WHERE linter_page = 702715
|
Sep 30 22:54:16 db2125 mysqld[30220]: InnoDB: Submit a detailed bug report to https://jira.mariadb.org/
|
Sep 30 23:08:12 db2125 mysqld[30220]: 2020-09-30 23:08:12 9 [ERROR] InnoDB: Record in index `cat_pages` of table `nowiki`.`category` was not found on update: TUPLE (info_bits=0, 2 fields): {[4] (0x8000101C),[4] m (0x006D8D03)} at: COMPACT RECORD(info_bits=32, 2 fields): {[4] (0x8000101C),[4] l (0x006CB006)}
|
Sep 30 23:08:12 db2125 mysqld[30220]: 2020-09-30 23:08:12 0x7fb23023d700 InnoDB: Assertion failure in file /root/mariadb-10.4.14/storage/innobase/row/row0ins.cc line 218
|
Sep 30 23:08:12 db2125 mysqld[30220]: InnoDB: Failing assertion: !cursor->index->is_committed()
|
Sep 30 23:08:12 db2125 mysqld[30220]: InnoDB: We intentionally generate a memory trap.
|
Sep 30 23:08:12 db2125 mysqld[30220]: InnoDB: Submit a detailed bug report to https://jira.mariadb.org/
|
Sep 30 23:08:12 db2125 mysqld[30220]: InnoDB: If you get repeated assertion failures or crashes, even
|
Sep 30 23:08:12 db2125 mysqld[30220]: InnoDB: immediately after the mysqld startup, there may be
|
Sep 30 23:08:12 db2125 mysqld[30220]: InnoDB: corruption in the InnoDB tablespace. Please refer to
|
Sep 30 23:08:12 db2125 mysqld[30220]: InnoDB: https://mariadb.com/kb/en/library/innodb-recovery-modes/
|
Sep 30 23:08:12 db2125 mysqld[30220]: InnoDB: about forcing recovery.
|
Sep 30 23:08:12 db2125 mysqld[30220]: 200930 23:08:12 [ERROR] mysqld got signal 6 ;
|
Sep 30 23:08:12 db2125 mysqld[30220]: This could be because you hit a bug. It is also possible that this binary
|
Sep 30 23:08:12 db2125 mysqld[30220]: or one of the libraries it was linked against is corrupt, improperly built,
|
Sep 30 23:08:12 db2125 mysqld[30220]: or misconfigured. This error can also be caused by malfunctioning hardware.
|
Sep 30 23:08:12 db2125 mysqld[30220]: To report this bug, see https://mariadb.com/kb/en/reporting-bugs
|
Sep 30 23:08:12 db2125 mysqld[30220]: We will try our best to scrape up some info that will hopefully help
|
Sep 30 23:08:12 db2125 mysqld[30220]: diagnose the problem, but since we have already crashed,
|
Sep 30 23:08:12 db2125 mysqld[30220]: something is definitely wrong and this may fail.
|
Sep 30 23:08:12 db2125 mysqld[30220]: Server version: 10.4.14-MariaDB-log
|
Sep 30 23:08:12 db2125 mysqld[30220]: key_buffer_size=134217728
|
Sep 30 23:08:12 db2125 mysqld[30220]: read_buffer_size=131072
|
Sep 30 23:08:12 db2125 mysqld[30220]: max_used_connections=13
|
Sep 30 23:08:12 db2125 mysqld[30220]: max_threads=2010
|
Sep 30 23:08:12 db2125 mysqld[30220]: thread_count=17
|
Sep 30 23:08:12 db2125 mysqld[30220]: It is possible that mysqld could use up to
|
Sep 30 23:08:12 db2125 mysqld[30220]: key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 4752266 K bytes of memory
|
Sep 30 23:08:12 db2125 mysqld[30220]: Hope that's ok; if not, decrease some variables in the equation.
|
Sep 30 23:08:12 db2125 mysqld[30220]: Thread pointer: 0x7f51f40014f8
|
Sep 30 23:08:12 db2125 mysqld[30220]: Attempting backtrace. You can use the following information to find out
|
Sep 30 23:08:12 db2125 mysqld[30220]: where mysqld died. If you see no messages after this, something went
|
Sep 30 23:08:12 db2125 mysqld[30220]: terribly wrong...
|
Sep 30 23:08:12 db2125 mysqld[30220]: stack_bottom = 0x7fb23023c698 thread_stack 0x30000
|
Sep 30 23:08:12 db2125 mysqld[30220]: *** buffer overflow detected ***: /opt/wmf-mariadb104/bin/mysqld terminated
|
Sep 30 23:08:13 db2125 systemd[1]: mariadb.service: Main process exited, code=killed, status=6/ABRT
|
-- Subject: Unit process exited
|
This host has always had innodb_purge_threads=1
As mentioned in my previous post, this host has had a HW malfunction, so I assume the data is corrupted, however we've not seen replicating getting broken or anything so I have left it replicating and tried to play with innodb_change_buffering=inserts to see if MySQL process would stop crashing, but it hasn't worked. It keeps crashing after a few hours (it is still replicating from production).
Maybe this is a good opportunity to gather data about this particular process crash.
We don't often run CHECK TABLES.
I will try to rebuild this host from a logical dump (and I will check tables on that source host to make sure we are good there just in case).
The configuration I will be using for this host once rebuilt will be:
innodb_purge_threads=1
|
innodb_change_buffering=inserts
|
Have encountered the same problem on 10.5.6.
Was running an import tool (doing roughly 9 inserts per second with 250 items each), while the DB was also being used by other sources (hope this helps a bit)
2020-10-13 9:51:49 1324189 [ERROR] InnoDB: Unable to find a record to delete-mark
|
InnoDB: tuple DATA TUPLE: 5 fields;
|
0: len 3; hex 000488; asc ;;
|
1: len 3; hex 000001; asc ;;
|
2: len 16; hex 4255533236323030304d475433363552; asc BUS262000MGT365R;;
|
3: len 4; hex 05e69ec4; asc ;;
|
4: len 4; hex 009daf0a; asc ;;
|
|
|
InnoDB: record PHYSICAL RECORD: n_fields 5; compact format; info bits 0
|
0: len 3; hex 000488; asc ;;
|
1: len 3; hex 000001; asc ;;
|
2: len 12; hex 425553323631393053424a50; asc BUS26190SBJP;;
|
3: len 4; hex 05e69ec4; asc ;;
|
4: len 4; hex 009897a4; asc ;;
|
2020-10-13 9:51:49 1324189 [ERROR] InnoDB: page [page id: space=505908, page number=12289] (272 records, index id 994674).
|
2020-10-13 9:51:49 1324189 [ERROR] InnoDB: Submit a detailed bug report to https://jira.mariadb.org/
|
2020-10-13 9:51:50 1322947 [Warning] Aborted connection 1322947 to db: 'provider' user: 'provider' host: 'localhost' (Got an error reading communication packets)
|
2020-10-13 9:52:02 0 [ERROR] InnoDB: Unable to find a record to delete-mark
|
InnoDB: tuple DATA TUPLE: 5 fields;
|
0: len 3; hex 0003f5; asc ;;
|
1: len 3; hex 000001; asc ;;
|
2: len 6; hex 313533353830; asc 153580;;
|
3: len 4; hex 05e69eea; asc ;;
|
4: len 4; hex 009f1774; asc t;;
|
|
|
InnoDB: record PHYSICAL RECORD: n_fields 5; compact format; info bits 0
|
0: len 3; hex 0003f5; asc ;;
|
1: len 3; hex 000001; asc ;;
|
2: len 6; hex 313533353830; asc 153580;;
|
3: len 4; hex 05e69ec5; asc ;;
|
4: len 4; hex 009fe628; asc (;;
|
2020-10-13 9:52:02 0 [ERROR] InnoDB: page [page id: space=505908, page number=17436] (593 records, index id 994674).
|
2020-10-13 9:52:02 0 [ERROR] InnoDB: Submit a detailed bug report to https://jira.mariadb.org/
|
2020-10-13 9:52:02 1324189 [ERROR] InnoDB: Record in index `Company_ID_2` of table `FYN_Rijwiel`.`Leverancier_Product` was not found on update: TUPLE (info_bits=0, 5 fields): {[3] (0x0003F5),[3] (0x000003),[6]500641(0x353030363431),[4] (0x05E69EEA),[4] (0x0099BA12)} at: COMPACT RECORD(info_bits=0, 5 fields): {[3] (0x0003F5),[3] (0x000003),[6]500640(0x353030363430),[4] (0x05E69EEA),[4] D(0x0098B144)}
|
2020-10-13 09:52:02 0x7fbc857fe700 InnoDB: Assertion failure in file /home/buildbot/buildbot/build/mariadb-10.5.6/storage/innobase/row/row0ins.cc line 218
|
InnoDB: Failing assertion: !cursor->index->is_committed()
|
InnoDB: We intentionally generate a memory trap.
|
InnoDB: Submit a detailed bug report to https://jira.mariadb.org/
|
InnoDB: If you get repeated assertion failures or crashes, even
|
InnoDB: immediately after the mysqld startup, there may be
|
InnoDB: corruption in the InnoDB tablespace. Please refer to
|
InnoDB: https://mariadb.com/kb/en/library/innodb-recovery-modes/
|
InnoDB: about forcing recovery.
|
201013 9:52:02 [ERROR] mysqld got signal 6 ;
|
This could be because you hit a bug. It is also possible that this binary
|
or one of the libraries it was linked against is corrupt, improperly built,
|
or misconfigured. This error can also be caused by malfunctioning hardware.
|
|
|
To report this bug, see https://mariadb.com/kb/en/reporting-bugs
|
|
|
We will try our best to scrape up some info that will hopefully help
|
diagnose the problem, but since we have already crashed,
|
something is definitely wrong and this may fail.
|
|
|
Server version: 10.5.6-MariaDB-1:10.5.6+maria~buster-log
|
key_buffer_size=8388608
|
read_buffer_size=131072
|
max_used_connections=375
|
max_threads=65537
|
thread_count=208
|
It is possible that mysqld could use up to
|
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 142741846 K bytes of memory
|
Hope that's ok; if not, decrease some variables in the equation.
|
|
|
Thread pointer: 0x7fbb84339cb8
|
Attempting backtrace. You can use the following information to find out
|
where mysqld died. If you see no messages after this, something went
|
terribly wrong...
|
stack_bottom = 0x7fbc857fdc98 thread_stack 0x49000
|
*** buffer overflow detected ***: terminated
|
2020-10-13 9:52:08 0 [Note] mariadbd: Aria engine: starting recovery
|
tables to flush: 1 0
|
(0.0 seconds);
|
2020-10-13 9:52:08 0 [Note] mariadbd: Aria engine: recovery done
|
We just had this crash happening again on two servers that were installed with 10.4.
They were cloned from a server running 10.1 at the same time (that 10.1 host never crashed) and these two servers crashed at the same time.
Some traces:
Nov 18 01:00:17 clouddb1017 mysqld[9432]: 2020-11-18 1:00:17 1 [ERROR] InnoDB: page [page id: space=328, page number=10484169] (319 records, index id 1122).
|
Nov 18 01:00:17 clouddb1017 mysqld[9432]: 2020-11-18 1:00:17 1 [ERROR] InnoDB: Submit a detailed bug report to https://jira.mariadb.org/
|
Nov 18 01:02:11 clouddb1017 mysqld[9432]: 2020-11-18 1:02:11 1 [ERROR] InnoDB: Unable to find a record to delete-mark
|
Nov 18 01:02:11 clouddb1017 mysqld[9432]: InnoDB: tuple DATA TUPLE: 3 fields;
|
Nov 18 01:02:11 clouddb1017 mysqld[9432]: 0: len 4; hex 80000003; asc ;;
|
Nov 18 01:02:11 clouddb1017 mysqld[9432]: 1: len 18; hex 462d31365f4a756e655f323030382e6a7067; asc F-16_June_2008.jpg;;
|
Nov 18 01:02:11 clouddb1017 mysqld[9432]: 2: len 4; hex 02468239; asc F 9;;
|
Nov 18 01:02:11 clouddb1017 mysqld[9432]: InnoDB: record PHYSICAL RECORD: n_fields 3; compact format; info bits 0
|
Nov 18 01:02:11 clouddb1017 mysqld[9432]: 0: len 4; hex 80000003; asc ;;
|
Nov 18 01:02:11 clouddb1017 mysqld[9432]: 1: len 18; hex 462d31365f4a756e655f323030382e6a7067; asc F-16_June_2008.jpg;;
|
Nov 18 01:02:11 clouddb1017 mysqld[9432]: 2: len 4; hex 0189110e; asc ;;
|
Nov 18 01:02:11 clouddb1017 mysqld[9432]: 2020-11-18 1:02:11 1 [ERROR] InnoDB: page [page id: space=279, page number=892137] (300 records, index id 963).
|
Nov 18 01:02:11 clouddb1017 mysqld[9432]: 2020-11-18 1:02:11 1 [ERROR] InnoDB: Submit a detailed bug report to https://jira.mariadb.org/
|
Nov 18 01:02:41 clouddb1017 mysqld[9432]: 2020-11-18 1:02:41 599 [ERROR] InnoDB: Record in index `pl_backlinks_namespace` of table `enwiki`.`pagelinks` was not found on update: TUPLE (info_bits=0, 4 fields): {[4] (0x80000002),[4] (0x80000000),[40]2010_Bihar_Legislative_Assembly_election(0x323031305F42696861725F4C656769736C61746976655F417373656D626C795F656C656374696F6E),[4] = (0x03E33DBE)} at: COMPACT RECORD(info_bits=0, 4 fields): {[4] (0x80000002),[4] (0x80000000),[40]2010_Bihar_Legislative_Assembly_election(0x323031305F42696861725F4C656769736C61746976655F417373656D626C795F656C656374696F6E),[4] ' (0x03C72703)}
|
Nov 18 01:02:41 clouddb1017 mysqld[9432]: 2020-11-18 1:02:41 599 [ERROR] InnoDB: Record in index `pl_backlinks_namespace` of table `enwiki`.`pagelinks` was not found on update: TUPLE (info_bits=0, 4 fields): {[4] (0x80000002),[4] (0x80000000),[4]Dara(0x44617261),[4] = (0x03E33DBE)} at: COMPACT RECORD(info_bits=0, 4 fields): {[4] (0x80000002),[4] (0x80000000),[4]Dara(0x44617261),[4] b (0x03B86281)}
|
Nov 18 01:02:41 clouddb1017 mysqld[9432]: 2020-11-18 1:02:41 599 [ERROR] InnoDB: Record in index `pl_backlinks_namespace` of table `enwiki`.`pagelinks` was not found on update: TUPLE (info_bits=0, 4 fields): {[4] (0x80000002),[4] (0x80000000),[12]Demographics(0x44656D6F6772617068696373),[4] = (0x03E33DBE)} at: COMPACT RECORD(info_bits=0, 4 fields): {[4] (0x80000002),[4] (0x80000000),[12]Demographics(0x44656D6F6772617068696373),[4] 0 (0x03E03018)}
|
Nov 18 01:02:41 clouddb1017 mysqld[9432]: 2020-11-18 1:02:41 599 [ERROR] InnoDB: Record in index `pl_namespace` of table `enwiki`.`pagelinks` was not found on update: TUPLE (info_bits=0, 3 fields): {[4] (0x80000000),[20]Hard_science_fiction(0x486172645F736369656E63655F66696374696F6E),[4] = (0x03E33DBE)} at: COMPACT RECORD(info_bits=0, 3 fields): {[4] (0x80000000),[20]Hard_science_fiction(0x486172645F736369656E63655F66696374696F6E),[4] (0x03D6CDBD)}
|
Nov 18 01:02:41 clouddb1017 mysqld[9432]: 2020-11-18 1:02:41 1 [ERROR] InnoDB: Unable to find a record to delete-mark
|
Nov 18 01:02:41 clouddb1017 mysqld[9432]: InnoDB: tuple DATA TUPLE: 4 fields;
|
Nov 18 01:02:41 clouddb1017 mysqld[9432]: 0: len 4; hex 80000002; asc ;;
|
Nov 18 01:02:41 clouddb1017 mysqld[9432]: 1: len 4; hex 80000000; asc ;;
|
Nov 18 01:02:41 clouddb1017 mysqld[9432]: 2: len 12; hex 46757272795f66616e646f6d; asc Furry_fandom;;
|
Nov 18 01:02:41 clouddb1017 mysqld[9432]: 3: len 4; hex 03e33dbe; asc = ;;
|
Nov 18 01:02:41 clouddb1017 mysqld[9432]: InnoDB: record PHYSICAL RECORD: n_fields 4; compact format; info bits 0
|
Nov 18 01:02:41 clouddb1017 mysqld[9432]: 0: len 4; hex 80000002; asc ;;
|
Nov 18 01:02:41 clouddb1017 mysqld[9432]: 1: len 4; hex 80000000; asc ;;
|
Nov 18 01:02:41 clouddb1017 mysqld[9432]: 2: len 12; hex 46757272795f66616e646f6d; asc Furry_fandom;;
|
Nov 18 01:02:41 clouddb1017 mysqld[9432]: 3: len 4; hex 03e0d760; asc `;;
|
Nov 18 01:02:41 clouddb1017 mysqld[9432]: 2020-11-18 1:02:41 1 [ERROR] InnoDB: page [page id: space=328, page number=7991194] (380 records, index id 1123).
|
Nov 18 01:02:41 clouddb1017 mysqld[9432]: 2020-11-18 1:02:41 1 [ERROR] InnoDB: Submit a detailed bug report to https://jira.mariadb.org/
|
Nov 18 01:02:41 clouddb1017 mysqld[9432]: 2020-11-18 1:02:41 0 [ERROR] InnoDB: Unable to find a record to delete-mark
|
Nov 18 01:02:41 clouddb1017 mysqld[9432]: InnoDB: tuple DATA TUPLE: 3 fields;
|
Nov 18 01:02:41 clouddb1017 mysqld[9432]: 0: len 4; hex 80000000; asc ;;
|
Nov 18 01:02:41 clouddb1017 mysqld[9432]: 1: len 8; hex 4e656f2d736f756c; asc Neo-soul;;
|
Nov 18 01:02:41 clouddb1017 mysqld[9432]: 2: len 4; hex 03e33dbe; asc = ;;
|
Nov 18 01:02:41 clouddb1017 mysqld[9432]: InnoDB: record PHYSICAL RECORD: n_fields 3; compact format; info bits 0
|
Nov 18 01:02:41 clouddb1017 mysqld[9432]: 0: len 4; hex 80000000; asc ;;
|
Nov 18 01:02:41 clouddb1017 mysqld[9432]: 1: len 8; hex 4e656f2d736f756c; asc Neo-soul;;
|
Nov 18 01:02:41 clouddb1017 mysqld[9432]: 2: len 4; hex 03d949a5; asc I ;;
|
Nov 18 01:02:41 clouddb1017 mysqld[9432]: 2020-11-18 1:02:41 0 [ERROR] InnoDB: page [page id: space=328, page number=4696746] (548 records, index id 1122).
|
Nov 18 01:02:41 clouddb1017 mysqld[9432]: 2020-11-18 1:02:41 0 [ERROR] InnoDB: Submit a detailed bug report to https://jira.mariadb.org/
|
Nov 18 01:02:41 clouddb1017 mysqld[9432]: 2020-11-18 1:02:41 1 [ERROR] InnoDB: Unable to find a record to delete-mark
|
Nov 18 01:02:41 clouddb1017 mysqld[9432]: InnoDB: tuple DATA TUPLE: 4 fields;
|
Nov 18 01:02:41 clouddb1017 mysqld[9432]: 0: len 4; hex 80000002; asc ;;
|
Nov 18 01:02:41 clouddb1017 mysqld[9432]: 1: len 4; hex 80000000; asc ;;
|
Nov 18 01:02:41 clouddb1017 mysqld[9432]: 2: len 13; hex 5472696c6c696e675f66726f67; asc Trilling_frog;;
|
Nov 18 01:02:41 clouddb1017 mysqld[9432]: 3: len 4; hex 03e33dbe; asc = ;;
|
Nov 18 01:02:41 clouddb1017 mysqld[9432]: InnoDB: record PHYSICAL RECORD: n_fields 4; compact format; info bits 0
|
Nov 18 01:02:41 clouddb1017 mysqld[9432]: 0: len 4; hex 80000002; asc ;;
|
Nov 18 01:02:41 clouddb1017 mysqld[9432]: 1: len 4; hex 80000000; asc ;;
|
Nov 18 01:02:41 clouddb1017 mysqld[9432]: 2: len 13; hex 5472696c6c696e675f66726f67; asc Trilling_frog;;
|
Nov 18 01:02:41 clouddb1017 mysqld[9432]: 3: len 4; hex 0357e17d; asc W };;
|
Nov 18 01:02:41 clouddb1017 mysqld[9432]: 2020-11-18 1:02:41 1 [ERROR] InnoDB: page [page id: space=328, page number=8038291] (326 records, index id 1123).
|
Nov 18 01:02:41 clouddb1017 mysqld[9432]: 2020-11-18 1:02:41 1 [ERROR] InnoDB: Submit a detailed bug report to https://jira.mariadb.org/
|
Nov 18 01:02:41 clouddb1017 mysqld[9432]: 2020-11-18 1:02:41 1 [ERROR] InnoDB: Unable to find a record to delete-mark
|
Nov 18 01:02:41 clouddb1017 mysqld[9432]: InnoDB: tuple DATA TUPLE: 3 fields;
|
Nov 18 01:02:41 clouddb1017 mysqld[9432]: 0: len 4; hex 80000000; asc ;;
|
Nov 18 01:02:41 clouddb1017 mysqld[9432]: 1: len 9; hex 53617373616e696473; asc Sassanids;;
|
Nov 18 01:02:41 clouddb1017 mysqld[9432]: 2: len 4; hex 03e33dbe; asc = ;;
|
Nov 18 01:02:41 clouddb1017 mysqld[9432]: InnoDB: record PHYSICAL RECORD: n_fields 3; compact format; info bits 0
|
Nov 18 01:02:41 clouddb1017 mysqld[9432]: 0: len 4; hex 80000000; asc ;;
|
Nov 18 01:02:41 clouddb1017 mysqld[9432]: 1: len 9; hex 53617373616e696473; asc Sassanids;;
|
Nov 18 01:02:41 clouddb1017 mysqld[9432]: 2: len 4; hex 03d551cc; asc Q ;;
|
Nov 18 01:02:41 clouddb1017 mysqld[9432]: 2020-11-18 1:02:41 1 [ERROR] InnoDB: page [page id: space=328, page number=4885314] (321 records, index id 1122).
|
Nov 18 01:02:41 clouddb1017 mysqld[9432]: 2020-11-18 1:02:41 1 [ERROR] InnoDB: Submit a detailed bug report to https://jira.mariadb.org/
|
Nov 18 01:02:41 clouddb1017 mysqld[9432]: 2020-11-18 1:02:41 1 [ERROR] InnoDB: Unable to find a record to delete-mark
|
Nov 18 01:02:41 clouddb1017 mysqld[9432]: InnoDB: tuple DATA TUPLE: 3 fields;
|
Nov 18 01:02:41 clouddb1017 mysqld[9432]: 0: len 4; hex 80000000; asc ;;
|
Nov 18 01:02:41 clouddb1017 mysqld[9432]: 1: len 16; hex 5265646669656c645f5265636f726473; asc Redfield_Records;;
|
Nov 18 01:02:41 clouddb1017 mysqld[9432]: 2: len 4; hex 03e33dbe; asc = ;;
|
Nov 18 01:02:41 clouddb1017 mysqld[9432]: InnoDB: record PHYSICAL RECORD: n_fields 3; compact format; info bits 0
|
Nov 18 01:02:41 clouddb1017 mysqld[9432]: 0: len 4; hex 80000000; asc ;;
|
Nov 18 01:02:41 clouddb1017 mysqld[9432]: 1: len 16; hex 5265646669656c645f5265636f726473; asc Redfield_Records;;
|
Nov 18 01:02:41 clouddb1017 mysqld[9432]: 2: len 4; hex 03cc121b; asc ;;
|
Nov 18 01:02:41 clouddb1017 mysqld[9432]: 2020-11-18 1:02:41 1 [ERROR] InnoDB: page [page id: space=328, page number=10095550] (501 records, index id 1122).
|
Nov 18 01:02:41 clouddb1017 mysqld[9432]: 2020-11-18 1:02:41 1 [ERROR] InnoDB: Submit a detailed bug report to https://jira.mariadb.org/
|
Nov 18 01:02:43 clouddb1017 mysqld[9432]: 2020-11-18 1:02:43 599 [ERROR] InnoDB: Record in index `iwl_prefix_from_title` of table `enwiki`.`iwlinks` was not found on update: TUPLE (info_bits=0, 3 fields): {[7]commons(0x636F6D6D6F6E73),[4] = (0x03E33DBE),[24]Category:Marmo_Botticino(0x43617465676F72793A4D61726D6F5F426F74746963696E6F)} at: COMPACT RECORD(info_bits=0, 3 fields): {[7]commons(0x636F6D6D6F6E73),[4] ; (0x03E33BFE),[33]Category:Murley_from_TomTom/Draft(0x43617465676F72793A4D75726C65795F66726F6D5F546F6D546F6D2F4472616674)}
|
Nov 18 01:03:13 clouddb1017 mysqld[9432]: 2020-11-18 1:03:13 1 [ERROR] InnoDB: Unable to find a record to delete-mark
|
Nov 18 01:03:13 clouddb1017 mysqld[9432]: InnoDB: tuple DATA TUPLE: 4 fields;
|
Nov 18 01:03:13 clouddb1017 mysqld[9432]: 0: len 4; hex 80000002; asc ;;
|
Nov 18 01:03:13 clouddb1017 mysqld[9432]: 1: len 4; hex 80000002; asc ;;
|
Nov 18 01:03:13 clouddb1017 mysqld[9432]: 2: len 7; hex 52656e614d696c; asc RenaMil;;
|
Nov 18 01:03:13 clouddb1017 mysqld[9432]: 3: len 4; hex 03e361ca; asc a ;;
|
Nov 18 01:03:13 clouddb1017 mysqld[9432]: InnoDB: record PHYSICAL RECORD: n_fields 4; compact format; info bits 0
|
Nov 18 01:03:13 clouddb1017 mysqld[9432]: 0: len 4; hex 80000002; asc ;;
|
Nov 18 01:03:13 clouddb1017 mysqld[9432]: 1: len 4; hex 80000002; asc ;;
|
Nov 18 01:03:13 clouddb1017 mysqld[9432]: 2: len 11; hex 52656e614372697370696e; asc RenaCrispin;;
|
Nov 18 01:03:13 clouddb1017 mysqld[9432]: 3: len 4; hex 03db554e; asc UN;;
|
Nov 18 01:03:13 clouddb1017 mysqld[9432]: 2020-11-18 1:03:13 1 [ERROR] InnoDB: page [page id: space=328, page number=10898301] (447 records, index id 1123).
|
Nov 18 01:03:13 clouddb1017 mysqld[9432]: 2020-11-18 1:03:13 1 [ERROR] InnoDB: Submit a detailed bug report to https://jira.mariadb.org/
|
Nov 18 01:03:13 clouddb1017 mysqld[9432]: 2020-11-18 1:03:13 1 [ERROR] InnoDB: Unable to find a record to delete-mark
|
Nov 18 01:03:13 clouddb1017 mysqld[9432]: InnoDB: tuple DATA TUPLE: 4 fields;
|
Nov 18 01:03:13 clouddb1017 mysqld[9432]: 0: len 4; hex 80000002; asc ;;
|
Nov 18 01:03:13 clouddb1017 mysqld[9432]: 1: len 4; hex 80000002; asc ;;
|
Nov 18 01:03:13 clouddb1017 mysqld[9432]: 2: len 7; hex 416c616e2e6361; asc Alan.ca;;
|
Nov 18 01:03:13 clouddb1017 mysqld[9432]: 3: len 4; hex 03e361ca; asc a ;;
|
Nov 18 01:03:13 clouddb1017 mysqld[9432]: InnoDB: record PHYSICAL RECORD: n_fields 4; compact format; info bits 0
|
Nov 18 01:03:13 clouddb1017 mysqld[9432]: 0: len 4; hex 80000002; asc ;;
|
Nov 18 01:03:13 clouddb1017 mysqld[9432]: 1: len 4; hex 80000002; asc ;;
|
Nov 18 01:03:13 clouddb1017 mysqld[9432]: 2: len 7; hex 416c616e2e6361; asc Alan.ca;;
|
Nov 18 01:03:13 clouddb1017 mysqld[9432]: 3: len 4; hex 031e6a7c; asc j|;;
|
Nov 18 01:03:13 clouddb1017 mysqld[9432]: 2020-11-18 1:03:13 1 [ERROR] InnoDB: page [page id: space=328, page number=8049415] (442 records, index id 1123).
|
Nov 18 01:03:13 clouddb1017 mysqld[9432]: 2020-11-18 1:03:13 1 [ERROR] InnoDB: Submit a detailed bug report to https://jira.mariadb.org/
|
Nov 18 01:04:59 clouddb1017 mysqld[9432]: 2020-11-18 01:04:59 0x7f72d4131700 InnoDB: Assertion failure in file /root/mariadb-10.4.15/storage/innobase/row/row0ins.cc line 218
|
Nov 18 01:04:59 clouddb1017 mysqld[9432]: InnoDB: Failing assertion: !cursor->index->is_committed()
|
Nov 18 01:04:59 clouddb1017 mysqld[9432]: InnoDB: We intentionally generate a memory trap.
|
Nov 18 01:04:59 clouddb1017 mysqld[9432]: InnoDB: Submit a detailed bug report to https://jira.mariadb.org/
|
Nov 18 01:04:59 clouddb1017 mysqld[9432]: InnoDB: If you get repeated assertion failures or crashes, even
|
Nov 18 01:04:59 clouddb1017 mysqld[9432]: InnoDB: immediately after the mysqld startup, there may be
|
Nov 18 01:04:59 clouddb1017 mysqld[9432]: InnoDB: corruption in the InnoDB tablespace. Please refer to
|
Nov 18 01:04:59 clouddb1017 mysqld[9432]: InnoDB: https://mariadb.com/kb/en/library/innodb-recovery-modes/
|
Nov 18 01:04:59 clouddb1017 mysqld[9432]: InnoDB: about forcing recovery.
|
Nov 18 01:04:59 clouddb1017 mysqld[9432]: 201118 1:04:59 [ERROR] mysqld got signal 6 ;
|
Nov 18 01:04:59 clouddb1017 mysqld[9432]: This could be because you hit a bug. It is also possible that this binary
|
Nov 18 01:04:59 clouddb1017 mysqld[9432]: or one of the libraries it was linked against is corrupt, improperly built,
|
Nov 18 01:04:59 clouddb1017 mysqld[9432]: or misconfigured. This error can also be caused by malfunctioning hardware.
|
Nov 18 01:04:59 clouddb1017 mysqld[9432]: To report this bug, see https://mariadb.com/kb/en/reporting-bugs
|
Nov 18 01:04:59 clouddb1017 mysqld[9432]: We will try our best to scrape up some info that will hopefully help
|
Nov 18 01:04:59 clouddb1017 mysqld[9432]: diagnose the problem, but since we have already crashed,
|
Nov 18 01:04:59 clouddb1017 mysqld[9432]: something is definitely wrong and this may fail.
|
Nov 18 01:04:59 clouddb1017 mysqld[9432]: Server version: 10.4.15-MariaDB
|
Nov 18 01:04:59 clouddb1017 mysqld[9432]: key_buffer_size=1048576
|
Nov 18 01:04:59 clouddb1017 mysqld[9432]: read_buffer_size=131072
|
Nov 18 01:04:59 clouddb1017 mysqld[9432]: max_used_connections=19
|
Nov 18 01:04:59 clouddb1017 mysqld[9432]: max_threads=252
|
Nov 18 01:04:59 clouddb1017 mysqld[9432]: thread_count=25
|
Nov 18 01:04:59 clouddb1017 mysqld[9432]: It is possible that mysqld could use up to
|
Nov 18 01:04:59 clouddb1017 mysqld[9432]: key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 555580 K bytes of memory
|
Nov 18 01:04:59 clouddb1017 mysqld[9432]: Hope that's ok; if not, decrease some variables in the equation.
|
Nov 18 01:04:59 clouddb1017 mysqld[9432]: Thread pointer: 0x7f3f5c0008d8
|
Nov 18 01:04:59 clouddb1017 mysqld[9432]: Attempting backtrace. You can use the following information to find out
|
Nov 18 01:04:59 clouddb1017 mysqld[9432]: where mysqld died. If you see no messages after this, something went
|
Nov 18 01:04:59 clouddb1017 mysqld[9432]: terribly wrong...
|
Nov 18 01:04:59 clouddb1017 mysqld[9432]: stack_bottom = 0x7f72d4130698 thread_stack 0x49000
|
Nov 18 01:05:04 clouddb1017 mysqld[9432]: /opt/wmf-mariadb104/bin/mysqld(my_print_stacktrace+0x2e)[0x5589fa2016fe]
|
Nov 18 01:05:04 clouddb1017 mysqld[9432]: /opt/wmf-mariadb104/bin/mysqld(handle_fatal_signal+0x54d)[0x5589f9cfa57d]
|
Nov 18 01:05:12 clouddb1017 mysqld[9432]: /lib/x86_64-linux-gnu/libpthread.so.0(+0x12730)[0x7f72dfe07730]
|
Nov 18 01:05:21 clouddb1017 mysqld[9432]: /lib/x86_64-linux-gnu/libc.so.6(gsignal+0x10b)[0x7f72df4e37bb]
|
Nov 18 01:05:21 clouddb1017 mysqld[9432]: /lib/x86_64-linux-gnu/libc.so.6(abort+0x121)[0x7f72df4ce535]
|
Nov 18 01:05:29 clouddb1017 mysqld[9432]: /opt/wmf-mariadb104/bin/mysqld(+0x5ad568)[0x5589f99f2568]
|
Nov 18 01:05:29 clouddb1017 mysqld[9432]: /opt/wmf-mariadb104/bin/mysqld(+0x59b83e)[0x5589f99e083e]
|
Nov 18 01:05:29 clouddb1017 mysqld[9432]: /opt/wmf-mariadb104/bin/mysqld(+0xafffce)[0x5589f9f44fce]
|
Nov 18 01:05:29 clouddb1017 mysqld[9432]: /opt/wmf-mariadb104/bin/mysqld(+0xb0067d)[0x5589f9f4567d]
|
Nov 18 01:05:29 clouddb1017 mysqld[9432]: /opt/wmf-mariadb104/bin/mysqld(+0xb1217b)[0x5589f9f5717b]
|
Nov 18 01:05:29 clouddb1017 mysqld[9432]: /opt/wmf-mariadb104/bin/mysqld(+0xa5e7b5)[0x5589f9ea37b5]
|
Nov 18 01:05:29 clouddb1017 mysqld[9432]: /opt/wmf-mariadb104/bin/mysqld(_ZN7handler12ha_write_rowEPKh+0x14d)[0x5589f9d0654d]
|
Nov 18 01:05:29 clouddb1017 mysqld[9432]: /opt/wmf-mariadb104/bin/mysqld(_ZN14Rows_log_event9write_rowEP14rpl_group_infob+0x174)[0x5589f9df87d4]
|
Nov 18 01:05:30 clouddb1017 mysqld[9432]: /opt/wmf-mariadb104/bin/mysqld(_ZN20Write_rows_log_event11do_exec_rowEP14rpl_group_info+0x7d)[0x5589f9df8d6d]
|
Nov 18 01:05:30 clouddb1017 mysqld[9432]: /opt/wmf-mariadb104/bin/mysqld(_ZN14Rows_log_event14do_apply_eventEP14rpl_group_info+0x23c)[0x5589f9decfec]
|
Nov 18 01:05:30 clouddb1017 mysqld[9432]: /opt/wmf-mariadb104/bin/mysqld(+0x607a52)[0x5589f9a4ca52]
|
Nov 18 01:05:30 clouddb1017 mysqld[9432]: /opt/wmf-mariadb104/bin/mysqld(handle_slave_sql+0x1322)[0x5589f9a55a22]
|
Nov 18 01:05:30 clouddb1017 mysqld[9432]: /opt/wmf-mariadb104/bin/mysqld(+0xd6d19b)[0x5589fa1b219b]
|
Nov 18 01:05:37 clouddb1017 mysqld[9432]: /lib/x86_64-linux-gnu/libpthread.so.0(+0x7fa3)[0x7f72dfdfcfa3]
|
Nov 18 01:05:45 clouddb1017 mysqld[9432]: /lib/x86_64-linux-gnu/libc.so.6(clone+0x3f)[0x7f72df5a54cf]
|
Nov 18 01:05:45 clouddb1017 mysqld[9432]: Trying to get some variables.
|
Nov 18 01:05:45 clouddb1017 mysqld[9432]: Some pointers may be invalid and cause the dump to abort.
|
Nov 18 01:05:45 clouddb1017 mysqld[9432]: Query (0x0):
|
Nov 18 01:05:45 clouddb1017 mysqld[9432]: Connection ID (thread ID): 599
|
Nov 18 01:05:45 clouddb1017 mysqld[9432]: Status: NOT_KILLED
|
Nov 18 01:05:45 clouddb1017 mysqld[9432]: Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=on,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=on,condition_pushdown_from_having=on
|
Nov 18 01:05:45 clouddb1017 mysqld[9432]: The manual page at https://mariadb.com/kb/en/how-to-produce-a-full-stack-trace-for-mysqld/ contains
|
Nov 18 01:05:45 clouddb1017 mysqld[9432]: information that should help you find out what is causing the crash.
|
Nov 18 01:05:45 clouddb1017 mysqld[9432]: Writing a core file...
|
Nov 18 01:05:45 clouddb1017 mysqld[9432]: Working directory at /srv/sqldata.s1
|
Nov 18 01:05:45 clouddb1017 mysqld[9432]: Resource Limits:
|
Nov 18 01:05:45 clouddb1017 mysqld[9432]: Limit Soft Limit Hard Limit Units
|
Nov 18 01:05:45 clouddb1017 mysqld[9432]: Max cpu time unlimited unlimited seconds
|
Nov 18 01:05:45 clouddb1017 mysqld[9432]: Max file size unlimited unlimited bytes
|
Nov 18 01:05:45 clouddb1017 mysqld[9432]: Max data size unlimited unlimited bytes
|
Nov 18 01:05:45 clouddb1017 mysqld[9432]: Max stack size 8388608 unlimited bytes
|
Nov 18 01:05:45 clouddb1017 mysqld[9432]: Max core file size 0 0 bytes
|
Nov 18 01:05:45 clouddb1017 mysqld[9432]: Max resident set unlimited unlimited bytes
|
Nov 18 01:05:45 clouddb1017 mysqld[9432]: Max processes 2058370 2058370 processes
|
Nov 18 01:05:45 clouddb1017 mysqld[9432]: Max open files 200001 200001 files
|
Nov 18 01:05:45 clouddb1017 mysqld[9432]: Max locked memory 65536 65536 bytes
|
Nov 18 01:05:45 clouddb1017 mysqld[9432]: Max address space unlimited unlimited bytes
|
Nov 18 01:05:45 clouddb1017 mysqld[9432]: Max file locks unlimited unlimited locks
|
Nov 18 01:05:45 clouddb1017 mysqld[9432]: Max pending signals 2058370 2058370 signals
|
Nov 18 01:05:45 clouddb1017 mysqld[9432]: Max msgqueue size 819200 819200 bytes
|
Nov 18 01:05:45 clouddb1017 mysqld[9432]: Max nice priority 0 0
|
Nov 18 01:05:45 clouddb1017 mysqld[9432]: Max realtime priority 0 0
|
Nov 18 01:05:45 clouddb1017 mysqld[9432]: Max realtime timeout unlimited unlimited us
|
Nov 18 01:05:45 clouddb1017 mysqld[9432]: Core pattern: /var/tmp/core/core.%h.%e.%p.%t
|
Nov 18 01:05:57 clouddb1017 systemd[1]: mariadb@s1.service: Main process exited, code=killed, status=6/ABRT
|
The server that is being used a clone source never had any issues.
I am going to try to reclone them, and start them with innodb_change_buffering=none
For what is worth: Answering to Marko's question about innodb_purge_threads=1 - we've never had anything different there. It is (and always has been our default.)
Looks as if an index inconsistency like this can also lead to wrong SELECT results without raising any log warnings at all.
E.g. a lookup on a secondary key may point to a row where the actual column value is a different one than the one in the index.
hholzgra, yes, this is pretty nasty. By design, the secondary index records may be trusted, unless an older version of a record is being retrieved.
We also have some ‘covering index’ read features that attempt to trust the secondary index records even more. There was a recent fix related to one such feature that is disabled by default (MDEV-12486, MDEV-20422, MDEV-23600).
I think that it would make sense to properly implement CHECK TABLE…EXTENDED for InnoDB, to actually check that each secondary index records matches something in the table, instead of just counting the records that exist in the current read view. Currently, InnoDB is ignoring any other CHECK TABLE attributes than QUICK.
When it comes to Galera, another possible error source are corruption that was injected during snapshot transfer. I only have a mild suspicion that something like that is possible, but no actual evidence.
marostegui, I have some potentially good news. I suspect that MDEV-24449 could explain this corruption, and that 10.5 is not affected by that bug.
Do you have any 10.5 instances in production?
No we don't, we are in process of finishing our migration to 10.4 
Any chances that what you found might be fixable on 10.4?
marostegui, sure, I think that MDEV-24449 is fixable, by waiting for the count of pending reads to reach 0. I was just hoping to get early validation (or rejection) of my hypothesis by a large-scale user of 10.5. I believe that we should succeed in repeating the corruption internally soon (possibly after a couple weeks of vacation period), to validate the hypothesis and the fix.
Marko, were you able to reach any conclusions?
marostegui, as you can read in MDEV-24449, I was able to repeat some form of corruption. I do not think I can do much more than wait for feedback from users like you, once releases with the fixes of MDEV-24449 and MDEV-24709 become available and widely used. The reporter feedback in MDEV-14643 makes me optimistic.
Thanks Marko. I will keep you posted. We will upgrade to 10.4.18 as soon as it gets released.
Thank you so much for investigating this bug!
We've seen this issue again on 10.4.18.
Upgrading a host from 10.4.15 to 10.4.18, starting mariadb, starting replication and then restarting the daemon ends up with errors:
Mar 05 09:13:22 db2146 mysqld[21494]: 2021-03-05 9:13:22 9 [ERROR] InnoDB: Record in index `el_to` of table `enwiki`.`externallinks` was not found on update: TUPLE (info_bits=0, 3 fields): {[60]https://www.jstor.org/action/doBasicSearch?Query=%22D.+Liam+(0x68747470733A2F2F7777772E6A73746F722E6F72672F616374696F6E2F646F42617369635365617263683F51756572793D253232442E2B4C69616D2B),[4] k (0x03FE6BFF),[4]&@^ (0x26405EFB)} at: COMPACT RECORD(info_bits=0, 3 fields): {[60]https://www.jstor.org/action/doBasicSearch?Query=%22D.+L.+Ha(0x68747470733A2F2F7777772E6A73746F722E6F72672F616374696F6E2F646F42617369635365617263683F51756572793D253232442E2B4C2E2B4861),[4] :T!(0x033A5421),[4] N (0x114EA7FB)}
|
Mar 05 09:13:22 db2146 mysqld[21494]: 2021-03-05 9:13:22 1 [ERROR] InnoDB: Unable to find a record to delete-mark
|
Mar 05 09:13:22 db2146 mysqld[21494]: InnoDB: tuple DATA TUPLE: 2 fields;
|
Mar 05 09:13:22 db2146 mysqld[21494]: 0: len 60; hex 68747470733a2f2f6f72672e6a73746f722e7777772e2f616374696f6e2f646f42617369635365617263683f51756572793d253232442e2b4c69616d; asc https://org.jstor.www./action/doBasicSearch?Query=%22D.+Liam;;
|
Mar 05 09:13:22 db2146 mysqld[21494]: 1: len 4; hex 26405efb; asc &@^ ;;
|
Mar 05 09:13:22 db2146 mysqld[21494]: InnoDB: record PHYSICAL RECORD: n_fields 2; compact format; info bits 0
|
Mar 05 09:13:22 db2146 mysqld[21494]: 0: len 30; hex 68747470733a2f2f6f72672e6a73746f722e7777772e2f616374696f6e2f; asc https://org.jstor.www./action/; (total 60 bytes);
|
Mar 05 09:13:22 db2146 mysqld[21494]: 1: len 4; hex 247f6f49; asc $ oI;;
|
Mar 05 09:13:22 db2146 mysqld[21494]: 2021-03-05 9:13:22 1 [ERROR] InnoDB: page [page id: space=693, page number=5776369] (145 records, index id 2309).
|
Mar 05 09:13:22 db2146 mysqld[21494]: 2021-03-05 9:13:22 1 [ERROR] InnoDB: Submit a detailed bug report to https://jira.mariadb.org/
|
Mar 05 09:13:22 db2146 mysqld[21494]: 2021-03-05 9:13:22 1 [ERROR] InnoDB: Unable to find a record to delete-mark
|
Mar 05 09:13:22 db2146 mysqld[21494]: InnoDB: tuple DATA TUPLE: 2 fields;
|
Mar 05 09:13:22 db2146 mysqld[21494]: 0: len 60; hex 687474703a2f2f636f6d2e676f6f676c652e7777772e2f7365617263683f7462733d626b733a3126713d253232442e2b4c69616d2b446f7272697325; asc http://com.google.www./search?tbs=bks:1&q=%22D.+Liam+Dorris%;;
|
Mar 05 09:13:22 db2146 mysqld[21494]: 1: len 4; hex 26405ef1; asc &@^ ;;
|
Mar 05 09:13:22 db2146 mysqld[21494]: InnoDB: record PHYSICAL RECORD: n_fields 2; compact format; info bits 0
|
Mar 05 09:13:22 db2146 mysqld[21494]: 0: len 30; hex 687474703a2f2f636f6d2e676f6f676c652e7777772e2f7365617263683f; asc http://com.google.www./search?; (total 60 bytes);
|
Mar 05 09:13:22 db2146 mysqld[21494]: 1: len 4; hex 247f6f3f; asc $ o?;;
|
Mar 05 09:13:22 db2146 mysqld[21494]: 2021-03-05 9:13:22 1 [ERROR] InnoDB: page [page id: space=693, page number=4700659] (194 records, index id 2308).
|
Mar 05 09:13:22 db2146 mysqld[21494]: 2021-03-05 9:13:22 1 [ERROR] InnoDB: Submit a detailed bug report to https://jira.mariadb.org/
|
Mar 05 09:16:37 db2146 mysqld[21494]: 2021-03-05 9:16:37 9 [ERROR] InnoDB: Record in index `el_from_index_60` of table `enwiki`.`externallinks` was not found on update: TUPLE (info_bits=0, 3 fields): {[4] W|(0x01F4577C),[43]https://com.google.www./search?q=Curlscurve(0x68747470733A2F2F636F6D2E676F6F676C652E7777772E2F7365617263683F713D4375726C736375727665),[4]&@U (0x26405514)} at: COMPACT RECORD(info_bits=0, 3 fields): {[4] W|(0x01F4577C),[43]https://com.google.www./search?q=Coolie+FAN(0x68747470733A2F2F636F6D2E676F6F676C652E7777772E2F7365617263683F713D436F6F6C69652B46414E),[4]&7 (0x2637F1B2)}
|
Mar 05 09:16:58 db2146 mysqld[21494]: 2021-03-05 9:16:58 1 [ERROR] InnoDB: Unable to find a record to delete-mark
|
Mar 05 09:16:58 db2146 mysqld[21494]: InnoDB: tuple DATA TUPLE: 3 fields;
|
Mar 05 09:16:58 db2146 mysqld[21494]: 0: len 4; hex 80000000; asc ;;
|
Mar 05 09:16:58 db2146 mysqld[21494]: 1: len 20; hex 447265616d636174636865725f2867726f757029; asc Dreamcatcher_(group);;
|
Mar 05 09:16:58 db2146 mysqld[21494]: 2: len 4; hex 03c3cdbb; asc ;;
|
Mar 05 09:16:58 db2146 mysqld[21494]: InnoDB: record PHYSICAL RECORD: n_fields 3; compact format; info bits 0
|
Mar 05 09:16:58 db2146 mysqld[21494]: 0: len 4; hex 80000000; asc ;;
|
Mar 05 09:16:58 db2146 mysqld[21494]: 1: len 20; hex 447265616d636174636865725f2867726f757029; asc Dreamcatcher_(group);;
|
Mar 05 09:16:58 db2146 mysqld[21494]: 2: len 4; hex 03b04d29; asc M);;
|
Mar 05 09:16:58 db2146 mysqld[21494]: 2021-03-05 9:16:58 1 [ERROR] InnoDB: page [page id: space=746, page number=70060] (333 records, index id 2481).
|
Mar 05 09:16:58 db2146 mysqld[21494]: 2021-03-05 9:16:58 1 [ERROR] InnoDB: Submit a detailed bug report to https://jira.mariadb.org/
|
We cannot know though if the corruption was present before or not.
marostegui, I am sad to read that. But, has latent corruption been ruled out? Can you try initializing the 10.4.18 server from a logical dump?
Yes, I have started to do so early today 
Hopefully will be ready by Monday.
I am also running a check table on another host that showed up corruption.
Will get back to you!
I have done some more testing:
- The host that got rebuilt from the logical dumps showed no issues.
- The hosts that had some InnoDB errors on the logs got a check table runs across all their databases and some tables had the indexes corrupted. I have rebuilt those tables and stopped/started mariadb a few times and so far so good, no more InnoDB errors.
I am now evaluating if there's any performance impact on setting 'innodb_change_buffering = none' and if there's no issues, I am inclined to deploy that across our fleet to prevent future issues.
marostegui, thank you. We are not currently aware of any other issues with the change buffer.
For completeness, I’d like to mention some recent changes in this area:
- Also
MDEV-24709could have caused corruption related to the change buffer. It was fixed in the same releases asMDEV-24449. MDEV-24913was a bogus debug assertion, nothing to worry about.- The 10.5.9 release (that release only) is affected by one change buffer related bug,
MDEV-24997caused by the fix ofMDEV-24569. Its impact should be minimal: crash recovery or Mariabackup could fail if the record is the first one that is written for a user .ibd file since the previous log checkpoint.
It would be very helpful if you could afford to enable the change buffer on at least one server, to confirm that it is now safe to use. I would keep this bug open until you have confirmed after a few weeks that the change buffer now appears safe to use. According to our benchmarks, the change buffer is sometimes helpful, and MDEV-11634 could improve it in a future release.
Thanks Marko for the extended feedback.
In production, all our hosts do have "innodb_change_buffering = all" (on 10.4.15 and now on 10.4.18). Even all the hosts I mentioned above that got their tables checked, those have "all" as a value too. So far so good.
I have recursively scanned all our mysql logs and I haven't found any other InnoDB corruptions on current (and previous mysql logs). We'll see once we keep upgrading from 10.4.15 to 10.4.18 and restart the daemon...
We only switched a few to "none" after all the crashes we faced months ago. Those hosts aren't production per se (they do replicate from production masters) but they get some users queries.
There's something I have noticed in a few clones we've done between hosts (different pair of hosts) (with mysql stopped on both)
Source host: 10.4.15
Target: 10.4.18
Data cloned successfully and once 10.4.18 gets started and replication flows:
Mar 11 08:21:34 db2148 mysqld[3190]: 2021-03-11 8:21:34 1 [ERROR] InnoDB: Unable to find a record to delete-mark
|
Mar 11 08:21:34 db2148 mysqld[3190]: InnoDB: tuple DATA TUPLE: 4 fields;
|
Mar 11 08:21:34 db2148 mysqld[3190]: 0: len 4; hex 80000002; asc ;;
|
Mar 11 08:21:34 db2148 mysqld[3190]: 1: len 4; hex 80000000; asc ;;
|
Mar 11 08:21:34 db2148 mysqld[3190]: 2: len 24; hex e58db0e7acace5ae89e4babae5a4b4e5838fe98791e5b881; asc ;;
|
Mar 11 08:21:34 db2148 mysqld[3190]: 3: len 4; hex 002fdea0; asc / ;;
|
Mar 11 08:21:34 db2148 mysqld[3190]: InnoDB: record PHYSICAL RECORD: n_fields 4; compact format; info bits 0
|
Mar 11 08:21:34 db2148 mysqld[3190]: 0: len 4; hex 80000002; asc ;;
|
Mar 11 08:21:34 db2148 mysqld[3190]: 1: len 4; hex 80000000; asc ;;
|
Mar 11 08:21:34 db2148 mysqld[3190]: 2: len 24; hex e58db0e7acace5ae89e4babae5a4b4e5838fe98791e5b881; asc ;;
|
Mar 11 08:21:34 db2148 mysqld[3190]: 3: len 4; hex 001389d7; asc ;;
|
A check table on the source host reveals no corruption.
A check table on the target host reveals corruption on some of the indexes.
I don't have an explanation for that, could be that the transfer corrupted the files, but that is very unlikely.
marostegui, a possible explanation is hidden corruption in the source host. I think that you have to treat anything that was not restored from a logical dump as corrupted.
The current hypothesis is that before the fix of MDEV-24449 (and MDEV-24709), if any other setting than innodb_change_buffering=none was ever used, and if crash recovery was ever involved (possibly as part of restoring a physical backup), then secondary indexes can become corrupted. We know of cases where CHECK TABLE does not catch such corruption. This is because the InnoDB implementation is only counting records in different indexes and not actually comparing the columns. The EXTENDED attribute of CHECK TABLE is ignored by InnoDB.
Thanks Marko.
We have started to run table rebuilds after recloning hosts.
For the record, I consider this ticket as free support for Wikipedia. I would keep this ticket open until this has been confirmed as a duplicate MDEV-24449. I expect that to take some time, and I have now set the `need_feedback` label, waiting for marostegui to respond after monitoring the situation for a few months.
Note: While fixing MDEV-24449 should prevent further corruption occurring due to change buffering, it will not magically fix corruption in old data files. Physical copying or mariabackup will also copy such corruption. The only reliable way to ‘uncorrupt’ everything (not only the secondary indexes of InnoDB tables, but also anything that is stored in the InnoDB system tablespace) is to restore from a logical dump.
In internal testing, we are seeing some secondary index corruption of the hidden FTS_DOC_ID_INDEX due to MDEV-25004 (involving SYSTEM VERSIONING and FULLTEXT INDEX).
This bug seems to affect 10.5.9 too, MDEV-24449 lists 10.2 - 10.4 in Fix Versions. So, either this is not a duplicate, or we have 10.5.x missing from that other bug. Can you, please, clarify this?
Happening at ServiceNow, can be reproduced in about 24 hours. We start with tables that pass CHECK TABLE, expose them to some load from the slave replication thread, then do a fast shutdown, restart, run CHECK TABLE on each table, now we have some tables (no particular pattern observed as to which one) that have a discrepancy in the record count between secondary index and the main part of the table. If the change buffering is only for inserts, then we find missing records in the secondary index. If we set it only for deletes, we end up with extra records in the secondary index.
We have been able to reproduce it on multiple variants of 10.4 including 10.4.18. We are not able to reproduce the issue on 10.2 at all, and we have tried many many times. So I am going through the diffs one by one to see if I can find something of interest...
The problem does not reproduce with debugging binaries - they die under load with timeouts.
Will be happy to provide more details, try a custom build, instrument whatever part of the code you think is the suspect with a diagnostic message/assert, etc...
spachev, if the database has not been restored with mariabackup and no crash recovery has ever been involved with a server that did not include a fix of MDEV-24449, then the corruption must be something else. Currently we have MDEV-25459 open, but I did not yet have enough time to find its root cause.
MDEV-25459 turned out to be a bug in the MVCC code that would cause some secondary index records to be missed. We have had similar cases earlier. In fact, I now remember from about 10 years ago that our InnoDB tester at Oracle would distinguish ‘transient corruption’ and ‘persistent corruption’ (one that would remain even after restarting the server). MDEV-25459 would definitely be a case of ‘transient corruption’. Examples of ‘persistent corruption’ would include MDEV-18272 and MDEV-24449.
I think that the case reported in this ticket is a case of persistent corruption and cannot be in any way caused by MDEV-25459, because locking operations (UPDATE or DELETE) would read the latest version from each index and simply skip delete-marked records in each index.
We should keep in mind that corruption does not disappear by itself; InnoDB was not designed with any self-healing capabilities in mind. If the database has been restored by a corrupted physical backup, bad things may continue to happen. When it comes to the nasty low-probability bug that was fixed in MDEV-24449, the only really safe way is to restore the database from a logical backup (such as mysqldump). If you read one of my last comments in MDEV-24449, you can see that I reproduced a ‘double free’ of a page in the system tablespace. This is a sign for very nasty things, such as a corrupted change buffer (see MDEV-23839) or undo logs (with symptoms that resemble MDEV-14407). A milder consequence of that bug is that a secondary index in a user table gets corrupted. I was unable to repeat that exact consequence.
If you are lucky and the system tablespace is not corrupted, it could be possible to heal the database by executing ALTER TABLE…FORCE, LOCK=SHARED on each corrupted table. That operation should only traverse the path from the clustered index B-tree page root to the first leaf page and then sequentially read the leaf pages and any BLOB pages.
Our corruption has very consistent predictable pattern:
- Only happens when the change buffer is enabled
- Only happens if the server gets shut down via fast shutdown
- Is discovered via CHECK TABLE (or if we happen to run the right type of query against the problem table)
- The problem that CHECK TABLE finds is always a discrepancy between the number of records in the clustered index and a secondary index.
- Can always be fixed by repairing with OPTIMIZE TABLE - or we should say, after running OPTIMIZE TABLE CHECK TABLE now succeeds and we do not see any failing queries.
- The bug does reproduce even with innodb_purge_threads=1
- We are not able to reproduce this type of corruption with any of the 10.2 versions, but can reproduce in under 4 hours with any 10.4 (including special builds with bug fix patches).
Running CHECK TABLE for each table after stopping the slave, and then performing the fast shutdown and restart results in no corruption (apparently because CHECK TABLE brings every page into the buffer pool causing the change buffer merge).
So the bug quite clearly is somewhere in the handling of unmerged change buffer + fast shutdown + restart.
What we have not yet tried, but will shortly is make sure all open transactions have been committed before we perform the fast shutdown + restart. We will report on the results once we have done this.
I believe we also experienced this last week on MariaDB 10.3.27 on Debian 10. Been a super hectic, stressful week and I'm still collecting my thoughts/notes but I thought I'd report our initial findings here.
Started to build out three new database secondaries. We use mariabackup. I restored the new secondary as I usually do (copy full, diff, and incr over, uncompress, prepare full, apply diff, apply incr) and started replication to have them catch up. Almost immediately after starting replication we started seeing these errors:
2021-04-23 0:42:20 0 [ERROR] InnoDB: Unable to find a record to delete-mark
|
InnoDB: tuple DATA TUPLE: 6 fields;
|
0: len 4; hex 80000001; asc ;;
|
1: len 4; hex 80002627; asc &';;
|
2: len 9; hex 556868206d656f773f; asc Uhh meow?;;
|
3: len 4; hex 8000e9b6; asc ;;
|
4: len 1; hex 44; asc D;;
|
5: len 4; hex 8085fdc9; asc ;;
|
|
|
InnoDB: record PHYSICAL RECORD: n_fields 6; compact format; info bits 0
|
0: len 4; hex 80000001; asc ;;
|
1: len 4; hex 80002627; asc &';;
|
2: len 9; hex 556868206d656f773f; asc Uhh meow?;;
|
3: len 4; hex 8000e9b6; asc ;;
|
4: len 1; hex 41; asc A;;
|
5: len 4; hex 8087e946; asc F;;
|
2021-04-23 0:42:20 0 [ERROR] InnoDB: page [page id: space=889, page number=243075] (370 records, index id 255).
|
2021-04-23 0:42:20 0 [ERROR] InnoDB: Submit a detailed bug report to https://jira.mariadb.org/
|
|
|
2021-04-23 0:43:42 51 [ERROR] InnoDB: Record in index `score_value` of table `ads`.`flashapi_scores_best` was not found on update: TUPLE (info_bits=0, 6 fields): {[4] i (0x800069B4),[4] % (0x80002582),[4] (0x80000001),[1]W(0x57),[4] l! (0x806C2198),[9]Uhh meow?(0x556868206D656F773F)} at: COMPACT RECORD(info_bits=0, 6 fields): {[4] i (0x800069B4),[4] % (0x80002582),[4] (0x80000001),[1]P(0x50),[4] 0(0x8089C530),[9]Uhh meow?(0x556868206D656F773F)}
|
Eventually MariaDB crashed:
2021-04-23 09:07:29 0x7f1320511700 InnoDB: Assertion failure in file /build/mariadb-10.3-Zg5BW3/mariadb-10.3-10.3.27/storage/innobase/row/row0ins.cc line 221
|
InnoDB: Failing assertion: !cursor->index->is_committed()
|
InnoDB: We intentionally generate a memory trap.
|
InnoDB: Submit a detailed bug report to https://jira.mariadb.org/
|
InnoDB: If you get repeated assertion failures or crashes, even
|
InnoDB: immediately after the mysqld startup, there may be
|
InnoDB: corruption in the InnoDB tablespace. Please refer to
|
InnoDB: https://mariadb.com/kb/en/library/innodb-recovery-modes/
|
InnoDB: about forcing recovery.
|
210423 9:07:29 [ERROR] mysqld got signal 6 ;
|
Thankfully after reading through this ticket a couple times (esp. @MG 's posts thank you SO much for taking the time to post here) I figured out that we had a bunch of tables with corrupt indexes. After repairing them and starting the secondary again everything got caught up and seems to be working fine now.
I checked all tables and found several with corrupted indexes. The corruption didn't seem to be the same between secondaries which I don't get. If mariabackup copies the corruption, I would expect the same exact tables to be corrupted on all servers restoring from the same backups, but that didn't appear to be the case.
The in-production secondaries aren't reporting any of these errors but I did indeed find that at least one table had corrupted indexes (I haven't checked the rest yet). But we've never had a problem with them. It seems like something in the restore process might be causing/triggering this issue. I know that crash recovery might be involved, but I know for a fact that these servers have gone through at least one crash recovery as a couple months back our hosting provider rebooted them all one morning.
I was in a panic after discovering that our data/backups could all be corrupted due to MDEV-24449 so I quickly upgraded one secondary to 10.3.28 from the MariaDB repos, set up a logical backup, and restored this secondary using that and we still got these same errors!
I was completely exhausted at this point so it's possible I messed something up with this restore. It's been a LONG time since I've had to do a logical backup/restore (but will be doing one from now one as an extra precautionary measure). I am going to test this again with mydumper so it hopefully takes less than 12+ hours.
Right now I have innodb_change_buffering = none on the 10.3.28 server that I used to restore a logical backup as well as two other secondaries with 10.3.27 just to be safe. I'm going to try reverting this setting to default here in a bit to see if this is indeed the problem. If this happens on 10.3.28 WITH the change buffer off and a logical backup restore, then I wonder if this was even the problem to begin with.
But as far as I know, rebuilding any corrupted indexes e.g. alter table X engine=innodb; seems to fix this issue. I also ran pt-table-checksum last night and everything seems fine. Every tool available to me seems to indicate that our data (including backups) is not corrupted beyond indexes in some tables.
Like I said, I'm going to attempt another logical backup and restore on this server with 10.3.28 to see if I get these same errors again. I am also going to re-enable the change buffer on one of the other two new secondaries to see if any indexes get corrupted this week.
If anyone wants me to test anything else please let me know.
@Bren
If you do CHECK TABLE for each table when you know that you are in a corrupted state, what type of corruption gets reported?
@Sasha Pachev
2021-04-25 15:48:04 767994659 [ERROR] InnoDB: Flagged corruption of `publisher_id` in table `ads`.`flashapi_user_scores` in CHECK TABLE; Wrong count
|
I'm doing a restore now with mydumper/myloader on the 10.3.28 node now. Will be interesting to see if I get the same errors like I did when I restored with mysqldump.
Actually, now that I've reviewed the logs with a bit clearer head, I think I know what happened on the 10.3.28 host I restored from logical backups. I think I entered the wrong gtid and started replication in the wrong place so naturally I started getting a lot of duplicate key errors and stuff like that. I hoped maybe if I skipped a few of those I would be able to salvage the restore but I quickly found out that it was a complete failure.
So I definitely was getting one of the same errors as I was on the host with the corrupted indexes but probably for different reasons.
My logical restore on the 10.3.28 host finally finished. Replication is still catching up but zero errors so far. Once it's caught up I'll re-enable the change buffer and see what happens.
On the 10.3.27 host I restored from a physical (maria) backup and where I originally got these errors, I checked all tables, repaired the ones that were corrupt, disabled the change buffer, and restarted replication. It's been running without error for over a day now.
Today I re-enabled the change buffer on this host, let replication run for a minute, then stopped replication and did a (fast) shutdown. Once I had it back up and replication started again, I started getting the exact same errors I listed above. It was almost immediate. This confirms @Sasha Pachev's observations. Thankfully doing an alter table seems to be the only thing that needs to be done to correct this.
This might also explain why the current production systems haven't been reporting these errors at all. I'm pretty sure that I have not done a fast shutdown on any of these since they were upgraded to 10.3.27. Thank goodness I discovered this before the next upgrade!
OK so I was able to get this to happen on 10.3.28 within a few seconds. Looks like I will have to keep the change buffer off for now.
@Bren:
Yes, keep the change buffer off unless you really need it. If you absolutely have to turn it on, I would recommend making sure it gets completely merged before you do hot backup or fast shutdown. You can monitor the change buffer state with:
SELECT COUNT(*) FROM information_schema.innodb_buffer_page WHERE page_type = 'IBUF_INDEX'; |
So you should be able to set innodb_change_buffering to "none" dynamically, then wait for the above query to return 0, then move forward with the shutdown/backup.
On the CHECK TABLE, you should have seen a message from this code:
push_warning_printf(
|
thd, Sql_condition::WARN_LEVEL_WARN,
|
ER_NOT_KEYFILE,
|
"InnoDB: Index '%-.200s' contains " ULINTPF |
" entries, should be " ULINTPF ".", |
index->name(), n_rows, n_rows_in_table);
|
in the output of the CHECK TABLE to the client. Do you happen to have that message around? It would provide more details on what was wrong with that index, namely the secondary vs clustered index (the "real" data) count discrepancy.
(Is it just me or are mentions not working?)
Sasha: I don't have the CHECK TABLE output available (will run that next time) but I do have mysqlcheck which looks similar to what you're asking for:
Warning : InnoDB: Index 'total' contains 3496043 entries, should be 3496044.
error : Corrupt
I've started disable the change buffer already and haven't noticed an impact so I will likely just disable across the entire environment. I ran the query you sent on these new secondaries that have the change buffer disabled and it doesn't return zero. Is that a problem?
@Bren:
I can think of three possibilities:
- The merging has not yet finished. When you turn it off, it only prevents the new writes to the change buffer, but the ones already there need to be merged, and it could take some time. Is the number going down with time?
- There is a subtlety about how the buffer pages are marked that I do not understand.
- There is a bug.
My take on the corruption you reported is that one insert merge was marked as done, removed from the change buffer, but did not actually get merged into the secondary index. We are having similar problems as I described in my posts earlier. I plan to do some code study over the next few days to see how this might be possible.
Mentions are not working for me either.
I should have mentioned that I ran that query on hosts where the change buffer was never on. The number doesn't appear to decrement at all. On one host for example, it was 351 hours ago and it's 351 now. Luckily this doesn't appear to be causing problems.
Thanks for posting this info. I really appreciate it!
medletan, yes, 10.3.27 would still lack the fix of MDEV-24449. If you want to be sure to not encounter this issue, you have to restore from a logical backup. Apparently you did that, on 10.3.28 (which I suppose what you mean by "3.3.28"). Are you claiming that the problem persists on MariaDB Server 10.3.28 even after the logical restore?
Marko: oops, yes, not sure how I messed up the version numbers not once but three times (been a long couple of weeks) but yes I was talking about 10.3.28. I found this ticket right away after we first experienced this on 10.3.27 and thought that MDEV-24449 was the fix so I switched one of the hosts over to the MariaDB repos and upgraded to 10.3.28.
Then I did a logical restore on this server which also had the change buffer set to "none" the entire time I did the restore and while I waited for replication to catch up. I let it run for the better part of a day, no errors. Restarted it a few times, still no errors.
Then I set the change buffer back to its default setting, restarted MariaDB, started replication again, did a fast shutdown, restarted replication, and within seconds I got all the same errors I did before:
2021-04-27 20:26:55 - started replication up with the change buffer set to "all"
|
|
|
2021-04-27 20:27:19 - stopped replication and did a fast shutdown
|
|
|
2021-04-27 20:28:13 - started replication again
|
|
|
2021-04-27 20:28:27 1 [ERROR] InnoDB: Unable to find a record to delete-mark
|
|
|
2021-04-27 20:28:27 1 [ERROR] InnoDB: page [page id: space=208, page number=184566] (695 records, index id 586).
|
|
|
2021-04-27 20:28:29 47 [ERROR] InnoDB: Record in index `score_value` of table `ads`.`flashapi_scores_best` was
|
not found on update: TUPLE....
|
|
|
[many more errors]
|
So MDEV-24449 doesn't appear to fix this.
medletan, thank you. Can you provide the table definition? There might be something in the secondary index definition that triggers a bug in the change buffer:
SELECT NAME,TABLE_ID FROM INFORMATION_SCHEMA.INNODB_SYS_TABLES WHERE SPACE=208; |
SELECT * FROM INFORMATION_SCHEMA.INNODB_SYS_INDEXES WHERE ID=586; |
I would like to see enough of the SHOW CREATE TABLE output to determine the layout of the secondary index. Secondary index records contain the columns (or column prefixes) that are part of the index, as well as all PRIMARY KEY columns.
spachev, when it comes to 10.4, one thing that is worth noting that due to MDEV-371, UNIQUE indexes that are defined on long string columns will internally define an index on a hidden virtual column. If you see column names like DB_ROW_HASH_1 in INFORMATION_SCHEMA.INNODB_SYS_VIRTUAL, then you are affected by that. Unfortunately, there currently is no way to disable MDEV-371 (so that you would get an error when trying to create such indexes), and there are many bugs related to indexed virtual columns, many of which MariaDB inherited from MySQL 5.7. In the InnoDB layer, I think that all logic related to evaluating virtual columns could be removed by MDEV-17598 and MDEV-22361.
We definitely have an issue with duplicate/overlapping indexes in a LOT of tables. I just opened a ticket about that today in fact. Here is one of the tables that got corrupted:
Table: flashapi_scores_best
|
Create Table: CREATE TABLE `flashapi_scores_best` (
|
`score_id` int(13) NOT NULL,
|
`publisher_id` int(4) NOT NULL,
|
`user_id` int(13) NOT NULL,
|
`user_name` varchar(20) NOT NULL,
|
`score_value` int(13) NOT NULL,
|
`archive_type` char(1) NOT NULL,
|
`tag` varchar(32) NOT NULL,
|
PRIMARY KEY (`score_id`,`publisher_id`,`archive_type`,`user_id`,`tag`),
|
KEY `score_id` (`score_id`,`publisher_id`,`archive_type`,`tag`),
|
KEY `archive_type` (`archive_type`,`publisher_id`),
|
KEY `archive_type_2` (`archive_type`),
|
KEY `publisher_id` (`publisher_id`,`score_id`,`tag`),
|
KEY `score_value` (`score_value`),
|
KEY `score_sort` (`publisher_id`,`score_id`,`tag`,`score_value`,`archive_type`),
|
KEY `score_sort_sans_tag` (`publisher_id`,`score_id`,`archive_type`,`score_value`),
|
KEY `score_id_sans_tag` (`publisher_id`,`score_id`,`archive_type`)
|
) ENGINE=InnoDB DEFAULT CHARSET=utf8
|
marko - we have nothing in INFORMATION_SCHEMA.INNODB_SYS_VIRTUAL.
I was playing around with the change buffer code to get an understanding of how it works and observed that the wrong count of corruption - exactly the kind we are getting - can be achieved by modifying `ibuf_merge_or_delete_for_page()` to unconditionally return, or by making `ibuf_insert_low()` to return DB_SUCCESS unconditionally without doing any work. I am thinking it is unlikely that the problem comes through `ibuf_insert_low()` path, but I am wondering if in some code path when doing fast shutdown + recovery, we end up not calling `ibuf_merge_or_delete_for_page()` when we should - either due to not writing an entry in the redo log, or not processing it correctly. Thoughts?
medletan - mentions work for me if I use the syntax
[~medletan]
|
medletan, thank you. I see nothing really special in your table definition: no virtual columns, no column prefix indexes.
In MDEV-11634, I have written down some thoughts on improving the change buffer. Starting with 10.5, the code already is a little more manageable, because due to MDEV-19514, change buffer merge normally occurs only when the secondary index with buffered changes needs to be accessed.
One more thought (which does not apply to medletan’s table definition): The setting SET unique_checks=0 is a ticking time-bomb. Most of the time, it would not disable any checks. But, if it happens so that a unique secondary index leaf page is not in the buffer pool, InnoDB could decide to buffer an insert into that unique index. Corruption due to the UNIQUE KEY violation would be noticed much later. I do not know how or when; I have not tested it lately.
spachev, it would be really helpful if you can generate a test case that would reproduce this. I would suggest that you use a modified non-debug executable where the setting innodb_change_buffering_debug would be available. The setting innodb_change_buffering_debug=1 would allow the change buffer to be exercised more. You should also try to tweak the SQL so that some of the secondary indexes are not accessed at all by queries. It could be helpful to define lots of redundant secondary indexes, like in medletan’s table. Concurrent to your workload (lots of UPDATE of indexed columns could work), you should run CHECK TABLE…QUICK, but not too frequently because it will force change buffer merges. If CHECK TABLE reports index count mismatch, we have won. But, be sure not to trigger MDEV-25459, or do apply that fix to your server code.
One more thing that you both could test easily is: Would setting innodb_adaptive_hash_index=OFF prevent the corruption? We have some open bugs in the adaptive hash index, in particular one simplified Random Query Generator grammar that can reproduce a race conditions. I hope that we will have a chance to work on that fairly soon. Both the adaptive hash index and the change buffer are unique InnoDB features that may improve performance in some cases but seem to be a constant source of very challenging bugs.
For the record, the adaptive hash index bug that I mentioned is MDEV-25527. It could be a regression due to MDEV-22456.
marko - thanks for the suggestions, will try those in our tests. In the meantime, does the fact that the wrong count corruption happens ONLY if you restart the server, ONLY if it is a fast shutdown, ONLY if the change buffer has something in it when the shutdown begins, and the corruption is immediately discovered with CHECK TABLE as soon as the server comes up post-restart tell you anything?
spachev, unfortunately your observations do not tell me anything yet. You probably are aware that any read from a secondary index leaf page will trigger merge if buffered changes exist. CHECK TABLE…QUICK would read each secondary index leaf page when counting the records.
If you can reproduce the problem very easily, then it might be a significant observation. Are you saying that if you shutdown and restart the server with innodb_fast_shutdown=0, CHECK TABLE after the restart never reports a problem, while with innodb_fast_shutdown=1 it sometimes would?
As long as we are comparing an orderly shutdown (not innodb_fast_shutdown=2 or killing the server), I do not think that there should be much difference. It is theoretically possible that if log-based recovery is involved, we might forget to write redo log for change buffer operations. Such bugs have in fact been found and fixed years ago.
marko - Yes, innodb_fast_shutdown=0 never produces a post-restart count mismatch corruption, while innodb_fast_shutdown=1 very reliably does. We have tried both with innodb_adaptive_hash_index ON and OFF obtaining the same corrupted results. I wondered if perhaps we are losing the records in the change buffer in the shutdown-restart, so I ported the debug record dump loop at the end of ibuf_init_at_db_start() into a separate function that counts the records and used it to print the change buffer record count at the end of ibuf_init_at_db_start() as well as at the start of innodb_shutdown(), and we are seeing the same number of records even in the cases of corruption - so it is not due to the loss of the records. I am assuming then the problem is that somehow it fails to merge during recovery. The record mismatch diff has the order of magnitude of several hundred per table with 10-20 tables corrupted. With the background merge disabled (via a patch that allows it to be disabled in a non-debug build ) we have about 80 million records in the change buffer when the shutdown is triggered.
Our next steps are to clean up our 4 hour test cycle removing unnecessary time-consuming steps and keep instrumenting mysqld with extra diagnostics. I would like to add a dump of the records that are present in the secondary index, but absent from the clustered and the other way around. Any tips on the technical details would be appreciated.
spachev, thank you. If there is an orderly shutdown, you should not see either of these messages on startup:
const char* msg = last_batch |
? "Starting final batch to recover " |
: "Starting a batch to recover "; |
ib::info() << msg << n << " pages from redo log."; |
I mention this message, because only during the final batch of recovery we may apply buffered changes to the pages that we loaded into the buffer pool. Before the final batch, we are not allowed to generate any redo log records. Merging the change buffer (moving data between persistent B-trees) generates redo log. But, once again, unless the server crashed during the shutdown, you should not see either message.
The other difference between innodb_fast_shutdown=1 and innodb_fast_shutdown=0 is that the slow shutdown will also force all transaction history to be purged. Maybe with innodb_fast_shutdown=0 the change buffer will be used less by purge, because the pages could already reside in the buffer pool.
I think that we have to use all possible means to narrow this down. My suggestions in ascending order of difficulty are:
- Does this occur with other values of innodb_change_buffering than the default all?
- First, try innodb_change_buffering=changes (disabling the purge-buffering that was introduced in MySQL 5.5)
- If that does not help, try innodb_change_buffering=inserts.
- (Did you already determine that innodb_change_buffering=none works?)
- Does a different value of innodb_purge_threads affect the probability?
- Did you try patching in some of the logic of innodb_change_buffering_debug (ibuf_debug) to make the use of change buffer more likely? This could allow the problem to be repeated with smaller amounts of data even when using a larger buffer pool.
- Is this repeatable at all on MariaDB 10.5? (
MDEV-19514and other changes may have affected this.)
Because we internally test with smaller amounts of data and typically reinitialize the database very frequently, it would be extremely unlikely for us to hit this in our internal testing. Also, the timing behavior of debug instrumented builds could be significantly different. I do not expect to be able to use https://rr-project.org/ on this one. (It would offer a huge productivity boost: diagnosing MDEV-15326 took several days, while analysing rr replay traces of failures typically is a matter of at most a few hours, as demonstrated by more complex bugs like MDEV-22139.)
I think that we really depend on you to narrow this down to something that I will be able to debug interactively.
marko Answers to your questions for the ones I have the info:
- innodb_change_buffering=none does not reproduce the problem
- innodb_change_buffering=inserts results in a secondary index having fewer records that the clustered
- innodb_change_buffering=deletes results in a secondary index having more records than the clustered
- Our results are 100% reproducible, just take a long time unfortunately (we are working on fixing that). With our without purge threads.
Will collect/obtain the other info you requested, and post when available.
It looks like are also running into this bug on 10.4.18. It seems to also only happen when innodb_change_buffering and innodb_fast_shutdown are enabled. We haven't observed this behavior on 10.1, 10.2, or 10.3. We are not running 10.5 or 10.6 anywhere. It seems to happen more frequently when restarting mysqld when there are active writes (like when the SQL thread is lagging behind and actively processing transactions). It doesn't seem to be affected by reads - the problem happens even when there are zero reads. Single we are using zfs it's somewhat easy for us to rollback and try different settings, versions, etc. We are happy to help provide any information that would help in debugging.
I have done extensive internal testing against the 10.4.12 release (in both optimized and debug) at the same revision as the release download available, and with the information provided here. Options used where:
--innodb_change_buffering=all --innodb_fast_shutdown=1 --sql_mode= --log-bin --innodb_purge_threads=32
|
I have thus far found a large number of bugs, though not the specific assert listed in this issue. Testing continues with post-shutdown restarts and subsequent check tables etc. Additionally, we will soon include a replication setup with mid-run shutdown whilst running multi-threaded workloads, as well as interleaved SQL heavily relying on tables with many secondary indexes. Still, given the issues I have seen in 10.4.12 which included various malloc/memory and deemed OOM issues, I would strongly encourage anyone who can "start afresh with reliable data from a dump" (or similar) to try 10.4.19 available from https://mariadb.org/download/ to see if this will prevent the issue from happening. Also please always double check there is no OOS (disk full) or OOM happening. I can especially say this given a number of deemed out of memory issues I saw whilst testing this release. It may make sense to run a script to keep track of memory allocation, for example by running the following in a loop every 30 seconds and logging the results to disk:
ps --sort -rss -eo pid,pmem,rss,vsz,comm | head -n20
|
And watching top process memory utilization. Finally, for anyone affected by this, there are also some workarounds possible, for example turning off innodb_fast_shutdown (set to =0) and innodb_change_buffering (set to =none). Any feedback/thoughts/pointers for further testing also welcome.
I am not sure yet if MDEV-25745 is something 10.5 specific (a regression from MDEV-12353 and related work) or if the user data that I analyzed demonstrates that the change buffer is occasionally bypassing the redo log and writing buffer pool pages directly. In any case, my suggestion there seems to have produced something, which I will try to analyze tomorrow:
…tiny buffer pool and a heavily indexed table (say, index(a,b,c,d),index(a,b,d,c),…,index(d,c,b,a) and UPDATE … SET a=random_data). Avoid any WHERE condition on any of those indexed columns, so that change buffer merges will be avoided for as long as possible.
mleich filed MDEV-25783 for some change buffer related corruption with a table that contains 10,000 rows. There was no luck on MDEV-25475 yet, though. Both these problems might be specific to 10.5.
Found a/another corruption specific to 10.4.12, seems unlikely to be related MDEV-25836
I completed the root cause analysis in MDEV-25783. During the execution that I analyzed, the change buffer record is lost due to the change that I made in MDEV-20934 (10.2.29, 10.3.20, 10.4.10, 10.5.0) in order to avoid a hang in change buffer merge when the data structures are corrupted. More than a year later we found and fixed the likely root cause of that in MDEV-24449 (10.2.37, 10.3.28, 10.4.18, 10.5.9).
This should explain why spachev experienced the corruption in 10.4.12 and 10.4.18 but not in a custom build of 10.2.27 (which includes a fix of MDEV-24449 but not the problematic work-around MDEV-20934).
I will leave this ticket open even after closing MDEV-25783, so that affected users can confirm whether the problem has finally been fixed.
Sorry for giving false hope. After some more investigation, I concluded that MDEV-25783 affects the normal operation of the 10.5 server, while in earlier versions of the server, it would only be possible (and probably extremely unlikely) during a shutdown with innodb_fast_shutdown=0. In this ticket, spachev claimed that a shutdown with innodb_fast_shutdown=0 works fine, while the default setting innodb_fast_shutdown=1 causes the trouble.
elenst provided a 10.4 rr replay trace and a data directory where a small table becomes corrupted after the last restart:
InnoDB: Index 'idx1289024_5' contains 3000 entries, should be 3001.
In fact, also 2 other indexes report the same error, but I will concentrate on the first one. Using the methods that I developed in MDEV-25783, I can find the mismatch. First, make CHECK TABLE…QUICK dump each visited row in the clustered index and in the secondary index of interest:
diff --git a/storage/innobase/row/row0mysql.cc b/storage/innobase/row/row0mysql.cc
|
index ce0746f20c6..dc24a02d4ad 100644
|
--- a/storage/innobase/row/row0mysql.cc
|
+++ b/storage/innobase/row/row0mysql.cc
|
@@ -4792,6 +4792,8 @@ row_scan_index_for_mysql(
|
|
offsets = rec_get_offsets(rec, index, offsets_, index->n_core_fields,
|
ULINT_UNDEFINED, &heap);
|
+ if (index->is_clust() || index->id == 101)
|
+ ib::info() << rec_printer(rec, offsets).str();
|
|
if (prev_entry != NULL) {
|
matched_fields = 0;
|
diff --git a/storage/innobase/rem/rem0rec.cc b/storage/innobase/rem/rem0rec.cc
|
index c4886101499..edf4de839cf 100644
|
--- a/storage/innobase/rem/rem0rec.cc
|
+++ b/storage/innobase/rem/rem0rec.cc
|
@@ -2584,12 +2584,12 @@ rec_print(
|
o << '['
|
<< local_len
|
<< '+' << BTR_EXTERN_FIELD_REF_SIZE << ']';
|
- ut_print_buf(o, data, local_len);
|
+ ut_print_buf_hex(o, data, local_len);
|
ut_print_buf_hex(o, data + local_len,
|
BTR_EXTERN_FIELD_REF_SIZE);
|
} else {
|
o << '[' << len << ']';
|
- ut_print_buf(o, data, len);
|
+ ut_print_buf_hex(o, data, len);
|
}
|
}
|
|
diff --git a/storage/innobase/include/dict0mem.h b/storage/innobase/include/dict0mem.h
|
index 9a8675a0a1d..8e599a57f06 100644
|
--- a/storage/innobase/include/dict0mem.h
|
+++ b/storage/innobase/include/dict0mem.h
|
@@ -2379,6 +2379,7 @@ inline bool dict_index_t::is_instant() const
|
|
inline bool dict_index_t::is_corrupted() const
|
{
|
+ return false;
|
return UNIV_UNLIKELY(online_status >= ONLINE_INDEX_ABORTED
|
|| (type & DICT_CORRUPT)
|
|| (table && table->corrupted)); |
The last hunk is needed for enabling access to an index that was marked as corrupted in the persistent metadata. We can filter the output with the following:
sed -ne 's/.*5 fields): {\([^,]*\),\([^,]*\),\([^,]*\),\[6\][^,]*,\[7\][^,]*}/{\1,\2,\3}/p' < mysqld.err |sort -u >sec.txt
|
sed -ne 's/.*3 fields): //p' < mysqld.err|sort -u|diff -u - sec.txt|grep '^[+-]{'
|
This will produce exactly one line:
+{[4](0x00000000),[0](0x),[0](0x)}
|
A secondary index record for a clustered index record is missing. The clustered index record is at the very start of the dump:
Version: '10.4.20-MariaDB-debug' socket: '/tmp/mysql.sock' port: 3306 Source distribution
|
2021-06-07 8:22:22 0 [Note] InnoDB: Buffer pool(s) load completed at 210607 8:22:22
|
2021-06-07 8:22:30 4 [Note] InnoDB: COMPACT RECORD(info_bits=0, 5 fields): {[4](0x00000000),[0](0x),[0](0x),[6](0x000000000000),[7](0x80000000000000)}
|
In this case, all columns of the table belong to the PRIMARY KEY, and the key of the secondary index record happens to be the first column of the PRIMARY KEY, so no permutation of the fields is needed. We can see that the DB_TRX_ID,DB_ROLL_PTR at the end of the record have already been reset by MDEV-12288.
In the data directory of the server immediately before the last restart, the option innodb_change_buffer_dump that is only available in debug builds included 3 records, which look very familiar:
PHYSICAL RECORD: n_fields 7; 1-byte offsets; info bits 0
|
0: len 4; hex 0000000b; asc ;;
|
1: len 1; hex 00; asc ;;
|
2: len 4; hex 0000001d; asc ;;
|
3: len 22; hex 00000001860300048000840f0014803f840f00ff803f; asc ? ?;;
|
4: len 4; hex 00000000; asc ;;
|
5: len 0; hex ; asc ;;
|
6: len 0; hex ; asc ;;
|
PHYSICAL RECORD: n_fields 7; 1-byte offsets; info bits 0
|
0: len 4; hex 0000000b; asc ;;
|
1: len 1; hex 00; asc ;;
|
2: len 4; hex 00000029; asc );;
|
3: len 22; hex 00000001860300048000840f0014803f840f00ff803f; asc ? ?;;
|
4: len 4; hex 00000000; asc ;;
|
5: len 0; hex ; asc ;;
|
6: len 0; hex ; asc ;;
|
PHYSICAL RECORD: n_fields 7; 1-byte offsets; info bits 0
|
0: len 4; hex 0000000b; asc ;;
|
1: len 1; hex 00; asc ;;
|
2: len 4; hex 0000002b; asc +;;
|
3: len 22; hex 00000001860300048000840f00ff803f840f0014803f; asc ? ?;;
|
4: len 4; hex 00000000; asc ;;
|
5: len 0; hex ; asc ;;
|
6: len 0; hex ; asc ;;
|
The last 3 fields of each record seem to match our record. By setting a breakpoint in rr replay inside ibuf_merge_or_delete_for_page(), we can confirm that page number 11:29 (the first record) belongs to our index. It is a delete-mark operation. But, in the page 11:29 (which is the leftmost leaf page of the index) that record is not found. It should be the first record.
Some deeper analysis of the 3 rr replay traces will be needed to figure out when exactly the corruption was introduced and why. A watchpoint on the clustered index record should be helpful. It is located at the logical start of the leftmost leaf page of the clustered index (page 11:0x13):
00013f30: 8000 0000 0000 0000 0000 0208 f141 0000 .............A..
|
00013f40: 0000 0000 0000 0000 8000 0000 0000 0000 ................
|
Of course, setting a watchpoint is easier said than done, because this page is loaded and evicted from the buffer pool several times during the second run. Finally, the interesting execution (inserting our row of interest) starts here:
Thread 47 hit Hardware watchpoint 4: -location $4.frame[0x61]@2
|
|
|
Old value = "\000\034"
|
New value = <incomplete sequence \333>
|
mach_write_to_2 (n=<optimized out>, b=<optimized out>) at storage/innobase/include/mach0data.ic:60
|
60 b[0] = (byte)(n >> 8);
|
(rr) bt
|
#0 mach_write_to_2 (n=<optimized out>, b=<optimized out>) at storage/innobase/include/mach0data.ic:60
|
#1 rec_set_next_offs_old (next=<optimized out>, rec=<optimized out>) at storage/innobase/include/rem0rec.ic:361
|
#2 page_rec_set_next (next=<optimized out>, rec=<optimized out>) at storage/innobase/include/page0page.ic:752
|
#3 page_cur_insert_rec_low (current_rec=0x7fff6e4be063 "infimum", index=index@entry=0x7fd5100ff6b8, rec=<optimized out>, offsets=<optimized out>, mtr=mtr@entry=0x7fff67ffb710)
|
at storage/innobase/page/page0cur.cc:1402
|
#4 0x000055589efd68cb in page_cur_tuple_insert (cursor=cursor@entry=0x7fff67ffad78, tuple=tuple@entry=0x37744101268, index=index@entry=0x7fd5100ff6b8, offsets=offsets@entry=0x7fff67ffad58,
|
heap=heap@entry=0x7fff67ffad50, n_ext=<optimized out>, mtr=0x7fff67ffb710) at storage/innobase/include/page0cur.ic:285
|
#5 0x000055589efdc479 in btr_cur_optimistic_insert (flags=flags@entry=0, cursor=cursor@entry=0x7fff67ffad70, offsets=offsets@entry=0x7fff67ffad58, heap=heap@entry=0x7fff67ffad50,
|
entry=entry@entry=0x37744101268, rec=rec@entry=0x7fff67ffb1b0, big_rec=0x7fff67ffad48, n_ext=<optimized out>, thr=0x7fd51010af38, mtr=0x7fff67ffb710) at storage/innobase/btr/btr0cur.cc:3607
|
#6 0x000055589ef3fb22 in row_ins_clust_index_entry_low (flags=flags@entry=0, mode=<optimized out>, mode@entry=2, index=index@entry=0x7fd5100ff6b8, n_uniq=n_uniq@entry=3, entry=entry@entry=0x37744101268,
|
n_ext=n_ext@entry=0, thr=0x7fd51010af38) at storage/innobase/row/row0ins.cc:2742
|
#7 0x000055589ef43fd1 in row_ins_clust_index_entry (index=index@entry=0x7fd5100ff6b8, entry=0x37744101268, thr=thr@entry=0x7fd51010af38, n_ext=n_ext@entry=0) at storage/innobase/row/row0ins.cc:3217
|
#8 0x000055589ef461a4 in row_ins_index_entry (thr=0x7fd51010af38, entry=<optimized out>, index=<optimized out>) at storage/innobase/row/row0ins.cc:3343
|
#9 row_ins_index_entry_step (thr=0x7fd51010af38, node=<optimized out>) at storage/innobase/row/row0ins.cc:3512
|
#10 row_ins (thr=<optimized out>, node=<optimized out>) at storage/innobase/row/row0ins.cc:3671
|
#11 row_ins_step (thr=thr@entry=0x7fd51010af38) at storage/innobase/row/row0ins.cc:3821
|
#12 0x000055589ef559ad in row_insert_for_mysql (mysql_rec=mysql_rec@entry=0x7fd5100fd558 "", prebuilt=0x7fd51010a408, ins_mode=ins_mode@entry=ROW_INS_NORMAL) at storage/innobase/row/row0mysql.cc:1420
|
#13 0x000055589eea68c1 in ha_innobase::write_row (this=0x7fd5100f89e0, record=0x7fd5100fd558 "") at storage/innobase/handler/ha_innodb.cc:8122
|
#14 0x000055589ed3016c in handler::ha_write_row (this=0x7fd5100f89e0, buf=0x7fd5100fd558 "") at sql/handler.cc:6757
|
#15 0x000055589eb36245 in write_record (thd=thd@entry=0x7fd518000c08, table=table@entry=0x7fd5100fc758, info=info@entry=0x7fff67ffc5c0) at sql/sql_insert.cc:2061
|
#16 0x000055589eb3bd5b in mysql_insert (thd=thd@entry=0x7fd518000c08, table_list=<optimized out>, fields=..., values_list=..., update_fields=..., update_values=..., duplic=<optimized out>,
|
ignore=<optimized out>) at sql/sql_insert.cc:1078
|
#17 0x000055589eb66f57 in mysql_execute_command (thd=thd@entry=0x7fd518000c08) at sql/sql_parse.cc:4600
|
#18 0x000055589eb6a885 in mysql_parse (thd=0x7fd518000c08, rawbuf=<optimized out>, length=<optimized out>, parser_state=<optimized out>, is_com_multi=<optimized out>, is_next_command=<optimized out>)
|
at sql/sql_parse.cc:7992
|
#19 0x000055589eb6c0d6 in dispatch_command (command=command@entry=COM_QUERY, thd=thd@entry=0x7fd518000c08,
|
packet=packet@entry=0x7fd518007559 "INSERT /* QNO 1049 CON_ID 42 */ IGNORE INTO `t5` ( `iwl_from` ) VALUES ( '' )", packet_length=packet_length@entry=77, is_com_multi=is_com_multi@entry=false,
|
is_next_command=is_next_command@entry=false) at sql/sql_parse.cc:1857
|
#20 0x000055589eb6d291 in do_command (thd=0x7fd518000c08) at sql/sql_parse.cc:1373
|
#21 0x000055589ec36bb4 in do_handle_one_connection (connect=<optimized out>) at sql/sql_connect.cc:1412
|
#22 0x000055589ec36c69 in handle_one_connection (arg=<optimized out>) at sql/sql_connect.cc:1316
|
#23 0x00007fd50def06db in start_thread (arg=0x7fff67fff700) at pthread_create.c:463
|
#24 0x00007fd50e22971f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
|
(rr) when
|
Current event: 667138
|
The insert of the record of interest is being buffered to change buffer page 1913:
#2 ibuf_insert_low (mode=mode@entry=36, op=op@entry=IBUF_OP_INSERT, no_counter=no_counter@entry=0, entry=entry@entry=0x37744101598, entry_size=entry_size@entry=11, index=index@entry=0x7fd5101152f8, page_id=
|
{m_space = 11, m_page_no = 29}, zip_size=<optimized out>, thr=<optimized out>) at storage/innobase/ibuf/ibuf0ibuf.cc:3546
|
#3 0x000055589eeddfcf in ibuf_insert (op=op@entry=IBUF_OP_INSERT, entry=entry@entry=0x37744101598, index=index@entry=0x7fd5101152f8, page_id=page_id@entry={m_space = 11, m_page_no = 29},
|
zip_size=zip_size@entry=0, thr=0x7fd51010af38) at storage/innobase/ibuf/ibuf0ibuf.cc:3690
|
#4 0x000055589efe0622 in btr_cur_search_to_nth_level_func (index=index@entry=0x7fd5101152f8, level=level@entry=0, tuple=tuple@entry=0x37744101598, mode=mode@entry=PAGE_CUR_LE, latch_mode=<optimized out>,
|
latch_mode@entry=514, cursor=cursor@entry=0x7fff67ffada0, file=<optimized out>, line=<optimized out>, mtr=<optimized out>, autoinc=<optimized out>) at storage/innobase/btr/btr0cur.cc:1659
|
#5 0x000055589ef442fb in row_ins_sec_index_entry_low (flags=flags@entry=0, mode=mode@entry=2, index=index@entry=0x7fd5101152f8, offsets_heap=<optimized out>, offsets_heap@entry=0x7fd518038a40,
|
heap=heap@entry=0x7fd518037880, entry=entry@entry=0x37744101598, trx_id=0, thr=<optimized out>) at storage/innobase/row/row0ins.cc:2973
|
A little later, that record is being moved around in the change buffer tree while an insert to an entirely different tablespace 7 is being buffered:
(rr) when
|
Current event: 693997
|
(rr) bt
|
#0 page_delete_rec_list_end (rec=rec@entry=0x7fff6e1ba295 "", block=block@entry=0x7fff6c7b5370, index=index@entry=0x5558a271e268, n_recs=<optimized out>, size=2189, mtr=mtr@entry=0x59916690faa0)
|
at storage/innobase/page/page0page.cc:1175
|
#1 0x000055589ef16255 in page_move_rec_list_end (new_block=new_block@entry=0x7fff6c744f80, block=block@entry=0x7fff6c7b5370, split_rec=split_rec@entry=0x7fff6e1ba295 "", index=0x5558a271e268,
|
mtr=mtr@entry=0x59916690faa0) at storage/innobase/page/page0page.cc:1321
|
#2 0x000055589efc9755 in btr_page_split_and_insert (flags=flags@entry=3, cursor=cursor@entry=0x59916690f830, offsets=offsets@entry=0x59916690f6e8, heap=heap@entry=0x59916690f6e0,
|
tuple=tuple@entry=0x5c74800317b8, n_ext=<optimized out>, mtr=<optimized out>) at storage/innobase/btr/btr0btr.cc:3095
|
#3 0x000055589efd9962 in btr_cur_pessimistic_insert (flags=flags@entry=3, cursor=cursor@entry=0x59916690f830, offsets=offsets@entry=0x59916690f6e8, heap=heap@entry=0x59916690f6e0,
|
entry=entry@entry=0x5c74800317b8, rec=rec@entry=0x59916690f6f8, big_rec=0x59916690f6d8, n_ext=<optimized out>, thr=0x7fd510024a58, mtr=0x59916690faa0) at storage/innobase/btr/btr0cur.cc:3823
|
#4 0x000055589eedd816 in ibuf_insert_low (mode=mode@entry=32801, op=op@entry=IBUF_OP_INSERT, no_counter=no_counter@entry=0, entry=entry@entry=0x5c7480026528, entry_size=entry_size@entry=74,
|
index=index@entry=0x7fd510031a18, page_id={m_space = 7, m_page_no = 2051}, zip_size=<optimized out>, thr=<optimized out>) at storage/innobase/ibuf/ibuf0ibuf.cc:3519
|
The record just moved to the end of new_block, so we will adjust our watchpoint and carry on. Another split:
(rr) when
|
Current event: 774219
|
(rr) bt
|
…
|
#2 0x000055589efc9755 in btr_page_split_and_insert (flags=flags@entry=3, cursor=cursor@entry=0x7fff6c043830, offsets=offsets@entry=0x7fff6c0436e8, heap=heap@entry=0x7fff6c0436e0,
|
tuple=tuple@entry=0x7fd4e4069248, n_ext=<optimized out>, mtr=<optimized out>) at storage/innobase/btr/btr0btr.cc:3095
|
#3 0x000055589efd9962 in btr_cur_pessimistic_insert (flags=flags@entry=3, cursor=cursor@entry=0x7fff6c043830, offsets=offsets@entry=0x7fff6c0436e8, heap=heap@entry=0x7fff6c0436e0,
|
entry=entry@entry=0x7fd4e4069248, rec=rec@entry=0x7fff6c0436f8, big_rec=0x7fff6c0436d8, n_ext=<optimized out>, thr=0x7fd4e4054da0, mtr=0x7fff6c043aa0) at storage/innobase/btr/btr0cur.cc:3823
|
#4 0x000055589eedd816 in ibuf_insert_low (mode=mode@entry=32801, op=op@entry=IBUF_OP_INSERT, no_counter=no_counter@entry=0, entry=entry@entry=0x7fd4e406add8, entry_size=entry_size@entry=29,
|
index=index@entry=0x7fd510114158, page_id={m_space = 11, m_page_no = 25}, zip_size=<optimized out>, thr=<optimized out>) at storage/innobase/ibuf/ibuf0ibuf.cc:3519
|
Finally, it gets a little more interesting: our change buffer record is no longer the last one in the page:
(rr) when
|
Current event: 1018223
|
(rr) bt
|
#0 mach_write_to_2 (n=<optimized out>, b=<optimized out>) at storage/innobase/include/mach0data.ic:60
|
#1 rec_set_next_offs_old (next=<optimized out>, rec=<optimized out>) at storage/innobase/include/rem0rec.ic:361
|
#2 page_rec_set_next (next=<optimized out>, rec=<optimized out>) at storage/innobase/include/page0page.ic:752
|
#3 page_cur_insert_rec_low (current_rec=0x7fff6c86c5db "", index=index@entry=0x5558a271e268, rec=<optimized out>, offsets=<optimized out>, mtr=mtr@entry=0x7fff6c044cf0)
|
at storage/innobase/page/page0cur.cc:1402
|
#4 0x000055589efd68cb in page_cur_tuple_insert (cursor=cursor@entry=0x7fff6c044a88, tuple=tuple@entry=0x7fd4e406cdf8, index=index@entry=0x5558a271e268, offsets=offsets@entry=0x7fff6c044938,
|
heap=heap@entry=0x7fff6c044930, n_ext=<optimized out>, mtr=0x7fff6c044cf0) at storage/innobase/include/page0cur.ic:285
|
#5 0x000055589efdc479 in btr_cur_optimistic_insert (flags=flags@entry=3, cursor=cursor@entry=0x7fff6c044a80, offsets=offsets@entry=0x7fff6c044938, heap=heap@entry=0x7fff6c044930,
|
entry=entry@entry=0x7fd4e406cdf8, rec=rec@entry=0x7fff6c044948, big_rec=0x7fff6c044928, n_ext=<optimized out>, thr=0x7fd4e4054da0, mtr=0x7fff6c044cf0) at storage/innobase/btr/btr0cur.cc:3607
|
#6 0x000055589eedd74b in ibuf_insert_low (mode=mode@entry=36, op=op@entry=IBUF_OP_DELETE_MARK, no_counter=no_counter@entry=0, entry=entry@entry=0x7fd4e4056628, entry_size=entry_size@entry=275,
|
index=index@entry=0x7fd5101152f8, page_id={m_space = 11, m_page_no = 30}, zip_size=<optimized out>, thr=<optimized out>) at storage/innobase/ibuf/ibuf0ibuf.cc:3483
|
Note: this is buffering a delete-mark for a different page (30, not 29). We will keep the watchpoint as is, because we expect it to be triggered when our record of interest is being deleted. The next event is the shutdown of the server, which initiates a rollback. Like I have pointed out in MDEV-11634, rollback never makes use of the change buffer (only INSERT, UPDATE, DELETE, purge do):
Thread 1 received signal SIGUSR1, User defined signal 1.
|
[Switching to Thread 26260.26260]
|
0x0000000070000002 in ?? ()
|
(rr) when
|
Current event: 1040522
|
(rr) c
|
Continuing.
|
2021-06-06 11:58:25 0 [Note] /home/mdbe/MDEV-22373/rqg-mdev22373/../10.4-instrumented/sql/mysqld (initiated by: rqg[rqg] @ localhost [127.0.0.1]): Normal shutdown
|
2021-06-06 11:58:25 0 [Note] Event Scheduler: Purging the queue. 0 events
|
2021-06-06 11:58:25 0 [Note] InnoDB: FTS optimize thread exiting.
|
[Switching to Thread 26260.26322]
|
|
|
Thread 62 hit Hardware watchpoint 12: -location $ib3.frame[0x5d9]@2
|
|
|
Old value = "\f_"
|
New value = "\r_"
|
mach_write_to_2 (n=<optimized out>, b=<optimized out>) at storage/innobase/include/mach0data.ic:61
|
61 b[1] = (byte)(n);
|
2: m_log = <error: current stack frame does not contain a variable named `this'>
|
(rr) bt
|
#0 mach_write_to_2 (n=<optimized out>, b=<optimized out>) at storage/innobase/include/mach0data.ic:61
|
#1 rec_set_next_offs_old (next=<optimized out>, rec=<optimized out>) at storage/innobase/include/rem0rec.ic:361
|
#2 page_rec_set_next (next=<optimized out>, rec=<optimized out>) at storage/innobase/include/page0page.ic:752
|
#3 page_cur_delete_rec (cursor=cursor@entry=0x7fff6c0438e8, index=0x5558a271e268, offsets=offsets@entry=0x7fff6c043430, mtr=mtr@entry=0x7fff6c044830) at storage/innobase/page/page0cur.cc:2412
|
#4 0x000055589efe3ce5 in btr_cur_optimistic_delete_func (cursor=cursor@entry=0x7fff6c0438e0, mtr=mtr@entry=0x7fff6c044830) at storage/innobase/btr/btr0cur.cc:5890
|
#5 0x000055589eed1ef8 in ibuf_delete_rec (space=11, page_no=30, pcur=pcur@entry=0x7fff6c0438e0, search_tuple=search_tuple@entry=0x7fd4e40406a8, mtr=mtr@entry=0x7fff6c044830)
|
at storage/innobase/ibuf/ibuf0ibuf.cc:4191
|
…
|
#29 0x000055589ec42b27 in trans_rollback_stmt (thd=thd@entry=0x7fd4e4016c08) at sql/transaction.cc:495
|
#30 0x000055589eb648ab in mysql_execute_command (thd=thd@entry=0x7fd4e4016c08) at sql/sql_parse.cc:6240
|
#31 0x000055589eb6a885 in mysql_parse (thd=0x7fd4e4016c08, rawbuf=<optimized out>, length=<optimized out>, parser_state=<optimized out>, is_com_multi=<optimized out>, is_next_command=<optimized out>)
|
at sql/sql_parse.cc:7992
|
#32 0x000055589eb6c0d6 in dispatch_command (command=command@entry=COM_QUERY, thd=thd@entry=0x7fd4e4016c08,
|
packet=packet@entry=0x7fd4e401d559 "UPDATE /* QNO 1024 CON_ID 31 */ IGNORE `t5` SET `iwl_title` = '45645', `iwl_title` = '2994' WHERE `iwl_from` >= '127'", packet_length=packet_length@entry=117,
|
is_com_multi=is_com_multi@entry=false, is_next_command=is_next_command@entry=false) at sql/sql_parse.cc:1857
|
This still is a false alarm: the change buffer record after ours (covering page 30, not page 29) is being deleted. After all records for page 5:30 are deleted by ibuf_merge_or_delete_for_page(), our record will again be the last in the change buffer page. (By the way, the rollback of active transactions before shutdown could be prevented by setting innodb_fast_shutdown=3.) Another rollback is causing a page reorganize, so we will have to adjust our watchpoint:
#4 0x000055589efc7c98 in btr_page_reorganize_low (recovery=recovery@entry=false, z_level=6, cursor=cursor@entry=0x7fff6c042f50, index=index@entry=0x5558a271e268, mtr=mtr@entry=0x7fff6c0447b0)
|
at storage/innobase/btr/btr0btr.cc:1484
|
#5 0x000055589efc88a7 in btr_page_reorganize_block (mtr=0x7fff6c0447b0, index=0x5558a271e268, block=<optimized out>, z_level=<optimized out>, recovery=false) at storage/innobase/btr/btr0btr.cc:1696
|
#6 btr_can_merge_with_page (cursor=cursor@entry=0x7fff6c043860, page_no=page_no@entry=1898, merge_block=merge_block@entry=0x7fff6c043010, mtr=mtr@entry=0x7fff6c0447b0) at storage/innobase/btr/btr0btr.cc:5431
|
#7 0x000055589efcfea8 in btr_can_merge_with_page (mtr=0x7fff6c0447b0, merge_block=0x7fff6c043010, page_no=1898, cursor=0x7fff6c043860) at storage/innobase/include/fil0fil.h:354
|
#8 btr_compress (cursor=cursor@entry=0x7fff6c043860, adjust=adjust@entry=0, mtr=mtr@entry=0x7fff6c0447b0) at storage/innobase/btr/btr0btr.cc:3683
|
#9 0x000055589efdbf4b in btr_cur_compress_if_useful (cursor=cursor@entry=0x7fff6c043860, adjust=adjust@entry=0, mtr=mtr@entry=0x7fff6c0447b0) at storage/innobase/btr/btr0cur.cc:5739
|
#10 0x000055589efe42a4 in btr_cur_pessimistic_delete (err=err@entry=0x7fff6c043698, has_reserved_extents=has_reserved_extents@entry=1, cursor=cursor@entry=0x7fff6c043860, flags=flags@entry=0,
|
rollback=rollback@entry=false, mtr=mtr@entry=0x7fff6c0447b0) at storage/innobase/btr/btr0cur.cc:6178
|
#11 0x000055589eed2026 in ibuf_delete_rec (space=<optimized out>, page_no=33, pcur=pcur@entry=0x7fff6c043860, search_tuple=search_tuple@entry=0x7fd4e40406a8, mtr=mtr@entry=0x7fff6c0447b0)
|
at storage/innobase/ibuf/ibuf0ibuf.cc:4244
|
#12 0x000055589eed9645 in ibuf_merge_or_delete_for_page (block=<optimized out>, block@entry=0x7fff6c655fc0, page_id={m_space = 11, m_page_no = 33}, zip_size=0) at storage/innobase/ibuf/ibuf0ibuf.cc:4611
|
After the shutdown of the second rr replay trace, the data is still there in the change buffer, starting at byte offset 0x966:
od -Ax -t x1 -j 0x76a000 -N 4096 data.2/ibdata1
|
76a920 0f 09 66 00 00 00 0b 00 00 00 00 1c 00 13 01 01
|
…
|
76a960 06 01 28 0f 09 96 00 00 00 0b 00 00 00 00 1d 00
|
76a970 00 00 01 86 03 00 04 80 00 84 0f 00 14 80 3f 84
|
76a980 0f 00 ff 80 3f 00 00 00 00 2e 2a 23 1f 09 05 04
|
76a990 00 01 30 0f 09 d1 00 00 00 0b 00 00 00 00 20 00
|
We can see that this is the only record for our secondary index leaf page 11:29; the preceding record is for 11:28 and the succeeding record is for 11:32.
The above record payload matches byte-for-byte the copy of the change buffer page that corresponds to the very first insert (event 693997).
In the rr replay trace of the run that reports the CHECK TABLE failure, the ibuf_merge_or_delete_for_page() around event 299429 notices that something needs to be done to page 11:29, but in the end it is doing nothing. I must debug that change buffer merge in detail and decode field 3 of the change buffer record to see what exactly is causing the record to be skipped:
PHYSICAL RECORD: n_fields 7; 1-byte offsets; info bits 0
|
0: len 4; hex 0000000b; asc ;;
|
1: len 1; hex 00; asc ;;
|
2: len 4; hex 0000001d; asc ;;
|
3: len 22; hex 00000001860300048000840f0014803f840f00ff803f; asc ? ?;;
|
4: len 4; hex 00000000; asc ;;
|
5: len 0; hex ; asc ;;
|
6: len 0; hex ; asc ;;
|
I was somewhat surprised that the data directory for which the rr replay trace shows corruption recovers just fine when I start it locally. This suggest that the corruption may be inflicted randomly on startup. Thanks to this trace being so small, all 3 change buffer records did fit in the change buffer root page (0:4). Finally, I got the idea to set a breakpoint on ibuf_delete_rec(). It will be invoked exactly 3 times (for all of the 3 records that were buffered). I did not check the other 2 corrupted indexes, but I would now assume that they were missing one record because also the INSERT operations for those records were discarded. Here is the stack trace for deleting our record (it was the last one):
|
10.4 ddddfc33c7825609a25ce9531183a0b0fb97f206 |
#0 ibuf_delete_rec (space=space@entry=11, page_no=page_no@entry=29,
|
pcur=pcur@entry=0x7f48487fee50,
|
search_tuple=search_tuple@entry=0x38fe14000c68,
|
mtr=mtr@entry=0x7f48487ff0c0) at storage/innobase/ibuf/ibuf0ibuf.cc:4181
|
#1 0x00005569d0033e3a in ibuf_delete_recs (page_id=<optimized out>)
|
at storage/innobase/ibuf/ibuf0ibuf.cc:4302
|
#2 0x00005569d05a1b3b in buf_read_ibuf_merge_pages (sync=sync@entry=false,
|
space_ids=space_ids@entry=0x7f48487ff800,
|
page_nos=page_nos@entry=0x7f48487ff7c0, n_stored=1)
|
at storage/innobase/buf/buf0rea.cc:777
|
#3 0x00005569d0467722 in ibuf_merge_pages (
|
n_pages=n_pages@entry=0x7f48487ffe58, sync=sync@entry=false)
|
at storage/innobase/ibuf/ibuf0ibuf.cc:2462
|
#4 0x00005569d046ae22 in ibuf_merge (sync=false, sync=false,
|
n_pages=0x7f48487ffe58) at storage/innobase/ibuf/ibuf0ibuf.cc:2567
|
#5 ibuf_merge_in_background (full=full@entry=true)
|
at storage/innobase/ibuf/ibuf0ibuf.cc:2638
|
#6 0x00005569d051bd59 in srv_master_do_idle_tasks ()
|
at storage/innobase/srv/srv0srv.cc:2240
|
#7 srv_master_thread (arg=<optimized out>)
|
at storage/innobase/srv/srv0srv.cc:2383
|
The reason is somewhat surprising:
775 if (UNIV_UNLIKELY(page_nos[i] >= space->size)) {
|
776 do {
|
777 ibuf_delete_recs(page_id_t(space_ids[i],
|
778 page_nos[i]));
|
779 } while (++i < n_stored
|
780 && space_ids[i - 1] == space_ids[i]
|
781 && page_nos[i] >= space->size);
|
(rr) p space->size
|
$23 = 0
|
(rr) p space->recv_size
|
$24 = 0
|
(rr) p space.name
|
$25 = 0x2b67167d3a0 "test/t5"
|
(rr) p space.chain.start.handle
|
$26 = {m_file = -1}
|
Because we did not open the file t5.ibd yet, we do not know the size of the file, and hence we wrongly determine that the changes can be deleted (because we wrongly decided that they are out of bounds for the tablespace). That code was added by me for MDEV-21069.
Because 10.5 does not merge the change buffer in the background (thanks to MDEV-19514), 10.5 is not affected by this bug. The following should fix this:
diff --git a/storage/innobase/buf/buf0rea.cc b/storage/innobase/buf/buf0rea.cc
|
index 9bd1b16a0a2..de51e7a1e20 100644
|
--- a/storage/innobase/buf/buf0rea.cc
|
+++ b/storage/innobase/buf/buf0rea.cc
|
@@ -772,13 +772,18 @@ buf_read_ibuf_merge_pages(
|
continue;
|
}
|
|
- if (UNIV_UNLIKELY(page_nos[i] >= space->size)) {
|
+ ulint size = space->size;
|
+ if (!size) {
|
+ size = fil_space_get_size(space->id);
|
+ }
|
+
|
+ if (UNIV_UNLIKELY(page_nos[i] >= size)) {
|
do {
|
ibuf_delete_recs(page_id_t(space_ids[i],
|
page_nos[i]));
|
} while (++i < n_stored
|
&& space_ids[i - 1] == space_ids[i]
|
- && page_nos[i] >= space->size);
|
+ && page_nos[i] >= size);
|
i--;
|
next:
|
space->release(); |
I filed MDEV-25869 for the 10.2, 10.3, 10.4 specific failure that I analyzed today.
For any MySQL or MariaDB version before 10.5.0, the race condition that was fixed in MDEV-24449 could corrupt anything in the InnoDB system tablespace as well as any secondary index pages during crash recovery. For MariaDB Server 10.2, 10.3, 10.4 but not 10.5, the change MDEV-21069 introduced MDEV-25869: wrongful deletion of change buffer records after server restart for files whose first page had not yet been read.
Finally, in MariaDB Server 10.5, the wrongful deletion of change buffer records (MDEV-25783) does not even require a server restart, only a very overloaded buffer pool so that pages are being evicted frequently enough. It might be difficult to repeat that bug without debug instrumentation (innodb_change_buffering_debug=1). The execution trace that I analyzed involved a change buffer merge for an index page, followed by immediate eviction of that page, buffering an operation for the page from another thread, and finally the wrong deletion of the change buffer by the thread that had just completed the change buffer merge.
Once more, this kind of corruption will not disappear by itself. If it is the MDEV-24449 corruption, in the worst case you must restore from a logical dump, because there is no easy way to rebuild the system tablespace. For the rest, it should suffice to drop the corrupted secondary indexes and to create them again.
We have now run multiple tests with and without the buf_read_ibuf_merge_pages fix, and in every one of our tests there was corruption without the fix, and no corruption with the fix. We are now trying with the addition of a message to stderr inside the size fixing if() block to prove conclusively that our tests are actually hitting that code.
Update:
We have verified with a fprintf to stderr that we are are getting zero size in {{buf_read_ibuf_merge_pages }} ! We still doing some extra testing to make sure that the fix does not break anything else, but things are looking good so far.
There is another place in the same fie (buf0rea.cc) where we use space->size without checking if it is 0. Do we need to update it as well?
buf_read_ahead_random function
if (high > space->size)
{ high = space->size; }spachev, thank you for testing it. I am relieved. But I would not close this bug yet until we have found more feedback.
yury.chaikou, can you please post an exact link to the source code? Maybe you are referring to https://github.com/MariaDB/server/blob/mariadb-10.2.38/storage/innobase/buf/buf0rea.cc#L282 which was last changed in MDEV-23190 almost a year ago?
Marko, it is mariadb-10.4.13
https://github.com/MariaDB/server/blob/1b18cddaa23711776537ee98f16529a74ff861c2/storage/innobase/buf/buf0rea.cc#L289
yury.chaikou, thank you. MariaDB 10.4.13 was released before the MDEV-23190 fix. The code before that fix should not have caused any loss of change buffer records. The read-ahead is optional, intended to prefetch pages into the buffer pool. Avoiding the read-ahead (such as when wrongly assuming that the tablespace size is 0 pages) is not going to lead to correctness trouble, but requesting a read-ahead of non-existing pages could trigger a crash.
The need_feedback label is mostly for marostegui and Barry to confirm that with the fixes of MDEV-24449 and MDEV-25869 or MDEV-25783 present, the change buffer related corruption will stay away. It will be a couple of months before the next minor releases will take place.
Thanks Marko.
We are in process of rolling 10.4.19 from 10.4.15 and 10.4.18 (we have that version running in about 80 hosts now).
We've got 10.4.18 in about 120, so far we've not seen any of them (we did change innodb_change_buffering=none; everywhere a few months ago though).
MDEV-25869's fix will be released with 10.4.20, right?
marostegui, that is right. We will have to wait for your feedback for several months, so that the fix will be available in a release, and that it has been tested long enough.
This one is biting us also (In a Galera setup) running 10.3.29. Will this fix be in the 10.3 branch (jira says 10.4> only).
rodehoed, please read my last comment on 2021-06-07 in this ticket. Multiple problems were repeated and fixed in different versions, starting with the 10.2 series.
I am waiting for marostegui and Barry to confirm that these problems no longer occur with a recent version of MariaDB.
We are now in a mix of 10.4.19, 10.4.18 and now starting to rollout 10.4.21.
Everytime we upgrade a host we do check all the tables just in case and as I mentioned a couple of months ago we have innodb_change_buffering=none everywhere.
So far we've not seen anymore crashes related to this.
marostegui, if you are not running with the default setting of innodb_change_buffering, you should not be able to encounter the originally reported issue, even when running with an affected server.
Barry, what about you?
I am closing this due to lack of feedback. I believe that the related problems have been fixed already; see my comment on 2021-06-07 for details.
For what is worth, today we just had a crash on 10.4.21 (this host was just upgraded from 10.1).
Sep 29 00:24:49 db2080 mysqld[4266]: 2021-09-29 0:24:49 55 [ERROR] InnoDB: In pages [page id: space=1059, page number=3618135] and [page id: space=1059, page number=3618150] of index `wbt_item_terms_item_id` of table `wikidatawiki`.`wbt_item_terms`
|
Sep 29 00:24:49 db2080 mysqld[4266]: InnoDB: records in wrong order on adjacent pages
|
Sep 29 00:24:49 db2080 mysqld[4266]: InnoDB: record PHYSICAL RECORD: n_fields 2; compact format; info bits 0
|
Sep 29 00:24:49 db2080 mysqld[4266]: 0: len 4; hex 00fc7d70; asc }p;;
|
Sep 29 00:24:49 db2080 mysqld[4266]: 1: len 8; hex 00000000fae2d10c; asc ;;
|
Sep 29 00:24:49 db2080 mysqld[4266]: InnoDB: record PHYSICAL RECORD: n_fields 2; compact format; info bits 0
|
Sep 29 00:24:49 db2080 mysqld[4266]: 0: len 4; hex 0066ad08; asc f ;;
|
Sep 29 00:24:49 db2080 mysqld[4266]: 1: len 8; hex 00000000aea663dc; asc c ;;
|
Sep 29 00:24:49 db2080 mysqld[4266]: 2021-09-29 0:24:49 55 [ERROR] InnoDB: Corruption of an index tree: table `wikidatawiki`.`wbt_item_terms` index `wbt_item_terms_item_id`, father ptr page no 3618114, child page no 3618150
|
Sep 29 00:24:49 db2080 mysqld[4266]: PHYSICAL RECORD: n_fields 2; compact format; info bits 0
|
Sep 29 00:24:49 db2080 mysqld[4266]: 0: len 4; hex 0066ad08; asc f ;;
|
Sep 29 00:24:49 db2080 mysqld[4266]: 1: len 8; hex 00000000aea663dc; asc c ;;
|
Sep 29 00:24:49 db2080 mysqld[4266]: 2021-09-29 0:24:49 55 [Note] InnoDB: n_owned: 0; heap_no: 466; next rec: 125
|
Sep 29 00:24:49 db2080 mysqld[4266]: PHYSICAL RECORD: n_fields 3; compact format; info bits 0
|
Sep 29 00:24:49 db2080 mysqld[4266]: 0: len 4; hex 0066ad02; asc f ;;
|
Sep 29 00:24:49 db2080 mysqld[4266]: 1: len 8; hex 000000000ec51a93; asc ;;
|
Sep 29 00:24:49 db2080 mysqld[4266]: 2: len 4; hex 00373542; asc 75B;;
|
Sep 29 00:24:49 db2080 mysqld[4266]: 2021-09-29 0:24:49 55 [Note] InnoDB: n_owned: 0; heap_no: 351; next rec: 7475
|
Sep 29 00:24:49 db2080 mysqld[4266]: 2021-09-29 0:24:49 55 [ERROR] [FATAL] InnoDB: You should dump + drop + reimport the table to fix the corruption. If the crash happens at database startup. Please refer to https://mariadb.com/kb/en/library/innodb-recovery-modes/ for in
|
Sep 29 00:24:49 db2080 mysqld[4266]: 210929 0:24:49 [ERROR] mysqld got signal 6 ;
|
Whether this corruption was already present and it simply crashed after the upgrade (and during mysqlcheck)...that's hard to know (we've been running innodb_change_buffering = none for months though).
marostegui, in MDEV-9663 you can find many tickets reporting some inconsistency of index trees. Most of them are related to change buffering, and the open ones are mostly related to indexed virtual columns (MDEV-5800, MariaDB 10.2). If you have a backup of the data before upgrading, I think that it would be useful to run CHECK TABLE on that, before enabling any writes. My best guess for this corruption would be the recovery race conditions that were fixed in MDEV-24449 and MDEV-24709. In that case, this corruption would have been dormant until you ran CHECK TABLE (without QUICK). If you never ran CHECK TABLE before upgrading, it is plausible that you never experienced other failures due to these non-leaf secondary index pages.
Apart from bugs related to change buffering, there was MDEV-18272, but it would cause a different type of failure, and I do not think that normal applications should be affected by it.
For the record, I believe that the original reported problem was caused by MDEV-21069 and fixed by MDEV-25869. The reported server version number 10.4.12, 10.4.13, 10.3.28 match that. The reported 10.5.6 and 10.5.9 failures ought to be due to MDEV-25783.
I am sorry if my intentions were unclear, but I was looking for feedback of the type “yes, the MDEV-25869 fix seems to work.” marostegui reported that he had set innodb_change_buffering=none, which means that he would not be able to encounter this class of bugs anymore, whether they are fixed or not. So, I was waiting for that kind of feedback from other affected users. It was surprisingly hard to repeat these types of errors in our internal testing. So, I was expecting some ‘statistical validation’ from the community. I closed this basically assuming that “no news is good news.”
For future corruption reports, I suggest that new bugs be filed. Before upgrading from an older major version of MariaDB server, I suggest a normal clean shutdown (and backing up the data, just in case). I would suggest to avoid innodb_fast_shutdown=0; it is unnecessary now that MDEV-15912 has been fixed, and it might make dormant corruption worse.