Details
-
Bug
-
Status: Closed (View Workflow)
-
Major
-
Resolution: Incomplete
-
10.4.20
-
None
Description
2021-11-05 10:07:51 131 [ERROR] InnoDB: Clustered record for sec rec not found index `items_1` of table `zabbix`.`items`
InnoDB: sec index record PHYSICAL RECORD: n_fields 3; compact format; info bits 0
0: len 8; hex 0000000000002f6d; asc /m;;
1: len 30; hex 646f636b65722e6d6f6e69746f725b757365642e6d656d2c736f6e64612d; asc docker.monitor[used.mem,sonda-; (total 41 bytes);
2: len 8; hex 000000000005f5be; asc ;;
InnoDB: clust index record PHYSICAL RECORD: n_fields 62; compact format; info bits 0
0: len 8; hex 000000000005f4bf; asc ;;
1: len 6; hex 000000000000; asc ;;
2: len 7; hex 80000000000000; asc ;;
3: len 4; hex 80000012; asc ;;
4: len 0; hex ; asc ;;
5: len 0; hex ; asc ;;
6: len 8; hex 0000000000002f99; asc / ;;
7: len 20; hex 436f66726520666f7261207375626e6574203332; asc Cofre fora subnet 32;;
8: len 9; hex 7375626e65745f3332; asc subnet_32;;
9: len 1; hex 30; asc 0;;
10: len 2; hex 3764; asc 7d;;
11: len 4; hex 33363564; asc 365d;;
12: len 4; hex 80000001; asc ;;
13: len 4; hex 80000003; asc ;;
14: len 0; hex ; asc ;;
15: len 0; hex ; asc ;;
16: len 0; hex ; asc ;;
17: len 4; hex 80000000; asc ;;
18: len 0; hex ; asc ;;
19: len 0; hex ; asc ;;
20: len 0; hex ; asc ;;
21: len 0; hex ; asc ;;
22: SQL NULL;
23: SQL NULL;
24: len 0; hex ; asc ;;
25: len 0; hex ; asc ;;
26: len 4; hex 80000000; asc ;;
27: len 0; hex ; asc ;;
28: len 0; hex ; asc ;;
29: len 0; hex ; asc ;;
30: len 0; hex ; asc ;;
31: len 4; hex 80000000; asc ;;
32: SQL NULL;
33: len 0; hex ; asc ;;
34: len 0; hex ; asc ;;
35: len 4; hex 80000000; asc ;;
36: len 3; hex 333064; asc 30d;;
37: len 4; hex 80000000; asc ;;
38: len 4; hex 80000000; asc ;;
39: len 0; hex ; asc ;;
40: len 4; hex 80000000; asc ;;
41: len 0; hex ; asc ;;
42: len 8; hex 000000000005f4b7; asc ;;
43: len 2; hex 3373; asc 3s;;
44: len 0; hex ; asc ;;
45: len 0; hex ; asc ;;
46: len 0; hex ; asc ;;
47: len 3; hex 323030; asc 200;;
48: len 4; hex 80000001; asc ;;
49: len 4; hex 80000000; asc ;;
50: len 0; hex ; asc ;;
51: len 0; hex ; asc ;;
52: len 4; hex 80000000; asc ;;
53: len 4; hex 80000000; asc ;;
54: len 4; hex 80000000; asc ;;
55: len 0; hex ; asc ;;
56: len 0; hex ; asc ;;
57: len 0; hex ; asc ;;
58: len 4; hex 80000000; asc ;;
59: len 4; hex 80000000; asc ;;
60: len 4; hex 80000000; asc ;;
61: SQL DEFAULT;
TRANSACTION 421703784035608, ACTIVE 3 sec fetching rows
mysql tables in use 3, locked 0
0 lock struct(s), heap size 1128, 0 row lock(s)
MySQL thread id 131, OS thread handle 140220371769088, query id 65567758 zabbix-infra02.sep.local 10.170.20.36 zabbix Sending data
select t.itemid,t.lastlogsize,t.mtime from item_rtdata t,items i,hosts h where t.itemid=i.itemid and i.hostid=h.hostid and h.proxy_hostid=12216 order by t.itemid
InnoDB: Submit a detailed bug report to https://jira.mariadb.org/
Attachments
Issue Links
- relates to
-
-
- Closed
-
Activity
MariaDB [(none)]> SHOW CREATE TABLE `zabbix`.`items`;
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Table | Create Table |
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| items | CREATE TABLE `items` ( `itemid` bigint(20) unsigned NOT NULL, `type` int(11) NOT NULL DEFAULT 0, `snmp_oid` varchar(512) COLLATE utf8_bin NOT NULL DEFAULT '', `hostid` bigint(20) unsigned NOT NULL, `name` varchar(255) COLLATE utf8_bin NOT NULL DEFAULT '', `key_` varchar(2048) COLLATE utf8_bin NOT NULL DEFAULT '', `delay` varchar(1024) COLLATE utf8_bin NOT NULL DEFAULT '0', `history` varchar(255) COLLATE utf8_bin NOT NULL DEFAULT '90d', `trends` varchar(255) COLLATE utf8_bin NOT NULL DEFAULT '365d', `status` int(11) NOT NULL DEFAULT 0, `value_type` int(11) NOT NULL DEFAULT 0, `trapper_hosts` varchar(255) COLLATE utf8_bin NOT NULL DEFAULT '', `units` varchar(255) COLLATE utf8_bin NOT NULL DEFAULT '', `formula` varchar(255) COLLATE utf8_bin NOT NULL DEFAULT '', `logtimefmt` varchar(64) COLLATE utf8_bin NOT NULL DEFAULT '', `templateid` bigint(20) unsigned DEFAULT NULL, `valuemapid` bigint(20) unsigned DEFAULT NULL, `params` text COLLATE utf8_bin NOT NULL, `ipmi_sensor` varchar(128) COLLATE utf8_bin NOT NULL DEFAULT '', `authtype` int(11) NOT NULL DEFAULT 0, `username` varchar(64) COLLATE utf8_bin NOT NULL DEFAULT '', `password` varchar(64) COLLATE utf8_bin NOT NULL DEFAULT '', `publickey` varchar(64) COLLATE utf8_bin NOT NULL DEFAULT '', `privatekey` varchar(64) COLLATE utf8_bin NOT NULL DEFAULT '', `flags` int(11) NOT NULL DEFAULT 0, `interfaceid` bigint(20) unsigned DEFAULT NULL, `description` text COLLATE utf8_bin NOT NULL, `inventory_link` int(11) NOT NULL DEFAULT 0, `lifetime` varchar(255) COLLATE utf8_bin NOT NULL DEFAULT '30d', `evaltype` int(11) NOT NULL DEFAULT 0, `jmx_endpoint` varchar(255) COLLATE utf8_bin NOT NULL DEFAULT '', `master_itemid` bigint(20) unsigned DEFAULT NULL, `timeout` varchar(255) COLLATE utf8_bin NOT NULL DEFAULT '3s', `url` varchar(2048) COLLATE utf8_bin NOT NULL DEFAULT '', `query_fields` varchar(2048) COLLATE utf8_bin NOT NULL DEFAULT '', `posts` text COLLATE utf8_bin NOT NULL, `status_codes` varchar(255) COLLATE utf8_bin NOT NULL DEFAULT '200', `follow_redirects` int(11) NOT NULL DEFAULT 1, `post_type` int(11) NOT NULL DEFAULT 0, `http_proxy` varchar(255) COLLATE utf8_bin NOT NULL DEFAULT '', `headers` text COLLATE utf8_bin NOT NULL, `retrieve_mode` int(11) NOT NULL DEFAULT 0, `request_method` int(11) NOT NULL DEFAULT 0, `output_format` int(11) NOT NULL DEFAULT 0, `ssl_cert_file` varchar(255) COLLATE utf8_bin NOT NULL DEFAULT '', `ssl_key_file` varchar(255) COLLATE utf8_bin NOT NULL DEFAULT '', `ssl_key_password` varchar(64) COLLATE utf8_bin NOT NULL DEFAULT '', `verify_peer` int(11) NOT NULL DEFAULT 0, `verify_host` int(11) NOT NULL DEFAULT 0, `allow_traps` int(11) NOT NULL DEFAULT 0, `discover` int(11) NOT NULL DEFAULT 0, PRIMARY KEY (`itemid`), KEY `items_3` (`status`), KEY `items_4` (`templateid`), KEY `items_5` (`valuemapid`), KEY `items_6` (`interfaceid`), KEY `items_7` (`master_itemid`), KEY `items_1` (`hostid`,`key_`(1021)), CONSTRAINT `c_items_1` FOREIGN KEY (`hostid`) REFERENCES `hosts` (`hostid`) ON DELETE CASCADE, CONSTRAINT `c_items_2` FOREIGN KEY (`templateid`) REFERENCES `items` (`itemid`) ON DELETE CASCADE, CONSTRAINT `c_items_3` FOREIGN KEY (`valuemapid`) REFERENCES `valuemaps` (`valuemapid`), CONSTRAINT `c_items_4` FOREIGN KEY (`interfaceid`) REFERENCES `interface` (`interfaceid`), CONSTRAINT `c_items_5` FOREIGN KEY (`master_itemid`) REFERENCES `items` (`itemid`) ON DELETE CASCADE ) ENGINE=InnoDB DEFAULT CHARSET=utf8 COLLATE=utf8_bin |
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
1 row in set (0.000 sec)
MariaDB [(none)]> use zabbix
Reading table information for completion of table and column names
You can turn off this feature to get a quicker startup with -A
Database changed
MariaDB [zabbix]> check table items;
ERROR 2013 (HY000): Lost connection to MySQL server during query
MariaDB [zabbix]>
in the mariadb log:
2021-11-05 13:37:36 331 [ERROR] InnoDB: Records in wrong order
InnoDB: previous record PHYSICAL RECORD: n_fields 3; compact format; info bits 0
0: len 8; hex 0000000000002dcb; asc - ;;
1: len 24; hex 687253746f7261676544657363725b2f7661722f6c6f675d; asc hrStorageDescr[/var/log];;
2: len 8; hex 0000000000049b4e; asc N;;
InnoDB: record PHYSICAL RECORD: n_fields 3; compact format; info bits 0
0: len 8; hex 0000000000002dcb; asc - ;;
1: len 17; hex 687253746f7261676544657363725b2f5d; asc hrStorageDescr[/];;
2: len 8; hex 0000000000049b4c; asc L;;
2021-11-05 13:37:36 331 [ERROR] InnoDB: In pages [page id: space=234, page number=587] and [page id: space=234, page number=593] of index `items_1` of table `zabbix`.`items`
InnoDB: records in wrong order on adjacent pages
InnoDB: record PHYSICAL RECORD: n_fields 3; compact format; info bits 0
0: len 8; hex 0000000000002dcb; asc - ;;
1: len 30; hex 687253746f7261676553697a65496e42797465735b2f7661722f6c6f675d; asc hrStorageSizeInBytes[/var/log];;
2: len 8; hex 000000000005a0cb; asc ;;
InnoDB: record PHYSICAL RECORD: n_fields 3; compact format; info bits 32
0: len 8; hex 0000000000002dcb; asc - ;;
1: len 23; hex 687253746f7261676553697a65496e42797465735b2f5d; asc hrStorageSizeInBytes[/];;
2: len 8; hex 000000000005a0c9; asc ;;
2021-11-05 13:37:36 331 [ERROR] InnoDB: Corruption of an index tree: table `zabbix`.`items` index `items_1`, father ptr page no 593, child page no 587
PHYSICAL RECORD: n_fields 3; compact format; info bits 0
0: len 8; hex 0000000000002da5; asc - ;;
1: len 15; hex 73797374656d2e686f73746e616d65; asc system.hostname;;
2: len 8; hex 0000000000048e5c; asc \;;
2021-11-05 13:37:36 331 [Note] InnoDB: n_owned: 0; heap_no: 2; next rec: 163
PHYSICAL RECORD: n_fields 4; compact format; info bits 0
0: len 8; hex 0000000000002dcb; asc - ;;
1: len 23; hex 687253746f7261676553697a65496e42797465735b2f5d; asc hrStorageSizeInBytes[/];;
2: len 8; hex 000000000005a0c9; asc ;;
3: len 4; hex 00000251; asc Q;;
2021-11-05 13:37:36 331 [Note] InnoDB: n_owned: 0; heap_no: 49; next rec: 1000
2021-11-05 13:37:36 331 [ERROR] [FATAL] InnoDB: You should dump + drop + reimport the table to fix the corruption. If the crash happens at database startup. Please refer to https://mariadb.com/kb/en/library/innodb-recovery-modes/ for information about forcing recovery. Then dump + drop + reimport.
211105 13:37:36 [ERROR] mysqld got signal 6 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
To report this bug, see https://mariadb.com/kb/en/reporting-bugs
We will try our best to scrape up some info that will hopefully help
diagnose the problem, but since we have already crashed,
something is definitely wrong and this may fail.
Server version: 10.4.20-MariaDB-log
key_buffer_size=134217728
read_buffer_size=131072
max_used_connections=32
max_threads=153
thread_count=40
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 467770 K bytes of memory
Hope that's ok; if not, decrease some variables in the equation.
Thread pointer: 0x7fbcc8000a88
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 0x7fbcdd4bfcf0 thread_stack 0x49000
/usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x558ba0260b8e]
/usr/sbin/mysqld(handle_fatal_signal+0x30f)[0x558b9fcebe1f]
/lib64/libpthread.so.0(+0xf630)[0x7fbecab37630]
/lib64/libc.so.6(gsignal+0x37)[0x7fbec9f82387]
/lib64/libc.so.6(abort+0x148)[0x7fbec9f83a78]
/usr/sbin/mysqld(+0xbd43b0)[0x558b9ffe93b0]
/usr/sbin/mysqld(+0xbda19e)[0x558b9ffef19e]
/usr/sbin/mysqld(+0xbe223c)[0x558b9fff723c]
/usr/sbin/mysqld(+0xa92d3a)[0x558b9fea7d3a]
/usr/sbin/mysqld(_ZN7handler8ha_checkEP3THDP15st_ha_check_opt+0x59)[0x558b9fcf4249]
/usr/sbin/mysqld(+0x7bc87f)[0x558b9fbd187f]
/usr/sbin/mysqld(_ZN19Sql_cmd_check_table7executeEP3THD+0xae)[0x558b9fbd2d6e]
/usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x1e92)[0x558b9fad9552]
/usr/sbin/mysqld(_Z11mysql_parseP3THDPcjP12Parser_statebb+0x22b)[0x558b9fadfb5b]
/usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcjbb+0x1df9)[0x558b9fae29f9]
/usr/sbin/mysqld(_Z10do_commandP3THD+0x11b)[0x558b9fae3c3b]
/usr/sbin/mysqld(_Z24do_handle_one_connectionP7CONNECT+0x1fa)[0x558b9fbc510a]
/usr/sbin/mysqld(handle_one_connection+0x3d)[0x558b9fbc51ed]
/lib64/libpthread.so.0(+0x7ea5)[0x7fbecab2fea5]
/lib64/libc.so.6(clone+0x6d)[0x7fbeca04ab0d]
Trying to get some variables.
Some pointers may be invalid and cause the dump to abort.
Query (0x7fbcc800ffe0): CHECK table `zabbix`.`items`
Connection ID (thread ID): 331
Status: NOT_KILLED
Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=on,condition_pushdown_from_having=on
The manual page at https://mariadb.com/kb/en/how-to-produce-a-full-stack-trace-for-mysqld/ contains
information that should help you find out what is causing the crash.
Writing a core file...
Working directory at /data/mysql-data
Resource Limits:
Limit Soft Limit Hard Limit Units
Max cpu time unlimited unlimited seconds
Max file size unlimited unlimited bytes
Max data size unlimited unlimited bytes
Max stack size 8388608 unlimited bytes
Max core file size 0 unlimited bytes
Max resident set unlimited unlimited bytes
Max processes 31119 31119 processes
Max open files 32768 32768 files
Max locked memory 65536 65536 bytes
Max address space unlimited unlimited bytes
Max file locks unlimited unlimited locks
Max pending signals 31119 31119 signals
Max msgqueue size 819200 819200 bytes
Max nice priority 0 0
Max realtime priority 0 0
Max realtime timeout unlimited unlimited us
Core pattern: core
The manual page at https://mariadb.com/kb/en/how-to-produce-a-full-stack-trace-for-mysqld/ contains
information that should help you find out what is causing the crash.
Writing a core file...
Working directory at /data/mysql-data
Resource Limits:
Limit Soft Limit Hard Limit Units
Max cpu time unlimited unlimited seconds
Max file size unlimited unlimited bytes
Max data size unlimited unlimited bytes
Max stack size 8388608 unlimited bytes
Max core file size 0 unlimited bytes
Max resident set unlimited unlimited bytes
Max processes 31119 31119 processes
Max open files 32768 32768 files
Max locked memory 65536 65536 bytes
Max address space unlimited unlimited bytes
Max file locks unlimited unlimited locks
Max pending signals 31119 31119 signals
Max msgqueue size 819200 819200 bytes
Max nice priority 0 0
Max realtime priority 0 0
Max realtime timeout unlimited unlimited us
Core pattern: core
-------------------------------------------------------------------------------
i didnt try to repair(rebuild) the corrupt table.
ivertongs, is this repatable after disabling the InnoDB change buffer and rebuilding the table, as suggested in MDEV-26917?
Please provide the output of CHECK table `zabbix`.`items`; + SHOW CREATE TABLE table `zabbix`.`items`;
Did you try to repair (rebuild) the corrupt table?