[MDEV-33205] [ERROR] InnoDB: We detected index corruption in an InnoDB type table. Created: 2024-01-09  Updated: 2024-01-19

Status: Needs Feedback
Project: MariaDB Server
Component/s: Galera, Server
Affects Version/s: 10.6.14
Fix Version/s: None

Type: Bug Priority: Major
Reporter: William Wong Assignee: Marko Mäkelä
Resolution: Unresolved Votes: 0
Labels: None
Environment:

RHEL7 on VMware


Issue Links:
Relates
relates to MDEV-30531 Corrupt index(es) on busy table when ... Closed
relates to MDEV-31767 InnoDB tables are being flagged as co... Closed

 Description   

Our environment is 3 nodes Galera cluster - 2 DB nodes + 1 arbitrator. HAProxy redirected traffic to one 1 DB node.

We encountered index corruption and, then, node inconsistent on standby node.

We needed to restart DB node manually and let it triggered implicitly SST to get resolve.

Kindly advise what we can check to troubleshot this case.

2024-01-03 14:52:10 7 [ERROR] InnoDB: We detected index corruption in an InnoDB type table. You have to dump + drop + reimport the table or, in a case of widespread corruption, dump all InnoDB tables and recreate the whole tablespace. If the mariadbd server crashes after the startup or when you dump the tables. Please refer to https://mariadb.com/kb/en/library/innodb-recovery-modes/ for information about forcing recovery.
2024-01-03 14:52:10 12 [ERROR] mariadbd: Index for table 'pa_pos_psdo' is corrupt; try to repair it
2024-01-03 14:52:10 7 [ERROR] mariadbd: Index for table 'pa_pos_psdo' is corrupt; try to repair it
2024-01-03 14:52:10 7 [ERROR] Slave SQL: Could not execute Write_rows_v1 event on table tes-public.pa_pos_psdo; Index for table 'pa_pos_psdo' is corrupt; try to repair it, Error_code: 1034; handler error HA_ERR_CRASHED; the event's master log FIRST, end_log_pos 9325, Internal MariaDB error code: 1034
2024-01-03 14:52:10 7 [Warning] WSREP: Event 6 Write_rows_v1 apply failed: 126, seqno 27074073
2024-01-03 14:52:10 12 [ERROR] Slave SQL: Could not execute Write_rows_v1 event on table tes-public.pa_pos_psdo; Table tes@002dpublic/pa_pos_psdo in tablespace 94889212653016 corrupted., Error_code: 126; Index for table 'pa_pos_psdo' is corrupt; try to repair it, Error_code: 1034; handler error HA_ERR_CRASHED; the event's master log FIRST, end_log_pos 9335, Internal MariaDB error code: 1034
2024-01-03 14:52:10 12 [Warning] WSREP: Event 6 Write_rows_v1 apply failed: 126, seqno 27074074
2024-01-03 14:52:10 6 [ERROR] mariadbd: Index for table 'pa_pos_psdo' is corrupt; try to repair it
2024-01-03 14:52:10 6 [ERROR] Slave SQL: Could not execute Write_rows_v1 event on table tes-public.pa_pos_psdo; Table tes@002dpublic/pa_pos_psdo in tablespace 94889212653016 corrupted., Error_code: 126; Index for table 'pa_pos_psdo' is corrupt; try to repair it, Error_code: 1034; handler error HA_ERR_CRASHED; the event's master log FIRST, end_log_pos 9325, Internal MariaDB error code: 1034
2024-01-03 14:52:10 6 [Warning] WSREP: Event 6 Write_rows_v1 apply failed: 126, seqno 27074076
2024-01-03 14:52:10 0 [Note] WSREP: Member 1(t2vdbs-hpftes-tesdbppd-2-01) initiates vote on 9b0bd445-e8dc-11ed-aff1-6ec3700a3571:27074073,9359a244aa1e2038:  Index for table 'pa_pos_psdo' is corrupt; try to repair it, Error_code: 1034;
 
xxx
 
2024-01-03 14:52:10 7 [ERROR] WSREP: Inconsistency detected: Inconsistent by consensus on 9b0bd445-e8dc-11ed-aff1-6ec3700a3571:27074073
         at /home/buildbot/buildbot/build/galera/src/replicator_smm.cpp:process_apply_error():1357



 Comments   
Comment by Marko Mäkelä [ 2024-01-10 ]

I think that this could be a duplicate of MDEV-31767 or MDEV-30531.

Comment by Marko Mäkelä [ 2024-01-10 ]

Have you tried to upgrade to MariaDB Server 10.6.16?

Comment by William Wong [ 2024-01-19 ]

Thanks Marko
We plan to upgrade to MariaDB Server 10.6.16

Generated at Thu Feb 08 10:37:10 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.