Alice Sherepa added a comment - 2023-05-11 17:29 Could you please provide the output of SHOW CREATE TABLE director.proc ;

Marko Mäkelä added a comment - 2023-05-12 05:36 This is basically a continuation of the infamous MDEV-9663 . Until MDEV-27734 in MariaDB Server 10.5.15 and 10.6.7, InnoDB change buffering was enabled by default. It has been the source of many corruption bugs, the MDEV-30009 "double buffering" bug being the latest that I am aware of. I think that it could explain this corruption and crash. If the affected index includes virtual columns (either explicitly or via MDEV-371 ), there could be other open bugs that would explain the corruption. The crash upon encountering this particular corruption has not been fixed in any version yet. MDEV-13542 in MariaDB Server 10.6 did remove many crashes on corruption, but not this one. It could make sense to go through each and every ut_a() in MariaDB Server 10.6.

Rob added a comment - 2023-05-12 06:50 Thanks for your review Here is the requested create table. As you can see it is a small table with few rows. The crash seemed to have occurred at a moment when SET FOREIGN_KEY_CHECKS=0 and the insert is performed after deleting the same records; The issue was also replicated so it affected all masters and slaves. The other masters and slaves end up in a loop of crashing, after each restart it attempts to do the query again, crashing the server. This was resolved by updating my.cnf not to start replication. There were about 40 tables restored like this and about 10 of them got corrupted (like Proc). This were all small tables with few rows. No such restore was attempted on large tables. Optimize table resolved the corruption after we downgrade to version 10.4.28. 24 hours later it is still working like sunshine on the 4.28. Looking at the release notes it is likely a replication failure. Given that we crashed 10 tables, probably on the master doing inserts it was working and the corruption occurred on the other master and slaves only. CREATE TABLE `proc` ( `PROCID` int(10) unsigned NOT NULL AUTO_INCREMENT, `GOALID` int(10) unsigned NOT NULL, `TEST` varchar(5) CHARACTER SET latin1 COLLATE latin1_swedish_ci NOT NULL, ..... PRIMARY KEY (`PROCID`) USING BTREE, KEY `new_fk_constraint` (`GOALID`), CONSTRAINT `new_fk_constraint` FOREIGN KEY (`GOALID`) REFERENCES `goal` (`GOALID`) ON DELETE CASCADE ON UPDATE CASCADE ) ENGINE=InnoDB AUTO_INCREMENT=611 DEFAULT CHARSET=utf8 COLLATE=utf8_general_ci ROW_FORMAT=COMPRESSED KEY_BLOCK_SIZE=4

Marko Mäkelä added a comment - 2023-05-12 07:38 Hrehora , have you executed ALTER TABLE…ADD INDEX or CREATE INDEX or added a foreign key constraint (which could implicitly create an index) on any of these tables in the past? If yes, this could be due to MDEV-30009 .

Rob added a comment - 2023-05-12 10:23 The addition or change of an index to the proc table is at least 10 years ago...

Marko Mäkelä added a comment - 2023-05-17 10:32 Without a repeatable test case, it is hard or impossible to say what caused the corruption in the first place. What I can do is to replace the crash with proper error propagation. This report duplicates MDEV-22739 .

Sergei Golubchik added a comment - 2023-06-06 11:51 This proper error propagation was implemented in 10.6.15 (planned release date — end of July). Could you try it, when it's out and comment here how it helped?

Rob added a comment - 2023-06-06 11:55 Oke I am going to check it when it comes out. Thanks

Elena Stepanova added a comment - 2023-09-10 18:53 Hrehora , 10.6.15 has been out for a while, did you have a chance to try it out?

Rob added a comment - 2023-09-13 10:22 Regretfully on v31 I continue to have the same problem. Rob

Rob added a comment - 2023-09-14 11:48 - edited What we did We were running 10.4.28 master- master on centos 7 We yum update to 10.4.31 When we perform a restore of a table we get the error below We downgrade to 10.4.28 We have to repair a set of innodb tables affected with optimize table the master master runs great again Let me know if I can help. 2023-09-13 07:42:14 0x7f4cae4c3700 InnoDB: Assertion failure in file /home/buildbot/buildbot/padding_for_CPACK_RPM_BUILD_SOURCE_DIRS_PREFIX/mariadb-10.4.31/storage/innobase/row/row0ins.cc line 219 InnoDB: Failing assertion: !cursor->index->is_committed() InnoDB: We intentionally generate a memory trap. InnoDB: Submit a detailed bug report to https://jira.mariadb.org/ InnoDB: If you get repeated assertion failures or crashes, even InnoDB: immediately after the mysqld startup, there may be InnoDB: corruption in the InnoDB tablespace. Please refer to InnoDB: https://mariadb.com/kb/en/library/innodb-recovery-modes/ InnoDB: about forcing recovery. 230913 7:42:14 [ERROR] mysqld got signal 6 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware.   To report this bug, see https://mariadb.com/kb/en/reporting-bugs   We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail.   Server version: 10.4.31-MariaDB-log source revision: 2aea9387497cecb5668ef605b8f80886f9de812c key_buffer_size=268435456 read_buffer_size=1048576 max_used_connections=264 max_threads=2002 thread_count=49 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 4412014 K bytes of memory Hope that's ok; if not, decrease some variables in the equation.   Thread pointer: 0x7f47580008c8 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0x7f4cae4c2cf0 thread_stack 0x49000 /usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x563c4684ce5e] /usr/sbin/mysqld(handle_fatal_signal+0x317)[0x563c4629aa07] sigaction.c:0(__restore_rt)[0x7f7fdd68e630] /lib64/libc.so.6(gsignal+0x37)[0x7f7fdcad9387] /lib64/libc.so.6(abort+0x148)[0x7f7fdcadaa78] /usr/sbin/mysqld(+0x5d6fb0)[0x563c45f98fb0] /usr/sbin/mysqld(+0xc69a35)[0x563c4662ba35] /usr/sbin/mysqld(+0xc6a0b3)[0x563c4662c0b3] /usr/sbin/mysqld(+0xc6a6a2)[0x563c4662c6a2] /usr/sbin/mysqld(+0xc7c610)[0x563c4663e610] /usr/sbin/mysqld(+0xbb78fd)[0x563c465798fd] /usr/sbin/mysqld(_ZN7handler12ha_write_rowEPKh+0x2af)[0x563c462a85ef] /usr/sbin/mysqld(_Z12write_recordP3THDP5TABLEP12st_copy_info+0x6a)[0x563c4606c53a] /usr/sbin/mysqld(_Z12mysql_insertP3THDP10TABLE_LISTR4ListI4ItemERS3_IS5_ES6_S6_15enum_duplicatesb+0x1284)[0x563c46073144] /usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x3e78)[0x563c460a2c58] /usr/sbin/mysqld(_Z11mysql_parseP3THDPcjP12Parser_statebb+0x39d)[0x563c460a838d] /usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcjbb+0x2113)[0x563c460ab403] /usr/sbin/mysqld(_Z10do_commandP3THD+0x112)[0x563c460acc42] /usr/sbin/mysqld(_Z24do_handle_one_connectionP7CONNECT+0x229)[0x563c46191ce9] /usr/sbin/mysqld(handle_one_connection+0x3d)[0x563c46191dad] pthread_create.c:0(start_thread)[0x7f7fdd686ea5] /lib64/libc.so.6(clone+0x6d)[0x7f7fdcba1b0d]   Trying to get some variables. Some pointers may be invalid and cause the dump to abort. Query (0x7f475801e1b0): insert into log.templateacttp (TEMPLATEID,ACTTYPEID) VALUES (8176,23863),(8176,23864),(8176,23868),(8176,23876),(8177,23851),(8177,23863),(8177,23864),(8177,23868),(8177,23876),(8178,23851),(8178,23863),(8178,23864),(8178,23868),(8178,23876),(8179,23851),(8179,23863),(8179,23864),(8179,23868),(8179,23876),(8180,23851)   Connection ID (thread ID): 3691832 Status: NOT_KILLED   Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=on,condition_pushdown_from_having=on   The manual page at https://mariadb.com/kb/en/how-to-produce-a-full-stack-trace-for-mysqld/ contains information that should help you find out what is causing the crash. Writing a core file... Working directory at /data/mysql Resource Limits: Limit Soft Limit Hard Limit Units Max cpu time unlimited unlimited seconds Max file size unlimited unlimited bytes Max data size unlimited unlimited bytes Max stack size 8388608 unlimited bytes Max core file size 0 unlimited bytes Max resident set unlimited unlimited bytes Max processes 1029880 1029880 processes Max open files 32768 32768 files Max locked memory 65536 65536 bytes Max address space unlimited unlimited bytes Max file locks unlimited unlimited locks Max pending signals 1029880 1029880 signals Max msgqueue size 819200 819200 bytes Max nice priority 0 0 Max realtime priority 0 0 Max realtime timeout unlimited unlimited us Kernel version: Linux version 3.10.0-1160.83.1.el7.x86_64 (mockbuild@kbuilder.bsys.centos.org) (gcc version 4.8.5 20150623 (Red Hat 4.8.5-44) (GCC) ) #1 SMP Wed Jan 25 16:41:43 UTC 2023

Marko Mäkelä added a comment - 2023-09-18 07:50 Hrehora , by design, InnoDB used to deal with corruption by intentionally crashing itself. This was fixed in MariaDB Server 10.6 by MDEV-13542 and some follow-up fixes, such as MDEV-22739 , where the assertion that fails for you in MariaDB Server 10.4 was removed. A possible cause of the corruption is MDEV-32132 . It is not feasible to port these fixes to earlier major versions than 10.6. Can you please try an upgrade to MariaDB Server 10.6.15?