Uploaded image for project: 'MariaDB Server'
  1. MariaDB Server
  2. MDEV-22986

[ERROR] InnoDB: Record in index was not found on update in mysqld.log on slave

Details

    Description

      [ERROR] InnoDB: Record in index `item_id` of table `item3`.`deleted_items` was not found on update: TUPLE (info_bits=0, 2 fields):

      {[4] x (0x819678F8),[4] ;J (0x003B4AD8)}

      at: COMPACT RECORD(info_bits=32, 2 fields):

      {[4] x (0x819678F8),[4] ;( (0x003B28F9)}

      Attachments

        Issue Links

          Activity

            rulotec Ruben Estrada Orozco added a comment - - edited

            I think I have a similar error and I already have innodb_change_buffering=none:

            2022-07-27  9:34:38 215 [ERROR] InnoDB: Record in index `vtiger_crmentity_labelidx` of table `vt_produccion`.`vtiger_crmentity` was not found on update: TUPLE (info_bits=0, 2 fields): {[30]Sandra Sandy Mu  oz Fe
            rn  ndez(0x53616E6472612053616E6479204D75C3B16F7A204665726EC3A16E64657A),[4] =  (0x803DD517)} at: COMPACT RECORD(info_bits=0, 2 fields): {[23]Sandra Saavedra Ch  vez(0x53616E647261205361617665647261204368C3A1766
            57A),[4] BU (0x804255E8)}
            2022-07-27 10:09:48 382 [ERROR] InnoDB: Corruption of an index tree: table `vt_produccion`.`vtiger_crmentity` index `vtiger_crmentity_labelidx`, father ptr page no 70022, child page no 273970
            PHYSICAL RECORD: n_fields 3; compact format; info bits 0
             0: len 30; hex 59612064656369646520706f72206f74726120756e697665727369646164; asc Ya decide por otra universidad; (total 90 bytes);
             1: len 4; hex 8005fd4d; asc    M;;
             2: len 4; hex 00044186; asc   A ;;
            2022-07-27 10:09:48 382 [Note] InnoDB: n_owned: 0; heap_no: 2; next rec: 233
            PHYSICAL RECORD: n_fields 3; compact format; info bits 0
             0: len 19; hex 554e49564552534944414420c38d5441434120; asc UNIVERSIDAD   TACA ;;
             1: len 4; hex 803fdbf4; asc  ?  ;;
             2: len 4; hex 00011186; asc     ;;
            2022-07-27 10:09:48 382 [Note] InnoDB: n_owned: 5; heap_no: 54; next rec: 13420
            2022-07-27 10:09:48 382 [ERROR] [FATAL] InnoDB: You should dump + drop + reimport the table to fix the corruption. If the crash happens at database startup. Please refer to https://mariadb.com/kb/en/library/innodb-recovery-modes/ for information about forcing recovery. Then dump + drop + reimport.
            220727 10:09:48 [ERROR] mysqld got signal 6 ;
            This could be because you hit a bug. It is also possible that this binary
            or one of the libraries it was linked against is corrupt, improperly built,
            or misconfigured. This error can also be caused by malfunctioning hardware.
             
            To report this bug, see https://mariadb.com/kb/en/reporting-bugs
             
            We will try our best to scrape up some info that will hopefully help
            diagnose the problem, but since we have already crashed, 
            something is definitely wrong and this may fail.
             
            Server version: 10.6.8-MariaDB
            key_buffer_size=134217728
            read_buffer_size=131072
            max_used_connections=61
            max_threads=153
            thread_count=48
            It is possible that mysqld could use up to 
            key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 467959 K  bytes of memory
            Hope that's ok; if not, decrease some variables in the equation.
             
            Thread pointer: 0x7f42e80008d8
            Attempting backtrace. You can use the following information to find out
            where mysqld died. If you see no messages after this, something went
            terribly wrong...
            stack_bottom = 0x7f43dc058cc0 thread_stack 0x49000
            Can't start addr2line
            /usr/sbin/mariadbd(my_print_stacktrace+0x2e)[0x556c739b2a7e]
            /usr/sbin/mariadbd(handle_fatal_signal+0x307)[0x556c733fa6f7]
            /lib64/libpthread.so.0(+0xf630)[0x7f46837bd630]
            /lib64/libc.so.6(gsignal+0x37)[0x7f4682c08387]
            /lib64/libc.so.6(abort+0x148)[0x7f4682c09a78]
            /usr/sbin/mariadbd(+0xe02e10)[0x556c7382ee10]
            /usr/sbin/mariadbd(+0xe08261)[0x556c73834261]
            /usr/sbin/mariadbd(+0xe0e415)[0x556c7383a415]
            /usr/sbin/mariadbd(+0xe21725)[0x556c7384d725]
            /usr/sbin/mariadbd(+0xe2cac6)[0x556c73858ac6]
            /usr/sbin/mariadbd(+0xe0ea40)[0x556c7383aa40]
            /usr/sbin/mariadbd(+0xe21725)[0x556c7384d725]
            /usr/sbin/mariadbd(+0xe2cac6)[0x556c73858ac6]
            /usr/sbin/mariadbd(+0xf01969)[0x556c7392d969]
            /usr/sbin/mariadbd(+0xf050ac)[0x556c739310ac]
            /usr/sbin/mariadbd(+0xdad6bb)[0x556c737d96bb]
            /usr/sbin/mariadbd(+0xd5a494)[0x556c73786494]
            /usr/sbin/mariadbd(+0xddebe2)[0x556c7380abe2]
            /usr/sbin/mariadbd(+0xcb0c9c)[0x556c736dcc9c]
            /usr/sbin/mariadbd(_Z17ha_rollback_transP3THDb+0x14f)[0x556c733fe4ef]
            /usr/sbin/mariadbd(_Z14trans_rollbackP3THD+0x36)[0x556c732e2de6]
            /usr/sbin/mariadbd(_Z21mysql_execute_commandP3THDb+0x25a3)[0x556c731d7363]
            /usr/sbin/mariadbd(_Z11mysql_parseP3THDPcjP12Parser_state+0x20b)[0x556c731da5bb]
            /usr/sbin/mariadbd(_Z16dispatch_command19enum_server_commandP3THDPcjb+0x1217)[0x556c731dc777]
            /usr/sbin/mariadbd(_Z10do_commandP3THDb+0x123)[0x556c731dde13]
            /usr/sbin/mariadbd(_Z24do_handle_one_connectionP7CONNECTb+0x187)[0x556c732d39f7]
            /usr/sbin/mariadbd(handle_one_connection+0x34)[0x556c732d3c94]
            /usr/sbin/mariadbd(+0xc1270c)[0x556c7363e70c]
            /lib64/libpthread.so.0(+0x7ea5)[0x7f46837b5ea5]
            /lib64/libc.so.6(clone+0x6d)[0x7f4682cd0b0d]
             
            Trying to get some variables.
            Some pointers may be invalid and cause the dump to abort.
            Query (0x7f42e80106e0): ROLLBACK
             
            Connection ID (thread ID): 382
            Status: NOT_KILLED
             
            Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=on,condition_pushdown_from_having=on,not_null_range_scan=off
             
            The manual page at https://mariadb.com/kb/en/how-to-produce-a-full-stack-trace-for-mysqld/ contains
            information that should help you find out what is causing the crash.
            Writing a core file...
            Working directory at /var/lib/mysql
            Resource Limits:
            Limit                     Soft Limit           Hard Limit           Units     
            Max cpu time              unlimited            unlimited            seconds   
            Max file size             unlimited            unlimited            bytes     
            Max data size             unlimited            unlimited            bytes     
            Max stack size            8388608              unlimited            bytes     
            Max core file size        0                    unlimited            bytes     
            Max resident set          unlimited            unlimited            bytes     
            Max processes             63343                63343                processes 
            Max open files            32768                32768                files     
            Max locked memory         65536                65536                bytes     
            Max address space         unlimited            unlimited            bytes     
            Max file locks            unlimited            unlimited            locks     
            Max pending signals       63343                63343                signals   
            Max msgqueue size         819200               819200               bytes     
            Max nice priority         0                    0                    
            Max realtime priority     0                    0                    
            Max realtime timeout      unlimited            unlimited            us 
            Core pattern: core
            

            could you help me out with this?

            rulotec Ruben Estrada Orozco added a comment - - edited I think I have a similar error and I already have innodb_change_buffering=none: 2022-07-27 9:34:38 215 [ERROR] InnoDB: Record in index `vtiger_crmentity_labelidx` of table `vt_produccion`.`vtiger_crmentity` was not found on update: TUPLE (info_bits=0, 2 fields): {[30]Sandra Sandy Mu oz Fe rn ndez(0x53616E6472612053616E6479204D75C3B16F7A204665726EC3A16E64657A),[4] = (0x803DD517)} at: COMPACT RECORD(info_bits=0, 2 fields): {[23]Sandra Saavedra Ch vez(0x53616E647261205361617665647261204368C3A1766 57A),[4] BU (0x804255E8)} 2022-07-27 10:09:48 382 [ERROR] InnoDB: Corruption of an index tree: table `vt_produccion`.`vtiger_crmentity` index `vtiger_crmentity_labelidx`, father ptr page no 70022, child page no 273970 PHYSICAL RECORD: n_fields 3; compact format; info bits 0 0: len 30; hex 59612064656369646520706f72206f74726120756e697665727369646164; asc Ya decide por otra universidad; (total 90 bytes); 1: len 4; hex 8005fd4d; asc M;; 2: len 4; hex 00044186; asc A ;; 2022-07-27 10:09:48 382 [Note] InnoDB: n_owned: 0; heap_no: 2; next rec: 233 PHYSICAL RECORD: n_fields 3; compact format; info bits 0 0: len 19; hex 554e49564552534944414420c38d5441434120; asc UNIVERSIDAD TACA ;; 1: len 4; hex 803fdbf4; asc ? ;; 2: len 4; hex 00011186; asc ;; 2022-07-27 10:09:48 382 [Note] InnoDB: n_owned: 5; heap_no: 54; next rec: 13420 2022-07-27 10:09:48 382 [ERROR] [FATAL] InnoDB: You should dump + drop + reimport the table to fix the corruption. If the crash happens at database startup. Please refer to https://mariadb.com/kb/en/library/innodb-recovery-modes/ for information about forcing recovery. Then dump + drop + reimport. 220727 10:09:48 [ERROR] mysqld got signal 6 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware.   To report this bug, see https://mariadb.com/kb/en/reporting-bugs   We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail.   Server version: 10.6.8-MariaDB key_buffer_size=134217728 read_buffer_size=131072 max_used_connections=61 max_threads=153 thread_count=48 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 467959 K bytes of memory Hope that's ok; if not, decrease some variables in the equation.   Thread pointer: 0x7f42e80008d8 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0x7f43dc058cc0 thread_stack 0x49000 Can't start addr2line /usr/sbin/mariadbd(my_print_stacktrace+0x2e)[0x556c739b2a7e] /usr/sbin/mariadbd(handle_fatal_signal+0x307)[0x556c733fa6f7] /lib64/libpthread.so.0(+0xf630)[0x7f46837bd630] /lib64/libc.so.6(gsignal+0x37)[0x7f4682c08387] /lib64/libc.so.6(abort+0x148)[0x7f4682c09a78] /usr/sbin/mariadbd(+0xe02e10)[0x556c7382ee10] /usr/sbin/mariadbd(+0xe08261)[0x556c73834261] /usr/sbin/mariadbd(+0xe0e415)[0x556c7383a415] /usr/sbin/mariadbd(+0xe21725)[0x556c7384d725] /usr/sbin/mariadbd(+0xe2cac6)[0x556c73858ac6] /usr/sbin/mariadbd(+0xe0ea40)[0x556c7383aa40] /usr/sbin/mariadbd(+0xe21725)[0x556c7384d725] /usr/sbin/mariadbd(+0xe2cac6)[0x556c73858ac6] /usr/sbin/mariadbd(+0xf01969)[0x556c7392d969] /usr/sbin/mariadbd(+0xf050ac)[0x556c739310ac] /usr/sbin/mariadbd(+0xdad6bb)[0x556c737d96bb] /usr/sbin/mariadbd(+0xd5a494)[0x556c73786494] /usr/sbin/mariadbd(+0xddebe2)[0x556c7380abe2] /usr/sbin/mariadbd(+0xcb0c9c)[0x556c736dcc9c] /usr/sbin/mariadbd(_Z17ha_rollback_transP3THDb+0x14f)[0x556c733fe4ef] /usr/sbin/mariadbd(_Z14trans_rollbackP3THD+0x36)[0x556c732e2de6] /usr/sbin/mariadbd(_Z21mysql_execute_commandP3THDb+0x25a3)[0x556c731d7363] /usr/sbin/mariadbd(_Z11mysql_parseP3THDPcjP12Parser_state+0x20b)[0x556c731da5bb] /usr/sbin/mariadbd(_Z16dispatch_command19enum_server_commandP3THDPcjb+0x1217)[0x556c731dc777] /usr/sbin/mariadbd(_Z10do_commandP3THDb+0x123)[0x556c731dde13] /usr/sbin/mariadbd(_Z24do_handle_one_connectionP7CONNECTb+0x187)[0x556c732d39f7] /usr/sbin/mariadbd(handle_one_connection+0x34)[0x556c732d3c94] /usr/sbin/mariadbd(+0xc1270c)[0x556c7363e70c] /lib64/libpthread.so.0(+0x7ea5)[0x7f46837b5ea5] /lib64/libc.so.6(clone+0x6d)[0x7f4682cd0b0d]   Trying to get some variables. Some pointers may be invalid and cause the dump to abort. Query (0x7f42e80106e0): ROLLBACK   Connection ID (thread ID): 382 Status: NOT_KILLED   Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=on,condition_pushdown_from_having=on,not_null_range_scan=off   The manual page at https://mariadb.com/kb/en/how-to-produce-a-full-stack-trace-for-mysqld/ contains information that should help you find out what is causing the crash. Writing a core file... Working directory at /var/lib/mysql Resource Limits: Limit Soft Limit Hard Limit Units Max cpu time unlimited unlimited seconds Max file size unlimited unlimited bytes Max data size unlimited unlimited bytes Max stack size 8388608 unlimited bytes Max core file size 0 unlimited bytes Max resident set unlimited unlimited bytes Max processes 63343 63343 processes Max open files 32768 32768 files Max locked memory 65536 65536 bytes Max address space unlimited unlimited bytes Max file locks unlimited unlimited locks Max pending signals 63343 63343 signals Max msgqueue size 819200 819200 bytes Max nice priority 0 0 Max realtime priority 0 0 Max realtime timeout unlimited unlimited us Core pattern: core could you help me out with this?

            rulotec, we can only help with this bug if someone produces a way of reproducing the error on a database that has been initialized, loading the data with SQL statements only (no copying of data files).

            If the affected table contains indexed virtual columns (MDEV-5800) or unique keys on long columns (MDEV-371), there may already exist a separate open bug that explains the corruption.

            marko Marko Mäkelä added a comment - rulotec , we can only help with this bug if someone produces a way of reproducing the error on a database that has been initialized, loading the data with SQL statements only (no copying of data files). If the affected table contains indexed virtual columns ( MDEV-5800 ) or unique keys on long columns ( MDEV-371 ), there may already exist a separate open bug that explains the corruption.

            MDEV-30009 is another bug fix in this area.

            marko Marko Mäkelä added a comment - MDEV-30009 is another bug fix in this area.

            InnoDB change buffer (a possible culprit) was removed in MDEV-29694. Let's close this issue.

            If anyone would still experience this bug, please add a comment here and we'll reopen the issue if needed.

            serg Sergei Golubchik added a comment - InnoDB change buffer (a possible culprit) was removed in MDEV-29694 . Let's close this issue. If anyone would still experience this bug, please add a comment here and we'll reopen the issue if needed.
            dcz01 Daniel Czadek added a comment -

            Hi,
            I got the exact same problem in a galera cluster with mariadb 10.5.21 with galera-4.
            Its only happening on server2 and not server1 and when i stopped the instance on server2 (because of infinity tries of restarts always with the same error) and full SST is done but error persists only on node server2.

            Is that normal?

            2024-07-03 11:17:00 2 [ERROR] InnoDB: Record in index `xxx` of table `xxx`.`media_contents_varchar` was not found on update: TUPLE (info_bits=0, 2 fields): {NULL,[8]      /f(0x0000000000162F66)} at: COMPACT RECORD(info_bits=0, 2 fields): {NULL,[8]      /d(0x0000000000162F64)}
            2024-07-03 11:17:00 0x7f72f81e3700  InnoDB: Assertion failure in file ./storage/innobase/row/row0ins.cc line 219
            InnoDB: Failing assertion: !cursor->index->is_committed()
            InnoDB: We intentionally generate a memory trap.
            InnoDB: Submit a detailed bug report to https://jira.mariadb.org/
            InnoDB: If you get repeated assertion failures or crashes, even
            InnoDB: immediately after the mariadbd startup, there may be
            InnoDB: corruption in the InnoDB tablespace. Please refer to
            InnoDB: https://mariadb.com/kb/en/library/innodb-recovery-modes/
            InnoDB: about forcing recovery.
            240703 11:17:00 [ERROR] mysqld got signal 6 ;
            This could be because you hit a bug. It is also possible that this binary
            or one of the libraries it was linked against is corrupt, improperly built,
            or misconfigured. This error can also be caused by malfunctioning hardware.
             
            To report this bug, see https://mariadb.com/kb/en/reporting-bugs
             
            We will try our best to scrape up some info that will hopefully help
            diagnose the problem, but since we have already crashed,
            something is definitely wrong and this may fail.
             
            Server version: 10.5.21-MariaDB-0+deb11u1 source revision: bed70468ea08c2820647f5e3ac006a9ff88144ac
            key_buffer_size=268435456
            read_buffer_size=131072
            max_used_connections=0
            max_threads=1252
            thread_count=2
            It is possible that mysqld could use up to
            key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 3018445 K  bytes of memory
            Hope that's ok; if not, decrease some variables in the equation.
             
            Thread pointer: 0x7f72d8000c58
            Attempting backtrace. You can use the following information to find out
            where mysqld died. If you see no messages after this, something went
            terribly wrong...
            stack_bottom = 0x7f72f81e2d98 thread_stack 0x49000
            /usr/sbin/mariadbd(my_print_stacktrace+0x2e)[0x55c8e11c2e2e]
            /usr/sbin/mariadbd(handle_fatal_signal+0x485)[0x55c8e0cbb0a5]
            /lib/x86_64-linux-gnu/libpthread.so.0(+0x13140)[0x7f73027d5140]
            /lib/x86_64-linux-gnu/libc.so.6(gsignal+0x141)[0x7f730230dce1]
            /lib/x86_64-linux-gnu/libc.so.6(abort+0x123)[0x7f73022f7537]
            /usr/sbin/mariadbd(+0x641e98)[0x55c8e0993e98]
            /usr/sbin/mariadbd(+0x630ff1)[0x55c8e0982ff1]
            /usr/sbin/mariadbd(+0xce7ecc)[0x55c8e1039ecc]
            /usr/sbin/mariadbd(+0xd19a92)[0x55c8e106ba92]
            /usr/sbin/mariadbd(+0xd1a18f)[0x55c8e106c18f]
            /usr/sbin/mariadbd(+0xcf84cb)[0x55c8e104a4cb]
            /usr/sbin/mariadbd(+0xc51e30)[0x55c8e0fa3e30]
            /usr/sbin/mariadbd(_ZN7handler13ha_update_rowEPKhS1_+0x2a2)[0x55c8e0cc9302]
            /usr/sbin/mariadbd(_ZN21Update_rows_log_event11do_exec_rowEP14rpl_group_info+0x241)[0x55c8e0ddef31]
            /usr/sbin/mariadbd(_ZN14Rows_log_event14do_apply_eventEP14rpl_group_info+0x33f)[0x55c8e0dd2b5f]
            /usr/sbin/mariadbd(_Z18wsrep_apply_eventsP3THDP14Relay_log_infoPKvm+0x1e9)[0x55c8e0f79dc9]
            /usr/sbin/mariadbd(+0xc0f5d0)[0x55c8e0f615d0]
            /usr/sbin/mariadbd(_ZN21Wsrep_applier_service15apply_write_setERKN5wsrep7ws_metaERKNS0_12const_bufferERNS0_14mutable_bufferE+0xa6)[0x55c8e0f623c6]
            /usr/sbin/mariadbd(+0xee86d7)[0x55c8e123a6d7]
            /usr/sbin/mariadbd(+0xef978e)[0x55c8e124b78e]
            /usr/lib/galera/libgalera_smm.so(+0x555a1)[0x7f72f9bea5a1]
            /usr/lib/galera/libgalera_smm.so(+0x6391f)[0x7f72f9bf891f]
            /usr/lib/galera/libgalera_smm.so(+0x7737e)[0x7f72f9c0c37e]
            /usr/lib/galera/libgalera_smm.so(+0x77ae1)[0x7f72f9c0cae1]
            /usr/lib/galera/libgalera_smm.so(+0x7cf91)[0x7f72f9c11f91]
            /usr/lib/galera/libgalera_smm.so(+0x69745)[0x7f72f9bfe745]
            /usr/lib/galera/libgalera_smm.so(+0x6a062)[0x7f72f9bff062]
            /usr/lib/galera/libgalera_smm.so(+0x6a5ef)[0x7f72f9bff5ef]
            /usr/lib/galera/libgalera_smm.so(+0x93fc2)[0x7f72f9c28fc2]
            /usr/lib/galera/libgalera_smm.so(+0x68c28)[0x7f72f9bfdc28]
            /usr/lib/galera/libgalera_smm.so(+0x43fed)[0x7f72f9bd8fed]
            /usr/sbin/mariadbd(_ZN5wsrep18wsrep_provider_v2611run_applierEPNS_21high_priority_serviceE+0xe)[0x55c8e124bdfe]
            /usr/sbin/mariadbd(+0xc29f8d)[0x55c8e0f7bf8d]
            /usr/sbin/mariadbd(_Z15start_wsrep_THDPv+0x26b)[0x55c8e0f6c59b]
            /usr/sbin/mariadbd(+0xbaa77b)[0x55c8e0efc77b]
            /lib/x86_64-linux-gnu/libpthread.so.0(+0x7ea7)[0x7f73027c9ea7]
            /lib/x86_64-linux-gnu/libc.so.6(clone+0x3f)[0x7f73023d0a2f]
             
            Trying to get some variables.
            Some pointers may be invalid and cause the dump to abort.
            Query (0x7f72ecba1743): UPDATE media_contents_varchar SET media_id=13146, tag_id=0, tag_internal_id=14, media_content_value='15-511031', media_content_dtupdate='2024-07-03 11:08:02' WHERE media_content_id=1453926
             
            Connection ID (thread ID): 2
            Status: NOT_KILLED
             
            Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=on,condition_pushdown_from_having=on,not_null_range_scan=off
             
            The manual page at https://mariadb.com/kb/en/how-to-produce-a-full-stack-trace-for-mysqld/ contains
            information that should help you find out what is causing the crash.
            Writing a core file...
            Working directory at /data/server2
            Resource Limits:
            Limit                     Soft Limit           Hard Limit           Units
            Max cpu time              unlimited            unlimited            seconds
            Max file size             unlimited            unlimited            bytes
            Max data size             unlimited            unlimited            bytes
            Max stack size            8388608              unlimited            bytes
            Max core file size        0                    unlimited            bytes
            Max resident set          unlimited            unlimited            bytes
            Max processes             128112               128112               processes
            Max open files            1048576              1048576              files
            Max locked memory         65536                65536                bytes
            Max address space         unlimited            unlimited            bytes
            Max file locks            unlimited            unlimited            locks
            Max pending signals       128112               128112               signals
            Max msgqueue size         819200               819200               bytes
            Max nice priority         0                    0
            Max realtime priority     0                    0
            Max realtime timeout      unlimited            unlimited            us
            Core pattern: core
             
            Kernel version: Linux version 5.10.0-26-amd64 (debian-kernel@lists.debian.org) (gcc-10 (Debian 10.2.1-6) 10.2.1 20210110, GNU ld (GNU Binutils for Debian) 2.35.2) #1 SMP Debian 5.10.197-1 (2023-09-29)
            

            dcz01 Daniel Czadek added a comment - Hi, I got the exact same problem in a galera cluster with mariadb 10.5.21 with galera-4. Its only happening on server2 and not server1 and when i stopped the instance on server2 (because of infinity tries of restarts always with the same error) and full SST is done but error persists only on node server2. Is that normal? 2024-07-03 11:17:00 2 [ERROR] InnoDB: Record in index `xxx` of table `xxx`.`media_contents_varchar` was not found on update: TUPLE (info_bits=0, 2 fields): {NULL,[8] /f(0x0000000000162F66)} at: COMPACT RECORD(info_bits=0, 2 fields): {NULL,[8] /d(0x0000000000162F64)} 2024-07-03 11:17:00 0x7f72f81e3700 InnoDB: Assertion failure in file ./storage/innobase/row/row0ins.cc line 219 InnoDB: Failing assertion: !cursor->index->is_committed() InnoDB: We intentionally generate a memory trap. InnoDB: Submit a detailed bug report to https://jira.mariadb.org/ InnoDB: If you get repeated assertion failures or crashes, even InnoDB: immediately after the mariadbd startup, there may be InnoDB: corruption in the InnoDB tablespace. Please refer to InnoDB: https://mariadb.com/kb/en/library/innodb-recovery-modes/ InnoDB: about forcing recovery. 240703 11:17:00 [ERROR] mysqld got signal 6 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware.   To report this bug, see https://mariadb.com/kb/en/reporting-bugs   We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail.   Server version: 10.5.21-MariaDB-0+deb11u1 source revision: bed70468ea08c2820647f5e3ac006a9ff88144ac key_buffer_size=268435456 read_buffer_size=131072 max_used_connections=0 max_threads=1252 thread_count=2 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 3018445 K bytes of memory Hope that's ok; if not, decrease some variables in the equation.   Thread pointer: 0x7f72d8000c58 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0x7f72f81e2d98 thread_stack 0x49000 /usr/sbin/mariadbd(my_print_stacktrace+0x2e)[0x55c8e11c2e2e] /usr/sbin/mariadbd(handle_fatal_signal+0x485)[0x55c8e0cbb0a5] /lib/x86_64-linux-gnu/libpthread.so.0(+0x13140)[0x7f73027d5140] /lib/x86_64-linux-gnu/libc.so.6(gsignal+0x141)[0x7f730230dce1] /lib/x86_64-linux-gnu/libc.so.6(abort+0x123)[0x7f73022f7537] /usr/sbin/mariadbd(+0x641e98)[0x55c8e0993e98] /usr/sbin/mariadbd(+0x630ff1)[0x55c8e0982ff1] /usr/sbin/mariadbd(+0xce7ecc)[0x55c8e1039ecc] /usr/sbin/mariadbd(+0xd19a92)[0x55c8e106ba92] /usr/sbin/mariadbd(+0xd1a18f)[0x55c8e106c18f] /usr/sbin/mariadbd(+0xcf84cb)[0x55c8e104a4cb] /usr/sbin/mariadbd(+0xc51e30)[0x55c8e0fa3e30] /usr/sbin/mariadbd(_ZN7handler13ha_update_rowEPKhS1_+0x2a2)[0x55c8e0cc9302] /usr/sbin/mariadbd(_ZN21Update_rows_log_event11do_exec_rowEP14rpl_group_info+0x241)[0x55c8e0ddef31] /usr/sbin/mariadbd(_ZN14Rows_log_event14do_apply_eventEP14rpl_group_info+0x33f)[0x55c8e0dd2b5f] /usr/sbin/mariadbd(_Z18wsrep_apply_eventsP3THDP14Relay_log_infoPKvm+0x1e9)[0x55c8e0f79dc9] /usr/sbin/mariadbd(+0xc0f5d0)[0x55c8e0f615d0] /usr/sbin/mariadbd(_ZN21Wsrep_applier_service15apply_write_setERKN5wsrep7ws_metaERKNS0_12const_bufferERNS0_14mutable_bufferE+0xa6)[0x55c8e0f623c6] /usr/sbin/mariadbd(+0xee86d7)[0x55c8e123a6d7] /usr/sbin/mariadbd(+0xef978e)[0x55c8e124b78e] /usr/lib/galera/libgalera_smm.so(+0x555a1)[0x7f72f9bea5a1] /usr/lib/galera/libgalera_smm.so(+0x6391f)[0x7f72f9bf891f] /usr/lib/galera/libgalera_smm.so(+0x7737e)[0x7f72f9c0c37e] /usr/lib/galera/libgalera_smm.so(+0x77ae1)[0x7f72f9c0cae1] /usr/lib/galera/libgalera_smm.so(+0x7cf91)[0x7f72f9c11f91] /usr/lib/galera/libgalera_smm.so(+0x69745)[0x7f72f9bfe745] /usr/lib/galera/libgalera_smm.so(+0x6a062)[0x7f72f9bff062] /usr/lib/galera/libgalera_smm.so(+0x6a5ef)[0x7f72f9bff5ef] /usr/lib/galera/libgalera_smm.so(+0x93fc2)[0x7f72f9c28fc2] /usr/lib/galera/libgalera_smm.so(+0x68c28)[0x7f72f9bfdc28] /usr/lib/galera/libgalera_smm.so(+0x43fed)[0x7f72f9bd8fed] /usr/sbin/mariadbd(_ZN5wsrep18wsrep_provider_v2611run_applierEPNS_21high_priority_serviceE+0xe)[0x55c8e124bdfe] /usr/sbin/mariadbd(+0xc29f8d)[0x55c8e0f7bf8d] /usr/sbin/mariadbd(_Z15start_wsrep_THDPv+0x26b)[0x55c8e0f6c59b] /usr/sbin/mariadbd(+0xbaa77b)[0x55c8e0efc77b] /lib/x86_64-linux-gnu/libpthread.so.0(+0x7ea7)[0x7f73027c9ea7] /lib/x86_64-linux-gnu/libc.so.6(clone+0x3f)[0x7f73023d0a2f]   Trying to get some variables. Some pointers may be invalid and cause the dump to abort. Query (0x7f72ecba1743): UPDATE media_contents_varchar SET media_id=13146, tag_id=0, tag_internal_id=14, media_content_value='15-511031', media_content_dtupdate='2024-07-03 11:08:02' WHERE media_content_id=1453926   Connection ID (thread ID): 2 Status: NOT_KILLED   Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=on,condition_pushdown_from_having=on,not_null_range_scan=off   The manual page at https://mariadb.com/kb/en/how-to-produce-a-full-stack-trace-for-mysqld/ contains information that should help you find out what is causing the crash. Writing a core file... Working directory at /data/server2 Resource Limits: Limit Soft Limit Hard Limit Units Max cpu time unlimited unlimited seconds Max file size unlimited unlimited bytes Max data size unlimited unlimited bytes Max stack size 8388608 unlimited bytes Max core file size 0 unlimited bytes Max resident set unlimited unlimited bytes Max processes 128112 128112 processes Max open files 1048576 1048576 files Max locked memory 65536 65536 bytes Max address space unlimited unlimited bytes Max file locks unlimited unlimited locks Max pending signals 128112 128112 signals Max msgqueue size 819200 819200 bytes Max nice priority 0 0 Max realtime priority 0 0 Max realtime timeout unlimited unlimited us Core pattern: core   Kernel version: Linux version 5.10.0-26-amd64 (debian-kernel@lists.debian.org) (gcc-10 (Debian 10.2.1-6) 10.2.1 20210110, GNU ld (GNU Binutils for Debian) 2.35.2) #1 SMP Debian 5.10.197-1 (2023-09-29)

            People

              marko Marko Mäkelä
              Moroz Vova Moroz
              Votes:
              3 Vote for this issue
              Watchers:
              12 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Git Integration

                  Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.