Details
-
Bug
-
Status: Open (View Workflow)
-
Critical
-
Resolution: Unresolved
-
10.5.23
-
Debian 11
Description
Hi,
We got now this error since last week very often and an fresh restart of the server with an SST from server1 doesn't work.
The server2 still keeps crashing and it's just only on server2, server1 isn't crashing.
2024-07-11 14:11:40 2 [ERROR] InnoDB: Record in index `IX_altref` of table `xxx`.`media_contents_varchar` was not found on update: TUPLE (info_bits=0, 2 fields): {NULL,[8] &(0x0000000000151A26)} at: COMPACT RECORD(info
|
_bits=0, 2 fields): {NULL,[8] $(0x0000000000151A24)}
|
2024-07-11 14:11:40 0x7f1d2c68d700 InnoDB: Assertion failure in file ./storage/innobase/row/row0ins.cc line 219
|
InnoDB: Failing assertion: !cursor->index->is_committed()
|
InnoDB: We intentionally generate a memory trap.
|
InnoDB: Submit a detailed bug report to https://jira.mariadb.org/
|
InnoDB: If you get repeated assertion failures or crashes, even
|
InnoDB: immediately after the mariadbd startup, there may be
|
InnoDB: corruption in the InnoDB tablespace. Please refer to
|
InnoDB: https://mariadb.com/kb/en/library/innodb-recovery-modes/
|
InnoDB: about forcing recovery.
|
240711 14:11:40 [ERROR] mysqld got signal 6 ;
|
Sorry, we probably made a mistake, and this is a bug.
|
|
|
Your assistance in bug reporting will enable us to fix this for the next release.
|
To report this bug, see https://mariadb.com/kb/en/reporting-bugs
|
|
|
We will try our best to scrape up some info that will hopefully help
|
diagnose the problem, but since we have already crashed,
|
something is definitely wrong and this may fail.
|
|
|
Server version: 10.5.23-MariaDB-0+deb11u1 source revision: 6cfd2ba397b0ca689d8ff1bdb9fc4a4dc516a5eb
|
key_buffer_size=268435456
|
read_buffer_size=131072
|
max_used_connections=8
|
max_threads=1252
|
thread_count=10
|
It is possible that mysqld could use up to
|
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 3018523 K bytes of memory
|
Hope that's ok; if not, decrease some variables in the equation.
|
|
|
Thread pointer: 0x7f1d0c000c58
|
Attempting backtrace. You can use the following information to find out
|
where mysqld died. If you see no messages after this, something went
|
terribly wrong...
|
stack_bottom = 0x7f1d2c68cd98 thread_stack 0x49000
|
/usr/sbin/mariadbd(my_print_stacktrace+0x2e)[0x56219b91a14e]
|
/usr/sbin/mariadbd(handle_fatal_signal+0x475)[0x56219b40f3b5]
|
/lib/x86_64-linux-gnu/libpthread.so.0(+0x13140)[0x7f1d3848d140]
|
/lib/x86_64-linux-gnu/libc.so.6(gsignal+0x141)[0x7f1d37fc5ce1]
|
/lib/x86_64-linux-gnu/libc.so.6(abort+0x123)[0x7f1d37faf537]
|
/usr/sbin/mariadbd(+0x64329a)[0x56219b0e029a]
|
/usr/sbin/mariadbd(+0x6323d9)[0x56219b0cf3d9]
|
/usr/sbin/mariadbd(+0xcf381c)[0x56219b79081c]
|
/usr/sbin/mariadbd(+0xd25382)[0x56219b7c2382]
|
/usr/sbin/mariadbd(+0xd25a7f)[0x56219b7c2a7f]
|
/usr/sbin/mariadbd(+0xd03dcb)[0x56219b7a0dcb]
|
/usr/sbin/mariadbd(+0xc5cc8e)[0x56219b6f9c8e]
|
/usr/sbin/mariadbd(_ZN7handler13ha_update_rowEPKhS1_+0x2a2)[0x56219b41d732]
|
/usr/sbin/mariadbd(_ZN21Update_rows_log_event11do_exec_rowEP14rpl_group_info+0x2b4)[0x56219b533ed4]
|
/usr/sbin/mariadbd(_ZN14Rows_log_event14do_apply_eventEP14rpl_group_info+0x33f)[0x56219b5279cf]
|
/usr/sbin/mariadbd(_Z18wsrep_apply_eventsP3THDP14Relay_log_infoPKvm+0x1e9)[0x56219b6cfa19]
|
/usr/sbin/mariadbd(+0xc19b30)[0x56219b6b6b30]
|
/usr/sbin/mariadbd(_ZN21Wsrep_applier_service15apply_write_setERKN5wsrep7ws_metaERKNS0_12const_bufferERNS0_14mutable_bufferE+0xa6)[0x56219b6b7966]
|
/usr/sbin/mariadbd(+0xef47c7)[0x56219b9917c7]
|
/usr/sbin/mariadbd(+0xf05c2e)[0x56219b9a2c2e]
|
/usr/lib/galera/libgalera_smm.so(+0x576b1)[0x7f1d2f8996b1]
|
/usr/lib/galera/libgalera_smm.so(+0x6591f)[0x7f1d2f8a791f]
|
/usr/lib/galera/libgalera_smm.so(+0x6a724)[0x7f1d2f8ac724]
|
/usr/lib/galera/libgalera_smm.so(+0x959df)[0x7f1d2f8d79df]
|
/usr/lib/galera/libgalera_smm.so(+0x964a2)[0x7f1d2f8d84a2]
|
/usr/lib/galera/libgalera_smm.so(+0x6ab78)[0x7f1d2f8acb78]
|
/usr/lib/galera/libgalera_smm.so(+0x459cd)[0x7f1d2f8879cd]
|
/usr/sbin/mariadbd(_ZN5wsrep18wsrep_provider_v2611run_applierEPNS_21high_priority_serviceE+0xe)[0x56219b9a329e]
|
/usr/sbin/mariadbd(+0xc34e0d)[0x56219b6d1e0d]
|
/usr/sbin/mariadbd(_Z15start_wsrep_THDPv+0x26b)[0x56219b6c1d0b]
|
/usr/sbin/mariadbd(+0xbb4ceb)[0x56219b651ceb]
|
/lib/x86_64-linux-gnu/libpthread.so.0(+0x7ea7)[0x7f1d38481ea7]
|
/lib/x86_64-linux-gnu/libc.so.6(clone+0x3f)[0x7f1d38088a2f]
|
|
|
Trying to get some variables.
|
Some pointers may be invalid and cause the dump to abort.
|
Query (0x7f1d27bc637b): UPDATE media_contents_varchar SET media_id=12820, tag_id=0, tag_internal_id=14, media_content_value='14-550337', media_content_dtupdate='2024-07-11 14:11:40' WHERE media_content_id=1382950
|
|
|
Connection ID (thread ID): 2
|
Status: NOT_KILLED
|
|
|
Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=on,condition_pushdown_from_having=on,not_null_range_scan=off
|
|
|
The manual page at https://mariadb.com/kb/en/how-to-produce-a-full-stack-trace-for-mariadbd/ contains
|
information that should help you find out what is causing the crash.
|
Writing a core file...
|
Working directory at /data/server2
|
Resource Limits:
|
Limit Soft Limit Hard Limit Units
|
Max cpu time unlimited unlimited seconds
|
Max file size unlimited unlimited bytes
|
Max data size unlimited unlimited bytes
|
Max stack size 8388608 unlimited bytes
|
Max core file size 0 unlimited bytes
|
Max resident set unlimited unlimited bytes
|
Max processes 128112 128112 processes
|
Max open files 1048576 1048576 files
|
Max locked memory 65536 65536 bytes
|
Max address space unlimited unlimited bytes
|
Max file locks unlimited unlimited locks
|
Max pending signals 128112 128112 signals
|
Max msgqueue size 819200 819200 bytes
|
Max nice priority 0 0
|
Max realtime priority 0 0
|
Max realtime timeout unlimited unlimited us
|
Core pattern: core
|
|
|
Kernel version: Linux version 5.10.0-26-amd64 (debian-kernel@lists.debian.org) (gcc-10 (Debian 10.2.1-6) 10.2.1 20210110, GNU ld (GNU Binutils for Debian) 2.35.2) #1 SMP Debian 5.10.197-1 (2023-09-29)
|
CREATE TABLE `media_contents_varchar` (
|
  `media_content_id` BIGINT(20) UNSIGNED NOT NULL AUTO_INCREMENT, |
  `media_id` BIGINT(20) NOT NULL, |
  `tag_id` BIGINT(20) UNSIGNED NOT NULL, |
  `tag_internal_id` BIGINT(20) NOT NULL, |
  `media_content_value` VARCHAR(255) NOT NULL COLLATE 'utf8_general_ci', |
  `media_content_dtcreate` DATETIME NOT NULL DEFAULT '0000-00-00 00:00:00', |
  `media_content_dtupdate` DATETIME NOT NULL DEFAULT '0000-00-00 00:00:00', |
  `media_content_altref` VARCHAR(255) AS (if(`tag_internal_id` = 14,`media_content_value`,NULL)) virtual, |
  PRIMARY KEY (`media_content_id`) USING BTREE,
|
  INDEX `ix_media_tag` (`media_id`, `tag_internal_id`) USING BTREE,
|
  INDEX `ix_media_id` (`media_id`) USING BTREE,
|
  INDEX `IX_altref` (`media_content_altref`(10)) USING BTREE |
)
|
COLLATE='utf8_general_ci' |
ENGINE=InnoDB
|
AUTO_INCREMENT=1512055 |
;
|
Attachments
Issue Links
- relates to
-
-
- Closed
-
-
-
- Confirmed
-
Activity
Hello dcz01 and thanks for the report!
This will take some time before I will be able to get to a full-time work on this bug. Meanwhile:
1. Since this problem affects your system severely, can you CHANGE media_content_altref to STORED? The trade-off for you will be that media_content_altref will be stored not only in the secondary index, but also in clustered index. We have a number of bugs with VIRTUAL column indexes, and most of them do not affect STORED.
2. I can't say much by "!cursor->index->is_committed()" assertion. Can you provide a core dump of a failing node?
It's debian 11, as I understood. x86_64?
Do you use the build from repository?
What's galera version?
What is your replica topology – 2 nodes?
All the specifics about the server will be useful – i.e. what server variables have non-default values, especially things like innodb persistent statistics.
Any server or galera logs may be also useful.
Best regards.
Hi Nikita,
Can i still generate a core dump of the node since it was restartet?
Yes it is Debian 11 x86_64 correct.
The mariadb-build used is from the Debian repository.
The galera version is 26.4.18-0+deb11u1.
The replication topology was an 2 node cluster without garb, but now we're using an garb in the cluster.
Regards
> Can i still generate a core dump of the node since it was restartet?
If core dumping was enabled, it hopefully can be found with coredumpctl.
Alright, then now core dumping is enabled with systemd-coredump.
Here are also our used config parameters for the server(s):
[mysqld]
|
key_buffer_size = 256M
|
table_open_cache = 8192
|
table_definition_cache = 8192
|
thread_cache_size = 32
|
max_user_connections = 0
|
tmp_table_size = 32M
|
max_heap_table_size = 32M
|
join_buffer_size = 2M
|
sort_buffer_size = 2M
|
read_rnd_buffer_size = 1M
|
|
|
query_cache_limit = 1M
|
query_cache_size = 0
|
query_cache_type = 0
|
|
|
innodb_buffer_pool_size = 16G
|
innodb_buffer_pool_instances = 16
|
innodb_thread_concurrency = 8
|
innodb_log_file_size = 6G
|
innodb_log_buffer_size = 16M
|
innodb_flush_log_at_trx_commit = 0
|
innodb_lock_wait_timeout = 300
|
innodb_io_capacity = 4000
|
innodb_io_capacity_max = 8000
|
innodb_read_io_threads = 8
|
innodb_write_io_threads = 8
|
|
|
binlog_format = ROW
|
default-storage-engine = InnoDB
|
innodb_autoinc_lock_mode = 2
|
innodb_file_per_table = ON
|
binlog-row-image = minimal
|
innodb_doublewrite = 1
|
|
|
wsrep_on = ON
|
wsrep_provider = /usr/lib/galera/libgalera_smm.so
|
wsrep_cluster_name = "sql-gc01"
|
wsrep_cluster_address = "gcomm://server1,server2,garb"
|
wsrep_node_address = "server2"
|
wsrep_node_name = "server2"
|
wsrep_sst_method = "mariabackup"
|
wsrep_sst_auth = user:password
|
ignore_db_dirs = .snapshot
|
wsrep_provider_options= "gcache.size=3G"
|
dcz01 hi,
I couldn't get to this bug yet.
Thanks for your patience.
Is there already any update?
We're having this error on our production system, in staging the error couldn't be reproduced with same version, config and an actual dump of the production data.