[MDEV-31078] MariaDB Services Crash When Starting Replication Services - Jira

XML

Word

Printable

Details

Type: Bug
Status: Closed (View Workflow)
Priority: Major
Resolution: Fixed
Affects Version/s: 10.6.12
Fix Version/s: 10.11.3
Component/s: Replication
Labels:
- innodb
- replication
Environment:
Linux 3.10.0-1160.88.1.el7.x86_64 #1 SMP Tue Mar 7 15:41:52 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux

Description

After installed Version 10.6.12 we're able to get the database running well. The master runs even when replication fails but when we run "start slave" on the newly-patched slave server to begin replication, we get the following output. We've run the query manually on the slave server and it doesn't crash the database.

Apr 18 15:59:54  systemd: Starting MariaDB 10.6.12 database server...

Apr 18 15:59:54  mariadbd: 2023-04-18 15:59:54 0 [Note] Starting MariaDB 10.6.12-MariaDB-log source revision 4c79e15cc3716f69c044d4287ad2160da8101cdc as process 17262

Apr 18 15:59:54  mariadbd: 2023-04-18 15:59:54 0 [Note] InnoDB: Compressed tables use zlib 1.2.7

Apr 18 15:59:54  mariadbd: 2023-04-18 15:59:54 0 [Note] InnoDB: Number of pools: 1

Apr 18 15:59:54  mariadbd: 2023-04-18 15:59:54 0 [Note] InnoDB: Using crc32 + pclmulqdq instructions

Apr 18 15:59:54  mariadbd: 2023-04-18 15:59:54 0 [Note] InnoDB: Using Linux native AIO

Apr 18 15:59:54  mariadbd: 2023-04-18 15:59:54 0 [Note] InnoDB: Initializing buffer pool, total size = 134217728, chunk size = 134217728

Apr 18 15:59:54  mariadbd: 2023-04-18 15:59:54 0 [Note] InnoDB: Completed initialization of buffer pool

Apr 18 15:59:54  mariadbd: 2023-04-18 15:59:54 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=42266,42266

Apr 18 15:59:54  mariadbd: 2023-04-18 15:59:54 0 [Note] InnoDB: Starting final batch to recover 24 pages from redo log.

Apr 18 15:59:54  mariadbd: 2023-04-18 15:59:54 0 [Note] InnoDB: Last binlog file './-bin.000002', position 13449

Apr 18 15:59:54  mariadbd: 2023-04-18 15:59:54 0 [Note] InnoDB: 128 rollback segments are active.

Apr 18 15:59:54  mariadbd: 2023-04-18 15:59:54 0 [Note] InnoDB: Removed temporary tablespace data file: "./ibtmp1"

Apr 18 15:59:54  mariadbd: 2023-04-18 15:59:54 0 [Note] InnoDB: Creating shared tablespace for temporary tables

Apr 18 15:59:54  mariadbd: 2023-04-18 15:59:54 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ...

Apr 18 15:59:54  mariadbd: 2023-04-18 15:59:54 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB.

Apr 18 15:59:54  mariadbd: 2023-04-18 15:59:54 0 [Note] InnoDB: 10.6.12 started; log sequence number 80770; transaction id 57

Apr 18 15:59:54  mariadbd: 2023-04-18 15:59:54 0 [Note] Plugin 'FEEDBACK' is disabled.

Apr 18 15:59:54  mariadbd: 2023-04-18 15:59:54 0 [Note] Recovering after a crash using -bin

Apr 18 15:59:54  mariadbd: 2023-04-18 15:59:54 0 [Note] Starting table crash recovery...

Apr 18 15:59:54  mariadbd: 2023-04-18 15:59:54 0 [Note] Crash table recovery finished.

Apr 18 15:59:54  mariadbd: 2023-04-18 15:59:54 0 [Note] InnoDB: Loading buffer pool(s) from /var/lib/mysql/ib_buffer_pool

Apr 18 15:59:54  mariadbd: 2023-04-18 15:59:54 0 [Note] InnoDB: Buffer pool(s) load completed at 230418 15:59:54

Apr 18 15:59:54  mariadbd: 2023-04-18 15:59:54 0 [Note] Server socket created on IP: '0.0.0.0'.

Apr 18 15:59:54  mariadbd: 2023-04-18 15:59:54 0 [Note] Server socket created on IP: '::'.

Apr 18 15:59:54  mariadbd: 2023-04-18 15:59:54 5 [Note] Slave I/O thread: Start asynchronous replication to master 'replication@10.10.25.51:3306' in log 'da3shml001-bin.000002' at position 1038795

Apr 18 15:59:54  mariadbd: 2023-04-18 15:59:54 6 [Note] Slave SQL thread initialized, starting replication in log 'da3shml001-bin.000001' at position 3733, relay log './-relay-bin.000002' position: 4037

Apr 18 15:59:54  mariadbd: 230418 15:59:54 [ERROR] mysqld got signal 11 ;

Apr 18 15:59:54  mariadbd: This could be because you hit a bug. It is also possible that this binary

Apr 18 15:59:54  mariadbd: or one of the libraries it was linked against is corrupt, improperly built,

Apr 18 15:59:54  mariadbd: or misconfigured. This error can also be caused by malfunctioning hardware.

Apr 18 15:59:54  mariadbd: To report this bug, see https://mariadb.com/kb/en/reporting-bugs

Apr 18 15:59:54  mariadbd: We will try our best to scrape up some info that will hopefully help

Apr 18 15:59:54  mariadbd: diagnose the problem, but since we have already crashed,

Apr 18 15:59:54  mariadbd: something is definitely wrong and this may fail.

Apr 18 15:59:54  mariadbd: Server version: 10.6.12-MariaDB-log source revision: 4c79e15cc3716f69c044d4287ad2160da8101cdc

Apr 18 15:59:54  mariadbd: key_buffer_size=134217728

Apr 18 15:59:54  mariadbd: read_buffer_size=131072

Apr 18 15:59:54  mariadbd: max_used_connections=0

Apr 18 15:59:54  mariadbd: max_threads=153

Apr 18 15:59:54  mariadbd: thread_count=3

Apr 18 15:59:54  mariadbd: It is possible that mysqld could use up to

Apr 18 15:59:54  mariadbd: key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 467967 K  bytes of memory

Apr 18 15:59:54  mariadbd: Hope that's ok; if not, decrease some variables in the equation.

Apr 18 15:59:54  mariadbd: Thread pointer: 0x7fad500013d8

Apr 18 15:59:54  mariadbd: Attempting backtrace. You can use the following information to find out

Apr 18 15:59:54  mariadbd: where mysqld died. If you see no messages after this, something went

Apr 18 15:59:54  mariadbd: terribly wrong...

Apr 18 15:59:54  mariadbd: stack_bottom = 0x7fad7c2b6520 thread_stack 0x49000

Apr 18 15:59:54  mariadbd: 2023-04-18 15:59:54 0 [Note] /usr/sbin/mariadbd: ready for connections.

Apr 18 15:59:54  mariadbd: Version: '10.6.12-MariaDB-log'  socket: '/var/lib/mysql/mysql.sock'  port: 3306  MariaDB Server

Apr 18 15:59:54  mariadbd: 2023-04-18 15:59:54 5 [Note] Slave I/O thread: connected to master 'replication@10.10.25.51:3306',replication started in log 'da3shml001-bin.000002' at position 1038795

Apr 18 15:59:54  mariadbd: ??:0(my_print_stacktrace)[0x55c84d78f8fe]

Apr 18 15:59:54  mariadbd: ??:0(handle_fatal_signal)[0x55c84d1ea997]

Apr 18 15:59:54  systemd: Started MariaDB 10.6.12 database server.

Apr 18 15:59:54  mariadbd: sigaction.c:0(__restore_rt)[0x7fad89758630]

Apr 18 15:59:54  mariadbd: ??:0(mysql_create_table(THD*, TABLE_LIST*, Table_specification_st*, Alter_info*))[0x55c84d0633a9]

Apr 18 15:59:54  mariadbd: ??:0(Sql_cmd_create_table_like::execute(THD*))[0x55c84d064c75]

Apr 18 15:59:54  mariadbd: ??:0(mysql_execute_command(THD*, bool))[0x55c84cfca498]

Apr 18 15:59:54  mariadbd: ??:0(mysql_parse(THD*, char*, unsigned int, Parser_state*))[0x55c84cfced8b]

Apr 18 15:59:54  mariadbd: ??:0(Query_log_event::do_apply_event(rpl_group_info*, char const*, unsigned int))[0x55c84d3016c5]

Apr 18 15:59:54  mariadbd: ??:0(non-virtual thunk to Item_string_sys::~Item_string_sys())[0x55c84cf1216f]

Apr 18 15:59:54  mariadbd: ??:0(handle_slave_sql)[0x55c84cf1c7c0]

Apr 18 15:59:54  mariadbd: ??:0(MyCTX_nopad::finish(unsigned char*, unsigned int*))[0x55c84d42f45c]

Apr 18 15:59:54  mariadbd: pthread_create.c:0(start_thread)[0x7fad89750ea5]

Apr 18 15:59:54  mariadbd: ??:0(__clone)[0x7fad88c6bb0d]

Apr 18 15:59:54  mariadbd: Trying to get some variables.

Apr 18 15:59:54  mariadbd: Some pointers may be invalid and cause the dump to abort.

Apr 18 15:59:54  mariadbd: Query (0x7fad50019584): CREATE TABLE IF NOT EXISTS user (   Host char(60) binary DEFAULT '' NOT NULL, User char(16) binary DEFAULT '' NOT NULL, Password char(41) character set latin1 collate latin1_bin DEFAULT '' NOT NULL, Select_priv enum('N','Y') COLLATE utf8_general_ci DEFAULT 'N' NOT NULL, Insert_priv enum('N','Y') COLLATE utf8_general_ci DEFAULT 'N' NOT NULL, Update_priv enum('N','Y') COLLATE utf8_general_ci DEFAULT 'N' NOT NULL, Delete_priv enum('N','Y') COLLATE utf8_general_ci DEFAULT 'N' NOT NULL, Create_priv enum('N','Y') COLLATE utf8_general_ci DEFAULT 'N' NOT NULL, Drop_priv enum('N','Y') COLLATE utf8_general_ci DEFAULT 'N' NOT NULL, Reload_priv enum('N','Y') COLLATE utf8_general_ci DEFAULT 'N' NOT NULL, Shutdown_priv enum('N','Y') COLLATE utf8_general_ci DEFAULT 'N' NOT NULL, Process_priv enum('N','Y') COLLATE utf8_general_ci DEFAULT 'N' NOT NULL, File_priv enum('N','Y') COLLATE utf8_general_ci DEFAULT 'N' NOT NULL, Grant_priv enum('N','Y') COLLATE utf8_general_ci DEFAULT 'N' NOT NULL, References_priv enum('N','Y') COLLATE utf8_general_ci DEFAULT 'N' NOT NULL, Index_priv enum('N','Y') COLLATE utf8_general_ci DEFAULT 'N' NOT NULL, Alter_priv enum('N','Y') COLLATE utf8_general_ci DEFAULT 'N' NOT NULL, Show_db_priv enum('N','Y') COLLATE utf8_general_ci DEFAULT 'N' NOT NULL, Super_priv enum('N','Y') COLLATE utf8_general_ci DEFAULT 'N' NOT NULL, Create_tmp_table_priv enum('N','Y') COLLATE utf8_general_ci DEFAULT 'N' NOT NULL, Lock_tables_priv enum('N','Y') COLLATE utf8_general_ci DEFAULT 'N' NOT NULL, Execute_priv enum('N','Y') COLLATE utf8_general_ci DEFAULT 'N' NOT NULL, Repl_slave_priv enum('N','Y') COLLATE utf8_general_ci DEFAULT 'N' NOT NULL, Repl_client_priv enum('N','Y') COLLATE utf8_general_ci DEFAULT 'N' NOT NULL, Create_view_priv enum('N','Y') COLLATE utf8_general_ci DEFAULT 'N' NOT NULL, Show_view_priv enum('N','Y') COLLATE utf8_general_ci DEFAULT 'N' NOT NULL, Create_routine_priv enum('N','Y') COLLATE utf8_general_ci DEFAULT 'N' NOT NULL, Alter_routine_priv enum('N','Y') COLLATE utf8_general_ci DEFAULT 'N' NOT NULL, Create_user_priv enum('N','Y') COLLATE utf8_general_ci DEFAULT 'N' NOT NULL, Event_priv enum('N','Y') COLLATE utf8_general_ci DEFAULT 'N' NOT NULL, Trigger_priv enum('N','Y') COLLATE utf8_general_ci DEFAULT 'N' NOT NULL, Create_tablespace_priv enum('N','Y') COLLATE utf8_general_ci DEFAULT 'N' NOT NULL, ssl_type enum('','ANY','X509', 'SPECIFIED') COLLATE utf8_general_ci DEFAULT '' NOT NULL, ssl_cipher BLOB NOT NULL, x509_issuer BLOB NOT NULL, x509_subject BLOB NOT NULL, max_questions int(11) unsigned DEFAULT 0  NOT NULL, max_updates int(11) unsigned DEFAULT 0  NOT NULL, max_connections int(11) unsigned DEFAULT 0  NOT NULL, max_user_connections int(11) DEFAULT 0  NOT NULL, plugin char(64) CHARACTER SET latin1 DEFAULT '' NOT NULL, authentication_string TEXT NOT NULL, PRIMARY KEY Host (Host,User) ) engine=MyISAM CHARACTER SET utf8 COLLATE utf8_bin comment='Users and global privileges'

Apr 18 15:59:54  mariadbd: Connection ID (thread ID): 6

Apr 18 15:59:54  mariadbd: Status: NOT_KILLED

Apr 18 15:59:54  mariadbd: Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=on,condition_pushdown_from_having=on,not_null_range_scan=off

Apr 18 15:59:54  mariadbd: The manual page at https://mariadb.com/kb/en/how-to-produce-a-full-stack-trace-for-mysqld/ contains

Apr 18 15:59:54  mariadbd: information that should help you find out what is causing the crash.

Apr 18 15:59:54  mariadbd: Writing a core file...

Apr 18 15:59:54  mariadbd: Working directory at /var/lib/mysql

Apr 18 15:59:54  mariadbd: Resource Limits:

Apr 18 15:59:54  mariadbd: Limit                     Soft Limit           Hard Limit           Units

Apr 18 15:59:54  mariadbd: Max cpu time              unlimited            unlimited            seconds

Apr 18 15:59:54  mariadbd: Max file size             unlimited            unlimited            bytes

Apr 18 15:59:54  mariadbd: Max data size             unlimited            unlimited            bytes

Apr 18 15:59:54  mariadbd: Max stack size            8388608              unlimited            bytes

Apr 18 15:59:54  mariadbd: Max core file size        0                    unlimited            bytes

Apr 18 15:59:54  mariadbd: Max resident set          unlimited            unlimited            bytes

Apr 18 15:59:54  mariadbd: Max processes             15059                15059                processes

Apr 18 15:59:54  mariadbd: Max open files            32768                32768                files

Apr 18 15:59:54  mariadbd: Max locked memory         65536                65536                bytes

Apr 18 15:59:54  mariadbd: Max address space         unlimited            unlimited            bytes

Apr 18 15:59:54  mariadbd: Max file locks            unlimited            unlimited            locks

Apr 18 15:59:54  mariadbd: Max pending signals       15059                15059                signals

Apr 18 15:59:54  mariadbd: Max msgqueue size         819200               819200               bytes

Apr 18 15:59:54  mariadbd: Max nice priority         0                    0

Apr 18 15:59:54  mariadbd: Max realtime priority     0                    0

Apr 18 15:59:54  mariadbd: Max realtime timeout      unlimited            unlimited            us

Apr 18 15:59:54  mariadbd: Core pattern: core

Apr 18 15:59:54  mariadbd: Kernel version: Linux version 3.10.0-1160.88.1.el7.x86_64 (mockbuild@kbuilder.bsys.centos.org) (gcc version 4.8.5 20150623 (Red Hat 4.8.5-44) (GCC) ) #1 SMP Tue Mar 7 15:41:52 UTC 2023

Apr 18 15:59:54  kernel: mariadbd[17276]: segfault at 102 ip 000055c84d0633a9 sp 00007fad7c2b5710 error 4 in mariadbd[55c84c813000+1639000]

Apr 18 15:59:54  systemd: mariadb.service: main process exited, code=killed, status=11/SEGV

Apr 18 15:59:54  systemd: Unit mariadb.service entered failed state.

Apr 18 15:59:54  systemd: mariadb.service failed.

Attachments

Activity

People

Assignee:: Unassigned

Reporter:: Roy Shelton

Votes:: 0 Vote for this issue

Watchers:: 4 Start watching this issue

Dates

Created:: 2023-04-18 21:09

Updated:: 2023-06-20 21:42

Resolved:: 2023-06-20 21:42

Git Integration

Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.