[MDEV-31078] MariaDB Services Crash When Starting Replication Services Created: 2023-04-18  Updated: 2023-06-20  Resolved: 2023-06-20

Status: Closed
Project: MariaDB Server
Component/s: Replication
Affects Version/s: 10.6.12
Fix Version/s: 10.11.3

Type: Bug Priority: Major
Reporter: Roy Shelton Assignee: Unassigned
Resolution: Fixed Votes: 0
Labels: innodb, replication
Environment:

Linux 3.10.0-1160.88.1.el7.x86_64 #1 SMP Tue Mar 7 15:41:52 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux


 Description   

After installed Version 10.6.12 we're able to get the database running well. The master runs even when replication fails but when we run "start slave" on the newly-patched slave server to begin replication, we get the following output. We've run the query manually on the slave server and it doesn't crash the database.

Apr 18 15:59:54  systemd: Starting MariaDB 10.6.12 database server...
 
Apr 18 15:59:54  mariadbd: 2023-04-18 15:59:54 0 [Note] Starting MariaDB 10.6.12-MariaDB-log source revision 4c79e15cc3716f69c044d4287ad2160da8101cdc as process 17262
 
Apr 18 15:59:54  mariadbd: 2023-04-18 15:59:54 0 [Note] InnoDB: Compressed tables use zlib 1.2.7
 
Apr 18 15:59:54  mariadbd: 2023-04-18 15:59:54 0 [Note] InnoDB: Number of pools: 1
 
Apr 18 15:59:54  mariadbd: 2023-04-18 15:59:54 0 [Note] InnoDB: Using crc32 + pclmulqdq instructions
 
Apr 18 15:59:54  mariadbd: 2023-04-18 15:59:54 0 [Note] InnoDB: Using Linux native AIO
 
Apr 18 15:59:54  mariadbd: 2023-04-18 15:59:54 0 [Note] InnoDB: Initializing buffer pool, total size = 134217728, chunk size = 134217728
 
Apr 18 15:59:54  mariadbd: 2023-04-18 15:59:54 0 [Note] InnoDB: Completed initialization of buffer pool
 
Apr 18 15:59:54  mariadbd: 2023-04-18 15:59:54 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=42266,42266
 
Apr 18 15:59:54  mariadbd: 2023-04-18 15:59:54 0 [Note] InnoDB: Starting final batch to recover 24 pages from redo log.
 
Apr 18 15:59:54  mariadbd: 2023-04-18 15:59:54 0 [Note] InnoDB: Last binlog file './-bin.000002', position 13449
 
Apr 18 15:59:54  mariadbd: 2023-04-18 15:59:54 0 [Note] InnoDB: 128 rollback segments are active.
 
Apr 18 15:59:54  mariadbd: 2023-04-18 15:59:54 0 [Note] InnoDB: Removed temporary tablespace data file: "./ibtmp1"
 
Apr 18 15:59:54  mariadbd: 2023-04-18 15:59:54 0 [Note] InnoDB: Creating shared tablespace for temporary tables
 
Apr 18 15:59:54  mariadbd: 2023-04-18 15:59:54 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ...
 
Apr 18 15:59:54  mariadbd: 2023-04-18 15:59:54 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB.
 
Apr 18 15:59:54  mariadbd: 2023-04-18 15:59:54 0 [Note] InnoDB: 10.6.12 started; log sequence number 80770; transaction id 57
 
Apr 18 15:59:54  mariadbd: 2023-04-18 15:59:54 0 [Note] Plugin 'FEEDBACK' is disabled.
 
Apr 18 15:59:54  mariadbd: 2023-04-18 15:59:54 0 [Note] Recovering after a crash using -bin
 
Apr 18 15:59:54  mariadbd: 2023-04-18 15:59:54 0 [Note] Starting table crash recovery...
 
Apr 18 15:59:54  mariadbd: 2023-04-18 15:59:54 0 [Note] Crash table recovery finished.
 
Apr 18 15:59:54  mariadbd: 2023-04-18 15:59:54 0 [Note] InnoDB: Loading buffer pool(s) from /var/lib/mysql/ib_buffer_pool
 
Apr 18 15:59:54  mariadbd: 2023-04-18 15:59:54 0 [Note] InnoDB: Buffer pool(s) load completed at 230418 15:59:54
 
Apr 18 15:59:54  mariadbd: 2023-04-18 15:59:54 0 [Note] Server socket created on IP: '0.0.0.0'.
 
Apr 18 15:59:54  mariadbd: 2023-04-18 15:59:54 0 [Note] Server socket created on IP: '::'.
 
Apr 18 15:59:54  mariadbd: 2023-04-18 15:59:54 5 [Note] Slave I/O thread: Start asynchronous replication to master 'replication@10.10.25.51:3306' in log 'da3shml001-bin.000002' at position 1038795
 
Apr 18 15:59:54  mariadbd: 2023-04-18 15:59:54 6 [Note] Slave SQL thread initialized, starting replication in log 'da3shml001-bin.000001' at position 3733, relay log './-relay-bin.000002' position: 4037
 
Apr 18 15:59:54  mariadbd: 230418 15:59:54 [ERROR] mysqld got signal 11 ;
 
Apr 18 15:59:54  mariadbd: This could be because you hit a bug. It is also possible that this binary
 
Apr 18 15:59:54  mariadbd: or one of the libraries it was linked against is corrupt, improperly built,
 
Apr 18 15:59:54  mariadbd: or misconfigured. This error can also be caused by malfunctioning hardware.
 
Apr 18 15:59:54  mariadbd: To report this bug, see https://mariadb.com/kb/en/reporting-bugs
 
Apr 18 15:59:54  mariadbd: We will try our best to scrape up some info that will hopefully help
 
Apr 18 15:59:54  mariadbd: diagnose the problem, but since we have already crashed,
 
Apr 18 15:59:54  mariadbd: something is definitely wrong and this may fail.
 
Apr 18 15:59:54  mariadbd: Server version: 10.6.12-MariaDB-log source revision: 4c79e15cc3716f69c044d4287ad2160da8101cdc
 
Apr 18 15:59:54  mariadbd: key_buffer_size=134217728
 
Apr 18 15:59:54  mariadbd: read_buffer_size=131072
 
Apr 18 15:59:54  mariadbd: max_used_connections=0
 
Apr 18 15:59:54  mariadbd: max_threads=153
 
Apr 18 15:59:54  mariadbd: thread_count=3
 
Apr 18 15:59:54  mariadbd: It is possible that mysqld could use up to
 
Apr 18 15:59:54  mariadbd: key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 467967 K  bytes of memory
 
Apr 18 15:59:54  mariadbd: Hope that's ok; if not, decrease some variables in the equation.
 
Apr 18 15:59:54  mariadbd: Thread pointer: 0x7fad500013d8
 
Apr 18 15:59:54  mariadbd: Attempting backtrace. You can use the following information to find out
 
Apr 18 15:59:54  mariadbd: where mysqld died. If you see no messages after this, something went
 
Apr 18 15:59:54  mariadbd: terribly wrong...
 
Apr 18 15:59:54  mariadbd: stack_bottom = 0x7fad7c2b6520 thread_stack 0x49000
 
Apr 18 15:59:54  mariadbd: 2023-04-18 15:59:54 0 [Note] /usr/sbin/mariadbd: ready for connections.
 
Apr 18 15:59:54  mariadbd: Version: '10.6.12-MariaDB-log'  socket: '/var/lib/mysql/mysql.sock'  port: 3306  MariaDB Server
 
Apr 18 15:59:54  mariadbd: 2023-04-18 15:59:54 5 [Note] Slave I/O thread: connected to master 'replication@10.10.25.51:3306',replication started in log 'da3shml001-bin.000002' at position 1038795
 
Apr 18 15:59:54  mariadbd: ??:0(my_print_stacktrace)[0x55c84d78f8fe]
 
Apr 18 15:59:54  mariadbd: ??:0(handle_fatal_signal)[0x55c84d1ea997]
 
Apr 18 15:59:54  systemd: Started MariaDB 10.6.12 database server.
 
Apr 18 15:59:54  mariadbd: sigaction.c:0(__restore_rt)[0x7fad89758630]
 
Apr 18 15:59:54  mariadbd: ??:0(mysql_create_table(THD*, TABLE_LIST*, Table_specification_st*, Alter_info*))[0x55c84d0633a9]
 
Apr 18 15:59:54  mariadbd: ??:0(Sql_cmd_create_table_like::execute(THD*))[0x55c84d064c75]
 
Apr 18 15:59:54  mariadbd: ??:0(mysql_execute_command(THD*, bool))[0x55c84cfca498]
 
Apr 18 15:59:54  mariadbd: ??:0(mysql_parse(THD*, char*, unsigned int, Parser_state*))[0x55c84cfced8b]
 
Apr 18 15:59:54  mariadbd: ??:0(Query_log_event::do_apply_event(rpl_group_info*, char const*, unsigned int))[0x55c84d3016c5]
 
Apr 18 15:59:54  mariadbd: ??:0(non-virtual thunk to Item_string_sys::~Item_string_sys())[0x55c84cf1216f]
 
Apr 18 15:59:54  mariadbd: ??:0(handle_slave_sql)[0x55c84cf1c7c0]
 
Apr 18 15:59:54  mariadbd: ??:0(MyCTX_nopad::finish(unsigned char*, unsigned int*))[0x55c84d42f45c]
 
Apr 18 15:59:54  mariadbd: pthread_create.c:0(start_thread)[0x7fad89750ea5]
 
Apr 18 15:59:54  mariadbd: ??:0(__clone)[0x7fad88c6bb0d]
 
Apr 18 15:59:54  mariadbd: Trying to get some variables.
 
Apr 18 15:59:54  mariadbd: Some pointers may be invalid and cause the dump to abort.
 
Apr 18 15:59:54  mariadbd: Query (0x7fad50019584): CREATE TABLE IF NOT EXISTS user (   Host char(60) binary DEFAULT '' NOT NULL, User char(16) binary DEFAULT '' NOT NULL, Password char(41) character set latin1 collate latin1_bin DEFAULT '' NOT NULL, Select_priv enum('N','Y') COLLATE utf8_general_ci DEFAULT 'N' NOT NULL, Insert_priv enum('N','Y') COLLATE utf8_general_ci DEFAULT 'N' NOT NULL, Update_priv enum('N','Y') COLLATE utf8_general_ci DEFAULT 'N' NOT NULL, Delete_priv enum('N','Y') COLLATE utf8_general_ci DEFAULT 'N' NOT NULL, Create_priv enum('N','Y') COLLATE utf8_general_ci DEFAULT 'N' NOT NULL, Drop_priv enum('N','Y') COLLATE utf8_general_ci DEFAULT 'N' NOT NULL, Reload_priv enum('N','Y') COLLATE utf8_general_ci DEFAULT 'N' NOT NULL, Shutdown_priv enum('N','Y') COLLATE utf8_general_ci DEFAULT 'N' NOT NULL, Process_priv enum('N','Y') COLLATE utf8_general_ci DEFAULT 'N' NOT NULL, File_priv enum('N','Y') COLLATE utf8_general_ci DEFAULT 'N' NOT NULL, Grant_priv enum('N','Y') COLLATE utf8_general_ci DEFAULT 'N' NOT NULL, References_priv enum('N','Y') COLLATE utf8_general_ci DEFAULT 'N' NOT NULL, Index_priv enum('N','Y') COLLATE utf8_general_ci DEFAULT 'N' NOT NULL, Alter_priv enum('N','Y') COLLATE utf8_general_ci DEFAULT 'N' NOT NULL, Show_db_priv enum('N','Y') COLLATE utf8_general_ci DEFAULT 'N' NOT NULL, Super_priv enum('N','Y') COLLATE utf8_general_ci DEFAULT 'N' NOT NULL, Create_tmp_table_priv enum('N','Y') COLLATE utf8_general_ci DEFAULT 'N' NOT NULL, Lock_tables_priv enum('N','Y') COLLATE utf8_general_ci DEFAULT 'N' NOT NULL, Execute_priv enum('N','Y') COLLATE utf8_general_ci DEFAULT 'N' NOT NULL, Repl_slave_priv enum('N','Y') COLLATE utf8_general_ci DEFAULT 'N' NOT NULL, Repl_client_priv enum('N','Y') COLLATE utf8_general_ci DEFAULT 'N' NOT NULL, Create_view_priv enum('N','Y') COLLATE utf8_general_ci DEFAULT 'N' NOT NULL, Show_view_priv enum('N','Y') COLLATE utf8_general_ci DEFAULT 'N' NOT NULL, Create_routine_priv enum('N','Y') COLLATE utf8_general_ci DEFAULT 'N' NOT NULL, Alter_routine_priv enum('N','Y') COLLATE utf8_general_ci DEFAULT 'N' NOT NULL, Create_user_priv enum('N','Y') COLLATE utf8_general_ci DEFAULT 'N' NOT NULL, Event_priv enum('N','Y') COLLATE utf8_general_ci DEFAULT 'N' NOT NULL, Trigger_priv enum('N','Y') COLLATE utf8_general_ci DEFAULT 'N' NOT NULL, Create_tablespace_priv enum('N','Y') COLLATE utf8_general_ci DEFAULT 'N' NOT NULL, ssl_type enum('','ANY','X509', 'SPECIFIED') COLLATE utf8_general_ci DEFAULT '' NOT NULL, ssl_cipher BLOB NOT NULL, x509_issuer BLOB NOT NULL, x509_subject BLOB NOT NULL, max_questions int(11) unsigned DEFAULT 0  NOT NULL, max_updates int(11) unsigned DEFAULT 0  NOT NULL, max_connections int(11) unsigned DEFAULT 0  NOT NULL, max_user_connections int(11) DEFAULT 0  NOT NULL, plugin char(64) CHARACTER SET latin1 DEFAULT '' NOT NULL, authentication_string TEXT NOT NULL, PRIMARY KEY Host (Host,User) ) engine=MyISAM CHARACTER SET utf8 COLLATE utf8_bin comment='Users and global privileges'
 
Apr 18 15:59:54  mariadbd: Connection ID (thread ID): 6
 
Apr 18 15:59:54  mariadbd: Status: NOT_KILLED
 
Apr 18 15:59:54  mariadbd: Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=on,condition_pushdown_from_having=on,not_null_range_scan=off
 
Apr 18 15:59:54  mariadbd: The manual page at https://mariadb.com/kb/en/how-to-produce-a-full-stack-trace-for-mysqld/ contains
 
Apr 18 15:59:54  mariadbd: information that should help you find out what is causing the crash.
 
Apr 18 15:59:54  mariadbd: Writing a core file...
 
Apr 18 15:59:54  mariadbd: Working directory at /var/lib/mysql
 
Apr 18 15:59:54  mariadbd: Resource Limits:
 
Apr 18 15:59:54  mariadbd: Limit                     Soft Limit           Hard Limit           Units
 
Apr 18 15:59:54  mariadbd: Max cpu time              unlimited            unlimited            seconds
 
Apr 18 15:59:54  mariadbd: Max file size             unlimited            unlimited            bytes
 
Apr 18 15:59:54  mariadbd: Max data size             unlimited            unlimited            bytes
 
Apr 18 15:59:54  mariadbd: Max stack size            8388608              unlimited            bytes
 
Apr 18 15:59:54  mariadbd: Max core file size        0                    unlimited            bytes
 
Apr 18 15:59:54  mariadbd: Max resident set          unlimited            unlimited            bytes
 
Apr 18 15:59:54  mariadbd: Max processes             15059                15059                processes
 
Apr 18 15:59:54  mariadbd: Max open files            32768                32768                files
 
Apr 18 15:59:54  mariadbd: Max locked memory         65536                65536                bytes
 
Apr 18 15:59:54  mariadbd: Max address space         unlimited            unlimited            bytes
 
Apr 18 15:59:54  mariadbd: Max file locks            unlimited            unlimited            locks
 
Apr 18 15:59:54  mariadbd: Max pending signals       15059                15059                signals
 
Apr 18 15:59:54  mariadbd: Max msgqueue size         819200               819200               bytes
 
Apr 18 15:59:54  mariadbd: Max nice priority         0                    0
 
Apr 18 15:59:54  mariadbd: Max realtime priority     0                    0
 
Apr 18 15:59:54  mariadbd: Max realtime timeout      unlimited            unlimited            us
 
Apr 18 15:59:54  mariadbd: Core pattern: core
 
Apr 18 15:59:54  mariadbd: Kernel version: Linux version 3.10.0-1160.88.1.el7.x86_64 (mockbuild@kbuilder.bsys.centos.org) (gcc version 4.8.5 20150623 (Red Hat 4.8.5-44) (GCC) ) #1 SMP Tue Mar 7 15:41:52 UTC 2023
 
Apr 18 15:59:54  kernel: mariadbd[17276]: segfault at 102 ip 000055c84d0633a9 sp 00007fad7c2b5710 error 4 in mariadbd[55c84c813000+1639000]
 
Apr 18 15:59:54  systemd: mariadb.service: main process exited, code=killed, status=11/SEGV
 
Apr 18 15:59:54  systemd: Unit mariadb.service entered failed state.
 
Apr 18 15:59:54  systemd: mariadb.service failed.



 Comments   
Comment by Daniel Black [ 2023-04-18 ]

Thanks for the bug report. To clarify, is the master server version the same (10.6.12?)?

Do you know how a pre-10.4 mysql.user table is ending up in the replication stream?

The stack trace make it look like a "CREATE TABLE user LIKE mysql.user" however I get even an error when attempting this. Can you include the master/relay log at the crashed position (probably text from is sufficient using mariadb-binlog).

After installing debuginfo symbols package, can you obtain a stack trace using the core file or just gdb attached to the server when you run "start slave".

Comment by Roy Shelton [ 2023-04-19 ]

To clarify, the master server version the same (10.6.12).

I do not know how a pre-10.4 mysql.user table is ending up in the replication stream. I've inherited this database so perhaps it was patched from pre-10.4 to 5.5.61-MariaDB-38.13?

/*!50530 SET @@SESSION.PSEUDO_SLAVE_MODE=1*/;
 
/*!40019 SET @@session.max_insert_delayed_threads=0*/;
 
/*!50003 SET @OLD_COMPLETION_TYPE=@@COMPLETION_TYPE,COMPLETION_TYPE=0*/;
 
DELIMITER /*!*/;
 
# at 4
 
#230418 16:25:01 server id 3  end_log_pos 256 CRC32 0x7691242a 	Start: binlog v 4, server v 10.6.12-MariaDB-log created 230418 16:25:01 at startup
 
ROLLBACK/*!*/;
 
BINLOG '
 
rQo/ZA8DAAAA/AAAAAABAABAAAQAMTAuNi4xMi1NYXJpYURCLWxvZwAAAAAAAAAAAAAAAAAAAAAA
 
AAAAAAAAAAAAAAAAAACtCj9kEzgNAAgAEgAEBAQEEgAA5AAEGggAAAAICAgCAAAACgoKAAAAAAAA
 
AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
 
AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
 
AAAAAAAAAAAEEwQADQgICAoKCgEqJJF2
 
'/*!*/;
 
DELIMITER ;
 
# End of log file
 
ROLLBACK /* added by mysqlbinlog */;
 
/*!50003 SET COMPLETION_TYPE=@OLD_COMPLETION_TYPE*/;
 
/*!50530 SET @@SESSION.PSEUDO_SLAVE_MODE=0*/;


Reading symbols from /usr/sbin/mariadbd...Reading symbols from /usr/lib/debug/usr/sbin/mariadbd.debug...done.
 
done.
 
BFD: Warning: /tmp/corefiles/core.11846 is truncated: expected core file size >= 380432384, found: 246255616.
 
[New LWP 11860]
 
[New LWP 11846]
 
[New LWP 11848]
 
[New LWP 11849]
 
[New LWP 11850]
 
[New LWP 11851]
 
[New LWP 11852]
 
[New LWP 11853]
 
[New LWP 11854]
 
[New LWP 11855]
 
[New LWP 11856]
 
[New LWP 11857]
 
[New LWP 11858]
 
[New LWP 11859]
 
Cannot access memory at address 0x7fe1925b6128
 
Cannot access memory at address 0x7fe1925b6120
 
Failed to read a valid object file image from memory.
 
Core was generated by `/usr/sbin/mariadbd'.
 
Program terminated with signal 11, Segmentation fault.
 
#0  0x00007fe190b6faa1 in ?? ()
 
(gdb) set logging file //var/tmp/gdb_output.log
 
(gdb) set logging on
 
Copying output to //var/tmp/gdb_output.log.
 
(gdb) bt -frame-arguments all full
 
No symbol "frame" in current context.
 
(gdb) thread apply all bt -frame-arguments all full
 
 
 
Thread 14 (LWP 11859):
 
No symbol "frame" in current context.
 
(gdb) bt all full
 
No symbol "all" in current context.
 
(gdb) bt full
 
#0  0x00007fe190b6faa1 in ?? ()
 
No symbol table info available.
 
#1  0x0000557d76102a0e in handle_fatal_signal (sig=11) at /usr/src/debug/MariaDB-/src_0/sql/signal_handler.cc:357
 
        curr_time = 1681914087
 
        tm = {tm_sec = 27, tm_min = 21, tm_hour = 9, tm_mday = 19, tm_mon = 3, tm_year = 123, tm_wday = 3, tm_yday = 108, tm_isdst = 1, tm_gmtoff = -18000, tm_zone = 0x557d7872c220 "CDT"}
 
        thd = 0x7fe158001a28
 
        print_invalid_query_pointer = false
 
#2  0x00007fe190b72630 in ?? ()
 
No symbol table info available.
 
#3  0x0000000000000001 in ?? ()
 
No symbol table info available.
 
#4  0x0000000000000000 in ?? ()
 
No symbol table info available.
 
(gdb) thread apply all full
 
 
 
Thread 14 (LWP 11859):
 
Undefined command: "full".  Try "help".
 
(gdb) thread apply all
 
Please specify a command following the thread ID list
 
(gdb) thread apply all full
 
 
 
Thread 14 (LWP 11859):
 
Undefined command: "full".  Try "help".
 
(gdb) help
 
List of classes of commands:
 
 
 
aliases -- Aliases of other commands
 
breakpoints -- Making program stop at certain points
 
data -- Examining data
 
files -- Specifying and examining files
 
internals -- Maintenance commands
 
obscure -- Obscure features
 
running -- Running the program
 
stack -- Examining the stack
 
status -- Status inquiries
 
support -- Support facilities
 
tracepoints -- Tracing of program execution without stopping the program
 
user-defined -- User-defined commands
 
 
 
Type "help" followed by a class name for a list of commands in that class.
 
Type "help all" for the list of all commands.
 
Type "help" followed by command name for full documentation.
 
Type "apropos word" to search for commands related to "word".
 
Command name abbreviations are allowed if unambiguous.
 
(gdb) set logging on
 
Already logging to //var/tmp/gdb_output.log.
 
(gdb) set pagination off
 
(gdb) set print frame-arguments all
 
(gdb) thread apply all bt full
 
 
 
Thread 14 (LWP 11859):
 
#0  0x00007fe19007addd in ?? ()
 
No symbol table info available.
 
#1  0x0000000000000000 in ?? ()
 
No symbol table info available.
 
 
 
Thread 13 (LWP 11858):
 
#0  0x00007fe18ffbe58a in ?? ()
 
No symbol table info available.
 
#1  0x000000000000001e in ?? ()
 
No symbol table info available.
 
#2  0x00007fe184368b90 in ?? ()
 
No symbol table info available.
 
#3  0x0000000000000010 in ?? ()
 
No symbol table info available.
 
#4  0x8ab15aae7c9a0c00 in ?? ()
 
No symbol table info available.
 
#5  0x00007fe154001088 in ?? ()
 
No symbol table info available.
 
#6  0x0000000000000020 in ?? ()
 
No symbol table info available.
 
#7  0x0000000000000020 in ?? ()
 
No symbol table info available.
 
#8  0x0000000000000006 in ?? ()
 
No symbol table info available.
 
#9  0x00007fe184368b30 in ?? ()
 
No symbol table info available.
 
#10 0x0000000000000000 in ?? ()
 
No symbol table info available.
 
 
 
Thread 12 (LWP 11857):
 
#0  0x00007fe190b6ede2 in ?? ()
 
No symbol table info available.
 
#1  0x0000000000000000 in ?? ()
 
No symbol table info available.
 
 
 
Thread 11 (LWP 11856):
 
#0  0x00007fe190b6ea35 in ?? ()
 
No symbol table info available.
 
#1  0x0000000000000000 in ?? ()
 
No symbol table info available.
 
 
 
Thread 10 (LWP 11855):
 
#0  0x00007fe190b6ea35 in ?? ()
 
No symbol table info available.
 
#1  0x0000000000000000 in ?? ()
 
No symbol table info available.
 
 
 
Thread 9 (LWP 11854):
 
#0  0x00007fe190b6ede2 in ?? ()
 
No symbol table info available.
 
#1  0x0000000000000000 in ?? ()
 
No symbol table info available.
 
 
 
Thread 8 (LWP 11853):
 
#0  0x00007fe190b6ede2 in ?? ()
 
No symbol table info available.
 
#1  0x0000000000000000 in ?? ()
 
No symbol table info available.
 
 
 
Thread 7 (LWP 11852):
 
#0  0x00007fe190b6ede2 in ?? ()
 
No symbol table info available.
 
#1  0x0000000000000000 in ?? ()
 
No symbol table info available.
 
 
 
Thread 6 (LWP 11851):
 
#0  0x00007fe190b6ea35 in ?? ()
 
No symbol table info available.
 
#1  0x0000000000000000 in ?? ()
 
No symbol table info available.
 
 
 
Thread 5 (LWP 11850):
 
#0  0x00007fe19007fe29 in ?? ()
 
No symbol table info available.
 
#1  0x0000557d76633018 in my_getevents (min_nr=1, nr=256, ev=0x7fe173ffccc0, ctx=<optimized out>) at /usr/src/debug/MariaDB-/src_0/tpool/aio_linux.cc:63
 
        saved_errno = 0
 
        ret = <optimized out>
 
#2  tpool::aio_linux::getevent_thread_routine (aio=0x557d78848df0) at /usr/src/debug/MariaDB-/src_0/tpool/aio_linux.cc:104
 
        events = {{data = 0x0, obj = 0x557d78866550, res = 16384, res2 = 0}, {data = 0x0, obj = 0x557d788662f8, res = 16384, res2 = 0}, {data = 0x0, obj = 0x557d78866230, res = 16384, res2 = 0}, {data = 0x0, obj = 0x557d78866488, res = 16384, res2 = 0}, {data = 0x0, obj = 0x0, res = 0, res2 = 0} <repeats 252 times>}
 
#3  0x00007fe19070c330 in ?? ()
 
No symbol table info available.
 
#4  0x0000000000000000 in ?? ()
 
No symbol table info available.
 
 
 
Thread 4 (LWP 11849):
 
#0  0x00007fe190b6ede2 in ?? ()
 
No symbol table info available.
 
#1  0x0000000000000000 in ?? ()
 
No symbol table info available.
 
 
 
Thread 3 (LWP 11848):
 
Python Exception <class 'gdb.MemoryError'> Cannot access memory at address 0x7fe19241dc70:
 
 
 
Thread 2 (LWP 11846):
 
Python Exception <class 'gdb.MemoryError'> Cannot access memory at address 0x7ffc51a593e0:
 
 
 
Thread 1 (LWP 11860):
 
#0  0x00007fe190b6faa1 in ?? ()
 
No symbol table info available.
 
#1  0x0000557d76102a0e in handle_fatal_signal (sig=11) at /usr/src/debug/MariaDB-/src_0/sql/signal_handler.cc:357
 
        curr_time = 1681914087
 
        tm = {tm_sec = 27, tm_min = 21, tm_hour = 9, tm_mday = 19, tm_mon = 3, tm_year = 123, tm_wday = 3, tm_yday = 108, tm_isdst = 1, tm_gmtoff = -18000, tm_zone = 0x557d7872c220 "CDT"}
 
        thd = 0x7fe158001a28
 
        print_invalid_query_pointer = false
 
#2  0x00007fe190b72630 in ?? ()
 
No symbol table info available.
 
#3  0x0000000000000001 in ?? ()
 
No symbol table info available.
 
#4  0x0000000000000000 in ?? ()
 
No symbol table info available.
 
(gdb) set logging off
 
Done logging to //var/tmp/gdb_output.log.
 
(gdb) quit
 
[root@wa3shml001 corefiles]# cat //var/tmp/gdb_output.log.
 
cat: //var/tmp/gdb_output.log.: No such file or directory
 
[root@wa3shml001 corefiles]# cat //var/tmp/gdb_output.log
 
No symbol "frame" in current context.
 
 
 
Thread 14 (LWP 11859):
 
No symbol "frame" in current context.
 
No symbol "all" in current context.
 
#0  0x00007fe190b6faa1 in ?? ()
 
No symbol table info available.
 
#1  0x0000557d76102a0e in handle_fatal_signal (sig=11) at /usr/src/debug/MariaDB-/src_0/sql/signal_handler.cc:357
 
        curr_time = 1681914087
 
        tm = {tm_sec = 27, tm_min = 21, tm_hour = 9, tm_mday = 19, tm_mon = 3, tm_year = 123, tm_wday = 3, tm_yday = 108, tm_isdst = 1, tm_gmtoff = -18000, tm_zone = 0x557d7872c220 "CDT"}
 
        thd = 0x7fe158001a28
 
        print_invalid_query_pointer = false
 
#2  0x00007fe190b72630 in ?? ()
 
No symbol table info available.
 
#3  0x0000000000000001 in ?? ()
 
No symbol table info available.
 
#4  0x0000000000000000 in ?? ()
 
No symbol table info available.
 
 
 
Thread 14 (LWP 11859):
 
Undefined command: "full".  Try "help".
 
Please specify a command following the thread ID list
 
 
 
Thread 14 (LWP 11859):
 
Undefined command: "full".  Try "help".
 
List of classes of commands:
 
 
 
aliases -- Aliases of other commands
 
breakpoints -- Making program stop at certain points
 
data -- Examining data
 
files -- Specifying and examining files
 
internals -- Maintenance commands
 
obscure -- Obscure features
 
running -- Running the program
 
stack -- Examining the stack
 
status -- Status inquiries
 
support -- Support facilities
 
tracepoints -- Tracing of program execution without stopping the program
 
user-defined -- User-defined commands
 
 
 
Type "help" followed by a class name for a list of commands in that class.
 
Type "help all" for the list of all commands.
 
Type "help" followed by command name for full documentation.
 
Type "apropos word" to search for commands related to "word".
 
Command name abbreviations are allowed if unambiguous.
 
Already logging to //var/tmp/gdb_output.log.
 
 
 
Thread 14 (LWP 11859):
 
#0  0x00007fe19007addd in ?? ()
 
No symbol table info available.
 
#1  0x0000000000000000 in ?? ()
 
No symbol table info available.
 
 
 
Thread 13 (LWP 11858):
 
#0  0x00007fe18ffbe58a in ?? ()
 
No symbol table info available.
 
#1  0x000000000000001e in ?? ()
 
No symbol table info available.
 
#2  0x00007fe184368b90 in ?? ()
 
No symbol table info available.
 
#3  0x0000000000000010 in ?? ()
 
No symbol table info available.
 
#4  0x8ab15aae7c9a0c00 in ?? ()
 
No symbol table info available.
 
#5  0x00007fe154001088 in ?? ()
 
No symbol table info available.
 
#6  0x0000000000000020 in ?? ()
 
No symbol table info available.
 
#7  0x0000000000000020 in ?? ()
 
No symbol table info available.
 
#8  0x0000000000000006 in ?? ()
 
No symbol table info available.
 
#9  0x00007fe184368b30 in ?? ()
 
No symbol table info available.
 
#10 0x0000000000000000 in ?? ()
 
No symbol table info available.
 
 
 
Thread 12 (LWP 11857):
 
#0  0x00007fe190b6ede2 in ?? ()
 
No symbol table info available.
 
#1  0x0000000000000000 in ?? ()
 
No symbol table info available.
 
 
 
Thread 11 (LWP 11856):
 
#0  0x00007fe190b6ea35 in ?? ()
 
No symbol table info available.
 
#1  0x0000000000000000 in ?? ()
 
No symbol table info available.
 
 
 
Thread 10 (LWP 11855):
 
#0  0x00007fe190b6ea35 in ?? ()
 
No symbol table info available.
 
#1  0x0000000000000000 in ?? ()
 
No symbol table info available.
 
 
 
Thread 9 (LWP 11854):
 
#0  0x00007fe190b6ede2 in ?? ()
 
No symbol table info available.
 
#1  0x0000000000000000 in ?? ()
 
No symbol table info available.
 
 
 
Thread 8 (LWP 11853):
 
#0  0x00007fe190b6ede2 in ?? ()
 
No symbol table info available.
 
#1  0x0000000000000000 in ?? ()
 
No symbol table info available.
 
 
 
Thread 7 (LWP 11852):
 
#0  0x00007fe190b6ede2 in ?? ()
 
No symbol table info available.
 
#1  0x0000000000000000 in ?? ()
 
No symbol table info available.
 
 
 
Thread 6 (LWP 11851):
 
#0  0x00007fe190b6ea35 in ?? ()
 
No symbol table info available.
 
#1  0x0000000000000000 in ?? ()
 
No symbol table info available.
 
 
 
Thread 5 (LWP 11850):
 
#0  0x00007fe19007fe29 in ?? ()
 
No symbol table info available.
 
#1  0x0000557d76633018 in my_getevents (min_nr=1, nr=256, ev=0x7fe173ffccc0, ctx=<optimized out>) at /usr/src/debug/MariaDB-/src_0/tpool/aio_linux.cc:63
 
        saved_errno = 0
 
        ret = <optimized out>
 
#2  tpool::aio_linux::getevent_thread_routine (aio=0x557d78848df0) at /usr/src/debug/MariaDB-/src_0/tpool/aio_linux.cc:104
 
        events = {{data = 0x0, obj = 0x557d78866550, res = 16384, res2 = 0}, {data = 0x0, obj = 0x557d788662f8, res = 16384, res2 = 0}, {data = 0x0, obj = 0x557d78866230, res = 16384, res2 = 0}, {data = 0x0, obj = 0x557d78866488, res = 16384, res2 = 0}, {data = 0x0, obj = 0x0, res = 0, res2 = 0} <repeats 252 times>}
 
#3  0x00007fe19070c330 in ?? ()
 
No symbol table info available.
 
#4  0x0000000000000000 in ?? ()
 
No symbol table info available.
 
 
 
Thread 4 (LWP 11849):
 
#0  0x00007fe190b6ede2 in ?? ()
 
No symbol table info available.
 
#1  0x0000000000000000 in ?? ()
 
No symbol table info available.
 
 
 
Thread 3 (LWP 11848):
 
Python Exception <class 'gdb.MemoryError'> Cannot access memory at address 0x7fe19241dc70:
 
 
 
Thread 2 (LWP 11846):
 
Python Exception <class 'gdb.MemoryError'> Cannot access memory at address 0x7ffc51a593e0:
 
 
 
Thread 1 (LWP 11860):
 
#0  0x00007fe190b6faa1 in ?? ()
 
No symbol table info available.
 
#1  0x0000557d76102a0e in handle_fatal_signal (sig=11) at /usr/src/debug/MariaDB-/src_0/sql/signal_handler.cc:357
 
        curr_time = 1681914087
 
        tm = {tm_sec = 27, tm_min = 21, tm_hour = 9, tm_mday = 19, tm_mon = 3, tm_year = 123, tm_wday = 3, tm_yday = 108, tm_isdst = 1, tm_gmtoff = -18000, tm_zone = 0x557d7872c220 "CDT"}
 
        thd = 0x7fe158001a28
 
        print_invalid_query_pointer = false
 
#2  0x00007fe190b72630 in ?? ()
 
No symbol table info available.
 
#3  0x0000000000000001 in ?? ()
 
No symbol table info available.
 
#4  0x0000000000000000 in ?? ()
 
No symbol table info available.

Comment by Roy Shelton [ 2023-04-28 ]

Any update on this item?

Comment by Roy Shelton [ 2023-05-26 ]

What do you recommend as a next step?

Comment by Daniel Black [ 2023-05-30 ]

I attempted to reproduce on the latest 10.6:

$ podman network create rpl
 
rpl
 
$ podman run -e MARIADB_REPLICATION_USER=bob -e MARIADB_REPLICATION_PASSWORD=jane -e MARIADB_ROOT_PASSWORD=jude --network rpl --name m106 -d quay.io/mariadb-foundation/mariadb-devel:10.6 --server-id=3000 --log-bin --log-basename=my-mariadb
 
$ podman run -e MARIADB_REPLICATION_USER=bob -e MARIADB_REPLICATION_PASSWORD=jane -e MARIADB_ROOT_PASSWORD=jude -e MARIADB_MASTER_HOST=m106 --network rpl --name m106rpl -d quay.io/mariadb-foundation/mariadb-devel:10.6 --server-id=3001
 
$ podman exec -ti m106rpl mariadb -pjude 
MariaDB [(none)]> show replica status\G
 
*************************** 1. row ***************************
 
                Slave_IO_State: Waiting for master to send event
 
                   Master_Host: m106
 
                   Master_User: bob
 
                   Master_Port: 3306
 
                 Connect_Retry: 10
 
               Master_Log_File: my-mariadb-bin.000002
 
           Read_Master_Log_Pos: 347
 
                Relay_Log_File: mysqld-relay-bin.000005
 
                 Relay_Log_Pos: 651
 
         Relay_Master_Log_File: my-mariadb-bin.000002
 
              Slave_IO_Running: Yes
 
             Slave_SQL_Running: Yes
 
..
 
$ podman exec -ti m106 mariadb -pjude mysql
 
MariaDB [mysql]> drop view user;
 
Query OK, 0 rows affected (0.006 sec)
 
 
 
MariaDB [mysql]> CREATE TABLE IF NOT EXISTS user (   Host char(60) binary DEFAULT '' NOT NULL, User char(80) binary DEFAULT '' NOT NULL, Password char(41) character set latin1 collate latin1_bin DEFAULT '' NOT NULL, Select_priv enum('N','Y') COLLATE utf8_general_ci DEFAULT 'N' NOT NULL, Insert_priv enum('N','Y') COLLATE utf8_general_ci DEFAULT 'N' NOT NULL, Update_priv enum('N','Y') COLLATE utf8_general_ci DEFAULT 'N' NOT NULL, Delete_priv enum('N','Y') COLLATE utf8_general_ci DEFAULT 'N' NOT NULL, Create_priv enum('N','Y') COLLATE utf8_general_ci DEFAULT 'N' NOT NULL, Drop_priv enum('N','Y') COLLATE utf8_general_ci DEFAULT 'N' NOT NULL, Reload_priv enum('N','Y') COLLATE utf8_general_ci DEFAULT 'N' NOT NULL, Shutdown_priv enum('N','Y') COLLATE utf8_general_ci DEFAULT 'N' NOT NULL, Process_priv enum('N','Y') COLLATE utf8_general_ci DEFAULT 'N' NOT NULL, File_priv enum('N','Y') COLLATE utf8_general_ci DEFAULT 'N' NOT NULL, Grant_priv enum('N','Y') COLLATE utf8_general_ci DEFAULT 'N' NOT NULL, References_priv enum('N','Y') COLLATE utf8_general_ci DEFAULT 'N' NOT NULL, Index_priv enum('N','Y') COLLATE utf8_general_ci DEFAULT 'N' NOT NULL, Alter_priv enum('N','Y') COLLATE utf8_general_ci DEFAULT 'N' NOT NULL, Show_db_priv enum('N','Y') COLLATE utf8_general_ci DEFAULT 'N' NOT NULL, Super_priv enum('N','Y') COLLATE utf8_general_ci DEFAULT 'N' NOT NULL, Create_tmp_table_priv enum('N','Y') COLLATE utf8_general_ci DEFAULT 'N' NOT NULL, Lock_tables_priv enum('N','Y') COLLATE utf8_general_ci DEFAULT 'N' NOT NULL, Execute_priv enum('N','Y') COLLATE utf8_general_ci DEFAULT 'N' NOT NULL, Repl_slave_priv enum('N','Y') COLLATE utf8_general_ci DEFAULT 'N' NOT NULL, Repl_client_priv enum('N','Y') COLLATE utf8_general_ci DEFAULT 'N' NOT NULL, Create_view_priv enum('N','Y') COLLATE utf8_general_ci DEFAULT 'N' NOT NULL, Show_view_priv enum('N','Y') COLLATE utf8_general_ci DEFAULT 'N' NOT NULL, Create_routine_priv enum('N','Y') COLLATE utf8_general_ci DEFAULT 'N' NOT NULL, Alter_routine_priv enum('N','Y') COLLATE utf8_general_ci DEFAULT 'N' NOT NULL, Create_user_priv enum('N','Y') COLLATE utf8_general_ci DEFAULT 'N' NOT NULL, Event_priv enum('N','Y') COLLATE utf8_general_ci DEFAULT 'N' NOT NULL, Trigger_priv enum('N','Y') COLLATE utf8_general_ci DEFAULT 'N' NOT NULL, Create_tablespace_priv enum('N','Y') COLLATE utf8_general_ci DEFAULT 'N' NOT NULL, Delete_history_priv enum('N','Y') COLLATE utf8_general_ci DEFAULT 'N' NOT NULL, ssl_type enum('','ANY','X509', 'SPECIFIED') COLLATE utf8_general_ci DEFAULT '' NOT NULL, ssl_cipher BLOB NOT NULL, x509_issuer BLOB NOT NULL, x509_subject BLOB NOT NULL, max_questions int(11) unsigned DEFAULT 0  NOT NULL, max_updates int(11) unsigned DEFAULT 0  NOT NULL, max_connections int(11) unsigned DEFAULT 0  NOT NULL, max_user_connections int(11) DEFAULT 0  NOT NULL, plugin char(64) CHARACTER SET latin1 DEFAULT '' NOT NULL, authentication_string TEXT NOT NULL, password_expired ENUM('N', 'Y') COLLATE utf8_general_ci DEFAULT 'N' NOT NULL, is_role enum('N', 'Y') COLLATE utf8_general_ci DEFAULT 'N' NOT NULL, default_role char(80) binary DEFAULT '' NOT NULL, max_statement_time decimal(12,6) DEFAULT 0 NOT NULL, PRIMARY KEY /*Host*/ (Host,User) ) engine=MyISAM CHARACTER SET utf8 COLLATE utf8_bin comment='Users and global privileges';
 
Query OK, 0 rows affected (0.006 sec)

So in short I couldn't crash the replica with the same query.

A truncated core dump and the resolution looks like it was using the wrong debug info packages.

If its still reproducable, can you ensure you have the right debug info packages install and attach gdb to the replica, and then "START REPLICA"

Comment by Roy Shelton [ 2023-06-01 ]

We may close this JIRA ticket.

We updated the slave MariaDB server to Version 10.11.3, removed the old database (which we believe was corrupted) and created a new database from scratch and restarted replication with the MariaDB Version 10.6.12 master and now it appears to be working without any errors.

Generated at Thu Feb 08 10:21:06 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.