DB2 crashed with Segmentation fault (Signal 11)error and VIP got shifted DB1

[root@ALEPO-DB2-ILB ~]# systemctl status mysqld

● mariadb.service - MariaDB 10.11.5 database server

   Loaded: loaded (/usr/lib/systemd/system/mariadb.service; enabled; vendor preset: disabled)

  Drop-In: /etc/systemd/system/mariadb.service.d

           └─migrated-from-my.cnf-settings.conf

   Active: active (running) since Sat 2024-12-14 05:52:46 PKT; 2 days ago

Dec 14 05:51:20 ALEPO-DB2-ILB NetworkManager[1266]: <info>  [1734137480.0437] dhcp4 (enp9s0): canceled DHCP transaction

Dec 14 05:51:44 ALEPO-DB2-ILB systemd[1]: mariadb.service: Main process exited, code=killed, status=11/SEGV

Dec 14 05:51:44 ALEPO-DB2-ILB systemd[1]: mariadb.service: Failed with result 'signal'.

Dec 14 05:51:49 ALEPO-DB2-ILB systemd[1]: mariadb.service: Service RestartSec=5s expired, scheduling restart.

Dec 14 05:51:49 ALEPO-DB2-ILB systemd[1]: mariadb.service: Scheduled restart job, restart counter is at 4.

Dec 14 05:51:49 ALEPO-DB2-ILB systemd[1]: Stopped MariaDB 10.11.5 database server.

Dec 14 05:51:49 ALEPO-DB2-ILB systemd[1]: Starting MariaDB 10.11.5 database server...

Dec 14 05:51:49 ALEPO-DB2-ILB mariadbd[685235]: 2024-12-14  5:51:49 0 [Warning] Could not increase number of max_open_files to more than 32768 (request: 65535)

Dec 14 05:52:11 ALEPO-DB2-ILB systemd[1]: session-418437.scope: Succeeded.

Dec 14 05:52:24 ALEPO-DB2-ILB snmpd[1499030]: truncating integer value > 32 bits

Dec 14 05:52:28 ALEPO-DB2-ILB snmpd[1499030]: truncating integer value > 32 bits

Dec 14 05:52:46 ALEPO-DB2-ILB systemd[1]: Started MariaDB 10.11.5 database server.

Dec 14 05:55:01 ALEPO-DB2-ILB systemd[1]: Started Session 418438 of user root.

Dec 14 05:55:01 ALEPO-DB2-ILB systemd[1]: Started Session 418439 of user root.

Dec 14 05:55:01 ALEPO-DB2-ILB systemd[1]: session-418439.scope: Succeeded.

Dec 14 05:55:01 ALEPO-DB2-ILB systemd[1]: session-418438.scope: Succeeded.

241214  5:41:49 [ERROR] mysqld got signal 11 ;

This could be because you hit a bug. It is also possible that this binary

or one of the libraries it was linked against is corrupt, improperly built,

or misconfigured. This error can also be caused by malfunctioning hardware.

To report this bug, see https://mariadb.com/kb/en/reporting-bugs

We will try our best to scrape up some info that will hopefully help

diagnose the problem, but since we have already crashed,

something is definitely wrong and this may fail.

Server version: 10.11.5-MariaDB-log source revision: 7875294b6b74b53dd3aaa723e6cc103d2bb47b2c

key_buffer_size=67108864

read_buffer_size=131072

max_used_connections=491

max_threads=802

thread_count=505

It is possible that mysqld could use up to

key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 1831964 K  bytes of memory

Hope that's ok; if not, decrease some variables in the equation.

Thread pointer: 0x7f1668000c58

Attempting backtrace. You can use the following information to find out

where mysqld died. If you see no messages after this, something went

terribly wrong...

stack_bottom = 0x7f19e1605b98 thread_stack 0x49000

/usr/sbin/mariadbd(my_print_stacktrace+0x2e)[0x5641f21a112e]

mysys/stacktrace.c:216(my_print_stacktrace)[0x5641f1c9a5b5]

/lib64/libpthread.so.0(+0x12cf0)[0x7f3fe1255cf0]

sql/multi_range_read.cc:1383(DsMrr_impl::close_second_handler())[0x5641f1ba06b0]

sql/multi_range_read.cc:1398(DsMrr_impl::dsmrr_close())[0x5641f1ba07c9]

handler/ha_innodb.cc:15770(wsrep_notify_status(wsrep::server_state::state, wsrep::view const*))[0x5641f1f443e7]

sql/ha_partition.cc:9520(MyCTX_nopad::finish(unsigned char*, unsigned int*))[0x5641f1e8a481]

sql/sql_base.cc:1022(close_thread_table(THD*, TABLE**))[0x5641f19e87d1]

sql/sql_base.cc:965(close_thread_tables(THD*))[0x5641f19e8aeb]

sql/sql_parse.cc:6097(mysql_execute_command(THD*, bool))[0x5641f1a58d7f]

sql/sql_parse.cc:8035(mysql_parse(THD*, char*, unsigned int, Parser_state*))[0x5641f1a4cb87]

sql/sql_parse.cc:1896(dispatch_command(enum_server_command, THD*, char*, unsigned int, bool))[0x5641f1a566e6]

sql/sql_parse.cc:1407(do_command(THD*, bool))[0x5641f1a57dab]

sql/sql_connect.cc:1416(do_handle_one_connection(CONNECT*, bool))[0x5641f1b6b977]

sql/sql_connect.cc:1318(handle_one_connection)[0x5641f1b6bcbd]

perfschema/pfs.cc:2204(MyCTX_nopad::finish(unsigned char*, unsigned int*))[0x5641f1ea1c9d]

/lib64/libpthread.so.0(+0x81ca)[0x7f3fe124b1ca]

/lib64/libc.so.6(clone+0x43)[0x7f3fe059be73]

Trying to get some variables.

Some pointers may be invalid and cause the dump to abort.

Query (0x7f1668010ad0): Insert Into AuthenticationLog (UserIndex,UserName,NasIp,NasPort,RequestId,RadiusCode,RejectReason,PortMessage,TimeStamp,RadiusServerName,AlepoSessionId,PasswordTyped,CallerId,CalledStationId,NasPortType,ErrorCode,CustomField1,CustomField2,customfield3,customfield4,customfield5,customnumfield1,customnumfield2,customnumfield3,customnumfield4,customnumfield5) Values(0,'mynet',3414627184,553649637,155,3,'USER_NOT_FOUND','Invalid User','2024-12-14 05:41:48','AAA4-ILB','HWI-HYD08213215090000043b6a2AAAJuAmynet','','c8:22:02:2e:c4:ff','',13,29,' - ',' - ','','','',0,0,0,0,0)

Connection ID (thread ID): 62154962

Status: NOT_KILLED

Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=on,condition_pushdown_from_having=on,not_null_range_scan=off,hash_join_cardinality=off

The manual page at https://mariadb.com/kb/en/how-to-produce-a-full-stack-trace-for-mysqld/ contains

information that should help you find out what is causing the crash.

Writing a core file...

Working directory at /var/lib/mysql

Resource Limits:

Limit                     Soft Limit           Hard Limit           Units

Max cpu time              unlimited            unlimited            seconds

Max file size             unlimited            unlimited            bytes

Max data size             unlimited            unlimited            bytes

Max stack size            8388608              unlimited            bytes

Max core file size        unlimited            unlimited            bytes

Max resident set          unlimited            unlimited            bytes

Max processes             2061710              2061710              processes

Max open files            32768                32768                files

Max locked memory         65536                65536                bytes

Max address space         unlimited            unlimited            bytes

Max file locks            unlimited            unlimited            locks

Max pending signals       2061710              2061710              signals

Max msgqueue size         819200               819200               bytes

Max nice priority         0                    0

Max realtime priority     0                    0

Max realtime timeout      unlimited            unlimited            us

Core pattern: /var/coredumps/core.%e.%p

Kernel version: Linux version 4.18.0-477.13.1.el8_8.x86_64 (mockbuild@x86-vm-08.build.eng.bos.redhat.com) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-18) (GCC)) #1 SMP Thu May 18 10:27:05 EDT 2023

2024-12-14  5:51:49 0 [Note] Starting MariaDB 10.11.5-MariaDB-log source revision 7875294b6b74b53dd3aaa723e6cc103d2bb47b2c as process 685235

2024-12-14  5:51:49 0 [Note] InnoDB: Compressed tables use zlib 1.2.11

2024-12-14  5:51:49 0 [Note] InnoDB: Number of transaction pools: 1

2024-12-14  5:51:49 0 [Note] InnoDB: Using crc32 + pclmulqdq instructions

2024-12-14  5:51:49 0 [Note] InnoDB: Using Linux native AIO

2024-12-14  5:51:49 0 [Note] InnoDB: Initializing buffer pool, total size = 150.000GiB, chunk size = 2.344GiB

2024-12-14  5:51:50 0 [Note] InnoDB: Completed initialization of buffer pool

2024-12-14  5:51:50 0 [Note] InnoDB: File system buffers for log disabled (block size=512 bytes)

2024-12-14  5:51:52 0 [Warning] InnoDB: 3663622144 bytes should have been read at 631345152 from (unknown file), but got only 2147479552. Retrying.

2024-12-14  5:51:53 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=46858414541845

2024-12-14  5:52:05 0 [Note] InnoDB: Parsed redo log up to LSN=46859496053550; to recover: 143915 pages

2024-12-14  5:52:20 0 [Note] InnoDB: Parsed redo log up to LSN=46860701570417; to recover: 174721 pages

2024-12-14  5:52:28 0 [Note] InnoDB: End of log at LSN=46861442798088

2024-12-14  5:52:30 0 [Note] InnoDB: 1 transaction(s) which must be rolled back or cleaned up in total 1 row operations to undo

2024-12-14  5:52:30 0 [Note] InnoDB: Trx id counter is 126414099237

2024-12-14  5:52:31 0 [Note] InnoDB: To recover: 196275 pages

2024-12-14  5:52:45 0 [Note] InnoDB: Last binlog file './db2-binary.284069', position 188882048

2024-12-14  5:52:45 0 [Note] InnoDB: 128 rollback segments are active.

2024-12-14  5:52:46 0 [Note] InnoDB: Starting in background the rollback of recovered transactions

2024-12-14  5:52:46 0 [Note] InnoDB: To roll back: 1 transactions, 1 rows

2024-12-14  5:52:46 0 [Note] InnoDB: Rolled back recovered transaction 126414021942

2024-12-14  5:52:46 0 [Note] InnoDB: Rollback of non-prepared transactions completed

2024-12-14  5:52:46 0 [Note] InnoDB: Removed temporary tablespace data file: "./ibtmp1"

2024-12-14  5:52:46 0 [Note] InnoDB: Setting file './ibtmp1' size to 12.000MiB. Physically writing the file full; Please wait ...

2024-12-14  5:52:46 0 [Note] InnoDB: File './ibtmp1' size is now 12.000MiB.

2024-12-14  5:52:46 0 [Note] InnoDB: log sequence number 46861442798088; transaction id 126414099239

2024-12-14  5:52:46 0 [Note] InnoDB: Loading buffer pool(s) from /var/lib/mysql/ib_buffer_pool

2024-12-14  5:52:46 0 [Note] Plugin 'FEEDBACK' is disabled.

2024-12-14  5:52:46 0 [Warning] 'thread-concurrency' was removed. It does nothing now and exists only for compatibility with old my.cnf files.

2024-12-14  5:52:46 0 [Warning] 'innodb-thread-concurrency' was removed. It does nothing now and exists only for compatibility with old my.cnf files.

2024-12-14  5:52:46 0 [Warning] 'innodb-log-files-in-group' was removed. It does nothing now and exists only for compatibility with old my.cnf files.

2024-12-14  5:52:46 0 [Note] Recovering after a crash using db2-binary

2024-12-14  5:52:46 0 [Note] Starting table crash recovery...

2024-12-14  5:52:46 0 [Note] Crash table recovery finished.

2024-12-14  5:52:46 0 [Note] Server socket created on IP: '0.0.0.0'.

2024-12-14  5:52:46 0 [Note] Server socket created on IP: '::'.

2024-12-14  5:52:46 0 [Warning] 'user' entry '@alepo-db2-ilb' ignored in --skip-name-resolve mode.

2024-12-14  5:52:46 0 [Warning] 'user' entry 'root@localhost.localdomain' ignored in --skip-name-resolve mode.

2024-12-14  5:52:46 0 [Warning] 'proxies_priv' entry '@% root@alepo-db2-ilb' ignored in --skip-name-resolve mode.

2024-12-14  5:52:46 3 [Note] Reading Master_info: 'master-s@002ddb3.info'  Relay_info:'relay-log-s@002ddb3.info'

2024-12-14  5:52:46 3 [Note] Initialized Master_info from 'master-s@002ddb3.info'

2024-12-14  5:52:46 4 [Note] Master 's-db3': Slave I/O thread: Start asynchronous replication to master '@:3306' in log '' at position 4

2024-12-14  5:52:46 4 [ERROR] Master 's-db3': Slave I/O: Fatal error: Invalid (empty) username when attempting to connect to the master server. Connection attempt terminated. Internal MariaDB error code: 1593

2024-12-14  5:52:46 4 [Note] Master 's-db3': Slave I/O thread killed while connecting to master

2024-12-14  5:52:46 4 [Note] Master 's-db3': Slave I/O thread exiting, read up to log 'FIRST', position 4, master :3306

2024-12-14  5:52:46 3 [Note] Started replication for 's-db3'

2024-12-14  5:52:46 5 [Note] Master 's-db3': Slave SQL thread initialized, starting replication in log 'FIRST' at position 0, relay log './db2-relay-s@002ddb3.000031' position: 4

2024-12-14  5:52:46 3 [Note] Reading Master_info: 'master-s@002ddb4.info'  Relay_info:'relay-log-s@002ddb4.info'

2024-12-14  5:52:46 3 [Note] Initialized Master_info from 'master-s@002ddb4.info'

2024-12-14  5:52:46 18 [Note] Master 's-db4': Slave I/O thread: Start asynchronous replication to master '@:3306' in log '' at position 4

2024-12-14  5:52:46 18 [ERROR] Master 's-db4': Slave I/O: Fatal error: Invalid (empty) username when attempting to connect to the master server. Connection attempt terminated. Internal MariaDB error code: 1593

2024-12-14  5:52:46 18 [Note] Master 's-db4': Slave I/O thread killed while connecting to master

2024-12-14  5:52:46 18 [Note] Master 's-db4': Slave I/O thread exiting, read up to log 'FIRST', position 4, master :3306

2024-12-14  5:52:46 3 [Note] Started replication for 's-db4'

2024-12-14  5:52:46 3 [Note] Reading Master_info: 'master-s@002ddb1.info'  Relay_info:'relay-log-s@002ddb1.info'

2024-12-14  5:52:46 19 [Note] Master 's-db4': Slave SQL thread initialized, starting replication in log 'FIRST' at position 0, relay log './db2-relay-s@002ddb4.000023' position: 4

2024-12-14  5:52:46 3 [Note] Initialized Master_info from 'master-s@002ddb1.info'

2024-12-14  5:52:46 20 [Note] Master 's-db1': Slave I/O thread: Start asynchronous replication to master '@:3306' in log '' at position 4

2024-12-14  5:52:46 20 [ERROR] Master 's-db1': Slave I/O: Fatal error: Invalid (empty) username when attempting to connect to the master server. Connection attempt terminated. Internal MariaDB error code: 1593

2024-12-14  5:52:46 20 [Note] Master 's-db1': Slave I/O thread killed while connecting to master

2024-12-14  5:52:46 20 [Note] Master 's-db1': Slave I/O thread exiting, read up to log 'FIRST', position 4, master :3306

2024-12-14  5:52:46 3 [Note] Started replication for 's-db1'

2024-12-14  5:52:46 21 [Note] Master 's-db1': Slave SQL thread initialized, starting replication in log 'FIRST' at position 0, relay log './db2-relay-s@002ddb1.000030' position: 4

2024-12-14  5:52:46 3 [Note] Reading Master_info: 'master-s@002ddb5.info'  Relay_info:'relay-log-s@002ddb5.info'

2024-12-14  5:52:46 3 [Note] Initialized Master_info from 'master-s@002ddb5.info'

2024-12-14  5:52:46 22 [Note] Master 's-db5': Slave I/O thread: Start asynchronous replication to master '@:3306' in log '' at position 4

2024-12-14  5:52:46 22 [ERROR] Master 's-db5': Slave I/O: Fatal error: Invalid (empty) username when attempting to connect to the master server. Connection attempt terminated. Internal MariaDB error code: 1593

2024-12-14  5:52:46 22 [Note] Master 's-db5': Slave I/O thread killed while connecting to master

2024-12-14  5:52:46 22 [Note] Master 's-db5': Slave I/O thread exiting, read up to log 'FIRST', position 4, master :3306

2024-12-14  5:52:46 3 [Note] Started replication for 's-db5'

2024-12-14  5:52:46 23 [Note] Master 's-db5': Slave SQL thread initialized, starting replication in log 'FIRST' at position 0, relay log './db2-relay-s@002ddb5.000023' position: 4

2024-12-14  5:52:46 25 [Note] Slave I/O thread: Start asynchronous replication to master 'gtid@10.76.24.71:3306' in log 'db1-binary.283405' at position 46576242

2024-12-14  5:52:46 0 [Note] /usr/sbin/mariadbd: ready for connections.

Version: '10.11.5-MariaDB-log'  socket: '/var/lib/mysql/mysql.sock'  port: 3306  MariaDB Server

2024-12-14  5:52:46 26 [Note] Slave SQL thread initialized, starting replication in log 'db1-binary.283405' at position 46576242, relay log './db2-relay.368740' position: 767

2024-12-14  5:52:46 25 [Note] Slave I/O thread: connected to master 'gtid@10.76.24.71:3306',replication started in log 'db1-binary.283405' at position 46576242

2024-12-14  5:52:49 28 [Note] Start binlog_dump to slave_server(1), pos(db2-binary.284069, 189004400), using_gtid(0), gtid('')

2024-12-14  5:52:56 0 [Note] InnoDB: Buffer pool(s) load completed at 241214  5:52:56