[MDEV-35396] DB1 again got crashed with Signal 11 (Segmentation Fault) Error - Jira

Details

Type: Bug
Status: Needs Feedback (View Workflow)
Priority: Major
Resolution: Unresolved
Affects Version/s: 10.11.5
Fix Version/s: None
Component/s: Parser
Labels:
- crash

Description

Hello Team,

We have again observed DB1 crash with same error as Singal 11 Segmentation Fault.

Please check this issue and provide solution.

Please refer the below logs.

/var/log/messages:

Nov 10 05:33:28 ALEPO-DB1-ILB NetworkManager[1281]: <info> [1731198808.9990] device (enp9s0): Activation: starting connection 'opt-net' (38c00c03-2f91-40f7-a4bb-1a9c28006b7b)
Nov 10 05:33:28 ALEPO-DB1-ILB NetworkManager[1281]: <info> [1731198808.9991] device (enp9s0): state change: disconnected -> prepare (reason 'none', sys-iface-state: 'managed')
Nov 10 05:33:28 ALEPO-DB1-ILB NetworkManager[1281]: <info> [1731198808.9992] device (enp9s0): state change: prepare -> config (reason 'none', sys-iface-state: 'managed')
Nov 10 05:33:29 ALEPO-DB1-ILB NetworkManager[1281]: <info> [1731198809.0047] device (enp9s0): state change: config -> ip-config (reason 'none', sys-iface-state: 'managed')
Nov 10 05:33:29 ALEPO-DB1-ILB NetworkManager[1281]: <info> [1731198809.0050] dhcp4 (enp9s0): activation: beginning transaction (timeout in 45 seconds)
Nov 10 05:33:34 ALEPO-DB1-ILB systemd[1]: mariadb.service: Main process exited, code=killed, status=11/SEGV
Nov 10 05:33:34 ALEPO-DB1-ILB systemd[1]: mariadb.service: Failed with result 'signal'.
Nov 10 05:33:39 ALEPO-DB1-ILB systemd[1]: mariadb.service: Service RestartSec=5s expired, scheduling restart.
Nov 10 05:33:39 ALEPO-DB1-ILB systemd[1]: mariadb.service: Scheduled restart job, restart counter is at 3.
Nov 10 05:33:39 ALEPO-DB1-ILB systemd[1]: Stopped MariaDB 10.11.5 database server.
Nov 10 05:33:39 ALEPO-DB1-ILB systemd[1]: Starting MariaDB 10.11.5 database server...
Nov 10 05:33:40 ALEPO-DB1-ILB mariadbd[484800]: 2024-11-10 5:33:40 0 [Warning] Could not increase number of max_open_files to more than 32768 (request: 65535)
Nov 10 05:34:11 ALEPO-DB1-ILB systemd[1]: Started MariaDB 10.11.5 database server.
Nov 10 05:34:13 ALEPO-DB1-ILB NetworkManager[1281]: <info> [1731198853.9165] device (enp9s0): state change: ip-config -> failed (reason 'ip-config-unavailable', sys-iface-state: 'managed')
Nov 10 05:34:13 ALEPO-DB1-ILB NetworkManager[1281]: <warn> [1731198853.9169] device (enp9s0): Activation: failed for connection 'opt-net'
Nov 10 05:34:13 ALEPO-DB1-ILB NetworkManager[1281]: <info> [1731198853.9170] device (enp9s0): state change: failed -> disconnected (reason 'none', sys-iface-state: 'managed')
Nov 10 05:34:13 ALEPO-DB1-ILB NetworkManager[1281]: <info> [1731198853.9260] dhcp4 (enp9s0): canceled DHCP transaction
Nov 10 05:34:13 ALEPO-DB1-ILB NetworkManager[1281]: <info> [1731198853.9267] policy: auto-activating connection 'opt-net' (38c00c03-2f91-40f7-a4bb-1a9c28006b7b)
Nov 10 05:34:13 ALEPO-DB1-ILB NetworkManager[1281]: <info> [1731198853.9270] device (enp9s0): Activation: starting connection 'opt-net' (38c00c03-2f91-40f7-a4bb-1a9c28006b7b)
@@@

/var/lib/mysql/mysqld.log

241110 5:29:51 [ERROR] mysqld got signal 11 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.

To report this bug, see MariaDB Community Bug Reports and Feature Requests

We will try our best to scrape up some info that will hopefully help
diagnose the problem, but since we have already crashed,
something is definitely wrong and this may fail.

Server version: 10.11.5-MariaDB-log source revision: 7875294b6b74b53dd3aaa723e6cc103d2bb47b2c
key_buffer_size=67108864
read_buffer_size=131072
max_used_connections=459
max_threads=802
thread_count=468
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 1831964 K bytes of memory
Hope that's ok; if not, decrease some variables in the equation.

Thread pointer: 0x7f4db8018fb8
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 0x7f4fd9139b98 thread_stack 0x49000
/usr/sbin/mariadbd(my_print_stacktrace+0x2e)[0x558dd648f12e]
mysys/stacktrace.c:216(my_print_stacktrace)[0x558dd5f885b5]
/lib64/libpthread.so.0(+0x12cf0)[0x7f78b8682cf0]
sql/multi_range_read.cc:1383(DsMrr_impl::close_second_handler())[0x558dd5e8e6b0]
sql/multi_range_read.cc:1398(DsMrr_impl::dsmrr_close())[0x558dd5e8e7c9]
handler/ha_innodb.cc:15770(wsrep_notify_status(wsrep::server_state::state, wsrep::view const*))[0x558dd62323e7]
sql/ha_partition.cc:9520(MyCTX_nopad::finish(unsigned char*, unsigned int*))[0x558dd6178481]
sql/sql_base.cc:1022(close_thread_table(THD*, TABLE**))[0x558dd5cd67d1]
sql/sql_base.cc:965(close_thread_tables(THD*))[0x558dd5cd6aeb]
sql/sql_parse.cc:6097(mysql_execute_command(THD*, bool))[0x558dd5d46d7f]
sql/sql_parse.cc:8035(mysql_parse(THD*, char*, unsigned int, Parser_state*))[0x558dd5d3ab87]
sql/sql_parse.cc:1896(dispatch_command(enum_server_command, THD*, char*, unsigned int, bool))[0x558dd5d446e6]
sql/sql_parse.cc:1407(do_command(THD*, bool))[0x558dd5d45dab]
sql/sql_connect.cc:1416(do_handle_one_connection(CONNECT*, bool))[0x558dd5e59977]
sql/sql_connect.cc:1318(handle_one_connection)[0x558dd5e59cbd]
perfschema/pfs.cc:2204(MyCTX_nopad::finish(unsigned char*, unsigned int*))[0x558dd618fc9d]
/lib64/libpthread.so.0(+0x81ca)[0x7f78b86781ca]
/lib64/libc.so.6(clone+0x43)[0x7f78b79c8e73]

Trying to get some variables.
Some pointers may be invalid and cause the dump to abort.
Query (0x7f4db8028e30): Insert Into AuthenticationLog (UserIndex,UserName,NasIp,NasPort,RequestId,RadiusCode,RejectReason,PortMessage,TimeStamp,RadiusServerName,AlepoSessionId,PasswordTyped,CallerId,CalledStationId,NasPortType,ErrorCode,CustomField1,CustomField2,CustomField3,CustomField5,customfield4,customnumfield1,customnumfield2,customnumfield3,customnumfield4,customnumfield5) Values(51883661,'03007915007GG',3414627232,339741342,132,2,'USER REJECTED - MAC NOT MATCHED: NAS-ERROR-CODE=0, MAC-ERROR-CODE=22','Access Accept','2024-11-10 05:29:49','AAA1-ILB','HWI-FSD081172718000008ba0e8AAAlnZ03007915007GG','','34:36:54:dd:8c:08','',13,10022,'10.139.4.46 - 0/9/30/0',' - ','34:36:54:dd:8c:07','10.139.4.46-FT-FSD-C11Am-H-M atm 0/9/0/30:0.103','NULL',1,2,0,0,0)

2024-11-10 5:34:11 0 [Note] Crash table recovery finished.
2024-11-10 5:34:11 0 [Note] Server socket created on IP: '0.0.0.0'.
2024-11-10 5:34:11 0 [Note] Server socket created on IP: '::'.
2024-11-10 5:34:11 0 [Warning] 'user' entry '@alepo-db2-ilb' ignored in --skip-name-resolve mode.
2024-11-10 5:34:11 0 [Warning] 'user' entry 'root@localhost.localdomain' ignored in --skip-name-resolve mode.
2024-11-10 5:34:11 0 [Warning] 'proxies_priv' entry '@% root@alepo-db2-ilb' ignored in --skip-name-resolve mode.
2024-11-10 5:34:11 4 [Note] Reading Master_info: 'master-s@002ddb3.info' Relay_info:'relay-log-s@002ddb3.info'
2024-11-10 5:34:11 4 [Note] Initialized Master_info from 'master-s@002ddb3.info'
2024-11-10 5:34:11 5 [Note] Master 's-db3': Slave I/O thread: Start asynchronous replication to master '@:3306' in log '' at position 4
2024-11-10 5:34:11 5 [ERROR] Master 's-db3': Slave I/O: Fatal error: Invalid (empty) username when attempting to connect to the master server. Connection attempt terminated. Internal MariaDB error code: 1593
2024-11-10 5:34:11 5 [Note] Master 's-db3': Slave I/O thread killed while connecting to master
2024-11-10 5:34:11 5 [Note] Master 's-db3': Slave I/O thread exiting, read up to log 'FIRST', position 4, master :3306
2024-11-10 5:34:11 4 [Note] Started replication for 's-db3'
2024-11-10 5:34:11 6 [Note] Master 's-db3': Slave SQL thread initialized, starting replication in log 'FIRST' at position 0, relay log './db1-relay-s@002ddb3.000040' position: 4
2024-11-10 5:34:11 4 [Note] Reading Master_info: 'master-s@002ddb4.info' Relay_info:'relay-log-s@002ddb4.info'
2024-11-10 5:34:11 4 [Note] Initialized Master_info from 'master-s@002ddb4.info'
2024-11-10 5:34:11 19 [Note] Master 's-db4': Slave I/O thread: Start asynchronous replication to master 'gtid@10.76.25.11:3306' in log 'db4-binary.236723' at position 157701627
2024-11-10 5:34:11 4 [Note] Started replication for 's-db4'
2024-11-10 5:34:11 4 [Note] Reading Master_info: 'master-s@002ddb2.info' Relay_info:'relay-log-s@002ddb2.info'

2024-11-10 5:34:11 4 [Note] Reading Master_info: 'master-s@002ddb4.info' Relay_info:'relay-log-s@002ddb4.info'
2024-11-10 5:34:11 4 [Note] Initialized Master_info from 'master-s@002ddb4.info'
2024-11-10 5:34:11 19 [Note] Master 's-db4': Slave I/O thread: Start asynchronous replication to master 'gtid@10.76.25.11:3306' in log 'db4-binary.236723' at position 157701627
2024-11-10 5:34:11 4 [Note] Started replication for 's-db4'
2024-11-10 5:34:11 4 [Note] Reading Master_info: 'master-s@002ddb2.info' Relay_info:'relay-log-s@002ddb2.info'
2024-11-10 5:34:11 20 [Note] Master 's-db4': Slave SQL thread initialized, starting replication in log 'db4-binary.236723' at position 157701627, relay log './db1-relay-s@002ddb4.171946' position: 783
2024-11-10 5:34:11 4 [Note] Initialized Master_info from 'master-s@002ddb2.info'
2024-11-10 5:34:11 21 [Note] Master 's-db2': Slave I/O thread: Start asynchronous replication to master '@:3306' in log '' at position 4
2024-11-10 5:34:11 21 [ERROR] Master 's-db2': Slave I/O: Fatal error: Invalid (empty) username when attempting to connect to the master server. Connection attempt terminated. Internal MariaDB error code: 1593
2024-11-10 5:34:11 21 [Note] Master 's-db2': Slave I/O thread killed while connecting to master
2024-11-10 5:34:11 21 [Note] Master 's-db2': Slave I/O thread exiting, read up to log 'FIRST', position 4, master :3306
2024-11-10 5:34:11 4 [Note] Started replication for 's-db2'
2024-11-10 5:34:11 4 [Note] Reading Master_info: 'master-s@002ddb5.info' Relay_info:'relay-log-s@002ddb5.info'
2024-11-10 5:34:11 22 [Note] Master 's-db2': Slave SQL thread initialized, starting replication in log 'FIRST' at position 0, relay log './db1-relay-s@002ddb2.000039' position: 4
2024-11-10 5:34:11 4 [Note] Initialized Master_info from 'master-s@002ddb5.info'
2024-11-10 5:34:11 23 [Note] Master 's-db5': Slave I/O thread: Start asynchronous replication to master '@:3306' in log '' at position 4
2024-11-10 5:34:11 23 [ERROR] Master 's-db5': Slave I/O: Fatal error: Invalid (empty) username when attempting to connect to the master server. Connection attempt terminated. Internal MariaDB error code: 1593
2024-11-10 5:34:11 23 [Note] Master 's-db5': Slave I/O thread killed while connecting to master
2024-11-10 5:34:11 23 [Note] Master 's-db5': Slave I/O thread exiting, read up to log 'FIRST', position 4, master :3306
2024-11-10 5:34:11 4 [Note] Started replication for 's-db5'
2024-11-10 5:34:11 24 [Note] Master 's-db5': Slave SQL thread initialized, starting replication in log 'FIRST' at position 0, relay log './db1-relay-s@002ddb5.000031' position: 4

MMM_Log

/var/log/mysql-mmm/ mmm_mond.log

2024/11/10 05:30:22 WARN Check 'rep_threads' on 'db1' is in unknown state! Message: UNKNOWN: Connect error (host = 10.76.24.71:3306, user = mmm_monitor)! Lost connection to server at 'handshake: reading initial communication packet', system error: 110
2024/11/10 05:30:23 WARN Check 'rep_backlog' on 'db1' is in unknown state! Message: UNKNOWN: Connect error (host = 10.76.24.71:3306, user = mmm_monitor)! Lost connection to server at 'handshake: reading initial communication packet', system error: 110
2024/11/10 05:30:39 ERROR Check 'mysql' on 'db1' has failed for 14 seconds! Message: ERROR: Connect error (host = 10.76.24.71:3306, user = mmm_monitor)! Lost connection to server at 'handshake: reading initial communication packet', system error: 110
2024/11/10 05:30:42 FATAL State of host 'db1' changed from ONLINE to HARD_OFFLINE (ping: OK, mysql: not OK)
2024/11/10 05:30:42 INFO Removing all roles from host 'db1':
2024/11/10 05:30:42 INFO Removed role 'writer(10.76.24.100)' from host 'db1'
2024/11/10 05:30:45 INFO Orphaned role 'writer(10.76.24.100)' has been assigned to 'db2'
2024/11/10 05:34:11 INFO Check 'rep_threads' on 'db1' is ok!
2024/11/10 05:34:13 INFO Check 'rep_backlog' on 'db1' is ok!
2024/11/10 05:34:14 INFO Check 'mysql' on 'db1' is ok!
2024/11/10 05:34:16 FATAL State of host 'db1' changed from HARD_OFFLINE to AWAITING_RECOVERY
2024/11/10 05:35:16 FATAL State of host 'db1' changed from AWAITING_RECOVERY to ONLINE because of auto_set_online(60 seconds). It was in state AWAITING_RECOVERY for 60 seconds

Coredump: