/var/log/messages: Nov 10 05:33:28 ALEPO-DB1-ILB NetworkManager[1281]: [1731198808.9990] device (enp9s0): Activation: starting connection 'opt-net' (38c00c03-2f91-40f7-a4bb-1a9c28006b7b) Nov 10 05:33:28 ALEPO-DB1-ILB NetworkManager[1281]: [1731198808.9991] device (enp9s0): state change: disconnected -> prepare (reason 'none', sys-iface-state: 'managed') Nov 10 05:33:28 ALEPO-DB1-ILB NetworkManager[1281]: [1731198808.9992] device (enp9s0): state change: prepare -> config (reason 'none', sys-iface-state: 'managed') Nov 10 05:33:29 ALEPO-DB1-ILB NetworkManager[1281]: [1731198809.0047] device (enp9s0): state change: config -> ip-config (reason 'none', sys-iface-state: 'managed') Nov 10 05:33:29 ALEPO-DB1-ILB NetworkManager[1281]: [1731198809.0050] dhcp4 (enp9s0): activation: beginning transaction (timeout in 45 seconds) Nov 10 05:33:34 ALEPO-DB1-ILB systemd[1]: mariadb.service: Main process exited, code=killed, status=11/SEGV Nov 10 05:33:34 ALEPO-DB1-ILB systemd[1]: mariadb.service: Failed with result 'signal'. Nov 10 05:33:39 ALEPO-DB1-ILB systemd[1]: mariadb.service: Service RestartSec=5s expired, scheduling restart. Nov 10 05:33:39 ALEPO-DB1-ILB systemd[1]: mariadb.service: Scheduled restart job, restart counter is at 3. Nov 10 05:33:39 ALEPO-DB1-ILB systemd[1]: Stopped MariaDB 10.11.5 database server. Nov 10 05:33:39 ALEPO-DB1-ILB systemd[1]: Starting MariaDB 10.11.5 database server... Nov 10 05:33:40 ALEPO-DB1-ILB mariadbd[484800]: 2024-11-10 5:33:40 0 [Warning] Could not increase number of max_open_files to more than 32768 (request: 65535) Nov 10 05:34:11 ALEPO-DB1-ILB systemd[1]: Started MariaDB 10.11.5 database server. Nov 10 05:34:13 ALEPO-DB1-ILB NetworkManager[1281]: [1731198853.9165] device (enp9s0): state change: ip-config -> failed (reason 'ip-config-unavailable', sys-iface-state: 'managed') Nov 10 05:34:13 ALEPO-DB1-ILB NetworkManager[1281]: [1731198853.9169] device (enp9s0): Activation: failed for connection 'opt-net' Nov 10 05:34:13 ALEPO-DB1-ILB NetworkManager[1281]: [1731198853.9170] device (enp9s0): state change: failed -> disconnected (reason 'none', sys-iface-state: 'managed') Nov 10 05:34:13 ALEPO-DB1-ILB NetworkManager[1281]: [1731198853.9260] dhcp4 (enp9s0): canceled DHCP transaction Nov 10 05:34:13 ALEPO-DB1-ILB NetworkManager[1281]: [1731198853.9267] policy: auto-activating connection 'opt-net' (38c00c03-2f91-40f7-a4bb-1a9c28006b7b) Nov 10 05:34:13 ALEPO-DB1-ILB NetworkManager[1281]: [1731198853.9270] device (enp9s0): Activation: starting connection 'opt-net' (38c00c03-2f91-40f7-a4bb-1a9c28006b7b) @@@ /var/lib/mysql/mysqld.log 241110 5:29:51 [ERROR] mysqld got signal 11 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. To report this bug, see https://mariadb.com/kb/en/reporting-bugs We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. Server version: 10.11.5-MariaDB-log source revision: 7875294b6b74b53dd3aaa723e6cc103d2bb47b2c key_buffer_size=67108864 read_buffer_size=131072 max_used_connections=459 max_threads=802 thread_count=468 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 1831964 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. Thread pointer: 0x7f4db8018fb8 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0x7f4fd9139b98 thread_stack 0x49000 /usr/sbin/mariadbd(my_print_stacktrace+0x2e)[0x558dd648f12e] mysys/stacktrace.c:216(my_print_stacktrace)[0x558dd5f885b5] /lib64/libpthread.so.0(+0x12cf0)[0x7f78b8682cf0] sql/multi_range_read.cc:1383(DsMrr_impl::close_second_handler())[0x558dd5e8e6b0] sql/multi_range_read.cc:1398(DsMrr_impl::dsmrr_close())[0x558dd5e8e7c9] handler/ha_innodb.cc:15770(wsrep_notify_status(wsrep::server_state::state, wsrep::view const*))[0x558dd62323e7] sql/ha_partition.cc:9520(MyCTX_nopad::finish(unsigned char*, unsigned int*))[0x558dd6178481] sql/sql_base.cc:1022(close_thread_table(THD*, TABLE**))[0x558dd5cd67d1] sql/sql_base.cc:965(close_thread_tables(THD*))[0x558dd5cd6aeb] sql/sql_parse.cc:6097(mysql_execute_command(THD*, bool))[0x558dd5d46d7f] sql/sql_parse.cc:8035(mysql_parse(THD*, char*, unsigned int, Parser_state*))[0x558dd5d3ab87] sql/sql_parse.cc:1896(dispatch_command(enum_server_command, THD*, char*, unsigned int, bool))[0x558dd5d446e6] sql/sql_parse.cc:1407(do_command(THD*, bool))[0x558dd5d45dab] sql/sql_connect.cc:1416(do_handle_one_connection(CONNECT*, bool))[0x558dd5e59977] sql/sql_connect.cc:1318(handle_one_connection)[0x558dd5e59cbd] perfschema/pfs.cc:2204(MyCTX_nopad::finish(unsigned char*, unsigned int*))[0x558dd618fc9d] /lib64/libpthread.so.0(+0x81ca)[0x7f78b86781ca] /lib64/libc.so.6(clone+0x43)[0x7f78b79c8e73] Trying to get some variables. Some pointers may be invalid and cause the dump to abort. Query (0x7f4db8028e30): Insert Into AuthenticationLog (UserIndex,UserName,NasIp,NasPort,RequestId,RadiusCode,RejectReason,PortMessage,TimeStamp,RadiusServerName,AlepoSessionId,PasswordTyped,CallerId,CalledStationId,NasPortType,ErrorCode,CustomField1,CustomField2,CustomField3,CustomField5,customfield4,customnumfield1,customnumfield2,customnumfield3,customnumfield4,customnumfield5) Values(51883661,'03007915007GG',3414627232,339741342,132,2,'USER REJECTED - MAC NOT MATCHED: NAS-ERROR-CODE=0, MAC-ERROR-CODE=22','Access Accept','2024-11-10 05:29:49','AAA1-ILB','HWI-FSD081172718000008ba0e8AAAlnZ03007915007GG','','34:36:54:dd:8c:08','',13,10022,'10.139.4.46 - 0/9/30/0',' - ','34:36:54:dd:8c:07','10.139.4.46-FT-FSD-C11Am-H-M atm 0/9/0/30:0.103','NULL',1,2,0,0,0) 2024-11-10 5:34:11 0 [Note] Crash table recovery finished. 2024-11-10 5:34:11 0 [Note] Server socket created on IP: '0.0.0.0'. 2024-11-10 5:34:11 0 [Note] Server socket created on IP: '::'. 2024-11-10 5:34:11 0 [Warning] 'user' entry '@alepo-db2-ilb' ignored in --skip-name-resolve mode. 2024-11-10 5:34:11 0 [Warning] 'user' entry 'root@localhost.localdomain' ignored in --skip-name-resolve mode. 2024-11-10 5:34:11 0 [Warning] 'proxies_priv' entry '@% root@alepo-db2-ilb' ignored in --skip-name-resolve mode. 2024-11-10 5:34:11 4 [Note] Reading Master_info: 'master-s@002ddb3.info' Relay_info:'relay-log-s@002ddb3.info' 2024-11-10 5:34:11 4 [Note] Initialized Master_info from 'master-s@002ddb3.info' 2024-11-10 5:34:11 5 [Note] Master 's-db3': Slave I/O thread: Start asynchronous replication to master '@:3306' in log '' at position 4 2024-11-10 5:34:11 5 [ERROR] Master 's-db3': Slave I/O: Fatal error: Invalid (empty) username when attempting to connect to the master server. Connection attempt terminated. Internal MariaDB error code: 1593 2024-11-10 5:34:11 5 [Note] Master 's-db3': Slave I/O thread killed while connecting to master 2024-11-10 5:34:11 5 [Note] Master 's-db3': Slave I/O thread exiting, read up to log 'FIRST', position 4, master :3306 2024-11-10 5:34:11 4 [Note] Started replication for 's-db3' 2024-11-10 5:34:11 6 [Note] Master 's-db3': Slave SQL thread initialized, starting replication in log 'FIRST' at position 0, relay log './db1-relay-s@002ddb3.000040' position: 4 2024-11-10 5:34:11 4 [Note] Reading Master_info: 'master-s@002ddb4.info' Relay_info:'relay-log-s@002ddb4.info' 2024-11-10 5:34:11 4 [Note] Initialized Master_info from 'master-s@002ddb4.info' 2024-11-10 5:34:11 19 [Note] Master 's-db4': Slave I/O thread: Start asynchronous replication to master 'gtid@10.76.25.11:3306' in log 'db4-binary.236723' at position 157701627 2024-11-10 5:34:11 4 [Note] Started replication for 's-db4' 2024-11-10 5:34:11 4 [Note] Reading Master_info: 'master-s@002ddb2.info' Relay_info:'relay-log-s@002ddb2.info' 2024-11-10 5:34:11 4 [Note] Reading Master_info: 'master-s@002ddb4.info' Relay_info:'relay-log-s@002ddb4.info' 2024-11-10 5:34:11 4 [Note] Initialized Master_info from 'master-s@002ddb4.info' 2024-11-10 5:34:11 19 [Note] Master 's-db4': Slave I/O thread: Start asynchronous replication to master 'gtid@10.76.25.11:3306' in log 'db4-binary.236723' at position 157701627 2024-11-10 5:34:11 4 [Note] Started replication for 's-db4' 2024-11-10 5:34:11 4 [Note] Reading Master_info: 'master-s@002ddb2.info' Relay_info:'relay-log-s@002ddb2.info' 2024-11-10 5:34:11 20 [Note] Master 's-db4': Slave SQL thread initialized, starting replication in log 'db4-binary.236723' at position 157701627, relay log './db1-relay-s@002ddb4.171946' position: 783 2024-11-10 5:34:11 4 [Note] Initialized Master_info from 'master-s@002ddb2.info' 2024-11-10 5:34:11 21 [Note] Master 's-db2': Slave I/O thread: Start asynchronous replication to master '@:3306' in log '' at position 4 2024-11-10 5:34:11 21 [ERROR] Master 's-db2': Slave I/O: Fatal error: Invalid (empty) username when attempting to connect to the master server. Connection attempt terminated. Internal MariaDB error code: 1593 2024-11-10 5:34:11 21 [Note] Master 's-db2': Slave I/O thread killed while connecting to master 2024-11-10 5:34:11 21 [Note] Master 's-db2': Slave I/O thread exiting, read up to log 'FIRST', position 4, master :3306 2024-11-10 5:34:11 4 [Note] Started replication for 's-db2' 2024-11-10 5:34:11 4 [Note] Reading Master_info: 'master-s@002ddb5.info' Relay_info:'relay-log-s@002ddb5.info' 2024-11-10 5:34:11 22 [Note] Master 's-db2': Slave SQL thread initialized, starting replication in log 'FIRST' at position 0, relay log './db1-relay-s@002ddb2.000039' position: 4 2024-11-10 5:34:11 4 [Note] Initialized Master_info from 'master-s@002ddb5.info' 2024-11-10 5:34:11 23 [Note] Master 's-db5': Slave I/O thread: Start asynchronous replication to master '@:3306' in log '' at position 4 2024-11-10 5:34:11 23 [ERROR] Master 's-db5': Slave I/O: Fatal error: Invalid (empty) username when attempting to connect to the master server. Connection attempt terminated. Internal MariaDB error code: 1593 2024-11-10 5:34:11 23 [Note] Master 's-db5': Slave I/O thread killed while connecting to master 2024-11-10 5:34:11 23 [Note] Master 's-db5': Slave I/O thread exiting, read up to log 'FIRST', position 4, master :3306 2024-11-10 5:34:11 4 [Note] Started replication for 's-db5' 2024-11-10 5:34:11 24 [Note] Master 's-db5': Slave SQL thread initialized, starting replication in log 'FIRST' at position 0, relay log './db1-relay-s@002ddb5.000031' position: 4 MMM_Log /var/log/mysql-mmm/ mmm_mond.log 2024/11/10 05:30:22 WARN Check 'rep_threads' on 'db1' is in unknown state! Message: UNKNOWN: Connect error (host = 10.76.24.71:3306, user = mmm_monitor)! Lost connection to server at 'handshake: reading initial communication packet', system error: 110 2024/11/10 05:30:23 WARN Check 'rep_backlog' on 'db1' is in unknown state! Message: UNKNOWN: Connect error (host = 10.76.24.71:3306, user = mmm_monitor)! Lost connection to server at 'handshake: reading initial communication packet', system error: 110 2024/11/10 05:30:39 ERROR Check 'mysql' on 'db1' has failed for 14 seconds! Message: ERROR: Connect error (host = 10.76.24.71:3306, user = mmm_monitor)! Lost connection to server at 'handshake: reading initial communication packet', system error: 110 2024/11/10 05:30:42 FATAL State of host 'db1' changed from ONLINE to HARD_OFFLINE (ping: OK, mysql: not OK) 2024/11/10 05:30:42 INFO Removing all roles from host 'db1': 2024/11/10 05:30:42 INFO Removed role 'writer(10.76.24.100)' from host 'db1' 2024/11/10 05:30:45 INFO Orphaned role 'writer(10.76.24.100)' has been assigned to 'db2' 2024/11/10 05:34:11 INFO Check 'rep_threads' on 'db1' is ok! 2024/11/10 05:34:13 INFO Check 'rep_backlog' on 'db1' is ok! 2024/11/10 05:34:14 INFO Check 'mysql' on 'db1' is ok! 2024/11/10 05:34:16 FATAL State of host 'db1' changed from HARD_OFFLINE to AWAITING_RECOVERY 2024/11/10 05:35:16 FATAL State of host 'db1' changed from AWAITING_RECOVERY to ONLINE because of auto_set_online(60 seconds). It was in state AWAITING_RECOVERY for 60 seconds