Details
-
Bug
-
Status: Closed (View Workflow)
-
Major
-
Resolution: Incomplete
-
10.11.5
Description
Hello Team,
We have observed DB2 crash on 2nd OCT with Segmentation fault -Signal 11 error and VIP got shifted to DB1
As we are facing this issue frequently kindly provide the solution for this issue.
Kindly find below details for your reference
[root@ALEPO-DB2-ILB tmp]# systemctl status mysqld
|
● mariadb.service - MariaDB 10.11.5 database server
|
Loaded: loaded (/usr/lib/systemd/system/mariadb.service; enabled; vendor preset: disabled)
|
Drop-In: /etc/systemd/system/mariadb.service.d
|
└─migrated-from-my.cnf-settings.conf
|
Active: active (running) since Wed 2024-10-02 16:06:54 PKT; 6 days ago
|
Docs: man:mariadbd(8)
|
https://mariadb.com/kb/en/library/systemd/
|
Process: 7607 ExecStartPost=/bin/sh -c systemctl unset-environment _WSREP_START_POSITION (code=exited, status=0/SUCCES>
|
Process: 7256 ExecStartPre=/bin/sh -c [ ! -e /usr/bin/galera_recovery ] && VAR= || VAR=`cd /usr/bin/..; /usr/bin/gal>
|
Process: 7253 ExecStartPre=/bin/sh -c systemctl unset-environment _WSREP_START_POSITION (code=exited, status=0/SUCCESS)
|
Main PID: 7476 (mariadbd)
|
Status: "Taking your SQL requests now..."
|
Tasks: 486 (limit: 3298736)
|
Memory: 461.2G
|
CGroup: /system.slice/mariadb.service
|
└─7476 /usr/sbin/mariadbd
|
|
|
[root@ALEPO-DB2-ILB mysql]# mysql --version
|
mysql Ver 15.1 Distrib 10.11.5-MariaDB, for Linux (x86_64) using readline 5.1
|
var/log/messages
|
Oct 2 16:06:27 ALEPO-DB2-ILB systemd[1]: mariadb.service: Main process exited, code=killed, status=11/SEGV
|
Oct 2 16:06:27 ALEPO-DB2-ILB systemd[1]: mariadb.service: Failed with result 'signal'.
|
Oct 2 16:06:32 ALEPO-DB2-ILB systemd[1]: mariadb.service: Service RestartSec=5s expired, scheduling restart.
|
Oct 2 16:06:32 ALEPO-DB2-ILB systemd[1]: mariadb.service: Scheduled restart job, restart counter is at 1.
|
Oct 2 16:06:32 ALEPO-DB2-ILB systemd[1]: Stopped MariaDB 10.11.5 database server.
|
Oct 2 16:06:32 ALEPO-DB2-ILB systemd[1]: Starting MariaDB 10.11.5 database server...
|
Oct 2 16:06:33 ALEPO-DB2-ILB mariadbd[7476]: 2024-10-02 16:06:33 0 [Warning] Could not increase number of max_open_files to more than 32768 (request: 65535)
|
Oct 2 16:06:34 ALEPO-DB2-ILB NetworkManager[1266]: <info> [1727867194.0230] device (enp9s0): state change: ip-config -> failed (reason 'ip-config-unavailable', sys-iface-state: 'managed')
|
Oct 2 16:06:34 ALEPO-DB2-ILB NetworkManager[1266]: <warn> [1727867194.0235] device (enp9s0): Activation: failed for connection 'opt-net'
|
Oct 2 16:06:34 ALEPO-DB2-ILB NetworkManager[1266]: <info> [1727867194.0237] device (enp9s0): state change: failed -> disconnected (reason 'none', sys-iface-state: 'managed')
|
Oct 2 16:06:34 ALEPO-DB2-ILB NetworkManager[1266]: <info> [1727867194.0337] dhcp4 (enp9s0): canceled DHCP transaction
|
Oct 2 16:06:34 ALEPO-DB2-ILB NetworkManager[1266]: <info> [1727867194.0345] policy: auto-activating connection 'opt-net' (3cd38034-58b2-40cd-8cc7-5afcda959b31)
|
Oct 2 16:06:34 ALEPO-DB2-ILB NetworkManager[1266]: <info> [1727867194.0349] device (enp9s0): Activation: starting connection 'opt-net' (3cd38034-58b2-40cd-8cc7-5afcda959b31)
|
Kindly find attached mysql logs, gdb output and sar report.
Please let us know if you required any additional information to debug the issue.
Attachments
Issue Links
- relates to
-
-
- Closed
-
-
-
- Confirmed
-
Thread 1 (Thread 0x7f5d49166700 (LWP 1908990)):#0 0x00007f5d5efe7a35 in pthread_kill () from /lib64/libpthread.so.0No symbol table info available.#1 0x0000555f126d0f27 in my_write_core (sig=<optimized out>) at /usr/src/debug/MariaDB-/src_0/mysys/stacktrace.c:424No locals.#2 0x0000555f121ca5d0 in handle_fatal_signal (sig=11) at /usr/src/debug/MariaDB-/src_0/sql/signal_handler.cc:360curr_time = 1727867038tm = {tm_sec = 58, tm_min = 3, tm_hour = 16, tm_mday = 2, tm_mon = 9, tm_year = 124, tm_wday = 3, tm_yday = 275, tm_isdst = 0, tm_gmtoff = 18000, tm_zone = 0x555f13c91300 "PKT"}thd = 0x7f339d11fb48print_invalid_query_pointer = false#3 <signal handler called>No symbol table info available.#4 change_item_list_context (context=0x7f339c8dfeb0, list=0x7f339d124d28) at /usr/src/debug/MariaDB-/src_0/sql/sql_lex.cc:9951it = {<base_list_iterator> = {list = <optimized out>, el = 0x7f339c8de250, prev = <optimized out>, current = <optimized out>}, <No data fields>}item = 0x3c2480it = <optimized out>item = <optimized out>#5 LEX::insert_select_hack (this=0x7f339d123d28, sel=<optimized out>) at /usr/src/debug/MariaDB-/src_0/sql/sql_lex.cc:9989insert_table = <optimized out>sel = 0x7f339c8dfe68this = 0x7f339d123d28insert_table = <optimized out>sel = <optimized out>insert_table = <optimized out>sel = <optimized out>#6 0x0000555f11f6f6db in LEX::parsed_insert_select (this=0x7f339d123d28, first_select=0x7f339c8dfe68) at /usr/src/debug/MariaDB-/src_0/sql/sql_lex.cc:10304blt = <optimized out>#7 0x0000555f1216d149 in MYSQLparse (thd=<optimized out>) at /usr/src/debug/MariaDB-/src_0/sql/sql_lex.h:983yychar = 261...#8 0x0000555f11f8081a in parse_sql (thd=thd@entry=0x7f339d11fb48, parser_state=parser_state@entry=0x7f5d49165260, creation_ctx=creation_ctx@entry=0x0, do_pfs_digest=do_pfs_digest@entry=true) at /usr/src/debug/MariaDB-/src_0/sql/sql_parse.cc:10448ret_value = <optimized out>backup_ctx = 0x0mysql_parse_status = <optimized out>#9 0x0000555f11f7ca1a in mysql_parse (thd=0x7f339d11fb48, rawbuf=<optimized out>, length=581, parser_state=0x7f5d49165260) at /usr/src/debug/MariaDB-/src_0/sql/sql_parse.cc:7986lex = 0x7f339d123d28err = <optimized out>#10 0x0000555f11f866e6 in dispatch_command (command=COM_QUERY, thd=0x7f339d11fb48, packet=<optimized out>, packet_length=<optimized out>, blocking=<optimized out>) at /usr/src/debug/MariaDB-/src_0/sql/sql_class.h:1369packet_end = 0x7f339c8dd495 ""parser_state = {m_lip = {lookahead_token = -1, lookahead_yylval = 0x0, m_thd = 0x7f339d11fb48, m_ptr = 0x7f339c8dd496 "\005", m_tok_start = 0x7f339c8dd495 "", m_tok_end = 0x7f339c8dd495 "", m_end_of_query = 0x7f339c8dd495 "", m_tok_start_prev = 0x7f339c8dd494 ")", m_buf = 0x7f339c8dd250 "Insert Into AuthenticationLog (UserIndex,UserName,NasIp,NasPort,RequestId,RadiusCode,RejectReason,PortMessage,TimeStamp,RadiusServerName,AlepoSessionId,PasswordTyped,CallerId,CalledStationId,NasPortTy"..., m_buf_length = 581, m_echo = true, m_echo_saved = false, m_cpp_buf = 0x7f339c8dd4f0 "Insert Into AuthenticationLog (UserIndex,UserName,NasIp,NasPort,RequestId,RadiusCode,RejectReason,PortMessage,TimeStamp,RadiusServerName,AlepoSessionId,PasswordTyped,CallerId,CalledStationId,NasPortTy"..., m_cpp_ptr = 0x7f339c8dd735 "", m_cpp_tok_start = 0x7f339c8dd735 "", m_cpp_tok_start_prev = 0x7f339c8dd734 ")", m_cpp_tok_end = 0x7f339c8dd735 "", m_body_utf8 = 0x0, m_body_utf8_ptr = 0x0, m_cpp_utf8_processed_ptr = 0x0, next_state = MY_LEX_END, found_semicolon = 0x0, ignore_space = false, stmt_prepare_mode = false, multi_statements = true, yylineno = 1, m_digest = 0x0, in_comment = NO_COMMENT, in_comment_saved = NO_COMMENT, m_cpp_text_start = 0x7f339c8dd733 "0)", m_cpp_text_end = 0x7f339c8dd734 ")", m_underscore_cs = 0x0}, m_yacc = {yacc_yyss = 0x0, yacc_yyvs = 0x0, m_set_signal_info = {m_item = {0x0 <repeats 13 times>}}, m_lock_type = TL_READ_DEFAULT, m_mdl_type = MDL_SHARED_READ}, m_digest_psi = 0x0}net = 0x7f339d11fe00error = falsedo_end_of_statement = true__FUNCTION__ = "dispatch_command"#11 0x0000555f11f87dab in do_command (thd=0x7f339d11fb48, blocking=blocking@entry=true) at /usr/src/debug/MariaDB-/src_0/sql/sql_parse.cc:1407return_value = <optimized out>packet = 0x7f339d1265a8 "\003Insert Into AuthenticationLog (UserIndex,UserName,NasIp,NasPort,RequestId,RadiusCode,RejectReason,PortMessage,TimeStamp,RadiusServerName,AlepoSessionId,PasswordTyped,CallerId,CalledStationId,NasPortT"...packet_length = <optimized out>net = 0x7f339d11fe00command = COM_QUERY__FUNCTION__ = "do_command"#12 0x0000555f1209b977 in do_handle_one_connection (connect=<optimized out>, connect@entry=0x555f1588d018, put_in_cache=put_in_cache@entry=true) at /usr/src/debug/MariaDB-/src_0/sql/sql_connect.cc:1416create_user = truethr_create_utime = <optimized out>thd = 0x7f339d11fb48#13 0x0000555f1209bcbd in handle_one_connection (arg=arg@entry=0x555f1588d018) at /usr/src/debug/MariaDB-/src_0/sql/sql_connect.cc:1318connect = 0x555f1588d018#14 0x0000555f123d1c9d in pfs_spawn_thread (arg=0x555f1582b8d8) at /usr/src/debug/MariaDB-/src_0/storage/perfschema/pfs.cc:2201typed_arg = 0x555f1582b8d8user_arg = 0x555f1588d018user_start_routine = 0x555f1209bc60 <handle_one_connection(void*)>pfs = <optimized out>klass = <optimized out>#15 0x00007f5d5efe01ca in start_thread () from /lib64/libpthread.so.0No symbol table info available.#16 0x00007f5d5e330e73 in clone () from /lib64/libc.so.6No symbol table info available.As its insert select, do you happen to know the the end part of the query? Insert Into AuthenticationLog .... SELECT ..
Just the form of it is fine, doesn't have to be the exact query (though it would be in the GDB core file if still there "frame 11", "print packet").