[MDEV-23034] 10.5.4 mariadbd crashes during shutdown Created: 2020-06-28  Updated: 2022-12-19  Resolved: 2020-07-30

Status: Closed
Project: MariaDB Server
Component/s: Server
Affects Version/s: 10.5.4
Fix Version/s: N/A

Type: Bug Priority: Major
Reporter: Xiaoguang Wang Assignee: Unassigned
Resolution: Incomplete Votes: 0
Labels: need_feedback, server, shutdown
Environment:

debian 10.4 amd64


Issue Links:
Relates
relates to MDEV-30260 Slave crashed:reload_acl_and_cache du... Stalled

 Description   

....
2020-06-28  0:45:33 0 [Note] InnoDB: Buffer pool(s) load completed at 200628  0:45:33
2020-06-28  0:45:46 0 [Note] /data1/app/mariadb/bin/mariadbd (initiated by: unknown): Normal shutdown
2020-06-28  0:45:46 0 [Note] Event Scheduler: Purging the queue. 0 events
2020-06-28  0:45:46 10 [Note] Slave SQL thread exiting, replication stopped in log 'mysql-bin.065769' at position 256; GTID position '0-100065-8777861145'
 
Status information:
 
Current dir: /data1/mysql/mysql-instance/data/
Running threads: 2  Cached threads: 2  Stack size: 299008
 
Key caches:
default
Buffer_size:      33554432
Block_size:           1024
Division_limit:        100
Age_threshold:         300
Partitions:              0
blocks used:             0
not flushed:             0
w_requests:              0
writes:                  0
r_requests:              0
reads:                   0
 
 
handler status:
read_key:         4531
read_next:        1023
read_rnd          1026
read_first:          3
write:            5218
delete               5
update:           3764
 
Table status:
Opened tables:         46
Open tables:           40
Open files:            35
Open streams:           4
 
Alarm status:
Active alarms:   0
Max used alarms: 0
Next alarm time: 0
 
Memory status:
Non-mmapped space allocated from system: 174833664
Number of free chunks:                   538
Number of fastbin blocks:                9
Number of mmapped regions:               15
Space in mmapped regions:                171819008
Maximum total allocated space:           0
Space available in freed fastbin blocks: 624
Total allocated space:                   159597856
Total free space:                        15235808
Top-most, releasable space:              200192
Estimated memory (with thread stack):    347250688
Global memory allocated by server:       34358400
Memory allocated by threads:             118384
 
 
 
Events status:
LLA = Last Locked At  LUA = Last Unlocked At
WOC = Waiting On Condition  DL = Data Locked
The Event Scheduler is disabled
 
2020-06-28  0:45:46 9 [Note] Slave I/O thread exiting, read up to log 'mysql-bin.065769', position 74476178; GTID position 0-100065-8777871952
2020-06-28  0:45:46 0 [Note] InnoDB: FTS optimize thread exiting.
200628  0:45:46 [ERROR] mysqld got signal 11 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
 
To report this bug, see https://mariadb.com/kb/en/reporting-bugs
 
We will try our best to scrape up some info that will hopefully help
diagnose the problem, but since we have already crashed,
something is definitely wrong and this may fail.
 
Server version: 10.5.4-MariaDB-log
key_buffer_size=33554432
read_buffer_size=16777216
max_used_connections=2
max_threads=2002
2020-06-28  0:45:46 0 [Note] InnoDB: Starting shutdown...
thread_count=0
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 65684313 K  bytes of memory
Hope that's ok; if not, decrease some variables in the equation.
 
Thread pointer: 0x0
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
2020-06-28  0:45:46 0 [Note] InnoDB: Dumping buffer pool(s) to /data1/mysql/mysql-instance/data/ib_buffer_pool
2020-06-28  0:45:46 0 [Note] InnoDB: Restricted to 128864 pages due to innodb_buf_pool_dump_pct=25
stack_bottom = 0x0 thread_stack 0x49000
2020-06-28  0:45:46 0 [Note] InnoDB: Buffer pool(s) dump completed at 200628  0:45:46
/data1/app/mariadb/bin/mariadbd(my_print_stacktrace+0x2e)[0x556bf79e849e]
/data1/app/mariadb/bin/mariadbd(handle_fatal_signal+0x307)[0x556bf73cf797]
??:0(__restore_rt)[0x7f79e329d730]
/data1/app/mariadb/bin/mariadbd(_Z22hostname_cache_refreshv+0x14)[0x556bf73e14c4]
/data1/app/mariadb/bin/mariadbd(_Z20reload_acl_and_cacheP3THDyP10TABLE_LISTPi+0x7c5)[0x556bf72e0e35]
2020-06-28  0:45:48 0 [Note] InnoDB: Shutdown completed; log sequence number 11724296462283; transaction id 16032419783
2020-06-28  0:45:48 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1"
2020-06-28  0:45:48 0 [Note] /data1/app/mariadb/bin/mariadbd: Shutdown complete
 
sql/hash_filo.h:87(hostname_cache_refresh())[0x556bf710180a]
sql/sql_reload.cc:351(reload_acl_and_cache(THD*, unsigned long long, TABLE_LIST*, int*))[0x556bf763fbad]
??:0(start_thread)[0x7f79e3292fa3]
??:0(clone)[0x7f79e2e6c4cf]
The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
information that should help you find out what is causing the crash.
Writing a core file...
Working directory at /data1/mysql/mysql-instance/data
Resource Limits:
Limit                     Soft Limit           Hard Limit           Units
Max cpu time              unlimited            unlimited            seconds
Max file size             unlimited            unlimited            bytes
Max data size             unlimited            unlimited            bytes
Max stack size            8388608              unlimited            bytes
Max core file size        0                    unlimited            bytes
Max resident set          unlimited            unlimited            bytes
Max processes             257066               257066               processes
Max open files            1024000              1024000              files
Max locked memory         67108864             67108864             bytes
Max address space         unlimited            unlimited            bytes
Max file locks            unlimited            unlimited            locks
Max pending signals       257066               257066               signals
Max msgqueue size         819200               819200               bytes
Max nice priority         0                    0
Max realtime priority     0                    0
Max realtime timeout      unlimited            unlimited            us
Core pattern: core
 
 
2020-06-28  0:48:54 0 [Note] InnoDB: Using Linux native AIO
2020-06-28  0:48:54 0 [Note] InnoDB: Uses event mutexes
....



 Comments   
Comment by Marko Mäkelä [ 2020-07-01 ]

wxiaoguang, would it be possible for you to attach a debugger to the process before you initiate shutdown? Something like this:

sudo gdb -p $(pgrep -x mariadbd)

Then, once the SIGSEGV is delivered, issue the following commands in gdb:

thread apply all backtrace
continue

The stack traces that mariadbd generates on crash are often unhelpful. Even if they contained a useful stack trace, it would be for the current thread only.

Comment by Xiaoguang Wang [ 2020-07-01 ]

This crash only occurred once here, I can not manage to reproduce it.

The only thing I can remember before this crash is that I copied the config from another slave with the same GTID to setup a new slave, and this mariadbd reports errors about duplicated GTIDs. Then I shutdown(kill) mariadbd and corrected the GTID and restarted mariadbd, maybe the crash occurred during these steps. Maybe I also changed the linux's hostname during these steps.

Just now I tried to kill mariadbd several times, no more crashes.

Comment by Marko Mäkelä [ 2020-07-01 ]

wxiaoguang, thank you! Based on your description, I feel that the problem is more likely to be in replication than in InnoDB (which is what I am working on).

Comment by Andrei Elkin [ 2020-07-01 ]

wxiaoguang Sounds like it even harder to reproduce it than to meet for the first time ! Thank you for reporting, yet the amount of info provided is insufficient to suspect the slave applier either.

Generated at Thu Feb 08 09:19:20 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.