[MDEV-24632] MariaDB 10.3.27 crash on windows Created: 2021-01-20  Updated: 2023-04-14  Resolved: 2023-04-14

Status: Closed
Project: MariaDB Server
Component/s: Storage Engine - InnoDB
Affects Version/s: 10.3.27
Fix Version/s: 10.6.5

Type: Bug Priority: Major
Reporter: srinivas chandolu Assignee: Marko Mäkelä
Resolution: Duplicate Votes: 0
Labels: Crash
Environment:

Microsoft Windows Server 2016 Standard
VMware x-64 based PC.
32 GB Memory
mysqld Ver 10.3.27-MariaDB-log for Win64 on AMD64 (mariadb.org binary distribution)


Attachments: File WIN-4RVIN69VNHC.err     File WIN-G57FI8VPMSD.err     Text File mdbvariables.txt     File my.ini     File mysqld.dmp     File mysqld.dmp     Zip Archive mysqldexe.zip     File mysqldpdb.7z     PNG File screenshot-1.png     Text File stacks.txt    
Issue Links:
Duplicate
duplicates MDEV-15020 Server hangs due to InnoDB persistent... Closed

 Description   

MariaDB server is getting crash by throwing the following error in .err file. Attached the .err file, crash dump file, my.ini file and "show variables" output for reference. It would be great if the reason and also the probable fix for the issue has been provided.

InnoDB: ###### Diagnostic info printed to the standard error stream
2021-01-19 22:19:04 0 [ERROR] [FATAL] InnoDB: Semaphore wait has lasted > 600 seconds. We intentionally crash the server because it appears to be hung.
210119 22:19:04 [ERROR] mysqld got exception 0x80000003 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
 
To report this bug, see https://mariadb.com/kb/en/reporting-bugs
 
We will try our best to scrape up some info that will hopefully help
diagnose the problem, but since we have already crashed, 
something is definitely wrong and this may fail.
 
Server version: 10.3.27-MariaDB-log
key_buffer_size=33554432
read_buffer_size=8388608
max_used_connections=66
max_threads=65537
thread_count=45
It is possible that mysqld could use up to 
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 52116 K  bytes of memory
Hope that's ok; if not, decrease some variables in the equation.
 
Thread pointer: 0x0
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
mysqld.exe!my_sigabrt_handler()[my_thr_init.c:485]
mysqld.exe!raise()[signal.cpp:547]
mysqld.exe!abort()[abort.cpp:71]
mysqld.exe!ib::fatal::~fatal()[ut0ut.cc:603]
mysqld.exe!srv_error_monitor_thread()[srv0srv.cc:1833]
KERNEL32.DLL!BaseThreadInitThunk()
ntdll.dll!RtlUserThreadStart()
The manual page at https://mariadb.com/kb/en/how-to-produce-a-full-stack-trace-for-mysqld/ contains
information that should help you find out what is causing the crash.
Writing a core file at C:\Program Files\McAfee\Network Security Manager\MariaDB\data\
Minidump written to C:\Program Files\McAfee\Network Security Manager\MariaDB\data\mysqld.dmp
 



 Comments   
Comment by srinivas chandolu [ 2021-02-16 ]

Hi Elena Stepanova,

Just curious to know any update on the mentioned issue? As a note, noticed that the MariaDB crash is happening if it has been installed on VM with thin provisioned disk. Not noticed such crashes in prior versions.

Any particular reason for this behavior?

Please update your findings.

Regards,
Srinivas

Comment by srinivas chandolu [ 2021-04-20 ]

Hi Marko Mäkelä,

Noticed that with MariaDB 10.3.28 few crash issues related to semaphore have been fixed. Upgraded the database from 10.3.27 to 10.3.28. Noticed that the crash is still happening. However prior to crash we have not noticed any query performance issues with 10.3.28. These db query performance issues have been noticed in 10.3.27.
Can you update the reason for this crash. Attached the crash dump and .err file for reference.

Regards,
Srinivas

Comment by srinivas chandolu [ 2021-06-10 ]

Hi Marko Mäkelä,
From the release notes looks like the semaphore crash fix has gone in 10.3.28 version. However as mentioned in earlier comment it has reoccurred while using MariaDB Version 10.3.28 (with the attached logs earlier). Just curious to know whether this issue has been addressed completely or still existing as per provided logs. Also curious to know the reason of failure. Please update.
Regards,
Srinivas

Comment by srinivas chandolu [ 2021-06-18 ]

Hi Marko Mäkelä,

Is it possible to restart database automatically for such abrupt shutdowns by configuring database or using a database configuration parameter? If so from what version of database this feature is (will be if it is planned) available?

Also, any update on the reason of failure on the raised semaphore issue?

Regards,
Srinivas

Comment by Marko Mäkelä [ 2021-06-28 ]

wlad, can you please check the .dmp files and comment on whether mysqld_safe like functionality is available on Microsoft Windows?

Comment by Vladislav Vaintroub [ 2021-06-29 ]

It is possible to restart ay windows service, if it crashes. https://serverfault.com/questions/48600/how-can-i-automatically-restart-a-windows-service-if-it-crashes

Comment by Vladislav Vaintroub [ 2021-06-29 ]

I can't load symbols using our official packages. I guess, this is something McAffee compiled themselves, and we'd need the mysqld.exe and mysqld.pdb to analyze

Comment by srinivas chandolu [ 2021-06-29 ]

Hi Vladislav Vaintroub,
Thanks for the updates.
As per provided reference, for restarting the database service during crash we need to utilize either (a) Operating System service properties or (b) a tool/agent that monitors this service and restart accordingly.

Attached mysqld.exe and mysqld.pdb files of Version 10.3.28 for reference. Also as a note these binary files are downloaded from downloads.mariadb.org website (part of mariadb-10.3.28-winx64.zip)

https://downloads.mariadb.org/mariadb/10.3.27/
https://downloads.mariadb.org/mariadb/10.3.28/

Comment by Vladislav Vaintroub [ 2021-06-30 ]

marko, the stacks.txt contains the stacks, and exception information, corresponding to innodb abort on 2021-04-20. There are several waits in this file, mostly bufferpool, not sure which one is interesting.

schandol, I suggest to peruse the service configuration to automatically configure restarts on crash. Perhaps, we should be doing this by default, when installing service, if you feel it is a good idea, please comment on it.

Comment by Marko Mäkelä [ 2023-04-14 ]

One of the threads in stacks.txt is executing dict_stats_save(). I suspect that this is a duplicate of the infamous MDEV-15020.

Generated at Thu Feb 08 09:31:28 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.