[MDEV-30707] MariaDB can'T be stopped Created: 2023-02-22  Updated: 2023-04-11  Resolved: 2023-04-11

Status: Closed
Project: MariaDB Server
Component/s: Platform Windows
Affects Version/s: 10.4.25
Fix Version/s: N/A

Type: Bug Priority: Major
Reporter: Richard Stracke Assignee: Richard Stracke
Resolution: Incomplete Votes: 0
Labels: None
Environment:

Windows

Attachments: Text File out.txt    
Issue Links:
Relates
relates to MDEV-15020 Server hangs due to InnoDB persistent... Closed

 Description   

Servive MariaDB was stopped , but stuck for multiple hours in status stoppeing without useful entries in the error log.

Attached full stacktrace from minidump.

 Comments   
Comment by Marko Mäkelä [ 2023-02-22 ]

This looks like a possible duplicate of MDEV-15020. In one thread, dict_stats_save() is updating InnoDB persistent statistics tables. Before MariaDB 10.6, that is prone to deadlocks due to lock order violation.

Comment by Marko Mäkelä [ 2023-03-08 ]

I was initially suspecting that this could be a duplicate of MDEV-15020, but a deeper analysis suggests that this is something else. Before MDEV-15020 was fixed in MariaDB 10.6.5, dict_stats_save() (thread 22 in out.txt) would hold dict_sys.mutex and exclusive dict_sys.latch. Inside transaction commit, it is waiting for log_write_up_to(), in log_write_mutex_enter().

That log_sys.write_mutex is basically only being used in log_write_up_to(). It turns out that several threads are waiting for the same mutex in log_write_up_to(). The mutex must be held by this thread:

mariadb-10.4.25

 
.175  Id: 1318.3824 Suspend: 0 Teb: 000000cc`8022c000 Unfrozen
 
      Start: ucrtbase!thread_start<unsigned int (__cdecl*)(void * __ptr64)> (00007ffb`5b182650)
 
      Priority: 0  Priority class: 32  Affinity: ffffffff
 
Child-SP          RetAddr               Call Site
 
000000cc`869fd358 00007ffb`5b7e93ad     ntdll!NtWriteFile+0x14
 
000000cc`869fd360 00007ff6`4b0cdd95     KERNELBASE!WriteFile+0xfd
 
000000cc`869fd3d0 00007ff6`4b0d160f     mysqld!SyncFileIO::execute(class IORequest * request = 0x000000cc`869fd488)+0x65 [D:\winx64-packages\build\src\storage\innobase\os\os0file.cc @ 3252]
 
000000cc`869fd430 00007ff6`4b0d1c6f     mysqld!os_file_io(class IORequest * in_type = <Value unavailable error>, void * file = 0x00000000`000001c4, void * buf = <Value unavailable error>, unsigned int64 n = 0x3800, unsigned int64 offset = 0x2140800, dberr_t * err = 0x000000cc`869fd660)+0x8f [D:\winx64-packages\build\src\storage\innobase\os\os0file.cc @ 4482]
 
000000cc`869fd5e0 00007ff6`4b0d2507     mysqld!os_file_pwrite(class IORequest * type = 0x000000cc`869fd888, void * file = 0x00000000`000001c4, unsigned char * buf = 0x0000029c`83010000 "--- memory read error at address 0x0000029c`83010000 ---", unsigned int64 n = 0x3800, unsigned int64 offset = 0x2140800, dberr_t * err = 0x000000cc`869fd660)+0xaf [D:\winx64-packages\build\src\storage\innobase\os\os0file.cc @ 4566]
 
000000cc`869fd630 00007ff6`4b0cea0c     mysqld!os_file_write_func(class IORequest * type = <Value unavailable error>, char * name = 0x0000029b`15a08420 "--- memory read error at address 0x0000029b`15a08420 ---", void * file = <Value unavailable error>, void * buf = <Value unavailable error>, unsigned int64 offset = 0x2140800, unsigned int64 n = 0x3800)+0x37 [D:\winx64-packages\build\src\storage\innobase\os\os0file.cc @ 4596]
 
000000cc`869fd780 00007ff6`4b10204e     mysqld!os_aio_func(class IORequest * type = 0x000000cc`869fd888, unsigned int64 mode = 0x18, char * name = 0x0000029b`15a08420 "--- memory read error at address 0x0000029b`15a08420 ---", struct pfs_os_file_t * file = 0x000000cc`869fdbc0, void * buf = 0x0000029c`83010000, unsigned int64 offset = 0x2140800, unsigned int64 n = 0x3800, bool read_only = false, struct fil_node_t * m1 = 0x00000290`7f79aa70, void * m2 = 0x00000000`00000000)+0x8c [D:\winx64-packages\build\src\storage\innobase\os\os0file.cc @ 6401]
 
(Inline Function) --------`--------     mysqld!pfs_os_aio_func(void)+0xa2 [D:\winx64-packages\build\src\storage\innobase\include\os0file.inl @ 253]
 
000000cc`869fd810 00007ff6`4b0e07e9     mysqld!fil_io(class IORequest * type = <Value unavailable error>, bool sync = false, class page_id_t page_id = class page_id_t, unsigned int64 zip_size = <Value unavailable error>, unsigned int64 byte_offset = 0x800, unsigned int64 len = 0x3800, void * buf = 0x0000029c`83010000, void * message = 0x00000000`00000000, bool ignore_missing_space = false)+0x6de [D:\winx64-packages\build\src\storage\innobase\fil\fil0fil.cc @ 4272]
 
000000cc`869fdc80 00007ff6`4b0e1345     mysqld!log_write_buf(unsigned char * buf = 0x0000029c`83010000 "--- memory read error at address 0x0000029c`83010000 ---", unsigned int64 len = 0x3800, unsigned int64 start_lsn = 0x0000650d`eaea9800, unsigned int64 new_data_offset = <Value unavailable error>)+0x1d9 [D:\winx64-packages\build\src\storage\innobase\log\log0log.cc @ 780]
 
000000cc`869fdd20 00007ff6`4b19187c     mysqld!log_write_up_to(unsigned int64 lsn = <Value unavailable error>, bool flush_to_disk = false, bool rotate_key = false)+0x575 [D:\winx64-packages\build\src\storage\innobase\log\log0log.cc @ 1022]
 
000000cc`869fddd0 00007ff6`4b19130a     mysqld!trx_flush_log_if_needed(unsigned int64 lsn = <Value unavailable error>, struct trx_t * trx = 0x00000290`7f9c2098)+0x1c [D:\winx64-packages\build\src\storage\innobase\trx\trx0trx.cc @ 1231]
 
000000cc`869fde00 00007ff6`4b080673     mysqld!trx_commit_complete_for_mysql(struct trx_t * trx = 0x00000290`7f9c2098)+0x3a [D:\winx64-packages\build\src\storage\innobase\trx\trx0trx.cc @ 1702]
 
000000cc`869fde30 00007ff6`4ac6f791     mysqld!innobase_commit(struct handlerton * hton = 0x00000000`00000000, class THD * thd = 0x0000029c`88bd5878, bool commit_trx = <Value unavailable error>)+0xb3 [D:\winx64-packages\build\src\storage\innobase\handler\ha_innodb.cc @ 4749]
 
000000cc`869fde60 00007ff6`4ac7168e     mysqld!commit_one_phase_2(class THD * thd = 0x0000029c`88bd5878, bool all = true, struct THD_TRANS * trans = 0x0000029c`88bd8340, bool is_real_trans = true)+0x61 [D:\winx64-packages\build\src\sql\handler.cc @ 1831]
 
000000cc`869fdeb0 00007ff6`4ac71b65     mysqld!ha_commit_one_phase(class THD * thd = 0x0000029c`88bd5878, bool all = true)+0xae [D:\winx64-packages\build\src\sql\handler.cc @ 1812]
 
000000cc`869fdee0 00007ff6`4afa3e33     mysqld!ha_commit_trans(class THD * thd = 0x0000029c`88bd5878, bool all = true)+0x465 [D:\winx64-packages\build\src\sql\handler.cc @ 1715]
 
000000cc`869fe800 00007ff6`4ae59ca6     mysqld!trans_commit(class THD * thd = 0x0000029c`88bd5878)+0x33 [D:\winx64-packages\build\src\sql\transaction.cc @ 248]
 
000000cc`869fe830 00007ff6`4ae5af34     mysqld!mysql_execute_command(class THD * thd = 0x0000029c`88bd5878)+0x3136 [D:\winx64-packages\build\src\sql\sql_parse.cc @ 5703]
 
000000cc`869feca0 00007ff6`4ae52417     mysqld!mysql_parse(class THD * thd = 0x0000029c`88bd5878, char * rawbuf = 0x0000029d`327b3af0 "--- memory read error at address 0x0000029d`327b3af0 ---", unsigned int length = <Value unavailable error>, class Parser_state * parser_state = 0x000000cc`869feea0, bool is_com_multi = false, bool is_next_command = false)+0x164 [D:\winx64-packages\build\src\sql\sql_parse.cc @ 7999]
 
000000cc`869fed00 00007ff6`4ae5388c     mysqld!dispatch_command(enum_server_command command = <Value unavailable error>, class THD * thd = 0x0000029c`88bd5878, char * packet = 0x0000029c`8aa33399 "--- memory read error at address 0x0000029c`8aa33399 ---", unsigned int packet_length = 6, bool is_com_multi = false, bool is_next_command = false)+0x987 [D:\winx64-packages\build\src\sql\sql_parse.cc @ 1860]
 
000000cc`869ffd60 00007ff6`4af27bc2     mysqld!do_command(class THD * thd = 0x0000029c`88bd5878)+0x1bc [D:\winx64-packages\build\src\sql\sql_parse.cc @ 1378]
 
000000cc`869ffda0 00007ff6`4af29409     mysqld!do_handle_one_connection(class CONNECT * connect = 0x0000029c`fff75528)+0x202 [D:\winx64-packages\build\src\sql\sql_connect.cc @ 1420]
 
000000cc`869ffe10 00007ff6`4b28859c     mysqld!handle_one_connection(void * arg = 0x0000029c`fff75528)+0x39 [D:\winx64-packages\build\src\sql\sql_connect.cc @ 1318]
 
000000cc`869ffe40 00007ffb`5b18268a     mysqld!pthread_start(void * p = <Value unavailable error>)+0x1c [D:\winx64-packages\build\src\mysys\my_winthread.c @ 62]
 
000000cc`869ffe70 00007ffb`5bd07af4     ucrtbase!thread_start<unsigned int +0x3a
 
000000cc`869ffea0 00007ffb`5e8fa371     kernel32!BaseThreadInitThunk+0x14
 
000000cc`869ffed0 00000000`00000000     ntdll!RtlUserThreadStart+0x21

It appears that a write could be stuck in the operating system.

The log writing is quite different in 10.6, thanks to MDEV-26789. Also the extra complication of sync_array was removed from 10.6.

Comment by Marko Mäkelä [ 2023-03-08 ]

Can you find out if any errors were reported by the operating system?

Generated at Thu Feb 08 10:18:17 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.