Details
-
Bug
-
Status: Closed (View Workflow)
-
Major
-
Resolution: Incomplete
-
10.4.25
-
None
-
Windows
Description
Servive MariaDB was stopped , but stuck for multiple hours in status stoppeing without useful entries in the error log.
Attached full stacktrace from minidump.
Attachments
- out.txt
- 125 kB
- Richard Stracke
Issue Links
- relates to
-
MDEV-15020 Server hangs due to InnoDB persistent statistics or innodb_stats_auto_recalc
-
- Closed
-
Activity
I was initially suspecting that this could be a duplicate of MDEV-15020, but a deeper analysis suggests that this is something else. Before MDEV-15020 was fixed in MariaDB 10.6.5, dict_stats_save() (thread 22 in out.txt) would hold dict_sys.mutex and exclusive dict_sys.latch. Inside transaction commit, it is waiting for log_write_up_to(), in log_write_mutex_enter().
That log_sys.write_mutex is basically only being used in log_write_up_to(). It turns out that several threads are waiting for the same mutex in log_write_up_to(). The mutex must be held by this thread:
mariadb-10.4.25 |
.175 Id: 1318.3824 Suspend: 0 Teb: 000000cc`8022c000 Unfrozen
|
Start: ucrtbase!thread_start<unsigned int (__cdecl*)(void * __ptr64)> (00007ffb`5b182650)
|
Priority: 0 Priority class: 32 Affinity: ffffffff
|
Child-SP RetAddr Call Site
|
000000cc`869fd358 00007ffb`5b7e93ad ntdll!NtWriteFile+0x14
|
000000cc`869fd360 00007ff6`4b0cdd95 KERNELBASE!WriteFile+0xfd
|
000000cc`869fd3d0 00007ff6`4b0d160f mysqld!SyncFileIO::execute(class IORequest * request = 0x000000cc`869fd488)+0x65 [D:\winx64-packages\build\src\storage\innobase\os\os0file.cc @ 3252]
|
000000cc`869fd430 00007ff6`4b0d1c6f mysqld!os_file_io(class IORequest * in_type = <Value unavailable error>, void * file = 0x00000000`000001c4, void * buf = <Value unavailable error>, unsigned int64 n = 0x3800, unsigned int64 offset = 0x2140800, dberr_t * err = 0x000000cc`869fd660)+0x8f [D:\winx64-packages\build\src\storage\innobase\os\os0file.cc @ 4482]
|
000000cc`869fd5e0 00007ff6`4b0d2507 mysqld!os_file_pwrite(class IORequest * type = 0x000000cc`869fd888, void * file = 0x00000000`000001c4, unsigned char * buf = 0x0000029c`83010000 "--- memory read error at address 0x0000029c`83010000 ---", unsigned int64 n = 0x3800, unsigned int64 offset = 0x2140800, dberr_t * err = 0x000000cc`869fd660)+0xaf [D:\winx64-packages\build\src\storage\innobase\os\os0file.cc @ 4566]
|
000000cc`869fd630 00007ff6`4b0cea0c mysqld!os_file_write_func(class IORequest * type = <Value unavailable error>, char * name = 0x0000029b`15a08420 "--- memory read error at address 0x0000029b`15a08420 ---", void * file = <Value unavailable error>, void * buf = <Value unavailable error>, unsigned int64 offset = 0x2140800, unsigned int64 n = 0x3800)+0x37 [D:\winx64-packages\build\src\storage\innobase\os\os0file.cc @ 4596]
|
000000cc`869fd780 00007ff6`4b10204e mysqld!os_aio_func(class IORequest * type = 0x000000cc`869fd888, unsigned int64 mode = 0x18, char * name = 0x0000029b`15a08420 "--- memory read error at address 0x0000029b`15a08420 ---", struct pfs_os_file_t * file = 0x000000cc`869fdbc0, void * buf = 0x0000029c`83010000, unsigned int64 offset = 0x2140800, unsigned int64 n = 0x3800, bool read_only = false, struct fil_node_t * m1 = 0x00000290`7f79aa70, void * m2 = 0x00000000`00000000)+0x8c [D:\winx64-packages\build\src\storage\innobase\os\os0file.cc @ 6401]
|
(Inline Function) --------`-------- mysqld!pfs_os_aio_func(void)+0xa2 [D:\winx64-packages\build\src\storage\innobase\include\os0file.inl @ 253]
|
000000cc`869fd810 00007ff6`4b0e07e9 mysqld!fil_io(class IORequest * type = <Value unavailable error>, bool sync = false, class page_id_t page_id = class page_id_t, unsigned int64 zip_size = <Value unavailable error>, unsigned int64 byte_offset = 0x800, unsigned int64 len = 0x3800, void * buf = 0x0000029c`83010000, void * message = 0x00000000`00000000, bool ignore_missing_space = false)+0x6de [D:\winx64-packages\build\src\storage\innobase\fil\fil0fil.cc @ 4272]
|
000000cc`869fdc80 00007ff6`4b0e1345 mysqld!log_write_buf(unsigned char * buf = 0x0000029c`83010000 "--- memory read error at address 0x0000029c`83010000 ---", unsigned int64 len = 0x3800, unsigned int64 start_lsn = 0x0000650d`eaea9800, unsigned int64 new_data_offset = <Value unavailable error>)+0x1d9 [D:\winx64-packages\build\src\storage\innobase\log\log0log.cc @ 780]
|
000000cc`869fdd20 00007ff6`4b19187c mysqld!log_write_up_to(unsigned int64 lsn = <Value unavailable error>, bool flush_to_disk = false, bool rotate_key = false)+0x575 [D:\winx64-packages\build\src\storage\innobase\log\log0log.cc @ 1022]
|
000000cc`869fddd0 00007ff6`4b19130a mysqld!trx_flush_log_if_needed(unsigned int64 lsn = <Value unavailable error>, struct trx_t * trx = 0x00000290`7f9c2098)+0x1c [D:\winx64-packages\build\src\storage\innobase\trx\trx0trx.cc @ 1231]
|
000000cc`869fde00 00007ff6`4b080673 mysqld!trx_commit_complete_for_mysql(struct trx_t * trx = 0x00000290`7f9c2098)+0x3a [D:\winx64-packages\build\src\storage\innobase\trx\trx0trx.cc @ 1702]
|
000000cc`869fde30 00007ff6`4ac6f791 mysqld!innobase_commit(struct handlerton * hton = 0x00000000`00000000, class THD * thd = 0x0000029c`88bd5878, bool commit_trx = <Value unavailable error>)+0xb3 [D:\winx64-packages\build\src\storage\innobase\handler\ha_innodb.cc @ 4749]
|
000000cc`869fde60 00007ff6`4ac7168e mysqld!commit_one_phase_2(class THD * thd = 0x0000029c`88bd5878, bool all = true, struct THD_TRANS * trans = 0x0000029c`88bd8340, bool is_real_trans = true)+0x61 [D:\winx64-packages\build\src\sql\handler.cc @ 1831]
|
000000cc`869fdeb0 00007ff6`4ac71b65 mysqld!ha_commit_one_phase(class THD * thd = 0x0000029c`88bd5878, bool all = true)+0xae [D:\winx64-packages\build\src\sql\handler.cc @ 1812]
|
000000cc`869fdee0 00007ff6`4afa3e33 mysqld!ha_commit_trans(class THD * thd = 0x0000029c`88bd5878, bool all = true)+0x465 [D:\winx64-packages\build\src\sql\handler.cc @ 1715]
|
000000cc`869fe800 00007ff6`4ae59ca6 mysqld!trans_commit(class THD * thd = 0x0000029c`88bd5878)+0x33 [D:\winx64-packages\build\src\sql\transaction.cc @ 248]
|
000000cc`869fe830 00007ff6`4ae5af34 mysqld!mysql_execute_command(class THD * thd = 0x0000029c`88bd5878)+0x3136 [D:\winx64-packages\build\src\sql\sql_parse.cc @ 5703]
|
000000cc`869feca0 00007ff6`4ae52417 mysqld!mysql_parse(class THD * thd = 0x0000029c`88bd5878, char * rawbuf = 0x0000029d`327b3af0 "--- memory read error at address 0x0000029d`327b3af0 ---", unsigned int length = <Value unavailable error>, class Parser_state * parser_state = 0x000000cc`869feea0, bool is_com_multi = false, bool is_next_command = false)+0x164 [D:\winx64-packages\build\src\sql\sql_parse.cc @ 7999]
|
000000cc`869fed00 00007ff6`4ae5388c mysqld!dispatch_command(enum_server_command command = <Value unavailable error>, class THD * thd = 0x0000029c`88bd5878, char * packet = 0x0000029c`8aa33399 "--- memory read error at address 0x0000029c`8aa33399 ---", unsigned int packet_length = 6, bool is_com_multi = false, bool is_next_command = false)+0x987 [D:\winx64-packages\build\src\sql\sql_parse.cc @ 1860]
|
000000cc`869ffd60 00007ff6`4af27bc2 mysqld!do_command(class THD * thd = 0x0000029c`88bd5878)+0x1bc [D:\winx64-packages\build\src\sql\sql_parse.cc @ 1378]
|
000000cc`869ffda0 00007ff6`4af29409 mysqld!do_handle_one_connection(class CONNECT * connect = 0x0000029c`fff75528)+0x202 [D:\winx64-packages\build\src\sql\sql_connect.cc @ 1420]
|
000000cc`869ffe10 00007ff6`4b28859c mysqld!handle_one_connection(void * arg = 0x0000029c`fff75528)+0x39 [D:\winx64-packages\build\src\sql\sql_connect.cc @ 1318]
|
000000cc`869ffe40 00007ffb`5b18268a mysqld!pthread_start(void * p = <Value unavailable error>)+0x1c [D:\winx64-packages\build\src\mysys\my_winthread.c @ 62]
|
000000cc`869ffe70 00007ffb`5bd07af4 ucrtbase!thread_start<unsigned int +0x3a
|
000000cc`869ffea0 00007ffb`5e8fa371 kernel32!BaseThreadInitThunk+0x14
|
000000cc`869ffed0 00000000`00000000 ntdll!RtlUserThreadStart+0x21
|
It appears that a write could be stuck in the operating system.
The log writing is quite different in 10.6, thanks to MDEV-26789. Also the extra complication of sync_array was removed from 10.6.
Can you find out if any errors were reported by the operating system?
This looks like a possible duplicate of
MDEV-15020. In one thread, dict_stats_save() is updating InnoDB persistent statistics tables. Before MariaDB 10.6, that is prone to deadlocks due to lock order violation.