Details

    • Bug
    • Status: Closed (View Workflow)
    • Major
    • Resolution: Incomplete
    • 10.4.25
    • N/A
    • Platform Windows
    • None
    • Windows

    Description

      Servive MariaDB was stopped , but stuck for multiple hours in status stoppeing without useful entries in the error log.

      Attached full stacktrace from minidump.

      Attachments

        1. out.txt
          125 kB
          Richard Stracke

        Issue Links

          Activity

            This looks like a possible duplicate of MDEV-15020. In one thread, dict_stats_save() is updating InnoDB persistent statistics tables. Before MariaDB 10.6, that is prone to deadlocks due to lock order violation.

            marko Marko Mäkelä added a comment - This looks like a possible duplicate of MDEV-15020 . In one thread, dict_stats_save() is updating InnoDB persistent statistics tables. Before MariaDB 10.6, that is prone to deadlocks due to lock order violation.

            I was initially suspecting that this could be a duplicate of MDEV-15020, but a deeper analysis suggests that this is something else. Before MDEV-15020 was fixed in MariaDB 10.6.5, dict_stats_save() (thread 22 in out.txt) would hold dict_sys.mutex and exclusive dict_sys.latch. Inside transaction commit, it is waiting for log_write_up_to(), in log_write_mutex_enter().

            That log_sys.write_mutex is basically only being used in log_write_up_to(). It turns out that several threads are waiting for the same mutex in log_write_up_to(). The mutex must be held by this thread:

            mariadb-10.4.25

            .175  Id: 1318.3824 Suspend: 0 Teb: 000000cc`8022c000 Unfrozen
                  Start: ucrtbase!thread_start<unsigned int (__cdecl*)(void * __ptr64)> (00007ffb`5b182650)
                  Priority: 0  Priority class: 32  Affinity: ffffffff
            Child-SP          RetAddr               Call Site
            000000cc`869fd358 00007ffb`5b7e93ad     ntdll!NtWriteFile+0x14
            000000cc`869fd360 00007ff6`4b0cdd95     KERNELBASE!WriteFile+0xfd
            000000cc`869fd3d0 00007ff6`4b0d160f     mysqld!SyncFileIO::execute(class IORequest * request = 0x000000cc`869fd488)+0x65 [D:\winx64-packages\build\src\storage\innobase\os\os0file.cc @ 3252]
            000000cc`869fd430 00007ff6`4b0d1c6f     mysqld!os_file_io(class IORequest * in_type = <Value unavailable error>, void * file = 0x00000000`000001c4, void * buf = <Value unavailable error>, unsigned int64 n = 0x3800, unsigned int64 offset = 0x2140800, dberr_t * err = 0x000000cc`869fd660)+0x8f [D:\winx64-packages\build\src\storage\innobase\os\os0file.cc @ 4482]
            000000cc`869fd5e0 00007ff6`4b0d2507     mysqld!os_file_pwrite(class IORequest * type = 0x000000cc`869fd888, void * file = 0x00000000`000001c4, unsigned char * buf = 0x0000029c`83010000 "--- memory read error at address 0x0000029c`83010000 ---", unsigned int64 n = 0x3800, unsigned int64 offset = 0x2140800, dberr_t * err = 0x000000cc`869fd660)+0xaf [D:\winx64-packages\build\src\storage\innobase\os\os0file.cc @ 4566]
            000000cc`869fd630 00007ff6`4b0cea0c     mysqld!os_file_write_func(class IORequest * type = <Value unavailable error>, char * name = 0x0000029b`15a08420 "--- memory read error at address 0x0000029b`15a08420 ---", void * file = <Value unavailable error>, void * buf = <Value unavailable error>, unsigned int64 offset = 0x2140800, unsigned int64 n = 0x3800)+0x37 [D:\winx64-packages\build\src\storage\innobase\os\os0file.cc @ 4596]
            000000cc`869fd780 00007ff6`4b10204e     mysqld!os_aio_func(class IORequest * type = 0x000000cc`869fd888, unsigned int64 mode = 0x18, char * name = 0x0000029b`15a08420 "--- memory read error at address 0x0000029b`15a08420 ---", struct pfs_os_file_t * file = 0x000000cc`869fdbc0, void * buf = 0x0000029c`83010000, unsigned int64 offset = 0x2140800, unsigned int64 n = 0x3800, bool read_only = false, struct fil_node_t * m1 = 0x00000290`7f79aa70, void * m2 = 0x00000000`00000000)+0x8c [D:\winx64-packages\build\src\storage\innobase\os\os0file.cc @ 6401]
            (Inline Function) --------`--------     mysqld!pfs_os_aio_func(void)+0xa2 [D:\winx64-packages\build\src\storage\innobase\include\os0file.inl @ 253]
            000000cc`869fd810 00007ff6`4b0e07e9     mysqld!fil_io(class IORequest * type = <Value unavailable error>, bool sync = false, class page_id_t page_id = class page_id_t, unsigned int64 zip_size = <Value unavailable error>, unsigned int64 byte_offset = 0x800, unsigned int64 len = 0x3800, void * buf = 0x0000029c`83010000, void * message = 0x00000000`00000000, bool ignore_missing_space = false)+0x6de [D:\winx64-packages\build\src\storage\innobase\fil\fil0fil.cc @ 4272]
            000000cc`869fdc80 00007ff6`4b0e1345     mysqld!log_write_buf(unsigned char * buf = 0x0000029c`83010000 "--- memory read error at address 0x0000029c`83010000 ---", unsigned int64 len = 0x3800, unsigned int64 start_lsn = 0x0000650d`eaea9800, unsigned int64 new_data_offset = <Value unavailable error>)+0x1d9 [D:\winx64-packages\build\src\storage\innobase\log\log0log.cc @ 780]
            000000cc`869fdd20 00007ff6`4b19187c     mysqld!log_write_up_to(unsigned int64 lsn = <Value unavailable error>, bool flush_to_disk = false, bool rotate_key = false)+0x575 [D:\winx64-packages\build\src\storage\innobase\log\log0log.cc @ 1022]
            000000cc`869fddd0 00007ff6`4b19130a     mysqld!trx_flush_log_if_needed(unsigned int64 lsn = <Value unavailable error>, struct trx_t * trx = 0x00000290`7f9c2098)+0x1c [D:\winx64-packages\build\src\storage\innobase\trx\trx0trx.cc @ 1231]
            000000cc`869fde00 00007ff6`4b080673     mysqld!trx_commit_complete_for_mysql(struct trx_t * trx = 0x00000290`7f9c2098)+0x3a [D:\winx64-packages\build\src\storage\innobase\trx\trx0trx.cc @ 1702]
            000000cc`869fde30 00007ff6`4ac6f791     mysqld!innobase_commit(struct handlerton * hton = 0x00000000`00000000, class THD * thd = 0x0000029c`88bd5878, bool commit_trx = <Value unavailable error>)+0xb3 [D:\winx64-packages\build\src\storage\innobase\handler\ha_innodb.cc @ 4749]
            000000cc`869fde60 00007ff6`4ac7168e     mysqld!commit_one_phase_2(class THD * thd = 0x0000029c`88bd5878, bool all = true, struct THD_TRANS * trans = 0x0000029c`88bd8340, bool is_real_trans = true)+0x61 [D:\winx64-packages\build\src\sql\handler.cc @ 1831]
            000000cc`869fdeb0 00007ff6`4ac71b65     mysqld!ha_commit_one_phase(class THD * thd = 0x0000029c`88bd5878, bool all = true)+0xae [D:\winx64-packages\build\src\sql\handler.cc @ 1812]
            000000cc`869fdee0 00007ff6`4afa3e33     mysqld!ha_commit_trans(class THD * thd = 0x0000029c`88bd5878, bool all = true)+0x465 [D:\winx64-packages\build\src\sql\handler.cc @ 1715]
            000000cc`869fe800 00007ff6`4ae59ca6     mysqld!trans_commit(class THD * thd = 0x0000029c`88bd5878)+0x33 [D:\winx64-packages\build\src\sql\transaction.cc @ 248]
            000000cc`869fe830 00007ff6`4ae5af34     mysqld!mysql_execute_command(class THD * thd = 0x0000029c`88bd5878)+0x3136 [D:\winx64-packages\build\src\sql\sql_parse.cc @ 5703]
            000000cc`869feca0 00007ff6`4ae52417     mysqld!mysql_parse(class THD * thd = 0x0000029c`88bd5878, char * rawbuf = 0x0000029d`327b3af0 "--- memory read error at address 0x0000029d`327b3af0 ---", unsigned int length = <Value unavailable error>, class Parser_state * parser_state = 0x000000cc`869feea0, bool is_com_multi = false, bool is_next_command = false)+0x164 [D:\winx64-packages\build\src\sql\sql_parse.cc @ 7999]
            000000cc`869fed00 00007ff6`4ae5388c     mysqld!dispatch_command(enum_server_command command = <Value unavailable error>, class THD * thd = 0x0000029c`88bd5878, char * packet = 0x0000029c`8aa33399 "--- memory read error at address 0x0000029c`8aa33399 ---", unsigned int packet_length = 6, bool is_com_multi = false, bool is_next_command = false)+0x987 [D:\winx64-packages\build\src\sql\sql_parse.cc @ 1860]
            000000cc`869ffd60 00007ff6`4af27bc2     mysqld!do_command(class THD * thd = 0x0000029c`88bd5878)+0x1bc [D:\winx64-packages\build\src\sql\sql_parse.cc @ 1378]
            000000cc`869ffda0 00007ff6`4af29409     mysqld!do_handle_one_connection(class CONNECT * connect = 0x0000029c`fff75528)+0x202 [D:\winx64-packages\build\src\sql\sql_connect.cc @ 1420]
            000000cc`869ffe10 00007ff6`4b28859c     mysqld!handle_one_connection(void * arg = 0x0000029c`fff75528)+0x39 [D:\winx64-packages\build\src\sql\sql_connect.cc @ 1318]
            000000cc`869ffe40 00007ffb`5b18268a     mysqld!pthread_start(void * p = <Value unavailable error>)+0x1c [D:\winx64-packages\build\src\mysys\my_winthread.c @ 62]
            000000cc`869ffe70 00007ffb`5bd07af4     ucrtbase!thread_start<unsigned int +0x3a
            000000cc`869ffea0 00007ffb`5e8fa371     kernel32!BaseThreadInitThunk+0x14
            000000cc`869ffed0 00000000`00000000     ntdll!RtlUserThreadStart+0x21
            

            It appears that a write could be stuck in the operating system.

            The log writing is quite different in 10.6, thanks to MDEV-26789. Also the extra complication of sync_array was removed from 10.6.

            marko Marko Mäkelä added a comment - I was initially suspecting that this could be a duplicate of MDEV-15020 , but a deeper analysis suggests that this is something else. Before MDEV-15020 was fixed in MariaDB 10.6.5, dict_stats_save() (thread 22 in out.txt ) would hold dict_sys.mutex and exclusive dict_sys.latch . Inside transaction commit, it is waiting for log_write_up_to() , in log_write_mutex_enter() . That log_sys.write_mutex is basically only being used in log_write_up_to() . It turns out that several threads are waiting for the same mutex in log_write_up_to() . The mutex must be held by this thread: mariadb-10.4.25 .175 Id: 1318.3824 Suspend: 0 Teb: 000000cc`8022c000 Unfrozen Start: ucrtbase!thread_start<unsigned int (__cdecl*)(void * __ptr64)> (00007ffb`5b182650) Priority: 0 Priority class: 32 Affinity: ffffffff Child-SP RetAddr Call Site 000000cc`869fd358 00007ffb`5b7e93ad ntdll!NtWriteFile+0x14 000000cc`869fd360 00007ff6`4b0cdd95 KERNELBASE!WriteFile+0xfd 000000cc`869fd3d0 00007ff6`4b0d160f mysqld!SyncFileIO::execute(class IORequest * request = 0x000000cc`869fd488)+0x65 [D:\winx64-packages\build\src\storage\innobase\os\os0file.cc @ 3252] 000000cc`869fd430 00007ff6`4b0d1c6f mysqld!os_file_io(class IORequest * in_type = <Value unavailable error>, void * file = 0x00000000`000001c4, void * buf = <Value unavailable error>, unsigned int64 n = 0x3800, unsigned int64 offset = 0x2140800, dberr_t * err = 0x000000cc`869fd660)+0x8f [D:\winx64-packages\build\src\storage\innobase\os\os0file.cc @ 4482] 000000cc`869fd5e0 00007ff6`4b0d2507 mysqld!os_file_pwrite(class IORequest * type = 0x000000cc`869fd888, void * file = 0x00000000`000001c4, unsigned char * buf = 0x0000029c`83010000 "--- memory read error at address 0x0000029c`83010000 ---", unsigned int64 n = 0x3800, unsigned int64 offset = 0x2140800, dberr_t * err = 0x000000cc`869fd660)+0xaf [D:\winx64-packages\build\src\storage\innobase\os\os0file.cc @ 4566] 000000cc`869fd630 00007ff6`4b0cea0c mysqld!os_file_write_func(class IORequest * type = <Value unavailable error>, char * name = 0x0000029b`15a08420 "--- memory read error at address 0x0000029b`15a08420 ---", void * file = <Value unavailable error>, void * buf = <Value unavailable error>, unsigned int64 offset = 0x2140800, unsigned int64 n = 0x3800)+0x37 [D:\winx64-packages\build\src\storage\innobase\os\os0file.cc @ 4596] 000000cc`869fd780 00007ff6`4b10204e mysqld!os_aio_func(class IORequest * type = 0x000000cc`869fd888, unsigned int64 mode = 0x18, char * name = 0x0000029b`15a08420 "--- memory read error at address 0x0000029b`15a08420 ---", struct pfs_os_file_t * file = 0x000000cc`869fdbc0, void * buf = 0x0000029c`83010000, unsigned int64 offset = 0x2140800, unsigned int64 n = 0x3800, bool read_only = false, struct fil_node_t * m1 = 0x00000290`7f79aa70, void * m2 = 0x00000000`00000000)+0x8c [D:\winx64-packages\build\src\storage\innobase\os\os0file.cc @ 6401] (Inline Function) --------`-------- mysqld!pfs_os_aio_func(void)+0xa2 [D:\winx64-packages\build\src\storage\innobase\include\os0file.inl @ 253] 000000cc`869fd810 00007ff6`4b0e07e9 mysqld!fil_io(class IORequest * type = <Value unavailable error>, bool sync = false, class page_id_t page_id = class page_id_t, unsigned int64 zip_size = <Value unavailable error>, unsigned int64 byte_offset = 0x800, unsigned int64 len = 0x3800, void * buf = 0x0000029c`83010000, void * message = 0x00000000`00000000, bool ignore_missing_space = false)+0x6de [D:\winx64-packages\build\src\storage\innobase\fil\fil0fil.cc @ 4272] 000000cc`869fdc80 00007ff6`4b0e1345 mysqld!log_write_buf(unsigned char * buf = 0x0000029c`83010000 "--- memory read error at address 0x0000029c`83010000 ---", unsigned int64 len = 0x3800, unsigned int64 start_lsn = 0x0000650d`eaea9800, unsigned int64 new_data_offset = <Value unavailable error>)+0x1d9 [D:\winx64-packages\build\src\storage\innobase\log\log0log.cc @ 780] 000000cc`869fdd20 00007ff6`4b19187c mysqld!log_write_up_to(unsigned int64 lsn = <Value unavailable error>, bool flush_to_disk = false, bool rotate_key = false)+0x575 [D:\winx64-packages\build\src\storage\innobase\log\log0log.cc @ 1022] 000000cc`869fddd0 00007ff6`4b19130a mysqld!trx_flush_log_if_needed(unsigned int64 lsn = <Value unavailable error>, struct trx_t * trx = 0x00000290`7f9c2098)+0x1c [D:\winx64-packages\build\src\storage\innobase\trx\trx0trx.cc @ 1231] 000000cc`869fde00 00007ff6`4b080673 mysqld!trx_commit_complete_for_mysql(struct trx_t * trx = 0x00000290`7f9c2098)+0x3a [D:\winx64-packages\build\src\storage\innobase\trx\trx0trx.cc @ 1702] 000000cc`869fde30 00007ff6`4ac6f791 mysqld!innobase_commit(struct handlerton * hton = 0x00000000`00000000, class THD * thd = 0x0000029c`88bd5878, bool commit_trx = <Value unavailable error>)+0xb3 [D:\winx64-packages\build\src\storage\innobase\handler\ha_innodb.cc @ 4749] 000000cc`869fde60 00007ff6`4ac7168e mysqld!commit_one_phase_2(class THD * thd = 0x0000029c`88bd5878, bool all = true, struct THD_TRANS * trans = 0x0000029c`88bd8340, bool is_real_trans = true)+0x61 [D:\winx64-packages\build\src\sql\handler.cc @ 1831] 000000cc`869fdeb0 00007ff6`4ac71b65 mysqld!ha_commit_one_phase(class THD * thd = 0x0000029c`88bd5878, bool all = true)+0xae [D:\winx64-packages\build\src\sql\handler.cc @ 1812] 000000cc`869fdee0 00007ff6`4afa3e33 mysqld!ha_commit_trans(class THD * thd = 0x0000029c`88bd5878, bool all = true)+0x465 [D:\winx64-packages\build\src\sql\handler.cc @ 1715] 000000cc`869fe800 00007ff6`4ae59ca6 mysqld!trans_commit(class THD * thd = 0x0000029c`88bd5878)+0x33 [D:\winx64-packages\build\src\sql\transaction.cc @ 248] 000000cc`869fe830 00007ff6`4ae5af34 mysqld!mysql_execute_command(class THD * thd = 0x0000029c`88bd5878)+0x3136 [D:\winx64-packages\build\src\sql\sql_parse.cc @ 5703] 000000cc`869feca0 00007ff6`4ae52417 mysqld!mysql_parse(class THD * thd = 0x0000029c`88bd5878, char * rawbuf = 0x0000029d`327b3af0 "--- memory read error at address 0x0000029d`327b3af0 ---", unsigned int length = <Value unavailable error>, class Parser_state * parser_state = 0x000000cc`869feea0, bool is_com_multi = false, bool is_next_command = false)+0x164 [D:\winx64-packages\build\src\sql\sql_parse.cc @ 7999] 000000cc`869fed00 00007ff6`4ae5388c mysqld!dispatch_command(enum_server_command command = <Value unavailable error>, class THD * thd = 0x0000029c`88bd5878, char * packet = 0x0000029c`8aa33399 "--- memory read error at address 0x0000029c`8aa33399 ---", unsigned int packet_length = 6, bool is_com_multi = false, bool is_next_command = false)+0x987 [D:\winx64-packages\build\src\sql\sql_parse.cc @ 1860] 000000cc`869ffd60 00007ff6`4af27bc2 mysqld!do_command(class THD * thd = 0x0000029c`88bd5878)+0x1bc [D:\winx64-packages\build\src\sql\sql_parse.cc @ 1378] 000000cc`869ffda0 00007ff6`4af29409 mysqld!do_handle_one_connection(class CONNECT * connect = 0x0000029c`fff75528)+0x202 [D:\winx64-packages\build\src\sql\sql_connect.cc @ 1420] 000000cc`869ffe10 00007ff6`4b28859c mysqld!handle_one_connection(void * arg = 0x0000029c`fff75528)+0x39 [D:\winx64-packages\build\src\sql\sql_connect.cc @ 1318] 000000cc`869ffe40 00007ffb`5b18268a mysqld!pthread_start(void * p = <Value unavailable error>)+0x1c [D:\winx64-packages\build\src\mysys\my_winthread.c @ 62] 000000cc`869ffe70 00007ffb`5bd07af4 ucrtbase!thread_start<unsigned int +0x3a 000000cc`869ffea0 00007ffb`5e8fa371 kernel32!BaseThreadInitThunk+0x14 000000cc`869ffed0 00000000`00000000 ntdll!RtlUserThreadStart+0x21 It appears that a write could be stuck in the operating system. The log writing is quite different in 10.6, thanks to MDEV-26789 . Also the extra complication of sync_array was removed from 10.6.

            Can you find out if any errors were reported by the operating system?

            marko Marko Mäkelä added a comment - Can you find out if any errors were reported by the operating system?

            People

              Richard Richard Stracke
              Richard Richard Stracke
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Git Integration

                  Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.