[MDEV-31566] All kinds of crash with RtlFreeHeap() Created: 2023-06-28  Updated: 2024-02-08

Status: In Review
Project: MariaDB Server
Component/s: None
Affects Version/s: 10.4.30
Fix Version/s: 10.4, 10.5, 10.6, 10.11, 11.0

Type: Bug Priority: Critical
Reporter: george he Assignee: Rucha Deodhar
Resolution: Unresolved Votes: 0
Labels: crash
Environment:

Windows Server 2008R2, Linux


 Description   

The server often crashes.
Each time the top error frame is RtlFreeHeap(), with different stack list.
The following are error stacks of version 10.4.30
Yet other softwares in the same server are stable.

1: exception 0xc0000005

 
ntdll.dll!RtlFreeHeap()
 
ucrtbase.DLL!_free_base()
 
mysqld.exe!dict_mem_table_free()[dict0mem.cc:248]
 
mysqld.exe!ibuf_rec_get_volume_func()[ibuf0ibuf.cc:1638]
 
mysqld.exe!ibuf_get_merge_page_nos_func()[ibuf0ibuf.cc:2306]
 
mysqld.exe!ibuf_merge_pages()[ibuf0ibuf.cc:2434]
 
mysqld.exe!ibuf_merge_in_background()[ibuf0ibuf.cc:2630]
 
mysqld.exe!srv_master_do_active_tasks()[srv0srv.cc:2115]
 
mysqld.exe!srv_master_thread()[srv0srv.cc:2349]
 
kernel32.dll!BaseThreadInitThunk()
 
ntdll.dll!RtlUserThreadStart()

2: exception 0xc0000005

 
ntdll.dll!RtlFreeHeap()
 
ucrtbase.DLL!_free_base()
 
mysqld.exe!Item_func_dyncol_json::val_str()[item_strfunc.cc:4802]
 
mysqld.exe!Item::save_str_in_field()[item.cc:6676]
 
mysqld.exe!Item::save_in_field()[item.cc:6724]
 
mysqld.exe!fill_record()[sql_base.cc:8932]
 
mysqld.exe!select_unit::send_data()[sql_union.cc:130]
 
mysqld.exe!end_send()[sql_select.cc:22057]
 
mysqld.exe!evaluate_join_record()[sql_select.cc:21090]
 
mysqld.exe!sub_select()[sql_select.cc:20901]
 
mysqld.exe!do_select()[sql_select.cc:20386]
 
mysqld.exe!JOIN::exec_inner()[sql_select.cc:4590]
 
mysqld.exe!JOIN::exec()[sql_select.cc:4373]
 
mysqld.exe!mysql_select()[sql_select.cc:4813]
 
mysqld.exe!mysql_derived_fill()[sql_derived.cc:1258]
 
mysqld.exe!mysql_handle_single_derived()[sql_derived.cc:200]
 
mysqld.exe!st_join_table::preread_init()[sql_select.cc:13838]
 
mysqld.exe!sub_select()[sql_select.cc:20826]
 
mysqld.exe!do_select()[sql_select.cc:20386]
 
mysqld.exe!JOIN::exec_inner()[sql_select.cc:4590]
 
mysqld.exe!JOIN::exec()[sql_select.cc:4373]
 
mysqld.exe!mysql_select()[sql_select.cc:4813]
 
mysqld.exe!handle_select()[sql_select.cc:442]
 
mysqld.exe!execute_sqlcom_select()[sql_parse.cc:6463]
 
mysqld.exe!mysql_execute_command()[sql_parse.cc:3966]
 
mysqld.exe!mysql_parse()[sql_parse.cc:8002]
 
mysqld.exe!dispatch_command()[sql_parse.cc:1860]
 
mysqld.exe!do_command()[sql_parse.cc:1378]
 
mysqld.exe!threadpool_process_request()[threadpool_common.cc:373]
 
mysqld.exe!tp_callback()[threadpool_common.cc:192]
 
ntdll.dll!RtlDllShutdownInProgress()
 
ntdll.dll!DbgUiRemoteBreakin()
 
kernel32.dll!BaseThreadInitThunk()
 
ntdll.dll!RtlUserThreadStart()

3: exception 0xc0000005

 
ntdll.dll!RtlFreeHeap()
 
ucrtbase.DLL!_free_base()
 
mysqld.exe!delete_dynamic()[array.c:305]
 
mysqld.exe!my_hash_free()[hash.c:159]
 
mysqld.exe!THD::cleanup()[sql_class.cc:1558]
 
mysqld.exe!THD::change_user()[sql_class.cc:1412]
 
mysqld.exe!dispatch_command()[sql_parse.cc:1709]
 
mysqld.exe!do_command()[sql_parse.cc:1378]
 
mysqld.exe!threadpool_process_request()[threadpool_common.cc:373]
 
mysqld.exe!tp_callback()[threadpool_common.cc:192]
 
ntdll.dll!RtlDllShutdownInProgress()
 
ntdll.dll!DbgUiRemoteBreakin()
 
kernel32.dll!BaseThreadInitThunk()
 
ntdll.dll!RtlUserThreadStart()



 Comments   
Comment by Marko Mäkelä [ 2023-06-28 ]

All stack traces are for cases when some dynamically allocated heap memory is being freed.

I suspect that the internal memory structures that ucrtbase.DLL or ntdll.dll uses for implementing primitives like malloc() and free() have been corrupted by a buffer overrun or a write through an invalid pointer. There are some open bugs mentioning AddressSanitizer diagnostics heap-use-after-free, heap-use-after-poison, heap-buffer-overflow.

I am not familiar with the debugging tools on Microsoft Windows, but I would expect there to be some debug version of the runtime library, which would help catch the actual culprit of the corruption, instead of seeing a crash in unrelated code later.

As always, an SQL test case for reproducing these failures would be helpful.

Comment by george he [ 2023-06-28 ]

Thanks for your reply!
I think that need a debug version of Mariadb which link to debug version of ucrtbase.DLL
Do you have such a version?

Comment by Oleg Smirnov [ 2023-06-28 ]

MariaDB supports Windows version in their active lifecycle (until the product reaches the Mainstream Support End Date). Unfortunately, Windows Server 2008R2 is far beyond that date. Is there any specific reason for using such an outdated version of the OS?

Comment by george he [ 2023-06-28 ]

Not upgraded just because I don't have the time, as it's a production server.
I just build a debug version and use it. Let's see if we can find the culprit.

Comment by Vladislav Vaintroub [ 2023-06-29 ]

I do not think debug compilation by itself is going to reveal much.

There are 2 options that can be more useful
1. Pageheap (no recompilation required)

gflags /p /enable mysqld.exe /full

execute the above in the elevated command line to enable pageheap on the mysqld, and restart the server. The effect is that program will crash when trying to access unallocated region, making culprit more obvious

when finished, execute
gflags /p /disable mysqld.exe

2. Address sanitizer compilation (works with relatively new VS2022)
cmake C:\path\to\source_dir "-DCMAKE_C_FLAGS=-fsanitize=address /W3" "-DCMAKE_CXX_FLAGS=-fsanitize=address /EHsc /W3" -DMSVC_CRT_TYPE=/MD

If you run it on a different machine then the one you compiled on, you'd need to copy
clang_rt.asan_dynamic-x86_64.dll from
C:\Program Files\Microsoft Visual Studio\2022\Community\VC\Tools\MSVC\14.36.32532\bin\Hostx64\x64\clang_rt.asan_dynamic-x86_64.dll' next to mysqld.exe (replace 14.36.32532 in this path with version of VS that you have on your compile host), or ensure that this clang_rt.asan_dynamic-x86_64.dll is in PATH

Don't use 1. and 2. at the same time, it would not work.
Both 1. and 2. are somewhat resource-intensive, and slow down the server thus unsure this could be run on a production host.

Comment by george he [ 2023-06-29 ]

Thank Vladislav Vaintroub!
I'll give it a try.

Comment by george he [ 2023-07-05 ]

Finally the reproducing sql is found. The following sql would always crash the server:

select column_json(0x0402000A0000000300030023076A736E7375626A6563742E0005006C0027000200290002002B0002002D0002002F0002000C31000C3B000C4B000C51000F62006631663266336634663509E5A79AE8BF9CE6B48B0FE8819AE9809AE98791E6A1A5E5BA970537343530301031313634332F393634352F31313630300C080000000000EFBFBDEFBFBD192E);

Could someone have a look at it? Thanks!

Comment by Oleg Smirnov [ 2023-07-05 ]

Can you attach the error call stack? I could not reproduce this on my Windows 10 machine with MariaDB 10.4.30.

Comment by Vladislav Vaintroub [ 2023-07-05 ]

Got this stack in ASAN build ( heap buffer overflow) trying to execute the query by reporter.

select column_json(0x0402000A0000000300030023076A736E7375626A6563742E0005006C0027000200290002002B0002002D0002002F0002000C31000C3B000C4B000C51000F62006631663266336634663509E5A79AE8BF9CE6B48B0FE8819AE9809AE98791E6A1A5E5BA970537343530301031313634332F393634352F31313630300C080000000000EFBFBDEFBFBD192E);

Mostly likely, the bug is portable , i.e can be reproduced outside of Windows, although I did not try

=================================================================
 
==40536==ERROR: AddressSanitizer: heap-buffer-overflow on address 0x11d7369209e0 at pc 0x7ff6a1985cae bp 0x007628ff7e50 sp 0x007628ff7e50
 
WRITE of size 1 at 0x11d7369209e0 thread T-1
 
    #0 0x7ff6a1985cad in dynstr_append_json_quoted C:\work\10.4\mysys\ma_dyncol.c:3847
 
    #1 0x7ff6a196e33a in mariadb_dyncol_val_str C:\work\10.4\mysys\ma_dyncol.c:3939
 
    #2 0x7ff6a1987104 in mariadb_dyncol_json_internal C:\work\10.4\mysys\ma_dyncol.c:4242
 
    #3 0x7ff6a196daaa in mariadb_dyncol_json C:\work\10.4\mysys\ma_dyncol.c:4265
 
    #4 0x7ff69ffc08ae in Item_func_dyncol_json::val_str(class String *) C:\work\10.4\sql\item_strfunc.cc:4791
 
    #5 0x7ff69f50767b in Type_handler::Item_send_str(class Item *, class Protocol *, struct st_value *) const C:\work\10.4\sql\sql_type.cc:7073
 
    #6 0x7ff69f541200 in Type_handler_string_result::Item_send(class Item *, class Protocol *, struct st_value *) const C:\work\10.4\sql\sql_type.h:4973
 
    #7 0x7ff69f583b92 in Item::send(class Protocol *, struct st_value *) C:\work\10.4\sql\item.h:1044
 
    #8 0x7ff69ffcbced in Protocol::send_result_set_row(class List<class Item> *) C:\work\10.4\sql\protocol.cc:1038
 
    #9 0x7ff69f732c6c in select_send::send_data(class List<class Item> &) C:\work\10.4\sql\sql_class.cc:3139
 
    #10 0x7ff69fe49032 in JOIN::exec_inner(void) C:\work\10.4\sql\sql_select.cc:4471
 
    #11 0x7ff69fe47b0c in JOIN::exec(void) C:\work\10.4\sql\sql_select.cc:4387
 
    #12 0x7ff69fe71387 in mysql_select(class THD *, struct TABLE_LIST *, unsigned int, class List<class Item> &, class Item *, unsigned int, struct st_order *, struct st_order *, class Item *, struct st_order *, unsigned __int64, class select_result *, class st_select_lex_unit *, class st_select_lex *) C:\work\10.4\sql\sql_select.cc:4826
 
    #13 0x7ff69fe7041d in handle_select(class THD *, struct LEX *, class select_result *, unsigned long) C:\work\10.4\sql\sql_select.cc:442
 
    #14 0x7ff6a002139a in execute_sqlcom_select C:\work\10.4\sql\sql_parse.cc:6473
 
    #15 0x7ff6a000318f in mysql_execute_command(class THD *) C:\work\10.4\sql\sql_parse.cc:3976
 
    #16 0x7ff69fffccd9 in mysql_parse(class THD *, char *, unsigned int, class Parser_state *, bool, bool) C:\work\10.4\sql\sql_parse.cc:8008
 
    #17 0x7ff6a001158f in dispatch_command(enum enum_server_command, class THD *, char *, unsigned int, bool, bool) C:\work\10.4\sql\sql_parse.cc:1857
 
    #18 0x7ff6a000f7ee in do_command(class THD *) C:\work\10.4\sql\sql_parse.cc:1378
 
    #19 0x7ff6a07bf729 in threadpool_process_request C:\work\10.4\sql\threadpool_common.cc:373
 
    #20 0x7ff6a07be952 in tp_callback(struct TP_connection *) C:\work\10.4\sql\threadpool_common.cc:192
 
    #21 0x7ff6a068c146 in tp_callback C:\work\10.4\sql\threadpool_win.cc:368
 
    #22 0x7ff6a068a0c1 in work_callback C:\work\10.4\sql\threadpool_win.cc:419
 
    #23 0x7ffd07812369  (C:\WINDOWS\SYSTEM32\ntdll.dll+0x180062369)
 
    #24 0x7ffd077e5975  (C:\WINDOWS\SYSTEM32\ntdll.dll+0x180035975)
 
    #25 0x7ffd05c926ac  (C:\WINDOWS\System32\KERNEL32.DLL+0x1800126ac)
 
    #26 0x7ffd0780a9f7  (C:\WINDOWS\SYSTEM32\ntdll.dll+0x18005a9f7)
 
 
 
0x11d7369209e0 is located 0 bytes to the right of 288-byte region [0x11d7369208c0,0x11d7369209e0)
 
allocated by thread T0 here:
 
    #0 0x7ffc63cbe988 in malloc D:\a\_work\1\s\src\vctools\asan\llvm\compiler-rt\lib\asan\asan_malloc_win.cpp:124
 
    #1 0x7ff6a19cb078 in my_malloc C:\work\10.4\mysys\my_malloc.c:101
 
    #2 0x7ff6a19873a2 in init_dynamic_string C:\work\10.4\mysys\string.c:39
 
    #3 0x7ff6a196da87 in mariadb_dyncol_json C:\work\10.4\mysys\ma_dyncol.c:4262
 
    #4 0x7ff69ffc08ae in Item_func_dyncol_json::val_str(class String *) C:\work\10.4\sql\item_strfunc.cc:4791
 
    #5 0x7ff69f50767b in Type_handler::Item_send_str(class Item *, class Protocol *, struct st_value *) const C:\work\10.4\sql\sql_type.cc:7073
 
    #6 0x7ff69f541200 in Type_handler_string_result::Item_send(class Item *, class Protocol *, struct st_value *) const C:\work\10.4\sql\sql_type.h:4973
 
    #7 0x7ff69f583b92 in Item::send(class Protocol *, struct st_value *) C:\work\10.4\sql\item.h:1044
 
    #8 0x7ff69ffcbced in Protocol::send_result_set_row(class List<class Item> *) C:\work\10.4\sql\protocol.cc:1038
 
    #9 0x7ff69f732c6c in select_send::send_data(class List<class Item> &) C:\work\10.4\sql\sql_class.cc:3139
 
    #10 0x7ff69fe49032 in JOIN::exec_inner(void) C:\work\10.4\sql\sql_select.cc:4471
 
    #11 0x7ff69fe47b0c in JOIN::exec(void) C:\work\10.4\sql\sql_select.cc:4387
 
    #12 0x7ff69fe71387 in mysql_select(class THD *, struct TABLE_LIST *, unsigned int, class List<class Item> &, class Item *, unsigned int, struct st_order *, struct st_order *, class Item *, struct st_order *, unsigned __int64, class select_result *, class st_select_lex_unit *, class st_select_lex *) C:\work\10.4\sql\sql_select.cc:4826
 
    #13 0x7ff69fe7041d in handle_select(class THD *, struct LEX *, class select_result *, unsigned long) C:\work\10.4\sql\sql_select.cc:442
 
    #14 0x7ff6a002139a in execute_sqlcom_select C:\work\10.4\sql\sql_parse.cc:6473
 
    #15 0x7ff6a000318f in mysql_execute_command(class THD *) C:\work\10.4\sql\sql_parse.cc:3976
 
    #16 0x7ff69fffccd9 in mysql_parse(class THD *, char *, unsigned int, class Parser_state *, bool, bool) C:\work\10.4\sql\sql_parse.cc:8008
 
    #17 0x7ff6a001158f in dispatch_command(enum enum_server_command, class THD *, char *, unsigned int, bool, bool) C:\work\10.4\sql\sql_parse.cc:1857
 
    #18 0x7ff6a000f7ee in do_command(class THD *) C:\work\10.4\sql\sql_parse.cc:1378
 
    #19 0x7ff6a07bf729 in threadpool_process_request C:\work\10.4\sql\threadpool_common.cc:373
 
    #20 0x7ff6a07be952 in tp_callback(struct TP_connection *) C:\work\10.4\sql\threadpool_common.cc:192
 
    #21 0x7ff6a068c146 in tp_callback C:\work\10.4\sql\threadpool_win.cc:368
 
    #22 0x7ff6a068a0c1 in work_callback C:\work\10.4\sql\threadpool_win.cc:419
 
    #23 0x7ffd07812369  (C:\WINDOWS\SYSTEM32\ntdll.dll+0x180062369)
 
    #24 0x7ffd077e5975  (C:\WINDOWS\SYSTEM32\ntdll.dll+0x180035975)
 
    #25 0x7ffd05c926ac  (C:\WINDOWS\System32\KERNEL32.DLL+0x1800126ac)
 
    #26 0x7ffd0780a9f7  (C:\WINDOWS\SYSTEM32\ntdll.dll+0x18005a9f7)
 
 
 
SUMMARY: AddressSanitizer: heap-buffer-overflow C:\work\10.4\mysys\ma_dyncol.c:3847 in dynstr_append_json_quoted
 
Shadow bytes around the buggy address:
 
  0x03ee1d5a40e0: fa fa fa fa fa fa fa fa 00 00 00 00 f7 00 00 00
 
  0x03ee1d5a40f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
 
  0x03ee1d5a4100: 00 00 00 00 00 00 00 00 00 00 00 00 00 fa fa fa
 
  0x03ee1d5a4110: fa fa fa fa fa fa fa fa 00 00 00 00 00 00 00 00
 
  0x03ee1d5a4120: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
 
=>0x03ee1d5a4130: 00 00 00 00 00 00 00 00 00 00 00 00[fa]fa fa fa
 
  0x03ee1d5a4140: fa fa fa fa fa fa fa fa fd fd fd fd fd fd fd fd
 
  0x03ee1d5a4150: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
 
  0x03ee1d5a4160: fd fd fd fd fd fd fd fd fd fa fa fa fa fa fa fa
 
  0x03ee1d5a4170: fa fa fa fa fa fa fa fa fd fd fd fd fd fd fd fd
 
  0x03ee1d5a4180: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
 
Shadow byte legend (one shadow byte represents 8 application bytes):
 
  Addressable:           00
 
  Partially addressable: 01 02 03 04 05 06 07
 
  Heap left redzone:       fa
 
  Freed heap region:       fd
 
  Stack left redzone:      f1
 
  Stack mid redzone:       f2
 
  Stack right redzone:     f3
 
  Stack after return:      f5
 
  Stack use after scope:   f8
 
  Global redzone:          f9
 
  Global init order:       f6
 
  Poisoned by user:        f7
 
  Container overflow:      fc
 
  Array cookie:            ac
 
  Intra object redzone:    bb
 
  ASan internal:           fe
 
  Left alloca redzone:     ca
 
  Right alloca redzone:    cb
 
==40536==ABORTING

Comment by Vladislav Vaintroub [ 2023-07-05 ]

yes, same thing with Linux/ASAN, trying

select column_json(0x0402000A0000000300030023076A736E7375626A6563742E0005006C0027000200290002002B0002002D0002002F0002000C31000C3B000C4B000C51000F62006631663266336634663509E5A79AE8BF9CE6B48B0FE8819AE9809AE98791E6A1A5E5BA970537343530301031313634332F393634352F31313630300C080000000000EFBFBDEFBFBD192E)

crashes on the same line

=================================================================
 
==10481==ERROR: AddressSanitizer: heap-buffer-overflow on address 0x612000023860 at pc 0x7f42dcedb246 bp 0x7f42cc87d2a0 sp 0x7f42cc87d298
 
WRITE of size 1 at 0x612000023860 thread T5
 
    #0 0x7f42dcedb245 in dynstr_append_json_quoted /mnt/c/work/10.4/mysys/ma_dyncol.c:3847
 
    #1 0x7f42dcedbcad in mariadb_dyncol_val_str /mnt/c/work/10.4/mysys/ma_dyncol.c:3939
 
    #2 0x7f42dceddd68 in mariadb_dyncol_json_internal /mnt/c/work/10.4/mysys/ma_dyncol.c:4241
 
    #3 0x7f42dceddf72 in mariadb_dyncol_json /mnt/c/work/10.4/mysys/ma_dyncol.c:4265
 
    #4 0x7f42dbe1534d in Item_func_dyncol_json::val_str(String*) /mnt/c/work/10.4/sql/item_strfunc.cc:4791
 
    #5 0x7f42db9fdbad in Type_handler::Item_send_str(Item*, Protocol*, st_value*) const /mnt/c/work/10.4/sql/sql_type.cc:7073
 
    #6 0x7f42dba13219 in Type_handler_string_result::Item_send(Item*, Protocol*, st_value*) const /mnt/c/work/10.4/sql/sql_type.h:4973
 
    #7 0x7f42db28afb1 in Item::send(Protocol*, st_value*) /mnt/c/work/10.4/sql/item.h:1044
 
    #8 0x7f42db27e18a in Protocol::send_result_set_row(List<Item>*) /mnt/c/work/10.4/sql/protocol.cc:1038
 
    #9 0x7f42db404cad in select_send::send_data(List<Item>&) /mnt/c/work/10.4/sql/sql_class.cc:3139
 
    #10 0x7f42db5c6e41 in JOIN::exec_inner() /mnt/c/work/10.4/sql/sql_select.cc:4473
 
    #11 0x7f42db5c5b13 in JOIN::exec() /mnt/c/work/10.4/sql/sql_select.cc:4387
 
    #12 0x7f42db5c971c in mysql_select(THD*, TABLE_LIST*, unsigned int, List<Item>&, Item*, unsigned int, st_order*, st_order*, Item*, st_order*, unsigned long long, select_result*, st_select_lex_unit*, st_select_lex*) /mnt/c/work/10.4/sql/sql_select.cc:4826
 
    #13 0x7f42db59e063 in handle_select(THD*, LEX*, select_result*, unsigned long) /mnt/c/work/10.4/sql/sql_select.cc:454
 
    #14 0x7f42db51bf0f in execute_sqlcom_select /mnt/c/work/10.4/sql/sql_parse.cc:6473
 
    #15 0x7f42db50addc in mysql_execute_command(THD*) /mnt/c/work/10.4/sql/sql_parse.cc:3976
 
    #16 0x7f42db52459f in mysql_parse(THD*, char*, unsigned int, Parser_state*, bool, bool) /mnt/c/work/10.4/sql/sql_parse.cc:8008
 
    #17 0x7f42db4fe234 in dispatch_command(enum_server_command, THD*, char*, unsigned int, bool, bool) /mnt/c/work/10.4/sql/sql_parse.cc:1858
 
    #18 0x7f42db4fb121 in do_command(THD*) /mnt/c/work/10.4/sql/sql_parse.cc:1379
 
    #19 0x7f42db8a4591 in do_handle_one_connection(CONNECT*) /mnt/c/work/10.4/sql/sql_connect.cc:1420
 
    #20 0x7f42db8a3ea3 in handle_one_connection /mnt/c/work/10.4/sql/sql_connect.cc:1324
 
    #21 0x7f42dc4af43f in pfs_spawn_thread /mnt/c/work/10.4/storage/perfschema/pfs.cc:1869
 
    #22 0x7f42d7ec74a3 in start_thread (/lib/x86_64-linux-gnu/libpthread.so.0+0x74a3)
 
    #23 0x7f42d7138d0e in __clone (/lib/x86_64-linux-gnu/libc.so.6+0xe8d0e)
 
 
 
0x612000023860 is located 0 bytes to the right of 288-byte region [0x612000023740,0x612000023860)
 
allocated by thread T5 here:
 
    #0 0x7f42d8f21d28 in malloc (/usr/lib/x86_64-linux-gnu/libasan.so.3+0xc1d28)
 
    #1 0x7f42dce92b95 in my_malloc /mnt/c/work/10.4/mysys/my_malloc.c:101
 
    #2 0x7f42dcea138b in init_dynamic_string /mnt/c/work/10.4/mysys/string.c:39
 
    #3 0x7f42dceddf4f in mariadb_dyncol_json /mnt/c/work/10.4/mysys/ma_dyncol.c:4262
 
    #4 0x7f42dbe1534d in Item_func_dyncol_json::val_str(String*) /mnt/c/work/10.4/sql/item_strfunc.cc:4791
 
    #5 0x7f42db9fdbad in Type_handler::Item_send_str(Item*, Protocol*, st_value*) const /mnt/c/work/10.4/sql/sql_type.cc:7073
 
    #6 0x7f42dba13219 in Type_handler_string_result::Item_send(Item*, Protocol*, st_value*) const /mnt/c/work/10.4/sql/sql_type.h:4973
 
    #7 0x7f42db28afb1 in Item::send(Protocol*, st_value*) /mnt/c/work/10.4/sql/item.h:1044
 
    #8 0x7f42db27e18a in Protocol::send_result_set_row(List<Item>*) /mnt/c/work/10.4/sql/protocol.cc:1038
 
    #9 0x7f42db404cad in select_send::send_data(List<Item>&) /mnt/c/work/10.4/sql/sql_class.cc:3139
 
    #10 0x7f42db5c6e41 in JOIN::exec_inner() /mnt/c/work/10.4/sql/sql_select.cc:4473
 
    #11 0x7f42db5c5b13 in JOIN::exec() /mnt/c/work/10.4/sql/sql_select.cc:4387
 
    #12 0x7f42db5c971c in mysql_select(THD*, TABLE_LIST*, unsigned int, List<Item>&, Item*, unsigned int, st_order*, st_order*, Item*, st_order*, unsigned long long, select_result*, st_select_lex_unit*, st_select_lex*) /mnt/c/work/10.4/sql/sql_select.cc:4826
 
    #13 0x7f42db59e063 in handle_select(THD*, LEX*, select_result*, unsigned long) /mnt/c/work/10.4/sql/sql_select.cc:454
 
    #14 0x7f42db51bf0f in execute_sqlcom_select /mnt/c/work/10.4/sql/sql_parse.cc:6473
 
    #15 0x7f42db50addc in mysql_execute_command(THD*) /mnt/c/work/10.4/sql/sql_parse.cc:3976
 
    #16 0x7f42db52459f in mysql_parse(THD*, char*, unsigned int, Parser_state*, bool, bool) /mnt/c/work/10.4/sql/sql_parse.cc:8008
 
    #17 0x7f42db4fe234 in dispatch_command(enum_server_command, THD*, char*, unsigned int, bool, bool) /mnt/c/work/10.4/sql/sql_parse.cc:1858
 
    #18 0x7f42db4fb121 in do_command(THD*) /mnt/c/work/10.4/sql/sql_parse.cc:1379
 
    #19 0x7f42db8a4591 in do_handle_one_connection(CONNECT*) /mnt/c/work/10.4/sql/sql_connect.cc:1420
 
    #20 0x7f42db8a3ea3 in handle_one_connection /mnt/c/work/10.4/sql/sql_connect.cc:1324
 
    #21 0x7f42dc4af43f in pfs_spawn_thread /mnt/c/work/10.4/storage/perfschema/pfs.cc:1869
 
    #22 0x7f42d7ec74a3 in start_thread (/lib/x86_64-linux-gnu/libpthread.so.0+0x74a3)
 
 
 
Thread T5 created by T0 here:
 
    #0 0x7f42d8e90f59 in __interceptor_pthread_create (/usr/lib/x86_64-linux-gnu/libasan.so.3+0x30f59)
 
    #1 0x7f42dc4af82c in spawn_thread_v1 /mnt/c/work/10.4/storage/perfschema/pfs.cc:1919
 
    #2 0x7f42db244f92 in inline_mysql_thread_create /mnt/c/work/10.4/include/mysql/psi/mysql_thread.h:1275
 
    #3 0x7f42db2598e6 in create_thread_to_handle_connection(CONNECT*) /mnt/c/work/10.4/sql/mysqld.cc:6287
 
    #4 0x7f42db259fc9 in create_new_thread(CONNECT*) /mnt/c/work/10.4/sql/mysqld.cc:6357
 
    #5 0x7f42db25a356 in handle_accepted_socket(st_mysql_socket, st_mysql_socket) /mnt/c/work/10.4/sql/mysqld.cc:6455
 
    #6 0x7f42db25afc7 in handle_connections_sockets() /mnt/c/work/10.4/sql/mysqld.cc:6613
 
    #7 0x7f42db259148 in mysqld_main(int, char**) /mnt/c/work/10.4/sql/mysqld.cc:5945
 
    #8 0x7f42db242f3f in main /mnt/c/work/10.4/sql/main.cc:25
 
    #9 0x7f42d70702e0 in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x202e0)
 
 
 
SUMMARY: AddressSanitizer: heap-buffer-overflow /mnt/c/work/10.4/mysys/ma_dyncol.c:3847 in dynstr_append_json_quoted
 
Shadow bytes around the buggy address:
 
  0x0c247fffc6b0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
 
  0x0c247fffc6c0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
 
  0x0c247fffc6d0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
 
  0x0c247fffc6e0: fa fa fa fa fa fa fa fa 00 00 00 00 00 00 00 00
 
  0x0c247fffc6f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
 
=>0x0c247fffc700: 00 00 00 00 00 00 00 00 00 00 00 00[fa]fa fa fa
 
  0x0c247fffc710: fa fa fa fa fa fa fa fa fd fd fd fd fd fd fd fd
 
  0x0c247fffc720: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
 
  0x0c247fffc730: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fa fa
 
  0x0c247fffc740: fa fa fa fa fa fa fa fa 00 00 00 00 00 00 00 00
 
  0x0c247fffc750: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
 
Shadow byte legend (one shadow byte represents 8 application bytes):
 
  Addressable:           00
 
  Partially addressable: 01 02 03 04 05 06 07 
  Heap left redzone:       fa
 
  Heap right redzone:      fb
 
  Freed heap region:       fd
 
  Stack left redzone:      f1
 
  Stack mid redzone:       f2
 
  Stack right redzone:     f3
 
  Stack partial redzone:   f4
 
  Stack after return:      f5
 
  Stack use after scope:   f8
 
  Global redzone:          f9
 
  Global init order:       f6
 
  Poisoned by user:        f7
 
  Container overflow:      fc
 
  Array cookie:            ac
 
  Intra object redzone:    bb
 
  ASan internal:           fe
 
  Left alloca redzone:     ca
 
  Right alloca redzone:    cb
 
==10481==ABORTING

Comment by george he [ 2023-11-13 ]

How is this going on? The server crashes now and then.

Comment by george he [ 2023-12-03 ]

Finally I find the bug is buffer overrun, and initiate the pull request #2892 to fix it.

Comment by Daniel Black [ 2024-02-08 ]

crashing bug, ready to review.

Generated at Thu Feb 08 10:24:50 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.