[MDEV-24901] SIGSEGV in fts_get_table_name, SIGSEGV in ib_vector_size, SIGSEGV in row_merge_fts_doc_tokenize, stack smashing Created: 2021-02-17  Updated: 2021-11-08  Resolved: 2021-10-29

Status: Closed
Project: MariaDB Server
Component/s: Character Sets, Full-text Search, Storage Engine - InnoDB
Affects Version/s: 10.2, 10.3, 10.4, 10.5, 10.6, 10.7
Fix Version/s: 10.2.42, 10.3.33, 10.4.23, 10.5.14, 10.6.6, 10.7.2

Type: Bug Priority: Critical
Reporter: Roel Van de Paar Assignee: Alexander Barkov
Resolution: Fixed Votes: 0
Labels: affects-tests, corruption, rr-profile, stack-smashing


 Description   

SET collation_connection='tis620_bin';
SET @@session.character_set_server='tis620';
CREATE DATABASE a;
USE a;
CREATE TABLE t(c TEXT,FULLTEXT KEY f(c)) ENGINE=InnoDB;
INSERT INTO t VALUES(100);
ALTER TABLE t ADD (c2 INT);

Leads to:

10.6.0 bfb4761ca04704d68dba51f76d7c9967f880a6ee (Debug)

Core was generated by `/test/MD110221-mariadb-10.6.0-linux-x86_64-dbg/bin/mysqld --no-defaults --core-'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0  0x000014ed559bbc30 in ?? () from /lib/x86_64-linux-gnu/libgcc_s.so.1
[Current thread is 1 (Thread 0x14ed31bfd700 (LWP 1630450))]
(gdb) bt
#0  0x000014ed559bbc30 in ?? () from /lib/x86_64-linux-gnu/libgcc_s.so.1
#1  0x000014ed559bd76b in _Unwind_Backtrace () from /lib/x86_64-linux-gnu/libgcc_s.so.1
#2  0x000014ed55c4b136 in __GI___backtrace (array=array@entry=0x14ed31bfbde0, size=size@entry=128) at backtrace.c:116
#3  0x000055817622e76d in my_print_stacktrace (stack_bottom=0x0, thread_stack=299008, silent=silent@entry=0 '\000') at /test/10.6_dbg/mysys/stacktrace.c:212
#4  0x00005581759c6221 in handle_fatal_signal (sig=11) at /test/10.6_dbg/sql/signal_handler.cc:208
#5  <signal handler called>
#6  0x0000558175efb556 in row_merge_fts_doc_tokenize (t_ctx=0x14ed31bfca80, opt_doc_id_size=<optimized out>, merge_file=<optimized out>, doc=0x14ed31bfc9d0, doc_id=<optimized out>, sort_buf=<optimized out>) at /test/10.6_dbg/storage/innobase/row/row0ftsort.cc:577
#7  fts_parallel_tokenization (arg=<optimized out>) at /test/10.6_dbg/storage/innobase/row/row0ftsort.cc:839
#8  0xb7c336e496240000 in ?? ()
#9  0x112e0be826d694b3 in ?? ()
#10 0x00005581761b7cc7 in std::condition_variable::__wait_until_impl<std::chrono::duration<long, std::ratio<1l, 1000000000l> > > (__atime=<synthetic pointer>: <optimized out>, __lock=<error reading variable: Cannot access memory at address 0x166476a9e69be61c>, this=0x558178a5a230) at /usr/include/x86_64-linux-gnu/c++/9/bits/gthr-default.h:872
#11 std::condition_variable::wait_until<std::chrono::_V2::steady_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > > (__atime=<optimized out>, __lock=<error reading variable: Cannot access memory at address 0x166476a9e69be61c>, this=0x558178a5a230) at /usr/include/c++/9/condition_variable:121
#12 std::condition_variable::wait_for<long, std::ratio<1l, 1000l> > (__rtime=@0x14ed31bfcea8: {__r = 0}, __lock=<error reading variable: Cannot access memory at address 0x166476a9e69be61c>, this=0x558178a5a230) at /usr/include/c++/9/condition_variable:152
#13 tpool::thread_pool_generic::wait_for_tasks (this=0x14ed31bfcd70, lk=<error reading variable: Cannot access memory at address 0x166476a9e69be61c>, thread_data=0x558178a5a230) at /test/10.6_dbg/tpool/tpool_generic.cc:446
Backtrace stopped: Cannot access memory at address 0x166476a9e69be674

10.2.37 (Debug)

Core was generated by `/test/MD260121-mariadb-10.2.37-linux-x86_64-dbg/bin/mysqld --no-defaults --core'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0  __pthread_kill (threadid=<optimized out>, signo=signo@entry=11)
    at ../sysdeps/unix/sysv/linux/pthread_kill.c:56
[Current thread is 1 (Thread 0x14d6d6cb5700 (LWP 1640311))]
(gdb) bt
#0  __pthread_kill (threadid=<optimized out>, signo=signo@entry=11) at ../sysdeps/unix/sysv/linux/pthread_kill.c:56
#1  0x000056162b81d087 in my_write_core (sig=sig@entry=11) at /data/builds/10.2_dbg/mysys/stacktrace.c:382
#2  0x000056162b114a91 in handle_fatal_signal (sig=11) at /data/builds/10.2_dbg/sql/signal_handler.cc:343
#3  <signal handler called>
#4  ib_vector_size (vec=0x0) at /data/builds/10.2_dbg/storage/innobase/include/ut0vec.ic:118
#5  fts_sync_write_words (unlock_cache=<optimized out>, index_cache=0x14d690046a10, trx=<optimized out>) at /data/builds/10.2_dbg/storage/innobase/fts/fts0fts.cc:4005
#6  fts_sync_index (sync=<optimized out>, index_cache=0x14d690046a10) at /data/builds/10.2_dbg/storage/innobase/fts/fts0fts.cc:4107
#7  0x000056162b81c033 in my_thread_var_dbug () at /data/builds/10.2_dbg/mysys/my_thr_init.c:444
#8  0x000056162b835054 in code_state () at /data/builds/10.2_dbg/dbug/dbug.c:375
#9  0x000014d690046680 in ?? ()
#10 0x0000000000000000 in ?? ()

10.2.37 (Optimized)

Core was generated by `/test/MD260121-mariadb-10.2.37-linux-x86_64-opt/bin/mysqld --no-defaults --core'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0  __pthread_kill (threadid=<optimized out>, signo=signo@entry=11)
    at ../sysdeps/unix/sysv/linux/pthread_kill.c:56
[Current thread is 1 (Thread 0x14f24e86d700 (LWP 1639811))]
(gdb) bt
#0  __pthread_kill (threadid=<optimized out>, signo=signo@entry=11) at ../sysdeps/unix/sysv/linux/pthread_kill.c:56
#1  0x000055a6ea4cd57f in my_write_core (sig=sig@entry=11) at /data/builds/10.2_opt/mysys/stacktrace.c:382
#2  0x000055a6e9f748a8 in handle_fatal_signal (sig=11) at /data/builds/10.2_opt/sql/signal_handler.cc:343
#3  <signal handler called>
#4  __memmove_avx_unaligned_erms () at ../sysdeps/x86_64/multiarch/memmove-vec-unaligned-erms.S:440
#5  0x000055a6ea323b5a in memcpy (__len=18446721043960435969, __src=0x14f20803eb00, __dest=0x14f24e86c560) at /usr/include/x86_64-linux-gnu/bits/string_fortified.h:34
#6  fts_get_table_name (fts_table=fts_table@entry=0x14f24e86c7f0, table_name=table_name@entry=0x14f24e86c560 "", dict_locked=dict_locked@entry=false) at /data/builds/10.2_opt/storage/innobase/fts/fts0sql.cc:124
#7  0x000055a6ea30b662 in fts_write_node (trx=0x14f208039390, graph=0x14f2080393f0, fts_table=0x14f24e86c7f0, word=0x14f208057f10, node=0x14f208039900) at /data/builds/10.2_opt/storage/innobase/fts/fts0fts.cc:3857
#8  0x000055a6ea30b8d5 in fts_sync_write_words (unlock_cache=<optimized out>, index_cache=0x14f208039390, trx=<optimized out>) at /data/builds/10.2_opt/storage/innobase/fts/fts0fts.cc:4023
#9  fts_sync_index (sync=<optimized out>, index_cache=0x14f208039390) at /data/builds/10.2_opt/storage/innobase/fts/fts0fts.cc:4107
#10 0x0000000000000000 in ?? ()

10.4.18 e626f511f9dc4faee9ae98fb5a8c8c6ddd06679b (Optimized)

Core was generated by `/test/MD260121-mariadb-10.4.18-linux-x86_64-opt/bin/mysqld --no-defaults --core'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0  0x000055c0e45b6823 in my_read (Filedes=Filedes@entry=70, 
    Buffer=Buffer@entry=0x14bef8ee22f0 "Limit", ' ' <repeats 21 times>, "Soft Limit", ' ' <repeats 11 times>, "Hard Limit", ' ' <repeats 11 times>, "Units     \nMax cpu time", ' ' <repeats 14 times>, "unlimited", ' ' <repeats 12 times>, "unlimited", ' ' <repeats 12 times>, "seconds   \nMax file size", ' ' <repeats 13 times>, "unlimited       "..., Count=Count@entry=4096, MyFlags=MyFlags@entry=0)
    at /data/builds/10.4_opt/mysys/my_read.c:63
[Current thread is 1 (Thread 0x14bef8ee4700 (LWP 1644189))]
(gdb) bt
#0  0x000055c0e45b6823 in my_read (Filedes=Filedes@entry=70, Buffer=Buffer@entry=0x14bef8ee22f0 "Limit", ' ' <repeats 21 times>, "Soft Limit", ' ' <repeats 11 times>, "Hard Limit", ' ' <repeats 11 times>, "Units     \nMax cpu time", ' ' <repeats 14 times>, "unlimited", ' ' <repeats 12 times>, "unlimited", ' ' <repeats 12 times>, "seconds   \nMax file size", ' ' <repeats 13 times>, "unlimited       "..., Count=Count@entry=4096, MyFlags=MyFlags@entry=0) at /data/builds/10.4_opt/mysys/my_read.c:63
#1  0x000055c0e3fc870a in output_core_info () at /data/builds/10.4_opt/sql/signal_handler.cc:66
#2  0x000055c0e3fc8b4e in handle_fatal_signal (sig=11) at /data/builds/10.4_opt/sql/signal_handler.cc:339
#3  <signal handler called>
#4  row_merge_fts_doc_tokenize (t_ctx=0x14bef8ee3b90, opt_doc_id_size=<optimized out>, merge_file=<optimized out>, doc=0x14bef8ee3ae0, doc_id=<optimized out>, sort_buf=<optimized out>) at /data/builds/10.4_opt/storage/innobase/row/row0ftsort.cc:586
#5  fts_parallel_tokenization (arg=<optimized out>) at /data/builds/10.4_opt/storage/innobase/row/row0ftsort.cc:854
#6  0x0000000000000000 in ?? ()

Note the various errors reading the backtrace. Some corruption going on.

Bug confirmed present in:
MariaDB: 10.2.37 (dbg), 10.2.37 (opt), 10.3.28 (dbg), 10.3.28 (opt), 10.4.18 (dbg), 10.4.18 (opt), 10.5.9 (dbg), 10.5.9 (opt), 10.6.0 (dbg), 10.6.0 (opt)

Bug (or feature/syntax) confirmed not present in:
MySQL: 5.5.62 (dbg), 5.5.62 (opt), 5.6.51 (dbg), 5.6.51 (opt), 5.7.33 (dbg), 5.7.33 (opt), 8.0.23 (dbg), 8.0.23 (opt)



 Comments   
Comment by Roel Van de Paar [ 2021-02-17 ]

Unique ID's seen so far

SIGSEGV|fts_get_table_name|fts_write_node|fts_sync_write_words|fts_sync_index
SIGSEGV|ib_vector_size|fts_sync_write_words|fts_sync_index|my_thread_var_dbug
SIGSEGV|row_merge_fts_doc_tokenize|fts_parallel_tokenization
SIGSEGV|row_merge_fts_doc_tokenize|fts_parallel_tokenization|std::condition_variable::__wait_until_impl<std::chrono::duration<long, std::ratio<1l, 1000000000l> > >|std::condition_variable::wait_until<std::chrono::_V2::steady_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >
SIGSEGV|row_merge_fts_doc_tokenize|fts_parallel_tokenization|purge_coordinator_callback|__GI___pthread_mutex_lock
SIGSEGV|row_merge_fts_doc_tokenize|fts_parallel_tokenization|__gthread_mutex_unlock|std::mutex::unlock
SIGSEGV|fts_parallel_tokenization|__gthread_mutex_unlock|std::mutex::unlock|std::unique_lock<std::mutex>::unlock

Comment by Roel Van de Paar [ 2021-02-17 ]

Yet other versions (different revisions) give other stacks. Please also try with this testcase

SET collation_connection='tis620_bin';
SET @@session.character_set_server='tis620';
CREATE DATABASE a;
USE a
CREATE TABLE t1 (col text, FULLTEXT KEY full_text (col)) ENGINE = InnoDB;
INSERT INTO t1 VALUES(7693);
ALTER TABLE t1 ADD (col2 varchar(100) character set latin1);

On different 10.6 revision, this leads to:

10.6.0 786bc312b85e58857cb26a24ab6e997ba0fdfc32 (Optimized)

Core was generated by `/test/MD100221-mariadb-10.6.0-linux-x86_64-opt/bin/mysqld --no-defaults --core-'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0  0x000014addcb7fc30 in ?? () from /lib/x86_64-linux-gnu/libgcc_s.so.1
[Current thread is 1 (Thread 0x14adb8df0700 (LWP 1989747))]
(gdb) bt
#0  0x000014addcb7fc30 in ?? () from /lib/x86_64-linux-gnu/libgcc_s.so.1
#1  0x000014addcb8176b in _Unwind_Backtrace () from /lib/x86_64-linux-gnu/libgcc_s.so.1
#2  0x000014addce0f136 in __GI___backtrace (array=array@entry=0x14adb8deee60, size=size@entry=128) at backtrace.c:116
#3  0x0000557ff6ab3fb2 in my_print_stacktrace (stack_bottom=0x0, thread_stack=299008, silent=silent@entry=0 '\000') at /test/10.6_opt/mysys/stacktrace.c:212
#4  0x0000557ff6529505 in handle_fatal_signal (sig=11) at /test/10.6_opt/sql/signal_handler.cc:208
#5  <signal handler called>
#6  0x0000557ff68e30bc in row_merge_fts_doc_tokenize (t_ctx=0x14adb8defa90, opt_doc_id_size=<optimized out>, merge_file=<optimized out>, doc=0x14adb8def9e0, doc_id=<optimized out>, sort_buf=<optimized out>) at /test/10.6_opt/storage/innobase/row/row0ftsort.cc:577
#7  fts_parallel_tokenization (arg=<optimized out>) at /test/10.6_opt/storage/innobase/row/row0ftsort.cc:839
#8  0x0000000008e454d6 in ?? ()
#9  0x0000557ff8aa4170 in ?? ()
#10 0x3943687e3e627300 in ?? ()
#11 0x000014adb8defd40 in ?? ()
#12 0x0000557ff6a46f77 in std::condition_variable::__wait_until_impl<std::chrono::duration<long, std::ratio<1l, 1000000000l> > > (__atime=<synthetic pointer>: <optimized out>, __lock=<error reading variable: Cannot access memory at address 0x602cc22f>, this=0x557ff7cb62d4 <purge_worker_task+116>) at /usr/include/x86_64-linux-gnu/c++/9/bits/gthr-default.h:872
#13 std::condition_variable::wait_until<std::chrono::_V2::steady_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > > (__atime=<optimized out>, __lock=<error reading variable: Cannot access memory at address 0x602cc22f>, this=0x557ff7cb62d4 <purge_worker_task+116>) at /usr/include/c++/9/condition_variable:121
#14 std::condition_variable::wait_for<long, std::ratio<1l, 1000l> > (__rtime=@0x14adb8defeb8: {__r = 0}, __lock=<error reading variable: Cannot access memory at address 0x602cc22f>, this=0x557ff7cb62d4 <purge_worker_task+116>) at /usr/include/c++/9/condition_variable:152
#15 tpool::thread_pool_generic::wait_for_tasks (this=0x14adb8defd80, lk=<error reading variable: Cannot access memory at address 0x602cc22f>, thread_data=0x557ff7cb62d4 <purge_worker_task+116>) at /test/10.6_opt/tpool/tpool_generic.cc:446
Backtrace stopped: Cannot access memory at address 0x602cc287

Comment by Roel Van de Paar [ 2021-02-17 ]

Please also try with:

SET collation_connection='tis620_bin';
SET @session_start_value=@@session.character_set_connection;
SET @@session.character_set_server=@session_start_value;
CREATE DATABASE `�\�\�\`;
USE `�\�\�\`;
CREATE TABLE t(col text,FULLTEXT KEY fullte (col));
INSERT INTO t VALUES(7693);
ALTER TABLE t ADD(col2 CHAR (100));

Leads to:

10.4.18 e626f511f9dc4faee9ae98fb5a8c8c6ddd06679b (Optimized)

Core was generated by `/test/MD260121-mariadb-10.4.18-linux-x86_64-opt/bin/mysqld --no-defaults --core'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0  0x000055d0f7afb823 in my_read (Filedes=Filedes@entry=70, 
    Buffer=Buffer@entry=0x14e830eea2f0 "Limit", ' ' <repeats 21 times>, "Soft Limit", ' ' <repeats 11 times>, "Hard Limit", ' ' <repeats 11 times>, "Units     \nMax cpu time", ' ' <repeats 14 times>, "unlimited", ' ' <repeats 12 times>, "unlimited", ' ' <repeats 12 times>, "seconds   \nMax file size", ' ' <repeats 13 times>, "unlimited       "..., Count=Count@entry=4096, MyFlags=MyFlags@entry=0)
    at /data/builds/10.4_opt/mysys/my_read.c:63
[Current thread is 1 (Thread 0x14e830eec700 (LWP 2765384))]
(gdb) bt
#0  0x000055d0f7afb823 in my_read (Filedes=Filedes@entry=70, Buffer=Buffer@entry=0x14e830eea2f0 "Limit", ' ' <repeats 21 times>, "Soft Limit", ' ' <repeats 11 times>, "Hard Limit", ' ' <repeats 11 times>, "Units     \nMax cpu time", ' ' <repeats 14 times>, "unlimited", ' ' <repeats 12 times>, "unlimited", ' ' <repeats 12 times>, "seconds   \nMax file size", ' ' <repeats 13 times>, "unlimited       "..., Count=Count@entry=4096, MyFlags=MyFlags@entry=0) at /data/builds/10.4_opt/mysys/my_read.c:63
#1  0x000055d0f750d70a in output_core_info () at /data/builds/10.4_opt/sql/signal_handler.cc:66
#2  0x000055d0f750db4e in handle_fatal_signal (sig=11) at /data/builds/10.4_opt/sql/signal_handler.cc:339
#3  <signal handler called>
#4  row_merge_fts_doc_tokenize (t_ctx=0x14e830eebb90, opt_doc_id_size=<optimized out>, merge_file=<optimized out>, doc=0x14e830eebae0, doc_id=<optimized out>, sort_buf=<optimized out>) at /data/builds/10.4_opt/storage/innobase/row/row0ftsort.cc:586
#5  fts_parallel_tokenization (arg=<optimized out>) at /data/builds/10.4_opt/storage/innobase/row/row0ftsort.cc:854
#6  0x0000000000000000 in ?? ()

10.6.0 bfb4761ca04704d68dba51f76d7c9967f880a6ee (Debug)

Core was generated by `/test/MD110221-mariadb-10.6.0-linux-x86_64-dbg/bin/mysqld --no-defaults --core-'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0  0x000014573a718c30 in ?? () from /lib/x86_64-linux-gnu/libgcc_s.so.1
[Current thread is 1 (Thread 0x145714cf6700 (LWP 2772568))]
(gdb) bt
#0  0x000014573a718c30 in ?? () from /lib/x86_64-linux-gnu/libgcc_s.so.1
#1  0x000014573a71a76b in _Unwind_Backtrace () from /lib/x86_64-linux-gnu/libgcc_s.so.1
#2  0x000014573a9a8136 in __GI___backtrace (array=array@entry=0x145714cf4de0, size=size@entry=128) at backtrace.c:116
#3  0x000055e26d0af76d in my_print_stacktrace (stack_bottom=0x0, thread_stack=299008, silent=silent@entry=0 '\000') at /test/10.6_dbg/mysys/stacktrace.c:212
#4  0x000055e26c847221 in handle_fatal_signal (sig=11) at /test/10.6_dbg/sql/signal_handler.cc:208
#5  <signal handler called>
#6  0x000055e26cd7c556 in row_merge_fts_doc_tokenize (t_ctx=0x145714cf5a80, opt_doc_id_size=<optimized out>, merge_file=<optimized out>, doc=0x145714cf59d0, doc_id=<optimized out>, sort_buf=<optimized out>) at /test/10.6_dbg/storage/innobase/row/row0ftsort.cc:577
#7  fts_parallel_tokenization (arg=<optimized out>) at /test/10.6_dbg/storage/innobase/row/row0ftsort.cc:839
#8  0x79d37975b9b47500 in ?? ()
#9  0x000055e26e3f9820 in ?? ()
#10 0x000055e26d03c0fe in __gthread_mutex_unlock (__mutex=0x55e26f66d030) at /usr/include/x86_64-linux-gnu/c++/9/bits/gthr-default.h:779
#11 std::mutex::unlock (this=0x55e26f66d030) at /usr/include/c++/9/bits/std_mutex.h:118
#12 std::unique_lock<std::mutex>::unlock (this=<synthetic pointer>) at /usr/include/c++/9/bits/unique_lock.h:197
#13 std::unique_lock<std::mutex>::~unique_lock (this=<synthetic pointer>, __in_chrg=<optimized out>) at /usr/include/c++/9/bits/unique_lock.h:106
#14 tpool::waitable_task::release (this=0x1456e8000b60) at /test/10.6_dbg/tpool/task.cc:70
Backtrace stopped: Cannot access memory at address 0x166479ff5097ca64

Bug confirmed present in:
MariaDB: 10.2.37 (dbg), 10.2.37 (opt), 10.3.28 (dbg), 10.3.28 (opt), 10.4.18 (dbg), 10.4.18 (opt), 10.5.9 (dbg), 10.5.9 (opt), 10.6.0 (dbg), 10.6.0 (opt)

Bug (or feature/syntax) confirmed not present in:
MySQL: 5.5.62 (dbg), 5.5.62 (opt), 5.6.51 (dbg), 5.6.51 (opt), 5.7.33 (dbg), 5.7.33 (opt), 8.0.23 (dbg), 8.0.23 (opt)

Comment by Thirunarayanan Balathandayuthapani [ 2021-02-23 ]

ASAN build reveals the failure:

=================================================================
==24082==ERROR: AddressSanitizer: stack-buffer-overflow on address 0x7f45fff2be52 at pc 0x55ab3cfc0e07 bp 0x7f45fff2bd20 sp 0x7f45fff2bd10
WRITE of size 1 at 0x7f45fff2be52 thread T8
    #0 0x55ab3cfc0e06 in strmake /home/thiru/mariarepo/10.6/10.6-work-1/strings/strmake.c:66
    #1 0x55ab3cf45e25 in my_strnxfrm_tis620 /home/thiru/mariarepo/10.6/10.6-work-1/strings/ctype-tis620.c:610
    #2 0x55ab3b96e2ec in charset_info_st::strnxfrm(unsigned char*, unsigned long, unsigned char const*, unsigned long) const /home/thiru/mariarepo/10.6/10.6-work-1/include/m_ctype.h:807
    #3 0x55ab3c4ad475 in innobase_strnxfrm(charset_info_st const*, unsigned char const*, unsigned long) /home/thiru/mariarepo/10.6/10.6-work-1/storage/innobase/handler/ha_innodb.cc:6001
    #4 0x55ab3c79536c in fts_select_index_by_range /home/thiru/mariarepo/10.6/10.6-work-1/storage/innobase/include/fts0types.ic:140
    #5 0x55ab3c79587a in fts_select_index /home/thiru/mariarepo/10.6/10.6-work-1/storage/innobase/include/fts0types.ic:214
    #6 0x55ab3c799a0c in row_merge_fts_doc_tokenize /home/thiru/mariarepo/10.6/10.6-work-1/storage/innobase/row/row0ftsort.cc:569
    #7 0x55ab3c79bfa7 in fts_parallel_tokenization /home/thiru/mariarepo/10.6/10.6-work-1/storage/innobase/row/row0ftsort.cc:839
    #8 0x55ab3cce3d0e in tpool::task::execute() /home/thiru/mariarepo/10.6/10.6-work-1/tpool/task.cc:52
    #9 0x55ab3ccd1e5d in tpool::thread_pool_generic::worker_main(tpool::worker_data*) /home/thiru/mariarepo/10.6/10.6-work-1/tpool/tpool_generic.cc:546
    #10 0x55ab3ccdb8e3 in void std::__invoke_impl<void, void (tpool::thread_pool_generic::*)(tpool::worker_data*), tpool::thread_pool_generic*, tpool::worker_data*>(std::__invoke_memfun_deref, void (tpool::thread_pool_generic::*&&)(tpool::worker_data*), tpool::thread_pool_generic*&&, tpool::worker_data*&&) (/home/thiru/mariarepo/10.6/10.6-work-1/basan/sql/mariadbd+0x36218e3)
    #11 0x55ab3ccd8f34 in std::__invoke_result<void (tpool::thread_pool_generic::*)(tpool::worker_data*), tpool::thread_pool_generic*, tpool::worker_data*>::type std::__invoke<void (tpool::thread_pool_generic::*)(tpool::worker_data*), tpool::thread_pool_generic*, tpool::worker_data*>(void (tpool::thread_pool_generic::*&&)(tpool::worker_data*), tpool::thread_pool_generic*&&, tpool::worker_data*&&) /usr/include/c++/8/bits/invoke.h:95
    #12 0x55ab3cce2d68 in decltype (__invoke((_S_declval<0ul>)(), (_S_declval<1ul>)(), (_S_declval<2ul>)())) std::thread::_Invoker<std::tuple<void (tpool::thread_pool_generic::*)(tpool::worker_data*), tpool::thread_pool_generic*, tpool::worker_data*> >::_M_invoke<0ul, 1ul, 2ul>(std::_Index_tuple<0ul, 1ul, 2ul>) /usr/include/c++/8/thread:244
    #13 0x55ab3cce2cbe in std::thread::_Invoker<std::tuple<void (tpool::thread_pool_generic::*)(tpool::worker_data*), tpool::thread_pool_generic*, tpool::worker_data*> >::operator()() /usr/include/c++/8/thread:253
    #14 0x55ab3cce2c27 in std::thread::_State_impl<std::thread::_Invoker<std::tuple<void (tpool::thread_pool_generic::*)(tpool::worker_data*), tpool::thread_pool_generic*, tpool::worker_data*> > >::_M_run() /usr/include/c++/8/thread:196
    #15 0x7f460cd2ad7f  (/usr/lib/x86_64-linux-gnu/libstdc++.so.6+0xd0d7f)
    #16 0x7f460d2396da in start_thread /build/glibc-2ORdQG/glibc-2.27/nptl/pthread_create.c:463
    #17 0x7f460c3d4a3e in __clone (/lib/x86_64-linux-gnu/libc.so.6+0x121a3e)

In mysql, this issue has been fixed in the file strings/ctype-tis620.c and their commit id is

commit 19bd66fe43c41f0bde5f36bc6b455a46693069fb
Author: bin.x.su@oracle.com <>
Date:   Fri Apr 4 11:35:27 2014 +0800
    BUG 18277082 - FTS: STACK BUFFER OVERFLOW IN INNOBASE_STRNXFRM AND TIS620
    When we run windows debug build or use a linux Address Sanitizer build,
    there would be a failure or stack-buffer-overflow in innobase_strnxfrm.
    The root cause is that my_strnxfrm_tis620 would access one more byte of
    the dst parameter and set it to '\0', which is unnecessary in this case.
    So we just do not set this terminating '\0'.
    Approved by Tor, rb#5051.

Comment by Roel Van de Paar [ 2021-02-23 ]

Perhaps sanja knows?

Comment by Marko Mäkelä [ 2021-03-01 ]

Because NUL is a valid code point in the tis620 encoding, the MySQL fix looks incorrect, in that it is terminating the copying on the first NUL byte.

Comment by Roel Van de Paar [ 2021-03-17 ]

Another stack seen:

10.6.0 03ff588d153f22f00ff00923e82498cbac63505f (Debug)

Core was generated by `/test/MD060321-mariadb-10.6.0-linux-x86_64-dbg/bin/mysqld --no-defaults --core-'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0  0x000055a33c264327 in row_merge_fts_doc_tokenize (t_ctx=0x153ec8374a80, 
    opt_doc_id_size=<optimized out>, merge_file=<optimized out>, 
    doc=0x153ec83749d0, doc_id=<optimized out>, sort_buf=<optimized out>)
    at /test/10.6_dbg/storage/innobase/row/row0ftsort.cc:577
577			mtuple_t* mtuple = &buf->tuples[buf->n_tuples + n_tuple[idx]];
[Current thread is 1 (Thread 0x153ec8375700 (LWP 356887))]
(gdb) bt
#0  0x000055a33c264327 in row_merge_fts_doc_tokenize (t_ctx=0x153ec8374a80, opt_doc_id_size=<optimized out>, merge_file=<optimized out>, doc=0x153ec83749d0, doc_id=<optimized out>, sort_buf=<optimized out>) at /test/10.6_dbg/storage/innobase/row/row0ftsort.cc:577
#1  fts_parallel_tokenization (arg=<optimized out>) at /test/10.6_dbg/storage/innobase/row/row0ftsort.cc:839
#2  0x000055a33d8e45a0 in ?? ()
#3  0x000055a33d8c6660 in ?? ()
#4  0x000055a33c51fff8 in __gthread_mutex_unlock (__mutex=0x153ec8374d30) at /usr/include/x86_64-linux-gnu/c++/9/bits/gthr-default.h:779
#5  std::mutex::unlock (this=0x153ec8374d30) at /usr/include/c++/9/bits/std_mutex.h:118
#6  std::unique_lock<std::mutex>::unlock (this=<synthetic pointer>) at /usr/include/c++/9/bits/unique_lock.h:197
#7  std::unique_lock<std::mutex>::~unique_lock (this=<synthetic pointer>, __in_chrg=<optimized out>) at /usr/include/c++/9/bits/unique_lock.h:106
#8  tpool::waitable_task::release (this=0x0) at /test/10.6_dbg/tpool/task.cc:70
Backtrace stopped: Cannot access memory at address 0x166d296eebf5cc4d

With this testcase (unfinished as the issue does not readily reproduce), running 200 threads, random SQL order:

SET collation_connection='tis620_bin';#NOERROR
SET @session_start_value = @@session.character_set_connection;#NOERROR
SET @@session.character_set_server = @session_start_value;#NOERROR
CREATE DATABASE `�\�\�\`;#NOERROR
USE `�\�\�\`;#NOERROR
CREATE TABLE t1 (col text, FULLTEXT KEY full_text (col)) ENGINE = InnoDB;#NOERROR
INSERT INTO t1 VALUES(7693);#NOERROR
ALTER TABLE t1 ADD (col2 varchar(100) character set latin1); ;
SELECT SLEEP(3);

And these options (unfiltered as the issue does not readily reproduce):

--max_allowed_packet=33554432 --maximum-bulk_insert_buffer_size=1M --maximum-join_buffer_size=1M --maximum-max_heap_table_size=1M --maximum-max_join_size=1M --maximum-myisam_max_sort_file_size=1M --maximum-myisam_mmap_size=1M --maximum-myisam_sort_buffer_size=1M --maximum-optimizer_trace_max_mem_size=1M --maximum-preload_buffer_size=1M --maximum-query_alloc_block_size=1M --maximum-query_prealloc_size=1M --maximum-range_alloc_block_size=1M --maximum-read_buffer_size=1M --maximum-read_rnd_buffer_size=1M --maximum-sort_buffer_size=1M --maximum-tmp_table_size=1M --maximum-transaction_alloc_block_size=1M --maximum-transaction_prealloc_size=1M --log-output=none --sql_mode=ONLY_FULL_GROUP_BY --innodb_file_per_table=1 --innodb_flush_method=O_DIRECT --innodb_stats_persistent=off --loose-idle_write_transaction_timeout=0 --loose-idle_transaction_timeout=0 --loose-idle_readonly_transaction_timeout=0 --connect_timeout=60 --interactive_timeout=28800 --slave_net_timeout=60 --net_read_timeout=30 --net_write_timeout=60 --loose-table_lock_wait_timeout=50 --wait_timeout=28800 --lock-wait-timeout=86400 --innodb-lock-wait-timeout=50 --log_output=FILE --log-bin --log_bin_trust_function_creators=1 --loose-max-statement-time=30 --loose-debug_assert_on_not_freed_memory=0 --innodb-buffer-pool-size=300M

This testcase is not the best, but adding it here for completeness and to include the new stack seen in this bug ticket. It can produce these two uniqueID's:

SIGSEGV|row_merge_fts_doc_tokenize|fts_parallel_tokenization|std::condition_variable::__wait_until_impl<std::chrono::duration<long, std::ratio<1l, 1000000000l> > >|std::condition_variable::wait_until<std::chrono::_V2::steady_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >
SIGSEGV|fts_parallel_tokenization|__gthread_mutex_unlock|std::mutex::unlock|std::unique_lock<std::mutex>::unlock

Comment by Marko Mäkelä [ 2021-03-17 ]

I added the component "Character sets" because I did not find any more descriptive category for an apparent bug in a collation. InnoDB is merely invoking that code.

Comment by Oleksandr Byelkin [ 2021-09-15 ]

--source include/have_innodb.inc
 
SET collation_connection='tis620_bin';
SET @@session.character_set_server='tis620';
CREATE DATABASE a;
USE a;
CREATE TABLE t(c TEXT,FULLTEXT KEY f(c)) ENGINE=InnoDB;
INSERT INTO t VALUES(100);
ALTER TABLE t ADD (c2 INT);
use test;
DROP database a;

Comment by Oleksandr Byelkin [ 2021-09-15 ]

I can not repeat it any more:

==============================================================================
 
TEST                                      RESULT   TIME (ms) or COMMENT
--------------------------------------------------------------------------
 
worker[1] Using MTR_BUILD_THREAD 300, with reserved ports 16000..16019
SET collation_connection='tis620_bin';
SET @@session.character_set_server='tis620';
CREATE DATABASE a;
USE a;
CREATE TABLE t(c TEXT,FULLTEXT KEY f(c)) ENGINE=InnoDB;
INSERT INTO t VALUES(100);
ALTER TABLE t ADD (c2 INT);
use test;
DROP database a;
main.test 'innodb'                       [ pass ]    106
--------------------------------------------------------------------------

Comment by Roel Van de Paar [ 2021-09-17 ]

Issue reproducible on trunk from today (all versions), either directly copy/paste in CLI, or using SOURCE in.sql:

10.7.0-dbg>source in.sql
Query OK, 0 rows affected (0.000 sec)
 
Query OK, 0 rows affected (0.000 sec)
 
Query OK, 1 row affected (0.002 sec)
 
Query OK, 0 rows affected (0.000 sec)
 
Query OK, 0 rows affected (0.076 sec)
 
Query OK, 1 row affected (0.012 sec)
 
ERROR 2013 (HY000) at line 7 in file: 'in.sql': Lost connection to server during query
10.7.0-dbg>

Or using a redirect like:

./bin/mysql -A -uroot -S./socket.sock --force --binary-mode test < ./in.sql

/test/MD180921-mariadb-10.7.0-linux-x86_64-dbg$ cat in.sql
SET collation_connection='tis620_bin';
SET @@session.character_set_server='tis620';
CREATE DATABASE a;
USE a;
CREATE TABLE t(c TEXT,FULLTEXT KEY f(c)) ENGINE=InnoDB;
INSERT INTO t VALUES(100);
ALTER TABLE t ADD (c2 INT);

Leads to:

10.7.0 d552e092c9f3e20da078d1b62b976f629f73d3a4 (Debug)

Core was generated by `/test/MD180921-mariadb-10.7.0-linux-x86_64-dbg/bin/mysqld --no-defaults --core-'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0  0x0000564a56acef79 in row_merge_fts_doc_tokenize (t_ctx=0x146cec94da80, 
    opt_doc_id_size=<optimized out>, merge_file=<optimized out>, 
    doc=0x146cec94d9d0, doc_id=<optimized out>, sort_buf=<optimized out>)
    at /test/10.7_dbg/storage/innobase/row/row0ftsort.cc:577
[Current thread is 1 (Thread 0x146cec94e700 (LWP 322041))]
(gdb) bt
#0  0x0000564a56acef79 in row_merge_fts_doc_tokenize (t_ctx=0x146cec94da80, opt_doc_id_size=<optimized out>, merge_file=<optimized out>, doc=0x146cec94d9d0, doc_id=<optimized out>, sort_buf=<optimized out>) at /test/10.7_dbg/storage/innobase/row/row0ftsort.cc:577
#1  fts_parallel_tokenization (arg=<optimized out>) at /test/10.7_dbg/storage/innobase/row/row0ftsort.cc:838
#2  0x361fce632e0f2700 in ?? ()
#3  0x112e0be826d694b3 in ?? ()
#4  0x0000564a56d7d601 in std::condition_variable::__wait_until_impl<std::chrono::duration<long, std::ratio<1l, 1000000000l> > > (__atime=<synthetic pointer>: <optimized out>, __lock=<error reading variable: Cannot access memory at address 0x16a5b9e31a9dd1de>, this=0x564a58962a90) at /usr/include/x86_64-linux-gnu/c++/9/bits/gthr-default.h:872
#5  std::condition_variable::wait_until<std::chrono::_V2::steady_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > > (__atime=<optimized out>, __lock=<error reading variable: Cannot access memory at address 0x16a5b9e31a9dd1de>, this=0x564a58962a90) at /usr/include/c++/9/condition_variable:121
#6  std::condition_variable::wait_for<long, std::ratio<1l, 1000l> > (__rtime=@0x146cec94dea8: {__r = 0}, __lock=<error reading variable: Cannot access memory at address 0x16a5b9e31a9dd1de>, this=0x564a58962a90) at /usr/include/c++/9/condition_variable:152
#7  tpool::thread_pool_generic::wait_for_tasks (this=0x146cec94dd70, lk=<error reading variable: Cannot access memory at address 0x16a5b9e31a9dd1de>, thread_data=0x564a58962a90) at /test/10.7_dbg/tpool/tpool_generic.cc:450
Backtrace stopped: Cannot access memory at address 0x16a5b9e31a9dd236

Bug confirmed present in:
MariaDB: 10.2.41 (dbg), 10.2.41 (opt), 10.3.32 (dbg), 10.3.32 (opt), 10.4.22 (dbg), 10.4.22 (opt), 10.5.13 (dbg), 10.5.13 (opt), 10.6.5 (dbg), 10.6.5 (opt), 10.7.0 (dbg), 10.7.0 (opt)

Bug (or feature/syntax) confirmed not present in:
MySQL: 5.5.62 (dbg), 5.5.62 (opt), 5.6.51 (dbg), 5.6.51 (opt), 5.7.35 (dbg), 5.7.35 (opt)

Comment by Roel Van de Paar [ 2021-09-17 ]

MTR Also crashes on same SQL.

/test/MD180921-mariadb-10.7.0-linux-x86_64-dbg/mysql-test$ cat main/test.test 
--source include/have_innodb.inc
SET collation_connection='tis620_bin';
SET @@session.character_set_server='tis620';
CREATE DATABASE a;
USE a;
CREATE TABLE t(c TEXT,FULLTEXT KEY f(c)) ENGINE=InnoDB;
INSERT INTO t VALUES(100);
ALTER TABLE t ADD (c2 INT);
/test/MD180921-mariadb-10.7.0-linux-x86_64-dbg/mysql-test$ ./mysql-test-run test

Leads to:

==============================================================================
 
TEST                                      RESULT   TIME (ms) or COMMENT
--------------------------------------------------------------------------
 
worker[1] Using MTR_BUILD_THREAD 300, with reserved ports 16000..16019
SET collation_connection='tis620_bin';
SET @@session.character_set_server='tis620';
CREATE DATABASE a;
USE a;
CREATE TABLE t(c TEXT,FULLTEXT KEY f(c)) ENGINE=InnoDB;
INSERT INTO t VALUES(100);
ALTER TABLE t ADD (c2 INT);
main.test 'innodb'                       [ fail ]
        Test ended at 2021-09-18 07:48:15
 
CURRENT_TEST: main.test
mysqltest: At line 8: query 'ALTER TABLE t ADD (c2 INT)' failed: <Unknown> (2013): Lost connection to server during query
...
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Core was generated by `/test/MD180921-mariadb-10.7.0-linux-x86_64-dbg/bin/mariadbd --defaults-group-su'.
Program terminated with signal SIGSEGV, Segmentation fault.
[Current thread is 1 (Thread 0x1522d031e700 (LWP 689375))]
#0  0x000055f724f41f79 in row_merge_fts_doc_tokenize (t_ctx=0x1522d031da80, opt_doc_id_size=<optimized out>, merge_file=<optimized out>, doc=0x1522d031d9d0, doc_id=<optimized out>, sort_buf=<optimized out>) at /test/10.7_dbg/storage/innobase/row/row0ftsort.cc:577
#1  fts_parallel_tokenization (arg=<optimized out>) at /test/10.7_dbg/storage/innobase/row/row0ftsort.cc:838
#2  0xcaa89188363c1a00 in ?? ()
#3  0x112e0be826d694b3 in ?? ()
#4  0x000055f7251f0601 in std::condition_variable::__wait_until_impl<std::chrono::duration<long, std::ratio<1l, 1000000000l> > > (__atime=<synthetic pointer>..., __lock=<error reading variable: Cannot access memory at address 0x16a5ba5d46e8dd3b>, this=0x55f7281aa6c0) at /usr/include/x86_64-linux-gnu/c++/9/bits/gthr-default.h:872
...

Comment by Marko Mäkelä [ 2021-10-27 ]

For some reason, the CREATE DATABASE is necessary for reproducing the crash. I was only able to simplify the test to request a simple table rebuild (no ADD COLUMN needed):

--source include/have_innodb.inc
SET collation_connection='tis620_bin';
SET @@session.character_set_server='tis620';
CREATE DATABASE a;
USE a;
CREATE TABLE t(c TEXT,FULLTEXT KEY f(c)) ENGINE=InnoDB;
INSERT INTO t VALUES(100);
OPTIMIZE TABLE t;
DROP DATABASE a;

10.7 da46c37bc7cf784c781d3c89d81b33911c10fedd

Version: '10.7.1-MariaDB-debug-log'  socket: '/dev/shm/10.7/mysql-test/var/tmp/mysqld.1.sock'  port: 16000  Source distribution
=================================================================
==2241578==ERROR: AddressSanitizer: stack-buffer-overflow on address 0x7f19c82e1f72 at pc 0x5570206a2483 bp 0x7f19c82e1ee0 sp 0x7f19c82e1ed8
WRITE of size 1 at 0x7f19c82e1f72 thread T5
    #0 0x5570206a2482 in strmake /mariadb/10.7/strings/strmake.c:66
    #1 0x5570206411cc in my_strnxfrm_tis620 /mariadb/10.7/strings/ctype-tis620.c:612
    #2 0x55701fb4bd5f in charset_info_st::strnxfrm(unsigned char*, unsigned long, unsigned char const*, unsigned long) const /mariadb/10.7/include/m_ctype.h:830
    #3 0x55701fb4bd5f in innobase_strnxfrm(charset_info_st const*, unsigned char const*, unsigned long) /mariadb/10.7/storage/innobase/handler/ha_innodb.cc:6478
    #4 0x55701fe55a59 in fts_select_index_by_range /mariadb/10.7/storage/innobase/include/fts0types.ic:140
    #5 0x55701fe55a59 in fts_select_index /mariadb/10.7/storage/innobase/include/fts0types.ic:214
    #6 0x55701fe60933 in row_merge_fts_doc_tokenize /mariadb/10.7/storage/innobase/row/row0ftsort.cc:569
    #7 0x55701fe63386 in fts_parallel_tokenization /mariadb/10.7/storage/innobase/row/row0ftsort.cc:838
Address 0x7f19c82e1f72 is located in stack of thread T5 at offset 34 in frame
    #0 0x55701fb4bc49 in innobase_strnxfrm(charset_info_st const*, unsigned char const*, unsigned long) /mariadb/10.7/storage/innobase/handler/ha_innodb.cc:6470
 
  This frame has 1 object(s):
    [32, 34) 'mystr' (line 6471) <== Memory access at offset 34 overflows this variable

Comment by Alexander Barkov [ 2021-10-27 ]

Marko suggested this script for 10.2. It crashes in the current 10.2 commit 2ed148c8d7b0133ecd17377587facadc7e76e9e8:

DROP TABLE IF EXISTS t1;
CREATE TABLE t1(c TEXT CHARACTER SET tis620) ENGINE=InnoDB;
INSERT INTO t1 VALUES('100');
ALTER TABLE t1 ADD FULLTEXT INDEX(c), ALGORITHM=INPLACE;

Comment by Alexander Barkov [ 2021-10-29 ]

marko, please review a patch:
https://github.com/MariaDB/server/commits/bb-10.2-bar-MDEV-24901

Comment by Marko Mäkelä [ 2021-10-29 ]

The InnoDB tests look good to me.

Generated at Thu Feb 08 09:33:34 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.