Another closely related issue in optimized builds (same testcase):

10.5.3>help 'it is going to crash';

ERROR 1290 (HY000): The MariaDB server is running with the --max-thread-mem-used=50000 option so it cannot execute this statement

Note the max-thread-mem-used vs max_session_mem_used

Could be a typo, variable name change without text update, or a different interaction between multiple variables. max_thread_mem_used is not listed in mysqld options nor online anywhere, so it looks like it is a typo or similar to be fixed.

Similar to previously closed bugs MDEV-13011 and MDEV-13179

Generic unique bug ID; may mask other bugs.

! is_set()|SIGABRT|Diagnostics_area::set_eof_status|my_eof|mysqld_help_internal|mysqld_help

I get a different stack trace leading to assertion failure:

Program terminated with signal SIGABRT, Aborted.

warning: Unexpected size of section `.reg-xstate/20826' in core file.
#0 __pthread_kill (threadid=<optimized out>, signo=6) at ../sysdeps/unix/sysv/linux/pthread_kill.c:57
57 ../sysdeps/unix/sysv/linux/pthread_kill.c: No such file or directory.
[Current thread is 1 (Thread 0x7f5de1aee700 (LWP 20826))]
#0 __pthread_kill (threadid=<optimized out>, signo=6) at ../sysdeps/unix/sysv/linux/pthread_kill.c:57
#1 0x00005609b4dd319d in my_write_core (sig=6) at /home/rucha/Workspace/10_1/server/mysys/stacktrace.c:387
#2 0x00005609b4783c67 in handle_fatal_signal (sig=6) at /home/rucha/Workspace/10_1/server/sql/signal_handler.cc:296
#3 <signal handler called>
#4 __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:51
#5 0x00007f5de79058b1 in __GI_abort () at abort.c:79
#6 0x00007f5de78f542a in _assert_fail_base (fmt=0x7f5de7a7ca38 "%s%s%s:%u: %s%sAssertion `%s' failed.\n%n", assertion=assertion@entry=0x5609b4f69cc0 "fixed == 1", file=file@entry=0x5609b4f6a0b0 "/home/rucha/Workspace/10_1/server/sql/item_cmpfunc.cc", line=line@entry=5222, function=function@entry=0x5609b4f6c360 <Item_func_like::val_int()::PRETTY_FUNCTION_> "virtual longlong Item_func_like::val_int()") at assert.c:92
#7 0x00007f5de78f54a2 in _GI_assert_fail (assertion=0x5609b4f69cc0 "fixed == 1", file=0x5609b4f6a0b0 "/home/rucha/Workspace/10_1/server/sql/item_cmpfunc.cc", line=5222, function=0x5609b4f6c360 <Item_func_like::val_int()::PRETTY_FUNCTION_> "virtual longlong Item_func_like::val_int()") at assert.c:101
#8 0x00005609b47c7a48 in Item_func_like::val_int (this=0x7f5dd014e7d8) at /home/rucha/Workspace/10_1/server/sql/item_cmpfunc.cc:5222
#9 0x00005609b48e4e96 in search_keyword (thd=0x5609b751c550, keywords=0x7f5dd0066880, find_fields=0x7f5de1aeb170, select=0x7f5dd014e928, key_id=0x7f5de1aeafc8) at /home/rucha/Workspace/10_1/server/sql/sql_help.cc:244
#10 0x00005609b48e65f5 in mysqld_help_internal (thd=0x5609b751c550, mask=0x7f5dd014d1a0 "it is going to crash") at /home/rucha/Workspace/10_1/server/sql/sql_help.cc:748
#11 0x00005609b48e6e54 in mysqld_help (thd=0x5609b751c550, mask=0x7f5dd014d1a0 "it is going to crash") at /home/rucha/Workspace/10_1/server/sql/sql_help.cc:857
#12 0x00005609b4545e3c in mysql_execute_command (thd=0x5609b751c550) at /home/rucha/Workspace/10_1/server/sql/sql_parse.cc:3061
#13 0x00005609b4552b6c in mysql_parse (thd=0x5609b751c550, rawbuf=0x7f5dd014d0d8 "help 'it is going to crash'", length=27, parser_state=0x7f5de1aed620) at /home/rucha/Workspace/10_1/server/sql/sql_parse.cc:7200
#14 0x00005609b4541d2d in dispatch_command (command=COM_QUERY, thd=0x5609b751c550, packet=0x5609b7522211 "help 'it is going to crash'", packet_length=27) at /home/rucha/Workspace/10_1/server/sql/sql_parse.cc:1491
#15 0x00005609b4540c19 in do_command (thd=0x5609b751c550) at /home/rucha/Workspace/10_1/server/sql/sql_parse.cc:1123
#16 0x00005609b467ec69 in do_handle_one_connection (thd_arg=0x5609b751c550) at /home/rucha/Workspace/10_1/server/sql/sql_connect.cc:1331
#17 0x00005609b467e9a4 in handle_one_connection (arg=0x5609b751c550) at /home/rucha/Workspace/10_1/server/sql/sql_connect.cc:1242
#18 0x00005609b4a9d0d5 in pfs_spawn_thread (arg=0x5609b75273a0) at /home/rucha/Workspace/10_1/server/storage/perfschema/pfs.cc:1868
#19 0x00007f5de83e46db in start_thread (arg=0x7f5de1aee700) at pthread_create.c:463
#20 0x00007f5de79e6a3f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
warning: Unexpected size of section `.reg-xstate/20818' in core file.

rucha174 any updates please?

SET SQL_MODE='';

SET GLOBAL wsrep_forced_binlog_format='STATEMENT';

HELP '%a';

CREATE TABLE t (c CHAR(8) NOT NULL) ENGINE=MEMORY;

SET max_session_mem_used = 50000;

REPLACE DELAYED t VALUES (5);

HELP 'a%';

Without --log-bin added to the startup, this testcase produces the same issue. With --log-bin added, it produces MDEV-22708.

Roel Yes, It crashes in the first search topics. I changed max_session_mem_used as suggested in the review to check if it can crash later because the code after first search_topics() looked fragile. Found out that it doesn't crash.

Patch: https://github.com/MariaDB/server/commit/9c2570e6c42

9c2570e6c42 is ok to push