[MDEV-29421] Thread (10.6+) and server hangs (10.4/10.5) in 'Opening tables' (on optimized builds) and SIGABRT in safe_mutex_lock (on debug) on I_S read when using Spider Created: 2022-08-31  Updated: 2023-11-24  Resolved: 2023-11-24

Status: Closed
Project: MariaDB Server
Component/s: Locking, Storage Engine - Spider
Affects Version/s: 10.4, 10.5, 10.6, 10.7, 10.8, 10.9, 10.10, 10.11, 11.1
Fix Version/s: 10.4.33, 10.5.24, 10.6.17, 10.11.7, 11.0.5, 11.1.4, 11.2.3

Type: Bug Priority: Critical
Reporter: Roel Van de Paar Assignee: Yuchen Pei
Resolution: Fixed Votes: 0
Labels: hang, locking, spider-sts-crd, thread_hang

Issue Links:
Blocks
is blocked by MDEV-28739 Trying to lock uninitialized mutex or... Closed
PartOf
is part of MDEV-28739 Trying to lock uninitialized mutex or... Closed
Relates
relates to MDEV-32807 Delete spider sts and crd system tables Open

 Description   

Please replace '../socket.sock' with the actual socket location.
The issue is not sporadic. On debug builds, 10.4 to 10.11, we will see the crash below.

INSTALL PLUGIN Spider SONAME 'ha_spider.so';
CREATE USER Spider@localhost IDENTIFIED BY 'PWD123';
CREATE SERVER srv FOREIGN DATA WRAPPER MYSQL OPTIONS (SOCKET '../socket.sock', DATABASE 'test', USER 'Spider', PASSWORD 'PWD123');
CREATE TABLE t (c INT);
CREATE TABLE t_s (c INT) ENGINE=Spider COMMENT='WRAPPER "mysql", srv "srv", TABLE "t"';
SET GLOBAL table_open_cache=10;
CREATE TABLE t1 (a INT) ENGINE=Spider;
SELECT * FROM t1;
SELECT * FROM information_schema.tables;

Leads to:

10.11.0 bc563f1a4b0b38de3b41fd0f0d3d8b7f1aacbd8b (Debug)

Core was generated by `/test/MD190822-mariadb-10.11.0-linux-x86_64-dbg/bin/mysqld --no-defaults --core'.
Program terminated with signal SIGABRT, Aborted.
#0  __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50
[Current thread is 1 (Thread 0x145d84b68700 (LWP 4144754))]
(gdb) bt
#0  __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50
#1  0x0000145d9e0c5859 in __GI_abort () at abort.c:79
#2  0x000055ccfc7d8b62 in safe_mutex_lock (mp=mp@entry=0x145d84b17ca0 <spider_tbl_mutex>, my_flags=my_flags@entry=0, file=file@entry=0x145d84ae7110 "/test/10.11_dbg/storage/spider/spd_table.cc", line=line@entry=5602) at /test/10.11_dbg/mysys/thr_mutex.c:262
#3  0x0000145d84a623f3 in inline_mysql_mutex_lock (src_line=5602, src_file=0x145d84ae7110 "/test/10.11_dbg/storage/spider/spd_table.cc", that=0x145d84b17ca0 <spider_tbl_mutex>) at /test/10.11_dbg/include/mysql/psi/mysql_thread.h:750
#4  spider_free_share (share=0x145d3407de58) at /test/10.11_dbg/storage/spider/spd_table.cc:5602
#5  0x0000145d84a86927 in ha_spider::close (this=0x145d3407c160) at /test/10.11_dbg/storage/spider/ha_spider.cc:577
#6  0x000055ccfc09ab30 in handler::ha_close (this=0x145d3407c160) at /test/10.11_dbg/sql/handler.cc:3401
#7  0x000055ccfbec25a1 in closefrm (table=table@entry=0x145d3402a068) at /test/10.11_dbg/sql/table.cc:4534
#8  0x000055ccfbfeec17 in intern_close_table (table=table@entry=0x145d3402a068) at /test/10.11_dbg/sql/table_cache.cc:225
#9  0x000055ccfbfeed0a in tc_remove_table (table=table@entry=0x145d3402a068) at /test/10.11_dbg/sql/table_cache.cc:263
#10 0x000055ccfbfeee9c in tc_add_table (thd=thd@entry=0x145d34000db8, table=0x145d340dba18) at /test/10.11_dbg/sql/table_cache.cc:363
#11 0x000055ccfbd21a83 in open_table (thd=thd@entry=0x145d34000db8, table_list=table_list@entry=0x145d84b61fe0, ot_ctx=ot_ctx@entry=0x145d84b61f00) at /test/10.11_dbg/sql/sql_base.cc:2208
#12 0x000055ccfbd24307 in open_ltable (thd=thd@entry=0x145d34000db8, table_list=table_list@entry=0x145d84b61fe0, lock_type=TL_WRITE, lock_flags=lock_flags@entry=2075) at /test/10.11_dbg/sql/sql_base.cc:5468
#13 0x0000145d84a01b1d in spider_sys_open_table (thd=thd@entry=0x145d34000db8, tables=tables@entry=0x145d84b61fe0, open_tables_backup=open_tables_backup@entry=0x145d84b62738) at /test/10.11_dbg/storage/spider/spd_sys_table.cc:587
#14 0x0000145d84a01db0 in spider_open_sys_table (thd=thd@entry=0x145d34000db8, table_name=<optimized out>, table_name@entry=0x145d84ae138a "spider_table_sts", table_name_length=table_name_length@entry=16, write=write@entry=true, open_tables_backup=open_tables_backup@entry=0x145d84b62738, need_lock=need_lock@entry=false, error_num=0x145d84b62734) at /test/10.11_dbg/storage/spider/spd_sys_table.cc:263
#15 0x0000145d84a0a65e in spider_sys_insert_or_update_table_sts (thd=thd@entry=0x145d34000db8, name=0x145d34013910 "./test/t_s", name_length=10, stat=stat@entry=0x145d3411e620, need_lock=need_lock@entry=false) at /test/10.11_dbg/storage/spider/spd_sys_table.cc:3240
#16 0x0000145d84a625c2 in spider_free_share (share=0x145d3411dde8) at /test/10.11_dbg/storage/spider/spd_table.cc:5633
#17 0x0000145d84a86927 in ha_spider::close (this=0x145d3411d230) at /test/10.11_dbg/storage/spider/ha_spider.cc:577
#18 0x000055ccfc09ab30 in handler::ha_close (this=0x145d3411d230) at /test/10.11_dbg/sql/handler.cc:3401
#19 0x000055ccfbec25a1 in closefrm (table=table@entry=0x145d34097dc8) at /test/10.11_dbg/sql/table.cc:4534
#20 0x000055ccfbfeec17 in intern_close_table (table=table@entry=0x145d34097dc8) at /test/10.11_dbg/sql/table_cache.cc:225
#21 0x000055ccfbfeed0a in tc_remove_table (table=table@entry=0x145d34097dc8) at /test/10.11_dbg/sql/table_cache.cc:263
#22 0x000055ccfbfeee9c in tc_add_table (thd=thd@entry=0x145d34000db8, table=0x145d340b6a88) at /test/10.11_dbg/sql/table_cache.cc:363
#23 0x000055ccfbd21a83 in open_table (thd=thd@entry=0x145d34000db8, table_list=table_list@entry=0x145d3419c230, ot_ctx=ot_ctx@entry=0x145d84b62fa0) at /test/10.11_dbg/sql/sql_base.cc:2208
#24 0x000055ccfbd261b5 in open_and_process_table (ot_ctx=0x145d84b62fa0, has_prelocking_list=false, prelocking_strategy=0x145d84b63038, flags=1090, counter=0x145d84b63034, tables=0x145d3419c230, thd=0x145d34000db8) at /test/10.11_dbg/sql/sql_base.cc:4106
#25 open_tables (thd=thd@entry=0x145d34000db8, options=@0x145d84b64758: {m_options = DDL_options_st::OPT_NONE}, start=start@entry=0x145d84b63028, counter=counter@entry=0x145d84b63034, flags=1090, prelocking_strategy=prelocking_strategy@entry=0x145d84b63038) at /test/10.11_dbg/sql/sql_base.cc:4593
#26 0x000055ccfbd2732d in open_tables (prelocking_strategy=0x145d84b63038, flags=<optimized out>, counter=0x145d84b63034, tables=0x145d84b63028, thd=0x145d34000db8) at /test/10.11_dbg/sql/sql_base.h:266
#27 open_normal_and_derived_tables (thd=thd@entry=0x145d34000db8, tables=<optimized out>, tables@entry=0x145d3419c230, flags=<optimized out>, dt_phases=dt_phases@entry=3) at /test/10.11_dbg/sql/sql_base.cc:5631
#28 0x000055ccfbd27420 in open_tables_only_view_structure (thd=thd@entry=0x145d34000db8, table_list=table_list@entry=0x145d3419c230, can_deadlock=can_deadlock@entry=false) at /test/10.11_dbg/sql/sql_base.cc:5681
#29 0x000055ccfbe41f0c in fill_schema_table_by_open (thd=thd@entry=0x145d34000db8, mem_root=mem_root@entry=0x145d84b64f30, is_show_fields_or_keys=is_show_fields_or_keys@entry=false, table=table@entry=0x145d341026d0, schema_table=schema_table@entry=0x55ccfd26dc80 <schema_tables+2368>, orig_db_name=orig_db_name@entry=0x145d34111f58, orig_table_name=0x145d34112198, open_tables_state_backup=0x145d84b64f70, can_deadlock=false) at /test/10.11_dbg/sql/sql_show.cc:4718
#30 0x000055ccfbe6573e in get_all_tables (thd=0x145d34000db8, tables=0x145d340142d8, cond=<optimized out>) at /test/10.11_dbg/sql/sql_show.cc:5389
#31 0x000055ccfbe66cd0 in get_schema_tables_result (join=join@entry=0x145d34015e28, executed_place=executed_place@entry=PROCESSED_BY_JOIN_EXEC) at /test/10.11_dbg/sql/sql_show.cc:8943
#32 0x000055ccfbe3b1ed in JOIN::exec_inner (this=this@entry=0x145d34015e28) at /test/10.11_dbg/sql/sql_string.h:687
#33 0x000055ccfbe3be28 in JOIN::exec (this=this@entry=0x145d34015e28) at /test/10.11_dbg/sql/sql_select.cc:4590
#34 0x000055ccfbe39bac in mysql_select (thd=thd@entry=0x145d34000db8, tables=0x145d340142d8, fields=@0x145d34013f88: {<base_list> = {<Sql_alloc> = {<No data fields>}, first = 0x145d34014278, last = 0x145d3410f658, elements = 23}, <No data fields>}, conds=0x0, og_num=0, order=0x0, group=0x0, having=0x0, proc_param=0x0, select_options=2701396736, result=0x145d34015e00, unit=0x145d34004ff0, select_lex=0x145d34013ce8) at /test/10.11_dbg/sql/sql_select.cc:5070
#35 0x000055ccfbe3a3a2 in handle_select (thd=thd@entry=0x145d34000db8, lex=lex@entry=0x145d34004f18, result=result@entry=0x145d34015e00, setup_tables_done_option=setup_tables_done_option@entry=0) at /test/10.11_dbg/sql/sql_select.cc:581
#36 0x000055ccfbda45a6 in execute_sqlcom_select (thd=thd@entry=0x145d34000db8, all_tables=0x145d340142d8) at /test/10.11_dbg/sql/sql_parse.cc:6261
#37 0x000055ccfbdb08c7 in mysql_execute_command (thd=thd@entry=0x145d34000db8, is_called_from_prepared_stmt=is_called_from_prepared_stmt@entry=false) at /test/10.11_dbg/sql/sql_parse.cc:3945
#38 0x000055ccfbd9e882 in mysql_parse (thd=thd@entry=0x145d34000db8, rawbuf=<optimized out>, length=<optimized out>, parser_state=parser_state@entry=0x145d84b67330) at /test/10.11_dbg/sql/sql_parse.cc:8035
#39 0x000055ccfbdabe6a in dispatch_command (command=command@entry=COM_QUERY, thd=thd@entry=0x145d34000db8, packet=packet@entry=0x145d3400b6e9 "SELECT * FROM information_schema.tables", packet_length=packet_length@entry=39, blocking=blocking@entry=true) at /test/10.11_dbg/sql/sql_class.h:1339
#40 0x000055ccfbdae574 in do_command (thd=0x145d34000db8, blocking=blocking@entry=true) at /test/10.11_dbg/sql/sql_parse.cc:1407
#41 0x000055ccfbf101da in do_handle_one_connection (connect=<optimized out>, connect@entry=0x55ccfedb1628, put_in_cache=put_in_cache@entry=true) at /test/10.11_dbg/sql/sql_connect.cc:1418
#42 0x000055ccfbf106e3 in handle_one_connection (arg=0x55ccfedb1628) at /test/10.11_dbg/sql/sql_connect.cc:1312
#43 0x0000145d9e5d6609 in start_thread (arg=<optimized out>) at pthread_create.c:477
#44 0x0000145d9e1c2133 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Optimized builds will thread-hang with:

10.10.2 87e8463e0454a04c2bbaa38d44227c491fb07dc1 (Optimized)

MariaDB [test]> SHOW PROCESSLIST;
+----+-------------+-----------+------+---------+------+----------------------------------------------------+-----------------------------------------+----------+
| Id | User        | Host      | db   | Command | Time | State                                              | Info                                    | Progress |
+----+-------------+-----------+------+---------+------+----------------------------------------------------+-----------------------------------------+----------+
|  4 | root        | localhost | test | Query   | 1289 | Opening tables                                     | SELECT * FROM information_schema.tables |    0.000 |
|  5 | system user |           | NULL | Sleep   | 1289 | Reset for next command                             | NULL                                    |    0.000 |
|  6 | system user |           | NULL | Daemon  | NULL | Spider table background statistics action handler  | NULL                                    |    0.000 |
|  7 | system user |           | NULL | Daemon  | NULL | Spider table background statistics action handler  | NULL                                    |    0.000 |
|  8 | system user |           | NULL | Daemon  | NULL | Spider table background statistics action handler  | NULL                                    |    0.000 |
| 11 | system user |           | NULL | Daemon  | NULL | Spider table background cardinality action handler | NULL                                    |    0.000 |
| 12 | system user |           | NULL | Daemon  | NULL | Spider table background cardinality action handler | NULL                                    |    0.000 |
| 14 | system user |           | NULL | Daemon  | NULL | Spider table background statistics action handler  | NULL                                    |    0.000 |
| 15 | system user |           | NULL | Daemon  | NULL | Spider table background cardinality action handler | NULL                                    |    0.000 |
| 16 | system user |           | NULL | Daemon  | NULL | Spider table background statistics action handler  | NULL                                    |    0.000 |
| 18 | system user |           | NULL | Daemon  | NULL | Spider table background statistics action handler  | NULL                                    |    0.000 |
|  9 | system user |           | NULL | Daemon  | NULL | Spider table background statistics action handler  | NULL                                    |    0.000 |
| 10 | system user |           | NULL | Daemon  | NULL | Spider table background cardinality action handler | NULL                                    |    0.000 |
| 13 | system user |           | NULL | Daemon  | NULL | Spider table background statistics action handler  | NULL                                    |    0.000 |
| 17 | system user |           | NULL | Daemon  | NULL | Spider table background statistics action handler  | NULL                                    |    0.000 |
| 19 | system user |           | NULL | Daemon  | NULL | Spider table background cardinality action handler | NULL                                    |    0.000 |
| 20 | system user |           | NULL | Daemon  | NULL | Spider table background cardinality action handler | NULL                                    |    0.000 |
| 21 | system user |           | NULL | Daemon  | NULL | Spider table background cardinality action handler | NULL                                    |    0.000 |
| 22 | system user |           | NULL | Daemon  | NULL | Spider table background cardinality action handler | NULL                                    |    0.000 |
| 23 | system user |           | NULL | Daemon  | NULL | Spider table background cardinality action handler | NULL                                    |    0.000 |
| 24 | system user |           | NULL | Daemon  | NULL | Spider table background cardinality action handler | NULL                                    |    0.000 |
| 25 | Spider      | localhost | test | Sleep   | 1288 |                                                    | NULL                                    |    0.000 |
| 26 | root        | localhost | test | Query   |    0 | starting                                           | SHOW PROCESSLIST                        |    0.000 |
+----+-------------+-----------+------+---------+------+----------------------------------------------------+-----------------------------------------+----------+
23 rows in set (0.001 sec)

A shutdown is still possible for 10.6 to 10.11 in this case. However, 10.4 and 10.5 will fully hang during shutdown.

Bug confirmed present in:
MariaDB: 10.4.27 (dbg), 10.4.27 (opt), 10.5.18 (dbg), 10.5.18 (opt), 10.6.10 (dbg), 10.6.10 (opt), 10.7.6 (dbg), 10.7.6 (opt), 10.8.5 (dbg), 10.8.5 (opt), 10.9.2 (dbg), 10.9.2 (opt), 10.10.2 (dbg), 10.10.2 (opt), 10.11.0 (dbg), 10.11.0 (opt)

Bug (or feature/syntax) confirmed not present in:
MySQL: 5.5.62 (dbg), 5.5.62 (opt), 5.6.51 (dbg), 5.6.51 (opt), 5.7.38 (dbg), 5.7.38 (opt), 8.0.29 (dbg), 8.0.29 (opt)

SIGABRT|safe_mutex_lock|inline_mysql_mutex_lock|spider_free_share|ha_spider::close



 Comments   
Comment by Nayuta Yanagisawa (Inactive) [ 2022-08-31 ]

I cannot reproduce the bug on 10.4 (cf1a944f5b5496434ebe2c19e0f436e1bdd94871) debug build neither by MTR nor mysql client in my development environment. Still trying to reproduce.

Comment by Nayuta Yanagisawa (Inactive) [ 2022-08-31 ]

Roel Are all the lines in the test case is necessary to reproduce the bug? If not, could you please try to simplify the test case?

Comment by Roel Van de Paar [ 2022-09-01 ]

nayuta-yanagisawa Ack. All lines are needed/the testcase was simplified. Both my and ramesh's confirmation were by CLI. Please note to update the socket location. I would recommend using socket and not port/host as OS networking rules/protections may be an issue.
Also, 10.10 or 10.11 may be a good choice to reproduce as the earlier version had some oddities with installation of the plugin etc. (the reason 10.3 is not listed in many bugs). 10.4 in principle should be ok though in that area; you would know better.
Ping me if you get stuck and we'll have a look at what happens. Thanks

Comment by Roel Van de Paar [ 2022-09-01 ]

To try and help, I tested 10.4 at cf1a944f5b5496434ebe2c19e0f436e1bdd94871 and it crashes for me. Perpahs the issue is socket location?

10.4.27 cf1a944f5b5496434ebe2c19e0f436e1bdd94871 (Debug)

10.4.27-dbg>SELECT * FROM t1;
ERROR 1429 (HY000): Unable to connect to foreign data source: localhost
10.4.27-dbg>SELECT * FROM information_schema.tables;;
ERROR 2013 (HY000): Lost connection to MySQL server during query
ERROR: No query specified

$ t
SIGABRT|safe_mutex_lock|inline_mysql_mutex_lock|spider_free_share|ha_spider::close

$ cat BUILD_CMD_CMAKE 
cmake . -DWITH_SSL=bundled -DCMAKE_BUILD_TYPE=Debug -DBUILD_CONFIG=mysql_release -DWITH_TOKUDB=0 -DWITH_JEMALLOC=no -DFEATURE_SET=community -DDEBUG_EXTNAME=OFF -DWITH_EMBEDDED_SERVER=0 -DENABLE_DOWNLOADS=1 -DDOWNLOAD_BOOST=1 -DWITH_BOOST=/tmp/boost_415179 -DENABLED_LOCAL_INFILE=1 -DENABLE_DTRACE=0 -DWITH_SAFEMALLOC=OFF -DPLUGIN_PERFSCHEMA=NO -DWITH_DBUG_TRACE=OFF -DWITH_ZLIB=bundled -DWITH_ROCKSDB=1 -DWITH_PAM=ON -DWITH_MARIABACKUP=0 -DFORCE_INSOURCE_BUILD=1 -DCMAKE_C_FLAGS='-Og -march=native -mtune=native' -DCMAKE_CXX_FLAGS='-Og -march=native -mtune=native' -DMYSQL_MAINTAINER_MODE=OFF

Comment by Nayuta Yanagisawa (Inactive) [ 2022-09-05 ]

Roel Thank you for your advice. I now can reproduce the problem.

Comment by Nayuta Yanagisawa (Inactive) [ 2022-09-05 ]

There seems to be a problem with the mechanism for persisting statistics for remote tables, and it is trying to lock the same mutex doubly.

Comment by Roel Van de Paar [ 2022-09-05 ]

nayuta-yanagisawa Great to hear. Thank you for looking into the bug

Comment by Nayuta Yanagisawa (Inactive) [ 2022-09-06 ]

It seems that the mutex spider_tbl_mutex is taken too long. I believe that the following patch fixes the problem:

diff --git a/storage/spider/spd_table.cc b/storage/spider/spd_table.cc
index 7a00061e3a2..408515810b7 100644
--- a/storage/spider/spd_table.cc
+++ b/storage/spider/spd_table.cc
@@ -5804,11 +5804,23 @@ int spider_free_share(
   SPIDER_SHARE *share
 ) {
   DBUG_ENTER("spider_free_share");
-  pthread_mutex_lock(&spider_tbl_mutex);
   bool do_delete_thd = false;
   THD *thd = current_thd;
-  if (!--share->use_count)
+  pthread_mutex_lock(&spider_tbl_mutex);
+  if (--share->use_count)
+  {
+    pthread_mutex_unlock(&spider_tbl_mutex);
+  }
+  else
   {
+#ifdef HASH_UPDATE_WITH_HASH_VALUE
+    my_hash_delete_with_hash_value(&spider_open_tables,
+      share->table_name_hash_value, (uchar*) share);
+#else
+    my_hash_delete(&spider_open_tables, (uchar*) share);
+#endif
+    pthread_mutex_unlock(&spider_tbl_mutex);
+
 #ifndef WITHOUT_SPIDER_BG_SEARCH
     spider_free_sts_thread(share);
     spider_free_crd_thread(share);
@@ -5866,12 +5878,6 @@ int spider_free_share(
       );
     }
     spider_free_share_alloc(share);
-#ifdef HASH_UPDATE_WITH_HASH_VALUE
-    my_hash_delete_with_hash_value(&spider_open_tables,
-      share->table_name_hash_value, (uchar*) share);
-#else
-    my_hash_delete(&spider_open_tables, (uchar*) share);
-#endif
     thr_lock_delete(&share->lock);
     pthread_mutex_destroy(&share->crd_mutex);
     pthread_mutex_destroy(&share->sts_mutex);
@@ -5881,7 +5887,6 @@ int spider_free_share(
   }
   if (do_delete_thd)
     spider_destroy_thd(thd);
-  pthread_mutex_unlock(&spider_tbl_mutex);
   DBUG_RETURN(0);
 }
 

Comment by Nayuta Yanagisawa (Inactive) [ 2022-09-06 ]

holyfoot Please review: https://github.com/MariaDB/server/commit/f3bce86f1722cd8d5ddabb6fc949701e54dc959d

I failed to reproduce the bug with MTR. So, the patch only includes the fix.

Comment by Alexey Botchkov [ 2022-09-27 ]

Ok to push with one comment.

Comment by Roel Van de Paar [ 2022-10-05 ]

MTR testcase (please adapt as deemed best, and please add to the patch)

--let $SOCKET= `SELECT @@global.socket`
--source include/have_innodb.inc
INSTALL PLUGIN Spider SONAME 'ha_spider.so';
CREATE USER Spider@localhost IDENTIFIED BY 'PWD123';
eval CREATE SERVER srv FOREIGN DATA WRAPPER MYSQL OPTIONS (SOCKET "$SOCKET", DATABASE 'test', USER 'Spider', PASSWORD 'PWD123');
CREATE TABLE t (c INT) ENGINE=InnoDB;
CREATE TABLE t_s (c INT) ENGINE=Spider COMMENT='WRAPPER "mysql", srv "srv", TABLE "t"';
SET GLOBAL table_open_cache=10;
CREATE TABLE t1 (a INT) ENGINE=Spider;
--error 1429
SELECT * FROM t1;
SELECT * FROM information_schema.tables;
 
DROP TABLE t, t_s, t1;

Comment by Yuchen Pei [ 2023-05-15 ]

Could get a crash following the testcase in the previous comment at 11.1 4e5b771e980edfdad5c5414aa62c81d409d585a4, taking over

Comment by Yuchen Pei [ 2023-11-24 ]

fixed by MDEV-28739

Generated at Thu Feb 08 10:08:27 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.