[MDEV-31793] LooseScan join on derived table crashes Created: 2023-07-28  Updated: 2024-01-30

Status: In Progress
Project: MariaDB Server
Component/s: None
Affects Version/s: 11.0, 11.1, 11.2, 11.3, 11.4
Fix Version/s: 11.0, 11.1, 11.2, 11.3, 11.4

Type: Bug Priority: Critical
Reporter: Oliver Assignee: Rex Johnston
Resolution: Unresolved Votes: 1
Labels: crash, regression
Environment:

MariaDB >= 11.0 docker image, Ubuntu 20.04/22.04 / Windows 10 host


Attachments: HTML File bad_trace     File crash-test.sql     HTML File good_trace    
Issue Links:
Relates
relates to MDEV-32785 background innodb stats recalculation... Open

 Description   

Some time ago I've noticed that my GitHub workflows that are using MariaDB where randomly failing. It seems to have started with version 11, at least I'm not able to reproduce it with earlier versions.

The basic gist is in the title, using a subquery with ranking and joins, MariaDB "randomly" segfaults. I wasn't able to reproduce it without either ranking or joins, and the columns of the involved tables seem to affect it too, if I remove any more of them, then it won't crash anymore.

Note that I say "randomly" because it can happen anywhere from immediately, to requiring 200+ attempts. I understand that these aren't the best prerequisites for recreating the problem, but it is somewhat reliable in that it will eventually crash when you just run the queries often enough.

Basic logs

2023-07-28 14:55:27+00:00 [Note] [Entrypoint]: Entrypoint script for MariaDB Server 1:11.2.0+maria~ubu2204 started.
2023-07-28 14:55:27+00:00 [Note] [Entrypoint]: Switching to dedicated user 'mysql'
2023-07-28 14:55:27+00:00 [Note] [Entrypoint]: Entrypoint script for MariaDB Server 1:11.2.0+maria~ubu2204 started.
2023-07-28 14:55:27+00:00 [Note] [Entrypoint]: Initializing database files
2023-07-28 14:55:27 0 [Warning] mariadbd: io_uring_queue_init() failed with ENOMEM: try larger memory locked limit, ulimit -l, or https://mariadb.com/kb/en/systemd/#configuring-limitmemlock under systemd (262144 bytes required)
2023-07-28 14:55:27 0 [Warning] InnoDB: liburing disabled: falling back to innodb_use_native_aio=OFF
 
 
PLEASE REMEMBER TO SET A PASSWORD FOR THE MariaDB root USER !
To do so, start the server, then issue the following command:
 
'/usr/bin/mariadb-secure-installation'
 
which will also give you the option of removing the test
databases and anonymous user created by default.  This is
strongly recommended for production servers.
 
See the MariaDB Knowledgebase at https://mariadb.com/kb
 
Please report any problems at https://mariadb.org/jira
 
The latest information about MariaDB is available at https://mariadb.org/.
 
Consider joining MariaDB's strong and vibrant community:
https://mariadb.org/get-involved/
 
2023-07-28 14:55:28+00:00 [Note] [Entrypoint]: Database files initialized
2023-07-28 14:55:28+00:00 [Note] [Entrypoint]: Starting temporary server
2023-07-28 14:55:28+00:00 [Note] [Entrypoint]: Waiting for server startup
2023-07-28 14:55:28 0 [Note] Starting MariaDB 11.2.0-MariaDB-1:11.2.0+maria~ubu2204 source revision e81fa345020ec6a067583db6a7019d6404b26f93 as process 99
2023-07-28 14:55:28 0 [Note] InnoDB: Compressed tables use zlib 1.2.11
2023-07-28 14:55:28 0 [Note] InnoDB: Number of transaction pools: 1
2023-07-28 14:55:28 0 [Note] InnoDB: Using crc32 + pclmulqdq instructions
2023-07-28 14:55:28 0 [Note] mariadbd: O_TMPFILE is not supported on /tmp (disabling future attempts)
2023-07-28 14:55:28 0 [Warning] mariadbd: io_uring_queue_init() failed with ENOMEM: try larger memory locked limit, ulimit -l, or https://mariadb.com/kb/en/systemd/#configuring-limitmemlock under systemd (262144 bytes required)
2023-07-28 14:55:28 0 [Warning] InnoDB: liburing disabled: falling back to innodb_use_native_aio=OFF
2023-07-28 14:55:28 0 [Note] InnoDB: Initializing buffer pool, total size = 128.000MiB, chunk size = 2.000MiB
2023-07-28 14:55:28 0 [Note] InnoDB: Completed initialization of buffer pool
2023-07-28 14:55:28 0 [Note] InnoDB: File system buffers for log disabled (block size=512 bytes)
2023-07-28 14:55:28 0 [Note] InnoDB: End of log at LSN=46125
2023-07-28 14:55:28 0 [Note] InnoDB: Opened 3 undo tablespaces
2023-07-28 14:55:28 0 [Note] InnoDB: 128 rollback segments in 3 undo tablespaces are active.
2023-07-28 14:55:28 0 [Note] InnoDB: Setting file './ibtmp1' size to 12.000MiB. Physically writing the file full; Please wait ...
2023-07-28 14:55:28 0 [Note] InnoDB: File './ibtmp1' size is now 12.000MiB.
2023-07-28 14:55:28 0 [Note] InnoDB: log sequence number 46125; transaction id 14
2023-07-28 14:55:28 0 [Note] Plugin 'FEEDBACK' is disabled.
2023-07-28 14:55:28 0 [Note] Plugin 'wsrep-provider' is disabled.
2023-07-28 14:55:28 0 [Warning] 'user' entry 'root@1ea12db333c8' ignored in --skip-name-resolve mode.
2023-07-28 14:55:28 0 [Warning] 'proxies_priv' entry '@% root@1ea12db333c8' ignored in --skip-name-resolve mode.
2023-07-28 14:55:28 0 [Note] mariadbd: Event Scheduler: Loaded 0 events
2023-07-28 14:55:28 0 [Note] mariadbd: ready for connections.
Version: '11.2.0-MariaDB-1:11.2.0+maria~ubu2204'  socket: '/run/mysqld/mysqld.sock'  port: 0  mariadb.org binary distribution
2023-07-28 14:55:29+00:00 [Note] [Entrypoint]: Temporary server started.
2023-07-28 14:55:30+00:00 [Note] [Entrypoint]: Creating database test
2023-07-28 14:55:30+00:00 [Note] [Entrypoint]: Securing system users (equivalent to running mysql_secure_installation)
 
2023-07-28 14:55:30+00:00 [Note] [Entrypoint]: Stopping temporary server
2023-07-28 14:55:30 0 [Note] mariadbd (initiated by: unknown): Normal shutdown
2023-07-28 14:55:30 0 [Note] InnoDB: FTS optimize thread exiting.
2023-07-28 14:55:30 0 [Note] InnoDB: Starting shutdown...
2023-07-28 14:55:30 0 [Note] InnoDB: Dumping buffer pool(s) to /var/lib/mysql/ib_buffer_pool
2023-07-28 14:55:30 0 [Note] InnoDB: Buffer pool(s) dump completed at 230728 14:55:30
2023-07-28 14:55:31 0 [Note] InnoDB: Removed temporary tablespace data file: "./ibtmp1"
2023-07-28 14:55:31 0 [Note] InnoDB: Shutdown completed; log sequence number 47375; transaction id 15
2023-07-28 14:55:31 0 [Note] mariadbd: Shutdown complete
 
2023-07-28 14:55:31+00:00 [Note] [Entrypoint]: Temporary server stopped
 
2023-07-28 14:55:31+00:00 [Note] [Entrypoint]: MariaDB init process done. Ready for start up.
 
2023-07-28 14:55:31 0 [Note] Starting MariaDB 11.2.0-MariaDB-1:11.2.0+maria~ubu2204 source revision e81fa345020ec6a067583db6a7019d6404b26f93 as process 1
2023-07-28 14:55:31 0 [Note] InnoDB: Compressed tables use zlib 1.2.11
2023-07-28 14:55:31 0 [Note] InnoDB: Number of transaction pools: 1
2023-07-28 14:55:31 0 [Note] InnoDB: Using crc32 + pclmulqdq instructions
2023-07-28 14:55:31 0 [Note] mariadbd: O_TMPFILE is not supported on /tmp (disabling future attempts)
2023-07-28 14:55:31 0 [Warning] mariadbd: io_uring_queue_init() failed with ENOMEM: try larger memory locked limit, ulimit -l, or https://mariadb.com/kb/en/systemd/#configuring-limitmemlock under systemd (262144 bytes required)
2023-07-28 14:55:31 0 [Warning] InnoDB: liburing disabled: falling back to innodb_use_native_aio=OFF
2023-07-28 14:55:31 0 [Note] InnoDB: Initializing buffer pool, total size = 128.000MiB, chunk size = 2.000MiB
2023-07-28 14:55:31 0 [Note] InnoDB: Completed initialization of buffer pool
2023-07-28 14:55:31 0 [Note] InnoDB: File system buffers for log disabled (block size=512 bytes)
2023-07-28 14:55:31 0 [Note] InnoDB: End of log at LSN=47375
2023-07-28 14:55:31 0 [Note] InnoDB: Opened 3 undo tablespaces
2023-07-28 14:55:31 0 [Note] InnoDB: 128 rollback segments in 3 undo tablespaces are active.
2023-07-28 14:55:31 0 [Note] InnoDB: Setting file './ibtmp1' size to 12.000MiB. Physically writing the file full; Please wait ...
2023-07-28 14:55:31 0 [Note] InnoDB: File './ibtmp1' size is now 12.000MiB.
2023-07-28 14:55:31 0 [Note] InnoDB: log sequence number 47375; transaction id 14
2023-07-28 14:55:31 0 [Note] InnoDB: Loading buffer pool(s) from /var/lib/mysql/ib_buffer_pool
2023-07-28 14:55:31 0 [Note] Plugin 'FEEDBACK' is disabled.
2023-07-28 14:55:31 0 [Note] Plugin 'wsrep-provider' is disabled.
2023-07-28 14:55:31 0 [Note] InnoDB: Buffer pool(s) load completed at 230728 14:55:31
2023-07-28 14:55:31 0 [Note] Server socket created on IP: '0.0.0.0'.
2023-07-28 14:55:31 0 [Note] Server socket created on IP: '::'.
2023-07-28 14:55:31 0 [Note] mariadbd: Event Scheduler: Loaded 0 events
2023-07-28 14:55:31 0 [Note] mariadbd: ready for connections.
Version: '11.2.0-MariaDB-1:11.2.0+maria~ubu2204'  socket: '/run/mysqld/mysqld.sock'  port: 3306  mariadb.org binary distribution
230728 14:55:41 [ERROR] mysqld got signal 11 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
 
To report this bug, see https://mariadb.com/kb/en/reporting-bugs
 
We will try our best to scrape up some info that will hopefully help
diagnose the problem, but since we have already crashed,
something is definitely wrong and this may fail.
 
Server version: 11.2.0-MariaDB-1:11.2.0+maria~ubu2204 source revision: e81fa345020ec6a067583db6a7019d6404b26f93
key_buffer_size=134217728
read_buffer_size=131072
max_used_connections=1
max_threads=153
thread_count=1
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 468063 K  bytes of memory
Hope that's ok; if not, decrease some variables in the equation.
 
Thread pointer: 0x7f4948000c68
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 0x7f4994119c38 thread_stack 0x49000
Printing to addr2line failed
mariadbd(my_print_stacktrace+0x32)[0x56389d4a5bd2]
mariadbd(handle_fatal_signal+0x488)[0x56389cf7ee98]
/lib/x86_64-linux-gnu/libc.so.6(+0x42520)[0x7f49abd45520]
mariadbd(_Z11copy_fieldsP15TMP_TABLE_PARAM+0x2b)[0x56389cd6b60b]
mariadbd(+0x8a6b62)[0x56389cd6bb62]
mariadbd(+0x87ade7)[0x56389cd3fde7]
mariadbd(_Z10sub_selectP4JOINP13st_join_tableb+0x24b)[0x56389cd5025b]
mariadbd(+0x87ade7)[0x56389cd3fde7]
mariadbd(_Z10sub_selectP4JOINP13st_join_tableb+0x24b)[0x56389cd5025b]
mariadbd(+0x87ade7)[0x56389cd3fde7]
mariadbd(_Z10sub_selectP4JOINP13st_join_tableb+0x2c4)[0x56389cd502d4]
mariadbd(_ZN4JOIN10exec_innerEv+0xfd4)[0x56389cd843d4]
mariadbd(_ZN4JOIN4execEv+0x3f)[0x56389cd8480f]
mariadbd(_Z12mysql_selectP3THDP10TABLE_LISTR4ListI4ItemEPS4_jP8st_orderS9_S7_S9_yP13select_resultP18st_select_lex_unitP13st_select_lex+0x12c)[0x56389cd8278c]
mariadbd(_Z13handle_selectP3THDP3LEXP13select_resulty+0x154)[0x56389cd82f84]
mariadbd(+0x8318f5)[0x56389ccf68f5]
mariadbd(_Z21mysql_execute_commandP3THDb+0x41ae)[0x56389cd0584e]
mariadbd(_Z11mysql_parseP3THDPcjP12Parser_state+0x1e7)[0x56389cd06b77]
mariadbd(_Z16dispatch_command19enum_server_commandP3THDPcjb+0x14bd)[0x56389cd0935d]
mariadbd(_Z10do_commandP3THDb+0x138)[0x56389cd0b158]
mariadbd(_Z24do_handle_one_connectionP7CONNECTb+0x3bf)[0x56389ce328cf]
mariadbd(handle_one_connection+0x5d)[0x56389ce32c1d]
mariadbd(+0xce5fa6)[0x56389d1aafa6]
/lib/x86_64-linux-gnu/libc.so.6(+0x94b43)[0x7f49abd97b43]
/lib/x86_64-linux-gnu/libc.so.6(clone+0x44)[0x7f49abe28bb4]
 
Trying to get some variables.
Some pointers may be invalid and cause the dump to abort.
Query (0x7f4948012fb0): SELECT
    TopComments.id AS TopComments__id,
    TopComments.article_id AS TopComments__article_id,
    TopComments.body AS TopComments__body,
    TopComments.votes AS TopComments__votes,
    TopComments_body_translation.id AS TopComments_body_translation__id,
    TopComments_body_translation.content AS TopComments_body_translation__content
FROM
    comments TopComments
INNER JOIN
    comments_i18n TopComments_body_translation ON (
        TopComments_body_translation.model = 'Comments'
        AND TopComments_body_translation.field = 'body'
        AND TopComments_body_translation.locale = 'de_DE'
        AND TopComments.id = TopComments_body_translation.foreign_key
    )
WHERE (
    TopComments.article_id in (1, 2)
    AND TopComments.id in (
        SELECT
            __ranked__TopComments.id AS id
        FROM (
            SELECT
                TopComments.id AS TopComments__id,
                TopComments.article_id AS TopComments__article_id,
                TopComments.body AS TopComments__body,
                TopComments.votes AS TopComments__votes,
                TopComments.id AS id,
                (
                    ROW_NUMBER() OVER (
                        PARTITION BY
                            TopComments.article_id
                        ORDER BY
                            TopComments.votes DESC,
                            TopComments.id ASC
                    )
                ) AS __row_number,
                TopComments_body_translation.id AS TopComments_body_translation__id,
                TopComments_body_translation.content AS TopComments_body_translation__content
            FROM
                comments TopComments
            INNER JOIN
                comments_i18n TopComments_body_translation ON (
                    TopComments_body_translation.model = 'Comments'
                    AND TopComments_body_translation.field = 'body'
                    AND TopComments_body_translation.locale = 'de_DE'
                    AND TopComments.id = TopComments_body_translation.foreign_key
                )
            WHERE
                TopComments.article_id in (1, 2)
        )
        __ranked__TopComments
        WHERE
            __ranked__TopComments.__row_number <= 3
    )
)
ORDER BY
    TopComments.votes DESC,
    TopComments.id ASC
 
Connection ID (thread ID): 11
Status: NOT_KILLED
 
Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin
=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exis
ts_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=on,condition_pushdown_from_having=on,not_null_range_scan=off,hash_join_cardinality=on
 
The manual page at https://mariadb.com/kb/en/how-to-produce-a-full-stack-trace-for-mysqld/ contains
information that should help you find out what is causing the crash.
Writing a core file...
Working directory at /var/lib/mysql
Resource Limits:
Limit                     Soft Limit           Hard Limit           Units
Max cpu time              unlimited            unlimited            seconds
Max file size             unlimited            unlimited            bytes
Max data size             unlimited            unlimited            bytes
Max stack size            8388608              unlimited            bytes
Max core file size        unlimited            unlimited            bytes
Max resident set          unlimited            unlimited            bytes
Max processes             unlimited            unlimited            processes
Max open files            1048576              1048576              files
Max locked memory         65536                65536                bytes
Max address space         unlimited            unlimited            bytes
Max file locks            unlimited            unlimited            locks
Max pending signals       15371                15371                signals
Max msgqueue size         819200               819200               bytes
Max nice priority         0                    0
Max realtime priority     0                    0
Max realtime timeout      unlimited            unlimited            us
Core pattern: |/usr/share/apport/apport -p%p -s%s -c%c -d%d -P%P -u%u -g%g -- %E
 
Kernel version: Linux version 5.4.0-124-generic (buildd@lcy02-amd64-089) (gcc version 9.4.0 (Ubuntu 9.4.0-1ubuntu1~20.04.1)) #140-Ubuntu SMP Thu Aug 4 02:23:37 UTC 2022

Full backtrace (all threads)

 
Thread 18 (Thread 0x7f9387fff640 (LWP 174) "mariadbd"):
#0  __futex_abstimed_wait_common64 (private=-1781142552, cancel=true, abstime=0x7f9387ffebf0, op=137, expected=0, futex_word=0x55c895a9cddc) at ./nptl/futex-internal.c:57
        sc_cancel_oldtype = 0
        sc_ret = <optimized out>
        resultvar = <optimized out>
        __arg6 = <optimized out>
        __arg5 = <optimized out>
        __arg4 = <optimized out>
        __arg3 = <optimized out>
        __arg2 = <optimized out>
        __arg1 = <optimized out>
        _a6 = <optimized out>
        _a5 = <optimized out>
        _a4 = <optimized out>
        _a3 = <optimized out>
        _a2 = <optimized out>
        _a1 = <optimized out>
#1  __futex_abstimed_wait_common (cancel=true, private=-1781142552, abstime=0x7f9387ffebf0, clockid=0, expected=0, futex_word=0x55c895a9cddc) at ./nptl/futex-internal.c:87
        err = <optimized out>
        clockbit = 256
        op = 137
        err = <optimized out>
        clockbit = <optimized out>
        op = <optimized out>
#2  __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x55c895a9cddc, expected=expected@entry=0, clockid=clockid@entry=1, abstime=abstime@entry=0x7f9387ffebf0, private=private@entry=0) at ./nptl/futex-internal.c:139
No locals.
#3  0x00007f93be0c835d in __pthread_cond_wait_common (abstime=0x7f9387ffebf0, clockid=1, mutex=0x55c895a9ca20, cond=0x55c895a9cdb0) at ./nptl/pthread_cond_wait.c:503
        spin = 0
        buffer = {__routine = 0x7f93be0c77a0 <__condvar_cleanup_waiting>, __arg = 0x7f9387ffeb70, __canceltype = 1, __prev = 0x0}
        cbuffer = {wseq = 125, cond = 0x55c895a9cdb0, mutex = 0x55c895a9ca20, private = 0}
        err = <optimized out>
        g = 1
        flags = <optimized out>
        g1_start = <optimized out>
        maxspin = 0
        signals = <optimized out>
        result = 0
        wseq = 125
        seq = 62
        private = 0
        maxspin = <optimized out>
        err = <optimized out>
        result = <optimized out>
        wseq = <optimized out>
        g = <optimized out>
        seq = <optimized out>
        flags = <optimized out>
        private = <optimized out>
        signals = <optimized out>
        done = <optimized out>
        g1_start = <optimized out>
        spin = <optimized out>
        buffer = <optimized out>
        cbuffer = <optimized out>
        s = <optimized out>
#4  ___pthread_cond_clockwait64 (abstime=0x7f9387ffebf0, clockid=1, mutex=0x55c895a9ca20, cond=0x55c895a9cdb0) at ./nptl/pthread_cond_wait.c:691
No locals.
#5  ___pthread_cond_clockwait64 (cond=0x55c895a9cdb0, mutex=0x55c895a9ca20, clockid=1, abstime=0x7f9387ffebf0) at ./nptl/pthread_cond_wait.c:679
No locals.
#6  0x000055c893b8232f in std::__condvar::wait_until (__abs_time=@0x7f9387ffebf0: {tv_sec = 11933, tv_nsec = 460221651}, __clock=1, __m=<optimized out>, this=0x55c895a9cdb0) at /usr/include/c++/11/bits/std_mutex.h:123
No locals.
#7  std::condition_variable::__wait_until_impl<std::chrono::duration<long, std::ratio<1l, 1000000000l> > > (__lock=@0x7f9387ffec40: {_M_device = 0x55c895a9ca20, _M_owns = true}, __lock=@0x7f9387ffec40: {_M_device = 0x55c895a9ca20, _M_owns = true}, __atime=<optimized out>, this=0x55c895a9cdb0) at /usr/include/c++/11/condition_variable:201
        __s = <optimized out>
        __ns = <optimized out>
        __ts = {tv_sec = 11933, tv_nsec = 460221651}
#8  std::condition_variable::wait_until<std::chrono::duration<long, std::ratio<1l, 1000000000l> > > (__atime=<optimized out>, __lock=@0x7f9387ffec40: {_M_device = 0x55c895a9ca20, _M_owns = true}, this=0x55c895a9cdb0) at /usr/include/c++/11/condition_variable:111
No locals.
#9  std::condition_variable::wait_for<long, std::ratio<1l, 1000l> > (__rtime=@0x55c895a9ca48: {__r = 60000}, __rtime=@0x55c895a9ca48: {__r = 60000}, __lock=@0x7f9387ffec40: {_M_device = 0x55c895a9ca20, _M_owns = true}, this=0x55c895a9cdb0) at /usr/include/c++/11/condition_variable:163
No locals.
#10 tpool::thread_pool_generic::wait_for_tasks (this=this@entry=0x55c895a9c900, lk=@0x7f9387ffec40: {_M_device = 0x55c895a9ca20, _M_owns = true}, thread_data=thread_data@entry=0x55c895a9cdb0) at ./tpool/tpool_generic.cc:480
        __PRETTY_FUNCTION__ = "bool tpool::thread_pool_generic::wait_for_tasks(std::unique_lock<std::mutex>&, tpool::worker_data*)"
#11 0x000055c893b82b68 in tpool::thread_pool_generic::get_task (this=0x55c895a9c900, thread_var=0x55c895a9cdb0, t=0x7f9387ffec88) at ./tpool/tpool_generic.cc:533
        lk = {_M_device = 0x55c895a9ca20, _M_owns = true}
#12 0x000055c893b82cd5 in tpool::thread_pool_generic::worker_main (this=0x55c895a9c900, thread_var=0x55c895a9cdb0) at ./tpool/tpool_generic.cc:578
        task = 0x7f93be0c54d8 <__nptl_free_stacks+168>
#13 0x00007f93be41f2b3 in ?? () from /lib/x86_64-linux-gnu/libstdc++.so.6
No symbol table info available.
#14 0x00007f93be0c8b43 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442
        ret = <optimized out>
        pd = <optimized out>
        out = <optimized out>
        unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140271585044528, 1783092640928857216, 140271618618944, 0, 140272525412432, 140271585044880, -1758129329567597440, -1758183593111225216}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
        not_first_call = <optimized out>
#15 0x00007f93be159bb4 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:100
No locals.
 
Thread 17 (Thread 0x7f93953f4640 (LWP 173) "mariadbd"):
#0  __futex_abstimed_wait_common64 (private=-1825326035, cancel=true, abstime=0x7f93953f3bf0, op=137, expected=0, futex_word=0x55c895a9ccd8) at ./nptl/futex-internal.c:57
        sc_cancel_oldtype = 0
        sc_ret = <optimized out>
        resultvar = <optimized out>
        __arg6 = <optimized out>
        __arg5 = <optimized out>
        __arg4 = <optimized out>
        __arg3 = <optimized out>
        __arg2 = <optimized out>
        __arg1 = <optimized out>
        _a6 = <optimized out>
        _a5 = <optimized out>
        _a4 = <optimized out>
        _a3 = <optimized out>
        _a2 = <optimized out>
        _a1 = <optimized out>
#1  __futex_abstimed_wait_common (cancel=true, private=-1825326035, abstime=0x7f93953f3bf0, clockid=0, expected=0, futex_word=0x55c895a9ccd8) at ./nptl/futex-internal.c:87
        err = <optimized out>
        clockbit = 256
        op = 137
        err = <optimized out>
        clockbit = <optimized out>
        op = <optimized out>
#2  __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x55c895a9ccd8, expected=expected@entry=0, clockid=clockid@entry=1, abstime=abstime@entry=0x7f93953f3bf0, private=private@entry=0) at ./nptl/futex-internal.c:139
No locals.
#3  0x00007f93be0c835d in __pthread_cond_wait_common (abstime=0x7f93953f3bf0, clockid=1, mutex=0x55c895a9ca20, cond=0x55c895a9ccb0) at ./nptl/pthread_cond_wait.c:503
        spin = 0
        buffer = {__routine = 0x7f93be0c77a0 <__condvar_cleanup_waiting>, __arg = 0x7f93953f3b70, __canceltype = 1, __prev = 0x0}
        cbuffer = {wseq = 474, cond = 0x55c895a9ccb0, mutex = 0x55c895a9ca20, private = 0}
        err = <optimized out>
        g = 0
        flags = <optimized out>
        g1_start = <optimized out>
        maxspin = 0
        signals = <optimized out>
        result = 0
        wseq = 474
        seq = 237
        private = 0
        maxspin = <optimized out>
        err = <optimized out>
        result = <optimized out>
        wseq = <optimized out>
        g = <optimized out>
        seq = <optimized out>
        flags = <optimized out>
        private = <optimized out>
        signals = <optimized out>
        done = <optimized out>
        g1_start = <optimized out>
        spin = <optimized out>
        buffer = <optimized out>
        cbuffer = <optimized out>
        s = <optimized out>
#4  ___pthread_cond_clockwait64 (abstime=0x7f93953f3bf0, clockid=1, mutex=0x55c895a9ca20, cond=0x55c895a9ccb0) at ./nptl/pthread_cond_wait.c:691
No locals.
#5  ___pthread_cond_clockwait64 (cond=0x55c895a9ccb0, mutex=0x55c895a9ca20, clockid=1, abstime=0x7f93953f3bf0) at ./nptl/pthread_cond_wait.c:679
No locals.
#6  0x000055c893b8232f in std::__condvar::wait_until (__abs_time=@0x7f93953f3bf0: {tv_sec = 11935, tv_nsec = 915261764}, __clock=1, __m=<optimized out>, this=0x55c895a9ccb0) at /usr/include/c++/11/bits/std_mutex.h:123
No locals.
#7  std::condition_variable::__wait_until_impl<std::chrono::duration<long, std::ratio<1l, 1000000000l> > > (__lock=@0x7f93953f3c40: {_M_device = 0x55c895a9ca20, _M_owns = true}, __lock=@0x7f93953f3c40: {_M_device = 0x55c895a9ca20, _M_owns = true}, __atime=<optimized out>, this=0x55c895a9ccb0) at /usr/include/c++/11/condition_variable:201
        __s = <optimized out>
        __ns = <optimized out>
        __ts = {tv_sec = 11935, tv_nsec = 915261764}
#8  std::condition_variable::wait_until<std::chrono::duration<long, std::ratio<1l, 1000000000l> > > (__atime=<optimized out>, __lock=@0x7f93953f3c40: {_M_device = 0x55c895a9ca20, _M_owns = true}, this=0x55c895a9ccb0) at /usr/include/c++/11/condition_variable:111
No locals.
#9  std::condition_variable::wait_for<long, std::ratio<1l, 1000l> > (__rtime=@0x55c895a9ca48: {__r = 60000}, __rtime=@0x55c895a9ca48: {__r = 60000}, __lock=@0x7f93953f3c40: {_M_device = 0x55c895a9ca20, _M_owns = true}, this=0x55c895a9ccb0) at /usr/include/c++/11/condition_variable:163
No locals.
#10 tpool::thread_pool_generic::wait_for_tasks (this=this@entry=0x55c895a9c900, lk=@0x7f93953f3c40: {_M_device = 0x55c895a9ca20, _M_owns = true}, thread_data=thread_data@entry=0x55c895a9ccb0) at ./tpool/tpool_generic.cc:480
        __PRETTY_FUNCTION__ = "bool tpool::thread_pool_generic::wait_for_tasks(std::unique_lock<std::mutex>&, tpool::worker_data*)"
#11 0x000055c893b82b68 in tpool::thread_pool_generic::get_task (this=0x55c895a9c900, thread_var=0x55c895a9ccb0, t=0x7f93953f3c88) at ./tpool/tpool_generic.cc:533
        lk = {_M_device = 0x55c895a9ca20, _M_owns = true}
#12 0x000055c893b82cd5 in tpool::thread_pool_generic::worker_main (this=0x55c895a9c900, thread_var=0x55c895a9ccb0) at ./tpool/tpool_generic.cc:578
        task = 0x55c895d42300
#13 0x00007f93be41f2b3 in ?? () from /lib/x86_64-linux-gnu/libstdc++.so.6
No symbol table info available.
#14 0x00007f93be0c8b43 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442
        ret = <optimized out>
        pd = <optimized out>
        out = <optimized out>
        unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140271585044704, 1783092640928857216, 140271840872000, 0, 140272525412432, 140271585045056, -1758097990801851264, -1758183593111225216}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
        not_first_call = <optimized out>
#15 0x00007f93be159bb4 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:100
No locals.
 
Thread 16 (Thread 0x7f93bc118640 (LWP 166) "mariadbd"):
#0  0x000055c8934b460b in copy_fields (param=0x7f93881151c0) at ./sql/sql_select.cc:28407
        ptr = 0x20000
        end = 0x1000000
        it = <optimized out>
        item = <optimized out>
#1  0x000055c8934b4b62 in end_write (join=0x7f93880e47c8, join_tab=0x7f9388107e08, end_of_records=<optimized out>) at ./sql/sql_select.cc:24972
        table = 0x7f938810a5e0
#2  0x000055c893488de7 in evaluate_join_record (join=0x7f93880e47c8, join_tab=0x7f9388107990, error=<optimized out>) at ./sql/sql_select.cc:23671
        rc = <optimized out>
        found = <optimized out>
        return_tab = <optimized out>
        shortcut_for_distinct = false
        found_records = 1
        select_cond = <optimized out>
        select_cond_result = <optimized out>
#3  0x000055c89349925b in sub_select (join=0x7f93880e47c8, join_tab=0x7f9388107990, end_of_records=false) at ./sql/sql_select.cc:23438
        error = <optimized out>
        rc = <optimized out>
        skip_over = <optimized out>
        info = <optimized out>
#4  0x000055c893488de7 in evaluate_join_record (join=0x7f93880e47c8, join_tab=0x7f9388107518, error=<optimized out>) at ./sql/sql_select.cc:23671
        rc = <optimized out>
        found = <optimized out>
        return_tab = <optimized out>
        shortcut_for_distinct = false
        found_records = 1
        select_cond = <optimized out>
        select_cond_result = <optimized out>
#5  0x000055c89349925b in sub_select (join=0x7f93880e47c8, join_tab=0x7f9388107518, end_of_records=false) at ./sql/sql_select.cc:23438
        error = <optimized out>
        rc = <optimized out>
        skip_over = <optimized out>
        info = <optimized out>
#6  0x000055c893488de7 in evaluate_join_record (join=0x7f93880e47c8, join_tab=0x7f93881070a0, error=<optimized out>) at ./sql/sql_select.cc:23671
        rc = <optimized out>
        found = <optimized out>
        return_tab = <optimized out>
        shortcut_for_distinct = false
        found_records = 1
        select_cond = <optimized out>
        select_cond_result = <optimized out>
#7  0x000055c8934992d4 in sub_select (join=0x7f93880e47c8, join_tab=0x7f93881070a0, end_of_records=false) at ./sql/sql_select.cc:23475
        error = <optimized out>
        rc = <optimized out>
        skip_over = <optimized out>
        info = 0x7f9388107170
#8  0x000055c8934cd3d4 in do_select (procedure=<optimized out>, join=0x7f93880e47c8) at ./sql/sql_select.cc:22955
        join_tab = <optimized out>
        rc = 0
        error = NESTED_LOOP_OK
        top_level_tables = <optimized out>
        rc = <optimized out>
        error = <optimized out>
        top_level_tables = <optimized out>
        res = <optimized out>
        last_tab = <optimized out>
        state = <optimized out>
        end_select = <optimized out>
        cleared_tables = <optimized out>
        columns_list = <optimized out>
        join_tab = <optimized out>
        sort_tab = <optimized out>
        join_tab = <optimized out>
        const_tables = <optimized out>
#9  JOIN::exec_inner (this=0x7f93880e47c8) at ./sql/sql_select.cc:4935
        columns_list = <optimized out>
        trace_wrapper = {<Json_writer_struct> = {_vptr.Json_writer_struct = <optimized out>, my_writer = 0x0, context = {writer = <optimized out>}, closed = false}, <No data fields>}
        trace_exec = {<Json_writer_struct> = {_vptr.Json_writer_struct = <optimized out>, my_writer = <optimized out>, context = {writer = <optimized out>}, closed = false}, <No data fields>}
        trace_steps = {<Json_writer_struct> = {_vptr.Json_writer_struct = 0x55c8943b9bf8 <vtable for Json_writer_array+16>, my_writer = 0x0, context = {writer = 0x0}, closed = false}, <No data fields>}
#10 0x000055c8934cd80f in JOIN::exec (this=this@entry=0x7f93880e47c8) at ./sql/sql_select.cc:4712
        res = <optimized out>
#11 0x000055c8934cb78c in mysql_select (thd=0x7f9388001128, tables=0x7f93880151b8, fields=<optimized out>, conds=0x7f93880e3988, og_num=2, order=0x7f93880e3c08, group=0x0, having=0x0, proc_param=0x0, select_options=<optimized out>, result=0x7f93880e3f10, unit=0x7f9388005478, select_lex=0x7f9388014268) at ./sql/sql_select.cc:5243
        err = 0
        free_join = true
        exec_error = false
        join = 0x7f93880e47c8
#12 0x000055c8934cbf84 in handle_select (thd=thd@entry=0x7f9388001128, lex=lex@entry=0x7f9388005398, result=result@entry=0x7f93880e3f10, setup_tables_done_option=setup_tables_done_option@entry=0) at ./sql/sql_select.cc:628
        unit = 0x7f9388005478
        res = <optimized out>
        select_lex = 0x7f9388014268
#13 0x000055c89343f8f5 in execute_sqlcom_select (thd=0x7f9388001128, all_tables=0x7f93880151b8) at ./sql/sql_parse.cc:6056
        save_protocol = 0x0
        lex = 0x7f9388005398
        result = 0x7f93880e3f10
        res = false
#14 0x000055c89344e84e in mysql_execute_command (thd=0x7f9388001128, is_called_from_prepared_stmt=<optimized out>) at ./sql/sql_parse.cc:3944
        privileges_requested = <optimized out>
        wsrep_error_label = <optimized out>
        error = <optimized out>
        res = 0
        lex = 0x7f9388005398
        select_lex = <optimized out>
        first_table = 0x7f93880151b8
        all_tables = 0x7f93880151b8
        unit = 0x7f9388005478
        rpl_filter = <optimized out>
        ots = {ctx = 0x7f9388005038, traceable = false}
        orig_binlog_format = BINLOG_FORMAT_MIXED
        orig_current_stmt_binlog_format = BINLOG_FORMAT_STMT
#15 0x000055c89344fb77 in mysql_parse (thd=0x7f9388001128, rawbuf=<optimized out>, length=<optimized out>, parser_state=<optimized out>) at ./sql/sql_parse.cc:7800
        found_semicolon = <optimized out>
        error = <optimized out>
        lex = 0x7f9388005398
        err = <optimized out>
#16 0x000055c89345235d in dispatch_command (command=COM_QUERY, thd=0x7f9388001128, packet=<optimized out>, packet_length=<optimized out>, blocking=<optimized out>) at ./sql/sql_parse.cc:1989
        beginning_of_next_stmt = <optimized out>
        length = <optimized out>
        packet_end = 0x7f93880138fd ""
        parser_state = {m_lip = {lookahead_token = -1, lookahead_yylval = 0x0, m_thd = 0x7f9388001128, m_ptr = 0x7f93880138fe "\004", m_tok_start = 0x7f93880138fe "\004", m_tok_end = 0x7f93880138fe "\004", m_end_of_query = 0x7f93880138fd "", m_tok_start_prev = 0x7f93880138fd "", m_buf = 0x7f9388012ff0 "SELECT\n    TopComments.id AS TopComments__id,\n    TopComments.article_id AS TopComments__article_id,\n    TopComments.body AS TopComments__body,\n    TopComments.votes AS TopComments__votes,\n    TopComm"..., m_buf_length = 2317, m_echo = true, m_echo_saved = false, m_cpp_buf = 0x7f9388013958 "SELECT\n    TopComments.id AS TopComments__id,\n    TopComments.article_id AS TopComments__article_id,\n    TopComments.body AS TopComments__body,\n    TopComments.votes AS TopComments__votes,\n    TopComm"..., m_cpp_ptr = 0x7f9388014265 "", m_cpp_tok_start = 0x7f9388014265 "", m_cpp_tok_start_prev = 0x7f9388014265 "", m_cpp_tok_end = 0x7f9388014265 "", m_body_utf8 = 0x0, m_body_utf8_ptr = 0x55c8945fa800 <dbname_cache_init()::buf> "", m_cpp_utf8_processed_ptr = 0x0, next_state = MY_LEX_END, found_semicolon = 0x0, ignore_space = false, stmt_prepare_mode = false, multi_statements = true, yylineno = 59, m_digest = 0x0, in_comment = NO_COMMENT, in_comment_saved = (PRESERVE_COMMENT | DISCARD_COMMENT | unknown: 0x7f90), m_cpp_text_start = 0x7f938801425f "id ASC", m_cpp_text_end = 0x7f9388014261 " ASC", m_underscore_cs = 0x0}, m_yacc = {yacc_yyss = 0x0, yacc_yyvs = 0x0, m_set_signal_info = {m_item = {0x0 <repeats 13 times>}}, m_lock_type = TL_READ_DEFAULT, m_mdl_type = MDL_SHARED_READ}, m_digest_psi = 0x0}
        net = 0x7f93880013e0
        error = false
        do_end_of_statement = true
        drop_more_results = false
        __FUNCTION__ = "dispatch_command"
        res = <optimized out>
#17 0x000055c893454158 in do_command (thd=0x7f9388001128, blocking=blocking@entry=true) at ./sql/sql_parse.cc:1405
        return_value = <optimized out>
        packet = 0x7f93880087a8 "\002"
        packet_length = <optimized out>
        net = 0x7f93880013e0
        command = COM_QUERY
        __FUNCTION__ = "do_command"
#18 0x000055c89357b8cf in do_handle_one_connection (connect=<optimized out>, put_in_cache=true) at ./sql/sql_connect.cc:1445
        create_user = true
        thr_create_utime = <optimized out>
        thd = 0x7f9388001128
#19 0x000055c89357bc1d in handle_one_connection (arg=arg@entry=0x55c895fcf5e8) at ./sql/sql_connect.cc:1347
        connect = 0x55c895fcf5e8
#20 0x000055c8938f3fa6 in pfs_spawn_thread (arg=0x55c895f78738) at ./storage/perfschema/pfs.cc:2201
        typed_arg = 0x55c895f78738
        user_arg = 0x55c895fcf5e8
        user_start_routine = 0x55c89357bbc0 <handle_one_connection(void*)>
        pfs = <optimized out>
        klass = <optimized out>
#21 0x00007f93be0c8b43 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442
        ret = <optimized out>
        pd = <optimized out>
        out = <optimized out>
        unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140723031921296, 1783092640928857216, 140272492185152, 2, 140272525412432, 140723031921648, -1758187890909807488, -1758183593111225216}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
        not_first_call = <optimized out>
#22 0x00007f93be159bb4 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:100
No locals.
 
Thread 15 (Thread 0x7f93bc163640 (LWP 145) "mariadbd"):
#0  0x00007f93be07721a in __GI___sigtimedwait (set=0x7f93bc162b50, info=0x7f93bc162bd0, timeout=0x0) at ../sysdeps/unix/sysv/linux/sigtimedwait.c:61
        result = <optimized out>
#1  0x000055c89333b11b in my_sigwait (code=<synthetic pointer>, sig=0x7f93bc162b48, set=0x7f93bc162b50) at ./include/my_pthread.h:195
        siginfo = {si_signo = 135168, si_errno = 0, si_code = 135168, __pad0 = 0, _sifields = {_pad = {10, 0, 41, 58, 1, 0, 0, 0, 0, 0, 124, 119, 110, 92, 2904, 0, -1571530800, 32764, -448, -1, 1744830512, 32659, 586290432, 1668133314, -1779233544, 21960, -448, -1}, _kill = {si_pid = 10, si_uid = 0}, _timer = {si_tid = 10, si_overrun = 0, si_sigval = {sival_int = 41, sival_ptr = 0x3a00000029}}, _rt = {si_pid = 10, si_uid = 0, si_sigval = {sival_int = 41, sival_ptr = 0x3a00000029}}, _sigchld = {si_pid = 10, si_uid = 0, si_status = 41, si_utime = 1, si_stime = 0}, _sigfault = {si_addr = 0xa, si_addr_lsb = 41, _bounds = {_addr_bnd = {_lower = 0x1, _upper = 0x0}, _pkey = 1}}, _sigpoll = {si_band = 10, si_fd = 41}, _sigsys = {_call_addr = 0xa, _syscall = 41, _arch = 58}}}
        siginfo = <optimized out>
#2  signal_hand (arg=arg@entry=0x0) at ./sql/mysqld.cc:3277
        error = <optimized out>
        origin = <optimized out>
        set = {__val = {548869, 0 <repeats 11 times>, 176093659136, 640, 0, 140272525466965}}
        sig = -1
#3  0x000055c8938f3fa6 in pfs_spawn_thread (arg=0x55c895f30cf8) at ./storage/perfschema/pfs.cc:2201
        typed_arg = 0x55c895f30cf8
        user_arg = 0x0
        user_start_routine = 0x55c89333afd0 <signal_hand(void*)>
        pfs = <optimized out>
        klass = <optimized out>
#4  0x00007f93be0c8b43 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442
        ret = <optimized out>
        pd = <optimized out>
        out = <optimized out>
        unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140723031922288, 1783092640928857216, 140272492492352, 2, 140272525412432, 140723031922640, -1758187936543835008, -1758183593111225216}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
        not_first_call = <optimized out>
#5  0x00007f93be159bb4 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:100
No locals.
 
Thread 13 (Thread 0x7f9385ffb640 (LWP 143) "mariadbd"):
#0  __futex_abstimed_wait_common64 (private=586290432, cancel=true, abstime=0x7f9385ffabf0, op=137, expected=0, futex_word=0x55c895a9cfd8) at ./nptl/futex-internal.c:57
        sc_cancel_oldtype = 0
        sc_ret = <optimized out>
        resultvar = <optimized out>
        __arg6 = <optimized out>
        __arg5 = <optimized out>
        __arg4 = <optimized out>
        __arg3 = <optimized out>
        __arg2 = <optimized out>
        __arg1 = <optimized out>
        _a6 = <optimized out>
        _a5 = <optimized out>
        _a4 = <optimized out>
        _a3 = <optimized out>
        _a2 = <optimized out>
        _a1 = <optimized out>
#1  __futex_abstimed_wait_common (cancel=true, private=586290432, abstime=0x7f9385ffabf0, clockid=0, expected=0, futex_word=0x55c895a9cfd8) at ./nptl/futex-internal.c:87
        err = <optimized out>
        clockbit = 256
        op = 137
        err = <optimized out>
        clockbit = <optimized out>
        op = <optimized out>
#2  __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x55c895a9cfd8, expected=expected@entry=0, clockid=clockid@entry=1, abstime=abstime@entry=0x7f9385ffabf0, private=private@entry=0) at ./nptl/futex-internal.c:139
No locals.
#3  0x00007f93be0c835d in __pthread_cond_wait_common (abstime=0x7f9385ffabf0, clockid=1, mutex=0x55c895a9ca20, cond=0x55c895a9cfb0) at ./nptl/pthread_cond_wait.c:503
        spin = 0
        buffer = {__routine = 0x7f93be0c77a0 <__condvar_cleanup_waiting>, __arg = 0x7f9385ffab70, __canceltype = 1, __prev = 0x0}
        cbuffer = {wseq = 424, cond = 0x55c895a9cfb0, mutex = 0x55c895a9ca20, private = 0}
        err = <optimized out>
        g = 0
        flags = <optimized out>
        g1_start = <optimized out>
        maxspin = 0
        signals = <optimized out>
        result = 0
        wseq = 424
        seq = 212
        private = 0
        maxspin = <optimized out>
        err = <optimized out>
        result = <optimized out>
        wseq = <optimized out>
        g = <optimized out>
        seq = <optimized out>
        flags = <optimized out>
        private = <optimized out>
        signals = <optimized out>
        done = <optimized out>
        g1_start = <optimized out>
        spin = <optimized out>
        buffer = <optimized out>
        cbuffer = <optimized out>
        s = <optimized out>
#4  ___pthread_cond_clockwait64 (abstime=0x7f9385ffabf0, clockid=1, mutex=0x55c895a9ca20, cond=0x55c895a9cfb0) at ./nptl/pthread_cond_wait.c:691
No locals.
#5  ___pthread_cond_clockwait64 (cond=0x55c895a9cfb0, mutex=0x55c895a9ca20, clockid=1, abstime=0x7f9385ffabf0) at ./nptl/pthread_cond_wait.c:679
No locals.
#6  0x000055c893b8232f in std::__condvar::wait_until (__abs_time=@0x7f9385ffabf0: {tv_sec = 11935, tv_nsec = 466378050}, __clock=1, __m=<optimized out>, this=0x55c895a9cfb0) at /usr/include/c++/11/bits/std_mutex.h:123
No locals.
#7  std::condition_variable::__wait_until_impl<std::chrono::duration<long, std::ratio<1l, 1000000000l> > > (__lock=@0x7f9385ffac40: {_M_device = 0x55c895a9ca20, _M_owns = true}, __lock=@0x7f9385ffac40: {_M_device = 0x55c895a9ca20, _M_owns = true}, __atime=<optimized out>, this=0x55c895a9cfb0) at /usr/include/c++/11/condition_variable:201
        __s = <optimized out>
        __ns = <optimized out>
        __ts = {tv_sec = 11935, tv_nsec = 466378050}
#8  std::condition_variable::wait_until<std::chrono::duration<long, std::ratio<1l, 1000000000l> > > (__atime=<optimized out>, __lock=@0x7f9385ffac40: {_M_device = 0x55c895a9ca20, _M_owns = true}, this=0x55c895a9cfb0) at /usr/include/c++/11/condition_variable:111
No locals.
#9  std::condition_variable::wait_for<long, std::ratio<1l, 1000l> > (__rtime=@0x55c895a9ca48: {__r = 60000}, __rtime=@0x55c895a9ca48: {__r = 60000}, __lock=@0x7f9385ffac40: {_M_device = 0x55c895a9ca20, _M_owns = true}, this=0x55c895a9cfb0) at /usr/include/c++/11/condition_variable:163
No locals.
#10 tpool::thread_pool_generic::wait_for_tasks (this=this@entry=0x55c895a9c900, lk=@0x7f9385ffac40: {_M_device = 0x55c895a9ca20, _M_owns = true}, thread_data=thread_data@entry=0x55c895a9cfb0) at ./tpool/tpool_generic.cc:480
        __PRETTY_FUNCTION__ = "bool tpool::thread_pool_generic::wait_for_tasks(std::unique_lock<std::mutex>&, tpool::worker_data*)"
#11 0x000055c893b82b68 in tpool::thread_pool_generic::get_task (this=0x55c895a9c900, thread_var=0x55c895a9cfb0, t=0x7f9385ffac88) at ./tpool/tpool_generic.cc:533
        lk = {_M_device = 0x55c895a9ca20, _M_owns = true}
#12 0x000055c893b82cd5 in tpool::thread_pool_generic::worker_main (this=0x55c895a9c900, thread_var=0x55c895a9cfb0) at ./tpool/tpool_generic.cc:578
        task = 0x55c894e85ce0 <purge_worker_task>
#13 0x00007f93be41f2b3 in ?? () from /lib/x86_64-linux-gnu/libstdc++.so.6
No symbol table info available.
#14 0x00007f93be0c8b43 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442
        ret = <optimized out>
        pd = <optimized out>
        out = <optimized out>
        unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140271618615344, 1783092640928857216, 140271585048128, 0, 140272525412432, 140271618615696, -1758133729761592192, -1758183593111225216}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
        not_first_call = <optimized out>
#15 0x00007f93be159bb4 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:100
No locals.
 
Thread 8 (Thread 0x7f93bc1ae640 (LWP 138) "mariadbd"):
#0  __futex_abstimed_wait_common64 (private=0, cancel=true, abstime=0x0, op=393, expected=0, futex_word=0x55c8945faf4c <COND_manager+44>) at ./nptl/futex-internal.c:57
        sc_cancel_oldtype = 0
        sc_ret = <optimized out>
        resultvar = <optimized out>
        __arg6 = <optimized out>
        __arg5 = <optimized out>
        __arg4 = <optimized out>
        __arg3 = <optimized out>
        __arg2 = <optimized out>
        __arg1 = <optimized out>
        _a6 = <optimized out>
        _a5 = <optimized out>
        _a4 = <optimized out>
        _a3 = <optimized out>
        _a2 = <optimized out>
        _a1 = <optimized out>
#1  __futex_abstimed_wait_common (cancel=true, private=0, abstime=0x0, clockid=0, expected=0, futex_word=0x55c8945faf4c <COND_manager+44>) at ./nptl/futex-internal.c:87
        err = <optimized out>
        clockbit = 256
        op = 393
        err = <optimized out>
        clockbit = <optimized out>
        op = <optimized out>
#2  __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x55c8945faf4c <COND_manager+44>, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x0, private=private@entry=0) at ./nptl/futex-internal.c:139
No locals.
#3  0x00007f93be0c7ac1 in __pthread_cond_wait_common (abstime=0x0, clockid=0, mutex=0x55c8945faf60 <LOCK_manager>, cond=0x55c8945faf20 <COND_manager>) at ./nptl/pthread_cond_wait.c:503
        spin = 0
        buffer = {__routine = 0x7f93be0c77a0 <__condvar_cleanup_waiting>, __arg = 0x7f93bc1adbd0, __canceltype = -2147455128, __prev = 0x0}
        cbuffer = {wseq = 3, cond = 0x55c8945faf20 <COND_manager>, mutex = 0x55c8945faf60 <LOCK_manager>, private = 0}
        err = <optimized out>
        g = 1
        flags = <optimized out>
        g1_start = <optimized out>
        maxspin = 0
        signals = <optimized out>
        result = 0
        wseq = 3
        seq = 1
        private = 0
        maxspin = <optimized out>
        err = <optimized out>
        result = <optimized out>
        wseq = <optimized out>
        g = <optimized out>
        seq = <optimized out>
        flags = <optimized out>
        private = <optimized out>
        signals = <optimized out>
        done = <optimized out>
        g1_start = <optimized out>
        spin = <optimized out>
        buffer = <optimized out>
        cbuffer = <optimized out>
        s = <optimized out>
#4  ___pthread_cond_wait (cond=0x55c8945faf20 <COND_manager>, mutex=0x55c8945faf60 <LOCK_manager>) at ./nptl/pthread_cond_wait.c:627
No locals.
#5  0x000055c89343e630 in inline_mysql_cond_wait (that=0x55c8945faf20 <COND_manager>, mutex=0x55c8945faf60 <LOCK_manager>, src_file=0x55c893cd4857 "./sql/sql_manager.cc", src_line=103) at ./include/mysql/psi/mysql_thread.h:1072
No locals.
#6  handle_manager (arg=arg@entry=0x0) at ./sql/sql_manager.cc:103
        cb = <optimized out>
        error = 0
        abstime = {tv_sec = 94319991316792, tv_nsec = 0}
        reset_flush_time = true
#7  0x000055c8938f3fa6 in pfs_spawn_thread (arg=0x55c89593e138) at ./storage/perfschema/pfs.cc:2201
        typed_arg = 0x55c89593e138
        user_arg = 0x0
        user_start_routine = 0x55c89343e580 <handle_manager(void*)>
        pfs = <optimized out>
        klass = <optimized out>
#8  0x00007f93be0c8b43 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442
        ret = <optimized out>
        pd = <optimized out>
        out = <optimized out>
        unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140723031913456, 1783092640928857216, 140272492799552, 0, 140272525412432, 140723031913808, -1758187834001490816, -1758183593111225216}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
        not_first_call = <optimized out>
#9  0x00007f93be159bb4 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:100
No locals.
 
Thread 4 (Thread 0x7f939c9ff640 (LWP 134) "mariadbd"):
#0  __futex_abstimed_wait_common64 (private=0, cancel=true, abstime=0x0, op=393, expected=0, futex_word=0x55c89454afa8 <buf_pool+17192>) at ./nptl/futex-internal.c:57
        sc_cancel_oldtype = 0
        sc_ret = <optimized out>
        resultvar = <optimized out>
        __arg6 = <optimized out>
        __arg5 = <optimized out>
        __arg4 = <optimized out>
        __arg3 = <optimized out>
        __arg2 = <optimized out>
        __arg1 = <optimized out>
        _a6 = <optimized out>
        _a5 = <optimized out>
        _a4 = <optimized out>
        _a3 = <optimized out>
        _a2 = <optimized out>
        _a1 = <optimized out>
#1  __futex_abstimed_wait_common (cancel=true, private=0, abstime=0x0, clockid=0, expected=0, futex_word=0x55c89454afa8 <buf_pool+17192>) at ./nptl/futex-internal.c:87
        err = <optimized out>
        clockbit = 256
        op = 393
        err = <optimized out>
        clockbit = <optimized out>
        op = <optimized out>
#2  __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x55c89454afa8 <buf_pool+17192>, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x0, private=private@entry=0) at ./nptl/futex-internal.c:139
No locals.
#3  0x00007f93be0c7ac1 in __pthread_cond_wait_common (abstime=0x0, clockid=0, mutex=0x55c89454af00 <buf_pool+17024>, cond=0x55c89454af80 <buf_pool+17152>) at ./nptl/pthread_cond_wait.c:503
        spin = 0
        buffer = {__routine = 0x7f93be0c77a0 <__condvar_cleanup_waiting>, __arg = 0x7f939c9febb0, __canceltype = -1806406528, __prev = 0x0}
        cbuffer = {wseq = 2, cond = 0x55c89454af80 <buf_pool+17152>, mutex = 0x55c89454af00 <buf_pool+17024>, private = 0}
        err = <optimized out>
        g = 0
        flags = <optimized out>
        g1_start = <optimized out>
        maxspin = 0
        signals = <optimized out>
        result = 0
        wseq = 2
        seq = 1
        private = 0
        maxspin = <optimized out>
        err = <optimized out>
        result = <optimized out>
        wseq = <optimized out>
        g = <optimized out>
        seq = <optimized out>
        flags = <optimized out>
        private = <optimized out>
        signals = <optimized out>
        done = <optimized out>
        g1_start = <optimized out>
        spin = <optimized out>
        buffer = <optimized out>
        cbuffer = <optimized out>
        s = <optimized out>
#4  ___pthread_cond_wait (cond=0x55c89454af80 <buf_pool+17152>, mutex=0x55c89454af00 <buf_pool+17024>) at ./nptl/pthread_cond_wait.c:627
No locals.
#5  0x000055c893af5069 in buf_flush_page_cleaner () at ./storage/innobase/buf/buf0flu.cc:2355
        soft_lsn_limit = <optimized out>
        oldest_lsn = <optimized out>
        pct_lwm = <optimized out>
        n_flushed = <optimized out>
        n = <optimized out>
        dirty_blocks = <optimized out>
        dirty_pct = <optimized out>
        last_pages = 0
        abstime = {tv_sec = 1690556399, tv_nsec = 513407000}
        lsn_limit = 0
        last_activity_count = 0
#6  0x00007f93be41f2b3 in ?? () from /lib/x86_64-linux-gnu/libstdc++.so.6
No symbol table info available.
#7  0x00007f93be0c8b43 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442
        ret = <optimized out>
        pd = <optimized out>
        out = <optimized out>
        unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140723031903824, 1783092640928857216, 140271964649024, 12, 140272525412432, 140723031904176, -1758118609329226624, -1758183593111225216}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
        not_first_call = <optimized out>
#8  0x00007f93be159bb4 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:100
No locals.
 
Thread 3 (Thread 0x7f93a9759640 (LWP 133) "mariadbd"):
#0  __futex_abstimed_wait_common64 (private=0, cancel=true, abstime=0x7f93a9758c20, op=393, expected=0, futex_word=0x55c894e2ece8 <COND_checkpoint+40>) at ./nptl/futex-internal.c:57
        sc_cancel_oldtype = 0
        sc_ret = <optimized out>
        resultvar = <optimized out>
        __arg6 = <optimized out>
        __arg5 = <optimized out>
        __arg4 = <optimized out>
        __arg3 = <optimized out>
        __arg2 = <optimized out>
        __arg1 = <optimized out>
        _a6 = <optimized out>
        _a5 = <optimized out>
        _a4 = <optimized out>
        _a3 = <optimized out>
        _a2 = <optimized out>
        _a1 = <optimized out>
#1  __futex_abstimed_wait_common (cancel=true, private=0, abstime=0x7f93a9758c20, clockid=0, expected=0, futex_word=0x55c894e2ece8 <COND_checkpoint+40>) at ./nptl/futex-internal.c:87
        err = <optimized out>
        clockbit = 256
        op = 393
        err = <optimized out>
        clockbit = <optimized out>
        op = <optimized out>
#2  __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x55c894e2ece8 <COND_checkpoint+40>, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x7f93a9758c20, private=private@entry=0) at ./nptl/futex-internal.c:139
No locals.
#3  0x00007f93be0c7f1b in __pthread_cond_wait_common (abstime=0x7f93a9758c20, clockid=0, mutex=0x55c894e2ed00 <LOCK_checkpoint>, cond=0x55c894e2ecc0 <COND_checkpoint>) at ./nptl/pthread_cond_wait.c:503
        spin = 0
        buffer = {__routine = 0x7f93be0c77a0 <__condvar_cleanup_waiting>, __arg = 0x7f93a9758bb0, __canceltype = 0, __prev = 0x0}
        cbuffer = {wseq = 14, cond = 0x55c894e2ecc0 <COND_checkpoint>, mutex = 0x55c894e2ed00 <LOCK_checkpoint>, private = 0}
        err = <optimized out>
        g = 0
        flags = <optimized out>
        g1_start = <optimized out>
        maxspin = 0
        signals = <optimized out>
        result = 0
        wseq = 14
        seq = 7
        private = 0
        maxspin = <optimized out>
        err = <optimized out>
        result = <optimized out>
        wseq = <optimized out>
        g = <optimized out>
        seq = <optimized out>
        flags = <optimized out>
        private = <optimized out>
        signals = <optimized out>
        done = <optimized out>
        g1_start = <optimized out>
        spin = <optimized out>
        buffer = <optimized out>
        cbuffer = <optimized out>
        s = <optimized out>
#4  ___pthread_cond_timedwait64 (cond=0x55c894e2ecc0 <COND_checkpoint>, mutex=0x55c894e2ed00 <LOCK_checkpoint>, abstime=abstime@entry=0x7f93a9758c20) at ./nptl/pthread_cond_wait.c:652
        flags = <optimized out>
        clockid = 0
#5  0x000055c893887df2 in inline_mysql_cond_timedwait (src_file=0x55c893e392b0 "./storage/maria/ma_servicethread.c", src_line=115, abstime=0x7f93a9758c20, mutex=<optimized out>, that=<optimized out>) at ./include/mysql/psi/mysql_thread.h:1088
No locals.
#6  my_service_thread_sleep (control=control@entry=0x55c894531dc0 <checkpoint_control>, sleep_time=sleep_time@entry=30000000000) at ./storage/maria/ma_servicethread.c:115
        abstime = {tv_sec = 1690556636, tv_nsec = 525930000}
        res = 0 '\000'
#7  0x000055c89388015e in ma_checkpoint_background (arg=0x1e) at ./storage/maria/ma_checkpoint.c:725
        interval = 30
        sleeps = 210
        sleep_time = 30
        log_horizon_at_last_checkpoint = 4311840914
        pagecache_flushes_at_last_checkpoint = 0
        pages_bunch_size = 0
        filter_param = {up_to_lsn = 4311840870, max_pages = 0}
        dfile = 0x0
        kfile = 0x0
#8  0x00007f93be0c8b43 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442
        ret = <optimized out>
        pd = <optimized out>
        out = <optimized out>
        unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140723031909424, 1783092640928857216, 140272179975744, 0, 140272525412432, 140723031909776, -1758229437739074432, -1758183593111225216}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
        not_first_call = <optimized out>
#9  0x00007f93be159bb4 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:100
No locals.
 
Thread 2 (Thread 0x7f93bdb80640 (LWP 132) "mariadbd"):
#0  __futex_abstimed_wait_common64 (private=0, cancel=true, abstime=0x7f93bdb7fc90, op=393, expected=0, futex_word=0x55c894ebbd8c <COND_timer+44>) at ./nptl/futex-internal.c:57
        sc_cancel_oldtype = 0
        sc_ret = <optimized out>
        resultvar = <optimized out>
        __arg6 = <optimized out>
        __arg5 = <optimized out>
        __arg4 = <optimized out>
        __arg3 = <optimized out>
        __arg2 = <optimized out>
        __arg1 = <optimized out>
        _a6 = <optimized out>
        _a5 = <optimized out>
        _a4 = <optimized out>
        _a3 = <optimized out>
        _a2 = <optimized out>
        _a1 = <optimized out>
#1  __futex_abstimed_wait_common (cancel=true, private=0, abstime=0x7f93bdb7fc90, clockid=0, expected=0, futex_word=0x55c894ebbd8c <COND_timer+44>) at ./nptl/futex-internal.c:87
        err = <optimized out>
        clockbit = 256
        op = 393
        err = <optimized out>
        clockbit = <optimized out>
        op = <optimized out>
#2  __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x55c894ebbd8c <COND_timer+44>, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x7f93bdb7fc90, private=private@entry=0) at ./nptl/futex-internal.c:139
No locals.
#3  0x00007f93be0c7f1b in __pthread_cond_wait_common (abstime=0x7f93bdb7fc90, clockid=0, mutex=0x55c894ebbda0 <LOCK_timer>, cond=0x55c894ebbd60 <COND_timer>) at ./nptl/pthread_cond_wait.c:503
        spin = 0
        buffer = {__routine = 0x7f93be0c77a0 <__condvar_cleanup_waiting>, __arg = 0x7f93bdb7fc00, __canceltype = -1784034784, __prev = 0x0}
        cbuffer = {wseq = 1755, cond = 0x55c894ebbd60 <COND_timer>, mutex = 0x55c894ebbda0 <LOCK_timer>, private = 0}
        err = <optimized out>
        g = 1
        flags = <optimized out>
        g1_start = <optimized out>
        maxspin = 0
        signals = <optimized out>
        result = 0
        wseq = 1755
        seq = 877
        private = 0
        maxspin = <optimized out>
        err = <optimized out>
        result = <optimized out>
        wseq = <optimized out>
        g = <optimized out>
        seq = <optimized out>
        flags = <optimized out>
        private = <optimized out>
        signals = <optimized out>
        done = <optimized out>
        g1_start = <optimized out>
        spin = <optimized out>
        buffer = <optimized out>
        cbuffer = <optimized out>
        s = <optimized out>
#4  ___pthread_cond_timedwait64 (cond=cond@entry=0x55c894ebbd60 <COND_timer>, mutex=mutex@entry=0x55c894ebbda0 <LOCK_timer>, abstime=abstime@entry=0x7f93bdb7fc90) at ./nptl/pthread_cond_wait.c:652
        flags = <optimized out>
        clockid = 0
#5  0x000055c893bf25a7 in inline_mysql_cond_timedwait (that=0x55c894ebbd60 <COND_timer>, mutex=0x55c894ebbda0 <LOCK_timer>, src_file=0x55c893eb1b70 "./mysys/thr_timer.c", src_line=321, abstime=0x7f93bdb7fc90) at ./include/mysql/psi/mysql_thread.h:1088
No locals.
#6  timer_handler (arg=<optimized out>) at ./mysys/thr_timer.c:321
        error = <optimized out>
        top_time = 0x55c895a9cb20
        now = <optimized out>
        abstime = {tv_sec = 1690556624, tv_nsec = 774721000}
#7  0x00007f93be0c8b43 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442
        ret = <optimized out>
        pd = <optimized out>
        out = <optimized out>
        unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140723031913424, 1783092640928857216, 140272519874112, 22, 140272525412432, 140723031913776, -1758187122110661504, -1758183593111225216}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
        not_first_call = <optimized out>
#8  0x00007f93be159bb4 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:100
No locals.
 
Thread 1 (Thread 0x7f93bdcfd8c0 (LWP 1) "mariadbd"):
#0  0x00007f93be14cd7f in __GI___poll (fds=0x55c895f4fa48, nfds=3, timeout=-1) at ../sysdeps/unix/sysv/linux/poll.c:29
        sc_ret = -516
        sc_cancel_oldtype = 0
        sc_ret = <optimized out>
#1  0x000055c8933437c2 in poll (__timeout=-1, __nfds=<optimized out>, __fds=<optimized out>) at /usr/include/x86_64-linux-gnu/bits/poll2.h:39
No locals.
#2  handle_connections_sockets () at ./sql/mysqld.cc:6362
        sock = {fd = 24, is_unix_domain_socket = <optimized out>, is_extra_port = <optimized out>, address_family = <optimized out>, m_psi = 0x0}
        error_count = 0
        cAddr = {ss_family = 1, __ss_padding = "\000\000\374\177\000\000\000\000\000\000\000\000\000\000Ї\371\225\310U\000\000\000\025\362\"±mc\340l_\224\310U\000\000\000\025\362\"±mc\000\000\000\000\000\000\000\000\240\332$\276\223\177\000\000$\nÌ“\310U\000\000O\221\024\276\223\177\000\000\000\000\000\000\000\000\000\000\310l_\224", '\000' <repeats 12 times>, "\353\320\v\276\223\177\000\000\244/\343\223\310U\000", __ss_align = 0}
        retval = <optimized out>
        fds = {array = {buffer = 0x55c895f4fa48 "\026", elements = 3, max_element = 16, alloc_increment = 16, size_of_element = 8, m_psi_key = 0, malloc_flags = 0}}
#3  0x000055c893344ab5 in mysqld_main (argc=<optimized out>, argv=<optimized out>) at ./sql/mysqld.cc:6064
        please_close_stdin = true
        ho_error = <optimized out>
        new_thread_stack_size = <optimized out>
        user = <optimized out>
#4  0x00007f93be05dd90 in __libc_start_call_main (main=main@entry=0x55c8932f7130 <main(int, char**)>, argc=argc@entry=1, argv=argv@entry=0x7ffca2545fa8) at ../sysdeps/nptl/libc_start_call_main.h:58
        self = <optimized out>
        result = <optimized out>
        unwind_buf = {cancel_jmp_buf = {{jmp_buf = {0, 1783092640928988288, 140723031924648, 94319951180080, 94319968675608, 140272537841728, -1781290304114164608, -1758183667871155072}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x55c8932f7130 <main(int, char**)>, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = -1825607376}}}
        not_first_call = <optimized out>
#5  0x00007f93be05de40 in __libc_start_main_impl (main=0x55c8932f7130 <main(int, char**)>, argc=1, argv=0x7ffca2545fa8, init=<optimized out>, fini=<optimized out>, rtld_fini=<optimized out>, stack_end=0x7ffca2545f98) at ../csu/libc-start.c:392
No locals.
#6  0x000055c893338585 in _start ()
No symbol table info available.
Warning: 'set logging off', an alias for the command 'set logging enabled', is deprecated.
Use 'set logging enabled off'.

How to reproduce

Image

Spin up a >= 11.0 docker image (the host OS seems to be irrelevant, I've been able to reproduce it on Ubuntu 20.04 and 22.04 as well as on Windows 10):

docker run --name=mariadb -e MARIADB_ROOT_PASSWORD=root -e MARIADB_DATABASE=test -p 3306:3306 quay.io/mariadb-foundation/mariadb-debug:11.2

Run SQL

Save the following SQL in a file (crash-test.sql in this example, also attached for convenience) and import/run it (don't mind the odd duplication in the query, it's from the context of an ORM, where transparently adding group wise limits via ranking is most safely done by duplicating the original query):

docker exec -i mariadb mariadb -uroot -proot test < crash-test.sql

DROP TABLE IF EXISTS `comments`;
CREATE TABLE `comments` (
    `id` INTEGER,
    `id2` INTEGER,
    `article_id` INTEGER,
    `article_id2` INTEGER,
    `author_id` INTEGER,
    `votes` INTEGER,
    `body` TEXT,
    `published` BOOLEAN,
    PRIMARY KEY (`id`),
    KEY `comments_article_id` (`article_id`, `article_id2`),
    KEY `comments_top_comments_sort` (`votes`, `id`)
);
 
DROP TABLE IF EXISTS `comments_i18n`;
CREATE TABLE `comments_i18n` (
    `id` INTEGER AUTO_INCREMENT,
    `locale` VARCHAR(255),
    `model` VARCHAR(255),
    `foreign_key` INTEGER,
    `field` VARCHAR(255),
    `content` TEXT,
    PRIMARY KEY (`id`),
    UNIQUE KEY `comments_I18N_LOCALE_FIELD` (`locale`, `model`, `foreign_key`, `field`),
    KEY `comments_I18N_FIELD` (`model`, `foreign_key`, `field`)
);
 
INSERT INTO `comments` (`id`, `id2`, `article_id`, `article_id2`, `author_id`, `votes`, `body`, `published`) VALUES
    (1, 2, 1, 2, 1, 1, 'Lorem ipsum dolor sit amet', 1),
    (2, 3, 1, 2, 1, 2, 'Lorem ipsum dolor sit amet', 1),
    (3, 4, 1, 2, 2, 3, 'Lorem ipsum dolor sit amet', 1),
    (4, 5, 1, 2, 2, 4, 'Lorem ipsum dolor sit amet', 0),
    (5, 6, 2, 3, 1, 10, 'Lorem ipsum dolor sit amet', 1),
    (6, 7, 2, 3, 1, 9, 'Lorem ipsum dolor sit amet', 1),
    (7, 8, 2, 3, 2, 8, 'Lorem ipsum dolor sit amet', 1),
    (8, 9, 2, 3, 2, 7, 'Lorem ipsum dolor sit amet', 0);
 
INSERT INTO `comments_i18n` (`locale`, `model`, `foreign_key`, `field`, `content`) VALUES
    ('de_DE', 'Comments', 1, 'body', 'Franz jagt im komplett verwahrlosten Taxi quer durch Bayern'),
    ('de_DE', 'Comments', 2, 'body', 'Der schnelle braune Fuchs springt über den faulen Hund'),
    ('de_DE', 'Comments', 5, 'body', 'Fix Schwyz! quäkt Jürgen blöd vom Paß'),
    ('de_DE', 'Comments', 6, 'body', 'Quod erat demonstrandum.');
 
SELECT
    TopComments.id AS TopComments__id,
    TopComments.article_id AS TopComments__article_id,
    TopComments.body AS TopComments__body,
    TopComments.votes AS TopComments__votes,
    TopComments_body_translation.id AS TopComments_body_translation__id,
    TopComments_body_translation.content AS TopComments_body_translation__content
FROM
    comments TopComments
INNER JOIN
    comments_i18n TopComments_body_translation ON (
        TopComments_body_translation.model = 'Comments'
        AND TopComments_body_translation.field = 'body'
        AND TopComments_body_translation.locale = 'de_DE'
        AND TopComments.id = TopComments_body_translation.foreign_key
    )
WHERE (
    TopComments.article_id in (1, 2)
    AND TopComments.id in (
        SELECT
            __ranked__TopComments.id AS id
        FROM (
            SELECT
                TopComments.id AS TopComments__id,
                TopComments.article_id AS TopComments__article_id,
                TopComments.body AS TopComments__body,
                TopComments.votes AS TopComments__votes,
                TopComments.id AS id,
                (
                    ROW_NUMBER() OVER (
                        PARTITION BY
                            TopComments.article_id
                        ORDER BY
                            TopComments.votes DESC,
                            TopComments.id ASC
                    )
                ) AS __row_number,
                TopComments_body_translation.id AS TopComments_body_translation__id,
                TopComments_body_translation.content AS TopComments_body_translation__content
            FROM
                comments TopComments
            INNER JOIN
                comments_i18n TopComments_body_translation ON (
                    TopComments_body_translation.model = 'Comments'
                    AND TopComments_body_translation.field = 'body'
                    AND TopComments_body_translation.locale = 'de_DE'
                    AND TopComments.id = TopComments_body_translation.foreign_key
                )
            WHERE
                TopComments.article_id in (1, 2)
        )
        __ranked__TopComments
        WHERE
            __ranked__TopComments.__row_number <= 3
    )
)
ORDER BY
    TopComments.votes DESC,
    TopComments.id ASC;
 
DROP TABLE `comments`;
DROP TABLE `comments_i18n`;

Repeat

Repeat the import step over and over again until the server eventually crashes. You might be able to reproduce by running only the last select query that triggers the crash, personally however I wasn't, I have to run the whole thing with table creation, so that might be related too. And as mentioned initially, it might happen immediately, it might take many many retries, but eventually it should happen. Run it in a loop if neccesary.

for (( i = 1; i <= 1000; i ++ ))
do
    echo "attempt $i"
    docker exec -i mariadb mariadb -uroot -proot test < crash-test.sql
    if [ $? -ne 0 ]; then break; fi
    sleep 0.2
done



 Comments   
Comment by Alice Sherepa [ 2023-07-31 ]

Thank you very much for the report!
I repeated on 11.0-11.2, not reproducible on earlier versions. I shortened test a little bit.
Please run it with --repeat=N (usually N=5 was enough) and please check the initial test case after the fix:

--source include/have_innodb.inc 
 
CREATE TABLE t1 (id int, a int, KEY (a), KEY (id)) engine=innodb;
INSERT INTO t1 VALUES  (1, 1), (2, 1), (3, 1), (4, 1), (5, 2),  (6, 2), (7, 2), (8, 2);
 
CREATE TABLE t2 (id2 int,  b TEXT) engine=innodb;
INSERT INTO t2 VALUES (1,'F'),( 2,'D'),( 5,'F'),( 6,'Q');
 
SELECT id, a 
FROM t1 JOIN t2 ON id = id2 
WHERE 
  t1.id IN (SELECT dt.id FROM  ( SELECT id, avg(a), b  FROM t1 JOIN t2 ON id = id2 )dt);
 
DROP TABLE t1,t2;

11.0 f2b4972bd4f9d3f013, -DCMAKE_BUILD_TYPE=Debug -DWITH_ASAN=ON

=================================================================
==2518469==ERROR: AddressSanitizer: use-after-poison on address 0x62d000a52f68 at pc 0x7f6177b1458d bp 0x7f61665138b0 sp 0x7f6166513058
WRITE of size 1 at 0x62d000a52f68 thread T15
    #0 0x7f6177b1458c in __interceptor_memcpy ../../../../src/libsanitizer/sanitizer_common/sanitizer_common_interceptors.inc:790
    #1 0x561063c6cee5 in Field_string::get_key_image(unsigned char*, unsigned int, unsigned char const*, Field::imagetype) const /11.0/src/sql/field.cc:7792
    #2 0x5610640096b5 in key_copy(unsigned char*, unsigned char const*, st_key const*, unsigned int, bool) /11.0/src/sql/key.cc:161
    #3 0x5610635604ca in sub_select(JOIN*, st_join_table*, bool) /11.0/src/sql/sql_select.cc:23385
    #4 0x5610635619d4 in evaluate_join_record /11.0/src/sql/sql_select.cc:23606
    #5 0x561063560249 in sub_select(JOIN*, st_join_table*, bool) /11.0/src/sql/sql_select.cc:23373
    #6 0x56106355db64 in do_select /11.0/src/sql/sql_select.cc:22890
    #7 0x5610634dc0c6 in JOIN::exec_inner() /11.0/src/sql/sql_select.cc:4924
    #8 0x5610634d9483 in JOIN::exec() /11.0/src/sql/sql_select.cc:4701
    #9 0x5610634ddb7f in mysql_select(THD*, TABLE_LIST*, List<Item>&, Item*, unsigned int, st_order*, st_order*, Item*, st_order*, unsigned long long, select_result*, st_select_lex_unit*, st_select_lex*) /11.0/src/sql/sql_select.cc:5182
    #10 0x5610634ad664 in handle_select(THD*, LEX*, select_result*, unsigned long long) /11.0/src/sql/sql_select.cc:626
    #11 0x5610633cf593 in execute_sqlcom_select /11.0/src/sql/sql_parse.cc:6279
    #12 0x5610633bdc72 in mysql_execute_command(THD*, bool) /11.0/src/sql/sql_parse.cc:3949
    #13 0x5610633da399 in mysql_parse(THD*, char*, unsigned int, Parser_state*) /11.0/src/sql/sql_parse.cc:8019
    #14 0x5610633b0380 in dispatch_command(enum_server_command, THD*, char*, unsigned int, bool) /11.0/src/sql/sql_parse.cc:1894
    #15 0x5610633ad0a6 in do_command(THD*, bool) /11.0/src/sql/sql_parse.cc:1407
    #16 0x561063882d6e in do_handle_one_connection(CONNECT*, bool) /11.0/src/sql/sql_connect.cc:1416
    #17 0x5610638826cb in handle_one_connection /11.0/src/sql/sql_connect.cc:1318
    #18 0x56106449db1b in pfs_spawn_thread /11.0/src/storage/perfschema/pfs.cc:2201
    #19 0x7f61775f9608 in start_thread /build/glibc-SzIz7B/glibc-2.31/nptl/pthread_create.c:477
    #20 0x7f61771ca132 in __clone (/lib/x86_64-linux-gnu/libc.so.6+0x11f132)
 
0x62d000a52f68 is located 11112 bytes inside of 32900-byte region [0x62d000a50400,0x62d000a58484)
allocated by thread T15 here:
    #0 0x7f6177b86808 in __interceptor_malloc ../../../../src/libsanitizer/asan/asan_malloc_linux.cc:144
    #1 0x5610650e3e7a in sf_malloc /11.0/src/mysys/safemalloc.c:126
    #2 0x5610650b1007 in my_malloc /11.0/src/mysys/my_malloc.c:88
    #3 0x56106508b9bd in root_alloc /11.0/src/mysys/my_alloc.c:71
    #4 0x56106508d260 in alloc_root /11.0/src/mysys/my_alloc.c:339
    #5 0x56106300eba1 in Query_arena::alloc(unsigned long) /11.0/src/sql/sql_class.h:1219
    #6 0x56106350fcf6 in JOIN::get_best_combination() /11.0/src/sql/sql_select.cc:12515
    #7 0x5610634c35cd in JOIN::optimize_stage2() /11.0/src/sql/sql_select.cc:2654
    #8 0x5610634c3023 in JOIN::optimize_inner() /11.0/src/sql/sql_select.cc:2633
    #9 0x5610634bbdf0 in JOIN::optimize() /11.0/src/sql/sql_select.cc:1935
    #10 0x5610634dd98e in mysql_select(THD*, TABLE_LIST*, List<Item>&, Item*, unsigned int, st_order*, st_order*, Item*, st_order*, unsigned long long, select_result*, st_select_lex_unit*, st_select_lex*) /11.0/src/sql/sql_select.cc:5168
    #11 0x5610634ad664 in handle_select(THD*, LEX*, select_result*, unsigned long long) /11.0/src/sql/sql_select.cc:626
    #12 0x5610633cf593 in execute_sqlcom_select /11.0/src/sql/sql_parse.cc:6279
    #13 0x5610633bdc72 in mysql_execute_command(THD*, bool) /11.0/src/sql/sql_parse.cc:3949
    #14 0x5610633da399 in mysql_parse(THD*, char*, unsigned int, Parser_state*) /11.0/src/sql/sql_parse.cc:8019
    #15 0x5610633b0380 in dispatch_command(enum_server_command, THD*, char*, unsigned int, bool) /11.0/src/sql/sql_parse.cc:1894
    #16 0x5610633ad0a6 in do_command(THD*, bool) /11.0/src/sql/sql_parse.cc:1407
    #17 0x561063882d6e in do_handle_one_connection(CONNECT*, bool) /11.0/src/sql/sql_connect.cc:1416
    #18 0x5610638826cb in handle_one_connection /11.0/src/sql/sql_connect.cc:1318
    #19 0x56106449db1b in pfs_spawn_thread /11.0/src/storage/perfschema/pfs.cc:2201
    #20 0x7f61775f9608 in start_thread /build/glibc-SzIz7B/glibc-2.31/nptl/pthread_create.c:477
 
Thread T15 created by T0 here:
    #0 0x7f6177ab3815 in __interceptor_pthread_create ../../../../src/libsanitizer/asan/asan_interceptors.cc:208
    #1 0x5610644996f8 in my_thread_create /11.0/src/storage/perfschema/my_thread.h:52
    #2 0x56106449df0e in pfs_spawn_thread_v1 /11.0/src/storage/perfschema/pfs.cc:2252
    #3 0x561062fe0f79 in inline_mysql_thread_create /11.0/src/include/mysql/psi/mysql_thread.h:1139
    #4 0x561062ff97c8 in create_thread_to_handle_connection(CONNECT*) /11.0/src/sql/mysqld.cc:6141
    #5 0x561062ff9e58 in create_new_thread(CONNECT*) /11.0/src/sql/mysqld.cc:6203
    #6 0x561062ffa1c5 in handle_accepted_socket(st_mysql_socket, st_mysql_socket) /11.0/src/sql/mysqld.cc:6265
    #7 0x561062ffabbb in handle_connections_sockets() /11.0/src/sql/mysqld.cc:6389
    #8 0x561062ff8fd5 in mysqld_main(int, char**) /11.0/src/sql/mysqld.cc:6036
    #9 0x561062fe002c in main /11.0/src/sql/main.cc:34
    #10 0x7f61770cf082 in __libc_start_main ../csu/libc-start.c:308
 
SUMMARY: AddressSanitizer: use-after-poison ../../../../src/libsanitizer/sanitizer_common/sanitizer_common_interceptors.inc:790 in __interceptor_memcpy
Shadow bytes around the buggy address:
  0x0c5a80142590: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x0c5a801425a0: 00 00 00 00 00 00 00 00 00 00 f7 00 00 00 00 00
  0x0c5a801425b0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x0c5a801425c0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x0c5a801425d0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
=>0x0c5a801425e0: 00 00 00 00 00 00 00 00 00 00 00 00 f7[f7]00 00
  0x0c5a801425f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x0c5a80142600: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 f7 00
  0x0c5a80142610: 00 00 00 00 00 00 00 f7 00 00 00 00 00 00 00 00
  0x0c5a80142620: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x0c5a80142630: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
Shadow byte legend (one shadow byte represents 8 application bytes):
  Addressable:           00
  Partially addressable: 01 02 03 04 05 06 07 
  Heap left redzone:       fa
  Freed heap region:       fd
  Stack left redzone:      f1
  Stack mid redzone:       f2
  Stack right redzone:     f3
  Stack after return:      f5
  Stack use after scope:   f8
  Global redzone:          f9
  Global init order:       f6
  Poisoned by user:        f7
  Container overflow:      fc
  Array cookie:            ac
  Intra object redzone:    bb
  ASan internal:           fe
  Left alloca redzone:     ca
  Right alloca redzone:    cb
  Shadow gap:              cc
==2518469==ABORTING

non-debug:

11.0.2

----------SERVER LOG START-----------
230731 15:30:22 [ERROR] mysqld got signal 11 ;
 
Server version: 11.0.2-MariaDB-log source revision: 0005f2f06c8e1aea4915887decad67885108a929
 
sql/signal_handler.cc:241(handle_fatal_signal)[0x55a5c3c8fb57]
sigaction.c:0(__restore_rt)[0x7efe1f9f8420]
sql/sql_select.cc:28254(copy_fields(TMP_TABLE_PARAM*))[0x55a5c3a970d3]
sql/sql_select.cc:24825(end_write(JOIN*, st_join_table*, bool))[0x55a5c3a97608]
sql/sql_class.h:4426(THD::get_stmt_da())[0x55a5c3a818fb]
sql/sql_select.cc:23313(sub_select(JOIN*, st_join_table*, bool))[0x55a5c3a87f86]
sql/sql_class.h:4426(THD::get_stmt_da())[0x55a5c3a818fb]
sql/sql_select.cc:23313(sub_select(JOIN*, st_join_table*, bool))[0x55a5c3a87f86]
sql/sql_class.h:4426(THD::get_stmt_da())[0x55a5c3a818fb]
sql/sql_select.cc:23313(sub_select(JOIN*, st_join_table*, bool))[0x55a5c3a87fe4]
sql/sql_select.cc:22826(JOIN::exec_inner())[0x55a5c3abb1ba]
sql/sql_select.cc:4681(JOIN::exec())[0x55a5c3abb5e3]
sql/sql_select.cc:5161(mysql_select(THD*, TABLE_LIST*, List<Item>&, Item*, unsigned int, st_order*, st_order*, Item*, st_order*, unsigned long long, select_result*, st_select_lex_unit*, st_select_lex*))[0x55a5c3ab96de]
sql/sql_select.cc:631(handle_select(THD*, LEX*, select_result*, unsigned long long))[0x55a5c3ab9f44]
sql/sql_parse.cc:6280(execute_sqlcom_select(THD*, TABLE_LIST*))[0x55a5c38cb379]
sql/sql_parse.cc:3949(mysql_execute_command(THD*, bool))[0x55a5c3a49608]
sql/sql_parse.cc:8015(mysql_parse(THD*, char*, unsigned int, Parser_state*))[0x55a5c3a4b9eb]
sql/sql_parse.cc:1953(dispatch_command(enum_server_command, THD*, char*, unsigned int, bool))[0x55a5c3a4de28]
sql/sql_parse.cc:1409(do_command(THD*, bool))[0x55a5c3a4f353]
sql/sql_connect.cc:1416(do_handle_one_connection(CONNECT*, bool))[0x55a5c3b5c2d7]
sql/sql_connect.cc:1324(handle_one_connection)[0x55a5c3b5c574]
perfschema/pfs.cc:2204(pfs_spawn_thread)[0x55a5c3ee211c]
nptl/pthread_create.c:478(start_thread)[0x7efe1f9ec609]
/lib/x86_64-linux-gnu/libc.so.6(clone+0x43)[0x7efe1f5d8133]
 
Trying to get some variables.
Some pointers may be invalid and cause the dump to abort.
Query (0x7efde4010a60): SELECT
TopComments.id AS TopComments__id,
TopComments.article_id AS TopComments__article_id,
TopComments.body AS TopComments__body,
TopComments.votes AS TopComments__votes,
TopComments_body_translation.id AS TopComments_body_translation__id,
TopComments_body_translation.content AS TopComments_body_translation__content
FROM
comments TopComments
INNER JOIN
comments_i18n TopComments_body_translation ON (
TopComments_body_translation.model = 'Comments'
        AND TopComments_body_translation.field = 'body'
        AND TopComments_body_translation.locale = 'de_DE'
        AND TopComments.id = TopComments_body_translation.foreign_key
)
WHERE (
TopComments.article_id in (1, 2)
AND TopComments.id in (
SELECT
__ranked__TopComments.id AS id
FROM (
SELECT
TopComments.id AS TopComments__id,
TopComments.article_id AS TopComments__article_id,
TopComments.body AS TopComments__body,
TopComments.votes AS TopComments__votes,
TopComments.id AS id,
(
ROW_NUMBER() OVER (
PARTITION BY
TopComments.article_id
ORDER BY
TopComments.votes DESC,
TopComments.id ASC
)
) AS __row_number,
TopComments_body_translation.id AS TopComments_body_translation__id,
TopComments_body_translation.content AS TopComments_body_translation__content
FROM
comments TopComments
INNER JOIN
comments_i18n TopComments_body_translation ON (
TopComments_body_translation.model = 'Comments'
                    AND TopComments_body_translation.field = 'body'
                    AND TopComments_body_translation.locale = 'de_DE'
                    AND TopComments.id = TopComments_body_translation.foreign_key
)
WHERE
TopComments.article_id in (1, 2)
)
__ranked__TopComments
WHERE
__ranked__TopComments.__row_number <= 3
)
)
ORDER BY
TopComments.votes DESC,
TopComments.id ASC
 
Connection ID (thread ID): 152
Status: NOT_KILLED

Comment by Rex Johnston [ 2023-11-10 ]

testing here suggests that launching the server with --innodb-stats-persistent=OFF makes the crash go away.
I'll explain once i understand why the bad query plan causes memory allocation issues.

Comment by Rex Johnston [ 2023-11-14 ]

Entropy in row count numbers is generated by background statistics collection, detailed in MDEV-32785.
Having a slightly inaccurate row count can cause unexpected query plans, in this case we are creating
a derived table and joining with 2 other tables.

SELECT id, a 
FROM t1 JOIN t2 ON id = id2 
WHERE 
t1.id IN (SELECT dt.id FROM (SELECT id, avg(a), b FROM t1 JOIN t2 ON id = id2) dt)

For the example given by Alice, dt is forced to be materialized and the IN subquery is transformed
into a semi-join, giving us a 3 way join.

In creating the derived table, 2 keys are created

"ref_optimizer_key_uses": [
  {
    "table": "t1",
    "index": "id",
    "field": "id",
    "equals": "dt.`id`",
    "null_rejecting": true
  },
  {
    "table": "t1",
    "index": "id",
    "field": "id",
    "equals": "t2.id2",
    "null_rejecting": true
  },
  {
    "table": "<derived3>",
    "index": "key1",
    "field": "id",
    "equals": "t1.`id`",
    "null_rejecting": true
  },
  {
    "table": "<derived3>",
    "index": "key2",
    "field": "id",
    "equals": "t2.id2",
    "null_rejecting": true
  }
]

The normal join order/method above is t2,<derived3> with a scan + join cache to t1.

In one observed trace, inaccurate row counts cause the plan to change to t2,<derived3> with a "LooseScan"
join to t1, using "key1" shown above.

These keys are created in Create_tmp_table::finalize(), but the code was missing
initialization of key length in the very first key created in some circumstances.

Attached are the 2 optimizer traces for Alice's example
good_trace: the normal optimizer trace
bad_trace: the optimizer trace collected before crashing.

The difference is that make_join_readinfo() when observing that a LooseScan join
is to be performed will now try to allocate some space for the key to be joined
in. Prior to the patch, this was zero, left that way from the bzero call
clearing the memory. Subsequently asking the alloc() routine to allocate zero
bytes seemed perfectly OK, but of course without any actual allocation there
were unpredictable results and asking to freeing this memory caused a crash.

Comment by Rex Johnston [ 2023-11-16 ]

The test case in the commit is a little contrived to simulate the conditions caused by random innodb statistics collection. We force a Loose Scan join on a derived table (the root conditions causing the crash). In order to observe the crash, remove the actual fix in make_join_readinfo

m_key_part_info->store_length= m_key_part_info->length;

and recompile, then run mysql test run on main.subselect_sj

Although this patch can be applied to 10.6 onward, it is not needed there.

Comment by Sergei Petrunia [ 2023-11-17 ]

@Rex, I agree that it's a good idea to set store_length.. but there are other odd things happening when running the query...

The fix is in Create_tmp_table::finalize() in this branch:

      /*                           
        Create an extra field to hold NULL bits so that unique indexes on
        blobs can distinguish NULL from 0. This extra field is not needed
        when we do not use UNIQUE indexes for blobs.
      */
      if (null_pack_length[distinct] && (keyinfo->flags & HA_UNIQUE_HASH))
      {

But should this query really have unique indexes on blobs?

It turns out, yes.

Item_in_subselect::Item_in_subselect(THD *thd, Item * left_exp,
                                     st_select_lex *select_lex):

has this logic:

  /*
    If the IN subquery (xxx IN (SELECT ...) is a join without grouping,
    we don't need duplicates from the tables it is joining. These
    tables can be derived tables, like shown in the following
    example. In this case, it's useful to indicate that we don't need
    duplicates from them either.
 
    Example:
     col IN (SELECT ...   -- this is the select_lex
              FROM
                (SELECT ... FROM t1) AS t1, -- child1, first_inner_init().
                (SELECT ... FROM t2) AS t2, -- child2
              WHERE
                ...
            )
 
     We don't need duplicates from either child1 or child2.
     We only indicate this to child1 (select_lex->first_inner_unit()), as that
     catches most of practically important use cases.
 
     (The check for item==NULL is to make sure the subquery is a derived table
     and not any other kind of subquery like another IN (SELECT ...) or a scalar-
     context (SELECT 'foo'))
  */
 
  select_lex->master_unit()->distinct= 1;
  if (!select_lex->with_sum_func &&
      select_lex->first_inner_unit() &&
      select_lex->first_inner_unit()->item == NULL)

which does:

    select_lex->first_inner_unit()->distinct= 1;

introduced in 11.0

commit 5e5a8eda1641eda1d915a7eb5736e494d2179795
Author:	Monty <monty@mariadb.org>  Wed May  4 17:26:43 2022
 
Derived tables and union can now create distinct keys

In the query that the testcase is using:

SELECT id, a 
FROM t1 JOIN t2 ON id = id2 
WHERE 
  t1.id IN (SELECT dt.id 
            FROM  
              (SELECT id, avg(a), b
               FROM t1 JOIN t2 ON id = id2
              ) dt
           );

it sets select_lex_unit->distinct=1 for the (SELECT id, avg(a), b FROM... ). b is a TEXT column.

Comment by Sergei Petrunia [ 2023-11-17 ]

Ok, but does this just cause some CPU overhead or is this harmful?

The query plan uses LooseScan over the temp. table:

+------+-------------+------------+------+-------------------+------+---------+-----------+------+-------------------------------------------------+
| id   | select_type | table      | type | possible_keys     | key  | key_len | ref       | rows | Extra                                           |
+------+-------------+------------+------+-------------------+------+---------+-----------+------+-------------------------------------------------+
|    1 | PRIMARY     | t2         | ALL  | NULL              | NULL | NULL    | NULL      | 4    | Using where                                     |
|    1 | PRIMARY     | <derived3> | ref  | key2,distinct_key | key2 | 5       | j1.t2.id2 | 3    | LooseScan                                       |
|    1 | PRIMARY     | t1         | ref  | id                | id   | 5       | j1.t2.id2 | 8    |                                                 |
|    3 | DERIVED     | t2         | ALL  | NULL              | NULL | NULL    | NULL      | 4    |                                                 |
|    3 | DERIVED     | t1         | ALL  | id                | NULL | NULL    | NULL      | 8    | Using where; Using join buffer (flat, BNL join) |
+------+-------------+------------+------+-------------------+------+---------+-----------+------+-------------------------------------------------+

I see the temp. table is created with just one index. That index starts with this extra field.
Can LooseScan semi-join strategy (https://mariadb.com/kb/en/loosescan-strategy/) work when using such index? I think it expects the index to start with IN subquery's select list columns... Note that column b is not in the IN-subquery's select list...

Will derived_with_keys optimization work with such index?

Johnston any thoughts about this?

Comment by Rex Johnston [ 2023-12-05 ]

psergei, you are right. It is possible to construct the contents of this derived table with NULL and empty strings sequentially and it produces the wrong result (and an odd one at that, still figuring this out).

Is banning the loose scan an option here? It doesn't appear this normally happens.
[Edit] never mind, figured out a way forward.

Comment by Sergei Petrunia [ 2023-12-13 ]

Take-aways from yesterday's discussion:

maybe, it's all of the above...

  • member un-initialization as in your initial fix
  • TABLE::use_index() re-numbering indexes but LooseScan plan still using the old index number.
  • I still doubt this combination of flags, is it really workable:

    (gdb) p keyuse->table->file->index_flags(0, 0, 1) & HA_READ_ORDER
      $74 = 4
    (gdb) p keyuse->table->key_info[0].flags & HA_UNIQUE_HASH
      $75 = 262144
    

Need to fix #2 and check #3.

Comment by Rex Johnston [ 2024-01-30 ]

HA_UNIQUE_HASH appears to be just a flag specific to derived tables to indicate that this key is to be unique. I don't see any issue using it for a loose scan semi-join as long any new 'unique-making' fields are placed on the end of key, not the beginning, providing the correct grouping while retrieving the contents of the derived table.

Observed behaviour prior to my last commit while executing the following (note values (3,''), (3,NULL) in t2)

CREATE TABLE t1 (id int, a int, KEY (a), KEY (id)) engine=innodb STATS_AUTO_RECALC=0;
 
INSERT INTO t1 VALUES  (1, 1), (2, 1), (3, 1), (4, 1);
 
CREATE TABLE t2 (id2 int,  b TEXT) engine=innodb STATS_AUTO_RECALC=0;
INSERT INTO t2 VALUES (1,'F'),( 2,'D'),(3,''), (3,NULL),( 4,'F'),( 5,'Q');
 
SET SESSION OPTIMIZER_WHERE_COST=1.0;
SET GLOBAL OPTIMIZER_ROW_COPY_COST=1.0;
set optimizer_switch='firstmatch=off,materialization=off,loosescan=on';
 
SELECT id, a , b
FROM t1 JOIN t2 ON id = id2 
WHERE 
  t1.id IN (SELECT dt.id FROM (SELECT id, a, b  FROM t1 JOIN t2 ON id = id2 limit 1000)dt);

had the code in sql_subselect (where loosescan is implemented) retrieving key values prefixed with null indicator values

Key:
 
Name:distinct_key,length:23,usable parts:4,parts:4,(KEY_PART_INFO *):{dt.(null)=\370}(KEY_PART_INFO *):{dt.id=1}(KEY_PART_INFO *):{dt.a=1}(KEY_PART_INFO *):{dt.b=F}
        skip_over = true, read_record returns end of 'same group' (-1)
        -> evaluated
 
Name:distinct_key,length:23,usable parts:4,parts:4,(KEY_PART_INFO *):{dt.(null)=\370}(KEY_PART_INFO *):{dt.id=2}(KEY_PART_INFO *):{dt.a=1}(KEY_PART_INFO *):{dt.b=D}
        same as above
 
Name:distinct_key,length:23,usable parts:4,parts:4,(KEY_PART_INFO *):{dt.(null)=\374}(KEY_PART_INFO *):{dt.id=3}(KEY_PART_INFO *):{dt.a=1}(KEY_PART_INFO *):{dt.b=NULL}
        skip_over = true, read_record returns zero (there is more in this group)
        key_cmp (comparing \370 from last read to \374)
          -> next group
 
Name:distinct_key,length:23,usable parts:4,parts:4,(KEY_PART_INFO *):{dt.(null)=\370}(KEY_PART_INFO *):{dt.id=3}(KEY_PART_INFO *):{dt.a=1}(KEY_PART_INFO *):{dt.b=}
Name:distinct_key,length:23,usable parts:4,parts:4,(KEY_PART_INFO *):{dt.(null)=\374}(KEY_PART_INFO *):{dt.id=3}(KEY_PART_INFO *):{dt.a=1}(KEY_PART_INFO *):{dt.b=NULL}
Name:distinct_key,length:23,usable parts:4,parts:4,(KEY_PART_INFO *):{dt.(null)=\370}(KEY_PART_INFO *):{dt.id=3}(KEY_PART_INFO *):{dt.a=1}(KEY_PART_INFO *):{dt.b=}

... illustrating that loosescan expects the grouping key indicator (join_tab->loosescan_key_len = 1) to be the first part of the key.

output is

+------+------+------+
| id   | a    | b    |
+------+------+------+
|    1 |    1 | F    |
|    2 |    1 | D    |
|    3 |    1 |      |
|    3 |    1 |      |
|    3 |    1 | NULL |
|    3 |    1 | NULL |
|    4 |    1 | F    |
+------+------+------+

which is clearly incorrect.

Altering the location of the NULL/empty key part to the end of the key appears to fix this problem.

Generated at Thu Feb 08 10:26:30 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.