Bonus info: It was OK in 10.6 and only slightly broken in 10.11. So it became worse over time...

oli, can you share a little more "bonus info", e.g all thread stacktrace, like harmut did in the ticket you quote

Can you please give me the command to get the gdb stacktrace output? Then I will do...

root@oli-1:~# gdb -batch -ex "thread apply all bt" -p $(pidof mariadbd)
[New LWP 150007]
[New LWP 150008]
[New LWP 150009]
[New LWP 150010]
[New LWP 150011]
[New LWP 150012]
[New LWP 150166]
[New LWP 150167]
[New LWP 150168]
[New LWP 150169]
[New LWP 150175]
[New LWP 150176]
[New LWP 150177]
[New LWP 150190]
[New LWP 150205]
[New LWP 150618]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
0x00007f6a0271b1df in poll () from target:/lib/x86_64-linux-gnu/libc.so.6

Thread 17 (Thread 0x7f69a552e6c0 (LWP 150618) "mariadbd"):
#0 0x00007f6a026a4f16 in ?? () from target:/lib/x86_64-linux-gnu/libc.so.6
#1 0x00007f6a026a7baf in pthread_cond_clockwait () from target:/lib/x86_64-linux-gnu/libc.so.6
#2 0x000055d785d7d951 in tpool::thread_pool_generic::wait_for_tasks(std::unique_lock<std::mutex>&, tpool::worker_data*) ()
#3 0x000055d785d7e5c0 in tpool::thread_pool_generic::get_task(tpool::worker_data*, tpool::task**) ()
#4 0x000055d785d7e6fd in tpool::thread_pool_generic::worker_main(tpool::worker_data*) ()
#5 0x00007f6a024d44a3 in ?? () from target:/lib/x86_64-linux-gnu/libstdc++.so.6
#6 0x00007f6a026a81c4 in ?? () from target:/lib/x86_64-linux-gnu/libc.so.6
#7 0x00007f6a0272885c in ?? () from target:/lib/x86_64-linux-gnu/libc.so.6

Thread 16 (Thread 0x7f69e40966c0 (LWP 150205) "mariadbd"):
#0 0x00007f6a0271b1df in poll () from target:/lib/x86_64-linux-gnu/libc.so.6
#1 0x000055d785a621cb in vio_io_wait ()
#2 0x000055d785a62290 in vio_socket_io_wait ()
#3 0x000055d785a62333 in vio_read ()
#4 0x000055d785a25673 in ?? ()
#5 0x000055d785a26a70 in my_net_read_packet_reallen ()
#6 0x000055d785a26b8d in my_net_read_packet ()
#7 0x000055d785655ff1 in do_command(THD*, bool) ()
#8 0x000055d7857831a5 in do_handle_one_connection(CONNECT*, bool) ()
#9 0x000055d7857834fd in handle_one_connection ()
#10 0x000055d785b02c37 in ?? ()
#11 0x00007f6a026a81c4 in ?? () from target:/lib/x86_64-linux-gnu/libc.so.6
#12 0x00007f6a0272885c in ?? () from target:/lib/x86_64-linux-gnu/libc.so.6

Thread 15 (Thread 0x7f69f405e6c0 (LWP 150190) "mariadbd"):
#0 0x00007f6a0271b1df in poll () from target:/lib/x86_64-linux-gnu/libc.so.6
#1 0x000055d785a621cb in vio_io_wait ()
#2 0x000055d785a62290 in vio_socket_io_wait ()
#3 0x000055d785a62333 in vio_read ()
#4 0x000055d785a25673 in ?? ()
#5 0x000055d785a26a70 in my_net_read_packet_reallen ()
#6 0x000055d785a26b8d in my_net_read_packet ()
#7 0x000055d785655ff1 in do_command(THD*, bool) ()
#8 0x000055d7857831a5 in do_handle_one_connection(CONNECT*, bool) ()
#9 0x000055d7857834fd in handle_one_connection ()
#10 0x000055d785b02c37 in ?? ()
#11 0x00007f6a026a81c4 in ?? () from target:/lib/x86_64-linux-gnu/libc.so.6
#12 0x00007f6a0272885c in ?? () from target:/lib/x86_64-linux-gnu/libc.so.6

Thread 14 (Thread 0x7f69f40a96c0 (LWP 150177) "mariadbd"):
#0 0x00007f6a0265bc82 in sigtimedwait () from target:/lib/x86_64-linux-gnu/libc.so.6
#1 0x000055d785534c9b in signal_hand ()
#2 0x000055d785b02c37 in ?? ()
#3 0x00007f6a026a81c4 in ?? () from target:/lib/x86_64-linux-gnu/libc.so.6
#4 0x00007f6a0272885c in ?? () from target:/lib/x86_64-linux-gnu/libc.so.6

Thread 13 (Thread 0x7f69f40f46c0 (LWP 150176) "mariadbd"):
#0 0x00007f6a026a4f16 in ?? () from target:/lib/x86_64-linux-gnu/libc.so.6
#1 0x00007f6a026a75d8 in pthread_cond_wait () from target:/lib/x86_64-linux-gnu/libc.so.6
#2 0x000055d7854e5663 in psi_cond_wait ()
#3 0x000055d7854991c5 in ?? ()
#4 0x000055d785b02c37 in ?? ()
#5 0x00007f6a026a81c4 in ?? () from target:/lib/x86_64-linux-gnu/libc.so.6
#6 0x00007f6a0272885c in ?? () from target:/lib/x86_64-linux-gnu/libc.so.6

Thread 12 (Thread 0x7f69f413f6c0 (LWP 150175) "mariadbd"):
#0 0x00007f6a026a4f16 in ?? () from target:/lib/x86_64-linux-gnu/libc.so.6
#1 0x00007f6a026a75d8 in pthread_cond_wait () from target:/lib/x86_64-linux-gnu/libc.so.6
#2 0x000055d7854e5663 in psi_cond_wait ()
#3 0x000055d785483272 in ?? ()
#4 0x000055d785b02c37 in ?? ()
#5 0x00007f6a026a81c4 in ?? () from target:/lib/x86_64-linux-gnu/libc.so.6
#6 0x00007f6a0272885c in ?? () from target:/lib/x86_64-linux-gnu/libc.so.6

Thread 11 (Thread 0x7f69a9ffe6c0 (LWP 150169) "mariadbd"):
#0 0x00007f6a026a4f16 in ?? () from target:/lib/x86_64-linux-gnu/libc.so.6
#1 0x00007f6a026a75d8 in pthread_cond_wait () from target:/lib/x86_64-linux-gnu/libc.so.6
#2 0x000055d785cf579c in ?? ()
#3 0x00007f6a024d44a3 in ?? () from target:/lib/x86_64-linux-gnu/libstdc++.so.6
#4 0x00007f6a026a81c4 in ?? () from target:/lib/x86_64-linux-gnu/libc.so.6
#5 0x00007f6a0272885c in ?? () from target:/lib/x86_64-linux-gnu/libc.so.6

Thread 10 (Thread 0x7f69aa7ff6c0 (LWP 150168) "mariadbd"):
#0 0x00007f6a0271b1df in poll () from target:/lib/x86_64-linux-gnu/libc.so.6
#1 0x000055d785ce5729 in ?? ()
#2 0x00007f6a024d44a3 in ?? () from target:/lib/x86_64-linux-gnu/libstdc++.so.6
#3 0x00007f6a026a81c4 in ?? () from target:/lib/x86_64-linux-gnu/libc.so.6
#4 0x00007f6a0272885c in ?? () from target:/lib/x86_64-linux-gnu/libc.so.6

Thread 9 (Thread 0x7f69e54e26c0 (LWP 150167) "mariadbd"):
#0 0x00007f6a02f669f7 in ?? () from target:/lib/x86_64-linux-gnu/liburing.so.2
#1 0x00007f6a02f66c39 in __io_uring_get_cqe () from target:/lib/x86_64-linux-gnu/liburing.so.2
#2 0x000055d785d804e0 in ?? ()
#3 0x00007f6a024d44a3 in ?? () from target:/lib/x86_64-linux-gnu/libstdc++.so.6
#4 0x00007f6a026a81c4 in ?? () from target:/lib/x86_64-linux-gnu/libc.so.6
#5 0x00007f6a0272885c in ?? () from target:/lib/x86_64-linux-gnu/libc.so.6

Thread 8 (Thread 0x7f69e60e46c0 (LWP 150166) "mariadbd"):
#0 0x00007f6a026a4f16 in ?? () from target:/lib/x86_64-linux-gnu/libc.so.6
#1 0x00007f6a026a78bc in pthread_cond_timedwait () from target:/lib/x86_64-linux-gnu/libc.so.6
#2 0x000055d7854e56d9 in psi_cond_timedwait ()
#3 0x000055d78549d8ae in ?? ()
#4 0x000055d785a91a49 in ?? ()
#5 0x000055d785b02c37 in ?? ()
#6 0x00007f6a026a81c4 in ?? () from target:/lib/x86_64-linux-gnu/libc.so.6
#7 0x00007f6a0272885c in ?? () from target:/lib/x86_64-linux-gnu/libc.so.6

Thread 7 (Thread 0x7f69f5a2e6c0 (LWP 150012) "mariadbd"):
#0 0x00007f6a026a4f16 in ?? () from target:/lib/x86_64-linux-gnu/libc.so.6
#1 0x00007f6a026a75d8 in pthread_cond_wait () from target:/lib/x86_64-linux-gnu/libc.so.6
#2 0x00007f69f7bdb179 in gu_cond_wait_SYS (mutex=0x55d799c089d0, cond=0x55d799c08a10) at ./galerautils/src/gu_threads.h:251
#3 fifo_lock_get (q=0x55d799c08950) at ./galerautils/src/gu_fifo.c:262
#4 gu_fifo_get_head (q=0x55d799c08950, err=0x7f69f5a2d664) at ./galerautils/src/gu_fifo.c:330
#5 0x00007f69f7ab7003 in gcs_recv (conn=0x55d799be2260, action=action@entry=0x7f69f5a2d6e0) at ./gcs/src/gcs.cpp:2146
#6 0x00007f69f7a9b384 in galera::Gcs::recv (act=..., this=<optimized out>) at ./galera/src/galera_gcs.hpp:133
#7 galera::GcsActionSource::process (this=0x55d7999745c0, recv_ctx=0x7f69f5a2dca0, exit_loop=@0x7f69f5a2d8df: false) at ./galera/src/gcs_action_source.cpp:165
#8 0x00007f69f7a6fc41 in galera::ReplicatorSMM::async_recv (this=0x55d79995a790, recv_ctx=0x7f69f5a2dca0) at ./galera/src/replicator_smm.cpp:404
#9 0x00007f69f7a4ad8b in galera_recv (gh=<optimized out>, recv_ctx=<optimized out>) at ./galera/src/wsrep_provider.cpp:265
#10 0x000055d785eb31ce in wsrep::wsrep_provider_v26::run_applier(wsrep::high_priority_service*) ()
#11 0x000055d785b8451d in ?? ()
#12 0x000055d785b72a66 in start_wsrep_THD(void*) ()
#13 0x000055d785b02c37 in ?? ()
#14 0x00007f6a026a81c4 in ?? () from target:/lib/x86_64-linux-gnu/libc.so.6
#15 0x00007f6a0272885c in ?? () from target:/lib/x86_64-linux-gnu/libc.so.6

Thread 6 (Thread 0x7f69f5a796c0 (LWP 150011) "mariadbd"):
#0 0x00007f6a026a4f16 in ?? () from target:/lib/x86_64-linux-gnu/libc.so.6
#1 0x00007f6a026a75d8 in pthread_cond_wait () from target:/lib/x86_64-linux-gnu/libc.so.6
#2 0x000055d785b83569 in ?? ()
#3 0x000055d785b72a66 in start_wsrep_THD(void*) ()
#4 0x000055d785b02c37 in ?? ()
#5 0x00007f6a026a81c4 in ?? () from target:/lib/x86_64-linux-gnu/libc.so.6
#6 0x00007f6a0272885c in ?? () from target:/lib/x86_64-linux-gnu/libc.so.6

Thread 5 (Thread 0x7f69f627a6c0 (LWP 150010) "mariadbd"):
#0 0x00007f6a026a4f16 in ?? () from target:/lib/x86_64-linux-gnu/libc.so.6
#1 0x00007f6a026a75d8 in pthread_cond_wait () from target:/lib/x86_64-linux-gnu/libc.so.6
#2 0x00007f69f7ad08ac in gu_cond_wait_SYS (mutex=0x55d799c0cca8, cond=0x55d799c0cce8) at ./galerautils/src/gu_threads.h:251
#3 gu::Lock::wait (this=<synthetic pointer>, cond=...) at ./galerautils/src/gu_lock.hpp:46
#4 RecvBuf::front (timeout=..., this=0x55d799c0cca8) at ./gcs/src/gcs_gcomm.cpp:116
#5 gcomm_recv (backend=<optimized out>, msg=0x55d799be2760, timeout=9223372035999999999) at ./gcs/src/gcs_gcomm.cpp:616
#6 0x00007f69f7abdbcf in core_msg_recv (timeout=9223372035999999999, recv_msg=0x55d799be2760, backend=0x55d799be2790) at ./gcs/src/gcs_core.cpp:522
#7 gcs_core_recv (conn=0x55d799be2540, recv_act=recv_act@entry=0x7f69f6279ca0, timeout=9223372035999999999) at ./gcs/src/gcs_core.cpp:1264
#8 0x00007f69f7ab77eb in gcs_recv_thread (arg=0x55d799be2260) at ./gcs/src/gcs.cpp:1479
#9 0x00007f6a026a81c4 in ?? () from target:/lib/x86_64-linux-gnu/libc.so.6
#10 0x00007f6a0272885c in ?? () from target:/lib/x86_64-linux-gnu/libc.so.6

Thread 4 (Thread 0x7f69f6a7b6c0 (LWP 150009) "mariadbd"):
#0 0x00007f6a02727ea6 in epoll_wait () from target:/lib/x86_64-linux-gnu/libc.so.6
#1 0x00007f69f7bb57e4 in asio::detail::epoll_reactor::run (this=0x55d799c12e90, usec=<optimized out>, ops=...) at /usr/include/asio/detail/impl/epoll_reactor.ipp:500
#2 0x00007f69f7bb27e8 in asio::detail::scheduler::do_run_one (this=this@entry=0x55d799c128f0, lock=..., this_thread=..., ec=...) at /usr/include/asio/detail/impl/scheduler.ipp:475
#3 0x00007f69f7bac0f1 in asio::detail::scheduler::run (this=0x55d799c128f0, ec=...) at /usr/include/asio/detail/impl/scheduler.ipp:209
#4 0x00007f69f7bac2aa in asio::io_context::run (this=<optimized out>) at /usr/include/asio/impl/io_context.ipp:62
#5 gu::AsioIoService::run (this=<optimized out>) at ./galerautils/src/gu_asio.cpp:764
#6 0x00007f69f7ad4320 in GCommConn::run (this=0x55d799c0cab0) at ./gcs/src/gcs_gcomm.cpp:441
#7 0x00007f69f7ad4866 in run_fn (arg=<optimized out>) at ./gcs/src/gcs_gcomm.cpp:309
#8 0x00007f6a026a81c4 in ?? () from target:/lib/x86_64-linux-gnu/libc.so.6
#9 0x00007f6a0272885c in ?? () from target:/lib/x86_64-linux-gnu/libc.so.6

Thread 3 (Thread 0x7f69f79ff6c0 (LWP 150008) "mariadbd"):
#0 0x00007f6a026a4f16 in ?? () from target:/lib/x86_64-linux-gnu/libc.so.6
#1 0x00007f6a026a75d8 in pthread_cond_wait () from target:/lib/x86_64-linux-gnu/libc.so.6
#2 0x00007f69f7a95e8c in gu_cond_wait_SYS (mutex=0x55d79995b0b0, cond=0x55d79995b0f0) at ./galerautils/src/gu_threads.h:251
#3 gu::Lock::wait (this=<synthetic pointer>, cond=...) at ./galerautils/src/gu_lock.hpp:46
#4 galera::ServiceThd::thd_func (arg=0x55d79995b090) at ./galera/src/galera_service_thd.cpp:31
#5 0x00007f6a026a81c4 in ?? () from target:/lib/x86_64-linux-gnu/libc.so.6
#6 0x00007f6a0272885c in ?? () from target:/lib/x86_64-linux-gnu/libc.so.6

Thread 2 (Thread 0x7f69f7dba6c0 (LWP 150007) "mariadbd"):
#0 0x00007f6a026a4f16 in ?? () from target:/lib/x86_64-linux-gnu/libc.so.6
#1 0x00007f6a026a78bc in pthread_cond_timedwait () from target:/lib/x86_64-linux-gnu/libc.so.6
#2 0x000055d7854e56d9 in psi_cond_timedwait ()
#3 0x000055d7854e58b5 in ?? ()
#4 0x000055d785b02c37 in ?? ()
#5 0x00007f6a026a81c4 in ?? () from target:/lib/x86_64-linux-gnu/libc.so.6
#6 0x00007f6a0272885c in ?? () from target:/lib/x86_64-linux-gnu/libc.so.6

Thread 1 (Thread 0x7f6a02c87840 (LWP 150006) "mariadbd"):
#0 0x00007f6a0271b1df in poll () from target:/lib/x86_64-linux-gnu/libc.so.6
#1 0x000055d78553d517 in handle_connections_sockets() ()
#2 0x000055d78553e915 in mysqld_main(int, char**) ()
#3 0x00007f6a0264624a in ?? () from target:/lib/x86_64-linux-gnu/libc.so.6
#4 0x00007f6a02646305 in __libc_start_main () from target:/lib/x86_64-linux-gnu/libc.so.6
#5 0x000055d785532431 in _start ()
[Inferior 1 (process 150006) detached]

I hope this helps

We have the machine still until tomorrow Friday 16:00 CET. Just in case you want something more...

Even if stack trace is less than ideal, it seems now to be a Galera thing, so assigning to sysprg

oli, there is no promise or obligation that all process threads are always instrumented, there are 3rd party components that create threads, that have no slightest idea of what is PS.

Hi Vlad, OK got it.

But some of theses 3rd party components are registering in the PS correctly:

thread/sql/wsrep_rollbacker_thread FOREGROUND 150011
thread/sql/wsrep_applier_thread FOREGROUND 150012

Should that not be somehow a standard if you create a thread that one should also register in PS???

Just for curiosity, what has happened to all the other BACKGROUND threads? They disappeared somehow! Or are they only started on demand nowadays???

thread/innodb/io_ibuf_thread
thread/innodb/io_read_thread
thread/innodb/io_read_thread
thread/innodb/io_read_thread
thread/innodb/io_read_thread
thread/innodb/io_write_thread
thread/innodb/io_write_thread
thread/innodb/io_write_thread
thread/innodb/io_write_thread
thread/innodb/page_flush_coordinator_thread
thread/innodb/log_checkpointer_thread
thread/innodb/log_flush_notifier_thread
thread/innodb/log_flusher_thread
thread/innodb/log_write_notifier_thread
thread/innodb/log_writer_thread
thread/innodb/log_files_governor_thread
thread/innodb/srv_lock_timeout_thread
thread/innodb/srv_error_monitor_thread
thread/innodb/srv_monitor_thread
thread/innodb/buf_resize_thread
thread/innodb/srv_master_thread
thread/innodb/dict_stats_thread
thread/innodb/fts_optimize_thread
thread/innodb/buf_dump_thread
thread/innodb/clone_gtid_thread
thread/innodb/srv_purge_thread

There are no innodb threads anymore that only do 1 thing and wait 99.9% of the time. The only dedicated thread that remains if the page cleaner, because it was easier. Anything else is served by internal threadpool inside the Innodb, since 10.5 . Threads are created when needed, and go away when idle. There are mechanisms to cap number of specific "groups" those handling io completion, or purge workers.

Some of 3rd party instrument their threads, with performance schema, like this galera (which I think is part of the server). But some don't , like another part of Galera, which I think is not part of the server.

I realize that statement that you can't expect all threads to be instrumented, is overly generic. But here are couple of illustrations

1. If you look into rocksdb - the library, they also have their threadpool for background work, and it was written without perfschema instrumentation in mind
2. Sometimes you'll see threads internal to Linux uring, and those won't be the threads we create, but they are still part of mariadbd process.
3. If you start any process on Windows, including mariadbd.exe, there are already 4 threads in it, only one of them executing main function, and 3 others idle.

This is just to illustrate that server does not have control over all possible threads, in some cases number of instrumented threads is equal to number of all threads, in other cases it might not, and without looking at the stacktraces, it is impossible to know whether it is a bug or not.

Thanks Vlad for this information!