Uploaded image for project: 'MariaDB Server'
  1. MariaDB Server
  2. MDEV-32537

Set thread names for MariaDB Server threads

Details

    Description

      Multi-threaded Linux programs (like Firefox or MySQL since 8.0.27 or so) usually set a name for each thread. This name will appear in the output of top and other utilities and would help to answer questions like "What thread uses most of CPU" etc.

      Consider this example of MySQL 8.0.29:

      openxs@ao756:~/dbs/8.0$ for task in $(ls /proc/$(pidof mysqld)/task/); do name=$(cat /proc/$(pidof mysqld)/task/${task}/comm); echo "TASK: ${task} (${name})"; done
      TASK: 567049 (mysqld)
      TASK: 567079 (ib_io_ibuf)
      TASK: 567080 (ib_io_log)
      TASK: 567081 (ib_io_rd-1)
      TASK: 567082 (ib_io_rd-2)
      TASK: 567083 (ib_io_rd-3)
      TASK: 567084 (ib_io_rd-4)
      TASK: 567085 (ib_io_wr-1)
      TASK: 567086 (ib_io_wr-2)
      TASK: 567087 (ib_io_wr-3)
      TASK: 567088 (ib_io_wr-4)
      TASK: 567089 (ib_pg_flush_co)
      TASK: 567096 (ib_log_checkpt)
      TASK: 567097 (ib_log_fl_notif)
      TASK: 567098 (ib_log_flush)
      TASK: 567099 (ib_log_wr_notif)
      TASK: 567100 (ib_log_writer)
      TASK: 567114 (ib_srv_lock_to)
      TASK: 567115 (ib_srv_err_mon)
      TASK: 567116 (ib_srv_mon)
      TASK: 567117 (ib_buf_resize)
      TASK: 567118 (ib_src_main)
      TASK: 567119 (ib_dict_stats)
      TASK: 567120 (ib_fts_opt)
      TASK: 567122 (xpl_worker-1)
      TASK: 567123 (xpl_worker-2)
      TASK: 567124 (xpl_accept-1)
      TASK: 567893 (ib_buf_dump)
      TASK: 567894 (ib_clone_gtid)
      TASK: 567895 (ib_srv_purge)
      TASK: 567896 (ib_srv_wkr-1)
      TASK: 567897 (ib_srv_wkr-2)
      TASK: 567898 (ib_srv_wkr-3)
      TASK: 567899 (evt_sched)
      TASK: 567900 (sig_handler)
      TASK: 567902 (xpl_accept-3)
      TASK: 567903 (gtid_zip)
      

      For MariaDB (10.6 here for example) we get:

      openxs@ao756:~/dbs/maria10.6$ for task in $(ls /proc/$(pidof mariadbd)/task/); do name=$(cat /proc/$(pidof mariadbd)/task/${task}/comm); echo "TASK: ${task} (${name})"; done
      TASK: 568093 (mariadbd)
      TASK: 568095 (mariadbd)
      TASK: 568096 (mariadbd)
      TASK: 568097 (mariadbd)
      TASK: 568098 (mariadbd)
      TASK: 568099 (mariadbd)
      TASK: 568104 (mariadbd)
      TASK: 568105 (mariadbd)
      openxs@ao756:~/dbs/maria10.6$
      

      See also https://bugs.mysql.com/bug.php?id=70858

      It makes sense to use the same thread name as used in Performance Schema.

      Attachments

        Activity

          valerii Valerii Kravchuk created issue -

          Compared to MySQL, MariaDB has only few single purpose threads. There is timer thread, there is page cleaner thread, there is a main thread, that accepts connections, and nothing interesting otherwise. there can be a bunch of background threads, from a threadpool, that handle innodb background work, and a bunch of foreground threads, that handle user queries ( from thread pool or not). I'm wondering if you can gain much from such info

          wlad Vladislav Vaintroub added a comment - Compared to MySQL, MariaDB has only few single purpose threads. There is timer thread, there is page cleaner thread, there is a main thread, that accepts connections, and nothing interesting otherwise. there can be a bunch of background threads, from a threadpool, that handle innodb background work, and a bunch of foreground threads, that handle user queries ( from thread pool or not). I'm wondering if you can gain much from such info

          I'd really be happy to know if CPU is used by a specific foreground user thread, some of slave applier threads or some background InnoDB thread.

          valerii Valerii Kravchuk added a comment - I'd really be happy to know if CPU is used by a specific foreground user thread, some of slave applier threads or some background InnoDB thread.
          wlad Vladislav Vaintroub made changes -
          Field Original Value New Value
          Assignee Vladislav Vaintroub [ wlad ]
          wlad Vladislav Vaintroub made changes -
          Fix Version/s 11.4 [ 29301 ]
          wlad Vladislav Vaintroub made changes -
          Status Open [ 1 ] In Progress [ 3 ]
          wlad Vladislav Vaintroub made changes -
          Assignee Vladislav Vaintroub [ wlad ] Nikita Malyavin [ nikitamalyavin ]
          Status In Progress [ 3 ] In Review [ 10002 ]
          nikitamalyavin Nikita Malyavin added a comment - left a few notes in https://github.com/MariaDB/server/commit/7e53e7729bf58a0e806424d538ed086f44304c1b
          nikitamalyavin Nikita Malyavin made changes -
          Assignee Nikita Malyavin [ nikitamalyavin ] Vladislav Vaintroub [ wlad ]
          Status In Review [ 10002 ] Stalled [ 10000 ]

          May be, let's avoid multiplying entities without a necessity?
          There's performance schema, it already has names for all threads. For example

          static PSI_thread_info all_mysys_threads[]=
          {
            { &key_thread_timer, "statement_timer", PSI_FLAG_GLOBAL}
          };
          

          the thread is started as

            if (mysql_thread_create(key_thread_timer, &timer_thread, &thr_attr,
                                    timer_handler, NULL))
          

          so in mysql_thread_create() performance schema can use pthread_setname_np() or whatever for the return value of pthread_create(), setting the name for the new thread. And users will see the same name in ps and in select * from performance_schema.threads. Less confusion.

          serg Sergei Golubchik added a comment - May be, let's avoid multiplying entities without a necessity? There's performance schema, it already has names for all threads. For example static PSI_thread_info all_mysys_threads[]= { { &key_thread_timer, "statement_timer" , PSI_FLAG_GLOBAL} }; the thread is started as if (mysql_thread_create(key_thread_timer, &timer_thread, &thr_attr, timer_handler, NULL)) so in mysql_thread_create() performance schema can use pthread_setname_np() or whatever for the return value of pthread_create() , setting the name for the new thread. And users will see the same name in ps and in select * from performance_schema.threads . Less confusion.
          wlad Vladislav Vaintroub added a comment - - edited

          serg, this does not work.

          • You assume the name can be always set from outside. It can't, portably. On macOS, our supported platform, there is no thread handle or ID that you can use this pthread_setname_np() with.
          • It can make sense, to change thread name, sometimes. For example, if the connection id is 10, one might want, in debug version say, change name of the thread to conn_10. Makes it easier to find your connection
          • Not all threads are created with mysql_thread_create. There are some with std::thread.
            There are some that are created by the OS-controlled threadpool , e.g on Windows where OS
            takes over the thread lifecycle. I'd like to have a thread name nonetheless, to distinguish between connection thread pool and Innodb's background threads.

          Thread names are for debugging and profiling, and that's it.

          wlad Vladislav Vaintroub added a comment - - edited serg , this does not work. You assume the name can be always set from outside. It can't, portably. On macOS, our supported platform, there is no thread handle or ID that you can use this pthread_setname_np() with. It can make sense, to change thread name, sometimes. For example, if the connection id is 10, one might want, in debug version say, change name of the thread to conn_10. Makes it easier to find your connection Not all threads are created with mysql_thread_create. There are some with std::thread. There are some that are created by the OS-controlled threadpool , e.g on Windows where OS takes over the thread lifecycle. I'd like to have a thread name nonetheless, to distinguish between connection thread pool and Innodb's background threads. Thread names are for debugging and profiling, and that's it.

          I thought another option could be to set the name inside the thread, but have a DBUG_ASSERT that it matches the name in perfschema. Then it'll still be "one entity" from the user point of view, even if we'll specify the name in two places (or one can #define STATEMENT_TIMER_THREAD_NAME and use it everywhere).

          As for many threads with the same name, I thought it'd be useful to print them like, say, "statement_timer:1234" where "1234" is, ideally, the id in the performance_schema.threads.

          The point is still to show not some arbitrary info but something that correlates with other information sources. For example, one can use ps to find the thread that takes 100% CPU and then use performance_schema to find out more.

          serg Sergei Golubchik added a comment - I thought another option could be to set the name inside the thread, but have a DBUG_ASSERT that it matches the name in perfschema. Then it'll still be "one entity" from the user point of view, even if we'll specify the name in two places (or one can #define STATEMENT_TIMER_THREAD_NAME and use it everywhere). As for many threads with the same name, I thought it'd be useful to print them like, say, "statement_timer:1234" where "1234" is, ideally, the id in the performance_schema.threads . The point is still to show not some arbitrary info but something that correlates with other information sources. For example, one can use ps to find the thread that takes 100% CPU and then use performance_schema to find out more.
          wlad Vladislav Vaintroub added a comment - - edited

          serg I sort-of did this verification in the last patch. Not sure it is a good idea, and strictly speaking we can't have the very same entity from the user's point of view.

          Because, Linux does not accept long names (where long is > 15 characters), one can't have thread names like "thread/innodb/page_cleaner", you can't even have names like 'innodb/page_cleaner'. Only names like 'page_cleaner' are possible, due to Linux limitations.

          I thought a bit about many threads with the same name, I actually would prefer them to have the same name by default (unless maybe an option is set).
          The profiler I use does a nice aggregation based on thread names, so I can figure out how much CPU foreground threads are using, and how much background ones, I'd rather not lose it. Besides, already some work was done to show OS thread id and what not in processlist and performance_schema.threads, this can help find thread during debugging

          wlad Vladislav Vaintroub added a comment - - edited serg I sort-of did this verification in the last patch. Not sure it is a good idea, and strictly speaking we can't have the very same entity from the user's point of view. Because, Linux does not accept long names (where long is > 15 characters), one can't have thread names like "thread/innodb/page_cleaner", you can't even have names like 'innodb/page_cleaner'. Only names like 'page_cleaner' are possible, due to Linux limitations. I thought a bit about many threads with the same name, I actually would prefer them to have the same name by default (unless maybe an option is set). The profiler I use does a nice aggregation based on thread names, so I can figure out how much CPU foreground threads are using, and how much background ones, I'd rather not lose it. Besides, already some work was done to show OS thread id and what not in processlist and performance_schema.threads, this can help find thread during debugging
          ralf.gebhardt Ralf Gebhardt made changes -
          Fix Version/s 11.5 [ 29506 ]
          Fix Version/s 11.4 [ 29301 ]

          Thanks. That explains why firefox names threads like

          AudioIP~ver RPC
          Backgro~ #51598
          BgIOThr~ol #600
          CanvasRenderer
          Compositor
          Cookie
          DNS Res~r #1879
          DNS Res~r #1882
          DNS Res~r #1883
          

          Agree about not appending numeric ids, not even for your profiler sake, but simply because the length is such a scarce resource, we shouldn't waste it on something that can be stored elsewhere, such a thread id.

          Then what about — the same name as the last component of name in perfschema. Assert if such a last component is longer than 15 characters, it'll limit our imagination a bit, but I'm sure we'll manage.

          PERFORMANCE_SCHEMA.THREADS has columns NAME and OS_THREAD_ID, they should match the values from

          ps H -o tid,comm
          

          ("match" in the above sense. for tid it's "equal", for "comm" it's "last component of")

          serg Sergei Golubchik added a comment - Thanks. That explains why firefox names threads like AudioIP~ver RPC Backgro~ #51598 BgIOThr~ol #600 CanvasRenderer Compositor Cookie DNS Res~r #1879 DNS Res~r #1882 DNS Res~r #1883 Agree about not appending numeric ids, not even for your profiler sake, but simply because the length is such a scarce resource, we shouldn't waste it on something that can be stored elsewhere, such a thread id. Then what about — the same name as the last component of name in perfschema. Assert if such a last component is longer than 15 characters, it'll limit our imagination a bit, but I'm sure we'll manage. PERFORMANCE_SCHEMA.THREADS has columns NAME and OS_THREAD_ID , they should match the values from ps H -o tid,comm ("match" in the above sense. for tid it's "equal", for "comm" it's "last component of")
          wlad Vladislav Vaintroub made changes -
          Assignee Vladislav Vaintroub [ wlad ] Sergei Golubchik [ serg ]
          Status Stalled [ 10000 ] In Review [ 10002 ]
          serg Sergei Golubchik made changes -
          Fix Version/s 11.6 [ 29515 ]
          Fix Version/s 11.5 [ 29506 ]
          serg Sergei Golubchik made changes -
          Priority Major [ 3 ] Critical [ 2 ]

          serg I would be happy to see current connection id in the thread name, if that's a connection thread, and if the worker is currently active or sleeping. Would really help navigating while debugging. That is, not necessary to have it in the release mode.

          nikitamalyavin Nikita Malyavin added a comment - serg I would be happy to see current connection id in the thread name, if that's a connection thread, and if the worker is currently active or sleeping. Would really help navigating while debugging. That is, not necessary to have it in the release mode.
          serg Sergei Golubchik made changes -
          Assignee Sergei Golubchik [ serg ] Vladislav Vaintroub [ wlad ]
          Status In Review [ 10002 ] Stalled [ 10000 ]
          wlad Vladislav Vaintroub made changes -
          Status Stalled [ 10000 ] In Testing [ 10301 ]
          wlad Vladislav Vaintroub made changes -
          Assignee Vladislav Vaintroub [ wlad ] Lena Startseva [ JIRAUSER50478 ]

          lstartseva, I' giving that out to testing , alas I do not know what exactly to test, apart from starting debugger and looking up thread names in all-threads-stacktraces. Perhaps valerii can come up with a creative idea, it is his request after all

          wlad Vladislav Vaintroub added a comment - lstartseva , I' giving that out to testing , alas I do not know what exactly to test, apart from starting debugger and looking up thread names in all-threads-stacktraces. Perhaps valerii can come up with a creative idea, it is his request after all

          To check what thread names are set (if any) we can use something like that command in the initial description:

          for task in $(ls /proc/$(pidof mariadbd)/task/); do name=$(cat /proc/$(pidof mariadbdd)/task/${task}/comm); echo "TASK: ${task} (${name})"; done
          

          valerii Valerii Kravchuk added a comment - To check what thread names are set (if any) we can use something like that command in the initial description: for task in $(ls /proc/$(pidof mariadbd)/task/); do name=$(cat /proc/$(pidof mariadbdd)/task/${task}/comm); echo "TASK: ${task} (${name})"; done

          When I do that (also after fixing typo "mariadbdd"), I get

          TASK: 241287 (mariadbd)
          TASK: 241288 (statement_timer)
          TASK: 241289 (checkpoint_bg)
          TASK: 241290 (my_getevents)
          TASK: 241291 (page_cleaner)
          TASK: 241292 (ib_tpool_worker)
          TASK: 241294 (manager)
          TASK: 241296 (signal_handler)
          TASK: 241339 (one_connection)
          

          Is this good enough?

          wlad Vladislav Vaintroub added a comment - When I do that (also after fixing typo "mariadbdd"), I get TASK: 241287 (mariadbd) TASK: 241288 (statement_timer) TASK: 241289 (checkpoint_bg) TASK: 241290 (my_getevents) TASK: 241291 (page_cleaner) TASK: 241292 (ib_tpool_worker) TASK: 241294 (manager) TASK: 241296 (signal_handler) TASK: 241339 (one_connection) Is this good enough?

          For me the names are OK, but some of them are not that obvious:

          TASK: 241294 (manager)

          and I see comments that for these:

          TASK: 241339 (one_connection)

          connection_id would be also good to see in the name...

          valerii Valerii Kravchuk added a comment - For me the names are OK, but some of them are not that obvious: TASK: 241294 (manager) and I see comments that for these: TASK: 241339 (one_connection) connection_id would be also good to see in the name...
          wlad Vladislav Vaintroub added a comment - - edited

          You're not alone guessing about reason a "manager" exists, but it is its name in perfschema, so it should not be new for you. I also have no idea what it does and why it exists.

          one_connection is also how it is called in perfschema. Actually, everything is like in performance schema, it was requested so by serg.

          I'm not sure about connection_id in thread name, I would not like to have it like this. xperf on Windows aggregates by thread names, and I rather see aggregate than 4000 individual threads, whenever I run a benchmark. In a debugger you might want to see one thing, in profiler another thing.

          UPD: "manager" is a single-threaded tpool, which is used exclusively by replication (as-if we do not have enough threadpool implementations)

          wlad Vladislav Vaintroub added a comment - - edited You're not alone guessing about reason a "manager" exists, but it is its name in perfschema, so it should not be new for you. I also have no idea what it does and why it exists. one_connection is also how it is called in perfschema. Actually, everything is like in performance schema, it was requested so by serg . I'm not sure about connection_id in thread name, I would not like to have it like this. xperf on Windows aggregates by thread names, and I rather see aggregate than 4000 individual threads, whenever I run a benchmark. In a debugger you might want to see one thing, in profiler another thing. UPD: "manager" is a single-threaded tpool, which is used exclusively by replication (as-if we do not have enough threadpool implementations)
          serg Sergei Golubchik made changes -
          serg Sergei Golubchik made changes -
          Issue Type Task [ 3 ] New Feature [ 2 ]
          ralf.gebhardt Ralf Gebhardt made changes -
          Labels Preview_11.6

          Checked threads name with Valerii's script. Everything looks good.
          About "one_connection" - in MySQL the same threads have the same name "connection", but the name "one_connection" looks a little strange because I didn't see "two_connection" or something like that.

          Also checked threads name under Windows with Visual Studio. Some of them had correct names, some were generated by the system. But as far as I understand, this is a feature of the implementation for Windows, which creates some threads itself.By the way, in Windows the process name is still "mysql.exe". Maybe it's time to rename, wlad, serg?

          I think that all the listed notes are not related to the implementation of this task, so this task can be pushed.

          lstartseva Lena Startseva added a comment - Checked threads name with Valerii's script. Everything looks good. About "one_connection" - in MySQL the same threads have the same name "connection", but the name "one_connection" looks a little strange because I didn't see "two_connection" or something like that. Also checked threads name under Windows with Visual Studio. Some of them had correct names, some were generated by the system. But as far as I understand, this is a feature of the implementation for Windows, which creates some threads itself.By the way, in Windows the process name is still "mysql.exe". Maybe it's time to rename, wlad , serg ? I think that all the listed notes are not related to the implementation of this task, so this task can be pushed.
          lstartseva Lena Startseva made changes -
          Assignee Lena Startseva [ JIRAUSER50478 ] Vladislav Vaintroub [ wlad ]
          Status In Testing [ 10301 ] Stalled [ 10000 ]
          wlad Vladislav Vaintroub made changes -
          Attachment screenshot-1.png [ 73815 ]
          wlad Vladislav Vaintroub added a comment - - edited

          About Windows threads that do not have "correct" names - those are the threads that did not run our code yet, or maybe never will.
          When I just start the mysqld.exe, and set breakpoint in main(), there are already 2 threads created by default process threadpool, probably when loading shared libraries.

          There is some discussion about it in https://stackoverflow.com/questions/34822072/why-does-windows-10-start-extra-threads-in-my-program

          wlad Vladislav Vaintroub added a comment - - edited About Windows threads that do not have "correct" names - those are the threads that did not run our code yet, or maybe never will. When I just start the mysqld.exe, and set breakpoint in main(), there are already 2 threads created by default process threadpool, probably when loading shared libraries. There is some discussion about it in https://stackoverflow.com/questions/34822072/why-does-windows-10-start-extra-threads-in-my-program

          About "one_connection" as thread name - unlike in MySQL, thread names are synchronized with performance schema, so it is just the result of this.

          wlad Vladislav Vaintroub added a comment - About "one_connection" as thread name - unlike in MySQL, thread names are synchronized with performance schema, so it is just the result of this.
          nikitamalyavin Nikita Malyavin made changes -
          Attachment screenshot-2.png [ 73818 ]

          wlad serg maybe later we can add a build-time option to add more details into a thread name, what'd you say?

          Sometimes it's tricky to find a thread that executes your query. Okay in thread-per-connection mode it's fine, but in tpool you'll have to traverse all thread stacks to find one, if, say, you're just pausing the execution in gdb.

          I solve this problem with parallel stacks in clion, it looks like this:

          Yet I would definitely love to rely on such mouse-addicted interface less.

          nikitamalyavin Nikita Malyavin added a comment - wlad serg maybe later we can add a build-time option to add more details into a thread name, what'd you say? Sometimes it's tricky to find a thread that executes your query. Okay in thread-per-connection mode it's fine, but in tpool you'll have to traverse all thread stacks to find one, if, say, you're just pausing the execution in gdb. I solve this problem with parallel stacks in clion, it looks like this: Yet I would definitely love to rely on such mouse-addicted interface less.

          I solve this problem with Visual Studio, and others solve this problem with "SHOW PROCESSLIST", if they need an id. the os thread id is also there. Is thread-per-connection much better? How do you find your thread with 100 of threads?

          wlad Vladislav Vaintroub added a comment - I solve this problem with Visual Studio, and others solve this problem with "SHOW PROCESSLIST", if they need an id. the os thread id is also there. Is thread-per-connection much better? How do you find your thread with 100 of threads?
          wlad Vladislav Vaintroub made changes -
          issue.field.resolutiondate 2024-07-09 17:35:01.0 2024-07-09 17:35:01.187
          wlad Vladislav Vaintroub made changes -
          Component/s Server [ 13907 ]
          Fix Version/s 11.6.1 [ 29847 ]
          Fix Version/s 11.6 [ 29515 ]
          Resolution Fixed [ 1 ]
          Status Stalled [ 10000 ] Closed [ 6 ]

          @nikita, I would say, changing thread name twice per query (as in threadpool, to indicate thread ID, and to indicate "idle/waiting" state without THD) would be too expensive for optimized binary. For debug, it probably does not make any difference, so personally I would not care (but I did not care about thread names being perfschema-compatible in the first place, just did it on @serg's request)

          wlad Vladislav Vaintroub added a comment - @nikita, I would say, changing thread name twice per query (as in threadpool, to indicate thread ID, and to indicate "idle/waiting" state without THD) would be too expensive for optimized binary. For debug, it probably does not make any difference, so personally I would not care (but I did not care about thread names being perfschema-compatible in the first place, just did it on @serg's request)

          People

            wlad Vladislav Vaintroub
            valerii Valerii Kravchuk
            Votes:
            1 Vote for this issue
            Watchers:
            7 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Git Integration

                Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.