[MDEV-14984] regression in connect performance Created: 2018-01-18  Updated: 2019-05-08  Resolved: 2019-05-03

Status: Closed
Project: MariaDB Server
Component/s: Server
Affects Version/s: 10.3.1
Fix Version/s: 10.3.15, 10.4.5

Type: Bug Priority: Critical
Reporter: Axel Schwenke Assignee: Sergey Vojtovich
Resolution: Fixed Votes: 0
Labels: None

Attachments: Zip Archive MDEV-14984.zip    
Sprint: 10.1.31

 Description   

This was found by the regression test suite. While we had a nice improvement in connect performance in MariaDB 10.2, a lot of that was lost between 10.3.0 and 10.3.1.

#Workload: connect; select 1; disconnect;
#numbers: connects per second (bigger=better)
#
#thread count           1       8       16      32      64      128     256
...
mariadb-10.2.10         9740.9  70063   115054  132711  131760  128702  123058
mariadb-10.2.11         9198.6  70738   114843  132299  131002  128688  122573
mariadb-10.3.0          8733.6  69989   112238  121092  120087  119833  114413
mariadb-10.3.1          8479.1  59291   80954   81046   80629   80344   79407
mariadb-10.3.2          8439.2  60418   81373   81412   80924   80519   80056
mariadb-10.3.3          8953.7  43717   80714   80412   78967   78009   75566                                                           
mariadb-10.3.4          8939.6  48065   80113   79996   78368   78175   74769



 Comments   
Comment by Sergei Golubchik [ 2018-01-23 ]

axel, could you please compare CPU load between mariadb-10.3.0 and mariadb-10.3.1 in your benchmarks?

Comment by Axel Schwenke [ 2018-01-25 ]

10.3.1 is actually using less cpu:

10.3.0: 30% user, 32% system, 38% idle
10.3.1: 24% user, 23% system, 53% idle

perf shows high mutex activity for 10.3.1:

Overhead  Shared Object        Symbol
   7.66%  libpthread-2.19.so   [.] pthread_mutex_lock
   3.05%  [kernel]             [k] entry_SYSCALL_64_fastpath
   2.51%  [kernel]             [k] entry_SYSCALL_64
   2.14%  libc-2.19.so         [.] __clock_gettime
   2.03%  [kernel]             [k] unix_sock_destructor
   1.78%  libpthread-2.19.so   [.] pthread_mutex_unlock
   1.74%  [kernel]             [k] native_queued_spin_lock_slowpath
   1.72%  [kernel]             [k] _raw_spin_lock_irqsave
   1.59%  [kernel]             [k] mutex_lock
   1.56%  libc-2.19.so         [.] _int_malloc
   1.56%  mysqld               [.] _Z16dispatch_command19enum_server_commandP3THDPcjbb

Performance schema indicates that waiting for LOCK_plugin takes about 25 times as long in 10.3.1 compared to 10.3.0:

+---------------------------------------------------+------------+----------------+
| event_name                                        | count_star | sum_timer_wait |
+---------------------------------------------------+------------+----------------+
| wait/synch/mutex/sql/LOCK_plugin                  |    3002661 | 20501471746110 |
| wait/synch/mutex/sql/LOCK_thread_cache            |    1999979 | 15839463650517 |
| wait/synch/mutex/sql/THD::LOCK_thd_data           |    9000513 |   762731016222 |
| wait/synch/mutex/sql/LOCK_global_system_variables |        753 |   560556884121 |
| wait/synch/mutex/sql/LOCK_thread_count            |    2000713 |   420640407081 |

in comparison 10.3.0:

+---------------------------------------------------+------------+----------------+
| event_name                                        | count_star | sum_timer_wait |
+---------------------------------------------------+------------+----------------+
| wait/synch/mutex/sql/LOCK_thread_cache            |    2000723 | 13625498619297 |
| wait/synch/mutex/sql/LOCK_plugin                  |    3001387 |   810259198791 |
| wait/synch/mutex/sql/THD::LOCK_thd_data           |    9003299 |   704091160587 |
| wait/synch/mutex/sql/LOCK_thread_count            |    2000775 |   390192030768 |
| wait/synch/rwlock/sql/LOCK_system_variables_hash  |    2000898 |   333832603434 |

Mutex stats are for 1 mio loops of the "connect; SELECT 1; disconnect" workload. LOCK_plugin is touched 3 times for each iteration.

Full results attached.

Comment by Axel Schwenke [ 2018-01-25 ]

SHOW PLUGINS list one additional plugin (53 vs. 52) in 10.3.1. And this is "user_variables".

Comment by Sergey Vojtovich [ 2018-02-07 ]

Number of mutexes acquisition per iteration in connect benchmark is horrible:

#1  0x00005589769f9e73 in inline_mysql_mutex_lock (that=0x558977a441a0 <LOCK_plugin>) at /home/svoj/devel/maria/mariadb/include/mysql/psi/mysql_thread.h:712
#2  plugin_thdvar_cleanup (thd=thd@entry=0x7eee540009a8) at /home/svoj/devel/maria/mariadb/sql/sql_plugin.cc:3248
#3  0x0000558976abc498 in end_connection (thd=0x7eee540009a8) at /home/svoj/devel/maria/mariadb/sql/sql_connect.cc:1191
#4  0x0000558976abcdbc in do_handle_one_connection (connect=<optimized out>) at /home/svoj/devel/maria/mariadb/sql/sql_connect.cc:1405
#5  0x0000558976abce69 in handle_one_connection (arg=<optimized out>) at /home/svoj/devel/maria/mariadb/sql/sql_connect.cc:1308
#6  0x00007efc8bc636ba in start_thread (arg=0x7eeee00e5700) at pthread_create.c:333
#7  0x00007efc8b30e41d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109
 
 
#1  0x00005589770b3a61 in rw_pr_rdlock (rwlock=rwlock@entry=0x558977a3e860 <LOCK_system_variables_hash>) at /home/svoj/devel/maria/mariadb/mysys/thr_rwlock.c:377
#2  0x00005589769f9f32 in inline_mysql_prlock_rdlock (that=<optimized out>) at /home/svoj/devel/maria/mariadb/include/mysql/psi/mysql_thread.h:928
#3  cleanup_variables (vars=0x7eee54001018) at /home/svoj/devel/maria/mariadb/sql/sql_plugin.cc:3210
#4  plugin_thdvar_cleanup (thd=thd@entry=0x7eee540009a8) at /home/svoj/devel/maria/mariadb/sql/sql_plugin.cc:3251
#5  0x0000558976abc498 in end_connection (thd=0x7eee540009a8) at /home/svoj/devel/maria/mariadb/sql/sql_connect.cc:1191
#6  0x0000558976abcdbc in do_handle_one_connection (connect=<optimized out>) at /home/svoj/devel/maria/mariadb/sql/sql_connect.cc:1405
#7  0x0000558976abce69 in handle_one_connection (arg=<optimized out>) at /home/svoj/devel/maria/mariadb/sql/sql_connect.cc:1308
#8  0x00007efc8bc636ba in start_thread (arg=0x7eeee00e5700) at pthread_create.c:333
#9  0x00007efc8b30e41d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109
 
 
#1  0x00005589770b3b05 in rw_pr_unlock (rwlock=rwlock@entry=0x558977a3e860 <LOCK_system_variables_hash>) at /home/svoj/devel/maria/mariadb/mysys/thr_rwlock.c:456
#2  0x00005589769f9f86 in inline_mysql_prlock_unlock (that=<optimized out>) at /home/svoj/devel/maria/mariadb/include/mysql/psi/mysql_thread.h:1097
#3  cleanup_variables (vars=0x7eee54001018) at /home/svoj/devel/maria/mariadb/sql/sql_plugin.cc:3229
#4  plugin_thdvar_cleanup (thd=thd@entry=0x7eee540009a8) at /home/svoj/devel/maria/mariadb/sql/sql_plugin.cc:3251
#5  0x0000558976abc498 in end_connection (thd=0x7eee540009a8) at /home/svoj/devel/maria/mariadb/sql/sql_connect.cc:1191
#6  0x0000558976abcdbc in do_handle_one_connection (connect=<optimized out>) at /home/svoj/devel/maria/mariadb/sql/sql_connect.cc:1405
#7  0x0000558976abce69 in handle_one_connection (arg=<optimized out>) at /home/svoj/devel/maria/mariadb/sql/sql_connect.cc:1308
#8  0x00007efc8bc636ba in start_thread (arg=0x7eeee00e5700) at pthread_create.c:333
#9  0x00007efc8b30e41d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109
 
 
#1  0x000055897695433e in inline_mysql_mutex_lock (that=0x558977a3eea0 <LOCK_status>) at /home/svoj/devel/maria/mariadb/include/mysql/psi/mysql_thread.h:712
#2  THD::add_status_to_global (this=0x7eee540009a8) at /home/svoj/devel/maria/mariadb/sql/sql_class.h:4326
#3  unlink_thd (thd=thd@entry=0x7eee540009a8) at /home/svoj/devel/maria/mariadb/sql/mysqld.cc:2932
#4  0x0000558976954409 in one_thread_per_connection_end (thd=0x7eee540009a8, put_in_cache=<optimized out>) at /home/svoj/devel/maria/mariadb/sql/mysqld.cc:3079
#5  0x0000558976abcd56 in do_handle_one_connection (connect=<optimized out>) at /home/svoj/devel/maria/mariadb/sql/sql_connect.cc:1421
#6  0x0000558976abce69 in handle_one_connection (arg=<optimized out>) at /home/svoj/devel/maria/mariadb/sql/sql_connect.cc:1308
#7  0x00007efc8bc636ba in start_thread (arg=0x7eeee00e5700) at pthread_create.c:333
#8  0x00007efc8b30e41d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109
 
 
#1  0x000055897695437a in inline_mysql_mutex_lock (that=0x558977a3ef60 <LOCK_thread_count>) at /home/svoj/devel/maria/mariadb/include/mysql/psi/mysql_thread.h:712
#2  unlink_not_visible_thd (thd=0x7eee540009a8) at /home/svoj/devel/maria/mariadb/sql/sql_class.h:4686
#3  unlink_thd (thd=thd@entry=0x7eee540009a8) at /home/svoj/devel/maria/mariadb/sql/mysqld.cc:2933
#4  0x0000558976954409 in one_thread_per_connection_end (thd=0x7eee540009a8, put_in_cache=<optimized out>) at /home/svoj/devel/maria/mariadb/sql/mysqld.cc:3079
#5  0x0000558976abcd56 in do_handle_one_connection (connect=<optimized out>) at /home/svoj/devel/maria/mariadb/sql/sql_connect.cc:1421
#6  0x0000558976abce69 in handle_one_connection (arg=<optimized out>) at /home/svoj/devel/maria/mariadb/sql/sql_connect.cc:1308
#7  0x00007efc8bc636ba in start_thread (arg=0x7eeee00e5700) at pthread_create.c:333
#8  0x00007efc8b30e41d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109
 
 
#1  0x00005589769542a9 in inline_mysql_mutex_lock (that=0x558977a3ebe0 <LOCK_connection_count>) at /home/svoj/devel/maria/mariadb/include/mysql/psi/mysql_thread.h:712
#2  dec_connection_count (scheduler=0x558977a41840 <thread_scheduler_struct>) at /home/svoj/devel/maria/mariadb/sql/mysqld.cc:2888
#3  0x00005589769543c8 in unlink_thd (thd=thd@entry=0x7eee540009a8) at /home/svoj/devel/maria/mariadb/sql/mysqld.cc:2940
#4  0x0000558976954409 in one_thread_per_connection_end (thd=0x7eee540009a8, put_in_cache=<optimized out>) at /home/svoj/devel/maria/mariadb/sql/mysqld.cc:3079
#5  0x0000558976abcd56 in do_handle_one_connection (connect=<optimized out>) at /home/svoj/devel/maria/mariadb/sql/sql_connect.cc:1421
#6  0x0000558976abce69 in handle_one_connection (arg=<optimized out>) at /home/svoj/devel/maria/mariadb/sql/sql_connect.cc:1308
#7  0x00007efc8bc636ba in start_thread (arg=0x7eeee00e5700) at pthread_create.c:333
#8  0x00007efc8b30e41d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109
 
 
#1  0x00005589769f9c92 in inline_mysql_mutex_lock (that=0x558977a441a0 <LOCK_plugin>) at /home/svoj/devel/maria/mariadb/include/mysql/psi/mysql_thread.h:712
#2  plugin_foreach_with_mask (thd=thd@entry=0x7eee540009a8, func=func@entry=0x558976bb0370 <closecon_handlerton(THD*, plugin_ref, void*)>, type=type@entry=1, state_mask=state_mask@entry=8,
    arg=arg@entry=0x0) at /home/svoj/devel/maria/mariadb/sql/sql_plugin.cc:2397
#3  0x0000558976bb237e in ha_close_connection (thd=thd@entry=0x7eee540009a8) at /home/svoj/devel/maria/mariadb/sql/handler.cc:801
#4  0x00005589769b6489 in THD::free_connection (this=this@entry=0x7eee540009a8) at /home/svoj/devel/maria/mariadb/sql/sql_class.cc:1635
#5  0x00005589769543d4 in unlink_thd (thd=thd@entry=0x7eee540009a8) at /home/svoj/devel/maria/mariadb/sql/mysqld.cc:2942
#6  0x0000558976954409 in one_thread_per_connection_end (thd=0x7eee540009a8, put_in_cache=<optimized out>) at /home/svoj/devel/maria/mariadb/sql/mysqld.cc:3079
#7  0x0000558976abcd56 in do_handle_one_connection (connect=<optimized out>) at /home/svoj/devel/maria/mariadb/sql/sql_connect.cc:1421
#8  0x0000558976abce69 in handle_one_connection (arg=<optimized out>) at /home/svoj/devel/maria/mariadb/sql/sql_connect.cc:1308
#9  0x00007efc8bc636ba in start_thread (arg=0x7eeee00e5700) at pthread_create.c:333
#10 0x00007efc8b30e41d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109
 
 
#1  0x00005589769f9b4e in inline_mysql_mutex_lock (that=0x558977a441a0 <LOCK_plugin>) at /home/svoj/devel/maria/mariadb/include/mysql/psi/mysql_thread.h:712
#2  plugin_unlock_list (thd=<optimized out>, list=0x7eeee00e4c00, count=9) at /home/svoj/devel/maria/mariadb/sql/sql_plugin.cc:1414
#3  0x00005589769f9d98 in plugin_foreach_with_mask (thd=thd@entry=0x7eee540009a8, func=func@entry=0x558976bb0370 <closecon_handlerton(THD*, plugin_ref, void*)>, type=type@entry=1,
    state_mask=state_mask@entry=8, arg=arg@entry=0x0) at /home/svoj/devel/maria/mariadb/sql/sql_plugin.cc:2434
#4  0x0000558976bb237e in ha_close_connection (thd=thd@entry=0x7eee540009a8) at /home/svoj/devel/maria/mariadb/sql/handler.cc:801
#5  0x00005589769b6489 in THD::free_connection (this=this@entry=0x7eee540009a8) at /home/svoj/devel/maria/mariadb/sql/sql_class.cc:1635
#6  0x00005589769543d4 in unlink_thd (thd=thd@entry=0x7eee540009a8) at /home/svoj/devel/maria/mariadb/sql/mysqld.cc:2942
#7  0x0000558976954409 in one_thread_per_connection_end (thd=0x7eee540009a8, put_in_cache=<optimized out>) at /home/svoj/devel/maria/mariadb/sql/mysqld.cc:3079
#8  0x0000558976abcd56 in do_handle_one_connection (connect=<optimized out>) at /home/svoj/devel/maria/mariadb/sql/sql_connect.cc:1421
#9  0x0000558976abce69 in handle_one_connection (arg=<optimized out>) at /home/svoj/devel/maria/mariadb/sql/sql_connect.cc:1308
#10 0x00007efc8bc636ba in start_thread (arg=0x7eeee00e5700) at pthread_create.c:333
#11 0x00007efc8b30e41d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109
 
 
#1  0x00005589769f9e73 in inline_mysql_mutex_lock (that=0x558977a441a0 <LOCK_plugin>) at /home/svoj/devel/maria/mariadb/include/mysql/psi/mysql_thread.h:712
#2  plugin_thdvar_cleanup (thd=thd@entry=0x7eee540009a8) at /home/svoj/devel/maria/mariadb/sql/sql_plugin.cc:3248
#3  0x00005589769b6491 in THD::free_connection (this=this@entry=0x7eee540009a8) at /home/svoj/devel/maria/mariadb/sql/sql_class.cc:1636
#4  0x00005589769543d4 in unlink_thd (thd=thd@entry=0x7eee540009a8) at /home/svoj/devel/maria/mariadb/sql/mysqld.cc:2942
#5  0x0000558976954409 in one_thread_per_connection_end (thd=0x7eee540009a8, put_in_cache=<optimized out>) at /home/svoj/devel/maria/mariadb/sql/mysqld.cc:3079
#6  0x0000558976abcd56 in do_handle_one_connection (connect=<optimized out>) at /home/svoj/devel/maria/mariadb/sql/sql_connect.cc:1421
#7  0x0000558976abce69 in handle_one_connection (arg=<optimized out>) at /home/svoj/devel/maria/mariadb/sql/sql_connect.cc:1308
#8  0x00007efc8bc636ba in start_thread (arg=0x7eeee00e5700) at pthread_create.c:333
#9  0x00007efc8b30e41d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109
 
 
#1  0x00005589770b3a61 in rw_pr_rdlock (rwlock=rwlock@entry=0x558977a3e860 <LOCK_system_variables_hash>) at /home/svoj/devel/maria/mariadb/mysys/thr_rwlock.c:377
#2  0x00005589769f9f32 in inline_mysql_prlock_rdlock (that=<optimized out>) at /home/svoj/devel/maria/mariadb/include/mysql/psi/mysql_thread.h:928
#3  cleanup_variables (vars=0x7eee54001018) at /home/svoj/devel/maria/mariadb/sql/sql_plugin.cc:3210
#4  plugin_thdvar_cleanup (thd=thd@entry=0x7eee540009a8) at /home/svoj/devel/maria/mariadb/sql/sql_plugin.cc:3251
#5  0x00005589769b6491 in THD::free_connection (this=this@entry=0x7eee540009a8) at /home/svoj/devel/maria/mariadb/sql/sql_class.cc:1636
#6  0x00005589769543d4 in unlink_thd (thd=thd@entry=0x7eee540009a8) at /home/svoj/devel/maria/mariadb/sql/mysqld.cc:2942
#7  0x0000558976954409 in one_thread_per_connection_end (thd=0x7eee540009a8, put_in_cache=<optimized out>) at /home/svoj/devel/maria/mariadb/sql/mysqld.cc:3079
#8  0x0000558976abcd56 in do_handle_one_connection (connect=<optimized out>) at /home/svoj/devel/maria/mariadb/sql/sql_connect.cc:1421
#9  0x0000558976abce69 in handle_one_connection (arg=<optimized out>) at /home/svoj/devel/maria/mariadb/sql/sql_connect.cc:1308
#10 0x00007efc8bc636ba in start_thread (arg=0x7eeee00e5700) at pthread_create.c:333
#11 0x00007efc8b30e41d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109
 
 
#1  0x00005589770b3b05 in rw_pr_unlock (rwlock=rwlock@entry=0x558977a3e860 <LOCK_system_variables_hash>) at /home/svoj/devel/maria/mariadb/mysys/thr_rwlock.c:456
#2  0x00005589769f9f86 in inline_mysql_prlock_unlock (that=<optimized out>) at /home/svoj/devel/maria/mariadb/include/mysql/psi/mysql_thread.h:1097
#3  cleanup_variables (vars=0x7eee54001018) at /home/svoj/devel/maria/mariadb/sql/sql_plugin.cc:3229
#4  plugin_thdvar_cleanup (thd=thd@entry=0x7eee540009a8) at /home/svoj/devel/maria/mariadb/sql/sql_plugin.cc:3251
#5  0x00005589769b6491 in THD::free_connection (this=this@entry=0x7eee540009a8) at /home/svoj/devel/maria/mariadb/sql/sql_class.cc:1636
#6  0x00005589769543d4 in unlink_thd (thd=thd@entry=0x7eee540009a8) at /home/svoj/devel/maria/mariadb/sql/mysqld.cc:2942
#7  0x0000558976954409 in one_thread_per_connection_end (thd=0x7eee540009a8, put_in_cache=<optimized out>) at /home/svoj/devel/maria/mariadb/sql/mysqld.cc:3079
#8  0x0000558976abcd56 in do_handle_one_connection (connect=<optimized out>) at /home/svoj/devel/maria/mariadb/sql/sql_connect.cc:1421
#9  0x0000558976abce69 in handle_one_connection (arg=<optimized out>) at /home/svoj/devel/maria/mariadb/sql/sql_connect.cc:1308
#10 0x00007efc8bc636ba in start_thread (arg=0x7eeee00e5700) at pthread_create.c:333
#11 0x00007efc8b30e41d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109
 
 
#1  0x00005589769b2879 in inline_mysql_mutex_lock (that=<optimized out>) at /home/svoj/devel/maria/mariadb/include/mysql/psi/mysql_thread.h:712
#2  Statement_map::reset (this=this@entry=0x7eee540023e0) at /home/svoj/devel/maria/mariadb/sql/sql_class.cc:4044
#3  0x00005589769b64b1 in THD::free_connection (this=this@entry=0x7eee540009a8) at /home/svoj/devel/maria/mariadb/sql/sql_class.cc:1640
#4  0x00005589769543d4 in unlink_thd (thd=thd@entry=0x7eee540009a8) at /home/svoj/devel/maria/mariadb/sql/mysqld.cc:2942
#5  0x0000558976954409 in one_thread_per_connection_end (thd=0x7eee540009a8, put_in_cache=<optimized out>) at /home/svoj/devel/maria/mariadb/sql/mysqld.cc:3079
#6  0x0000558976abcd56 in do_handle_one_connection (connect=<optimized out>) at /home/svoj/devel/maria/mariadb/sql/sql_connect.cc:1421
#7  0x0000558976abce69 in handle_one_connection (arg=<optimized out>) at /home/svoj/devel/maria/mariadb/sql/sql_connect.cc:1308
#8  0x00007efc8bc636ba in start_thread (arg=0x7eeee00e5700) at pthread_create.c:333
#9  0x00007efc8b30e41d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109
 
 
#1  0x000055897695443f in inline_mysql_mutex_lock (that=0x558977a3eee0 <LOCK_thread_cache>) at /home/svoj/devel/maria/mariadb/include/mysql/psi/mysql_thread.h:712
#2  cache_thread (thd=0x7eee540009a8) at /home/svoj/devel/maria/mariadb/sql/mysqld.cc:2971
#3  one_thread_per_connection_end (thd=0x7eee540009a8, put_in_cache=<optimized out>) at /home/svoj/devel/maria/mariadb/sql/mysqld.cc:3080
#4  0x0000558976abcd56 in do_handle_one_connection (connect=<optimized out>) at /home/svoj/devel/maria/mariadb/sql/sql_connect.cc:1421
#5  0x0000558976abce69 in handle_one_connection (arg=<optimized out>) at /home/svoj/devel/maria/mariadb/sql/sql_connect.cc:1308
#6  0x00007efc8bc636ba in start_thread (arg=0x7eeee00e5700) at pthread_create.c:333
#7  0x00007efc8b30e41d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109
 
 
#1  0x00005589769b30ff in inline_mysql_mutex_lock (that=<optimized out>) at /home/svoj/devel/maria/mariadb/include/mysql/psi/mysql_thread.h:712
#2  THD::init (this=this@entry=0x7eee540009a8, skip_lock=skip_lock@entry=false) at /home/svoj/devel/maria/mariadb/sql/sql_class.cc:1272
#3  0x00005589769b6289 in THD::change_user (this=this@entry=0x7eee540009a8) at /home/svoj/devel/maria/mariadb/sql/sql_class.cc:1466
#4  0x00005589769b6389 in THD::reset_for_reuse (this=this@entry=0x7eee540009a8) at /home/svoj/devel/maria/mariadb/sql/sql_class.cc:1658
#5  0x0000558976abc9e4 in CONNECT::create_thd (this=this@entry=0x5589ce5a4e88, thd=thd@entry=0x7eee540009a8) at /home/svoj/devel/maria/mariadb/sql/sql_connect.cc:1498
#6  0x00005589769545a2 in cache_thread (thd=0x7eee540009a8) at /home/svoj/devel/maria/mariadb/sql/mysqld.cc:3016
#7  one_thread_per_connection_end (thd=0x7eee540009a8, put_in_cache=<optimized out>) at /home/svoj/devel/maria/mariadb/sql/mysqld.cc:3080
#8  0x0000558976abcd56 in do_handle_one_connection (connect=<optimized out>) at /home/svoj/devel/maria/mariadb/sql/sql_connect.cc:1421
#9  0x0000558976abce69 in handle_one_connection (arg=<optimized out>) at /home/svoj/devel/maria/mariadb/sql/sql_connect.cc:1308
#10 0x00007efc8bc636ba in start_thread (arg=0x7eeee00e5700) at pthread_create.c:333
#11 0x00007efc8b30e41d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109
 
 
#1  0x00005589770b3a61 in rw_pr_rdlock (rwlock=0x558977a3e860 <LOCK_system_variables_hash>) at /home/svoj/devel/maria/mariadb/mysys/thr_rwlock.c:377
#2  0x00005589769f8fb9 in inline_mysql_prlock_rdlock (that=<optimized out>) at /home/svoj/devel/maria/mariadb/include/mysql/psi/mysql_thread.h:928
#3  cleanup_variables (vars=0x7eee54001018) at /home/svoj/devel/maria/mariadb/sql/sql_plugin.cc:3210
#4  plugin_thdvar_init (thd=thd@entry=0x7eee540009a8) at /home/svoj/devel/maria/mariadb/sql/sql_plugin.cc:3160
#5  0x00005589769b3107 in THD::init (this=this@entry=0x7eee540009a8, skip_lock=skip_lock@entry=false) at /home/svoj/devel/maria/mariadb/sql/sql_class.cc:1273
#6  0x00005589769b6289 in THD::change_user (this=this@entry=0x7eee540009a8) at /home/svoj/devel/maria/mariadb/sql/sql_class.cc:1466
#7  0x00005589769b6389 in THD::reset_for_reuse (this=this@entry=0x7eee540009a8) at /home/svoj/devel/maria/mariadb/sql/sql_class.cc:1658
#8  0x0000558976abc9e4 in CONNECT::create_thd (this=this@entry=0x5589ce5a4e88, thd=thd@entry=0x7eee540009a8) at /home/svoj/devel/maria/mariadb/sql/sql_connect.cc:1498
#9  0x00005589769545a2 in cache_thread (thd=0x7eee540009a8) at /home/svoj/devel/maria/mariadb/sql/mysqld.cc:3016
#10 one_thread_per_connection_end (thd=0x7eee540009a8, put_in_cache=<optimized out>) at /home/svoj/devel/maria/mariadb/sql/mysqld.cc:3080
#11 0x0000558976abcd56 in do_handle_one_connection (connect=<optimized out>) at /home/svoj/devel/maria/mariadb/sql/sql_connect.cc:1421
#12 0x0000558976abce69 in handle_one_connection (arg=<optimized out>) at /home/svoj/devel/maria/mariadb/sql/sql_connect.cc:1308
#13 0x00007efc8bc636ba in start_thread (arg=0x7eeee00e5700) at pthread_create.c:333
#14 0x00007efc8b30e41d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109
 
 
#1  0x00005589770b3b05 in rw_pr_unlock (rwlock=0x558977a3e860 <LOCK_system_variables_hash>) at /home/svoj/devel/maria/mariadb/mysys/thr_rwlock.c:456
#2  0x00005589769f9013 in inline_mysql_prlock_unlock (that=<optimized out>) at /home/svoj/devel/maria/mariadb/include/mysql/psi/mysql_thread.h:1097
#3  cleanup_variables (vars=0x7eee54001018) at /home/svoj/devel/maria/mariadb/sql/sql_plugin.cc:3229
#4  plugin_thdvar_init (thd=thd@entry=0x7eee540009a8) at /home/svoj/devel/maria/mariadb/sql/sql_plugin.cc:3160
#5  0x00005589769b3107 in THD::init (this=this@entry=0x7eee540009a8, skip_lock=skip_lock@entry=false) at /home/svoj/devel/maria/mariadb/sql/sql_class.cc:1273
#6  0x00005589769b6289 in THD::change_user (this=this@entry=0x7eee540009a8) at /home/svoj/devel/maria/mariadb/sql/sql_class.cc:1466
#7  0x00005589769b6389 in THD::reset_for_reuse (this=this@entry=0x7eee540009a8) at /home/svoj/devel/maria/mariadb/sql/sql_class.cc:1658
#8  0x0000558976abc9e4 in CONNECT::create_thd (this=this@entry=0x5589ce5a4e88, thd=thd@entry=0x7eee540009a8) at /home/svoj/devel/maria/mariadb/sql/sql_connect.cc:1498
#9  0x00005589769545a2 in cache_thread (thd=0x7eee540009a8) at /home/svoj/devel/maria/mariadb/sql/mysqld.cc:3016
#10 one_thread_per_connection_end (thd=0x7eee540009a8, put_in_cache=<optimized out>) at /home/svoj/devel/maria/mariadb/sql/mysqld.cc:3080
#11 0x0000558976abcd56 in do_handle_one_connection (connect=<optimized out>) at /home/svoj/devel/maria/mariadb/sql/sql_connect.cc:1421
#12 0x0000558976abce69 in handle_one_connection (arg=<optimized out>) at /home/svoj/devel/maria/mariadb/sql/sql_connect.cc:1308
#13 0x00007efc8bc636ba in start_thread (arg=0x7eeee00e5700) at pthread_create.c:333
#14 0x00007efc8b30e41d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109
 
 
#1  0x00005589769f9064 in inline_mysql_mutex_lock (that=0x558977a441a0 <LOCK_plugin>) at /home/svoj/devel/maria/mariadb/include/mysql/psi/mysql_thread.h:712
#2  plugin_thdvar_init (thd=thd@entry=0x7eee540009a8) at /home/svoj/devel/maria/mariadb/sql/sql_plugin.cc:3169
#3  0x00005589769b3107 in THD::init (this=this@entry=0x7eee540009a8, skip_lock=skip_lock@entry=false) at /home/svoj/devel/maria/mariadb/sql/sql_class.cc:1273
#4  0x00005589769b6289 in THD::change_user (this=this@entry=0x7eee540009a8) at /home/svoj/devel/maria/mariadb/sql/sql_class.cc:1466
#5  0x00005589769b6389 in THD::reset_for_reuse (this=this@entry=0x7eee540009a8) at /home/svoj/devel/maria/mariadb/sql/sql_class.cc:1658
#6  0x0000558976abc9e4 in CONNECT::create_thd (this=this@entry=0x5589ce5a4e88, thd=thd@entry=0x7eee540009a8) at /home/svoj/devel/maria/mariadb/sql/sql_connect.cc:1498
#7  0x00005589769545a2 in cache_thread (thd=0x7eee540009a8) at /home/svoj/devel/maria/mariadb/sql/mysqld.cc:3016
#8  one_thread_per_connection_end (thd=0x7eee540009a8, put_in_cache=<optimized out>) at /home/svoj/devel/maria/mariadb/sql/mysqld.cc:3080
#9  0x0000558976abcd56 in do_handle_one_connection (connect=<optimized out>) at /home/svoj/devel/maria/mariadb/sql/sql_connect.cc:1421
#10 0x0000558976abce69 in handle_one_connection (arg=<optimized out>) at /home/svoj/devel/maria/mariadb/sql/sql_connect.cc:1308
#11 0x00007efc8bc636ba in start_thread (arg=0x7eeee00e5700) at pthread_create.c:333
#12 0x00007efc8b30e41d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109
 
 
#1  0x0000558976966fe0 in inline_mysql_mutex_lock (that=<optimized out>) at /home/svoj/devel/maria/mariadb/include/mysql/psi/mysql_thread.h:712
#2  Session_sysvars_tracker::enable (this=0x7eee54006690, thd=0x7eee540009a8) at /home/svoj/devel/maria/mariadb/sql/session_tracker.cc:668
#3  0x00005589769684e4 in Session_tracker::enable (this=this@entry=0x7eee54005dc0, thd=thd@entry=0x7eee540009a8) at /home/svoj/devel/maria/mariadb/sql/session_tracker.cc:1640
#4  0x00005589769b3082 in THD::init (this=this@entry=0x7eee540009a8, skip_lock=skip_lock@entry=false) at /home/svoj/devel/maria/mariadb/sql/sql_class.cc:1360
#5  0x00005589769b6289 in THD::change_user (this=this@entry=0x7eee540009a8) at /home/svoj/devel/maria/mariadb/sql/sql_class.cc:1466
#6  0x00005589769b6389 in THD::reset_for_reuse (this=this@entry=0x7eee540009a8) at /home/svoj/devel/maria/mariadb/sql/sql_class.cc:1658
#7  0x0000558976abc9e4 in CONNECT::create_thd (this=this@entry=0x5589ce5a4e88, thd=thd@entry=0x7eee540009a8) at /home/svoj/devel/maria/mariadb/sql/sql_connect.cc:1498
#8  0x00005589769545a2 in cache_thread (thd=0x7eee540009a8) at /home/svoj/devel/maria/mariadb/sql/mysqld.cc:3016
#9  one_thread_per_connection_end (thd=0x7eee540009a8, put_in_cache=<optimized out>) at /home/svoj/devel/maria/mariadb/sql/mysqld.cc:3080
#10 0x0000558976abcd56 in do_handle_one_connection (connect=<optimized out>) at /home/svoj/devel/maria/mariadb/sql/sql_connect.cc:1421
#11 0x0000558976abce69 in handle_one_connection (arg=<optimized out>) at /home/svoj/devel/maria/mariadb/sql/sql_connect.cc:1308
#12 0x00007efc8bc636ba in start_thread (arg=0x7eeee00e5700) at pthread_create.c:333
#13 0x00007efc8b30e41d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109
 
 
#1  0x00005589770b3a61 in rw_pr_rdlock (rwlock=0x558977a3e860 <LOCK_system_variables_hash>) at /home/svoj/devel/maria/mariadb/mysys/thr_rwlock.c:377
#2  0x00005589769f8615 in inline_mysql_prlock_rdlock (that=<optimized out>) at /home/svoj/devel/maria/mariadb/include/mysql/psi/mysql_thread.h:928
#3  find_sys_var_ex (thd=thd@entry=0x7eee540009a8, str=0x5589771a8180 "autocommit,character_set_client,character_set_connection,character_set_results,time_zone", length=10,
    throw_error=throw_error@entry=true, locked=locked@entry=true) at /home/svoj/devel/maria/mariadb/sql/sql_plugin.cc:2846
#4  0x0000558976966ed0 in Session_sysvars_tracker::vars_list::parse_var_list (this=0x7eee540067a0, thd=thd@entry=0x7eee540009a8, var_list=..., throw_error=throw_error@entry=true,
    char_set=0x5589779ee700 <my_charset_latin1>, take_mutex=false) at /home/svoj/devel/maria/mariadb/sql/session_tracker.cc:442
#5  0x0000558976967025 in Session_sysvars_tracker::enable (this=0x7eee54006690, thd=0x7eee540009a8) at /home/svoj/devel/maria/mariadb/sql/session_tracker.cc:672
#6  0x00005589769684e4 in Session_tracker::enable (this=this@entry=0x7eee54005dc0, thd=thd@entry=0x7eee540009a8) at /home/svoj/devel/maria/mariadb/sql/session_tracker.cc:1640
#7  0x00005589769b3082 in THD::init (this=this@entry=0x7eee540009a8, skip_lock=skip_lock@entry=false) at /home/svoj/devel/maria/mariadb/sql/sql_class.cc:1360
#8  0x00005589769b6289 in THD::change_user (this=this@entry=0x7eee540009a8) at /home/svoj/devel/maria/mariadb/sql/sql_class.cc:1466
#9  0x00005589769b6389 in THD::reset_for_reuse (this=this@entry=0x7eee540009a8) at /home/svoj/devel/maria/mariadb/sql/sql_class.cc:1658
#10 0x0000558976abc9e4 in CONNECT::create_thd (this=this@entry=0x5589ce5a4e88, thd=thd@entry=0x7eee540009a8) at /home/svoj/devel/maria/mariadb/sql/sql_connect.cc:1498
#11 0x00005589769545a2 in cache_thread (thd=0x7eee540009a8) at /home/svoj/devel/maria/mariadb/sql/mysqld.cc:3016
#12 one_thread_per_connection_end (thd=0x7eee540009a8, put_in_cache=<optimized out>) at /home/svoj/devel/maria/mariadb/sql/mysqld.cc:3080
#13 0x0000558976abcd56 in do_handle_one_connection (connect=<optimized out>) at /home/svoj/devel/maria/mariadb/sql/sql_connect.cc:1421
#14 0x0000558976abce69 in handle_one_connection (arg=<optimized out>) at /home/svoj/devel/maria/mariadb/sql/sql_connect.cc:1308
#15 0x00007efc8bc636ba in start_thread (arg=0x7eeee00e5700) at pthread_create.c:333
#16 0x00007efc8b30e41d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109
 
 
#1  0x00005589770b3b05 in rw_pr_unlock (rwlock=0x558977a3e860 <LOCK_system_variables_hash>) at /home/svoj/devel/maria/mariadb/mysys/thr_rwlock.c:456
#2  0x00005589769f870c in inline_mysql_prlock_unlock (that=<optimized out>) at /home/svoj/devel/maria/mariadb/include/mysql/psi/mysql_thread.h:1097
#3  find_sys_var_ex (thd=thd@entry=0x7eee540009a8, str=0x5589771a8180 "autocommit,character_set_client,character_set_connection,character_set_results,time_zone", length=10,
    throw_error=throw_error@entry=true, locked=locked@entry=true) at /home/svoj/devel/maria/mariadb/sql/sql_plugin.cc:2863
#4  0x0000558976966ed0 in Session_sysvars_tracker::vars_list::parse_var_list (this=0x7eee540067a0, thd=thd@entry=0x7eee540009a8, var_list=..., throw_error=throw_error@entry=true,
    char_set=0x5589779ee700 <my_charset_latin1>, take_mutex=false) at /home/svoj/devel/maria/mariadb/sql/session_tracker.cc:442
#5  0x0000558976967025 in Session_sysvars_tracker::enable (this=0x7eee54006690, thd=0x7eee540009a8) at /home/svoj/devel/maria/mariadb/sql/session_tracker.cc:672
#6  0x00005589769684e4 in Session_tracker::enable (this=this@entry=0x7eee54005dc0, thd=thd@entry=0x7eee540009a8) at /home/svoj/devel/maria/mariadb/sql/session_tracker.cc:1640
#7  0x00005589769b3082 in THD::init (this=this@entry=0x7eee540009a8, skip_lock=skip_lock@entry=false) at /home/svoj/devel/maria/mariadb/sql/sql_class.cc:1360
#8  0x00005589769b6289 in THD::change_user (this=this@entry=0x7eee540009a8) at /home/svoj/devel/maria/mariadb/sql/sql_class.cc:1466
#9  0x00005589769b6389 in THD::reset_for_reuse (this=this@entry=0x7eee540009a8) at /home/svoj/devel/maria/mariadb/sql/sql_class.cc:1658
#10 0x0000558976abc9e4 in CONNECT::create_thd (this=this@entry=0x5589ce5a4e88, thd=thd@entry=0x7eee540009a8) at /home/svoj/devel/maria/mariadb/sql/sql_connect.cc:1498
#11 0x00005589769545a2 in cache_thread (thd=0x7eee540009a8) at /home/svoj/devel/maria/mariadb/sql/mysqld.cc:3016
#12 one_thread_per_connection_end (thd=0x7eee540009a8, put_in_cache=<optimized out>) at /home/svoj/devel/maria/mariadb/sql/mysqld.cc:3080
#13 0x0000558976abcd56 in do_handle_one_connection (connect=<optimized out>) at /home/svoj/devel/maria/mariadb/sql/sql_connect.cc:1421
#14 0x0000558976abce69 in handle_one_connection (arg=<optimized out>) at /home/svoj/devel/maria/mariadb/sql/sql_connect.cc:1308
#15 0x00007efc8bc636ba in start_thread (arg=0x7eeee00e5700) at pthread_create.c:333
#16 0x00007efc8b30e41d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109
 
 
#1  0x00005589770b3a61 in rw_pr_rdlock (rwlock=0x558977a3e860 <LOCK_system_variables_hash>) at /home/svoj/devel/maria/mariadb/mysys/thr_rwlock.c:377
#2  0x00005589769f8615 in inline_mysql_prlock_rdlock (that=<optimized out>) at /home/svoj/devel/maria/mariadb/include/mysql/psi/mysql_thread.h:928
#3  find_sys_var_ex (thd=thd@entry=0x7eee540009a8, str=0x5589771a818b "character_set_client,character_set_connection,character_set_results,time_zone", length=20, throw_error=throw_error@entry=true,
    locked=locked@entry=true) at /home/svoj/devel/maria/mariadb/sql/sql_plugin.cc:2846
#4  0x0000558976966ed0 in Session_sysvars_tracker::vars_list::parse_var_list (this=0x7eee540067a0, thd=thd@entry=0x7eee540009a8, var_list=..., throw_error=throw_error@entry=true,
    char_set=0x5589779ee700 <my_charset_latin1>, take_mutex=false) at /home/svoj/devel/maria/mariadb/sql/session_tracker.cc:442
#5  0x0000558976967025 in Session_sysvars_tracker::enable (this=0x7eee54006690, thd=0x7eee540009a8) at /home/svoj/devel/maria/mariadb/sql/session_tracker.cc:672
#6  0x00005589769684e4 in Session_tracker::enable (this=this@entry=0x7eee54005dc0, thd=thd@entry=0x7eee540009a8) at /home/svoj/devel/maria/mariadb/sql/session_tracker.cc:1640
#7  0x00005589769b3082 in THD::init (this=this@entry=0x7eee540009a8, skip_lock=skip_lock@entry=false) at /home/svoj/devel/maria/mariadb/sql/sql_class.cc:1360
#8  0x00005589769b6289 in THD::change_user (this=this@entry=0x7eee540009a8) at /home/svoj/devel/maria/mariadb/sql/sql_class.cc:1466
#9  0x00005589769b6389 in THD::reset_for_reuse (this=this@entry=0x7eee540009a8) at /home/svoj/devel/maria/mariadb/sql/sql_class.cc:1658
#10 0x0000558976abc9e4 in CONNECT::create_thd (this=this@entry=0x5589ce5a4e88, thd=thd@entry=0x7eee540009a8) at /home/svoj/devel/maria/mariadb/sql/sql_connect.cc:1498
#11 0x00005589769545a2 in cache_thread (thd=0x7eee540009a8) at /home/svoj/devel/maria/mariadb/sql/mysqld.cc:3016
#12 one_thread_per_connection_end (thd=0x7eee540009a8, put_in_cache=<optimized out>) at /home/svoj/devel/maria/mariadb/sql/mysqld.cc:3080
#13 0x0000558976abcd56 in do_handle_one_connection (connect=<optimized out>) at /home/svoj/devel/maria/mariadb/sql/sql_connect.cc:1421
#14 0x0000558976abce69 in handle_one_connection (arg=<optimized out>) at /home/svoj/devel/maria/mariadb/sql/sql_connect.cc:1308
#15 0x00007efc8bc636ba in start_thread (arg=0x7eeee00e5700) at pthread_create.c:333
#16 0x00007efc8b30e41d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109
 
 
#1  0x00005589770b3b05 in rw_pr_unlock (rwlock=0x558977a3e860 <LOCK_system_variables_hash>) at /home/svoj/devel/maria/mariadb/mysys/thr_rwlock.c:456
#2  0x00005589769f870c in inline_mysql_prlock_unlock (that=<optimized out>) at /home/svoj/devel/maria/mariadb/include/mysql/psi/mysql_thread.h:1097
#3  find_sys_var_ex (thd=thd@entry=0x7eee540009a8, str=0x5589771a818b "character_set_client,character_set_connection,character_set_results,time_zone", length=20, throw_error=throw_error@entry=true,
    locked=locked@entry=true) at /home/svoj/devel/maria/mariadb/sql/sql_plugin.cc:2863
#4  0x0000558976966ed0 in Session_sysvars_tracker::vars_list::parse_var_list (this=0x7eee540067a0, thd=thd@entry=0x7eee540009a8, var_list=..., throw_error=throw_error@entry=true,
    char_set=0x5589779ee700 <my_charset_latin1>, take_mutex=false) at /home/svoj/devel/maria/mariadb/sql/session_tracker.cc:442
#5  0x0000558976967025 in Session_sysvars_tracker::enable (this=0x7eee54006690, thd=0x7eee540009a8) at /home/svoj/devel/maria/mariadb/sql/session_tracker.cc:672
#6  0x00005589769684e4 in Session_tracker::enable (this=this@entry=0x7eee54005dc0, thd=thd@entry=0x7eee540009a8) at /home/svoj/devel/maria/mariadb/sql/session_tracker.cc:1640
#7  0x00005589769b3082 in THD::init (this=this@entry=0x7eee540009a8, skip_lock=skip_lock@entry=false) at /home/svoj/devel/maria/mariadb/sql/sql_class.cc:1360
#8  0x00005589769b6289 in THD::change_user (this=this@entry=0x7eee540009a8) at /home/svoj/devel/maria/mariadb/sql/sql_class.cc:1466
#9  0x00005589769b6389 in THD::reset_for_reuse (this=this@entry=0x7eee540009a8) at /home/svoj/devel/maria/mariadb/sql/sql_class.cc:1658
#10 0x0000558976abc9e4 in CONNECT::create_thd (this=this@entry=0x5589ce5a4e88, thd=thd@entry=0x7eee540009a8) at /home/svoj/devel/maria/mariadb/sql/sql_connect.cc:1498
#11 0x00005589769545a2 in cache_thread (thd=0x7eee540009a8) at /home/svoj/devel/maria/mariadb/sql/mysqld.cc:3016
#12 one_thread_per_connection_end (thd=0x7eee540009a8, put_in_cache=<optimized out>) at /home/svoj/devel/maria/mariadb/sql/mysqld.cc:3080
#13 0x0000558976abcd56 in do_handle_one_connection (connect=<optimized out>) at /home/svoj/devel/maria/mariadb/sql/sql_connect.cc:1421
#14 0x0000558976abce69 in handle_one_connection (arg=<optimized out>) at /home/svoj/devel/maria/mariadb/sql/sql_connect.cc:1308
#15 0x00007efc8bc636ba in start_thread (arg=0x7eeee00e5700) at pthread_create.c:333
#16 0x00007efc8b30e41d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109
 
 
#1  0x00005589770b3a61 in rw_pr_rdlock (rwlock=0x558977a3e860 <LOCK_system_variables_hash>) at /home/svoj/devel/maria/mariadb/mysys/thr_rwlock.c:377
#2  0x00005589769f8615 in inline_mysql_prlock_rdlock (that=<optimized out>) at /home/svoj/devel/maria/mariadb/include/mysql/psi/mysql_thread.h:928
#3  find_sys_var_ex (thd=thd@entry=0x7eee540009a8, str=0x5589771a81a0 "character_set_connection,character_set_results,time_zone", length=24, throw_error=throw_error@entry=true, locked=locked@entry=true)
    at /home/svoj/devel/maria/mariadb/sql/sql_plugin.cc:2846
#4  0x0000558976966ed0 in Session_sysvars_tracker::vars_list::parse_var_list (this=0x7eee540067a0, thd=thd@entry=0x7eee540009a8, var_list=..., throw_error=throw_error@entry=true,
    char_set=0x5589779ee700 <my_charset_latin1>, take_mutex=false) at /home/svoj/devel/maria/mariadb/sql/session_tracker.cc:442
#5  0x0000558976967025 in Session_sysvars_tracker::enable (this=0x7eee54006690, thd=0x7eee540009a8) at /home/svoj/devel/maria/mariadb/sql/session_tracker.cc:672
#6  0x00005589769684e4 in Session_tracker::enable (this=this@entry=0x7eee54005dc0, thd=thd@entry=0x7eee540009a8) at /home/svoj/devel/maria/mariadb/sql/session_tracker.cc:1640
#7  0x00005589769b3082 in THD::init (this=this@entry=0x7eee540009a8, skip_lock=skip_lock@entry=false) at /home/svoj/devel/maria/mariadb/sql/sql_class.cc:1360
#8  0x00005589769b6289 in THD::change_user (this=this@entry=0x7eee540009a8) at /home/svoj/devel/maria/mariadb/sql/sql_class.cc:1466
#9  0x00005589769b6389 in THD::reset_for_reuse (this=this@entry=0x7eee540009a8) at /home/svoj/devel/maria/mariadb/sql/sql_class.cc:1658
#10 0x0000558976abc9e4 in CONNECT::create_thd (this=this@entry=0x5589ce5a4e88, thd=thd@entry=0x7eee540009a8) at /home/svoj/devel/maria/mariadb/sql/sql_connect.cc:1498
#11 0x00005589769545a2 in cache_thread (thd=0x7eee540009a8) at /home/svoj/devel/maria/mariadb/sql/mysqld.cc:3016
#12 one_thread_per_connection_end (thd=0x7eee540009a8, put_in_cache=<optimized out>) at /home/svoj/devel/maria/mariadb/sql/mysqld.cc:3080
#13 0x0000558976abcd56 in do_handle_one_connection (connect=<optimized out>) at /home/svoj/devel/maria/mariadb/sql/sql_connect.cc:1421
#14 0x0000558976abce69 in handle_one_connection (arg=<optimized out>) at /home/svoj/devel/maria/mariadb/sql/sql_connect.cc:1308
#15 0x00007efc8bc636ba in start_thread (arg=0x7eeee00e5700) at pthread_create.c:333
#16 0x00007efc8b30e41d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109
 
 
#1  0x00005589770b3b05 in rw_pr_unlock (rwlock=0x558977a3e860 <LOCK_system_variables_hash>) at /home/svoj/devel/maria/mariadb/mysys/thr_rwlock.c:456
#2  0x00005589769f870c in inline_mysql_prlock_unlock (that=<optimized out>) at /home/svoj/devel/maria/mariadb/include/mysql/psi/mysql_thread.h:1097
#3  find_sys_var_ex (thd=thd@entry=0x7eee540009a8, str=0x5589771a81a0 "character_set_connection,character_set_results,time_zone", length=24, throw_error=throw_error@entry=true, locked=locked@entry=true)
    at /home/svoj/devel/maria/mariadb/sql/sql_plugin.cc:2863
#4  0x0000558976966ed0 in Session_sysvars_tracker::vars_list::parse_var_list (this=0x7eee540067a0, thd=thd@entry=0x7eee540009a8, var_list=..., throw_error=throw_error@entry=true,
    char_set=0x5589779ee700 <my_charset_latin1>, take_mutex=false) at /home/svoj/devel/maria/mariadb/sql/session_tracker.cc:442
#5  0x0000558976967025 in Session_sysvars_tracker::enable (this=0x7eee54006690, thd=0x7eee540009a8) at /home/svoj/devel/maria/mariadb/sql/session_tracker.cc:672
#6  0x00005589769684e4 in Session_tracker::enable (this=this@entry=0x7eee54005dc0, thd=thd@entry=0x7eee540009a8) at /home/svoj/devel/maria/mariadb/sql/session_tracker.cc:1640
#7  0x00005589769b3082 in THD::init (this=this@entry=0x7eee540009a8, skip_lock=skip_lock@entry=false) at /home/svoj/devel/maria/mariadb/sql/sql_class.cc:1360
#8  0x00005589769b6289 in THD::change_user (this=this@entry=0x7eee540009a8) at /home/svoj/devel/maria/mariadb/sql/sql_class.cc:1466
#9  0x00005589769b6389 in THD::reset_for_reuse (this=this@entry=0x7eee540009a8) at /home/svoj/devel/maria/mariadb/sql/sql_class.cc:1658
#10 0x0000558976abc9e4 in CONNECT::create_thd (this=this@entry=0x5589ce5a4e88, thd=thd@entry=0x7eee540009a8) at /home/svoj/devel/maria/mariadb/sql/sql_connect.cc:1498
#11 0x00005589769545a2 in cache_thread (thd=0x7eee540009a8) at /home/svoj/devel/maria/mariadb/sql/mysqld.cc:3016
#12 one_thread_per_connection_end (thd=0x7eee540009a8, put_in_cache=<optimized out>) at /home/svoj/devel/maria/mariadb/sql/mysqld.cc:3080
#13 0x0000558976abcd56 in do_handle_one_connection (connect=<optimized out>) at /home/svoj/devel/maria/mariadb/sql/sql_connect.cc:1421
#14 0x0000558976abce69 in handle_one_connection (arg=<optimized out>) at /home/svoj/devel/maria/mariadb/sql/sql_connect.cc:1308
#15 0x00007efc8bc636ba in start_thread (arg=0x7eeee00e5700) at pthread_create.c:333
#16 0x00007efc8b30e41d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109
 
 
#1  0x00005589770b3a61 in rw_pr_rdlock (rwlock=0x558977a3e860 <LOCK_system_variables_hash>) at /home/svoj/devel/maria/mariadb/mysys/thr_rwlock.c:377
#2  0x00005589769f8615 in inline_mysql_prlock_rdlock (that=<optimized out>) at /home/svoj/devel/maria/mariadb/include/mysql/psi/mysql_thread.h:928
#3  find_sys_var_ex (thd=thd@entry=0x7eee540009a8, str=0x5589771a81b9 "character_set_results,time_zone", length=21, throw_error=throw_error@entry=true, locked=locked@entry=true)
    at /home/svoj/devel/maria/mariadb/sql/sql_plugin.cc:2846
#4  0x0000558976966ed0 in Session_sysvars_tracker::vars_list::parse_var_list (this=0x7eee540067a0, thd=thd@entry=0x7eee540009a8, var_list=..., throw_error=throw_error@entry=true,
    char_set=0x5589779ee700 <my_charset_latin1>, take_mutex=false) at /home/svoj/devel/maria/mariadb/sql/session_tracker.cc:442
#5  0x0000558976967025 in Session_sysvars_tracker::enable (this=0x7eee54006690, thd=0x7eee540009a8) at /home/svoj/devel/maria/mariadb/sql/session_tracker.cc:672
#6  0x00005589769684e4 in Session_tracker::enable (this=this@entry=0x7eee54005dc0, thd=thd@entry=0x7eee540009a8) at /home/svoj/devel/maria/mariadb/sql/session_tracker.cc:1640
#7  0x00005589769b3082 in THD::init (this=this@entry=0x7eee540009a8, skip_lock=skip_lock@entry=false) at /home/svoj/devel/maria/mariadb/sql/sql_class.cc:1360
#8  0x00005589769b6289 in THD::change_user (this=this@entry=0x7eee540009a8) at /home/svoj/devel/maria/mariadb/sql/sql_class.cc:1466
#9  0x00005589769b6389 in THD::reset_for_reuse (this=this@entry=0x7eee540009a8) at /home/svoj/devel/maria/mariadb/sql/sql_class.cc:1658
#10 0x0000558976abc9e4 in CONNECT::create_thd (this=this@entry=0x5589ce5a4e88, thd=thd@entry=0x7eee540009a8) at /home/svoj/devel/maria/mariadb/sql/sql_connect.cc:1498
#11 0x00005589769545a2 in cache_thread (thd=0x7eee540009a8) at /home/svoj/devel/maria/mariadb/sql/mysqld.cc:3016
#12 one_thread_per_connection_end (thd=0x7eee540009a8, put_in_cache=<optimized out>) at /home/svoj/devel/maria/mariadb/sql/mysqld.cc:3080
#13 0x0000558976abcd56 in do_handle_one_connection (connect=<optimized out>) at /home/svoj/devel/maria/mariadb/sql/sql_connect.cc:1421
#14 0x0000558976abce69 in handle_one_connection (arg=<optimized out>) at /home/svoj/devel/maria/mariadb/sql/sql_connect.cc:1308
#15 0x00007efc8bc636ba in start_thread (arg=0x7eeee00e5700) at pthread_create.c:333
#16 0x00007efc8b30e41d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109
 
 
#1  0x00005589770b3b05 in rw_pr_unlock (rwlock=0x558977a3e860 <LOCK_system_variables_hash>) at /home/svoj/devel/maria/mariadb/mysys/thr_rwlock.c:456
#2  0x00005589769f870c in inline_mysql_prlock_unlock (that=<optimized out>) at /home/svoj/devel/maria/mariadb/include/mysql/psi/mysql_thread.h:1097
#3  find_sys_var_ex (thd=thd@entry=0x7eee540009a8, str=0x5589771a81b9 "character_set_results,time_zone", length=21, throw_error=throw_error@entry=true, locked=locked@entry=true)
    at /home/svoj/devel/maria/mariadb/sql/sql_plugin.cc:2863
#4  0x0000558976966ed0 in Session_sysvars_tracker::vars_list::parse_var_list (this=0x7eee540067a0, thd=thd@entry=0x7eee540009a8, var_list=..., throw_error=throw_error@entry=true,
    char_set=0x5589779ee700 <my_charset_latin1>, take_mutex=false) at /home/svoj/devel/maria/mariadb/sql/session_tracker.cc:442
#5  0x0000558976967025 in Session_sysvars_tracker::enable (this=0x7eee54006690, thd=0x7eee540009a8) at /home/svoj/devel/maria/mariadb/sql/session_tracker.cc:672
#6  0x00005589769684e4 in Session_tracker::enable (this=this@entry=0x7eee54005dc0, thd=thd@entry=0x7eee540009a8) at /home/svoj/devel/maria/mariadb/sql/session_tracker.cc:1640
#7  0x00005589769b3082 in THD::init (this=this@entry=0x7eee540009a8, skip_lock=skip_lock@entry=false) at /home/svoj/devel/maria/mariadb/sql/sql_class.cc:1360
#8  0x00005589769b6289 in THD::change_user (this=this@entry=0x7eee540009a8) at /home/svoj/devel/maria/mariadb/sql/sql_class.cc:1466
#9  0x00005589769b6389 in THD::reset_for_reuse (this=this@entry=0x7eee540009a8) at /home/svoj/devel/maria/mariadb/sql/sql_class.cc:1658
#10 0x0000558976abc9e4 in CONNECT::create_thd (this=this@entry=0x5589ce5a4e88, thd=thd@entry=0x7eee540009a8) at /home/svoj/devel/maria/mariadb/sql/sql_connect.cc:1498
#11 0x00005589769545a2 in cache_thread (thd=0x7eee540009a8) at /home/svoj/devel/maria/mariadb/sql/mysqld.cc:3016
#12 one_thread_per_connection_end (thd=0x7eee540009a8, put_in_cache=<optimized out>) at /home/svoj/devel/maria/mariadb/sql/mysqld.cc:3080
#13 0x0000558976abcd56 in do_handle_one_connection (connect=<optimized out>) at /home/svoj/devel/maria/mariadb/sql/sql_connect.cc:1421
#14 0x0000558976abce69 in handle_one_connection (arg=<optimized out>) at /home/svoj/devel/maria/mariadb/sql/sql_connect.cc:1308
#15 0x00007efc8bc636ba in start_thread (arg=0x7eeee00e5700) at pthread_create.c:333
#16 0x00007efc8b30e41d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109
 
 
#1  0x00005589770b3a61 in rw_pr_rdlock (rwlock=0x558977a3e860 <LOCK_system_variables_hash>) at /home/svoj/devel/maria/mariadb/mysys/thr_rwlock.c:377
#2  0x00005589769f8615 in inline_mysql_prlock_rdlock (that=<optimized out>) at /home/svoj/devel/maria/mariadb/include/mysql/psi/mysql_thread.h:928
#3  find_sys_var_ex (thd=thd@entry=0x7eee540009a8, str=0x5589771a81cf "time_zone", length=9, throw_error=throw_error@entry=true, locked=locked@entry=true)
    at /home/svoj/devel/maria/mariadb/sql/sql_plugin.cc:2846
#4  0x0000558976966ed0 in Session_sysvars_tracker::vars_list::parse_var_list (this=0x7eee540067a0, thd=thd@entry=0x7eee540009a8, var_list=..., throw_error=throw_error@entry=true,
    char_set=0x5589779ee700 <my_charset_latin1>, take_mutex=false) at /home/svoj/devel/maria/mariadb/sql/session_tracker.cc:442
#5  0x0000558976967025 in Session_sysvars_tracker::enable (this=0x7eee54006690, thd=0x7eee540009a8) at /home/svoj/devel/maria/mariadb/sql/session_tracker.cc:672
#6  0x00005589769684e4 in Session_tracker::enable (this=this@entry=0x7eee54005dc0, thd=thd@entry=0x7eee540009a8) at /home/svoj/devel/maria/mariadb/sql/session_tracker.cc:1640
#7  0x00005589769b3082 in THD::init (this=this@entry=0x7eee540009a8, skip_lock=skip_lock@entry=false) at /home/svoj/devel/maria/mariadb/sql/sql_class.cc:1360
#8  0x00005589769b6289 in THD::change_user (this=this@entry=0x7eee540009a8) at /home/svoj/devel/maria/mariadb/sql/sql_class.cc:1466
#9  0x00005589769b6389 in THD::reset_for_reuse (this=this@entry=0x7eee540009a8) at /home/svoj/devel/maria/mariadb/sql/sql_class.cc:1658
#10 0x0000558976abc9e4 in CONNECT::create_thd (this=this@entry=0x5589ce5a4e88, thd=thd@entry=0x7eee540009a8) at /home/svoj/devel/maria/mariadb/sql/sql_connect.cc:1498
#11 0x00005589769545a2 in cache_thread (thd=0x7eee540009a8) at /home/svoj/devel/maria/mariadb/sql/mysqld.cc:3016
#12 one_thread_per_connection_end (thd=0x7eee540009a8, put_in_cache=<optimized out>) at /home/svoj/devel/maria/mariadb/sql/mysqld.cc:3080
#13 0x0000558976abcd56 in do_handle_one_connection (connect=<optimized out>) at /home/svoj/devel/maria/mariadb/sql/sql_connect.cc:1421
#14 0x0000558976abce69 in handle_one_connection (arg=<optimized out>) at /home/svoj/devel/maria/mariadb/sql/sql_connect.cc:1308
#15 0x00007efc8bc636ba in start_thread (arg=0x7eeee00e5700) at pthread_create.c:333
#16 0x00007efc8b30e41d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109
 
 
#1  0x00005589770b3b05 in rw_pr_unlock (rwlock=0x558977a3e860 <LOCK_system_variables_hash>) at /home/svoj/devel/maria/mariadb/mysys/thr_rwlock.c:456
#2  0x00005589769f870c in inline_mysql_prlock_unlock (that=<optimized out>) at /home/svoj/devel/maria/mariadb/include/mysql/psi/mysql_thread.h:1097
#3  find_sys_var_ex (thd=thd@entry=0x7eee540009a8, str=0x5589771a81cf "time_zone", length=9, throw_error=throw_error@entry=true, locked=locked@entry=true)
    at /home/svoj/devel/maria/mariadb/sql/sql_plugin.cc:2863
#4  0x0000558976966ed0 in Session_sysvars_tracker::vars_list::parse_var_list (this=0x7eee540067a0, thd=thd@entry=0x7eee540009a8, var_list=..., throw_error=throw_error@entry=true,
    char_set=0x5589779ee700 <my_charset_latin1>, take_mutex=false) at /home/svoj/devel/maria/mariadb/sql/session_tracker.cc:442
#5  0x0000558976967025 in Session_sysvars_tracker::enable (this=0x7eee54006690, thd=0x7eee540009a8) at /home/svoj/devel/maria/mariadb/sql/session_tracker.cc:672
#6  0x00005589769684e4 in Session_tracker::enable (this=this@entry=0x7eee54005dc0, thd=thd@entry=0x7eee540009a8) at /home/svoj/devel/maria/mariadb/sql/session_tracker.cc:1640
#7  0x00005589769b3082 in THD::init (this=this@entry=0x7eee540009a8, skip_lock=skip_lock@entry=false) at /home/svoj/devel/maria/mariadb/sql/sql_class.cc:1360
#8  0x00005589769b6289 in THD::change_user (this=this@entry=0x7eee540009a8) at /home/svoj/devel/maria/mariadb/sql/sql_class.cc:1466
#9  0x00005589769b6389 in THD::reset_for_reuse (this=this@entry=0x7eee540009a8) at /home/svoj/devel/maria/mariadb/sql/sql_class.cc:1658
#10 0x0000558976abc9e4 in CONNECT::create_thd (this=this@entry=0x5589ce5a4e88, thd=thd@entry=0x7eee540009a8) at /home/svoj/devel/maria/mariadb/sql/sql_connect.cc:1498
#11 0x00005589769545a2 in cache_thread (thd=0x7eee540009a8) at /home/svoj/devel/maria/mariadb/sql/mysqld.cc:3016
#12 one_thread_per_connection_end (thd=0x7eee540009a8, put_in_cache=<optimized out>) at /home/svoj/devel/maria/mariadb/sql/mysqld.cc:3080
#13 0x0000558976abcd56 in do_handle_one_connection (connect=<optimized out>) at /home/svoj/devel/maria/mariadb/sql/sql_connect.cc:1421
#14 0x0000558976abce69 in handle_one_connection (arg=<optimized out>) at /home/svoj/devel/maria/mariadb/sql/sql_connect.cc:1308
#15 0x00007efc8bc636ba in start_thread (arg=0x7eeee00e5700) at pthread_create.c:333
#16 0x00007efc8b30e41d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109
 
 
#1  0x00005589769b2879 in inline_mysql_mutex_lock (that=<optimized out>) at /home/svoj/devel/maria/mariadb/include/mysql/psi/mysql_thread.h:712
#2  Statement_map::reset (this=this@entry=0x7eee540023e0) at /home/svoj/devel/maria/mariadb/sql/sql_class.cc:4044
#3  0x00005589769b6295 in THD::change_user (this=this@entry=0x7eee540009a8) at /home/svoj/devel/maria/mariadb/sql/sql_class.cc:1467
#4  0x00005589769b6389 in THD::reset_for_reuse (this=this@entry=0x7eee540009a8) at /home/svoj/devel/maria/mariadb/sql/sql_class.cc:1658
#5  0x0000558976abc9e4 in CONNECT::create_thd (this=this@entry=0x5589ce5a4e88, thd=thd@entry=0x7eee540009a8) at /home/svoj/devel/maria/mariadb/sql/sql_connect.cc:1498
#6  0x00005589769545a2 in cache_thread (thd=0x7eee540009a8) at /home/svoj/devel/maria/mariadb/sql/mysqld.cc:3016
#7  one_thread_per_connection_end (thd=0x7eee540009a8, put_in_cache=<optimized out>) at /home/svoj/devel/maria/mariadb/sql/mysqld.cc:3080
#8  0x0000558976abcd56 in do_handle_one_connection (connect=<optimized out>) at /home/svoj/devel/maria/mariadb/sql/sql_connect.cc:1421
#9  0x0000558976abce69 in handle_one_connection (arg=<optimized out>) at /home/svoj/devel/maria/mariadb/sql/sql_connect.cc:1308
#10 0x00007efc8bc636ba in start_thread (arg=0x7eeee00e5700) at pthread_create.c:333
#11 0x00007efc8b30e41d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109
 
 
#1  0x0000558976954608 in inline_mysql_mutex_lock (that=0x558977a3ef60 <LOCK_thread_count>) at /home/svoj/devel/maria/mariadb/include/mysql/psi/mysql_thread.h:712
#2  add_to_active_threads (thd=0x7eee540009a8) at /home/svoj/devel/maria/mariadb/sql/sql_class.h:4672
#3  cache_thread (thd=0x7eee540009a8) at /home/svoj/devel/maria/mariadb/sql/mysqld.cc:3043
#4  one_thread_per_connection_end (thd=0x7eee540009a8, put_in_cache=<optimized out>) at /home/svoj/devel/maria/mariadb/sql/mysqld.cc:3080
#5  0x0000558976abcd56 in do_handle_one_connection (connect=<optimized out>) at /home/svoj/devel/maria/mariadb/sql/sql_connect.cc:1421
#6  0x0000558976abce69 in handle_one_connection (arg=<optimized out>) at /home/svoj/devel/maria/mariadb/sql/sql_connect.cc:1308
#7  0x00007efc8bc636ba in start_thread (arg=0x7eeee00e5700) at pthread_create.c:333
#8  0x00007efc8b30e41d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109

Comment by Sergei Golubchik [ 2019-02-18 ]

A couple of ideas how to reduce the number of LOCK_plugin's above:

  • don't call plugin_thdvar_cleanup twice
  • intern_plugin_lock requires the caller to take LOCK_plugin even for built-in plugin. It can be changed to intern_plugin_lock itself locking the mutex unless it was already locked or the plugin was a built-in. It should returning the mutex status, so that the caller could pass it to the next intern_plugin_lock call and would know whether to unlock the mutex.
Comment by Sergey Vojtovich [ 2019-02-20 ]

Removing plugin_thdvar_cleanup seems to be doable and seem to give up to ~10% improvement. At the same time this is applicable to 10.2 as well and if it gives similar effect with 10.2, then we're in a problem again.

I didn't quite get suggestion re intern_plugin_lock.

IIRC this performance regression is generally due to session state tracker.

Comment by Sergey Vojtovich [ 2019-03-13 ]

Pushed removal of redundant plugin_thdvar_cleanup() to 10.2. Gives ~10% boost in 10.3 (when merged), still ~30% regression remaining.

Comment by Sergey Vojtovich [ 2019-03-19 ]

serg, please review 11 patches tagged with MDEV-14984 in bb-10.3-svoj-MDEV-14984.

Comment by Sergey Vojtovich [ 2019-03-20 ]

13 patches

Comment by Sergey Vojtovich [ 2019-04-01 ]

Awaiting for:
1. review of "Session_sysvars_tracker::session_track_system_variables", rev. 2f1c0ecb86d8727a2d53dcc93aa0f2d25d311cd3
2. review of group commit change, rev. 21fde6b551db705212b888efbb40a60bb2a796da
3. suggestions wrt THD_count::orig_thd

Comment by Sergei Golubchik [ 2019-05-02 ]

1 and 2 look good

Generated at Thu Feb 08 08:17:49 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.