[#MDEV-32484] Handlersocket Memory Leak in 10.11

[MDEV-32484] Handlersocket Memory Leak in 10.11 Created: 2023-10-17 Updated: 2023-10-24
Status:	Stalled
Project:	MariaDB Server
Component/s:	OTHER
Affects Version/s:	10.11.5
Fix Version/s:	10.11

Type:

Bug

Priority:

Major

Reporter:

Yutaro Iwasaki

Assignee:

Sergei Golubchik

Resolution:

Unresolved

Votes:

Labels:

None

Environment:

Rocky Linux 8.8

Attachments:

Test_CMD.txt

mariadb_log.txt

mariadb_mleak_log.txt

Description

After enable the HandlerSocket plugin, stopping the service causes a memory leak.
I have pasted the log when I tried enabling ASAN in the attached file.
mariadb_log.txt

2023-09-28 16:48:33 0 [Note] InnoDB: Buffer pool(s) dump completed at 230928 16:48:33

2023-09-28 16:48:33 0 [Note] InnoDB: Removed temporary tablespace data file: "./ibtmp1"

2023-09-28 16:48:33 0 [Note] InnoDB: Shutdown completed; log sequence number 96909540; transaction id 61897

handlersocket: terminated

2023-09-28 16:48:33 0 [Note] /usr/sbin/mariadbd: Shutdown complete

Warning: Memory not freed: 48

How to repeat:

1) Enable HandlerSocket Setting in my.cnf

[mysqld]

plugin_maturity=beta

loose_handlersocket_port = 9998

loose_handlersocket_port_wr = 9999

loose_handlersocket_threads = 16

loose_handlersocket_threads_wr = 1

open_files_limit = 65535

2) Install HandlerSocket Plugin

> install plugin handlersocket soname 'handlersocket.so';

3) Create DB data for testing

CREATE DATABASE testdb;

CREATE TABLE testdb.table1 (k int key, v char(20));

INSERT INTO testdb.table1 values (234,'foo'),(678,'bar');

4) Check operation with telnet
All are specified using TAB delimiters.
$ telnet localhost 9999

P	0	testdb	table1	PRIMARY	k,v

0	1

0	=	1	234

0	2	234	foo

0	+	2	789	hoge

0	1

0	=	1	789

0	2	789	hoge

^]

telnet> quit

Connection closed.

5) Stop MariaDB Service

Comments

Comment by Yutaro Iwasaki [ 2023-10-17 ]

I I've tried several versions.
I confirmed the issue with the following three versions I tried:

10.4: Warning: Memory not freed: is displayed
10.6: Warning: Memory not freed: is displayed
10.11: Warning: Memory not freed: is displayed

Comment by Marko Mäkelä [ 2023-10-17 ]

One thing that could be worth trying is a build with cmake -DPLUGIN_HANDLERSOCKET=STATIC. This should hopefully replace the <unknown module>> in mariadb_log.txt with properly resolved stack frames.

Comment by Anel Husakovic [ 2023-10-19 ]

Hi,
I couldn't reproduce the problem.
I have tried with:
1. Dynamic library (default) and static linking
2. For Debug, Release, RelWithDebug build types.

Procedure:
A) Configuration file .my104-hs.cnf

$ cat ~/.my104-hs.cnf

[mariadb]

datadir=/tmp/10.4

lc_messages_dir=/home/anel/GitHub/mariadb/server/build/10.4/sql/share

# Plugin is compiled statically

plugin_maturity=beta

loose_handlersocket_port = 9998

loose_handlersocket_port_wr = 9999

loose_handlersocket_threads = 16

loose_handlersocket_threads_wr = 1

open_files_limit = 65535

plugin_dir="/home/anel/GitHub/mariadb/server/build/10.4/plugin/handler_socket"

core_file

log_error

B) Make sure binary is compiled with WITH_ASAN and check library

$ ldd ./sql/mariadbd|grep asan

	libasan.so.5 => /lib/x86_64-linux-gnu/libasan.so.5 (0x00007fbf6472d000)

# or

# $ ASAN_OPTIONS=help=1 ./sql/mariadbd|less

C) Install system tables and start the server

$ rm -rf /tmp/10.4 && ./scripts/mariadb-install-db --srcdir=../../src/10.4 --defaults-file=~/.my104-hs.cnf

Installing MariaDB/MySQL system tables in '/tmp/10.4' ...

OK

C.1) When installing without loose_ in configuration file:

# this fails

$ rm -rf /tmp/10.4 && ./scripts/mariadb-install-db --srcdir=../../src/10.4 --defaults-file=~/.my104-hs.cnf

$ cat /tmp/10.4/anel.err

2023-10-19 12:12:17 0 [ERROR] ./sql/mysqld: unknown variable 'handlersocket_port=9998'

2023-10-19 12:12:17 0 [ERROR] Aborting

D) Start the server

Server start with ASAN
Show all
$ ASAN_OPTIONS="abort_on_error=1:fast_unwind_on_malloc=0" ./sql/mariadbd --defaults-file=~/.my104-hs.cnf
2023-10-19 10:24:56 0 [Note] Starting MariaDB 10.4.32-MariaDB-debug source revision 699cfee595b917ab05a99426bc6e9adbaa342e71 as process 102040
2023-10-19 10:24:56 0 [Note] InnoDB: !!!!!!!! UNIV_DEBUG switched on !!!!!!!!!
2023-10-19 10:24:56 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
2023-10-19 10:24:56 0 [Note] InnoDB: Uses event mutexes
2023-10-19 10:24:56 0 [Note] InnoDB: Compressed tables use zlib 1.2.11
2023-10-19 10:24:56 0 [Note] InnoDB: Number of pools: 1
2023-10-19 10:24:56 0 [Note] InnoDB: Using SSE2 crc32 instructions
2023-10-19 10:24:56 0 [Note] InnoDB: Initializing buffer pool, total size = 128M, instances = 1, chunk size = 128M
2023-10-19 10:24:56 0 [Note] InnoDB: Completed initialization of buffer pool
2023-10-19 10:24:56 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority().
2023-10-19 10:24:57 0 [Note] InnoDB: 128 out of 128 rollback segments are active.
2023-10-19 10:24:57 0 [Note] InnoDB: Creating shared tablespace for temporary tables
2023-10-19 10:24:57 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ...
2023-10-19 10:24:57 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB.
2023-10-19 10:24:57 0 [Note] InnoDB: Waiting for purge to start
2023-10-19 10:24:57 0 [Note] InnoDB: 10.4.32 started; log sequence number 61544; transaction id 20
2023-10-19 10:24:57 0 [Note] InnoDB: Loading buffer pool(s) from /tmp/10.4/ib_buffer_pool
2023-10-19 10:24:57 0 [Note] Plugin 'FEEDBACK' is disabled.
2023-10-19 10:24:57 0 [Note] InnoDB: Buffer pool(s) load completed at 231019 10:24:57
2023-10-19 10:24:57 0 [Warning] ./sql/mariadbd: unknown variable 'loose_handlersocket_port=9998'
2023-10-19 10:24:57 0 [Warning] ./sql/mariadbd: unknown variable 'loose_handlersocket_port_wr=9999'
2023-10-19 10:24:57 0 [Warning] ./sql/mariadbd: unknown variable 'loose_handlersocket_threads=16'
2023-10-19 10:24:57 0 [Warning] ./sql/mariadbd: unknown variable 'loose_handlersocket_threads_wr=1'
2023-10-19 10:24:57 0 [Note] Server socket created on IP: '::'.
2023-10-19 10:24:57 0 [Note] Reading of all Master_info entries succeeded
2023-10-19 10:24:57 0 [Note] Added new Master_info '' to hash table
2023-10-19 10:24:57 0 [Note] ./sql/mariadbd: ready for connections.
Version: '10.4.32-MariaDB-debug' socket: '/tmp/mysql.sock' port: 3306 Source distribution

E) Client

$ ./client/mysql -S /tmp/mysql.sock

MariaDB [(none)]> select @@plugin_dir; # plugin_dir is read-only

+--------------------------------------------------------------------+

| @@plugin_dir                                                       |

+--------------------------------------------------------------------+

| /home/anel/GitHub/mariadb/server/build/10.4/plugin/handler_socket/ |

+--------------------------------------------------------------------+

1 row in set (0.003 sec)

MariaDB [(none)]> install plugin handlersocket soname 'handlersocket.so';

Query OK, 0 rows affected (0.031 sec)

MariaDB [(none)]> select * from mysql.plugin;

+---------------+------------------+

| name          | dl               |

+---------------+------------------+

| handlersocket | handlersocket.so |

+---------------+------------------+

1 row in set (0.006 sec)

MariaDB [(none)]> CREATE DATABASE testdb;

MariaDB [(none)]> CREATE TABLE testdb.table1 (k int key, v char(20));

MariaDB [(none)]> INSERT INTO testdb.table1 values (234,'foo'),(678,'bar');

MariaDB [test]> SHOW GLOBAL VARIABLES LIKE 'core_file';

+---------------+-------+

| Variable_name | Value |

+---------------+-------+

| core_file     | ON    |

+---------------+-------+

1 row in set (0.003 sec)

F) Kill the server

$ kill -9 `pgrep -x mariadbd`

Default - dynamic library

$ cmake . -LAH|grep HANDLER

// How to build plugin HANDLERSOCKET. Options are: NO STATIC DYNAMIC YES AUTO.

PLUGIN_HANDLERSOCKET:STRING=DYNAMIC

Static library (not working)

$ cmake -LAH . |grep -E "HANDLERSOCKET|WITH_ASAN"

// How to build plugin HANDLERSOCKET. Options are: NO STATIC DYNAMIC YES AUTO.

PLUGIN_HANDLERSOCKET:STRING=STATIC

WITH_ASAN:BOOL=ON

WITH_ASAN_SCOPE:BOOL=OFF

WSREP_LIB_WITH_ASAN:BOOL=OFF

No archive created and no symbol found (only for test purposes from dynamic library)

$ nm -C plugin/handler_socket/handlersocket.so  # list of symbols like daemon_handlersocket_status_variables or daemon_handlersocket_init

$ nm -C sql/mariadbd |grep daemon_handlersocket_init

Core file in both cases doesn't generated memory leak

Core File
Show all
$ cat /tmp/10.4/anel.err
2023-10-19 12:15:54 0 [Note] Starting MariaDB 10.4.32-MariaDB-debug source revision 699cfee595b917ab05a99426bc6e9adbaa342e71 as process 167993
2023-10-19 12:15:54 0 [Note] InnoDB: !!!!!!!! UNIV_DEBUG switched on !!!!!!!!!
2023-10-19 12:15:54 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
2023-10-19 12:15:54 0 [Note] InnoDB: Uses event mutexes
2023-10-19 12:15:54 0 [Note] InnoDB: Compressed tables use zlib 1.2.11
2023-10-19 12:15:54 0 [Note] InnoDB: Number of pools: 1
2023-10-19 12:15:54 0 [Note] InnoDB: Using SSE2 crc32 instructions
2023-10-19 12:15:54 0 [Note] InnoDB: Initializing buffer pool, total size = 128M, instances = 1, chunk size = 128M
2023-10-19 12:15:54 0 [Note] InnoDB: Completed initialization of buffer pool
2023-10-19 12:15:54 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority().
2023-10-19 12:15:55 0 [Note] InnoDB: 128 out of 128 rollback segments are active.
2023-10-19 12:15:55 0 [Note] InnoDB: Creating shared tablespace for temporary tables
2023-10-19 12:15:55 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ...
2023-10-19 12:15:55 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB.
2023-10-19 12:15:55 0 [Note] InnoDB: Waiting for purge to start
2023-10-19 12:15:55 0 [Note] InnoDB: 10.4.32 started; log sequence number 61535; transaction id 20
2023-10-19 12:15:55 0 [Note] InnoDB: Loading buffer pool(s) from /tmp/10.4/ib_buffer_pool
2023-10-19 12:15:55 0 [Note] Plugin 'FEEDBACK' is disabled.
2023-10-19 12:15:55 0 [Warning] ./sql/mariadbd: unknown variable 'loose_handlersocket_port=9998'
2023-10-19 12:15:55 0 [Warning] ./sql/mariadbd: unknown variable 'loose_handlersocket_port_wr=9999'
2023-10-19 12:15:55 0 [Warning] ./sql/mariadbd: unknown variable 'loose_handlersocket_threads=16'
2023-10-19 12:15:55 0 [Warning] ./sql/mariadbd: unknown variable 'loose_handlersocket_threads_wr=1'
2023-10-19 12:15:55 0 [Note] InnoDB: Buffer pool(s) load completed at 231019 12:15:55
2023-10-19 12:15:55 0 [Note] Server socket created on IP: '::'.
2023-10-19 12:15:55 0 [Note] Reading of all Master_info entries succeeded
2023-10-19 12:15:55 0 [Note] Added new Master_info '' to hash table
2023-10-19 12:15:55 0 [Note] ./sql/mariadbd: ready for connections.
Version: '10.4.32-MariaDB-debug' socket: '/tmp/mysql.sock' port: 3306 Source distribution
2023-10-19 12:16:11 8 [Warning] Plugin 'handlersocket' is of maturity level beta while the server is stable
handlersocket: initialized
CONFIG: num_threads=16
CONFIG: nonblocking=1(default)
CONFIG: use_epoll=1
CONFIG: readsize=0
CONFIG: conn_per_thread=1024(default)
CONFIG: for_write=0(default)
CONFIG: plain_secret=(default)
CONFIG: timeout=300
CONFIG: listen_backlog=32768
CONFIG: host=(default)
CONFIG: port=9998
CONFIG: sndbuf=0
CONFIG: rcvbuf=0
CONFIG: stack_size=1048576(default)
CONFIG: wrlock_timeout=12
CONFIG: accept_balance=0
CONFIG: wrlock_timeout=12
CONFIG: accept_balance=0
CONFIG: wrlock_timeout=12
CONFIG: accept_balance=0
CONFIG: wrlock_timeout=12
CONFIG: accept_balance=0
CONFIG: wrlock_timeout=12
CONFIG: accept_balance=0
CONFIG: wrlock_timeout=12
CONFIG: accept_balance=0
CONFIG: wrlock_timeout=12
CONFIG: accept_balance=0
CONFIG: wrlock_timeout=12
CONFIG: accept_balance=0
CONFIG: wrlock_timeout=12
CONFIG: accept_balance=0
CONFIG: wrlock_timeout=12
CONFIG: accept_balance=0
CONFIG: wrlock_timeout=12
CONFIG: accept_balance=0
CONFIG: wrlock_timeout=12
CONFIG: accept_balance=0
CONFIG: wrlock_timeout=12
CONFIG: accept_balance=0
CONFIG: wrlock_timeout=12
CONFIG: accept_balance=0
CONFIG: wrlock_timeout=12
CONFIG: accept_balance=0
CONFIG: wrlock_timeout=12
CONFIG: accept_balance=0
CONFIG: num_threads=1
CONFIG: nonblocking=1(default)
CONFIG: use_epoll=1
CONFIG: readsize=0
CONFIG: conn_per_thread=1024(default)
CONFIG: for_write=1
CONFIG: plain_secret=
CONFIG: timeout=300
CONFIG: listen_backlog=32768
CONFIG: host=(default)
CONFIG: port=9999
CONFIG: sndbuf=0
CONFIG: rcvbuf=0
CONFIG: stack_size=1048576(default)
CONFIG: wrlock_timeout=12
CONFIG: accept_balance=0

So please let me know how properly to reproduce and which build type to use.
Regards,
Anel

Comment by Yutaro Iwasaki [ 2023-10-20 ]

Thank you for Test!!

Since there is a big difference in the environment when building, I will describe the steps to reproduce using the official repository.
There may be differences depending on the distribution, but could you please check whether logs are output using the following method?

# curl -LsS https://r.mariadb.com/downloads/mariadb_repo_setup | sudo bash -s -- --mariadb-server-version="mariadb-10.11"

# dnf install mariadb-server

# useradd mysql

# mkdir /var/lib/mysql

# mariadb-install-db --user=mysql --datadir=/var/lib/mysql

# vi /etc/my.cnf

[mysqld]

plugin_maturity=beta

loose_handlersocket_port = 9998

loose_handlersocket_port_wr = 9999

loose_handlersocket_threads = 16

loose_handlersocket_threads_wr = 1

open_files_limit = 65535

# systemctl start mariadb

# mysql -u root

Welcome to the MariaDB monitor.  Commands end with ; or \g.

Your MariaDB connection id is 3

Server version: 10.11.5-MariaDB MariaDB Server

Copyright (c) 2000, 2018, Oracle, MariaDB Corporation Ab and others.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

MariaDB [(none)]> install plugin handlersocket soname 'handlersocket.so';

Query OK, 0 rows affected (0.006 sec)

MariaDB [(none)]> CREATE DATABASE testdb;

Query OK, 1 row affected (0.000 sec)

MariaDB [(none)]> CREATE TABLE testdb.table1 (k int key, v char(20));

Query OK, 0 rows affected (0.009 sec)

MariaDB [(none)]> INSERT INTO testdb.table1 values (234,'foo'),(678,'bar');

Query OK, 2 rows affected (0.001 sec)

Records: 2  Duplicates: 0  Warnings: 0

MariaDB [(none)]> quit

Bye

# telnet localhost 9999

Trying ::1...

telnet: connect to address ::1: Connection refused

Trying 127.0.0.1...

Connected to localhost.

Escape character is '^]'.

P       0       testdb  table1  PRIMARY k,v

0       1

0       =       1       234

0       2       234     foo

^]

telnet> quit

Connection closed.

# systemctl stop mariadb

# less /var/log/messages

Oct 20 09:50:31 ip-172-31-46-220 mariadbd[15253]: 2023-10-20  9:50:31 0 [Note] /usr/sbin/mariadbd (initiated by: unknown): Normal shutdown

Oct 20 09:50:32 ip-172-31-46-220 mariadbd[15253]: 2023-10-20  9:50:32 0 [Note] InnoDB: FTS optimize thread exiting.

Oct 20 09:50:32 ip-172-31-46-220 mariadbd[15253]: 2023-10-20  9:50:32 0 [Note] InnoDB: Starting shutdown...

Oct 20 09:50:32 ip-172-31-46-220 mariadbd[15253]: 2023-10-20  9:50:32 0 [Note] InnoDB: Dumping buffer pool(s) to /var/lib/mysql/ib_buffer_pool

Oct 20 09:50:32 ip-172-31-46-220 mariadbd[15253]: 2023-10-20  9:50:32 0 [Note] InnoDB: Buffer pool(s) dump completed at 231020  9:50:32

Oct 20 09:50:32 ip-172-31-46-220 mariadbd[15253]: 2023-10-20  9:50:32 0 [Note] InnoDB: Removed temporary tablespace data file: "./ibtmp1"

Oct 20 09:50:32 ip-172-31-46-220 mariadbd[15253]: 2023-10-20  9:50:32 0 [Note] InnoDB: Shutdown completed; log sequence number 49886; transaction id 25

Oct 20 09:50:32 ip-172-31-46-220 mariadbd[15253]: handlersocket: terminated

Oct 20 09:50:32 ip-172-31-46-220 mariadbd[15253]: 2023-10-20  9:50:32 0 [Note] /usr/sbin/mariadbd: Shutdown complete

Oct 20 09:50:32 ip-172-31-46-220 mariadbd[15253]: Warning: Memory not freed: 48

Oct 20 09:50:32 ip-172-31-46-220 systemd[1]: mariadb.service: Succeeded.

Oct 20 09:50:32 ip-172-31-46-220 systemd[1]: Stopped MariaDB 10.11.5 database server.

"Warning: Memory not freed: 48" displayed

Comment by Anel Husakovic [ 2023-10-20 ]

I have found client memory leaks, not server's as in your case and server abortion too.
Seems there are:
1. problem with readline library and this plugin regarding the client mariadb.
2. uninstalling the plugin crashes the server.

Client memory leaks - install plugin and quit the client connection

Uninstall the plugin - bug

Show all
MariaDB [(none)]> uninstall plugin handlersocket;
ERROR 2013 (HY000): Lost connection to server during query

$ ./sql/mariadbd --defaults-file=~/.my1011-hs.cnf
2023-10-20 13:52:44 0 [Note] Starting MariaDB 10.11.6-MariaDB-debug source revision 22c869f65f143bbd0bb3811b4113fdb0f483eef0 as process 13744
2023-10-20 13:52:44 0 [Note] InnoDB: !!!!!!!! UNIV_DEBUG switched on !!!!!!!!!
2023-10-20 13:52:44 0 [Note] InnoDB: Compressed tables use zlib 1.2.11
2023-10-20 13:52:44 0 [Note] InnoDB: Number of transaction pools: 1
2023-10-20 13:52:44 0 [Note] InnoDB: Using crc32 + pclmulqdq instructions
2023-10-20 13:52:44 0 [Note] InnoDB: Using Linux native AIO
2023-10-20 13:52:44 0 [Note] InnoDB: Initializing buffer pool, total size = 128.000MiB, chunk size = 2.000MiB
2023-10-20 13:52:44 0 [Note] InnoDB: Completed initialization of buffer pool
2023-10-20 13:52:44 0 [Note] InnoDB: File system buffers for log disabled (block size=512 bytes)
2023-10-20 13:52:44 0 [Note] InnoDB: End of log at LSN=55942
2023-10-20 13:52:44 0 [Note] InnoDB: 128 rollback segments are active.
2023-10-20 13:52:44 0 [Note] InnoDB: Setting file './ibtmp1' size to 12.000MiB. Physically writing the file full; Please wait ...
2023-10-20 13:52:44 0 [Note] InnoDB: File './ibtmp1' size is now 12.000MiB.
2023-10-20 13:52:44 0 [Note] InnoDB: log sequence number 55942; transaction id 14
2023-10-20 13:52:44 0 [Note] InnoDB: Loading buffer pool(s) from /tmp/10.11/ib_buffer_pool
2023-10-20 13:52:44 0 [Note] Plugin 'FEEDBACK' is disabled.
2023-10-20 13:52:45 0 [Warning] ./sql/mariadbd: unknown variable 'loose_handlersocket_port=9998'
2023-10-20 13:52:45 0 [Warning] ./sql/mariadbd: unknown variable 'loose_handlersocket_port_wr=9999'
2023-10-20 13:52:45 0 [Warning] ./sql/mariadbd: unknown variable 'loose_handlersocket_threads=16'
2023-10-20 13:52:45 0 [Warning] ./sql/mariadbd: unknown variable 'loose_handlersocket_threads_wr=1'
2023-10-20 13:52:45 0 [Note] InnoDB: Buffer pool(s) load completed at 231020 13:52:45
2023-10-20 13:52:45 0 [Note] Server socket created on IP: '0.0.0.0'.
2023-10-20 13:52:45 0 [Note] Server socket created on IP: '::'.
2023-10-20 13:52:45 0 [Note] ./sql/mariadbd: ready for connections.
Version: '10.11.6-MariaDB-debug' socket: '/tmp/mysql.sock' port: 3306 Source distribution
2023-10-20 13:53:29 4 [Warning] Plugin 'handlersocket' is of maturity level beta while the server is stable
handlersocket: initialized
CONFIG: num_threads=16
CONFIG: nonblocking=1(default)
CONFIG: use_epoll=1
CONFIG: readsize=0
CONFIG: conn_per_thread=1024(default)
CONFIG: for_write=0(default)
CONFIG: plain_secret=(default)
CONFIG: timeout=300
CONFIG: listen_backlog=32768
CONFIG: host=(default)
CONFIG: port=9998
CONFIG: sndbuf=0
CONFIG: rcvbuf=0
CONFIG: stack_size=1048576(default)
CONFIG: wrlock_timeout=12
CONFIG: accept_balance=0
CONFIG: wrlock_timeout=12
CONFIG: accept_balance=0
CONFIG: wrlock_timeout=12
CONFIG: accept_balance=0
CONFIG: wrlock_timeout=12
CONFIG: accept_balance=0
CONFIG: wrlock_timeout=12
CONFIG: accept_balance=0
CONFIG: wrlock_timeout=12
CONFIG: accept_balance=0
CONFIG: wrlock_timeout=12
CONFIG: accept_balance=0
CONFIG: wrlock_timeout=12
CONFIG: accept_balance=0
CONFIG: wrlock_timeout=12
CONFIG: accept_balance=0
CONFIG: wrlock_timeout=12
CONFIG: accept_balance=0
CONFIG: wrlock_timeout=12
CONFIG: accept_balance=0
CONFIG: wrlock_timeout=12
CONFIG: accept_balance=0
CONFIG: wrlock_timeout=12
CONFIG: accept_balance=0
CONFIG: wrlock_timeout=12
CONFIG: accept_balance=0
CONFIG: wrlock_timeout=12
CONFIG: accept_balance=0
CONFIG: wrlock_timeout=12
CONFIG: accept_balance=0
CONFIG: num_threads=1
CONFIG: nonblocking=1(default)
CONFIG: use_epoll=1
CONFIG: readsize=0
CONFIG: conn_per_thread=1024(default)
CONFIG: for_write=1
CONFIG: plain_secret=
CONFIG: timeout=300
CONFIG: listen_backlog=32768
CONFIG: host=(default)
CONFIG: port=9999
CONFIG: sndbuf=0
CONFIG: rcvbuf=0
CONFIG: stack_size=1048576(default)
CONFIG: wrlock_timeout=12
CONFIG: accept_balance=0
handlersocket: terminated
mariadbd: /home/anel/GitHub/mariadb/server/src/10.11/sql/sql_list.h:705: void ilink::assert_not_linked(): Assertion `prev == 0 && next == 0' failed.
231020 13:55:23 [ERROR] mysqld got signal 6 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.

To report this bug, see https://mariadb.com/kb/en/reporting-bugs

We will try our best to scrape up some info that will hopefully help
diagnose the problem, but since we have already crashed,
something is definitely wrong and this may fail.

Server version: 10.11.6-MariaDB-debug source revision: 22c869f65f143bbd0bb3811b4113fdb0f483eef0
key_buffer_size=134217728
read_buffer_size=131072
max_used_connections=1
max_threads=153
thread_count=18
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 468162 K bytes of memory
Hope that's ok; if not, decrease some variables in the equation.

Thread pointer: 0x0
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 0x0 thread_stack 0x100000
sanitizer_common/sanitizer_common_interceptors.inc:4022(__interceptor_backtrace.part.0)[0x7fcda6078d40]
addr2line: './sql/mariadbd': No such file
Printing to addr2line failed
./sql/mariadbd(my_print_stacktrace+0xec)[0x55f017611d89]
./sql/mariadbd(handle_fatal_signal+0xa29)[0x55f0161d5788]
sigaction.c:0(__restore_rt)[0x7fcda5b98420]
addr2line: DWARF error: section .debug_info is larger than its filesize! (0x93ef57 vs 0x530ea0)
/lib/x86_64-linux-gnu/libc.so.6(gsignal+0xcb)[0x7fcda568100b]
/lib/x86_64-linux-gnu/libc.so.6(abort+0x12b)[0x7fcda5660859]
/lib/x86_64-linux-gnu/libc.so.6(+0x22729)[0x7fcda5660729]
/lib/x86_64-linux-gnu/libc.so.6(+0x33fd6)[0x7fcda5671fd6]
addr2line: './sql/mariadbd': No such file
./sql/mariadbd(_ZN5ilink17assert_not_linkedEv+0x140)[0x55f01564df42]
./sql/mariadbd(_ZN3THDD1Ev+0x28c)[0x55f01578f094]
./sql/mariadbd(_ZN3THDD0Ev+0x1c)[0x55f01578fd40]
handlersocket/database.cpp:341(dena::dbcontext::term_thread())[0x7fcd81daec4c]
handlersocket/hstcpsvr_worker.cpp:318(dena::(anonymous namespace)::thr_init::~thr_init())[0x7fcd81ddb5ce]
handlersocket/hstcpsvr_worker.cpp:347(dena::hstcpsvr_worker::run())[0x7fcd81ddba43]
handlersocket/hstcpsvr.cpp:31(dena::worker_throbj::operator()())[0x7fcd81df3b86]
libhsclient/thread.hpp:72(dena::thread<dena::worker_throbj>::thread_main(void*))[0x7fcd81df7073]
nptl/pthread_create.c:478(start_thread)[0x7fcda5b8c609]
addr2line: DWARF error: section .debug_info is larger than its filesize! (0x93ef57 vs 0x530ea0)
/lib/x86_64-linux-gnu/libc.so.6(clone+0x43)[0x7fcda575d133]
The manual page at https://mariadb.com/kb/en/how-to-produce-a-full-stack-trace-for-mysqld/ contains
information that should help you find out what is causing the crash.
Writing a core file...
Working directory at /tmp/10.11
Resource Limits:
Limit Soft Limit Hard Limit Units
Max cpu time unlimited unlimited seconds
Max file size unlimited unlimited bytes
Max data size unlimited unlimited bytes
Max stack size 8388608 unlimited bytes
Max core file size 0 0 bytes
Max resident set unlimited unlimited bytes
Max processes 127651 127651 processes
Max open files 65535 65535 files
Max locked memory 67108864 67108864 bytes
Max address space unlimited unlimited bytes
Max file locks unlimited unlimited locks
Max pending signals 127651 127651 signals
Max msgqueue size 819200 819200 bytes
Max nice priority 0 0
Max realtime priority 0 0
Max realtime timeout unlimited unlimited us
Core pattern: \|/usr/share/apport/apport -p%p -s%s -c%c -d%d -P%P -u%u -g%g -- %E

Kernel version: Linux version 5.15.0-87-generic (buildd@bos03-amd64-016) (gcc (Ubuntu 9.4.0-1ubuntu1~20.04.2) 9.4.0, GNU ld (GNU Binutils for Ubuntu) 2.34) #97~20.04.1-Ubuntu SMP Thu Oct 5 08:25:28 UTC 2023

Aborted (core dumped)

Quit client connection - client memory leaks with readline library

Show all
MariaDB [(none)]> uninstall plugin handlersocket;
ERROR 2013 (HY000): Lost connection to server during query
MariaDB [(none)]> select * from mysql.plugin;
ERROR 2006 (HY000): Server has gone away
No connection. Trying to reconnect...
ERROR 2002 (HY000): Can't connect to local server through socket '/tmp/mysql.sock' (111)
ERROR: Can't connect to the server

unknown [(none)]> select * from mysql.plugin;
No connection. Trying to reconnect...
ERROR 2002 (HY000): Can't connect to local server through socket '/tmp/mysql.sock' (111)
ERROR: Can't connect to the server

unknown [(none)]> exit5
-> Ctrl-C -- exit!
Aborted

=================================================================
==15349==ERROR: LeakSanitizer: detected memory leaks

Direct leak of 128 byte(s) in 1 object(s) allocated from:
#0 0x7f02d656fa06 in __interceptor_calloc ../../../../src/libsanitizer/asan/asan_malloc_linux.cc:153
#1 0x55691435fd2c in mysql_init /home/anel/GitHub/mariadb/server/src/10.11/libmariadb/libmariadb/mariadb_lib.c:1282
#2 0x556914349be3 in sql_real_connect /home/anel/GitHub/mariadb/server/src/10.11/client/mysql.cc:4856
#3 0x55691434a1b8 in sql_connect /home/anel/GitHub/mariadb/server/src/10.11/client/mysql.cc:4941
#4 0x556914347ce1 in com_connect /home/anel/GitHub/mariadb/server/src/10.11/client/mysql.cc:4520
#5 0x55691434025a in reconnect /home/anel/GitHub/mariadb/server/src/10.11/client/mysql.cc:3100
#6 0x556914340434 in mysql_real_query_for_lazy(char const*, unsigned long) /home/anel/GitHub/mariadb/server/src/10.11/client/mysql.cc:3148
#7 0x556914342013 in com_go /home/anel/GitHub/mariadb/server/src/10.11/client/mysql.cc:3421
#8 0x55691433d5ee in add_line /home/anel/GitHub/mariadb/server/src/10.11/client/mysql.cc:2574
#9 0x55691433b6db in read_and_execute /home/anel/GitHub/mariadb/server/src/10.11/client/mysql.cc:2269
#10 0x5569143391c3 in main /home/anel/GitHub/mariadb/server/src/10.11/client/mysql.cc:1373
#11 0x7f02d5afd082 in __libc_start_main ../csu/libc-start.c:308
#12 0x55691433734d in _start (/home/anel/GitHub/mariadb/server/build/10.11/client/mariadb+0xef34d)

Direct leak of 32 byte(s) in 1 object(s) allocated from:
#0 0x7f02d656f808 in __interceptor_malloc ../../../../src/libsanitizer/asan/asan_malloc_linux.cc:144
#1 0x7f02d644beac in xmalloc (/lib/x86_64-linux-gnu/libreadline.so.5+0x29eac)
#2 0x7f02d6446902 in rl_add_undo (/lib/x86_64-linux-gnu/libreadline.so.5+0x24902)
#3 0x7f02d64491f0 in rl_delete_text (/lib/x86_64-linux-gnu/libreadline.so.5+0x271f0)
#4 0x7f02d644a937 in _rl_rubout_char (/lib/x86_64-linux-gnu/libreadline.so.5+0x28937)
#5 0x7f02d6434455 in _rl_dispatch_subseq (/lib/x86_64-linux-gnu/libreadline.so.5+0x12455)
#6 0x7f02d64347c0 in readline_internal_char (/lib/x86_64-linux-gnu/libreadline.so.5+0x127c0)
#7 0x7f02d6434dc4 in readline (/lib/x86_64-linux-gnu/libreadline.so.5+0x12dc4)
#8 0x55691433b3d0 in read_and_execute /home/anel/GitHub/mariadb/server/src/10.11/client/mysql.cc:2230
#9 0x5569143391c3 in main /home/anel/GitHub/mariadb/server/src/10.11/client/mysql.cc:1373
#10 0x7f02d5afd082 in __libc_start_main ../csu/libc-start.c:308
#11 0x55691433734d in _start (/home/anel/GitHub/mariadb/server/build/10.11/client/mariadb+0xef34d)

Direct leak of 24 byte(s) in 1 object(s) allocated from:
#0 0x7f02d656fa06 in __interceptor_calloc ../../../../src/libsanitizer/asan/asan_malloc_linux.cc:153
#1 0x55691435fcd4 in mysql_init /home/anel/GitHub/mariadb/server/src/10.11/libmariadb/libmariadb/mariadb_lib.c:1280
#2 0x556914349be3 in sql_real_connect /home/anel/GitHub/mariadb/server/src/10.11/client/mysql.cc:4856
#3 0x55691434a1b8 in sql_connect /home/anel/GitHub/mariadb/server/src/10.11/client/mysql.cc:4941
#4 0x556914347ce1 in com_connect /home/anel/GitHub/mariadb/server/src/10.11/client/mysql.cc:4520
#5 0x55691434025a in reconnect /home/anel/GitHub/mariadb/server/src/10.11/client/mysql.cc:3100
#6 0x556914340434 in mysql_real_query_for_lazy(char const*, unsigned long) /home/anel/GitHub/mariadb/server/src/10.11/client/mysql.cc:3148
#7 0x556914342013 in com_go /home/anel/GitHub/mariadb/server/src/10.11/client/mysql.cc:3421
#8 0x55691433d5ee in add_line /home/anel/GitHub/mariadb/server/src/10.11/client/mysql.cc:2574
#9 0x55691433b6db in read_and_execute /home/anel/GitHub/mariadb/server/src/10.11/client/mysql.cc:2269
#10 0x5569143391c3 in main /home/anel/GitHub/mariadb/server/src/10.11/client/mysql.cc:1373
#11 0x7f02d5afd082 in __libc_start_main ../csu/libc-start.c:308
#12 0x55691433734d in _start (/home/anel/GitHub/mariadb/server/build/10.11/client/mariadb+0xef34d)

Indirect leak of 96 byte(s) in 3 object(s) allocated from:
#0 0x7f02d656f808 in __interceptor_malloc ../../../../src/libsanitizer/asan/asan_malloc_linux.cc:144
#1 0x7f02d644beac in xmalloc (/lib/x86_64-linux-gnu/libreadline.so.5+0x29eac)
#2 0x7f02d6446902 in rl_add_undo (/lib/x86_64-linux-gnu/libreadline.so.5+0x24902)
#3 0x7f02d64491f0 in rl_delete_text (/lib/x86_64-linux-gnu/libreadline.so.5+0x271f0)
#4 0x7f02d644a937 in _rl_rubout_char (/lib/x86_64-linux-gnu/libreadline.so.5+0x28937)
#5 0x7f02d6434455 in _rl_dispatch_subseq (/lib/x86_64-linux-gnu/libreadline.so.5+0x12455)
#6 0x7f02d64347c0 in readline_internal_char (/lib/x86_64-linux-gnu/libreadline.so.5+0x127c0)
#7 0x7f02d6434dc4 in readline (/lib/x86_64-linux-gnu/libreadline.so.5+0x12dc4)
#8 0x55691433b3d0 in read_and_execute /home/anel/GitHub/mariadb/server/src/10.11/client/mysql.cc:2230
#9 0x5569143391c3 in main /home/anel/GitHub/mariadb/server/src/10.11/client/mysql.cc:1373
#10 0x7f02d5afd082 in __libc_start_main ../csu/libc-start.c:308
#11 0x55691433734d in _start (/home/anel/GitHub/mariadb/server/build/10.11/client/mariadb+0xef34d)

Indirect leak of 32 byte(s) in 1 object(s) allocated from:
#0 0x7f02d656f808 in __interceptor_malloc ../../../../src/libsanitizer/asan/asan_malloc_linux.cc:144
#1 0x7f02d644beac in xmalloc (/lib/x86_64-linux-gnu/libreadline.so.5+0x29eac)
#2 0x7f02d6446902 in rl_add_undo (/lib/x86_64-linux-gnu/libreadline.so.5+0x24902)
#3 0x7f02d64490ec in rl_insert_text (/lib/x86_64-linux-gnu/libreadline.so.5+0x270ec)
#4 0x7f02d644a079 in _rl_insert_char (/lib/x86_64-linux-gnu/libreadline.so.5+0x28079)
#5 0x7f02d6434455 in _rl_dispatch_subseq (/lib/x86_64-linux-gnu/libreadline.so.5+0x12455)
#6 0x7f02d64347c0 in readline_internal_char (/lib/x86_64-linux-gnu/libreadline.so.5+0x127c0)
#7 0x7f02d6434dc4 in readline (/lib/x86_64-linux-gnu/libreadline.so.5+0x12dc4)
#8 0x55691433b3d0 in read_and_execute /home/anel/GitHub/mariadb/server/src/10.11/client/mysql.cc:2230
#9 0x5569143391c3 in main /home/anel/GitHub/mariadb/server/src/10.11/client/mysql.cc:1373
#10 0x7f02d5afd082 in __libc_start_main ../csu/libc-start.c:308
#11 0x55691433734d in _start (/home/anel/GitHub/mariadb/server/build/10.11/client/mariadb+0xef34d)

Indirect leak of 8 byte(s) in 4 object(s) allocated from:
#0 0x7f02d656f808 in __interceptor_malloc ../../../../src/libsanitizer/asan/asan_malloc_linux.cc:144
#1 0x7f02d644beac in xmalloc (/lib/x86_64-linux-gnu/libreadline.so.5+0x29eac)
#2 0x7f02d64458c5 in rl_copy_text (/lib/x86_64-linux-gnu/libreadline.so.5+0x238c5)
#3 0x7f02d6449199 in rl_delete_text (/lib/x86_64-linux-gnu/libreadline.so.5+0x27199)
#4 0x7f02d644a937 in _rl_rubout_char (/lib/x86_64-linux-gnu/libreadline.so.5+0x28937)
#5 0x7f02d6434455 in _rl_dispatch_subseq (/lib/x86_64-linux-gnu/libreadline.so.5+0x12455)
#6 0x7f02d64347c0 in readline_internal_char (/lib/x86_64-linux-gnu/libreadline.so.5+0x127c0)
#7 0x7f02d6434dc4 in readline (/lib/x86_64-linux-gnu/libreadline.so.5+0x12dc4)
#8 0x55691433b3d0 in read_and_execute /home/anel/GitHub/mariadb/server/src/10.11/client/mysql.cc:2230
#9 0x5569143391c3 in main /home/anel/GitHub/mariadb/server/src/10.11/client/mysql.cc:1373
#10 0x7f02d5afd082 in __libc_start_main ../csu/libc-start.c:308
#11 0x55691433734d in _start (/home/anel/GitHub/mariadb/server/build/10.11/client/mariadb+0xef34d)

SUMMARY: AddressSanitizer: 320 byte(s) leaked in 11 allocation(s).
Aborted (core dumped)

Load plugin and insert queries,quit the server

Show all
MariaDB [(none)]> # Added queries
MariaDB [(none)]> DROP DATABASE IF EXISTS testdb; CREATE DATABASE testdb; CREATE TABLE testdb.table1 (k int key, v char(20)); INSERT INTO testdb.table1 values (234,'foo'),(678,'bar');

$ kill -3 `pgrep -x mariadbd`

2023-10-20 14:02:32 0 [Note] ./sql/mariadbd: ready for connections.
Version: '10.11.6-MariaDB-debug' socket: '/tmp/mysql.sock' port: 3306 Source distribution
2023-10-20 14:03:44 0 [Note] ./sql/mariadbd (initiated by: unknown): Normal shutdown
mariadbd: /home/anel/GitHub/mariadb/server/src/10.11/sql/sql_list.h:705: void ilink::assert_not_linked(): Assertion `prev == 0 && next == 0' failed.
mariadbd: /home/anel/GitHub/mariadb/server/src/10.11/sql/sql_list.h:705: void ilink::assert_not_linked(): Assertion `prev == 0 && next == 0' failed.
231020 14:03:44 [ERROR] mysqld got signal 6 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.

To report this bug, see https://mariadb.com/kb/en/reporting-bugs

We will try our best to scrape up some info that will hopefully help
diagnose the problem, but since we have already crashed,
something is definitely wrong and this may fail.

Server version: 10.11.6-MariaDB-debug source revision: 22c869f65f143bbd0bb3811b4113fdb0f483eef0
key_buffer_size=134217728
read_buffer_size=131072
max_used_connections=1
max_threads=153
thread_count=17
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 468162 K bytes of memory
Hope that's ok; if not, decrease some variables in the equation.

Thread pointer: 0x0
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
Aborted (core dumped)

Quit the client

Show all
MariaDB [(none)]> Aborted

=================================================================
==16516==ERROR: LeakSanitizer: detected memory leaks

Direct leak of 421 byte(s) in 17 object(s) allocated from:
#0 0x7f61a79af808 in __interceptor_malloc ../../../../src/libsanitizer/asan/asan_malloc_linux.cc:144
#1 0x7f61a788beac in xmalloc (/lib/x86_64-linux-gnu/libreadline.so.5+0x29eac)
#2 0x7f61a7880c61 (/lib/x86_64-linux-gnu/libreadline.so.5+0x1ec61)
#3 0x7f61a7881ba7 in rl_message (/lib/x86_64-linux-gnu/libreadline.so.5+0x1fba7)
#4 0x7f61a787faeb (/lib/x86_64-linux-gnu/libreadline.so.5+0x1daeb)
#5 0x7f61a78804f0 in _rl_isearch_dispatch (/lib/x86_64-linux-gnu/libreadline.so.5+0x1e4f0)
#6 0x7f61a788081a (/lib/x86_64-linux-gnu/libreadline.so.5+0x1e81a)
#7 0x7f61a7874455 in _rl_dispatch_subseq (/lib/x86_64-linux-gnu/libreadline.so.5+0x12455)
#8 0x7f61a78747c0 in readline_internal_char (/lib/x86_64-linux-gnu/libreadline.so.5+0x127c0)
#9 0x7f61a7874dc4 in readline (/lib/x86_64-linux-gnu/libreadline.so.5+0x12dc4)
#10 0x5595d4b7d3d0 in read_and_execute /home/anel/GitHub/mariadb/server/src/10.11/client/mysql.cc:2230
#11 0x5595d4b7b1c3 in main /home/anel/GitHub/mariadb/server/src/10.11/client/mysql.cc:1373
#12 0x7f61a6f3d082 in __libc_start_main ../csu/libc-start.c:308
#13 0x5595d4b7934d in _start (/home/anel/GitHub/mariadb/server/build/10.11/client/mariadb+0xef34d)

Direct leak of 23 byte(s) in 1 object(s) allocated from:
#0 0x7f61a79af808 in __interceptor_malloc ../../../../src/libsanitizer/asan/asan_malloc_linux.cc:144
#1 0x7f61a788beac in xmalloc (/lib/x86_64-linux-gnu/libreadline.so.5+0x29eac)
#2 0x7f61a7880c61 (/lib/x86_64-linux-gnu/libreadline.so.5+0x1ec61)
#3 0x7f61a7881ba7 in rl_message (/lib/x86_64-linux-gnu/libreadline.so.5+0x1fba7)
#4 0x7f61a787faeb (/lib/x86_64-linux-gnu/libreadline.so.5+0x1daeb)
#5 0x7f61a78807f8 (/lib/x86_64-linux-gnu/libreadline.so.5+0x1e7f8)
#6 0x7f61a7874455 in _rl_dispatch_subseq (/lib/x86_64-linux-gnu/libreadline.so.5+0x12455)
#7 0x7f61a78747c0 in readline_internal_char (/lib/x86_64-linux-gnu/libreadline.so.5+0x127c0)
#8 0x7f61a7874dc4 in readline (/lib/x86_64-linux-gnu/libreadline.so.5+0x12dc4)
#9 0x5595d4b7d3d0 in read_and_execute /home/anel/GitHub/mariadb/server/src/10.11/client/mysql.cc:2230
#10 0x5595d4b7b1c3 in main /home/anel/GitHub/mariadb/server/src/10.11/client/mysql.cc:1373
#11 0x7f61a6f3d082 in __libc_start_main ../csu/libc-start.c:308
#12 0x5595d4b7934d in _start (/home/anel/GitHub/mariadb/server/build/10.11/client/mariadb+0xef34d)

SUMMARY: AddressSanitizer: 444 byte(s) leaked in 18 allocation(s).
Aborted (core dumped)

Comment by Yutaro Iwasaki [ 2023-10-24 ]

I have attached information about all the tested commands and the output results from the initialized server.
Test_CMD.txt
This is the procedure when testing on AWS.

mariadb_mleak_log.txt
The result is the output log.

The difference is that we have applied the patch provided by zulipchat.

Comment by Marko Mäkelä [ 2023-10-24 ]

iytarou, thank you, mariadb_mleak_log.txt looks very useful. Can you apply the following patch and check if any leaks remain?

diff --git a/plugin/handler_socket/handlersocket/hstcpsvr.cpp b/plugin/handler_socket/handlersocket/hstcpsvr.cpp

index 250ef2c7be5..9bfa42dfa86 100644

--- a/plugin/handler_socket/handlersocket/hstcpsvr.cpp

+++ b/plugin/handler_socket/handlersocket/hstcpsvr.cpp

@@ -133,6 +133,7 @@ hstcpsvr::stop_workers()

   vshared.shutdown = 1;

   for (size_t i = 0; i < threads.size(); ++i) {

     threads[i]->join();

+    delete threads[i];

   threads.clear();

If any leaks remain, please attach another report of the remaining memory leaks.

Comment by Yutaro Iwasaki [ 2023-10-24 ]

Thank you for your reply!
After applying the patch you provided, the following warning no longer appears when closing,
and the ASAN error no longer appears.
Warning: Memory not freed: is displayed

It seems that the problem has been resolved,
but I have confirmed that the error "heap-use-after-free" may be displayed
depending on the operation when starting mariadb for the first time after starting the OS.

An error has occured
1) reboot
2) export ASAN_OPTIONS=fast_unwind_on_malloc=0
3) systemctl start mariadb
4) systemctl stop mariadb

No error occurred
1) reboot
2) export ASAN_OPTIONS=fast_unwind_on_malloc=0
3) systemctl start mariadb
===Wait for 1 minute===
4) systemctl stop mariadb

Oct 24 06:54:33 ip-172-31-47-64 systemd[1]: Stopping MariaDB 10.11.6 database server...

Oct 24 06:54:33 ip-172-31-47-64 mariadbd[1520]: 2023-10-24  6:54:33 0 [Note] /usr/local/mysql/bin/mariadbd (initiated by: unknown): Normal shutdown

Oct 24 06:54:33 ip-172-31-47-64 mariadbd[1520]: =================================================================

Oct 24 06:54:33 ip-172-31-47-64 mariadbd[1520]: ==1520==ERROR: AddressSanitizer: heap-use-after-free on address 0x62b00000e220 at pc 0x000000a473cd bp 0x7fccef1e28d0 sp 0x7fccef1e28c0

Oct 24 06:54:33 ip-172-31-47-64 mariadbd[1520]: WRITE of size 8 at 0x62b00000e220 thread T10

Oct 24 06:54:34 ip-172-31-47-64 mariadbd[1520]:    #0 0xa473cc in ilink::unlink() /root/server/sql/sql_list.h:696

Oct 24 06:54:34 ip-172-31-47-64 mariadbd[1520]:    #1 0xa473cc in ilink::~ilink() /root/server/sql/sql_list.h:707

Oct 24 06:54:34 ip-172-31-47-64 mariadbd[1520]:    #2 0xa473cc in Statement::~Statement() /root/server/sql/sql_class.h:1341

Oct 24 06:54:34 ip-172-31-47-64 mariadbd[1520]:    #3 0xa7699a in THD::~THD() /root/server/sql/sql_class.cc:1663

Oct 24 06:54:34 ip-172-31-47-64 mariadbd[1520]:    #4 0xa76f10 in THD::~THD() /root/server/sql/sql_class.cc:1756

Oct 24 06:54:34 ip-172-31-47-64 mariadbd[1520]:    #5 0x7fccf022589c in dena::dbcontext::term_thread() /root/server/plugin/handler_socket/handlersocket/database.cpp:340

Oct 24 06:54:34 ip-172-31-47-64 mariadbd[1520]:    #6 0x7fccf025483c in ~thr_init /root/server/plugin/handler_socket/handlersocket/hstcpsvr_worker.cpp:317

Oct 24 06:54:34 ip-172-31-47-64 mariadbd[1520]:    #7 0x7fccf025483c in dena::hstcpsvr_worker::run() /root/server/plugin/handler_socket/handlersocket/hstcpsvr_worker.cpp:327

Oct 24 06:54:34 ip-172-31-47-64 mariadbd[1520]:    #8 0x7fccf025d5d0 in dena::worker_throbj::operator()() /root/server/plugin/handler_socket/handlersocket/hstcpsvr.cpp:30

Oct 24 06:54:34 ip-172-31-47-64 mariadbd[1520]:    #9 0x7fccf025d5d0 in dena::thread<dena::worker_throbj>::thread_main(void*) /root/server/plugin/handler_socket/libhsclient/thread.hpp:71

Oct 24 06:54:34 ip-172-31-47-64 mariadbd[1520]:    #10 0x7fcd132e81c9 in start_thread (/lib64/libpthread.so.0+0x81c9)

Oct 24 06:54:34 ip-172-31-47-64 mariadbd[1520]:    #11 0x7fcd12625e72 in __clone (/lib64/libc.so.6+0x39e72)

Oct 24 06:54:34 ip-172-31-47-64 mariadbd[1520]: 0x62b00000e220 is located 32 bytes inside of 27352-byte region [0x62b00000e200,0x62b000014cd8)

Oct 24 06:54:34 ip-172-31-47-64 mariadbd[1520]: freed by thread T8 here:

Oct 24 06:54:34 ip-172-31-47-64 mariadbd[1520]:    #0 0x7fcd14b857f0 in __interceptor_free (/lib64/libasan.so.5+0xef7f0)

Oct 24 06:54:34 ip-172-31-47-64 mariadbd[1520]:    #1 0x7fccf022589c in dena::dbcontext::term_thread() /root/server/plugin/handler_socket/handlersocket/database.cpp:340

Oct 24 06:54:34 ip-172-31-47-64 mariadbd[1520]:    #2 0x7fccf025483c in ~thr_init /root/server/plugin/handler_socket/handlersocket/hstcpsvr_worker.cpp:317

Oct 24 06:54:34 ip-172-31-47-64 mariadbd[1520]:    #3 0x7fccf025483c in dena::hstcpsvr_worker::run() /root/server/plugin/handler_socket/handlersocket/hstcpsvr_worker.cpp:327

Oct 24 06:54:34 ip-172-31-47-64 mariadbd[1520]:    #4 0x7fccf025d5d0 in dena::worker_throbj::operator()() /root/server/plugin/handler_socket/handlersocket/hstcpsvr.cpp:30

Oct 24 06:54:34 ip-172-31-47-64 mariadbd[1520]:    #5 0x7fccf025d5d0 in dena::thread<dena::worker_throbj>::thread_main(void*) /root/server/plugin/handler_socket/libhsclient/thread.hpp:71

Oct 24 06:54:34 ip-172-31-47-64 mariadbd[1520]:    #6 0x7fcd132e81c9 in start_thread (/lib64/libpthread.so.0+0x81c9)

Oct 24 06:54:34 ip-172-31-47-64 mariadbd[1520]: previously allocated by thread T8 here:

Oct 24 06:54:34 ip-172-31-47-64 mariadbd[1520]:    #0 0x7fcd14b85bb8 in __interceptor_malloc (/lib64/libasan.so.5+0xefbb8)

Oct 24 06:54:34 ip-172-31-47-64 mariadbd[1520]:    #1 0x2323d44 in my_malloc /root/server/mysys/my_malloc.c:92

Oct 24 06:54:34 ip-172-31-47-64 mariadbd[1520]:    #2 0x7fccf02282e5 in ilink::operator new(unsigned long) /root/server/sql/sql_list.h:680

Oct 24 06:54:34 ip-172-31-47-64 mariadbd[1520]:    #3 0x7fccf02282e5 in dena::dbcontext::init_thread(void const*, int volatile&) /root/server/plugin/handler_socket/handlersocket/database.cpp:274

Oct 24 06:54:34 ip-172-31-47-64 mariadbd[1520]:    #4 0x7fccf02546d4 in thr_init /root/server/plugin/handler_socket/handlersocket/hstcpsvr_worker.cpp:314

Oct 24 06:54:34 ip-172-31-47-64 mariadbd[1520]:    #5 0x7fccf02546d4 in dena::hstcpsvr_worker::run() /root/server/plugin/handler_socket/handlersocket/hstcpsvr_worker.cpp:327

Oct 24 06:54:34 ip-172-31-47-64 mariadbd[1520]:    #6 0x7fccf025d5d0 in dena::worker_throbj::operator()() /root/server/plugin/handler_socket/handlersocket/hstcpsvr.cpp:30

Oct 24 06:54:34 ip-172-31-47-64 mariadbd[1520]:    #7 0x7fccf025d5d0 in dena::thread<dena::worker_throbj>::thread_main(void*) /root/server/plugin/handler_socket/libhsclient/thread.hpp:71

Oct 24 06:54:34 ip-172-31-47-64 mariadbd[1520]:    #8 0x7fcd132e81c9 in start_thread (/lib64/libpthread.so.0+0x81c9)

Oct 24 06:54:34 ip-172-31-47-64 mariadbd[1520]: Thread T10 created by T0 here:

Oct 24 06:54:34 ip-172-31-47-64 mariadbd[1520]:    #0 0x7fcd14ae8eb3 in __interceptor_pthread_create (/lib64/libasan.so.5+0x52eb3)

Oct 24 06:54:34 ip-172-31-47-64 mariadbd[1520]:    #1 0x7fccf0259810 in dena::thread<dena::worker_throbj>::start_nothrow() /root/server/plugin/handler_socket/libhsclient/thread.hpp:46

Oct 24 06:54:34 ip-172-31-47-64 mariadbd[1520]:    #2 0x7fccf0259810 in dena::thread<dena::worker_throbj>::start() /root/server/plugin/handler_socket/libhsclient/thread.hpp:30

Oct 24 06:54:34 ip-172-31-47-64 mariadbd[1520]:    #3 0x7fccf0259810 in dena::hstcpsvr::start_listen[abi:cxx11]() /root/server/plugin/handler_socket/handlersocket/hstcpsvr.cpp:124

Oct 24 06:54:34 ip-172-31-47-64 mariadbd[1520]:    #4 0x7fccf023e108 in daemon_handlersocket_init /root/server/plugin/handler_socket/handlersocket/handlersocket.cpp:86

Oct 24 06:54:34 ip-172-31-47-64 mariadbd[1520]:    #5 0xbd9a10 in plugin_do_initialize /root/server/sql/sql_plugin.cc:1451

Oct 24 06:54:34 ip-172-31-47-64 mariadbd[1520]:    #6 0xbe8c86 in plugin_initialize /root/server/sql/sql_plugin.cc:1504

Oct 24 06:54:34 ip-172-31-47-64 mariadbd[1520]:    #7 0xbeb9a0 in plugin_init(int*, char**, int) /root/server/sql/sql_plugin.cc:1763

Oct 24 06:54:34 ip-172-31-47-64 mariadbd[1520]:    #8 0x871744 in init_server_components /root/server/sql/mysqld.cc:5224

Oct 24 06:54:34 ip-172-31-47-64 mariadbd[1520]:    #9 0x880ce9 in mysqld_main(int, char**) /root/server/sql/mysqld.cc:5842

Oct 24 06:54:34 ip-172-31-47-64 mariadbd[1520]:    #10 0x7fcd12626d84 in __libc_start_main (/lib64/libc.so.6+0x3ad84)

Oct 24 06:54:34 ip-172-31-47-64 mariadbd[1520]: Thread T8 created by T0 here:

Oct 24 06:54:34 ip-172-31-47-64 mariadbd[1520]:    #0 0x7fcd14ae8eb3 in __interceptor_pthread_create (/lib64/libasan.so.5+0x52eb3)

Oct 24 06:54:34 ip-172-31-47-64 mariadbd[1520]:    #1 0x7fccf0259810 in dena::thread<dena::worker_throbj>::start_nothrow() /root/server/plugin/handler_socket/libhsclient/thread.hpp:46

Oct 24 06:54:34 ip-172-31-47-64 mariadbd[1520]:    #2 0x7fccf0259810 in dena::thread<dena::worker_throbj>::start() /root/server/plugin/handler_socket/libhsclient/thread.hpp:30

Oct 24 06:54:34 ip-172-31-47-64 mariadbd[1520]:    #3 0x7fccf0259810 in dena::hstcpsvr::start_listen[abi:cxx11]() /root/server/plugin/handler_socket/handlersocket/hstcpsvr.cpp:124

Oct 24 06:54:34 ip-172-31-47-64 mariadbd[1520]:    #4 0x7fccf023e108 in daemon_handlersocket_init /root/server/plugin/handler_socket/handlersocket/handlersocket.cpp:86

Oct 24 06:54:34 ip-172-31-47-64 mariadbd[1520]:    #5 0xbd9a10 in plugin_do_initialize /root/server/sql/sql_plugin.cc:1451

Oct 24 06:54:34 ip-172-31-47-64 mariadbd[1520]:    #6 0xbe8c86 in plugin_initialize /root/server/sql/sql_plugin.cc:1504

Oct 24 06:54:34 ip-172-31-47-64 mariadbd[1520]:    #7 0xbeb9a0 in plugin_init(int*, char**, int) /root/server/sql/sql_plugin.cc:1763

Oct 24 06:54:34 ip-172-31-47-64 mariadbd[1520]:    #8 0x871744 in init_server_components /root/server/sql/mysqld.cc:5224

Oct 24 06:54:34 ip-172-31-47-64 mariadbd[1520]:    #9 0x880ce9 in mysqld_main(int, char**) /root/server/sql/mysqld.cc:5842

Oct 24 06:54:34 ip-172-31-47-64 mariadbd[1520]:    #10 0x7fcd12626d84 in __libc_start_main (/lib64/libc.so.6+0x3ad84)

Oct 24 06:54:34 ip-172-31-47-64 mariadbd[1520]: SUMMARY: AddressSanitizer: heap-use-after-free /root/server/sql/sql_list.h:696 in ilink::unlink()

Oct 24 06:54:34 ip-172-31-47-64 mariadbd[1520]: Shadow bytes around the buggy address:

Oct 24 06:54:34 ip-172-31-47-64 mariadbd[1520]:  0x0c567fff9bf0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa

Oct 24 06:54:34 ip-172-31-47-64 mariadbd[1520]:  0x0c567fff9c00: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa

Oct 24 06:54:34 ip-172-31-47-64 mariadbd[1520]:  0x0c567fff9c10: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa

Oct 24 06:54:34 ip-172-31-47-64 mariadbd[1520]:  0x0c567fff9c20: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa

Oct 24 06:54:34 ip-172-31-47-64 mariadbd[1520]:  0x0c567fff9c30: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa

Oct 24 06:54:34 ip-172-31-47-64 mariadbd[1520]: =>0x0c567fff9c40: fd fd fd fd[fd]fd fd fd fd fd fd fd fd fd fd fd

Oct 24 06:54:34 ip-172-31-47-64 mariadbd[1520]:  0x0c567fff9c50: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd

Oct 24 06:54:34 ip-172-31-47-64 mariadbd[1520]:  0x0c567fff9c60: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd

Oct 24 06:54:34 ip-172-31-47-64 mariadbd[1520]:  0x0c567fff9c70: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd

Oct 24 06:54:34 ip-172-31-47-64 mariadbd[1520]:  0x0c567fff9c80: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd

Oct 24 06:54:34 ip-172-31-47-64 mariadbd[1520]:  0x0c567fff9c90: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd

Oct 24 06:54:34 ip-172-31-47-64 mariadbd[1520]: Shadow byte legend (one shadow byte represents 8 application bytes):

Oct 24 06:54:34 ip-172-31-47-64 mariadbd[1520]:  Addressable:           00

Oct 24 06:54:34 ip-172-31-47-64 mariadbd[1520]:  Partially addressable: 01 02 03 04 05 06 07

Oct 24 06:54:34 ip-172-31-47-64 mariadbd[1520]:  Heap left redzone:       fa

Oct 24 06:54:34 ip-172-31-47-64 mariadbd[1520]:  Freed heap region:       fd

Oct 24 06:54:34 ip-172-31-47-64 mariadbd[1520]:  Stack left redzone:      f1

Oct 24 06:54:34 ip-172-31-47-64 mariadbd[1520]:  Stack mid redzone:       f2

Oct 24 06:54:34 ip-172-31-47-64 mariadbd[1520]:  Stack right redzone:     f3

Oct 24 06:54:34 ip-172-31-47-64 mariadbd[1520]:  Stack after return:      f5

Oct 24 06:54:34 ip-172-31-47-64 mariadbd[1520]:  Stack use after scope:   f8

Oct 24 06:54:34 ip-172-31-47-64 mariadbd[1520]:  Global redzone:          f9

Oct 24 06:54:34 ip-172-31-47-64 mariadbd[1520]:  Global init order:       f6

Oct 24 06:54:34 ip-172-31-47-64 mariadbd[1520]:  Poisoned by user:        f7

Oct 24 06:54:34 ip-172-31-47-64 mariadbd[1520]:  Container overflow:      fc

Oct 24 06:54:34 ip-172-31-47-64 mariadbd[1520]:  Array cookie:            ac

Oct 24 06:54:34 ip-172-31-47-64 mariadbd[1520]:  Intra object redzone:    bb

Oct 24 06:54:34 ip-172-31-47-64 mariadbd[1520]:  ASan internal:           fe

Oct 24 06:54:34 ip-172-31-47-64 mariadbd[1520]:  Left alloca redzone:     ca

Oct 24 06:54:34 ip-172-31-47-64 mariadbd[1520]:  Right alloca redzone:    cb

Oct 24 06:54:34 ip-172-31-47-64 mariadbd[1520]: ==1520==ABORTING

Generated at Thu Feb 08 10:31:42 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.

[MDEV-32484] Handlersocket Memory Leak in 10.11 Created: 2023-10-17 Updated: 2023-10-24