[MCOL-5491] Enable StringStore in JSON_ARRAYAGG() function. Created: 2023-05-12  Updated: 2023-05-24  Resolved: 2023-05-24

Status: Closed
Project: MariaDB ColumnStore
Component/s: PrimProc
Affects Version/s: 23.02.3
Fix Version/s: 23.02.4

Type: Bug Priority: Major
Reporter: Gagan Goel (Inactive) Assignee: Gagan Goel (Inactive)
Resolution: Fixed Votes: 0
Labels: None

Issue Links:
Relates
relates to MCOL-5498 Potential memory leak in GROUP_CONCAT... Confirmed
Sprint: 2023-6
Assigned for Review: Roman Roman
Assigned for Testing: Daniel Lee Daniel Lee (Inactive)

 Description   

In MCOL-5385, we fixed a PrimProc crash scenario in GROUP_CONCAT() processing for queries that exceeded the available memory limits. This was done by enabling usage of StringStore for handling long strings. This ticket is to duplicate that change but for JSON_ARRAYAGG(), which has a similar implementation as GROUP_CONCAT.

 Comments   
Comment by Daniel Lee (Inactive) [ 2023-05-22 ]

Build tested: Develop branch

engine: a90535e1a7ffefa0e5ae808fdd0d38d30cffc017
server: 805750b3a90ed4aecbf475025e63674aaab7f7f7
buildNo: 7829

4gb VM memory

10g dbt3 database

MariaDB [tpch10]> select JSON_ARRAYAGG(l_comment) from lineitem;
ERROR 1815 (HY000): Internal error: InetStreamSocket::readToMagic: Remote is closed

crit.log

May 22 14:26:31 rocky8 ddlpackageproc[9686]: 31.412200 |0|0|0| C 23 CAL0008: DMLProc main process has started
 
May 22 14:26:31 rocky8 ddlpackageproc[9706]: 31.461581 |0|0|0| C 23 CAL0008: DDLProc main process has started
 
May 22 14:27:01 rocky8 controllernode[11832]: 01.549135 |0|0|0| C 29 CAL0000: InetStreamSocket::readToMagic(): I/O error1: rc-1; poll signal interrupt ( POLLHUP POLLERR )         %%10%%
 
May 22 14:27:01 rocky8 ddlpackageproc[11910]: 01.591131 |0|0|0| C 23 CAL0008: DMLProc main process has started
 
May 22 14:27:01 rocky8 controllernode[11832]: 01.607006 |0|0|0| C 29 CAL0000: InetStreamSocket::readToMagic(): I/O error1: rc-1; poll signal interrupt ( POLLHUP POLLERR )         %%10%%
 
May 22 14:27:01 rocky8 ddlpackageproc[11933]: 01.642599 |0|0|0| C 23 CAL0008: DDLProc main process has started
 
May 22 14:27:18 rocky8 controllernode[12160]: 18.685533 |0|0|0| C 29 CAL0000: InetStreamSocket::readToMagic(): I/O error1: rc-1; poll signal interrupt ( POLLHUP POLLERR )         %%10%%
 
May 22 14:27:18 rocky8 ddlpackageproc[12238]: 18.754231 |0|0|0| C 23 CAL0008: DMLProc main process has started
 
May 22 14:27:18 rocky8 controllernode[12160]: 18.786018 |0|0|0| C 29 CAL0000: InetStreamSocket::readToMagic(): I/O error1: rc-1; poll signal interrupt ( POLLHUP POLLERR )         %%10%%
 
May 22 14:27:18 rocky8 ddlpackageproc[12261]: 18.861306 |0|0|0| C 23 CAL0008: DDLProc main process has started

err.log

[rocky8:root@rocky8~]# cat err.log 
May 22 14:26:31 rocky8 ddlpackageproc[9686]: 31.412200 |0|0|0| C 23 CAL0008: DMLProc main process has started
 
May 22 14:26:31 rocky8 ddlpackageproc[9706]: 31.461581 |0|0|0| C 23 CAL0008: DDLProc main process has started
 
May 22 14:27:01 rocky8 controllernode[11832]: 01.549135 |0|0|0| C 29 CAL0000: InetStreamSocket::readToMagic(): I/O error1: rc-1; poll signal interrupt ( POLLHUP POLLERR )         %%10%%
 
May 22 14:27:01 rocky8 ddlpackageproc[11910]: 01.591131 |0|0|0| C 23 CAL0008: DMLProc main process has started
 
May 22 14:27:01 rocky8 controllernode[11832]: 01.607006 |0|0|0| C 29 CAL0000: InetStreamSocket::readToMagic(): I/O error1: rc-1; poll signal interrupt ( POLLHUP POLLERR )         %%10%%
 
May 22 14:27:01 rocky8 ddlpackageproc[11933]: 01.642599 |0|0|0| C 23 CAL0008: DDLProc main process has started
 
May 22 14:27:18 rocky8 controllernode[12160]: 18.685533 |0|0|0| C 29 CAL0000: InetStreamSocket::readToMagic(): I/O error1: rc-1; poll signal interrupt ( POLLHUP POLLERR )         %%10%%
 
May 22 14:27:18 rocky8 ddlpackageproc[12238]: 18.754231 |0|0|0| C 23 CAL0008: DMLProc main process has started
 
May 22 14:27:18 rocky8 controllernode[12160]: 18.786018 |0|0|0| C 29 CAL0000: InetStreamSocket::readToMagic(): I/O error1: rc-1; poll signal interrupt ( POLLHUP POLLERR )         %%10%%
 
May 22 14:27:18 rocky8 ddlpackageproc[12261]: 18.861306 |0|0|0| C 23 CAL0008: DDLProc main process has started
 
May 22 14:28:46 rocky8 ddlpackageproc[12263]: 46.932671 |0|0|0| E 23 CAL0001: Table does not exist in ColumnStore.
 
May 22 14:28:47 rocky8 ddlpackageproc[12263]: 47.195845 |0|0|0| E 23 CAL0001: Table does not exist in ColumnStore.
 
May 22 14:28:47 rocky8 ddlpackageproc[12263]: 47.367928 |0|0|0| E 23 CAL0001: Table does not exist in ColumnStore.
 
May 22 14:28:47 rocky8 ddlpackageproc[12263]: 47.581143 |0|0|0| E 23 CAL0001: Table does not exist in ColumnStore.
 
May 22 14:28:47 rocky8 ddlpackageproc[12263]: 47.779364 |0|0|0| E 23 CAL0001: Table does not exist in ColumnStore.
 
May 22 14:28:48 rocky8 ddlpackageproc[12263]: 48.025607 |0|0|0| E 23 CAL0001: Table does not exist in ColumnStore.
 
May 22 14:28:48 rocky8 ddlpackageproc[12263]: 48.165697 |0|0|0| E 23 CAL0001: Table does not exist in ColumnStore.
 
May 22 14:28:48 rocky8 ddlpackageproc[12263]: 48.307281 |0|0|0| E 23 CAL0001: Table does not exist in ColumnStore.

Release 23.02.3 ran successfully

With 8 gb VM memory, execution got further, but it seems to be not releasing memory. Execution takes longer
and longer and eventually crashed on the 4th run.

develop

1 row in set (3.741 sec)
 
1 row in set (5.948 sec)
 
1 row in set (8.132 sec)
 
ERROR 1815 (HY000): Internal error: InetStreamSocket::readToMagic: Remote is closed
 
 
 
Start to track free memory after the first crash.  Third run crashed
 
 
 
MiB Mem :   7763.3 total,   7372.1 free,    220.4 used,    170.9 buff/cache
 
MiB Mem :   7763.3 total,   2193.9 free,   5396.9 used,    172.5 buff/cache
 
MiB Mem :   7763.3 total,   1299.3 free,   6291.5 used,    172.5 buff/cache
 
MiB Mem :   7763.3 total,   7436.8 free,    173.7 used,    152.8 buff/cache

23.02.3

1 row in set (2.653 sec)
 
1 row in set (2.641 sec)
 
1 row in set (2.659 sec)
 
1 row in set (2.430 sec)

Comment by Daniel Lee (Inactive) [ 2023-05-24 ]

Enable StringStore for JSON_ARRAYAGG() has been done. Identified memory leak issue for both GROUP_CONCAT and JSON_ARRAYAGG() is being track by MCOL-5498.

Generated at Thu Feb 08 02:58:16 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.