[MDEV-26712] row events never reset thd->mem_root - Jira

Details

Type: Bug
Status: Closed (View Workflow)
Priority: Blocker
Resolution: Fixed
Affects Version/s: 10.2(EOL), 10.3(EOL), 10.4(EOL), 10.5, 10.6
Fix Version/s: 10.2.41, 10.3.32, 10.4.22, 10.5.13, 10.6.5
Component/s: Replication
Labels:
- Memory_leak
- not-10.7

Description

row event execution starts from Rows_log_event::do_apply_event which calls open_and_lock_tables() which calls lock_tables(), which does

if (!(ptr=start=(TABLE**) thd->alloc(sizeof(TABLE*)*count)))

thus allocating an array of points of the thd's memroot. This is just few bytes, 8 in the test below. But this memory is never freed, not by row events at least. So, by creating a transaction with a huge number of row events we can consume any amount of memory.

source include/have_innodb.inc;

source include/have_binlog_format_row.inc;

source include/master-slave.inc;

create table t1 (a int) engine=innodb;

insert t1 values (1);

sync_slave_with_master;

stop slave;

set global max_session_mem_used=10*1024*1024;

start slave;

connection master;

let $a=1000000;

start transaction;

disable_query_log;

while ($a) {

  update t1 set a=a+1;

  dec $a;

enable_query_log;

commit;

drop table t1;

sync_slave_with_master;

set global max_session_mem_used=default;

source include/rpl_end.inc;

To fix that, rbr events must do

free_root(thd->mem_root, MYF(MY_KEEP_PREALLOC));

somewhere. Where it's safe, because it'll destroy anything allocated on the thd memroot.
This patch appears to be working fine, it frees thd memroot after closing all opened tables:

--- a/sql/rpl_rli.cc

+++ b/sql/rpl_rli.cc

@@ -2326,6 +2326,7 @@ void rpl_group_info::cleanup_context(THD *thd, bool error)

*/

   reset_row_stmt_start_timestamp();

   unset_long_find_row_note_printed();

+  free_root(thd->mem_root, MYF(MY_KEEP_PREALLOC));

   DBUG_EXECUTE_IF("inject_sleep_gtid_100_x_x", {

       if (current_gtid.domain_id == 100)

Attachments

Issue Links

relates to

MDEV-20455 memory leak in 10.4 series

Closed

mentioned in: Page Failed to load

Activity

Ascending order - Click to sort in descending order

Sergei Golubchik created issue - 2021-09-28 19:34

Sergei Golubchik made changes - 2021-09-28 19:35

Field	Original Value	New Value
Description	_I didn't verify in other versions, please, check_ row event execution starts from {{Rows_log_event::do_apply_event}} which calls {{open_and_lock_tables()}} which calls {{lock_tables()}}, which does {code:c++} if (!(ptr=start=(TABLE*) thd->alloc(sizeof(TABLE)count))) {code} thus allocating an array of points of the thd's memroot. This is just few bytes, 8 in the test below. But this memory is never freed, not by row events at least. So, by creating a transaction with a huge number of row events we can consume any amount of memory. {code:sql} source include/have_innodb.inc; source include/have_binlog_format_row.inc; source include/master-slave.inc; create table t1 (a int) engine=innodb; insert t1 values (1); sync_slave_with_master; stop slave; set global max_session_mem_used=1010241024; start slave; connection master; let $a=1000000; start transaction; disable_query_log; while ($a) { update t1 set a=a+1; dec $a; } enable_query_log; commit; drop table t1; sync_slave_with_master; set global max_session_mem_used=default; source include/rpl_end.inc; {code} To fix that, rbr events must* do {code:c++} free_root(thd->mem_root, MYF(MY_KEEP_PREALLOC)); {code} somewhere. Where it's safe, because it'll destroy anything allocated on the thd memroot. This patch appears to be working fine, it frees thd memroot after closing all opened tables: {code:c++} --- a/sql/rpl_rli.cc +++ b/sql/rpl_rli.cc @@ -2326,6 +2326,7 @@ void rpl_group_info::cleanup_context(THD thd, bool error) / reset_row_stmt_start_timestamp(); unset_long_find_row_note_printed(); + free_root(thd->mem_root, MYF(MY_KEEP_PREALLOC)); DBUG_EXECUTE_IF("inject_sleep_gtid_100_x_x", { if (current_gtid.domain_id == 100) {code}	_I didn't try other versions, please, check_ row event execution starts from {{Rows_log_event::do_apply_event}} which calls {{open_and_lock_tables()}} which calls {{lock_tables()}}, which does {code:c++} if (!(ptr=start=(TABLE*) thd->alloc(sizeof(TABLE)count))) {code} thus allocating an array of points of the thd's memroot. This is just few bytes, 8 in the test below. But this memory is never freed, not by row events at least. So, by creating a transaction with a huge number of row events we can consume any amount of memory. {code:sql} source include/have_innodb.inc; source include/have_binlog_format_row.inc; source include/master-slave.inc; create table t1 (a int) engine=innodb; insert t1 values (1); sync_slave_with_master; stop slave; set global max_session_mem_used=1010241024; start slave; connection master; let $a=1000000; start transaction; disable_query_log; while ($a) { update t1 set a=a+1; dec $a; } enable_query_log; commit; drop table t1; sync_slave_with_master; set global max_session_mem_used=default; source include/rpl_end.inc; {code} To fix that, rbr events must* do {code:c++} free_root(thd->mem_root, MYF(MY_KEEP_PREALLOC)); {code} somewhere. Where it's safe, because it'll destroy anything allocated on the thd memroot. This patch appears to be working fine, it frees thd memroot after closing all opened tables: {code:c++} --- a/sql/rpl_rli.cc +++ b/sql/rpl_rli.cc @@ -2326,6 +2326,7 @@ void rpl_group_info::cleanup_context(THD thd, bool error) / reset_row_stmt_start_timestamp(); unset_long_find_row_note_printed(); + free_root(thd->mem_root, MYF(MY_KEEP_PREALLOC)); DBUG_EXECUTE_IF("inject_sleep_gtid_100_x_x", { if (current_gtid.domain_id == 100) {code}

Sergei Golubchik made changes - 2021-09-28 19:36

Link

This issue blocks MENT-1315 [ MENT-1315 ]

Sergei Golubchik made changes - 2021-09-29 07:44

Status

Open [ 1 ]

Confirmed [ 10101 ]

Roel Van de Paar made changes - 2021-09-29 10:34

Affects Version/s

10.4 [ 22408 ]

Roel Van de Paar made changes - 2021-09-29 10:34

Fix Version/s

10.4 [ 22408 ]

Roel Van de Paar made changes - 2021-09-29 10:36

Link

This issue relates to ~~MDEV-20455~~ [ ~~MDEV-20455~~ ]

Roel Van de Paar made changes - 2021-09-29 10:57

Affects Version/s		10.2 [ 14601 ]
Affects Version/s		10.3 [ 22126 ]
Affects Version/s		10.6 [ 24028 ]

Roel Van de Paar made changes - 2021-09-29 10:58

Fix Version/s		10.2 [ 14601 ]
Fix Version/s		10.3 [ 22126 ]
Fix Version/s		10.6 [ 24028 ]

Roel Van de Paar made changes - 2021-09-29 10:58

Labels

not-10.7

Roel Van de Paar made changes - 2021-09-29 10:58

Labels

not-10.7

Memory_leak not-10.7

Roel Van de Paar made changes - 2021-09-29 10:58

Description

_I didn't try other versions, please, check_

row event execution starts from {{Rows_log_event::do_apply_event}} which calls {{open_and_lock_tables()}} which calls {{lock_tables()}}, which does
{code:c++}
if (!(ptr=start=(TABLE**) thd->alloc(sizeof(TABLE*)*count)))
{code}
thus allocating an array of points of the thd's memroot. This is just few bytes, 8 in the test below. But this memory is never freed, not by row events at least. So, by creating a transaction with a huge number of row events we can consume any amount of memory.
{code:sql}
source include/have_innodb.inc;
source include/have_binlog_format_row.inc;
source include/master-slave.inc;
create table t1 (a int) engine=innodb;
insert t1 values (1);

sync_slave_with_master;
stop slave;
set global max_session_mem_used=10*1024*1024;
start slave;

connection master;
let $a=1000000;
start transaction;
disable_query_log;
while ($a) {
  update t1 set a=a+1;
  dec $a;
}
enable_query_log;
commit;
drop table t1;
sync_slave_with_master;
set global max_session_mem_used=default;
source include/rpl_end.inc;
{code}

To fix that, rbr events *must* do
{code:c++}
free_root(thd->mem_root, MYF(MY_KEEP_PREALLOC));
{code}
somewhere. Where it's safe, because it'll destroy anything allocated on the thd memroot.
This patch appears to be working fine, it frees thd memroot after closing all opened tables:
{code:c++}
--- a/sql/rpl_rli.cc
+++ b/sql/rpl_rli.cc
@@ -2326,6 +2326,7 @@ void rpl_group_info::cleanup_context(THD *thd, bool error)
   */
   reset_row_stmt_start_timestamp();
   unset_long_find_row_note_printed();
+ free_root(thd->mem_root, MYF(MY_KEEP_PREALLOC));

   DBUG_EXECUTE_IF("inject_sleep_gtid_100_x_x", {
       if (current_gtid.domain_id == 100)
{code}

row event execution starts from {{Rows_log_event::do_apply_event}} which calls {{open_and_lock_tables()}} which calls {{lock_tables()}}, which does
{code:c++}
if (!(ptr=start=(TABLE**) thd->alloc(sizeof(TABLE*)*count)))
{code}
thus allocating an array of points of the thd's memroot. This is just few bytes, 8 in the test below. But this memory is never freed, not by row events at least. So, by creating a transaction with a huge number of row events we can consume any amount of memory.
{code:sql}
source include/have_innodb.inc;
source include/have_binlog_format_row.inc;
source include/master-slave.inc;
create table t1 (a int) engine=innodb;
insert t1 values (1);

sync_slave_with_master;
stop slave;
set global max_session_mem_used=10*1024*1024;
start slave;

connection master;
let $a=1000000;
start transaction;
disable_query_log;
while ($a) {
  update t1 set a=a+1;
  dec $a;
}
enable_query_log;
commit;
drop table t1;
sync_slave_with_master;
set global max_session_mem_used=default;
source include/rpl_end.inc;
{code}

To fix that, rbr events *must* do
{code:c++}
free_root(thd->mem_root, MYF(MY_KEEP_PREALLOC));
{code}
somewhere. Where it's safe, because it'll destroy anything allocated on the thd memroot.
This patch appears to be working fine, it frees thd memroot after closing all opened tables:
{code:c++}
--- a/sql/rpl_rli.cc
+++ b/sql/rpl_rli.cc
@@ -2326,6 +2326,7 @@ void rpl_group_info::cleanup_context(THD *thd, bool error)
   */
   reset_row_stmt_start_timestamp();
   unset_long_find_row_note_printed();
+ free_root(thd->mem_root, MYF(MY_KEEP_PREALLOC));

   DBUG_EXECUTE_IF("inject_sleep_gtid_100_x_x", {
       if (current_gtid.domain_id == 100)
{code}

Andrei Elkin made changes - 2021-09-29 17:00

Status

Confirmed [ 10101 ]

In Progress [ 3 ]

Andrei Elkin made changes - 2021-09-29 17:03

Assignee	Andrei Elkin [ elkin ]	Sergei Golubchik [ serg ]
Status	In Progress [ 3 ]	In Review [ 10002 ]

Sergei Golubchik made changes - 2021-10-01 22:01

Assignee	Sergei Golubchik [ serg ]	Andrei Elkin [ elkin ]
Status	In Review [ 10002 ]	Stalled [ 10000 ]

Andrei Elkin made changes - 2021-10-13 19:47

Fix Version/s		10.2.41 [ 26032 ]
Fix Version/s		10.3.32 [ 26029 ]
Fix Version/s		10.4.22 [ 26031 ]
Fix Version/s		10.5.13 [ 26026 ]
Fix Version/s		10.6.5 [ 26034 ]
Fix Version/s	10.2 [ 14601 ]
Fix Version/s	10.3 [ 22126 ]
Fix Version/s	10.4 [ 22408 ]
Fix Version/s	10.5 [ 23123 ]
Fix Version/s	10.6 [ 24028 ]
Resolution		Fixed [ 1 ]
Status	Stalled [ 10000 ]	Closed [ 6 ]

Sergei Golubchik made changes - 2021-12-06 21:54

Workflow

MariaDB v3 [ 125713 ]

MariaDB v4 [ 159726 ]

Rob Schwyzer (Inactive) made changes - 2024-02-08 20:48

Remote Link

This issue links to "Page (MariaDB Confluence)" [ 36610 ]

Jira Automation (IT) made changes - 2024-07-04 02:11

Zendesk Related Tickets

172587

People

Assignee:: Andrei Elkin

Reporter:: Sergei Golubchik

Votes:: 2 Vote for this issue

Watchers:: 8 Start watching this issue

Dates

Created:: 2021-09-28 19:34

Updated:: 2024-07-07 21:01

Resolved:: 2021-10-13 19:47

Git Integration

Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.

MariaDB Server

Details

Description

Attachments

Issue Links

Activity

People

Dates

Git Integration