[CONPY-108] Memory leak while continuously inserting/updating data Created: 2020-08-20  Updated: 2020-08-29  Resolved: 2020-08-29

Status: Closed
Project: MariaDB Connector/Python
Component/s: DBAPI 2.0
Affects Version/s: 1.0.1
Fix Version/s: 1.0.2

Type: Bug Priority: Critical
Reporter: Sven Siegel Assignee: Georg Richter
Resolution: Fixed Votes: 0
Labels: None
Environment:

Ubuntu 19.10, MariaDB 10.5.5-MariaDB-1:10.5.5+maria~focal - mariadb.org binary distribution


Attachments: File demo.py    

 Description   

I'm running the connector to feed IOT data from a broker into MariaDB tables. It opens a single connection and uses a single cursor. The values for "pyformat" style are specified as a dict.

While running the script I can watch the script continuously growing up to several GB during a few hours.

I already checked that it is related to the MariaDB actions by commenting out the three execute() calls, what is completely stopping memory growth.

I also checked what would happen if I do a cyclic cursor.close() and connection.close(), but no change, memory is not getting released.

If any additional information is required please let me know.

Update 2020-08-22:
In order to help to track this down I just changed the whole insertion process to just one line which calls a stored procedure with 4 parameters (varchar(15), datetime, char(1), varchar(512)). The memory leak is still clearly visible, memory usage is continuously increasing.

CREATE TABLE `leak` (
  `a` varchar(15) COLLATE utf8_bin NOT NULL,
  `b` datetime NOT NULL,
  `c` char(1) COLLATE utf8_bin NOT NULL,
  `d` varchar(512) COLLATE utf8_bin NOT NULL,
  KEY `a` (`a`,`b`)
) ENGINE=ROCKSDB DEFAULT CHARSET=utf8 COLLATE=utf8_bin

Memory size during script run:

Just started:
F S   UID   PID  PPID  C PRI  NI ADDR SZ WCHAN  TTY          TIME CMD
0 S  1000 19814 26276 14  80   0 -  7667 poll_s pts/0    00:00:00 demo.py
 
2:30 runtime:
F S   UID   PID  PPID  C PRI  NI ADDR SZ WCHAN  TTY          TIME CMD
0 S  1000 19814 26276  3  80   0 -  9185 poll_s pts/0    00:00:05 demo.py
 
5:10 runtime:
F S   UID   PID  PPID  C PRI  NI ADDR SZ WCHAN  TTY          TIME CMD
0 S  1000 19814 26276  3  80   0 - 10835 poll_s pts/0    00:00:09 demo.py



 Comments   
Comment by Georg Richter [ 2020-08-22 ]

Thank for your bug report.
Could you please attach a small python script which reproduces the memory leak?
Thanks!

Comment by Sven Siegel [ 2020-08-27 ]

Some more info:

  1. I checked this scenario on a Raspberry PI 4 - same problem, memory usage of the python script is constantly increasing with every execute()
  2. I checked with other DB engine - same problem even with InnoDB
  3. Type of SQL command seems not to matter - same problem if calling a procedure, running INSERT or UPDATE
Comment by Georg Richter [ 2020-08-29 ]

Fixed in rev. bd560c2bb91df7679e60c6f12a38c4a2aea4cb32

Generated at Thu Feb 08 03:30:19 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.