Uploaded image for project: 'MariaDB Connector/Python'
  1. MariaDB Connector/Python
  2. CONPY-108

Memory leak while continuously inserting/updating data

Details

    • Bug
    • Status: Closed (View Workflow)
    • Critical
    • Resolution: Fixed
    • 1.0.1
    • 1.0.2
    • DBAPI 2.0
    • None
    • Ubuntu 19.10, MariaDB 10.5.5-MariaDB-1:10.5.5+maria~focal - mariadb.org binary distribution

    Description

      I'm running the connector to feed IOT data from a broker into MariaDB tables. It opens a single connection and uses a single cursor. The values for "pyformat" style are specified as a dict.

      While running the script I can watch the script continuously growing up to several GB during a few hours.

      I already checked that it is related to the MariaDB actions by commenting out the three execute() calls, what is completely stopping memory growth.

      I also checked what would happen if I do a cyclic cursor.close() and connection.close(), but no change, memory is not getting released.

      If any additional information is required please let me know.

      Update 2020-08-22:
      In order to help to track this down I just changed the whole insertion process to just one line which calls a stored procedure with 4 parameters (varchar(15), datetime, char(1), varchar(512)). The memory leak is still clearly visible, memory usage is continuously increasing.

      CREATE TABLE `leak` (
        `a` varchar(15) COLLATE utf8_bin NOT NULL,
        `b` datetime NOT NULL,
        `c` char(1) COLLATE utf8_bin NOT NULL,
        `d` varchar(512) COLLATE utf8_bin NOT NULL,
        KEY `a` (`a`,`b`)
      ) ENGINE=ROCKSDB DEFAULT CHARSET=utf8 COLLATE=utf8_bin
      

      Memory size during script run:

      Just started:
      F S   UID   PID  PPID  C PRI  NI ADDR SZ WCHAN  TTY          TIME CMD
      0 S  1000 19814 26276 14  80   0 -  7667 poll_s pts/0    00:00:00 demo.py
       
      2:30 runtime:
      F S   UID   PID  PPID  C PRI  NI ADDR SZ WCHAN  TTY          TIME CMD
      0 S  1000 19814 26276  3  80   0 -  9185 poll_s pts/0    00:00:05 demo.py
       
      5:10 runtime:
      F S   UID   PID  PPID  C PRI  NI ADDR SZ WCHAN  TTY          TIME CMD
      0 S  1000 19814 26276  3  80   0 - 10835 poll_s pts/0    00:00:09 demo.py
      

      Attachments

        Activity

          georg Georg Richter added a comment -

          Thank for your bug report.
          Could you please attach a small python script which reproduces the memory leak?
          Thanks!

          georg Georg Richter added a comment - Thank for your bug report. Could you please attach a small python script which reproduces the memory leak? Thanks!
          svens Sven Siegel added a comment - - edited

          Some more info:

          1. I checked this scenario on a Raspberry PI 4 - same problem, memory usage of the python script is constantly increasing with every execute()
          2. I checked with other DB engine - same problem even with InnoDB
          3. Type of SQL command seems not to matter - same problem if calling a procedure, running INSERT or UPDATE
          svens Sven Siegel added a comment - - edited Some more info: I checked this scenario on a Raspberry PI 4 - same problem, memory usage of the python script is constantly increasing with every execute() I checked with other DB engine - same problem even with InnoDB Type of SQL command seems not to matter - same problem if calling a procedure, running INSERT or UPDATE
          georg Georg Richter added a comment -

          Fixed in rev. bd560c2bb91df7679e60c6f12a38c4a2aea4cb32

          georg Georg Richter added a comment - Fixed in rev. bd560c2bb91df7679e60c6f12a38c4a2aea4cb32

          People

            georg Georg Richter
            svens Sven Siegel
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Git Integration

                Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.