Uploaded image for project: 'MariaDB Connector/Python'
  1. MariaDB Connector/Python
  2. CONPY-108

Memory leak while continuously inserting/updating data

    XMLWordPrintable

Details

    • Bug
    • Status: Closed (View Workflow)
    • Critical
    • Resolution: Fixed
    • 1.0.1
    • 1.0.2
    • DBAPI 2.0
    • None
    • Ubuntu 19.10, MariaDB 10.5.5-MariaDB-1:10.5.5+maria~focal - mariadb.org binary distribution

    Description

      I'm running the connector to feed IOT data from a broker into MariaDB tables. It opens a single connection and uses a single cursor. The values for "pyformat" style are specified as a dict.

      While running the script I can watch the script continuously growing up to several GB during a few hours.

      I already checked that it is related to the MariaDB actions by commenting out the three execute() calls, what is completely stopping memory growth.

      I also checked what would happen if I do a cyclic cursor.close() and connection.close(), but no change, memory is not getting released.

      If any additional information is required please let me know.

      Update 2020-08-22:
      In order to help to track this down I just changed the whole insertion process to just one line which calls a stored procedure with 4 parameters (varchar(15), datetime, char(1), varchar(512)). The memory leak is still clearly visible, memory usage is continuously increasing.

      CREATE TABLE `leak` (
        `a` varchar(15) COLLATE utf8_bin NOT NULL,
        `b` datetime NOT NULL,
        `c` char(1) COLLATE utf8_bin NOT NULL,
        `d` varchar(512) COLLATE utf8_bin NOT NULL,
        KEY `a` (`a`,`b`)
      ) ENGINE=ROCKSDB DEFAULT CHARSET=utf8 COLLATE=utf8_bin
      

      Memory size during script run:

      Just started:
      F S   UID   PID  PPID  C PRI  NI ADDR SZ WCHAN  TTY          TIME CMD
      0 S  1000 19814 26276 14  80   0 -  7667 poll_s pts/0    00:00:00 demo.py
       
      2:30 runtime:
      F S   UID   PID  PPID  C PRI  NI ADDR SZ WCHAN  TTY          TIME CMD
      0 S  1000 19814 26276  3  80   0 -  9185 poll_s pts/0    00:00:05 demo.py
       
      5:10 runtime:
      F S   UID   PID  PPID  C PRI  NI ADDR SZ WCHAN  TTY          TIME CMD
      0 S  1000 19814 26276  3  80   0 - 10835 poll_s pts/0    00:00:09 demo.py
      

      Attachments

        Activity

          People

            georg Georg Richter
            svens Sven Siegel
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Git Integration

                Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.