Uploaded image for project: 'MariaDB Connector/Python'
  1. MariaDB Connector/Python
  2. CONPY-108

Memory leak while continuously inserting/updating data

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Closed (View Workflow)
    • Priority: Critical
    • Resolution: Fixed
    • Affects Version/s: 1.0.1
    • Fix Version/s: 1.0.2
    • Component/s: DBAPI 2.0
    • Labels:
      None
    • Environment:
      Ubuntu 19.10, MariaDB 10.5.5-MariaDB-1:10.5.5+maria~focal - mariadb.org binary distribution

      Description

      I'm running the connector to feed IOT data from a broker into MariaDB tables. It opens a single connection and uses a single cursor. The values for "pyformat" style are specified as a dict.

      While running the script I can watch the script continuously growing up to several GB during a few hours.

      I already checked that it is related to the MariaDB actions by commenting out the three execute() calls, what is completely stopping memory growth.

      I also checked what would happen if I do a cyclic cursor.close() and connection.close(), but no change, memory is not getting released.

      If any additional information is required please let me know.

      Update 2020-08-22:
      In order to help to track this down I just changed the whole insertion process to just one line which calls a stored procedure with 4 parameters (varchar(15), datetime, char(1), varchar(512)). The memory leak is still clearly visible, memory usage is continuously increasing.

      CREATE TABLE `leak` (
        `a` varchar(15) COLLATE utf8_bin NOT NULL,
        `b` datetime NOT NULL,
        `c` char(1) COLLATE utf8_bin NOT NULL,
        `d` varchar(512) COLLATE utf8_bin NOT NULL,
        KEY `a` (`a`,`b`)
      ) ENGINE=ROCKSDB DEFAULT CHARSET=utf8 COLLATE=utf8_bin
      

      Memory size during script run:

      Just started:
      F S   UID   PID  PPID  C PRI  NI ADDR SZ WCHAN  TTY          TIME CMD
      0 S  1000 19814 26276 14  80   0 -  7667 poll_s pts/0    00:00:00 demo.py
       
      2:30 runtime:
      F S   UID   PID  PPID  C PRI  NI ADDR SZ WCHAN  TTY          TIME CMD
      0 S  1000 19814 26276  3  80   0 -  9185 poll_s pts/0    00:00:05 demo.py
       
      5:10 runtime:
      F S   UID   PID  PPID  C PRI  NI ADDR SZ WCHAN  TTY          TIME CMD
      0 S  1000 19814 26276  3  80   0 - 10835 poll_s pts/0    00:00:09 demo.py
      

        Attachments

          Activity

            People

            Assignee:
            georg Georg Richter
            Reporter:
            svens Sven Siegel
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Dates

              Created:
              Updated:
              Resolved:

                Git Integration