Uploaded image for project: 'MariaDB Connector/J'
  1. MariaDB Connector/J
  2. CONJ-929

OOM happening when querying large datasets and calling isValid on connection

    XMLWordPrintable

Details

    • Bug
    • Status: Closed (View Workflow)
    • Blocker
    • Resolution: Won't Fix
    • 2.7.5
    • N/A
    • performance
    • None

    Description

      I am using the following

      • AWS Aurora MySQL 5.6
      • mariadb-java-client 2.7.5
      • Kafka Connect JDBC Source 10.0.0 (also reproduces locally with simple java process)

      Hello

      I'm getting the same issue with Kafka Connect JDBC Source. (https://jira.mariadb.org/browse/CONJ-715)
      We have migrated our connector from Oracle to Aurora (RDS) and now using the MariaDB Connector.

      We have a large table to read with connector and we are facing OOM.
      A #isValid call of the current connection (when iterating a result set) produces an OOM because of

      // load active result if any so buffer are clean for next query
      if (activeStreamingResult != null)

      { activeStreamingResult.loadFully(false, this); activeStreamingResult = null; }

      Do you have any idea why calling isValid method has this impact ?
      Thank you

      STACK :
      java.lang.OutOfMemoryError: Java heap space
      at java.base/java.util.Arrays.copyOf(Arrays.java:3689)
      at org.mariadb.jdbc.internal.com.read.resultset.SelectResultSet.growDataArray(SelectResultSet.java:521)
      at org.mariadb.jdbc.internal.com.read.resultset.SelectResultSet.readNextValue(SelectResultSet.java:435)
      at org.mariadb.jdbc.internal.com.read.resultset.SelectResultSet.addStreamingValue(SelectResultSet.java:364)
      at org.mariadb.jdbc.internal.com.read.resultset.SelectResultSet.fetchRemaining(SelectResultSet.java:311)
      at org.mariadb.jdbc.internal.com.read.dao.Results.loadFully(Results.java:274)
      at org.mariadb.jdbc.internal.protocol.AbstractQueryProtocol.cmdPrologue(AbstractQueryProtocol.java:1918)
      at org.mariadb.jdbc.internal.protocol.AbstractQueryProtocol.ping(AbstractQueryProtocol.java:1225)
      at org.mariadb.jdbc.internal.protocol.AbstractQueryProtocol.isValid(AbstractQueryProtocol.java:1272)
      at org.mariadb.jdbc.MariaDbConnection.isValid(MariaDbConnection.java:1228)
      at io.confluent.connect.jdbc.dialect.GenericDatabaseDialect.isConnectionValid(GenericDatabaseDialect.java:253)
      at io.confluent.connect.jdbc.util.CachedConnectionProvider.isConnectionValid(CachedConnectionProvider.java:70)
      at io.confluent.connect.jdbc.util.CachedConnectionProvider.getConnection(CachedConnectionProvider.java:53)
      at io.confluent.connect.jdbc.source.JdbcSourceTask.poll(JdbcSourceTask.java:364)

      Attachments

        Activity

          People

            diego dupin Diego Dupin
            cyril.joui Cyril Joui
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Git Integration

                Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.