Uploaded image for project: 'MariaDB Connector/J'
  1. MariaDB Connector/J
  2. CONJ-860

Possible Deadlocks due to AbstractQueryProtocol.setTimeout

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Closed (View Workflow)
    • Priority: Major
    • Resolution: Cannot Reproduce
    • Affects Version/s: 2.6.0
    • Fix Version/s: N/A
    • Component/s: aurora
    • Labels:
      None

      Description

      Hi,

      we are occasionally seeing (roughly once per day) threads get stuck in our Application that don't recover until we restart the app. Looking at the threads via `jstack <pid> ` we see that they seem to be blocked due to `AbstractQueryProtocol.setTimeout`.

      We use an RDS Aurora Cluster with engine version 5.7.mysql_aurora.2.07.2.

      Examples stack traces:

      Thread 20226: (state = BLOCKED)
       - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information may be imprecise)
       - java.util.concurrent.locks.LockSupport.park(java.lang.Object) @bci=14, line=175 (Compiled frame)
       - java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt() @bci=1, line=836 (Compiled frame)
       - java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(java.util.concurrent.locks.AbstractQueuedSynchronizer$Node, int) @bci=67, line=870 (Compiled frame)
       - java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(int) @bci=17, line=1199 (Compiled frame)
       - java.util.concurrent.locks.ReentrantLock$NonfairSync.lock() @bci=21, line=209 (Compiled frame)
       - java.util.concurrent.locks.ReentrantLock.lock() @bci=4, line=285 (Compiled frame)
       - org.mariadb.jdbc.internal.protocol.AbstractQueryProtocol.setTimeout(int) @bci=4, line=1444 (Compiled frame)
       - sun.reflect.GeneratedMethodAccessor57.invoke(java.lang.Object, java.lang.Object[]) @bci=114 (Compiled frame)
       - sun.reflect.DelegatingMethodAccessorImpl.invoke(java.lang.Object, java.lang.Object[]) @bci=6, line=43 (Compiled frame)
       - java.lang.reflect.Method.invoke(java.lang.Object, java.lang.Object[]) @bci=56, line=498 (Compiled frame)
       - org.mariadb.jdbc.internal.failover.impl.MastersSlavesListener.invoke(java.lang.reflect.Method, java.lang.Object[]) @bci=142, line=233 (Compiled frame)
       - org.mariadb.jdbc.internal.failover.FailoverProxy.executeInvocation(java.lang.reflect.Method, java.lang.Object[], boolean) @bci=17, line=301 (Compiled frame)
       - java.lang.String.hashCode() @bci=33, line=1471 (Compiled frame)
       - org.mariadb.jdbc.internal.failover.FailoverProxy.invoke(java.lang.Object, java.lang.reflect.Method, java.lang.Object[]) @bci=15, line=163 (Compiled frame)
       - com.sun.proxy.$Proxy55.setTimeout(int) @bci=19 (Compiled frame)
       - org.mariadb.jdbc.MariaDbConnection.setNetworkTimeout(java.util.concurrent.Executor, int) @bci=72, line=1669 (Compiled frame)
       - com.zaxxer.hikari.pool.PoolBase.setNetworkTimeout(java.sql.Connection, long) @bci=15, line=556 (Compiled frame)
       - com.zaxxer.hikari.pool.PoolBase.isConnectionAlive(java.sql.Connection) @bci=6, line=152 (Compiled frame)
       - com.zaxxer.hikari.pool.HikariPool.getConnection(long) @bci=76, line=185 (Compiled frame)
       - com.zaxxer.hikari.pool.HikariPool.getConnection() @bci=5, line=161 (Compiled frame)
       - com.zaxxer.hikari.HikariDataSource.getConnection() @bci=50, line=100 (Compiled frame)
      ...
      

      Thread 20472: (state = BLOCKED)
       - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information may be imprecise)
       - java.util.concurrent.locks.LockSupport.park(java.lang.Object) @bci=14, line=175 (Compiled frame)
       - java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt() @bci=1, line=836 (Compiled frame)
       - java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(java.util.concurrent.locks.AbstractQueuedSynchronizer$Node, int) @bci=67, line=870 (Compiled frame)
       - java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(int) @bci=17, line=1199 (Compiled frame)
       - java.util.concurrent.locks.ReentrantLock$NonfairSync.lock() @bci=21, line=209 (Compiled frame)
       - java.util.concurrent.locks.ReentrantLock.lock() @bci=4, line=285 (Compiled frame)
       - org.mariadb.jdbc.internal.protocol.AbstractQueryProtocol.setTimeout(int) @bci=4, line=1444 (Compiled frame)
       - sun.reflect.GeneratedMethodAccessor57.invoke(java.lang.Object, java.lang.Object[]) @bci=114 (Compiled frame)
       - sun.reflect.DelegatingMethodAccessorImpl.invoke(java.lang.Object, java.lang.Object[]) @bci=6, line=43 (Compiled frame)
       - java.lang.reflect.Method.invoke(java.lang.Object, java.lang.Object[]) @bci=56, line=498 (Compiled frame)
       - org.mariadb.jdbc.internal.failover.impl.MastersSlavesListener.invoke(java.lang.reflect.Method, java.lang.Object[]) @bci=142, line=233 (Compiled frame)
       - org.mariadb.jdbc.internal.failover.FailoverProxy.executeInvocation(java.lang.reflect.Method, java.lang.Object[], boolean) @bci=17, line=301 (Compiled frame)
       - java.lang.String.hashCode() @bci=33, line=1471 (Compiled frame)
       - org.mariadb.jdbc.internal.failover.FailoverProxy.invoke(java.lang.Object, java.lang.reflect.Method, java.lang.Object[]) @bci=15, line=163 (Compiled frame)
       - com.sun.proxy.$Proxy55.setTimeout(int) @bci=19 (Compiled frame)
       - org.mariadb.jdbc.MariaDbConnection.setNetworkTimeout(java.util.concurrent.Executor, int) @bci=72, line=1669 (Compiled frame)
       - com.zaxxer.hikari.pool.PoolBase.setNetworkTimeout(java.sql.Connection, long) @bci=15, line=556 (Compiled frame)
       - com.zaxxer.hikari.pool.PoolBase.isConnectionAlive(java.sql.Connection) @bci=6, line=152 (Interpreted frame)
       - com.zaxxer.hikari.pool.HikariPool.getConnection(long) @bci=76, line=185 (Interpreted frame)
       - com.zaxxer.hikari.pool.HikariPool.getConnection() @bci=5, line=161 (Interpreted frame)
       - com.zaxxer.hikari.HikariDataSource.getConnection() @bci=50, line=100 (Interpreted frame)
      ...
      

      We have the following parameters configured:

      ImmutableMap.<String, String>builder()
                      .put("cachePrepStmts", "true")
                      .put("prepStmtCacheSize", "500")
                      .put("prepStmtCacheSqlLimit", "2048")
                      .put("useServerPrepStmts", "true")
                      .put("rewriteBatchedStatements", "true")
                      .put("cacheResultSetMetadata", "true")
                      .put("cacheServerConfiguration", "true")
                      .put("socketTimeout", "60000")
                      .put("serverTimezone", "UTC")
                      .put("dumpQueriesOnException", "true")
                      .put("useCompression", "true")
                      .put("log", "true")
                      .put("useUnicode", "true")
                      .put("connectionCollation", "utf8mb4_unicode_ci")
                      .put("characterSetResults", "utf8")
                      .put("zeroDateTimeBehavior", "convertToNull")
                      .put("enablePacketDebug", "true")
                      .build();
      

      Any help would be appreciated.

        Attachments

          Activity

            People

            Assignee:
            diego dupin Diego Dupin
            Reporter:
            kartoffelsup M Z
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Dates

              Created:
              Updated:
              Resolved:

                Git Integration

                Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.