[CONJ-860] Possible Deadlocks due to AbstractQueryProtocol.setTimeout - Jira

XML

Word

Printable

Details

Type: Bug
Status: Closed (View Workflow)
Priority: Major
Resolution: Cannot Reproduce
Affects Version/s: 2.6.0
Fix Version/s: N/A
Component/s: aurora
Labels:
None

Description

Hi,

we are occasionally seeing (roughly once per day) threads get stuck in our Application that don't recover until we restart the app. Looking at the threads via `jstack <pid> ` we see that they seem to be blocked due to `AbstractQueryProtocol.setTimeout`.

We use an RDS Aurora Cluster with engine version 5.7.mysql_aurora.2.07.2.

Examples stack traces:

Thread 20226: (state = BLOCKED)

 - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information may be imprecise)

 - java.util.concurrent.locks.LockSupport.park(java.lang.Object) @bci=14, line=175 (Compiled frame)

 - java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt() @bci=1, line=836 (Compiled frame)

 - java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(java.util.concurrent.locks.AbstractQueuedSynchronizer$Node, int) @bci=67, line=870 (Compiled frame)

 - java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(int) @bci=17, line=1199 (Compiled frame)

 - java.util.concurrent.locks.ReentrantLock$NonfairSync.lock() @bci=21, line=209 (Compiled frame)

 - java.util.concurrent.locks.ReentrantLock.lock() @bci=4, line=285 (Compiled frame)

 - org.mariadb.jdbc.internal.protocol.AbstractQueryProtocol.setTimeout(int) @bci=4, line=1444 (Compiled frame)

 - sun.reflect.GeneratedMethodAccessor57.invoke(java.lang.Object, java.lang.Object[]) @bci=114 (Compiled frame)

 - sun.reflect.DelegatingMethodAccessorImpl.invoke(java.lang.Object, java.lang.Object[]) @bci=6, line=43 (Compiled frame)

 - java.lang.reflect.Method.invoke(java.lang.Object, java.lang.Object[]) @bci=56, line=498 (Compiled frame)

 - org.mariadb.jdbc.internal.failover.impl.MastersSlavesListener.invoke(java.lang.reflect.Method, java.lang.Object[]) @bci=142, line=233 (Compiled frame)

 - org.mariadb.jdbc.internal.failover.FailoverProxy.executeInvocation(java.lang.reflect.Method, java.lang.Object[], boolean) @bci=17, line=301 (Compiled frame)

 - java.lang.String.hashCode() @bci=33, line=1471 (Compiled frame)

 - org.mariadb.jdbc.internal.failover.FailoverProxy.invoke(java.lang.Object, java.lang.reflect.Method, java.lang.Object[]) @bci=15, line=163 (Compiled frame)

 - com.sun.proxy.$Proxy55.setTimeout(int) @bci=19 (Compiled frame)

 - org.mariadb.jdbc.MariaDbConnection.setNetworkTimeout(java.util.concurrent.Executor, int) @bci=72, line=1669 (Compiled frame)

 - com.zaxxer.hikari.pool.PoolBase.setNetworkTimeout(java.sql.Connection, long) @bci=15, line=556 (Compiled frame)

 - com.zaxxer.hikari.pool.PoolBase.isConnectionAlive(java.sql.Connection) @bci=6, line=152 (Compiled frame)

 - com.zaxxer.hikari.pool.HikariPool.getConnection(long) @bci=76, line=185 (Compiled frame)

 - com.zaxxer.hikari.pool.HikariPool.getConnection() @bci=5, line=161 (Compiled frame)

 - com.zaxxer.hikari.HikariDataSource.getConnection() @bci=50, line=100 (Compiled frame)

...

Thread 20472: (state = BLOCKED)

 - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information may be imprecise)

 - java.util.concurrent.locks.LockSupport.park(java.lang.Object) @bci=14, line=175 (Compiled frame)

 - java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt() @bci=1, line=836 (Compiled frame)

 - java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(java.util.concurrent.locks.AbstractQueuedSynchronizer$Node, int) @bci=67, line=870 (Compiled frame)

 - java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(int) @bci=17, line=1199 (Compiled frame)

 - java.util.concurrent.locks.ReentrantLock$NonfairSync.lock() @bci=21, line=209 (Compiled frame)

 - java.util.concurrent.locks.ReentrantLock.lock() @bci=4, line=285 (Compiled frame)

 - org.mariadb.jdbc.internal.protocol.AbstractQueryProtocol.setTimeout(int) @bci=4, line=1444 (Compiled frame)

 - sun.reflect.GeneratedMethodAccessor57.invoke(java.lang.Object, java.lang.Object[]) @bci=114 (Compiled frame)

 - sun.reflect.DelegatingMethodAccessorImpl.invoke(java.lang.Object, java.lang.Object[]) @bci=6, line=43 (Compiled frame)

 - java.lang.reflect.Method.invoke(java.lang.Object, java.lang.Object[]) @bci=56, line=498 (Compiled frame)

 - org.mariadb.jdbc.internal.failover.impl.MastersSlavesListener.invoke(java.lang.reflect.Method, java.lang.Object[]) @bci=142, line=233 (Compiled frame)

 - org.mariadb.jdbc.internal.failover.FailoverProxy.executeInvocation(java.lang.reflect.Method, java.lang.Object[], boolean) @bci=17, line=301 (Compiled frame)

 - java.lang.String.hashCode() @bci=33, line=1471 (Compiled frame)

 - org.mariadb.jdbc.internal.failover.FailoverProxy.invoke(java.lang.Object, java.lang.reflect.Method, java.lang.Object[]) @bci=15, line=163 (Compiled frame)

 - com.sun.proxy.$Proxy55.setTimeout(int) @bci=19 (Compiled frame)

 - org.mariadb.jdbc.MariaDbConnection.setNetworkTimeout(java.util.concurrent.Executor, int) @bci=72, line=1669 (Compiled frame)

 - com.zaxxer.hikari.pool.PoolBase.setNetworkTimeout(java.sql.Connection, long) @bci=15, line=556 (Compiled frame)

 - com.zaxxer.hikari.pool.PoolBase.isConnectionAlive(java.sql.Connection) @bci=6, line=152 (Interpreted frame)

 - com.zaxxer.hikari.pool.HikariPool.getConnection(long) @bci=76, line=185 (Interpreted frame)

 - com.zaxxer.hikari.pool.HikariPool.getConnection() @bci=5, line=161 (Interpreted frame)

 - com.zaxxer.hikari.HikariDataSource.getConnection() @bci=50, line=100 (Interpreted frame)

...

We have the following parameters configured:

ImmutableMap.<String, String>builder()

                .put("cachePrepStmts", "true")

                .put("prepStmtCacheSize", "500")

                .put("prepStmtCacheSqlLimit", "2048")

                .put("useServerPrepStmts", "true")

                .put("rewriteBatchedStatements", "true")

                .put("cacheResultSetMetadata", "true")

                .put("cacheServerConfiguration", "true")

                .put("socketTimeout", "60000")

                .put("serverTimezone", "UTC")

                .put("dumpQueriesOnException", "true")

                .put("useCompression", "true")

                .put("log", "true")

                .put("useUnicode", "true")

                .put("connectionCollation", "utf8mb4_unicode_ci")

                .put("characterSetResults", "utf8")

                .put("zeroDateTimeBehavior", "convertToNull")

                .put("enablePacketDebug", "true")

                .build();

Any help would be appreciated.

Attachments

- Sort By Name
- Sort By Date
- Ascending
- Descending
- Thumbnails
- List
- Download All

bgthreads.dmp
130 kB
2021-02-19 16:01

Activity

People

Assignee:: Diego Dupin

Reporter:: M Z

Votes:: 0 Vote for this issue

Watchers:: 2 Start watching this issue

Dates

Created:: 2021-02-17 07:26

Updated:: 2021-05-17 08:33

Resolved:: 2021-05-17 08:33

Git Integration

Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.