[CONJ-770] [AURORA] - Seeing seemingly random connection resets when using aurora read/write cluster Created: 2020-03-10  Updated: 2022-08-16  Resolved: 2022-08-16

Status: Closed
Project: MariaDB Connector/J
Component/s: aurora
Affects Version/s: 2.5.3
Fix Version/s: N/A

Type: Bug Priority: Major
Reporter: M Z Assignee: Diego Dupin
Resolution: Not a Bug Votes: 0
Labels: None

Issue Links:
Relates
relates to CONJ-771 enablePacketDebug reset stack when re... Closed

 Description   

We have recently switched to the MariaDB Java Client for our Aurora RDS cluster to utilize the read/write balancing. However, we occasionally (roughly once a day) see connection reset errors. We enabled `enablePacketDebug` to see if we can find what is causing this, and it seems to be a transition from write to read only. The following are 3 examples of what was executed according to `enablePacketDebug` before the connection reset errors occurred:

send at -exchange:15913- 2020-03-10T06:45:14.811Z (compressed protocol - packet not compressed)
 
3D 00 00 00 00 00 00                                 =......
 
39 00 00 00                                          9...
 
03 53 45 54 20 53 45 53  53 49 4F 4E 20 54 52 41     .SET SESSION TRA
 
4E 53 41 43 54 49 4F 4E  20 49 53 4F 4C 41 54 49     NSACTION ISOLATI
 
4F 4E 20 4C 45 56 45 4C  20 52 45 41 44 20 55 4E     ON LEVEL READ UN
 
43 4F 4D 4D 49 54 54 45  44                          COMMITTED
 
 
 
read at -exchange:15914- 2020-03-10T06:45:14.812Z (compressed protocol - packet not compressed)
 
2C 00 00 01 00 00 00                                 ,......
 
28 00 00 01 00 00 00 02  40 00 00 00 1F 00 1D 0C     (.......@.......
 
74 78 5F 69 73 6F 6C 61  74 69 6F 6E 0F 52 45 50     tx_isolation.REP
 
45 41 54 41 42 4C 45 2D  52 45 41 44                 EATABLE-READ


send at -exchange:279198- 2020-03-10T08:36:38.608Z (compressed protocol - packet not compressed)
 
3D 00 00 00 00 00 00                                 =......
 
39 00 00 00                                          9...
 
03 53 45 54 20 53 45 53  53 49 4F 4E 20 54 52 41     .SET SESSION TRA
 
4E 53 41 43 54 49 4F 4E  20 49 53 4F 4C 41 54 49     NSACTION ISOLATI
 
4F 4E 20 4C 45 56 45 4C  20 52 45 41 44 20 55 4E     ON LEVEL READ UN
 
43 4F 4D 4D 49 54 54 45  44                          COMMITTED
 
 
 
read at -exchange:279199- 2020-03-10T08:36:38.609Z (compressed protocol - packet not compressed)
 
2C 00 00 01 00 00 00                                 ,......
 
28 00 00 01 00 00 00 02  40 00 00 00 1F 00 1D 0C     (.......@.......
 
74 78 5F 69 73 6F 6C 61  74 69 6F 6E 0F 52 45 50     tx_isolation.REP
 
45 41 54 41 42 4C 45 2D  52 45 41 44                 EATABLE-READ


send at -exchange:115923- 2020-03-10T08:37:50.187Z (compressed protocol - packet not compressed)
 
3D 00 00 00 00 00 00                                 =......
 
39 00 00 00                                          9...
 
03 53 45 54 20 53 45 53  53 49 4F 4E 20 54 52 41     .SET SESSION TRA
 
4E 53 41 43 54 49 4F 4E  20 49 53 4F 4C 41 54 49     NSACTION ISOLATI
 
4F 4E 20 4C 45 56 45 4C  20 52 45 41 44 20 55 4E     ON LEVEL READ UN
 
43 4F 4D 4D 49 54 54 45  44                          COMMITTED
 
 
 
read at -exchange:115924- 2020-03-10T08:37:50.190Z (compressed protocol - packet not compressed)
 
2C 00 00 01 00 00 00                                 ,......
 
28 00 00 01 00 00 00 02  40 00 00 00 1F 00 1D 0C     (.......@.......
 
74 78 5F 69 73 6F 6C 61  74 69 6F 6E 0F 52 45 50     tx_isolation.REP
 
45 41 54 41 42 4C 45 2D  52 45 41 44                 EATABLE-READ

any help with this would be highly appreciated.



 Comments   
Comment by Diego Dupin [ 2020-03-10 ]

What do you mean exactly by connection reset ?
The query logged just indicate that 

Connection.setTransactionIsolation(Connection.TRANSACTION_READ_UNCOMMITTED)

has been called. That can be by application, or more probably occurs after a socket fails: connectors automatically create a new connection, and set database, transaction level according to the state of previous connection.

Comment by M Z [ 2020-03-10 ]

Ah sorry, what I mean is the following exception occurs:

ERROR o.m.j.MariaDbStatement - error executing query
 
java.sql.SQLException: (conn=382843) Connection reset
 
	at org.mariadb.jdbc.internal.util.exceptions.ExceptionMapper.get(ExceptionMapper.java:262) ~[Mariadb-java-client-2.5.x.jar:?]
 
	at org.mariadb.jdbc.internal.util.exceptions.ExceptionMapper.getException(ExceptionMapper.java:164) ~[Mariadb-java-client-2.5.x.jar:?]
 
	at org.mariadb.jdbc.MariaDbStatement.executeExceptionEpilogue(MariaDbStatement.java:258) ~[Mariadb-java-client-2.5.x.jar:?]
 
	at org.mariadb.jdbc.ClientSidePreparedStatement.executeInternal(ClientSidePreparedStatement.java:225) ~[Mariadb-java-client-2.5.x.jar:?]
 
	at org.mariadb.jdbc.ClientSidePreparedStatement.execute(ClientSidePreparedStatement.java:145) ~[Mariadb-java-client-2.5.x.jar:?]
 
	at org.mariadb.jdbc.ClientSidePreparedStatement.executeQuery(ClientSidePreparedStatement.java:159) ~[Mariadb-java-client-2.5.x.jar:?]
 
	at com.zaxxer.hikari.pool.ProxyPreparedStatement.executeQuery(ProxyPreparedStatement.java:52) ~[HikariCP-3.3.1.jar:?]
 
	at com.zaxxer.hikari.pool.HikariProxyPreparedStatement.executeQuery(HikariProxyPreparedStatement.java) ~[HikariCP-3.3.1.jar:?]
 
	at org.hibernate.engine.jdbc.internal.ResultSetReturnImpl.extract(ResultSetReturnImpl.java:70) ~[hibernate-core-5.1.0.Final.jar:5.1.0.Final]
 
       ....
 
Caused by: java.net.SocketException: Connection reset
 
	at java.net.SocketInputStream.read(SocketInputStream.java:210) ~[?:1.8.0_242]
 
	at java.net.SocketInputStream.read(SocketInputStream.java:141) ~[?:1.8.0_242]
 
	at java.io.FilterInputStream.read(FilterInputStream.java:133) ~[?:1.8.0_242]
 
	at org.mariadb.jdbc.internal.io.input.ReadAheadBufferedStream.fillBuffer(ReadAheadBufferedStream.java:129) ~[Mariadb-java-client-2.5.x.jar:?]
 
	at org.mariadb.jdbc.internal.io.input.ReadAheadBufferedStream.read(ReadAheadBufferedStream.java:102) ~[Mariadb-java-client-2.5.x.jar:?]
 
	at org.mariadb.jdbc.internal.io.input.DecompressPacketInputStream.readBlocking(DecompressPacketInputStream.java:192) ~[Mariadb-java-client-2.5.x.jar:?]
 
	at org.mariadb.jdbc.internal.io.input.DecompressPacketInputStream.getPacketArray(DecompressPacketInputStream.java:111) ~[Mariadb-java-client-2.5.x.jar:?]
 
	at org.mariadb.jdbc.internal.io.input.DecompressPacketInputStream.getPacket(DecompressPacketInputStream.java:91) ~[Mariadb-java-client-2.5.x.jar:?]
 
	at org.mariadb.jdbc.internal.protocol.AbstractQueryProtocol.readPacket(AbstractQueryProtocol.java:1443) ~[Mariadb-java-client-2.5.x.jar:?]
 
	at org.mariadb.jdbc.internal.protocol.AbstractQueryProtocol.getResult(AbstractQueryProtocol.java:1424) ~[Mariadb-java-client-2.5.x.jar:?]
 
	at org.mariadb.jdbc.internal.protocol.AbstractQueryProtocol.executeQuery(AbstractQueryProtocol.java:240) ~[Mariadb-java-client-2.5.x.jar:?]

The query logged just indicate that

I thought the enablePacketDebug option logs the packets that were sent before the error occured?

Comment by Diego Dupin [ 2020-03-11 ]

This error means that a TCP RST was received, so socket was closed by server, not because of some network issue.

After that you've got to discover why server has close connection.
It generally involve timeout, like wait_timeout, or if you are streaming very big resultset, check net_write_timeout too.

I would recommand to check those server values. default value is 28800s (8h) might it be possible that some connection have been idle for that amount of time ?
If this can be the case, this is generally avoid setting pool to timeout connection after some time.

Comment by M Z [ 2020-03-11 ]

Hi,

thank you for your reply.

The wait_timeout is the default at 8h and our HikariPool has:

config.setMaxLifetime(TimeUnit.MINUTES.toMillis(20));

the 

net_write_timeout

is 60 seconds.
And according to the timestamp of the last successful packet:

read at -exchange:99389- 2020-03-11T01:54:48.155Z (compressed protocol - packet not compressed)
 
2C 00 00 01 00 00 00                                 ,......
 
28 00 00 01 00 00 00 02  40 00 00 00 1F 00 1D 0C     (.......@.......
 
74 78 5F 69 73 6F 6C 61  74 69 6F 6E 0F 52 45 50     tx_isolation.REP
 
45 41 54 41 42 4C 45 2D  52 45 41 44                 EATABLE-READ

and the timestamp of the exception:

2020-03-11T01:54:48Z

it doesn't seem to be a timeout issue.

Could it be something else that's causing this? Maybe it's the transition from writer to reader that sometimes resets the tcp socket?

Comment by Diego Dupin [ 2020-03-13 ]

What i don't understands is that "java.net.SocketException: Connection reset" occurs when reading from socket, so RST was sent from server AFTER driver send a command (driver doesn't read from socket if there is no command send)
And writing to socket is the surest thing to ensure socket state... so socket was ok when sending data. What are the change that server send a RST after that...

Comment by Diego Dupin [ 2020-03-13 ]

For your previous message, I think those timestamps doesn't say anything:

  • Driver send a command successfully. Socket state is ok
  • Server send a TCP RST
  • socket is closed, driver create a new connection and set the state (tx_isolation) to previous connection. That corresponds to the trace indicated.

We can remote wait_timeout from the equation, but net_write_timeout can still be the problem:

One example :

  • Driver send a command with a lots of result
  • after 60s (net_write_timeout) server still hasn't sent all results, then force close of connection.
  • socket is closed, driver create a new connection and set the state (tx_isolation) to previous connection. That corresponds to the trace indicated...

This usually goes with command with big resultset and using setFetchSize : when using fetchSize, driver read only one row at a time, to avoid staturating memory with all data. Problem is that if application traitement on those rows takes more than 60s, this will result in this kind of issue.

Could you identify if there is some queries that would correspong to this at this time (batches usually)
If this might be the case, you could change session variable net_write_timeout to a bigger value using connection string like :
"jdbc:mariadb:..../db?user=myUser&sessionVariables=net_write_timeout=3600"

Comment by M Z [ 2020-03-13 ]

Thank you, I will increase `net_write_timeout` and observe and report back next week.

Comment by Diego Dupin [ 2020-03-17 ]

btw, CONJ-771 has been created because enablePacketDebug reset stack when reconnecting and there is no reason for that. It will permit easier identification of issue like this

Comment by M Z [ 2020-03-17 ]

Ah yes, that explains why the ConnectionID reported by the packetDebug only shows up after the issue occurred in the audit log.
I've set the `net_write_timeout` on global level via RDS to '3600' on friday, but the connection resets still happened on the weekend. So to make sure I set it as session variable on the connections yesterday and it hasn't happened since. I will report back in 1-2 days if that fixed it. Then it's up to us to find the query that takes too long. Thank You!

Comment by M Z [ 2020-03-18 ]

Unfortunately, the connection resets happened again this morning. 3 in a row. We probably have to wait for 2.6.0 to further debug this?

Comment by Diego Dupin [ 2020-03-18 ]

When server send an RST, MariaDB server logs the reason. I don't know the logs aurora offer, but i would recommend go that way first, but sure, 2.6.0 can help. (release will be probably end of this week, of the week after)

Comment by M Z [ 2020-03-18 ]

Unfortunately, I do not see any logging related to RST. Hopefully the fixed packet debugging will shed some light onto this.

Comment by Diego Dupin [ 2020-03-23 ]

hm, i'm thinking of another possibility : if your application use connection.abort.

Aurora send a fake thread id, on connection. Abort will kill the connection with this thread id.
With aurora that would mean a random number...

Do you use connection.abort ?

Comment by M Z [ 2020-03-23 ]

Not knowlingly, we do not work on Connection level. We use Spring JPA/Hibernate, but I'm not sure if these Frameworks use Connection.abort to be honest.

Comment by Diego Dupin [ 2020-03-24 ]

In fact this is more a pool thing, and i've forgot that you use hikaricp that only does abort when closing the pool which is only done for testing.

Comment by M Z [ 2020-03-31 ]

Due to the current situation, we are not allowed to deploy to the production environment. Luckily (or maybe un-luckily) this issue now also happened on our dev environment with the 2.6.0 Version. Unfortunately, the packetDebug only shows a bunch of cryptic numbers and select queries from a background job that worked 4 hours prior without issue. Is there anything else I can do to get to the bottom of this?

Comment by Diego Dupin [ 2020-03-31 ]

If you have some packetDebug please post them here. we never know, it might help.

Since server close the connection, the best thing is to check server logs. I don't know for aurora, but normally server logs why it close a connection.
Since you use aurora, aurora seems to use some proxy. I don't know how work this proxy, but since you indicate there is no server logs, that problem might reside in this proxy. And for that i might not help.

That might be completly wrong, but this is the first time i've seen a task for aurora with protocol compression.
I would test disabling compression.
this option is generally not to activate. It is recommended for "long distance" connections, and this is rarely the case for connections to a database where the application and data servers are in the same data center.

Comment by M Z [ 2020-04-01 ]

I'm afraid that I am probably not allowed to share the packetDebug logs due to company policy. I will ask around to check.
The compression is disabled in the application that faces these issues in the testing environment. We have at least 3 Applications that are affected by this, only one of them uses compression.

Comment by M Z [ 2020-04-16 ]

Below is a packetDebug log where I removed the query due to company policy:

Caused by: java.sql.SQLTransientConnectionException: Connection reset
 
thread:0 read at -exchange:59450- 2020-04-11T05:54:10.448Z
 
+--------------------------------------------------+
 
|  0  1  2  3  4  5  6  7   8  9  a  b  c  d  e  f |
 
+--------------------------------------------------+------------------+
 
| 8D 00 00 17 03 31 32 30  03 31 37 39 03 31 37 39 | .....120.179.179 |
 
| 03 31 32 30 01 31 03 30  2E 31 22 31 2E 33 2E 36 | .120.1.0.1"1.3.6 |
 
| 2E 31 2E 34 2E 31 2E 33  31 38 2E 31 2E 31 2E 31 | .1.4.1.318.1.1.1 |
 
| 30 2E 35 2E 31 2E 31 2E  31 2E 38 2E 32 22 31 2E | 0.5.1.1.1.8.2"1. |
 
| 33 2E 36 2E 31 2E 34 2E  31 2E 33 31 38 2E 31 2E | 3.6.1.4.1.318.1. |
 
| 31 2E 31 30 2E 35 2E 31  2E 31 2E 31 2E 35 2E 32 | 1.10.5.1.1.1.5.2 |
 
| 20 31 2E 33 2E 36 2E 31  2E 34 2E 31 2E 33 31 38 |  1.3.6.1.4.1.318 |
 
| 2E 31 2E 31 2E 31 30 2E  33 2E 31 32 2E 31 31 2E | .1.1.10.3.12.11. |
 
| 30 0F 57 69 72 65 6C 65  73 73 20 50 6F 72 74 20 | 0.Wireless Port  |
 
| 32                                               | 2                |
 
+--------------------------------------------------+------------------+
 
 
thread:0 read at -exchange:59451- 2020-04-11T05:54:10.448Z
 
+--------------------------------------------------+
 
|  0  1  2  3  4  5  6  7   8  9  a  b  c  d  e  f |
 
+--------------------------------------------------+------------------+
 
| 8D 00 00 18 03 31 32 30  03 31 38 30 03 31 38 30 | .....120.180.180 |
 
| 03 31 32 30 01 31 03 30  2E 31 22 31 2E 33 2E 36 | .120.1.0.1"1.3.6 |
 
| 2E 31 2E 34 2E 31 2E 33  31 38 2E 31 2E 31 2E 31 | .1.4.1.318.1.1.1 |
 
| 30 2E 35 2E 31 2E 31 2E  31 2E 38 2E 33 22 31 2E | 0.5.1.1.1.8.3"1. |
 
| 33 2E 36 2E 31 2E 34 2E  31 2E 33 31 38 2E 31 2E | 3.6.1.4.1.318.1. |
 
| 31 2E 31 30 2E 35 2E 31  2E 31 2E 31 2E 35 2E 33 | 1.10.5.1.1.1.5.3 |
 
| 20 31 2E 33 2E 36 2E 31  2E 34 2E 31 2E 33 31 38 |  1.3.6.1.4.1.318 |
 
| 2E 31 2E 31 2E 31 30 2E  33 2E 31 32 2E 31 31 2E | .1.1.10.3.12.11. |
 
| 30 0F 57 69 72 65 6C 65  73 73 20 50 6F 72 74 20 | 0.Wireless Port  |
 
| 33                                               | 3                |
 
+--------------------------------------------------+------------------+
 
 
thread:0 read at -exchange:59452- 2020-04-11T05:54:10.448Z
 
+--------------------------------------------------+
 
|  0  1  2  3  4  5  6  7   8  9  a  b  c  d  e  f |
 
+--------------------------------------------------+------------------+
 
| 8D 00 00 19 03 31 32 30  03 31 38 31 03 31 38 31 | .....120.181.181 |
 
| 03 31 32 30 01 31 03 30  2E 31 22 31 2E 33 2E 36 | .120.1.0.1"1.3.6 |
 
| 2E 31 2E 34 2E 31 2E 33  31 38 2E 31 2E 31 2E 31 | .1.4.1.318.1.1.1 |
 
| 30 2E 35 2E 31 2E 31 2E  31 2E 38 2E 34 22 31 2E | 0.5.1.1.1.8.4"1. |
 
| 33 2E 36 2E 31 2E 34 2E  31 2E 33 31 38 2E 31 2E | 3.6.1.4.1.318.1. |
 
| 31 2E 31 30 2E 35 2E 31  2E 31 2E 31 2E 35 2E 34 | 1.10.5.1.1.1.5.4 |
 
| 20 31 2E 33 2E 36 2E 31  2E 34 2E 31 2E 33 31 38 |  1.3.6.1.4.1.318 |
 
| 2E 31 2E 31 2E 31 30 2E  33 2E 31 32 2E 31 31 2E | .1.1.10.3.12.11. |
 
| 30 0F 57 69 72 65 6C 65  73 73 20 50 6F 72 74 20 | 0.Wireless Port  |
 
| 34                                               | 4                |
 
+--------------------------------------------------+------------------+
 
 
thread:0 read at -exchange:59453- 2020-04-11T05:54:10.448Z
 
+--------------------------------------------------+
 
|  0  1  2  3  4  5  6  7   8  9  a  b  c  d  e  f |
 
+--------------------------------------------------+------------------+
 
| 8D 00 00 1A 03 31 32 30  03 31 38 32 03 31 38 32 | .....120.182.182 |
 
| 03 31 32 30 01 31 03 30  2E 31 22 31 2E 33 2E 36 | .120.1.0.1"1.3.6 |
 
| 2E 31 2E 34 2E 31 2E 33  31 38 2E 31 2E 31 2E 31 | .1.4.1.318.1.1.1 |
 
| 30 2E 35 2E 31 2E 31 2E  31 2E 38 2E 35 22 31 2E | 0.5.1.1.1.8.5"1. |
 
| 33 2E 36 2E 31 2E 34 2E  31 2E 33 31 38 2E 31 2E | 3.6.1.4.1.318.1. |
 
| 31 2E 31 30 2E 35 2E 31  2E 31 2E 31 2E 35 2E 35 | 1.10.5.1.1.1.5.5 |
 
| 20 31 2E 33 2E 36 2E 31  2E 34 2E 31 2E 33 31 38 |  1.3.6.1.4.1.318 |
 
| 2E 31 2E 31 2E 31 30 2E  33 2E 31 32 2E 31 31 2E | .1.1.10.3.12.11. |
 
| 30 0F 57 69 72 65 6C 65  73 73 20 50 6F 72 74 20 | 0.Wireless Port  |
 
| 35                                               | 5                |
 
+--------------------------------------------------+------------------+
 
 
thread:0 read at -exchange:59454- 2020-04-11T05:54:10.448Z
 
+--------------------------------------------------+
 
|  0  1  2  3  4  5  6  7   8  9  a  b  c  d  e  f |
 
+--------------------------------------------------+------------------+
 
| 8D 00 00 1B 03 31 32 30  03 31 38 33 03 31 38 33 | .....120.183.183 |
 
| 03 31 32 30 01 31 03 30  2E 31 22 31 2E 33 2E 36 | .120.1.0.1"1.3.6 |
 
| 2E 31 2E 34 2E 31 2E 33  31 38 2E 31 2E 31 2E 31 | .1.4.1.318.1.1.1 |
 
| 30 2E 35 2E 31 2E 31 2E  31 2E 38 2E 36 22 31 2E | 0.5.1.1.1.8.6"1. |
 
| 33 2E 36 2E 31 2E 34 2E  31 2E 33 31 38 2E 31 2E | 3.6.1.4.1.318.1. |
 
| 31 2E 31 30 2E 35 2E 31  2E 31 2E 31 2E 35 2E 36 | 1.10.5.1.1.1.5.6 |
 
| 20 31 2E 33 2E 36 2E 31  2E 34 2E 31 2E 33 31 38 |  1.3.6.1.4.1.318 |
 
| 2E 31 2E 31 2E 31 30 2E  33 2E 31 32 2E 31 31 2E | .1.1.10.3.12.11. |
 
| 30 0F 57 69 72 65 6C 65  73 73 20 50 6F 72 74 20 | 0.Wireless Port  |
 
| 36                                               | 6                |
 
+--------------------------------------------------+------------------+
 
 
thread:0 read at -exchange:59455- 2020-04-11T05:54:10.448Z
 
+--------------------------------------------------+
 
|  0  1  2  3  4  5  6  7   8  9  a  b  c  d  e  f |
 
+--------------------------------------------------+------------------+
 
| 8D 00 00 1C 03 31 32 30  03 31 38 34 03 31 38 34 | .....120.184.184 |
 
| 03 31 32 30 01 31 03 30  2E 31 22 31 2E 33 2E 36 | .120.1.0.1"1.3.6 |
 
| 2E 31 2E 34 2E 31 2E 33  31 38 2E 31 2E 31 2E 31 | .1.4.1.318.1.1.1 |
 
| 30 2E 35 2E 31 2E 31 2E  31 2E 38 2E 37 22 31 2E | 0.5.1.1.1.8.7"1. |
 
| 33 2E 36 2E 31 2E 34 2E  31 2E 33 31 38 2E 31 2E | 3.6.1.4.1.318.1. |
 
| 31 2E 31 30 2E 35 2E 31  2E 31 2E 31 2E 35 2E 37 | 1.10.5.1.1.1.5.7 |
 
| 20 31 2E 33 2E 36 2E 31  2E 34 2E 31 2E 33 31 38 |  1.3.6.1.4.1.318 |
 
| 2E 31 2E 31 2E 31 30 2E  33 2E 31 32 2E 31 31 2E | .1.1.10.3.12.11. |
 
| 30 0F 57 69 72 65 6C 65  73 73 20 50 6F 72 74 20 | 0.Wireless Port  |
 
| 37                                               | 7                |
 
+--------------------------------------------------+------------------+
 
 
thread:0 read at -exchange:59456- 2020-04-11T05:54:10.448Z
 
+--------------------------------------------------+
 
|  0  1  2  3  4  5  6  7   8  9  a  b  c  d  e  f |
 
+--------------------------------------------------+------------------+
 
| 8D 00 00 1D 03 31 32 30  03 31 38 35 03 31 38 35 | .....120.185.185 |
 
| 03 31 32 30 01 31 03 30  2E 31 22 31 2E 33 2E 36 | .120.1.0.1"1.3.6 |
 
| 2E 31 2E 34 2E 31 2E 33  31 38 2E 31 2E 31 2E 31 | .1.4.1.318.1.1.1 |
 
| 30 2E 35 2E 31 2E 31 2E  31 2E 38 2E 38 22 31 2E | 0.5.1.1.1.8.8"1. |
 
| 33 2E 36 2E 31 2E 34 2E  31 2E 33 31 38 2E 31 2E | 3.6.1.4.1.318.1. |
 
| 31 2E 31 30 2E 35 2E 31  2E 31 2E 31 2E 35 2E 38 | 1.10.5.1.1.1.5.8 |
 
| 20 31 2E 33 2E 36 2E 31  2E 34 2E 31 2E 33 31 38 |  1.3.6.1.4.1.318 |
 
| 2E 31 2E 31 2E 31 30 2E  33 2E 31 32 2E 31 31 2E | .1.1.10.3.12.11. |
 
| 30 0F 57 69 72 65 6C 65  73 73 20 50 6F 72 74 20 | 0.Wireless Port  |
 
| 38                                               | 8                |
 
+--------------------------------------------------+------------------+
 
 
thread:0 read at -exchange:59457- 2020-04-11T05:54:10.448Z
 
+--------------------------------------------------+
 
|  0  1  2  3  4  5  6  7   8  9  a  b  c  d  e  f |
 
+--------------------------------------------------+------------------+
 
| 8D 00 00 1E 03 31 32 30  03 31 38 36 03 31 38 36 | .....120.186.186 |
 
| 03 31 32 30 01 31 03 30  2E 31 22 31 2E 33 2E 36 | .120.1.0.1"1.3.6 |
 
| 2E 31 2E 34 2E 31 2E 33  31 38 2E 31 2E 31 2E 31 | .1.4.1.318.1.1.1 |
 
| 30 2E 35 2E 31 2E 31 2E  31 2E 38 2E 39 22 31 2E | 0.5.1.1.1.8.9"1. |
 
| 33 2E 36 2E 31 2E 34 2E  31 2E 33 31 38 2E 31 2E | 3.6.1.4.1.318.1. |
 
| 31 2E 31 30 2E 35 2E 31  2E 31 2E 31 2E 35 2E 39 | 1.10.5.1.1.1.5.9 |
 
| 20 31 2E 33 2E 36 2E 31  2E 34 2E 31 2E 33 31 38 |  1.3.6.1.4.1.318 |
 
| 2E 31 2E 31 2E 31 30 2E  33 2E 31 32 2E 31 31 2E | .1.1.10.3.12.11. |
 
| 30 0F 57 69 72 65 6C 65  73 73 20 50 6F 72 74 20 | 0.Wireless Port  |
 
| 39                                               | 9                |
 
+--------------------------------------------------+------------------+
 
 
thread:0 read at -exchange:59458- 2020-04-11T05:54:10.448Z
 
+--------------------------------------------------+
 
|  0  1  2  3  4  5  6  7   8  9  a  b  c  d  e  f |
 
+--------------------------------------------------+------------------+
 
| 07 00 00 1F FE 00 00 01  00 00 00                | ...........      |
 
+--------------------------------------------------+------------------+
 
 
thread:0 send at -exchange:59459- 2020-04-11T05:54:17.887Z
 
+--------------------------------------------------+
 
|  0  1  2  3  4  5  6  7   8  9  a  b  c  d  e  f |
 
+--------------------------------------------------+------------------+
 
a rather long select was here that I removed due to company policy
 
+--------------------------------------------------+------------------+

And these are the parameters we use for our connection, maybe we configure something wrong? Because it really happens in all our applications that recently switched to MariaDB and they all use very similar parameters:

"loginTimeout", "0"
 
"serverTimezone", "UTC"
 
"cachePrepStmts", "true"
 
"prepStmtCacheSize", "500"
 
"prepStmtCacheSqlLimit", "2048"
 
"useServerPrepStmts", "true"
 
"rewriteBatchedStatements", "true"
 
"cacheServerConfiguration", "true"
 
"autoReconnect", "true"
 
"useUnicode", "true"
 
"connectionCollation", "utf8_general_ci"
 
"zeroDateTimeBehavior", "convertToNull"
 
"enablePacketDebug", "true"

Comment by M Z [ 2020-04-16 ]

Another interesting thing we just found in the audit log of aurora:
~5 minutes after the connection reset exception occured, the audit log logged this for the same connection id:

2020-04-11T05:59:10.726Z...2174288,0,DISCONNECT,...,,0

reset timestamp:

2020-04-11T05:54:17Z Caused by: java.sql.SQLTransientConnectionException: (conn=2174288) Connection reset

Does that mean the connection reset happened on client side?

Comment by M Z [ 2020-04-23 ]

Gentle ping

Comment by M Z [ 2020-05-25 ]

We have 'reverted' back to the mysql connector and are now seeeing:
Caused by: com.mysql.jdbc.exceptions.jdbc4.MySQLNonTransientConnectionException: Communications link failure
instead of connection resets. So imho it is not a MariaDB issue. This can probably be closed.

Comment by Diego Dupin [ 2022-08-16 ]

closing as requested

Generated at Thu Feb 08 03:18:11 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.