[CONJ-726] NullPointerException after failover happening on aurora cluster Created: 2019-08-29  Updated: 2019-09-13  Resolved: 2019-09-13

Status: Closed
Project: MariaDB Connector/J
Component/s: aurora, Failover
Affects Version/s: 2.3.0, 2.4.1, 2.4.2
Fix Version/s: 2.4.4

Type: Bug Priority: Major
Reporter: Ionut Ichim Assignee: Diego Dupin
Resolution: Fixed Votes: 0
Labels: None
Environment:

Java 8, Windows 10, AWS, Aurora MySql 5.7



 Description   

When failover happens on aurora db cluster, no new connection can be initiated to the db, each attempt generating NPE.

This is the failover triggered exception:
java.sql.SQLException: (conn=322421) Communications link failure with secondary host xxxxx.amazonaws.com:3306. Connection timed out. Driver has reconnect connection
at org.mariadb.jdbc.internal.util.exceptions.ExceptionMapper.get(ExceptionMapper.java:264)
at org.mariadb.jdbc.internal.util.exceptions.ExceptionMapper.getException(ExceptionMapper.java:171)
at org.mariadb.jdbc.MariaDbStatement.executeExceptionEpilogue(MariaDbStatement.java:248)
at org.mariadb.jdbc.ServerSidePreparedStatement.executeInternal(ServerSidePreparedStatement.java:432)
at org.mariadb.jdbc.ServerSidePreparedStatement.execute(ServerSidePreparedStatement.java:386)
at org.mariadb.jdbc.ServerSidePreparedStatement.executeQuery(ServerSidePreparedStatement.java:365)
....
Caused by: java.sql.SQLException: Communications link failure with secondary host xxxxx.amazonaws.com:3306. Connection timed out. Driver has reconnect connection
at org.mariadb.jdbc.internal.failover.AbstractMastersListener.throwFailoverMessage(AbstractMastersListener.java:545)
at org.mariadb.jdbc.internal.failover.FailoverProxy.handleFailOver(FailoverProxy.java:367)
at org.mariadb.jdbc.internal.failover.FailoverProxy.invoke(FailoverProxy.java:238)
at com.sun.proxy.$Proxy13.executePreparedQuery(Unknown Source)
at org.mariadb.jdbc.ServerSidePreparedStatement.executeInternal(ServerSidePreparedStatement.java:425)
... 26 more
Caused by: java.sql.SQLNonTransientConnectionException: Read timed out
at org.mariadb.jdbc.internal.protocol.AbstractQueryProtocol.handleIoException(AbstractQueryProtocol.java:1925)
at org.mariadb.jdbc.internal.protocol.AbstractQueryProtocol.readPacket(AbstractQueryProtocol.java:1437)
at org.mariadb.jdbc.internal.protocol.AbstractQueryProtocol.getResult(AbstractQueryProtocol.java:1415)
at org.mariadb.jdbc.internal.protocol.AbstractQueryProtocol.executePreparedQuery(AbstractQueryProtocol.java:1050)
at sun.reflect.GeneratedMethodAccessor31.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.mariadb.jdbc.internal.failover.AbstractMastersListener.invoke(AbstractMastersListener.java:401)
at org.mariadb.jdbc.internal.failover.FailoverProxy.invoke(FailoverProxy.java:233)
... 28 more
Caused by: java.net.SocketTimeoutException: Read timed out
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
at java.net.SocketInputStream.read(SocketInputStream.java:171)
at java.net.SocketInputStream.read(SocketInputStream.java:141)
at java.io.FilterInputStream.read(FilterInputStream.java:133)
at org.mariadb.jdbc.internal.io.input.ReadAheadBufferedStream.fillBuffer(ReadAheadBufferedStream.java:130)
at org.mariadb.jdbc.internal.io.input.ReadAheadBufferedStream.read(ReadAheadBufferedStream.java:103)
at org.mariadb.jdbc.internal.io.input.StandardPacketInputStream.getPacketArray(StandardPacketInputStream.java:244)
at org.mariadb.jdbc.internal.io.input.StandardPacketInputStream.getPacket(StandardPacketInputStream.java:215)
at org.mariadb.jdbc.internal.protocol.AbstractQueryProtocol.readPacket(AbstractQueryProtocol.java:1435)

And then, on any attempt to reconnect:
java.lang.NullPointerException
at org.mariadb.jdbc.internal.protocol.AbstractConnectProtocol.setHostAddress(AbstractConnectProtocol.java:1168)
at org.mariadb.jdbc.internal.protocol.AuroraProtocol.loop(AuroraProtocol.java:166)
at org.mariadb.jdbc.internal.failover.impl.AuroraListener.reconnectFailedConnection(AuroraListener.java:210)
at org.mariadb.jdbc.internal.failover.impl.MastersSlavesListener.initializeConnection(MastersSlavesListener.java:170)
at org.mariadb.jdbc.internal.failover.FailoverProxy.<init>(FailoverProxy.java:116)
at org.mariadb.jdbc.internal.util.Utils.retrieveProxy(Utils.java:543)
at org.mariadb.jdbc.MariaDbConnection.newConnection(MariaDbConnection.java:175)
at org.mariadb.jdbc.MariaDbDataSource.getConnection(MariaDbDataSource.java:288)
at org.mariadb.jdbc.MariaDbDataSource.getPooledConnection(MariaDbDataSource.java:451)

This is happening on 90% of failover cases and it requires application restart as the driver does not recover from this state.

Thanks,
Ionut



 Comments   
Comment by Diego Dupin [ 2019-08-30 ]

I fail to see what can cause this for now.
Could you indicate how many servers do you have in your environment?

Comment by Ionut Ichim [ 2019-08-30 ]

Hi,
We have 2 servers, one writer and one read-only replica.

Generated at Thu Feb 08 03:17:52 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.