Details
-
Bug
-
Status: Closed (View Workflow)
-
Critical
-
Resolution: Incomplete
-
1.5.9
-
None
-
Aurora
Description
Hi all,
I'm running Aurora failover testing with the help of MariaDB connector recently and discovered this weird issue.
Preparation:
- Aurora cluster with 2 instances: test-01 and test-02, cluster end point is test-cluster.
- jdbcUrl="jdbc:mysql:aurora://test-cluster.rds.amazonaws.com:3306/Test?log=true"
- Using MariaDB connector version 1.5.9 due to concern on usePipeAuth.
- Java server application have all kinds of queries hitting db server.
Test1:
- test-01 is the writer and test-02 is the reader.
- launch server application.
Result: fine.
(Clean up by terminating server application)
Test2:
- Click failover in AWS console for test cluster.
- Wait until it completes: test-02 is the writer and test-01 is the reader
- launch server application
Result: getting exception.
(Clean up by terminating server application)
Exception:
Caused by: java.lang.ArrayIndexOutOfBoundsException: -10 |
at org.mariadb.jdbc.internal.util.buffer.Buffer.readShort(Buffer.java:98) ~[server.jar:?] |
at org.mariadb.jdbc.internal.packet.dao.ColumnInformation.<init>(ColumnInformation.java:148) ~[server.jar:?] |
at org.mariadb.jdbc.internal.protocol.AbstractQueryProtocol.readResultSet(AbstractQueryProtocol.java:1195) ~[server.jar:?] |
at org.mariadb.jdbc.internal.protocol.AbstractQueryProtocol.readPacket(AbstractQueryProtocol.java:1089) ~[server.jar:?] |
at org.mariadb.jdbc.internal.protocol.AbstractQueryProtocol.getResult(AbstractQueryProtocol.java:1031) ~[server.jar:?] |
at org.mariadb.jdbc.internal.protocol.AbstractQueryProtocol.executeQuery(AbstractQueryProtocol.java:203) ~[server.jar:?] |
at sun.reflect.GeneratedMethodAccessor25.invoke(Unknown Source) ~[?:?]
|
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_121] |
at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_121] |
at org.mariadb.jdbc.internal.failover.AbstractMastersListener.invoke(AbstractMastersListener.java:355) ~[server.jar:?] |
at org.mariadb.jdbc.internal.failover.FailoverProxy.executeInvocation(FailoverProxy.java:200) ~[server.jar:?] |
at org.mariadb.jdbc.internal.failover.FailoverProxy.invoke(FailoverProxy.java:192) ~[server.jar:?] |
at com.sun.proxy.$Proxy42.executeQuery(Unknown Source) ~[?:?]
|
at org.mariadb.jdbc.MariaDbClientPreparedStatement.executeInternal(MariaDbClientPreparedStatement.java:224) ~[server.jar:?] |
at org.mariadb.jdbc.MariaDbClientPreparedStatement.executeQuery(MariaDbClientPreparedStatement.java:177) ~[server.jar:?] |
Note:
- Tested on the same environment with com.mysql.jdbc driver and removed "aurora" from jdbc url and things are working fine => issue comes from mariadb driver.
- My server application has nothing instance specific: the only information aurora specific is the jdbc url and it was using cluster end point
- For the query that triggered this, I've compared the table description and data between test-01 and test-02 and they are identical.
- Query="select * from table_timestamp"
- Table description(table_timestamp):
+--------------+------------------+------+-----+-------------------+-----------------------------+
| Field | Type | Null | Key | Default | Extra |
+--------------+------------------+------+-----+-------------------+-----------------------------+
| last_updated | timestamp | NO | | CURRENT_TIMESTAMP | on update CURRENT_TIMESTAMP |
| update_count | int(10) unsigned | NO | | 0 | |
+--------------+------------------+------+-----+-------------------+-----------------------------+
2 rows in set (0.00 sec)
- Manually tested the same query using mysql command line, getting identical results from test-01 and test-02
- I also got another different exception, but it's from the same query.
java.sql.SQLException: (conn:4027) Packets out of order when reading field packets, expected was EOF stream. Packet contents (hex) = 03 |
Query is: CALL Test.sp_get_timestamp()
|
at org.mariadb.jdbc.internal.util.ExceptionMapper.get(ExceptionMapper.java:150) ~[server.jar:?] |
at org.mariadb.jdbc.internal.util.ExceptionMapper.getException(ExceptionMapper.java:101) ~[server.jar:?] |
at org.mariadb.jdbc.internal.util.ExceptionMapper.throwAndLogException(ExceptionMapper.java:77) ~[server.jar:?] |
at org.mariadb.jdbc.MariaDbStatement.executeQueryEpilog(MariaDbStatement.java:226) ~[server.jar:?] |
at org.mariadb.jdbc.MariaDbServerPreparedStatement.executeInternal(MariaDbServerPreparedStatement.java:413) ~[server.jar:?] |
at org.mariadb.jdbc.MariaDbProcedureStatement.execute(MariaDbProcedureStatement.java:168) ~[server.jar:?] |
at org.mariadb.jdbc.MariaDbServerPreparedStatement.executeQuery(MariaDbServerPreparedStatement.java:343) ~[server.jar:?] |
- The sproc is as simple as one select query as listed above.
BEGINselect * from gift_timestamp;
END
I've tested with other slightly older versions as well and have seen the same result.
Can someone help me out here? Many thanks in advance!
Please don't hesitate to reach out to me if you need more information. I'm subscribed to this thread and should be updated if you leave a comment.
Update
With the same test I'm also seeing this exception periodically:
Caused by: org.mariadb.jdbc.internal.util.dao.QueryException: Access denied for user 'someusername'@'10.%' (using password: YES) |
Query is: CALL Test.sp_get_timestamp()
|
at org.mariadb.jdbc.internal.protocol.AbstractQueryProtocol.readErrorPacket(AbstractQueryProtocol.java:1163) ~[server.jar:?] |
at org.mariadb.jdbc.internal.protocol.AbstractQueryProtocol.readPacket(AbstractQueryProtocol.java:1093) ~[server.jar:?] |
at org.mariadb.jdbc.internal.protocol.AbstractQueryProtocol.getResult(AbstractQueryProtocol.java:1044) ~[server.jar:?] |
at org.mariadb.jdbc.internal.protocol.AbstractQueryProtocol.executePreparedQuery(AbstractQueryProtocol.java:628) ~[server.jar:?] |
at sun.reflect.GeneratedMethodAccessor16.invoke(Unknown Source) ~[?:?]
|
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_121] |
at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_121] |
at org.mariadb.jdbc.internal.failover.AbstractMastersListener.invoke(AbstractMastersListener.java:351) ~[server.jar:?] |
at org.mariadb.jdbc.internal.failover.FailoverProxy.invoke(FailoverProxy.java:156) ~[server.jar:?] |
at com.sun.proxy.$Proxy43.executePreparedQuery(Unknown Source) ~[?:?]
|
at org.mariadb.jdbc.MariaDbServerPreparedStatement.executeInternal(MariaDbServerPreparedStatement.java:401) ~[server.jar:?] |
... 36 more |
Note:
This is solvable by restarting the server several times but it will be a blocker for me to deploy mariadb driver to production.
Can someone fix this as well?
Attachments
Activity
I agree with you. what format of tcpdump result are you most comfortable with? if you have a preferred group of tcpdump args, please let me know.
any will do with the -X option (packet content with hexa and ASCII) if i remember that right
sry for the late reply. I was out of office and did not get a chance to do this.
my test environment was tore apart and I need to set those up again to reproduce this issue. will attach tcpdump info when I have that.
I cannot point to a specific place, the method "executeQuery" in the error log is one of the most used method.
if you could post the tcpdump / log with trace level here, that would be great. That would permit to identify that place.
Because strangely it seems a protocol issue.