Details
-
Bug
-
Status: Closed (View Workflow)
-
Minor
-
Resolution: Fixed
-
2.1.1
-
None
Description
When loading a local file larger than max_allowed_packet it fails:
java.sql.SQLException: (conn=77811) Could not send query:...
|
java.sql.SQLException: (conn=77811) Could not send query: query size is >= to max_allowed_packet (16777216)
|
at org.mariadb.jdbc.internal.util.exceptions.ExceptionMapper.get(ExceptionMapper.java:179)
|
at org.mariadb.jdbc.internal.util.exceptions.ExceptionMapper.getException(ExceptionMapper.java:118)
|
at org.mariadb.jdbc.MariaDbStatement.executeExceptionEpilogue(MariaDbStatement.java:235)
|
at org.mariadb.jdbc.MariaDbPreparedStatementClient.executeInternal(MariaDbPreparedStatementClient.java:224)
|
at org.mariadb.jdbc.MariaDbPreparedStatementClient.execute(MariaDbPreparedStatementClient.java:159)
|
at org.mariadb.jdbc.MariaDbPreparedStatementClient.executeUpdate(MariaDbPreparedStatementClient.java:192)
|
at com.inditex.distribucion.cloud.mariadb.ClientStandalone.executeUpdate(ClientStandalone.java:48)
|
at com.inditex.distribucion.cloud.mariadb.ClientStandalone.main(ClientStandalone.java:116)
|
Caused by: java.sql.SQLException: Could not send query: query size is >= to max_allowed_packet (16777216)
|
Query is: LOAD DATA LOCAL INFILE 'c://pub/file.dat' IGNORE INTO TABLE tmp.data CHARACTER SET utf8 FIELDS TERMINATED BY '|'
|
at org.mariadb.jdbc.internal.util.LogQueryTool.exceptionWithQuery(LogQueryTool.java:146)
|
at org.mariadb.jdbc.internal.protocol.AbstractQueryProtocol.executeQuery(AbstractQueryProtocol.java:217)
|
at org.mariadb.jdbc.MariaDbPreparedStatementClient.executeInternal(MariaDbPreparedStatementClient.java:218)
|
... 4 more
|
Caused by: org.mariadb.jdbc.internal.util.exceptions.MaxAllowedPacketException: query size (22021201) is >= to max_allowed_packet (16777216)
|
at org.mariadb.jdbc.internal.io.output.AbstractPacketOutputStream.flush(AbstractPacketOutputStream.java:179)
|
at org.mariadb.jdbc.internal.protocol.AbstractQueryProtocol.readLocalInfilePacket(AbstractQueryProtocol.java:1491)
|
at org.mariadb.jdbc.internal.protocol.AbstractQueryProtocol.readPacket(AbstractQueryProtocol.java:1297)
|
at org.mariadb.jdbc.internal.protocol.A
|
I changed class org.mariadb.jdbc.internal.protocol.AbstractQueryProtocol
method readLocalInfilePacket line ~1489 where the fileInputStream is read, to read it in chunks and send it in several packets of 1Mb size instead of sending it in only one.
Instead of
writer.write(is, false, false); |
writer.flush();
|
send it in chunks:
byte[] buff = new byte[1024 * 1024]; //arbitrary size |
int read = 0; |
while ((read = is.read(buff)) != -1) { |
writer.startPacket(seq);
|
writer.write(buff, 0, read); |
writer.flush();
|
seq ++;
|
}
|
There is a test LocalInfileInputStreamTest test2xMaxAllowedPacketLocalInfileInputStream that explicitally checks that it must fail, I don´t understand why:
try { |
checkBigLocalInfile(maxAllowedPacket * 2); |
fail("must have fail"); |
} catch (SQLException sqle) { |
assertTrue(sqle.getMessage().contains("Could not send query: query size ") |
&& sqle.getMessage().contains(" is >= to max_allowed_packet")); |
}
|
Patch attached, thanks.