Uploaded image for project: 'MariaDB Connector/J'
  1. MariaDB Connector/J
  2. CONJ-528

Error executing LOAD DATA LOCAL INFILE when file is larger than max_allowed_packet

    XMLWordPrintable

Details

    • Bug
    • Status: Closed (View Workflow)
    • Minor
    • Resolution: Fixed
    • 2.1.1
    • 1.6.5, 2.1.2
    • Other
    • None

    Description

      When loading a local file larger than max_allowed_packet it fails:

      java.sql.SQLException: (conn=77811) Could not send query:...
      java.sql.SQLException: (conn=77811) Could not send query: query size is >= to max_allowed_packet (16777216)
      	at org.mariadb.jdbc.internal.util.exceptions.ExceptionMapper.get(ExceptionMapper.java:179)
      	at org.mariadb.jdbc.internal.util.exceptions.ExceptionMapper.getException(ExceptionMapper.java:118)
      	at org.mariadb.jdbc.MariaDbStatement.executeExceptionEpilogue(MariaDbStatement.java:235)
      	at org.mariadb.jdbc.MariaDbPreparedStatementClient.executeInternal(MariaDbPreparedStatementClient.java:224)
      	at org.mariadb.jdbc.MariaDbPreparedStatementClient.execute(MariaDbPreparedStatementClient.java:159)
      	at org.mariadb.jdbc.MariaDbPreparedStatementClient.executeUpdate(MariaDbPreparedStatementClient.java:192)
      	at com.inditex.distribucion.cloud.mariadb.ClientStandalone.executeUpdate(ClientStandalone.java:48)
      	at com.inditex.distribucion.cloud.mariadb.ClientStandalone.main(ClientStandalone.java:116)
      Caused by: java.sql.SQLException: Could not send query: query size is >= to max_allowed_packet (16777216)
      Query is: LOAD DATA LOCAL INFILE 'c://pub/file.dat' IGNORE INTO TABLE tmp.data CHARACTER SET utf8 FIELDS TERMINATED BY '|'
      	at org.mariadb.jdbc.internal.util.LogQueryTool.exceptionWithQuery(LogQueryTool.java:146)
      	at org.mariadb.jdbc.internal.protocol.AbstractQueryProtocol.executeQuery(AbstractQueryProtocol.java:217)
      	at org.mariadb.jdbc.MariaDbPreparedStatementClient.executeInternal(MariaDbPreparedStatementClient.java:218)
      	... 4 more
      Caused by: org.mariadb.jdbc.internal.util.exceptions.MaxAllowedPacketException: query size (22021201) is >= to max_allowed_packet (16777216)
      	at org.mariadb.jdbc.internal.io.output.AbstractPacketOutputStream.flush(AbstractPacketOutputStream.java:179)
      	at org.mariadb.jdbc.internal.protocol.AbstractQueryProtocol.readLocalInfilePacket(AbstractQueryProtocol.java:1491)
      	at org.mariadb.jdbc.internal.protocol.AbstractQueryProtocol.readPacket(AbstractQueryProtocol.java:1297)
      	at org.mariadb.jdbc.internal.protocol.A 
      

      I changed class org.mariadb.jdbc.internal.protocol.AbstractQueryProtocol
      method readLocalInfilePacket line ~1489 where the fileInputStream is read, to read it in chunks and send it in several packets of 1Mb size instead of sending it in only one.

      Instead of

         writer.write(is, false, false);
         writer.flush();
      

      send it in chunks:

                      byte[] buff = new byte[1024 * 1024];  //arbitrary size
                      int read = 0;
                      while ((read = is.read(buff)) != -1) {
                          writer.startPacket(seq);
                          writer.write(buff, 0, read);
                          writer.flush();
                          seq ++;
                      }  
      

      There is a test LocalInfileInputStreamTest test2xMaxAllowedPacketLocalInfileInputStream that explicitally checks that it must fail, I don´t understand why:

      try {
                  checkBigLocalInfile(maxAllowedPacket * 2);
                  fail("must have fail");
              } catch (SQLException sqle) {
                  assertTrue(sqle.getMessage().contains("Could not send query: query size ")
                          && sqle.getMessage().contains(" is >= to max_allowed_packet"));
              }
      

      Patch attached, thanks.

      Attachments

        Activity

          People

            diego dupin Diego Dupin
            rafalopez79 Rafael Lopez Fernández
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Git Integration

                Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.