Uploaded image for project: 'MariaDB Connector/J'
  1. MariaDB Connector/J
  2. CONJ-651

Stream LOB columns from the socket instead of loading them in memory

    XMLWordPrintable

Details

    • Bug
    • Status: Closed (View Workflow)
    • Major
    • Resolution: Fixed
    • 2.3.0
    • 4.0.0
    • Other
    • None
    • Java 8, Windows/RedHat

    Description

      ResultSet.getCharacterStream(), getBinaryStream(), getBlob() and getClob() only return a stream once the whole field, and in fact the whole row, has been loaded in memory. Reading a large LOB requires a heap several times its size and ends with OutOfMemoryError. The returned stream is misleading: nothing is streamed. Same root cause as CONJ-1146.

      Solution

      A statement with fetch size Integer.MIN_VALUE (it passes through Hibernate's org.hibernate.fetchSize hint and Spring JdbcTemplate) returns a "sequential access" result-set:

      • rows are read from the socket one field at a time, never kept in memory,
      • getBinaryStream(), getBlob(), getCharacterStream(), getClob(), getNClob(), getAsciiStream() and getObject() with those classes return objects that stream the field bytes directly from the socket, including for values split over several 16 MB packets,
      • other getters load only the requested field.

      Enabled per statement with stmt.setFetchSize(Integer.MIN_VALUE) or for all statements with the defaultFetchSize=-2147483648 connection option.

      Restrictions (comparable to .NET CommandBehavior.SequentialAccess)

      • fields must be read in increasing column index order,
      • a field returned as a stream can be read once; moving to another field or row skips what remains and invalidates the stream,
      • forward-only, read-only result-sets only: ignored for scrollable or updatable statements, refused on callable statements,
      • Clob.length() needs the character count, so it loads the remaining content in memory,
      • Blob/Clob are read-only and forward-only (position() and setXxx() are not supported).

      Executing another command on the connection while such a result-set is open loads the remaining rows in memory, as for other streaming result-sets, and the result-set stays usable.

      Attachments

        Issue Links

          Activity

            People

              diego dupin Diego Dupin
              squadgazzz Ilya Zhavoronkov
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Git Integration

                  Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.