Details

    • Bug
    • Status: Closed (View Workflow)
    • Major
    • Resolution: Fixed
    • 1.4.2, 1.4.6
    • 2.0.0-RC
    • Other
    • None
    • Windows 7

    Description

      The MariaDB J connector documentation says:

      Streaming result sets
      By default, Statement.executeQuery() will read the full result set from the server before returning. With large result sets, this will require large amounts of memory. Better behavior in this case would be reading row-by-row, with ResultSet.next(), so called "streaming". This is activated using Statement.setFetchSize(Integer.MIN_VALUE)

      The problem is that if an application decides to stop reading the result set, the connector still insists to fetch all the rows of the result set before closing it. This can make very long queries with big tables even a small part of it is useful.

      The here attached application JdbcCancel.java demonstrates this. It is written to read and dispay the first rows of a big table (the original one has 10 millions rows) until a row containing a "-" in the rln column is met.

      In previous versions of the MySQL J connector (for instance version 5.1.17) a turnaround was to cancel the statement before closing it. Unfortunately later versions (for instance 6.0.2) as well as the MariaDB J connector (I tested versions 1.4.2 and 1.4.6) fail to stop the reading. Even if a cancel command is executed the reading continues.

      Here is the console output of the program execution:

      Connection org.mariadb.jdbc.MariaDbConnection@20c684 established
      recordnb, diagnb, rln, diag, dp
      339858, 5, A49U9MQ94, 0, 0
      985921, 1, LFFXXDGOL, 0, 0
      1441640, 1, 8W887WRWG, 0, 0
      1760557, 0, 2MMXOF1JU, 0, 0
      1760558, 0, 5DDDSJDTD, 0, 0
      1760559, 0, 7UQQS7SKK, 0, 0
      Canceling query
      Closing statement
      Statement closed
      

      All connectors display the same output, the difference is that between the last two lines, more than 3 minutes are spent reading the rest of the file by the new versions.

      Trying the cancel the statement from another thread does not work either.

      Attached is the java application and the first 100 rows of the file that was used the make the diag table.

      Attachments

        Issue Links

          Activity

            bertrandop Olivier Bertrand created issue -
            bertrandop Olivier Bertrand made changes -
            Field Original Value New Value
            Attachment JdbcCancel.java [ 42219 ]
            bertrandop Olivier Bertrand made changes -
            Attachment diagsamp.txt [ 42220 ]
            bertrandop Olivier Bertrand made changes -
            Description The MariaDB J connector documentation says:

            ??*Streaming result sets*??
            ??By default, Statement.executeQuery() will read the full result set from the server before returning. With large result sets, this will require large amounts of memory. Better behavior in this case would be reading row-by-row, with ResultSet.next(), so called "streaming". This is activated using Statement.setFetchSize(Integer.MIN_VALUE)??

            The problem is that if an application decides to stop reading the result set, the connector still insists to fetch all the rows of the result set before closing it. This can make very long queries with big tables even a small part of it is useful.

            The here attached application _JdbcCancel.java_ demonstrates this. It is written to read and dispay the first rows of a big table (the original one has 10 millions rows) until a row containing a "-" in the _rln_ column is met.

            In previous versions of the MySQL J connector (for instance version 5.1.17) a turnaround was to cancel the statement before closing it. Unfortunately later versions (for instance 6.0.2) as well as the MariaDB J connector (I tested versions 1.4.2 and 1.4.6) fail to stop the reading. Even the _cancel_ command is executed the reading continues.

            Here is the console output of the program execution:
            {code}
            Connection org.mariadb.jdbc.MariaDbConnection@20c684 established
            recordnb, diagnb, rln, diag, dp
            339858, 5, A49U9MQ94, 0, 0
            985921, 1, LFFXXDGOL, 0, 0
            1441640, 1, 8W887WRWG, 0, 0
            1760557, 0, 2MMXOF1JU, 0, 0
            1760558, 0, 5DDDSJDTD, 0, 0
            1760559, 0, 7UQQS7SKK, 0, 0
            Canceling query
            Closing statement
            Statement closed
            {code}
            All connectors diplay the same output, the difference is that between the last two lines, more than 3 minutes are spent reading the rest of the file by the new versions.

            Trying the cancel the statement from another thread does not work either.

            Attached is the java application and the first 100 rows of the file that was used the make the _diag_ table.
             
            The MariaDB J connector documentation says:

            ??*Streaming result sets*??
            ??By default, Statement.executeQuery() will read the full result set from the server before returning. With large result sets, this will require large amounts of memory. Better behavior in this case would be reading row-by-row, with ResultSet.next(), so called "streaming". This is activated using Statement.setFetchSize(Integer.MIN_VALUE)??

            The problem is that if an application decides to stop reading the result set, the connector still insists to fetch all the rows of the result set before closing it. This can make very long queries with big tables even a small part of it is useful.

            The here attached application _JdbcCancel.java_ demonstrates this. It is written to read and dispay the first rows of a big table (the original one has 10 millions rows) until a row containing a "-" in the _rln_ column is met.

            In previous versions of the MySQL J connector (for instance version 5.1.17) a turnaround was to cancel the statement before closing it. Unfortunately later versions (for instance 6.0.2) as well as the MariaDB J connector (I tested versions 1.4.2 and 1.4.6) fail to stop the reading. Even if a _cancel_ command is executed the reading continues.

            Here is the console output of the program execution:
            {code}
            Connection org.mariadb.jdbc.MariaDbConnection@20c684 established
            recordnb, diagnb, rln, diag, dp
            339858, 5, A49U9MQ94, 0, 0
            985921, 1, LFFXXDGOL, 0, 0
            1441640, 1, 8W887WRWG, 0, 0
            1760557, 0, 2MMXOF1JU, 0, 0
            1760558, 0, 5DDDSJDTD, 0, 0
            1760559, 0, 7UQQS7SKK, 0, 0
            Canceling query
            Closing statement
            Statement closed
            {code}
            All connectors display the same output, the difference is that between the last two lines, more than 3 minutes are spent reading the rest of the file by the new versions.

            Trying the cancel the statement from another thread does not work either.

            Attached is the java application and the first 100 rows of the file that was used the make the _diag_ table.
             

            This is not addressed by the new version 1.5.0. Any chance to have it considered for the next one?

            bertrandop Olivier Bertrand added a comment - This is not addressed by the new version 1.5.0. Any chance to have it considered for the next one?
            diego dupin Diego Dupin made changes -
            Fix Version/s 1.6.0 [ 22119 ]
            diego dupin Diego Dupin added a comment -

            Not sure this can be done.
            There is some protocol solution for that (COM_STMT_FETCH), but this solution is a lot slower than actual implementation, and does break in 5.7 and 10.2.

            Best solution would be that application handle that with LIMIT and OFFSET. See also pagination

            diego dupin Diego Dupin added a comment - Not sure this can be done. There is some protocol solution for that (COM_STMT_FETCH), but this solution is a lot slower than actual implementation, and does break in 5.7 and 10.2. Best solution would be that application handle that with LIMIT and OFFSET . See also pagination
            diego dupin Diego Dupin made changes -
            Fix Version/s 2.0.0-RC [ 22119 ]

            The same problem occurs when limit is used.

            bertrandop Olivier Bertrand added a comment - The same problem occurs when limit is used.

            Not sure this can be done
            Come on! Other JDBC drivers do it and older version of the MySQL JDBC driver also did it. Just do the same. It is not normal to continue reading a table when the query is stopped...

            bertrandop Olivier Bertrand added a comment - Not sure this can be done Come on! Other JDBC drivers do it and older version of the MySQL JDBC driver also did it. Just do the same. It is not normal to continue reading a table when the query is stopped...
            diego dupin Diego Dupin added a comment -

            Right,
            I was reluctant to set up a solution with "kill query", but it is definitely the right solution for this case.
            this can definitively be done : Commit 0d6e4db
            will be released in 2.0.0

            diego dupin Diego Dupin added a comment - Right, I was reluctant to set up a solution with "kill query", but it is definitely the right solution for this case. this can definitively be done : Commit 0d6e4db will be released in 2.0.0
            diego dupin Diego Dupin made changes -
            Fix Version/s 2.0.0-RC [ 22119 ]
            diego dupin Diego Dupin made changes -
            Component/s Other [ 12201 ]
            Resolution Fixed [ 1 ]
            Status Open [ 1 ] Closed [ 6 ]
            adrian.tarau Adrian Tarau made changes -
            serg Sergei Golubchik made changes -
            Workflow MariaDB v3 [ 76129 ] MariaDB v4 [ 134884 ]

            People

              diego dupin Diego Dupin
              bertrandop Olivier Bertrand
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Git Integration

                  Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.