[CONJ-26] [Feature Request] Implement configurable fetch size and fetch direction for Statement/ResultSet Created: 2013-02-14  Updated: 2016-04-04  Resolved: 2016-04-04

Status: Closed
Project: MariaDB Connector/J
Component/s: Other
Affects Version/s: 1.1.0
Fix Version/s: 1.4.0

Type: Epic Priority: Minor
Reporter: Elena Stepanova Assignee: Diego Dupin
Resolution: Fixed Votes: 0
Labels: None

Issue Links:
Blocks
is blocked by CONJ-22 Java Client library does not support ... Closed
Duplicate
duplicates CONJ-125 Optimize cached ResultSet memory foot... Closed
is duplicated by CONJ-173 [Feature Request] Implement a mechani... Closed
Relates
Epic Name: Implement multiple row fetching
Sprint: Sprint connector/j 1.3.0

 Description   

As discussed earlier, currently fetch size is "one or all". It would be good to implement it fully.
Also, setFetchDirection for ResultSet and Statement are currently just stubs.



 Comments   
Comment by Vladislav Vaintroub [ 2013-04-28 ]

I think fetch direction can safely be ignored. after looking again at spec. this is a hint to the driver, which has no visible effects during runtime (i.e ResultSet.next() will still move forward, and previous() will move backward). We cannot use this hint - to move backwards with ResultSet.previous(), we have to read and cache the whole result, no way around it.

Comment by Paolo Bazzi [ 2016-02-24 ]

+1 for solving this issue....
We run into a similiar problem while reading a large set of data from a MariaDB and were forced to switch to the streamed mode, since we ran into OutOfMemory exceptions when fetching all records at once (~8m records). Other JDBC drivers (like Oracle) only fetch the records, when iterating over the result set and therefore require a lot less of memory if the records are processed in a loop and then discarded.
It would be great to support a configurable fetch size instead of force the user to decide wether to read all or nothing.

Comment by Diego Dupin [ 2016-02-24 ]

Some good news Paolo : that's in the roadmap for next version 1.4.0.

Comment by Vladislav Vaintroub [ 2016-02-24 ]

bazzip ,alas, is no principal difference between streamed mode and configurable fetch size. streaming mode requires least memory though
Are you unhappy to be "forced" into this mode?

Comment by Paolo Bazzi [ 2016-02-24 ]

@Diego very nice to hear!

@Vladislav
Two issues with the streamed mode

  • We use shared java code which is executed on both MariaDB and Oracle databases (some kind of data replication). The fetch size is set for each statement according to business logic and expected statement result size. With this setup we run into OutOfMemoryException problems with large data sets and the MariaDB JDBC driver, since the driver tried to load all data into memory. We were forced to implement a "if oracle then use fetchSize else use Integer.MIN_VALUE" Hack to solve the problem
  • I would expect a performance gain using an adequate fetch size instead of the streaming mode, which requires a JDBC driver <-> database server network round trip for each fetched result row
Comment by Vladislav Vaintroub [ 2016-02-24 ]

I agree on portability, but I doubt you will gain any performance

the driver does exactly the same amount of network reads, and the server the same amount of writes.
The server writes whole result set, the client reads the whole result set, Oracle may and actually does perform very differently.

Comment by Diego Dupin [ 2016-04-04 ]

This is now implemented on version 1.4.0.

Like Vladislav say, since all datas have to be read, performance doesn't change a lot,

JMH results (source https://codeshare.io/OlBRO) when streaming 100,000 rows

Bench.fetchSizeBy1000 : 50.274 ± 0.206 ms/op (read with fetch size 1000)
Bench.fetchSizeAll : 50.593 ± 0.252 ms/op (read all data)
Bench.fetchSizeOneByOne : 51.641 ± 0.299 ms/op (fetch one by one)

No big difference, but avoiding to create a big buffer permit to gain a small 1%, ( and avoid loading all in memory)

Generated at Thu Feb 08 03:12:37 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.