Details
-
New Feature
-
Status: Closed (View Workflow)
-
Critical
-
Resolution: Fixed
Description
The bulk command COM_STMT_BULK_STMT actually returns one OK_Packet that contains the first insert ID and the total of affected rows.
Using java there is 2 behaviors that are not compatible :
- When all auto increment id's needs to be retrieved.
- when using optimistic batch.
Explaination of those use cases :
insert ids needed
Prepared statement can be created with explicit indicator that auto increment ids are needed : https://docs.oracle.com/en/java/javase/18/docs/api/java.sql/java/sql/Connection.html#prepareStatement(java.lang.String,int)
example :
try (PreparedStatement ps = |
sharedConn.prepareStatement(
|
"INSERT INTO prepare2(t2) VALUES (?)", java.sql.Statement.RETURN_GENERATED_KEYS)) { |
ps.setInt(1, 10); |
ps.addBatch();
|
ps.setInt(1, 20); |
ps.addBatch();
|
ps.executeBatch();
|
|
ResultSet rs = ps.getGeneratedKeys();
|
assertTrue(rs.next());
|
assertEquals(1, rs.getInt(1)); |
assertTrue(rs.next());
|
assertEquals(2, rs.getInt(1)); |
assertFalse(rs.next());
|
}
|
Actual java connector implementation is to just loop with COM_STMT_EXECUTE, since driver is aware that generated id's are needed.
Second case is more problematic.
optimistic locking.
In java, when using multiple client servers, a stragegy named "Optimistic locking" to have faster result is to cache some data client side. "Optimistic locking" doesn't shared cache information between client servers: This is more complex than that, but to simplify, all tables have an additional 'version' column that is incremented at any changes. All operations on data will check primary AND that version equality, then increment version.
example :
Table basket_item DDL :
create table people ( |
name varchar(32) not null, |
age int, |
version int not null, |
PRIMARY KEY (name)); |
When updating some Entity, update command will use primary key + version like, incrementing version :
UPDATE people SET age=26, version=11 WHERE name='john Doe' AND version = 10; |
Application check that affected rows is exactly 1 to ensure value has not changed by another client during the time this data has been in client cache. If Affected row is 0, then application knows changes has occurs, then will handle that.
the problem is when using BULK, actual server implementation only returns the total number of changes, not unitary change. Java connector then returns an array of Statement.SUCCESS_NO_INFO, indicating that command has succedded,not knowing exact affected rows. This isn't compatible with Optimistic locking.
Solution
There is a flag STMT_BULK_FLAG_SEND_UNIT_RESULTS for bulk that has not been implemented.
Client need to know that server permit that functionality (new capability MARIADB_CLIENT_BULK_UNIT_RESULTS) in order to know if can use that functionality.
Link is a proposed implementation (based on 11.0).
Proposition is to add an ending resulset containing all unitary changes, since BULK can be use with command using RETURNING.
Attachments
Issue Links
- blocks
-
CONJS-275 permit returning all Bulk insert id's
-
- Closed
-
- causes
-
CONCPP-128 Bulk implementation returning individual results
-
- Open
-
-
CONJ-1173 Bulk implementation returning individual results
-
- Closed
-
Activity
Field | Original Value | New Value |
---|---|---|
Description |
For bulk command COM_STMT_BULK_STMT actually returns one OK_Packet containing first insert ID and all affected rows.
Using java there is 2 behaviors that are not compatible : * When all auto increment ids needs to be retrieved. * when using optimistic batch. Explaination of those use cases : h3. insert ids needed Prepared statement can be created with explicit indicator that auto increment ids are needed : https://docs.oracle.com/en/java/javase/18/docs/api/java.sql/java/sql/Connection.html#prepareStatement(java.lang.String,int) example : {code:java} try (PreparedStatement ps = sharedConn.prepareStatement( "INSERT INTO prepare2(t2) VALUES (?)", java.sql.Statement.RETURN_GENERATED_KEYS)) { ps.setInt(1, 10); ps.addBatch(); ps.setInt(1, 20); ps.addBatch(); ps.executeBatch(); ResultSet rs = ps.getGeneratedKeys(); assertTrue(rs.next()); assertEquals(1, rs.getInt(1)); assertTrue(rs.next()); assertEquals(2, rs.getInt(1)); assertFalse(rs.next()); } {code} Actual java connector implementation is to just loop with COM_STMT_EXECUTE, since driver is aware that generated id's are needed. Second case is more problematic. h3. optimistic locking. In java, when using multiple client servers, a stragegy named "Optimistic locking" to have faster result is to cache some data client side. "Optimistic locking" doesn't shared cache information between client servers: This is more complex than that, but to simplify, all tables have an additional 'version' column that is incremented at any changes. All operations on data will check primary AND that version equality, then increment version. example : Table basket_item DDL : {code:sql} create table people ( name varchar(32) not null, age int, version int not null, PRIMARY KEY (name)); {code} client receive a value : 'john Doe', 25, 10. update will be like : {code:sql} UPDATE people SET age=26, version=11 WHERE name='john Doe' AND version = 10; {code} Application check that affected rows is exactly 1 to ensure value has not changed by another client during the time this data has been in client cache. h3. Solution Add a specific flag (or reuse 64 : Return generated auto-increment IDs that isn't implemented) to indicate that COM_STMT_BULK_EXECUTE has to returns as many OK_Packet than bunch of parameters, returning unitary generated ids and affected rows. This feature would be advertised to connector using a specific capability flag. |
The bulk command COM_STMT_BULK_STMT actually returns one OK_Packet that contains the first insert ID and the total of affected rows.
Using java there is 2 behaviors that are not compatible : * When all auto increment id's needs to be retrieved. * when using optimistic batch. Explaination of those use cases : h3. insert ids needed Prepared statement can be created with explicit indicator that auto increment ids are needed : https://docs.oracle.com/en/java/javase/18/docs/api/java.sql/java/sql/Connection.html#prepareStatement(java.lang.String,int) example : {code:java} try (PreparedStatement ps = sharedConn.prepareStatement( "INSERT INTO prepare2(t2) VALUES (?)", java.sql.Statement.RETURN_GENERATED_KEYS)) { ps.setInt(1, 10); ps.addBatch(); ps.setInt(1, 20); ps.addBatch(); ps.executeBatch(); ResultSet rs = ps.getGeneratedKeys(); assertTrue(rs.next()); assertEquals(1, rs.getInt(1)); assertTrue(rs.next()); assertEquals(2, rs.getInt(1)); assertFalse(rs.next()); } {code} Actual java connector implementation is to just loop with COM_STMT_EXECUTE, since driver is aware that generated id's are needed. Second case is more problematic. h3. optimistic locking. In java, when using multiple client servers, a stragegy named "Optimistic locking" to have faster result is to cache some data client side. "Optimistic locking" doesn't shared cache information between client servers: This is more complex than that, but to simplify, all tables have an additional 'version' column that is incremented at any changes. All operations on data will check primary AND that version equality, then increment version. example : Table basket_item DDL : {code:sql} create table people ( name varchar(32) not null, age int, version int not null, PRIMARY KEY (name)); {code} client receive a value : 'john Doe', 25, 10. update will be like : {code:sql} UPDATE people SET age=26, version=11 WHERE name='john Doe' AND version = 10; {code} Application check that affected rows is exactly 1 to ensure value has not changed by another client during the time this data has been in client cache. h3. Solution Add a specific flag (or reuse 64 : Return generated auto-increment IDs that isn't implemented) to indicate that COM_STMT_BULK_EXECUTE has to returns as many OK_Packet than bunch of parameters, returning unitary generated ids and affected rows. This feature would be advertised to connector using a specific capability flag. |
Attachment | [MDEV-30366]_COM_STMT_BULK_STMT_new_flag_to_server_to_returns_all_unitary_results.patch [ 67788 ] |
Description |
The bulk command COM_STMT_BULK_STMT actually returns one OK_Packet that contains the first insert ID and the total of affected rows.
Using java there is 2 behaviors that are not compatible : * When all auto increment id's needs to be retrieved. * when using optimistic batch. Explaination of those use cases : h3. insert ids needed Prepared statement can be created with explicit indicator that auto increment ids are needed : https://docs.oracle.com/en/java/javase/18/docs/api/java.sql/java/sql/Connection.html#prepareStatement(java.lang.String,int) example : {code:java} try (PreparedStatement ps = sharedConn.prepareStatement( "INSERT INTO prepare2(t2) VALUES (?)", java.sql.Statement.RETURN_GENERATED_KEYS)) { ps.setInt(1, 10); ps.addBatch(); ps.setInt(1, 20); ps.addBatch(); ps.executeBatch(); ResultSet rs = ps.getGeneratedKeys(); assertTrue(rs.next()); assertEquals(1, rs.getInt(1)); assertTrue(rs.next()); assertEquals(2, rs.getInt(1)); assertFalse(rs.next()); } {code} Actual java connector implementation is to just loop with COM_STMT_EXECUTE, since driver is aware that generated id's are needed. Second case is more problematic. h3. optimistic locking. In java, when using multiple client servers, a stragegy named "Optimistic locking" to have faster result is to cache some data client side. "Optimistic locking" doesn't shared cache information between client servers: This is more complex than that, but to simplify, all tables have an additional 'version' column that is incremented at any changes. All operations on data will check primary AND that version equality, then increment version. example : Table basket_item DDL : {code:sql} create table people ( name varchar(32) not null, age int, version int not null, PRIMARY KEY (name)); {code} client receive a value : 'john Doe', 25, 10. update will be like : {code:sql} UPDATE people SET age=26, version=11 WHERE name='john Doe' AND version = 10; {code} Application check that affected rows is exactly 1 to ensure value has not changed by another client during the time this data has been in client cache. h3. Solution Add a specific flag (or reuse 64 : Return generated auto-increment IDs that isn't implemented) to indicate that COM_STMT_BULK_EXECUTE has to returns as many OK_Packet than bunch of parameters, returning unitary generated ids and affected rows. This feature would be advertised to connector using a specific capability flag. |
The bulk command COM_STMT_BULK_STMT actually returns one OK_Packet that contains the first insert ID and the total of affected rows.
Using java there is 2 behaviors that are not compatible : * When all auto increment id's needs to be retrieved. * when using optimistic batch. Explaination of those use cases : h3. insert ids needed Prepared statement can be created with explicit indicator that auto increment ids are needed : https://docs.oracle.com/en/java/javase/18/docs/api/java.sql/java/sql/Connection.html#prepareStatement(java.lang.String,int) example : {code:java} try (PreparedStatement ps = sharedConn.prepareStatement( "INSERT INTO prepare2(t2) VALUES (?)", java.sql.Statement.RETURN_GENERATED_KEYS)) { ps.setInt(1, 10); ps.addBatch(); ps.setInt(1, 20); ps.addBatch(); ps.executeBatch(); ResultSet rs = ps.getGeneratedKeys(); assertTrue(rs.next()); assertEquals(1, rs.getInt(1)); assertTrue(rs.next()); assertEquals(2, rs.getInt(1)); assertFalse(rs.next()); } {code} Actual java connector implementation is to just loop with COM_STMT_EXECUTE, since driver is aware that generated id's are needed. Second case is more problematic. h3. optimistic locking. In java, when using multiple client servers, a stragegy named "Optimistic locking" to have faster result is to cache some data client side. "Optimistic locking" doesn't shared cache information between client servers: This is more complex than that, but to simplify, all tables have an additional 'version' column that is incremented at any changes. All operations on data will check primary AND that version equality, then increment version. example : Table basket_item DDL : {code:sql} create table people ( name varchar(32) not null, age int, version int not null, PRIMARY KEY (name)); {code} client receive a value : 'john Doe', 25, 10. update will be like : {code:sql} UPDATE people SET age=26, version=11 WHERE name='john Doe' AND version = 10; {code} Application check that affected rows is exactly 1 to ensure value has not changed by another client during the time this data has been in client cache. h3. Solution There is a flag STMT_BULK_FLAG_SEND_UNIT_RESULTS for bulk that has not been implemented. Client need to know that server permit that functionality (new capability MARIADB_CLIENT_BULK_UNIT_RESULTS) in order to know if can use that functionality. Link is a proposed implementation (based on 11.0). Proposition is to add an ending resulset containing all unitary changes, since BULK can be use with command using RETURNING. |
Attachment |
[ |
Description |
The bulk command COM_STMT_BULK_STMT actually returns one OK_Packet that contains the first insert ID and the total of affected rows.
Using java there is 2 behaviors that are not compatible : * When all auto increment id's needs to be retrieved. * when using optimistic batch. Explaination of those use cases : h3. insert ids needed Prepared statement can be created with explicit indicator that auto increment ids are needed : https://docs.oracle.com/en/java/javase/18/docs/api/java.sql/java/sql/Connection.html#prepareStatement(java.lang.String,int) example : {code:java} try (PreparedStatement ps = sharedConn.prepareStatement( "INSERT INTO prepare2(t2) VALUES (?)", java.sql.Statement.RETURN_GENERATED_KEYS)) { ps.setInt(1, 10); ps.addBatch(); ps.setInt(1, 20); ps.addBatch(); ps.executeBatch(); ResultSet rs = ps.getGeneratedKeys(); assertTrue(rs.next()); assertEquals(1, rs.getInt(1)); assertTrue(rs.next()); assertEquals(2, rs.getInt(1)); assertFalse(rs.next()); } {code} Actual java connector implementation is to just loop with COM_STMT_EXECUTE, since driver is aware that generated id's are needed. Second case is more problematic. h3. optimistic locking. In java, when using multiple client servers, a stragegy named "Optimistic locking" to have faster result is to cache some data client side. "Optimistic locking" doesn't shared cache information between client servers: This is more complex than that, but to simplify, all tables have an additional 'version' column that is incremented at any changes. All operations on data will check primary AND that version equality, then increment version. example : Table basket_item DDL : {code:sql} create table people ( name varchar(32) not null, age int, version int not null, PRIMARY KEY (name)); {code} client receive a value : 'john Doe', 25, 10. update will be like : {code:sql} UPDATE people SET age=26, version=11 WHERE name='john Doe' AND version = 10; {code} Application check that affected rows is exactly 1 to ensure value has not changed by another client during the time this data has been in client cache. h3. Solution There is a flag STMT_BULK_FLAG_SEND_UNIT_RESULTS for bulk that has not been implemented. Client need to know that server permit that functionality (new capability MARIADB_CLIENT_BULK_UNIT_RESULTS) in order to know if can use that functionality. Link is a proposed implementation (based on 11.0). Proposition is to add an ending resulset containing all unitary changes, since BULK can be use with command using RETURNING. |
The bulk command COM_STMT_BULK_STMT actually returns one OK_Packet that contains the first insert ID and the total of affected rows.
Using java there is 2 behaviors that are not compatible : * When all auto increment id's needs to be retrieved. * when using optimistic batch. Explaination of those use cases : h3. insert ids needed Prepared statement can be created with explicit indicator that auto increment ids are needed : https://docs.oracle.com/en/java/javase/18/docs/api/java.sql/java/sql/Connection.html#prepareStatement(java.lang.String,int) example : {code:java} try (PreparedStatement ps = sharedConn.prepareStatement( "INSERT INTO prepare2(t2) VALUES (?)", java.sql.Statement.RETURN_GENERATED_KEYS)) { ps.setInt(1, 10); ps.addBatch(); ps.setInt(1, 20); ps.addBatch(); ps.executeBatch(); ResultSet rs = ps.getGeneratedKeys(); assertTrue(rs.next()); assertEquals(1, rs.getInt(1)); assertTrue(rs.next()); assertEquals(2, rs.getInt(1)); assertFalse(rs.next()); } {code} Actual java connector implementation is to just loop with COM_STMT_EXECUTE, since driver is aware that generated id's are needed. Second case is more problematic. h3. optimistic locking. In java, when using multiple client servers, a stragegy named "Optimistic locking" to have faster result is to cache some data client side. "Optimistic locking" doesn't shared cache information between client servers: This is more complex than that, but to simplify, all tables have an additional 'version' column that is incremented at any changes. All operations on data will check primary AND that version equality, then increment version. example : Table basket_item DDL : {code:sql} create table people ( name varchar(32) not null, age int, version int not null, PRIMARY KEY (name)); {code} When updating some Entity, update command will use primary key + version like, incrementing version : {code:sql} UPDATE people SET age=26, version=11 WHERE name='john Doe' AND version = 10; {code} Application check that affected rows is exactly 1 to ensure value has not changed by another client during the time this data has been in client cache. If Affected row is 0, then application knows changes has occurs, then will handle that. the problem is when using BULK, actual server implementation only returns the total number of changes, not unitary change. Java connector then returns an array of Statement.SUCCESS_NO_INFO, indicating that command has succedded,not knowing exact affected rows. This isn't compatible with Optimistic locking. h3. Solution There is a flag STMT_BULK_FLAG_SEND_UNIT_RESULTS for bulk that has not been implemented. Client need to know that server permit that functionality (new capability MARIADB_CLIENT_BULK_UNIT_RESULTS) in order to know if can use that functionality. Link is a proposed implementation (based on 11.0). Proposition is to add an ending resulset containing all unitary changes, since BULK can be use with command using RETURNING. |
Fix Version/s | 11.1 [ 28549 ] |
Component/s | Protocol [ 14604 ] |
Assignee | Sergei Golubchik [ serg ] |
Status | Open [ 1 ] | In Progress [ 3 ] |
Status | In Progress [ 3 ] | In Review [ 10002 ] |
Priority | Major [ 3 ] | Critical [ 2 ] |
Assignee | Sergei Golubchik [ serg ] | Diego Dupin [ diego dupin ] |
Status | In Review [ 10002 ] | Stalled [ 10000 ] |
Assignee | Diego Dupin [ diego dupin ] | Sergei Golubchik [ serg ] |
Fix Version/s | 11.3 [ 28565 ] | |
Fix Version/s | 11.1 [ 28549 ] |
Fix Version/s | 11.4 [ 29301 ] | |
Fix Version/s | 11.3 [ 28565 ] |
Status | Stalled [ 10000 ] | In Review [ 10002 ] |
Issue Type | Task [ 3 ] | New Feature [ 2 ] |
Assignee | Sergei Golubchik [ serg ] | Oleksandr Byelkin [ sanja ] |
Status | In Review [ 10002 ] | Stalled [ 10000 ] |
Fix Version/s | 11.5 [ 29506 ] | |
Fix Version/s | 11.4 [ 29301 ] |
Status | Stalled [ 10000 ] | In Progress [ 3 ] |
Assignee | Oleksandr Byelkin [ sanja ] | Diego Dupin [ diego dupin ] |
Assignee | Diego Dupin [ diego dupin ] | Georg Richter [ georg ] |
Assignee | Georg Richter [ georg ] | Oleksandr Byelkin [ sanja ] |
Status | In Progress [ 3 ] | In Testing [ 10301 ] |
Assignee | Oleksandr Byelkin [ sanja ] | Ramesh Sivaraman [ JIRAUSER48189 ] |
Link | This issue is part of TODO-4630 [ TODO-4630 ] |
Labels | Preview_11.5 |
Assignee | Ramesh Sivaraman [ JIRAUSER48189 ] | Diego Dupin [ diego dupin ] |
Status | In Testing [ 10301 ] | Stalled [ 10000 ] |
Assignee | Diego Dupin [ diego dupin ] | Sergei Golubchik [ serg ] |
Fix Version/s | 11.5.1 [ 29634 ] | |
Fix Version/s | 11.5 [ 29506 ] | |
Resolution | Fixed [ 1 ] | |
Status | Stalled [ 10000 ] | Closed [ 6 ] |
Assignee | Sergei Golubchik [ serg ] | Diego Dupin [ diego dupin ] |
Link | This issue causes CONCPP-128 [ CONCPP-128 ] |