Uploaded image for project: 'MariaDB Server'
  1. MariaDB Server
  2. MDEV-30366

Permit bulk implementation to return ALL individual results

Details

    Description

      The bulk command COM_STMT_BULK_STMT actually returns one OK_Packet that contains the first insert ID and the total of affected rows.

      Using java there is 2 behaviors that are not compatible :

      • When all auto increment id's needs to be retrieved.
      • when using optimistic batch.

      Explaination of those use cases :

      insert ids needed

      Prepared statement can be created with explicit indicator that auto increment ids are needed : https://docs.oracle.com/en/java/javase/18/docs/api/java.sql/java/sql/Connection.html#prepareStatement(java.lang.String,int)

      example :

          try (PreparedStatement ps =
              sharedConn.prepareStatement(
                  "INSERT INTO prepare2(t2) VALUES (?)", java.sql.Statement.RETURN_GENERATED_KEYS)) {
            ps.setInt(1, 10);
            ps.addBatch();
            ps.setInt(1, 20);
            ps.addBatch();
            ps.executeBatch();
            
            ResultSet rs = ps.getGeneratedKeys();
            assertTrue(rs.next());
            assertEquals(1, rs.getInt(1));
            assertTrue(rs.next());
            assertEquals(2, rs.getInt(1));
            assertFalse(rs.next());
          }
      

      Actual java connector implementation is to just loop with COM_STMT_EXECUTE, since driver is aware that generated id's are needed.

      Second case is more problematic.

      optimistic locking.

      In java, when using multiple client servers, a stragegy named "Optimistic locking" to have faster result is to cache some data client side. "Optimistic locking" doesn't shared cache information between client servers: This is more complex than that, but to simplify, all tables have an additional 'version' column that is incremented at any changes. All operations on data will check primary AND that version equality, then increment version.
      example :

      Table basket_item DDL :

      create table people (
          name varchar(32) not null,
          age int,
          version int not null,
          PRIMARY KEY (name));
      

      When updating some Entity, update command will use primary key + version like, incrementing version :

       UPDATE people SET age=26, version=11 WHERE name='john Doe' AND version = 10;
      

      Application check that affected rows is exactly 1 to ensure value has not changed by another client during the time this data has been in client cache. If Affected row is 0, then application knows changes has occurs, then will handle that.

      the problem is when using BULK, actual server implementation only returns the total number of changes, not unitary change. Java connector then returns an array of Statement.SUCCESS_NO_INFO, indicating that command has succedded,not knowing exact affected rows. This isn't compatible with Optimistic locking.

      Solution

      There is a flag STMT_BULK_FLAG_SEND_UNIT_RESULTS for bulk that has not been implemented.
      Client need to know that server permit that functionality (new capability MARIADB_CLIENT_BULK_UNIT_RESULTS) in order to know if can use that functionality.

      Link is a proposed implementation (based on 11.0).
      Proposition is to add an ending resulset containing all unitary changes, since BULK can be use with command using RETURNING.

      Attachments

        Issue Links

          Activity

            diego dupin Diego Dupin created issue -
            diego dupin Diego Dupin made changes -
            Field Original Value New Value
            Description For bulk command COM_STMT_BULK_STMT actually returns one OK_Packet containing first insert ID and all affected rows.

            Using java there is 2 behaviors that are not compatible :
            * When all auto increment ids needs to be retrieved.
            * when using optimistic batch.

            Explaination of those use cases :
            h3. insert ids needed

            Prepared statement can be created with explicit indicator that auto increment ids are needed : https://docs.oracle.com/en/java/javase/18/docs/api/java.sql/java/sql/Connection.html#prepareStatement(java.lang.String,int)

            example :

            {code:java}
                try (PreparedStatement ps =
                    sharedConn.prepareStatement(
                        "INSERT INTO prepare2(t2) VALUES (?)", java.sql.Statement.RETURN_GENERATED_KEYS)) {
                  ps.setInt(1, 10);
                  ps.addBatch();
                  ps.setInt(1, 20);
                  ps.addBatch();
                  ps.executeBatch();
                  
                  ResultSet rs = ps.getGeneratedKeys();
                  assertTrue(rs.next());
                  assertEquals(1, rs.getInt(1));
                  assertTrue(rs.next());
                  assertEquals(2, rs.getInt(1));
                  assertFalse(rs.next());
                }
            {code}

            Actual java connector implementation is to just loop with COM_STMT_EXECUTE, since driver is aware that generated id's are needed.


            Second case is more problematic.

            h3. optimistic locking.
            In java, when using multiple client servers, a stragegy named "Optimistic locking" to have faster result is to cache some data client side. "Optimistic locking" doesn't shared cache information between client servers: This is more complex than that, but to simplify, all tables have an additional 'version' column that is incremented at any changes. All operations on data will check primary AND that version equality, then increment version.
            example :

            Table basket_item DDL :

            {code:sql}
            create table people (
                name varchar(32) not null,
                age int,
                version int not null,
                PRIMARY KEY (name));
            {code}

            client receive a value :
            'john Doe', 25, 10.

            update will be like :

            {code:sql}
             UPDATE people SET age=26, version=11 WHERE name='john Doe' AND version = 10;
            {code}

            Application check that affected rows is exactly 1 to ensure value has not changed by another client during the time this data has been in client cache.

            h3. Solution

            Add a specific flag (or reuse 64 : Return generated auto-increment IDs that isn't implemented) to indicate that COM_STMT_BULK_EXECUTE has to returns as many OK_Packet than bunch of parameters, returning unitary generated ids and affected rows.

            This feature would be advertised to connector using a specific capability flag.




            The bulk command COM_STMT_BULK_STMT actually returns one OK_Packet that contains the first insert ID and the total of affected rows.

            Using java there is 2 behaviors that are not compatible :
            * When all auto increment id's needs to be retrieved.
            * when using optimistic batch.

            Explaination of those use cases :
            h3. insert ids needed

            Prepared statement can be created with explicit indicator that auto increment ids are needed : https://docs.oracle.com/en/java/javase/18/docs/api/java.sql/java/sql/Connection.html#prepareStatement(java.lang.String,int)

            example :

            {code:java}
                try (PreparedStatement ps =
                    sharedConn.prepareStatement(
                        "INSERT INTO prepare2(t2) VALUES (?)", java.sql.Statement.RETURN_GENERATED_KEYS)) {
                  ps.setInt(1, 10);
                  ps.addBatch();
                  ps.setInt(1, 20);
                  ps.addBatch();
                  ps.executeBatch();
                  
                  ResultSet rs = ps.getGeneratedKeys();
                  assertTrue(rs.next());
                  assertEquals(1, rs.getInt(1));
                  assertTrue(rs.next());
                  assertEquals(2, rs.getInt(1));
                  assertFalse(rs.next());
                }
            {code}

            Actual java connector implementation is to just loop with COM_STMT_EXECUTE, since driver is aware that generated id's are needed.


            Second case is more problematic.

            h3. optimistic locking.
            In java, when using multiple client servers, a stragegy named "Optimistic locking" to have faster result is to cache some data client side. "Optimistic locking" doesn't shared cache information between client servers: This is more complex than that, but to simplify, all tables have an additional 'version' column that is incremented at any changes. All operations on data will check primary AND that version equality, then increment version.
            example :

            Table basket_item DDL :

            {code:sql}
            create table people (
                name varchar(32) not null,
                age int,
                version int not null,
                PRIMARY KEY (name));
            {code}

            client receive a value :
            'john Doe', 25, 10.

            update will be like :

            {code:sql}
             UPDATE people SET age=26, version=11 WHERE name='john Doe' AND version = 10;
            {code}

            Application check that affected rows is exactly 1 to ensure value has not changed by another client during the time this data has been in client cache.

            h3. Solution

            Add a specific flag (or reuse 64 : Return generated auto-increment IDs that isn't implemented) to indicate that COM_STMT_BULK_EXECUTE has to returns as many OK_Packet than bunch of parameters, returning unitary generated ids and affected rows.

            This feature would be advertised to connector using a specific capability flag.




            diego dupin Diego Dupin made changes -
            Attachment [MDEV-30366]_COM_STMT_BULK_STMT_new_flag_to_server_to_returns_all_unitary_results.patch [ 67788 ]
            diego dupin Diego Dupin made changes -
            Description The bulk command COM_STMT_BULK_STMT actually returns one OK_Packet that contains the first insert ID and the total of affected rows.

            Using java there is 2 behaviors that are not compatible :
            * When all auto increment id's needs to be retrieved.
            * when using optimistic batch.

            Explaination of those use cases :
            h3. insert ids needed

            Prepared statement can be created with explicit indicator that auto increment ids are needed : https://docs.oracle.com/en/java/javase/18/docs/api/java.sql/java/sql/Connection.html#prepareStatement(java.lang.String,int)

            example :

            {code:java}
                try (PreparedStatement ps =
                    sharedConn.prepareStatement(
                        "INSERT INTO prepare2(t2) VALUES (?)", java.sql.Statement.RETURN_GENERATED_KEYS)) {
                  ps.setInt(1, 10);
                  ps.addBatch();
                  ps.setInt(1, 20);
                  ps.addBatch();
                  ps.executeBatch();
                  
                  ResultSet rs = ps.getGeneratedKeys();
                  assertTrue(rs.next());
                  assertEquals(1, rs.getInt(1));
                  assertTrue(rs.next());
                  assertEquals(2, rs.getInt(1));
                  assertFalse(rs.next());
                }
            {code}

            Actual java connector implementation is to just loop with COM_STMT_EXECUTE, since driver is aware that generated id's are needed.


            Second case is more problematic.

            h3. optimistic locking.
            In java, when using multiple client servers, a stragegy named "Optimistic locking" to have faster result is to cache some data client side. "Optimistic locking" doesn't shared cache information between client servers: This is more complex than that, but to simplify, all tables have an additional 'version' column that is incremented at any changes. All operations on data will check primary AND that version equality, then increment version.
            example :

            Table basket_item DDL :

            {code:sql}
            create table people (
                name varchar(32) not null,
                age int,
                version int not null,
                PRIMARY KEY (name));
            {code}

            client receive a value :
            'john Doe', 25, 10.

            update will be like :

            {code:sql}
             UPDATE people SET age=26, version=11 WHERE name='john Doe' AND version = 10;
            {code}

            Application check that affected rows is exactly 1 to ensure value has not changed by another client during the time this data has been in client cache.

            h3. Solution

            Add a specific flag (or reuse 64 : Return generated auto-increment IDs that isn't implemented) to indicate that COM_STMT_BULK_EXECUTE has to returns as many OK_Packet than bunch of parameters, returning unitary generated ids and affected rows.

            This feature would be advertised to connector using a specific capability flag.




            The bulk command COM_STMT_BULK_STMT actually returns one OK_Packet that contains the first insert ID and the total of affected rows.

            Using java there is 2 behaviors that are not compatible :
            * When all auto increment id's needs to be retrieved.
            * when using optimistic batch.

            Explaination of those use cases :
            h3. insert ids needed

            Prepared statement can be created with explicit indicator that auto increment ids are needed : https://docs.oracle.com/en/java/javase/18/docs/api/java.sql/java/sql/Connection.html#prepareStatement(java.lang.String,int)

            example :

            {code:java}
                try (PreparedStatement ps =
                    sharedConn.prepareStatement(
                        "INSERT INTO prepare2(t2) VALUES (?)", java.sql.Statement.RETURN_GENERATED_KEYS)) {
                  ps.setInt(1, 10);
                  ps.addBatch();
                  ps.setInt(1, 20);
                  ps.addBatch();
                  ps.executeBatch();
                  
                  ResultSet rs = ps.getGeneratedKeys();
                  assertTrue(rs.next());
                  assertEquals(1, rs.getInt(1));
                  assertTrue(rs.next());
                  assertEquals(2, rs.getInt(1));
                  assertFalse(rs.next());
                }
            {code}

            Actual java connector implementation is to just loop with COM_STMT_EXECUTE, since driver is aware that generated id's are needed.


            Second case is more problematic.

            h3. optimistic locking.
            In java, when using multiple client servers, a stragegy named "Optimistic locking" to have faster result is to cache some data client side. "Optimistic locking" doesn't shared cache information between client servers: This is more complex than that, but to simplify, all tables have an additional 'version' column that is incremented at any changes. All operations on data will check primary AND that version equality, then increment version.
            example :

            Table basket_item DDL :

            {code:sql}
            create table people (
                name varchar(32) not null,
                age int,
                version int not null,
                PRIMARY KEY (name));
            {code}

            client receive a value :
            'john Doe', 25, 10.

            update will be like :

            {code:sql}
             UPDATE people SET age=26, version=11 WHERE name='john Doe' AND version = 10;
            {code}

            Application check that affected rows is exactly 1 to ensure value has not changed by another client during the time this data has been in client cache.

            h3. Solution

            There is a flag STMT_BULK_FLAG_SEND_UNIT_RESULTS for bulk that has not been implemented.
            Client need to know that server permit that functionality (new capability MARIADB_CLIENT_BULK_UNIT_RESULTS) in order to know if can use that functionality.

            Link is a proposed implementation (based on 11.0).
            Proposition is to add an ending resulset containing all unitary changes, since BULK can be use with command using RETURNING.





            diego dupin Diego Dupin made changes -
            Attachment [MDEV-30366]_COM_STMT_BULK_STMT_new_flag_to_server_to_returns_all_unitary_results.patch [ 67788 ]
            diego dupin Diego Dupin made changes -
            Description The bulk command COM_STMT_BULK_STMT actually returns one OK_Packet that contains the first insert ID and the total of affected rows.

            Using java there is 2 behaviors that are not compatible :
            * When all auto increment id's needs to be retrieved.
            * when using optimistic batch.

            Explaination of those use cases :
            h3. insert ids needed

            Prepared statement can be created with explicit indicator that auto increment ids are needed : https://docs.oracle.com/en/java/javase/18/docs/api/java.sql/java/sql/Connection.html#prepareStatement(java.lang.String,int)

            example :

            {code:java}
                try (PreparedStatement ps =
                    sharedConn.prepareStatement(
                        "INSERT INTO prepare2(t2) VALUES (?)", java.sql.Statement.RETURN_GENERATED_KEYS)) {
                  ps.setInt(1, 10);
                  ps.addBatch();
                  ps.setInt(1, 20);
                  ps.addBatch();
                  ps.executeBatch();
                  
                  ResultSet rs = ps.getGeneratedKeys();
                  assertTrue(rs.next());
                  assertEquals(1, rs.getInt(1));
                  assertTrue(rs.next());
                  assertEquals(2, rs.getInt(1));
                  assertFalse(rs.next());
                }
            {code}

            Actual java connector implementation is to just loop with COM_STMT_EXECUTE, since driver is aware that generated id's are needed.


            Second case is more problematic.

            h3. optimistic locking.
            In java, when using multiple client servers, a stragegy named "Optimistic locking" to have faster result is to cache some data client side. "Optimistic locking" doesn't shared cache information between client servers: This is more complex than that, but to simplify, all tables have an additional 'version' column that is incremented at any changes. All operations on data will check primary AND that version equality, then increment version.
            example :

            Table basket_item DDL :

            {code:sql}
            create table people (
                name varchar(32) not null,
                age int,
                version int not null,
                PRIMARY KEY (name));
            {code}

            client receive a value :
            'john Doe', 25, 10.

            update will be like :

            {code:sql}
             UPDATE people SET age=26, version=11 WHERE name='john Doe' AND version = 10;
            {code}

            Application check that affected rows is exactly 1 to ensure value has not changed by another client during the time this data has been in client cache.

            h3. Solution

            There is a flag STMT_BULK_FLAG_SEND_UNIT_RESULTS for bulk that has not been implemented.
            Client need to know that server permit that functionality (new capability MARIADB_CLIENT_BULK_UNIT_RESULTS) in order to know if can use that functionality.

            Link is a proposed implementation (based on 11.0).
            Proposition is to add an ending resulset containing all unitary changes, since BULK can be use with command using RETURNING.





            The bulk command COM_STMT_BULK_STMT actually returns one OK_Packet that contains the first insert ID and the total of affected rows.

            Using java there is 2 behaviors that are not compatible :
            * When all auto increment id's needs to be retrieved.
            * when using optimistic batch.

            Explaination of those use cases :
            h3. insert ids needed

            Prepared statement can be created with explicit indicator that auto increment ids are needed : https://docs.oracle.com/en/java/javase/18/docs/api/java.sql/java/sql/Connection.html#prepareStatement(java.lang.String,int)

            example :

            {code:java}
                try (PreparedStatement ps =
                    sharedConn.prepareStatement(
                        "INSERT INTO prepare2(t2) VALUES (?)", java.sql.Statement.RETURN_GENERATED_KEYS)) {
                  ps.setInt(1, 10);
                  ps.addBatch();
                  ps.setInt(1, 20);
                  ps.addBatch();
                  ps.executeBatch();
                  
                  ResultSet rs = ps.getGeneratedKeys();
                  assertTrue(rs.next());
                  assertEquals(1, rs.getInt(1));
                  assertTrue(rs.next());
                  assertEquals(2, rs.getInt(1));
                  assertFalse(rs.next());
                }
            {code}

            Actual java connector implementation is to just loop with COM_STMT_EXECUTE, since driver is aware that generated id's are needed.


            Second case is more problematic.

            h3. optimistic locking.
            In java, when using multiple client servers, a stragegy named "Optimistic locking" to have faster result is to cache some data client side. "Optimistic locking" doesn't shared cache information between client servers: This is more complex than that, but to simplify, all tables have an additional 'version' column that is incremented at any changes. All operations on data will check primary AND that version equality, then increment version.
            example :

            Table basket_item DDL :

            {code:sql}
            create table people (
                name varchar(32) not null,
                age int,
                version int not null,
                PRIMARY KEY (name));
            {code}

            When updating some Entity, update command will use primary key + version like, incrementing version :
            {code:sql}
             UPDATE people SET age=26, version=11 WHERE name='john Doe' AND version = 10;
            {code}

            Application check that affected rows is exactly 1 to ensure value has not changed by another client during the time this data has been in client cache. If Affected row is 0, then application knows changes has occurs, then will handle that.

            the problem is when using BULK, actual server implementation only returns the total number of changes, not unitary change. Java connector then returns an array of Statement.SUCCESS_NO_INFO, indicating that command has succedded,not knowing exact affected rows. This isn't compatible with Optimistic locking.

            h3. Solution

            There is a flag STMT_BULK_FLAG_SEND_UNIT_RESULTS for bulk that has not been implemented.
            Client need to know that server permit that functionality (new capability MARIADB_CLIENT_BULK_UNIT_RESULTS) in order to know if can use that functionality.

            Link is a proposed implementation (based on 11.0).
            Proposition is to add an ending resulset containing all unitary changes, since BULK can be use with command using RETURNING.





            serg Sergei Golubchik made changes -
            Fix Version/s 11.1 [ 28549 ]
            serg Sergei Golubchik made changes -
            Component/s Protocol [ 14604 ]
            serg Sergei Golubchik made changes -
            Assignee Sergei Golubchik [ serg ]
            serg Sergei Golubchik made changes -
            Status Open [ 1 ] In Progress [ 3 ]
            serg Sergei Golubchik made changes -
            Status In Progress [ 3 ] In Review [ 10002 ]
            serg Sergei Golubchik made changes -
            Priority Major [ 3 ] Critical [ 2 ]
            serg Sergei Golubchik made changes -
            Assignee Sergei Golubchik [ serg ] Diego Dupin [ diego dupin ]
            Status In Review [ 10002 ] Stalled [ 10000 ]
            diego dupin Diego Dupin made changes -
            Assignee Diego Dupin [ diego dupin ] Sergei Golubchik [ serg ]
            serg Sergei Golubchik made changes -
            Fix Version/s 11.3 [ 28565 ]
            Fix Version/s 11.1 [ 28549 ]
            serg Sergei Golubchik made changes -
            Fix Version/s 11.4 [ 29301 ]
            Fix Version/s 11.3 [ 28565 ]
            serg Sergei Golubchik made changes -
            Status Stalled [ 10000 ] In Review [ 10002 ]
            serg Sergei Golubchik made changes -
            Issue Type Task [ 3 ] New Feature [ 2 ]
            serg Sergei Golubchik made changes -
            Assignee Sergei Golubchik [ serg ] Oleksandr Byelkin [ sanja ]
            Status In Review [ 10002 ] Stalled [ 10000 ]
            diego dupin Diego Dupin made changes -
            diego dupin Diego Dupin made changes -
            serg Sergei Golubchik made changes -
            Fix Version/s 11.5 [ 29506 ]
            Fix Version/s 11.4 [ 29301 ]
            sanja Oleksandr Byelkin made changes -
            Status Stalled [ 10000 ] In Progress [ 3 ]
            sanja Oleksandr Byelkin made changes -
            Assignee Oleksandr Byelkin [ sanja ] Diego Dupin [ diego dupin ]
            diego dupin Diego Dupin made changes -
            Assignee Diego Dupin [ diego dupin ] Georg Richter [ georg ]
            diego dupin Diego Dupin made changes -
            Assignee Georg Richter [ georg ] Oleksandr Byelkin [ sanja ]
            sanja Oleksandr Byelkin made changes -
            Status In Progress [ 3 ] In Testing [ 10301 ]
            sanja Oleksandr Byelkin made changes -
            Assignee Oleksandr Byelkin [ sanja ] Ramesh Sivaraman [ JIRAUSER48189 ]
            serg Sergei Golubchik made changes -
            ralf.gebhardt Ralf Gebhardt made changes -
            Labels Preview_11.5
            ramesh Ramesh Sivaraman made changes -
            Assignee Ramesh Sivaraman [ JIRAUSER48189 ] Diego Dupin [ diego dupin ]
            Status In Testing [ 10301 ] Stalled [ 10000 ]
            ramesh Ramesh Sivaraman made changes -
            Assignee Diego Dupin [ diego dupin ] Sergei Golubchik [ serg ]
            diego dupin Diego Dupin made changes -
            serg Sergei Golubchik made changes -
            Fix Version/s 11.5.1 [ 29634 ]
            Fix Version/s 11.5 [ 29506 ]
            Resolution Fixed [ 1 ]
            Status Stalled [ 10000 ] Closed [ 6 ]
            serg Sergei Golubchik made changes -
            Assignee Sergei Golubchik [ serg ] Diego Dupin [ diego dupin ]
            ralf.gebhardt Ralf Gebhardt made changes -
            ralf.gebhardt Ralf Gebhardt made changes -

            People

              diego dupin Diego Dupin
              diego dupin Diego Dupin
              Votes:
              3 Vote for this issue
              Watchers:
              10 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Git Integration

                  Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.