Uploaded image for project: 'MariaDB Server'
  1. MariaDB Server
  2. MDEV-9347

Not all rows returned by the C API

Details

    • 10.1.11

    Description

      We noticed after upgrading from MySQL 5.5 to MariaDB 10.1 that looping over a table through the C API doesn't return all rows. It only happens on a sufficiently large table.

      Attached is a small Python script which produces such a table, and a C program that triggers the bug. Running make will compile the C file and produce a file setup.sql which you can import in your database. When it is imported, MYSQLPASSWORD= ./main -s (other flags might be needed, see ./main -h) shows the number of rows received, which should of course agree with SELECT COUNT(*) FROM foo; from the mysql client.

      We have bisected the issue to this commit: https://github.com/MariaDB/server/commit/7a387c01586ae5107f2c2c64d3cdd420de83e168

      Attachments

        Activity

          Ascending order - Click to sort in descending order
          Timvde Tim Van den Eynde created issue -
          Timvde Tim Van den Eynde made changes -
          Field Original Value New Value
          Description We noticed after upgrading from MySQL 5.5 to MariaDB 10.1 that looping over a table through the C API doesn't return all rows. It only happens on a sufficiently large table.

          Attached is a small Python script which produces such a table, and a C program that triggers the bug. Running {{make}} will compile the C file and produce a file {{setup.sql}} which you can import in your database. When it is imported, {{MYSQLPASSWORD= ./main -s}} shows the number of rows received, which should of course agree with {{SELECT COUNT(*) FROM foo;}} from the mysql client.

          We have bisected the issue to this commit: https://github.com/MariaDB/server/commit/7a387c01586ae5107f2c2c64d3cdd420de83e168           		We noticed after upgrading from MySQL 5.5 to MariaDB 10.1 that looping over a table through the C API doesn't return all rows. It only happens on a sufficiently large table.

          Attached is a small Python script which produces such a table, and a C program that triggers the bug. Running {{make}} will compile the C file and produce a file {{setup.sql}} which you can import in your database. When it is imported, {{MYSQLPASSWORD= ./main -s}} (other flags might be needed, see {{./main -h}}) shows the number of rows received, which should of course agree with {{SELECT COUNT(*) FROM foo;}} from the mysql client.

          We have bisected the issue to this commit: https://github.com/MariaDB/server/commit/7a387c01586ae5107f2c2c64d3cdd420de83e168
          serg Sergei Golubchik made changes -
          Description We noticed after upgrading from MySQL 5.5 to MariaDB 10.1 that looping over a table through the C API doesn't return all rows. It only happens on a sufficiently large table.

          Attached is a small Python script which produces such a table, and a C program that triggers the bug. Running {{make}} will compile the C file and produce a file {{setup.sql}} which you can import in your database. When it is imported, {{MYSQLPASSWORD= ./main -s}} (other flags might be needed, see {{./main -h}}) shows the number of rows received, which should of course agree with {{SELECT COUNT(*) FROM foo;}} from the mysql client.

          We have bisected the issue to this commit: https://github.com/MariaDB/server/commit/7a387c01586ae5107f2c2c64d3cdd420de83e168           		We noticed after upgrading from MySQL 5.5 to MariaDB 10.1 that looping over a table through the C API doesn't return all rows. It only happens on a sufficiently large table.

          Attached is a small Python script which produces such a table, and a C program that triggers the bug. Running {{make}} will compile the C file and produce a file {{setup.sql}} which you can import in your database. When it is imported, {{MYSQLPASSWORD= ./main -s}} (other flags might be needed, see {{./main -h}}) shows the number of rows received, which should of course agree with {{SELECT COUNT(\*) FROM foo;}} from the mysql client.

          We have bisected the issue to this commit: https://github.com/MariaDB/server/commit/7a387c01586ae5107f2c2c64d3cdd420de83e168
          serg Sergei Golubchik made changes -
          Affects Version/s 10.1.4 [ 18400 ]
          serg Sergei Golubchik made changes -
          Fix Version/s 10.1 [ 16100 ]
          serg Sergei Golubchik made changes -
          Affects Version/s 10.1 [ 16100 ]
          Affects Version/s 10.1.4 [ 18400 ]
          serg Sergei Golubchik made changes -
          Assignee Sergei Golubchik [ serg ]
          ratzpo Rasmus Johansson (Inactive) made changes -
          Sprint 10.1.11 [ 30 ]
          ratzpo Rasmus Johansson (Inactive) made changes -
          Rank Ranked lower
          serg Sergei Golubchik made changes -
          Assignee Sergei Golubchik [ serg ] Michael Widenius [ monty ]
          serg Sergei Golubchik added a comment - - edited

          What happens here:

          • a large table (1*10 6 rows)
          • main.c prepares a simple SELECT columns FROM table
          • and executes it with the CURSOR_TYPE_READ_ONLY flag
          • on the server this goes to mysql_open_cursor() that creates a Select_materialize object
          • Select_materialize::send_result_set_metadata() invokes create_result_table() with keep_row_order=TRUE
          • before my commit (referenced above) keep_row_order caused Aria tables to be created with the DYNAMIC record format
          • with the google encryption patch Aria can preserve the row order in the BLOCK format too
          • so now all temporary tables use BLOCK format to avoid leaking unencrypted data to disk (and BLOCK is faster anyway)
          • in the debug build that query hits an assert in allocate_head() function in ma_bitmap.c: 

              if (insert_order)
            		 
              {
            		 
                uint last_insert_page= share->last_insert_page;
            		 
                uint byte= 6 * (last_insert_page / 16);
            		 
                first_pattern= last_insert_page % 16;
            		 
                DBUG_ASSERT(data + byte < end);
            		 
                data+= byte;
            		 
              }

          serg Sergei Golubchik added a comment - - edited What happens here: a large table (1*10 6 rows) main.c prepares a simple SELECT columns FROM table and executes it with the CURSOR_TYPE_READ_ONLY flag on the server this goes to mysql_open_cursor() that creates a Select_materialize object Select_materialize::send_result_set_metadata() invokes create_result_table() with keep_row_order=TRUE before my commit (referenced above) keep_row_order caused Aria tables to be created with the DYNAMIC record format with the google encryption patch Aria can preserve the row order in the BLOCK format too so now all temporary tables use BLOCK format to avoid leaking unencrypted data to disk (and BLOCK is faster anyway) in the debug build that query hits an assert in allocate_head() function in ma_bitmap.c : if (insert_order) { uint last_insert_page= share->last_insert_page; uint byte= 6 * (last_insert_page / 16); first_pattern= last_insert_page % 16; DBUG_ASSERT(data + byte < end); data+= byte; }
          serg Sergei Golubchik made changes -
          Status Open [ 1 ] In Progress [ 3 ]
          monty Michael Widenius added a comment -

          roblem was that insert-order (enforced by the optimizer) did not handle the case where the bitmap changed to a new one.

          Fixed by remembering the last bitmap page used and to force usage of this when inserting new rows

          monty Michael Widenius added a comment - roblem was that insert-order (enforced by the optimizer) did not handle the case where the bitmap changed to a new one. Fixed by remembering the last bitmap page used and to force usage of this when inserting new rows
          monty Michael Widenius added a comment -

          Fix pushed

          monty Michael Widenius added a comment - Fix pushed
          monty Michael Widenius made changes -
          Component/s Optimizer [ 10200 ]
          Component/s Storage Engine - Aria [ 10126 ]
          Fix Version/s 10.1.11 [ 21202 ]
          Fix Version/s 10.1 [ 16100 ]
          Resolution Fixed [ 1 ]
          Status In Progress [ 3 ] Closed [ 6 ]
          serg Sergei Golubchik made changes -
          Workflow MariaDB v3 [ 73347 ] MariaDB v4 [ 149952 ]

          People

            monty Michael Widenius
            Timvde Tim Van den Eynde
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Git Integration

                Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.