Uploaded image for project: 'MariaDB Server'
  1. MariaDB Server
  2. MDEV-9347

Not all rows returned by the C API

Details

    • 10.1.11

    Description

      We noticed after upgrading from MySQL 5.5 to MariaDB 10.1 that looping over a table through the C API doesn't return all rows. It only happens on a sufficiently large table.

      Attached is a small Python script which produces such a table, and a C program that triggers the bug. Running make will compile the C file and produce a file setup.sql which you can import in your database. When it is imported, MYSQLPASSWORD= ./main -s (other flags might be needed, see ./main -h) shows the number of rows received, which should of course agree with SELECT COUNT(*) FROM foo; from the mysql client.

      We have bisected the issue to this commit: https://github.com/MariaDB/server/commit/7a387c01586ae5107f2c2c64d3cdd420de83e168

      Attachments

        Activity

          Ascending order - Click to sort in descending order
          serg Sergei Golubchik added a comment - - edited

          What happens here:

          • a large table (1*10 6 rows)
          • main.c prepares a simple SELECT columns FROM table
          • and executes it with the CURSOR_TYPE_READ_ONLY flag
          • on the server this goes to mysql_open_cursor() that creates a Select_materialize object
          • Select_materialize::send_result_set_metadata() invokes create_result_table() with keep_row_order=TRUE
          • before my commit (referenced above) keep_row_order caused Aria tables to be created with the DYNAMIC record format
          • with the google encryption patch Aria can preserve the row order in the BLOCK format too
          • so now all temporary tables use BLOCK format to avoid leaking unencrypted data to disk (and BLOCK is faster anyway)
          • in the debug build that query hits an assert in allocate_head() function in ma_bitmap.c: 

              if (insert_order)
            		 
              {
            		 
                uint last_insert_page= share->last_insert_page;
            		 
                uint byte= 6 * (last_insert_page / 16);
            		 
                first_pattern= last_insert_page % 16;
            		 
                DBUG_ASSERT(data + byte < end);
            		 
                data+= byte;
            		 
              }

          serg Sergei Golubchik added a comment - - edited What happens here: a large table (1*10 6 rows) main.c prepares a simple SELECT columns FROM table and executes it with the CURSOR_TYPE_READ_ONLY flag on the server this goes to mysql_open_cursor() that creates a Select_materialize object Select_materialize::send_result_set_metadata() invokes create_result_table() with keep_row_order=TRUE before my commit (referenced above) keep_row_order caused Aria tables to be created with the DYNAMIC record format with the google encryption patch Aria can preserve the row order in the BLOCK format too so now all temporary tables use BLOCK format to avoid leaking unencrypted data to disk (and BLOCK is faster anyway) in the debug build that query hits an assert in allocate_head() function in ma_bitmap.c : if (insert_order) { uint last_insert_page= share->last_insert_page; uint byte= 6 * (last_insert_page / 16); first_pattern= last_insert_page % 16; DBUG_ASSERT(data + byte < end); data+= byte; }
          monty Michael Widenius added a comment -

          roblem was that insert-order (enforced by the optimizer) did not handle the case where the bitmap changed to a new one.

          Fixed by remembering the last bitmap page used and to force usage of this when inserting new rows

          monty Michael Widenius added a comment - roblem was that insert-order (enforced by the optimizer) did not handle the case where the bitmap changed to a new one. Fixed by remembering the last bitmap page used and to force usage of this when inserting new rows
          monty Michael Widenius added a comment -

          Fix pushed

          monty Michael Widenius added a comment - Fix pushed

          People

            monty Michael Widenius
            Timvde Tim Van den Eynde
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Git Integration

                Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.