Uploaded image for project: 'MariaDB Server'
  1. MariaDB Server
  2. MDEV-33450

Assertion fails in main.alter_table_online_debug

Details

    Attachments

      Activity

        I've been getting a similar assertion failure in concurrent tests on Linux since they were switched from bb-11.2-release to 11.2 main branch on Feb 8

        11.2 1553a9dd

        mysqld: /home/vsts/src/sql/sql_table.cc:12138: int copy_data_between_tables(THD*, TABLE*, TABLE*, bool, uint, ORDER*, ha_rows*, ha_rows*, Alter_info*, Alter_table_ctx*, bool, uint64): Assertion `from->s->online_alter_binlog == __null' failed.
         
        #7  0x00007ff0e8645fd6 in __GI___assert_fail (assertion=0x558e9fc22be0 "from->s->online_alter_binlog == __null", file=0x558e9fc1b360 "/home/vsts/src/sql/sql_table.cc", line=12138, function=0x558e9fc22ac0 "int copy_data_between_tables(THD*, TABLE*, TABLE*, bool, uint, ORDER*, ha_rows*, ha_rows*, Alter_info*, Alter_table_ctx*, bool, uint64)") at assert.c:101
        #8  0x0000558e9dca61e8 in copy_data_between_tables (thd=0x62c000450218, from=0x619000b0cc98, to=0x61900026b698, ignore=false, order_num=0, order=0x0, copied=0x7ff0b77871a0, deleted=0x7ff0b77871c0, alter_info=0x7ff0b778a0b0, alter_ctx=0x7ff0b7788d00, online=true, start_alter_id=0) at /home/vsts/src/sql/sql_table.cc:12138
        #9  0x0000558e9dc9fd41 in mysql_alter_table (thd=0x62c000450218, new_db=0x62c000455030, new_name=0x62c000455490, create_info=0x7ff0b778a260, table_list=0x6290007a8658, recreate_info=0x7ff0b778a050, alter_info=0x7ff0b778a0b0, order_num=0, order=0x0, ignore=false, if_exists=false) at /home/vsts/src/sql/sql_table.cc:11376
        #10 0x0000558e9de42c91 in Sql_cmd_alter_table::execute (this=0x6290007a9090, thd=0x62c000450218) at /home/vsts/src/sql/sql_alter.cc:701
        #11 0x0000558e9d9cf89b in mysql_execute_command (thd=0x62c000450218, is_called_from_prepared_stmt=false) at /home/vsts/src/sql/sql_parse.cc:5842
        #12 0x0000558e9d9db85f in mysql_parse (thd=0x62c000450218, rawbuf=0x6290007a8238 "/* WRK-2 QNO 1840 */  ALTER TABLE `advanced_db`.`t8` WAIT 2 ADD CONSTRAINT FOREIGN KEY x ( `id` ) REFERENCES `advanced_db`.`t6` (`id`)", length=134, parser_state=0x7ff0b778b910) at /home/vsts/src/sql/sql_parse.cc:7874
        #13 0x0000558e9d9b5a3d in dispatch_command (command=COM_QUERY, thd=0x62c000450218, packet=0x62900079e219 "/* WRK-2 QNO 1840 */  ALTER TABLE `advanced_db`.`t8` WAIT 2 ADD CONSTRAINT FOREIGN KEY x ( `id` ) REFERENCES `advanced_db`.`t6` (`id`) ", packet_length=135, blocking=true) at /home/vsts/src/sql/sql_parse.cc:1893
        #14 0x0000558e9d9b2e7b in do_command (thd=0x62c000450218, blocking=true) at /home/vsts/src/sql/sql_parse.cc:1406
        #15 0x0000558e9de27340 in do_handle_one_connection (connect=0x608000002f38, put_in_cache=true) at /home/vsts/src/sql/sql_connect.cc:1437
        #16 0x0000558e9de26ccd in handle_one_connection (arg=0x608000002f38) at /home/vsts/src/sql/sql_connect.cc:1339
        #17 0x0000558e9e997560 in pfs_spawn_thread (arg=0x617000007e98) at /home/vsts/src/storage/perfschema/pfs.cc:2201
        #18 0x00007ff0e8b5e609 in start_thread (arg=<optimized out>) at pthread_create.c:477
        #19 0x00007ff0e8731353 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
        

        elenst Elena Stepanova added a comment - I've been getting a similar assertion failure in concurrent tests on Linux since they were switched from bb-11.2-release to 11.2 main branch on Feb 8 11.2 1553a9dd mysqld: /home/vsts/src/sql/sql_table.cc:12138: int copy_data_between_tables(THD*, TABLE*, TABLE*, bool, uint, ORDER*, ha_rows*, ha_rows*, Alter_info*, Alter_table_ctx*, bool, uint64): Assertion `from->s->online_alter_binlog == __null' failed.   #7 0x00007ff0e8645fd6 in __GI___assert_fail (assertion=0x558e9fc22be0 "from->s->online_alter_binlog == __null", file=0x558e9fc1b360 "/home/vsts/src/sql/sql_table.cc", line=12138, function=0x558e9fc22ac0 "int copy_data_between_tables(THD*, TABLE*, TABLE*, bool, uint, ORDER*, ha_rows*, ha_rows*, Alter_info*, Alter_table_ctx*, bool, uint64)") at assert.c:101 #8 0x0000558e9dca61e8 in copy_data_between_tables (thd=0x62c000450218, from=0x619000b0cc98, to=0x61900026b698, ignore=false, order_num=0, order=0x0, copied=0x7ff0b77871a0, deleted=0x7ff0b77871c0, alter_info=0x7ff0b778a0b0, alter_ctx=0x7ff0b7788d00, online=true, start_alter_id=0) at /home/vsts/src/sql/sql_table.cc:12138 #9 0x0000558e9dc9fd41 in mysql_alter_table (thd=0x62c000450218, new_db=0x62c000455030, new_name=0x62c000455490, create_info=0x7ff0b778a260, table_list=0x6290007a8658, recreate_info=0x7ff0b778a050, alter_info=0x7ff0b778a0b0, order_num=0, order=0x0, ignore=false, if_exists=false) at /home/vsts/src/sql/sql_table.cc:11376 #10 0x0000558e9de42c91 in Sql_cmd_alter_table::execute (this=0x6290007a9090, thd=0x62c000450218) at /home/vsts/src/sql/sql_alter.cc:701 #11 0x0000558e9d9cf89b in mysql_execute_command (thd=0x62c000450218, is_called_from_prepared_stmt=false) at /home/vsts/src/sql/sql_parse.cc:5842 #12 0x0000558e9d9db85f in mysql_parse (thd=0x62c000450218, rawbuf=0x6290007a8238 "/* WRK-2 QNO 1840 */ ALTER TABLE `advanced_db`.`t8` WAIT 2 ADD CONSTRAINT FOREIGN KEY x ( `id` ) REFERENCES `advanced_db`.`t6` (`id`)", length=134, parser_state=0x7ff0b778b910) at /home/vsts/src/sql/sql_parse.cc:7874 #13 0x0000558e9d9b5a3d in dispatch_command (command=COM_QUERY, thd=0x62c000450218, packet=0x62900079e219 "/* WRK-2 QNO 1840 */ ALTER TABLE `advanced_db`.`t8` WAIT 2 ADD CONSTRAINT FOREIGN KEY x ( `id` ) REFERENCES `advanced_db`.`t6` (`id`) ", packet_length=135, blocking=true) at /home/vsts/src/sql/sql_parse.cc:1893 #14 0x0000558e9d9b2e7b in do_command (thd=0x62c000450218, blocking=true) at /home/vsts/src/sql/sql_parse.cc:1406 #15 0x0000558e9de27340 in do_handle_one_connection (connect=0x608000002f38, put_in_cache=true) at /home/vsts/src/sql/sql_connect.cc:1437 #16 0x0000558e9de26ccd in handle_one_connection (arg=0x608000002f38) at /home/vsts/src/sql/sql_connect.cc:1339 #17 0x0000558e9e997560 in pfs_spawn_thread (arg=0x617000007e98) at /home/vsts/src/storage/perfschema/pfs.cc:2201 #18 0x00007ff0e8b5e609 in start_thread (arg=<optimized out>) at pthread_create.c:477 #19 0x00007ff0e8731353 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
        wlad Vladislav Vaintroub added a comment - - edited It is all-debug-builders problem https://buildbot.mariadb.net/ci/reports/cross_reference#branch=&revision=&platform=&fail_name=main.alter_table_online_debug+&fail_variant=&fail_info_full=from-%3Es-%3Eonline_alter_binlog&typ=&info=&dt=&limit=100&fail_info_short=

        The memory leak assertion has got a good catch up.

        serg, please review these two commits:

        f1a28b57 fix race in the MDEV-32614 test
        65c6417e MDEV-33450 Assertion fails in main.alter_table_online_debug
        

        nikitamalyavin Nikita Malyavin added a comment - The memory leak assertion has got a good catch up. serg , please review these two commits: f1a28b57 fix race in the MDEV-32614 test 65c6417e MDEV-33450 Assertion fails in main.alter_table_online_debug

        ok to push, but I suggest to change the commit comment to something like

        MDEV-33450 Assertion fails in main.alter_table_online_debug

        `TABLE_SHARE` that is being online-altered has a shared `s->online_alter_binlog` member that
        all concurrent DMLs are writing to. Online alter thread deletes it under the MDL_EXCLUSIVE.
        If upgrading the lock to MDL_EXCLUSIVE fails, table as marked as `flushed` and it's freed
        automatically when its usage drops to zero.

        In 3059f27 the lock upgrade was relaxed to MDL_SHARED_NO_WRITE to allow concurrent SELECT
        threads during the final `online_alter_read_from_binlog()` pass. An attempt to upgrade the lock
        to MDL_EXCLUSIVE was still happening, but much later — after the code that marked the table `flushed`.
        That is, if the upgrade failed, the table was left with a stale `s->online_alter_binlog` triggering an assert
        in a future online alter.

        To fix this, let's upgrade the lock to MDL_EXCLUSIVE earlier. After the final `online_alter_read_from_binlog()`
        pass, but before the table is marked `flushed`.

        serg Sergei Golubchik added a comment - ok to push, but I suggest to change the commit comment to something like MDEV-33450 Assertion fails in main.alter_table_online_debug `TABLE_SHARE` that is being online-altered has a shared `s->online_alter_binlog` member that all concurrent DMLs are writing to. Online alter thread deletes it under the MDL_EXCLUSIVE. If upgrading the lock to MDL_EXCLUSIVE fails, table as marked as `flushed` and it's freed automatically when its usage drops to zero. In 3059f27 the lock upgrade was relaxed to MDL_SHARED_NO_WRITE to allow concurrent SELECT threads during the final `online_alter_read_from_binlog()` pass. An attempt to upgrade the lock to MDL_EXCLUSIVE was still happening, but much later — after the code that marked the table `flushed`. That is, if the upgrade failed, the table was left with a stale `s->online_alter_binlog` triggering an assert in a future online alter. To fix this, let's upgrade the lock to MDL_EXCLUSIVE earlier. After the final `online_alter_read_from_binlog()` pass, but before the table is marked `flushed`.

        People

          nikitamalyavin Nikita Malyavin
          nikitamalyavin Nikita Malyavin
          Votes:
          0 Vote for this issue
          Watchers:
          4 Start watching this issue

          Dates

            Created:
            Updated:
            Resolved:

            Git Integration

              Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.