Uploaded image for project: 'MariaDB Server'
  1. MariaDB Server
  2. MDEV-31101

spider/bugfix.mdev_29904 fails with "Server shutdown in progress" in CI

Details

    Description

      E.g. https://buildbot.mariadb.net/buildbot/builders/kvm-asan/builds/10208/steps/mtr_nm/logs/stdio

      I am not sure why this happens and I cannot reproduce it at the same commit. Here is the complete test:

      --echo #
      --echo # MDEV-29904 SPIDER plugin initialization fails upon startup
      --echo #
       
      --let $restart_parameters=--plugin-load-add=ha_spider
      --source include/restart_mysqld.inc
      

      According to [1] the error message means some query is still running while the server is shutting down. Maybe mtr tries to shutdown the server after --source include/restart_mysqld.inc while the spider init queries are still running in another thread? In that case perhaps we need to introduce a wait at the end of the test

      [1] https://www.percona.com/blog/error-mysqld-sort-aborted-server-shutdown-in-progress/

      To add a wait condition perhaps we could use include/wait_condition.inc, but that would require a sql statement to determine the completion, but looking at at spd_init_query.h, I don't see which statement can achieve this, so maybe we'll need to add an init query that signals completion.

      On the other hand, if spider init queries are no longer running in a background thread, then this bug may be fixed automatically, so let's block this ticket with the init bug ticket MDEV-22979 where we are considering such change.

      Attachments

        Issue Links

          Activity

            ycp Yuchen Pei added a comment - - edited

            Happened again today locally on a 10.5 patched with commit to MDEV-30435 and running multiple tests with parallel=auto (mtr chose 8)

            ./mysql-test/mtr --parallel=auto --force --max-test-fail=0 spider/handler.direct_aggregate_part spider/handler.spider_fixes spider.direct_aggregate_part spider/bugfix.mdev_29904 spider.direct_aggregate spider.direct_aggregate_part spider.checksum_table_with_quick_mode_3 spider.basic_sql spider.basic_sql_part spider.auto_increment
            

            ycp Yuchen Pei added a comment - - edited Happened again today locally on a 10.5 patched with commit to MDEV-30435 and running multiple tests with parallel=auto (mtr chose 8) ./mysql-test/mtr --parallel=auto --force --max-test-fail=0 spider/handler.direct_aggregate_part spider/handler.spider_fixes spider.direct_aggregate_part spider/bugfix.mdev_29904 spider.direct_aggregate spider.direct_aggregate_part spider.checksum_table_with_quick_mode_3 spider.basic_sql spider.basic_sql_part spider.auto_increment
            ycp Yuchen Pei added a comment -

            I'm going to disable mdev_29904.test until the present issue is no longer blocked by MDEV-22979

            ycp Yuchen Pei added a comment - I'm going to disable mdev_29904.test until the present issue is no longer blocked by MDEV-22979
            ycp Yuchen Pei added a comment -

            I have a patch on top of my current patches for MDEV-29870 which is
            under review. It works locally - neither spider/bugfix.mdev_29904 nor
            spider/bugfix.mdev_27575 fails. Will need to check CI.

            e9ea9353d53 upstream/bb-10.4-mdev-31101 MDEV-31101 Re-enable spider/bugfix.mdev_29904

            ycp Yuchen Pei added a comment - I have a patch on top of my current patches for MDEV-29870 which is under review. It works locally - neither spider/bugfix.mdev_29904 nor spider/bugfix.mdev_27575 fails. Will need to check CI. e9ea9353d53 upstream/bb-10.4-mdev-31101 MDEV-31101 Re-enable spider/bugfix.mdev_29904
            ycp Yuchen Pei added a comment -

            Hi holyfoot, ptal this trivial patch, thanks

            bb-10.4-mdev-31101 5710179d323e58e8f549fb48635be0600574b211
            MDEV-31101 Re-enable spider/bugfix.mdev_29904
             
            The spider init bug fixes remove any race conditions during spider
            init.
             
            Also remove the add_suppressions in spider/bugfix.mdev_27575 which is
            a similar issue.
            

            ycp Yuchen Pei added a comment - Hi holyfoot , ptal this trivial patch, thanks bb-10.4-mdev-31101 5710179d323e58e8f549fb48635be0600574b211 MDEV-31101 Re-enable spider/bugfix.mdev_29904   The spider init bug fixes remove any race conditions during spider init.   Also remove the add_suppressions in spider/bugfix.mdev_27575 which is a similar issue.

            ok to push.

            holyfoot Alexey Botchkov added a comment - ok to push.
            ycp Yuchen Pei added a comment -

            pushed d277a63c749bd49c9025da4efdd3008d22180dc0 to 10.4

            ycp Yuchen Pei added a comment - pushed d277a63c749bd49c9025da4efdd3008d22180dc0 to 10.4

            People

              ycp Yuchen Pei
              ycp Yuchen Pei
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Git Integration

                  Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.