Uploaded image for project: 'MariaDB Server'
  1. MariaDB Server
  2. MDEV-30084

Shutdown hangs in some tests

    XMLWordPrintable

Details

    Description

      The test main.bootstrap_innodb is often failing with a message like this:

      10.7 22ab79c43079d9fe0ca136148fa6fd05a5aa4d5a

      main.bootstrap_innodb 'innodb'           w23 [ fail ]  Found warnings/errors in server log file!
              Test ended at 2022-11-24 06:57:37
      line
      Attempting backtrace. You can use the following information to find out
      

      In the corresponding server error log, we can find a hint of the real cause:

      CURRENT_TEST: main.bootstrap_innodb
      2022-11-24  6:56:34 0 [Note] /home/buildbot/amd64-ubuntu-2004-debug/build/sql/mariadbd (initiated by: root[root] @ localhost []): Normal shutdown
      221124  6:57:34 [ERROR] mysqld got signal 6 ;
      

      Because 60 seconds (1 minute) passed between the two messages, the server process must have been killed by mysqlclient. That is, the 60-second shutdown timeout elapsed.

      In either log, I am only seeing the stack trace of a single thread. The cause of the hang is thus unknown.

      Another hang:

      10.9 d48db97d0abec7b310fed362da93c575cbbd7124

      main.statistics_upgrade_not_done 'innodb' w27 [ fail ]  Found warnings/errors in server log file!
              Test ended at 2022-11-24 07:07:19
      line
      Attempting backtrace. You can use the following information to find out
      

      Again, the server error log shows a time difference of 60 seconds (1 minute) between shutdown initiation and the kill -ABRT by the test harness:

      CURRENT_TEST: main.statistics_upgrade_not_done
      2022-11-24  7:06:16 317 [ERROR] Incorrect definition of table mysql.column_stats: expected column 'hist_type' at position 9 to have type enum('SINGLE_PREC_HB','DOUBLE_PREC_HB','JSON_HB'), found type enum('SINGLE_PREC_HB','DOUBLE_PREC_HB').
      2022-11-24  7:06:16 317 [ERROR] Incorrect definition of table mysql.column_stats: expected column 'histogram' at position 10 to have type longblob, found type varbinary(255).
      2022-11-24  7:06:16 317 [ERROR] Incorrect definition of table mysql.column_stats: expected column 'hist_type' at position 9 to have type enum('SINGLE_PREC_HB','DOUBLE_PREC_HB','JSON_HB'), found type enum('SINGLE_PREC_HB','DOUBLE_PREC_HB').
      2022-11-24  7:06:16 317 [ERROR] Incorrect definition of table mysql.column_stats: expected column 'histogram' at position 10 to have type longblob, found type varbinary(255).
      2022-11-24  7:06:16 0 [Note] /home/buildbot/amd64-ubuntu-2204-debug-ps/build/sql/mariadbd (initiated by: root[root] @ localhost []): Normal shutdown
      221124  7:07:16 [ERROR] mysqld got signal 6 ;
      

      Because both hangs occurred on mandatory staging builders and required the builds to be rerun (incurring an extra wait of over 30 minutes) I think that this needs to be treated at a high priority.

      Attachments

        Issue Links

          Activity

            People

              serg Sergei Golubchik
              marko Marko Mäkelä
              Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Git Integration

                  Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.