Uploaded image for project: 'MariaDB Server'
  1. MariaDB Server
  2. MDEV-29141

InnoDB hangs some time after performing live VM migration

    XMLWordPrintable

Details

    Description

      Hello,

      The first 10.6 release we used was 10.6.5. After upgrading to 10.6.5, whenever we perform live VM migrations we sometimes see that InnoDB will at some point no longer respond. We also see this behaviour in 10.6.8.

      We perform a live VM migration on XCP-ng (xenserver). We have done this for about ~300 VMs, and about ~40 had issues. We would see the issues ranging from about a few minutes after the migration up to ~2 weeks after the migration (usually on less busy servers it would manifest later).

      Queries remain stuck in the states Updating, Sending Data, Statistics, Filling Schema table, Commit and just never complete.
      Running the command "Show engine innodb status" will hang indefinitely and never give any output.

      A restart of mysql also will not work. we have to kill -9 in order to restart mysql. After that it works again.

      We have not noticed this on MariaDB 10.5 and 10.4.

      We have noticed this on single instances as well as instances running galera.

      I did make a gcore of one of the instances that has issues (it is almost 5GB). Perhaps I can do anything with that, but I'm not sure what.

      Any ideas on what is wrong here?

      Attachments

        1. deadlock1-broke-state.json
          2 kB
          Niels Hendriks
        2. deadlock2-after-restart-working-state.json
          2 kB
          Niels Hendriks
        3. gdb.txt
          292 kB
          Niels Hendriks

        Issue Links

          Activity

            People

              marko Marko Mäkelä
              nielsh Niels Hendriks
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Git Integration

                  Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.