Uploaded image for project: 'MariaDB Server'
  1. MariaDB Server
  2. MDEV-16067

Invalid deadlock detection

Details

    Description

      After upgrading from Debian Jessie to Debian Stretch using MariaDB 10.1.32 packages from mariadb.org, we immediately started having problems with our backup software, Bacula ( www.bacula.org ) - the backup jobs are failing with this error:
      Deadlock found when trying to get lock; try restarting transaction

      This is happening every night for different backup jobs (probably random) and has NEVER happened before. While upgrading Debian, there was only a minor version change of Bacula (7.4.3 to 7.4.4). From my observation, error is happening for randomly for same SQL commands, sometimes after 300 seconds, sometimes after 2000 seconds of execution/waiting for locks. This is happening even after i set this:
      innodb_lock_wait_timeout = 100000

      I also tried upgrading to 10.2.14 and set this:
      innodb_deadlock_detect = 0

      Which does nothing, problem persists.

      Attachments

        Issue Links

          duplicates

          Bug - A problem which impairs or prevents the functions of the product. MDEV-13333 Deadlock failure that does not occur elsewhere

          • Major - Major loss of function.
          • Closed

          Activity

            Ascending order - Click to sort in descending order
            azurit azurit created issue -
            elenst Elena Stepanova added a comment -

            azurit,

            • from which version did you upgrade?
            • please paste or attach your config file(s).

            mleich, could you please take a look into this, whether we have a regression in 10.1.32 comparing to the version that azurit used before?

            marko, do you think it might be related to recent changes in persistent statistics?

            elenst Elena Stepanova added a comment - azurit , from which version did you upgrade? please paste or attach your config file(s). mleich , could you please take a look into this, whether we have a regression in 10.1.32 comparing to the version that azurit used before? marko , do you think it might be related to recent changes in persistent statistics?
            elenst Elena Stepanova made changes -
            Field Original Value New Value
            Component/s Locking [ 10900 ]
            Component/s Storage Engine - InnoDB [ 10129 ]
            Assignee Matthias Leich [ mleich ]
            azurit azurit added a comment -

            Before Debian upgrade, we used MariaDB 10.1.32 for Debian Jessie from mariadb.org, after upgrade we used the same version (10.1.32) also from mariadb.org but for Debian Stretch (we correctly set the Debian version in repository settings before upgrade). Attaching my.cnf.

            I know that the version of MariaDB didn't change but, for example, why am i getting that error even with innodb_deadlock_detect = 0 ?

            azurit azurit added a comment - Before Debian upgrade, we used MariaDB 10.1.32 for Debian Jessie from mariadb.org, after upgrade we used the same version (10.1.32) also from mariadb.org but for Debian Stretch (we correctly set the Debian version in repository settings before upgrade). Attaching my.cnf. I know that the version of MariaDB didn't change but, for example, why am i getting that error even with innodb_deadlock_detect = 0 ?
            azurit azurit made changes -
            Attachment my.cnf [ 45575 ]
            azurit azurit added a comment -

            I just found MDEV-13333 , which was opened by Bacula author.

            azurit azurit added a comment - I just found MDEV-13333 , which was opened by Bacula author.
            elenst Elena Stepanova added a comment -

            alice, you did a lot of work on MDEV-13333, do you remember anything about it?

            From comments there it appears that the problem occurred on 10.2 but not on 10.1, which is somewhat understandable. Can you think of any explanation why the behavior could change on the same MariaDB version only after Bacula upgrade and switching from MariaDB server packaged by MariaDB to Debian packages? Some changes in Bacula scripts, maybe? Or a difference in configuration files in MariaDB packages vs Debian's?

            elenst Elena Stepanova added a comment - alice , you did a lot of work on MDEV-13333 , do you remember anything about it? From comments there it appears that the problem occurred on 10.2 but not on 10.1, which is somewhat understandable. Can you think of any explanation why the behavior could change on the same MariaDB version only after Bacula upgrade and switching from MariaDB server packaged by MariaDB to Debian packages? Some changes in Bacula scripts, maybe? Or a difference in configuration files in MariaDB packages vs Debian's?
            elenst Elena Stepanova made changes -
            azurit azurit added a comment -

            I was able to downgrade to 10.1 and problem seems to be gone.

            azurit azurit added a comment - I was able to downgrade to 10.1 and problem seems to be gone.
            serg Sergei Golubchik made changes -
            Assignee Matthias Leich [ mleich ] Alice Sherepa [ alice ]
            alice Alice Sherepa added a comment -

            azurit, it looks like it is the same problem as MDEV-13333, so I suggest to wait for MDEV-13333 to be fixed.
            If the bug will be still reproducible, then it will be investigated further.

            alice Alice Sherepa added a comment - azurit , it looks like it is the same problem as MDEV-13333 , so I suggest to wait for MDEV-13333 to be fixed. If the bug will be still reproducible, then it will be investigated further.
            jplindst Jan Lindström (Inactive) added a comment -

            @azurit try set global innodb_lock_schedule_algorithm=FCFS;

            jplindst Jan Lindström (Inactive) added a comment - @azurit try set global innodb_lock_schedule_algorithm=FCFS;
            azurit azurit added a comment -

            I'm, currently, on 10.1 so it is already set to 'fcfs'. The problem is still occuring but it's not so often as on 10.2. Any suggestions how to bypass it on 10.1? It is really doing us lots of trouble.

            azurit azurit added a comment - I'm, currently, on 10.1 so it is already set to 'fcfs'. The problem is still occuring but it's not so often as on 10.2. Any suggestions how to bypass it on 10.1? It is really doing us lots of trouble.
            alice Alice Sherepa added a comment -

            azurit MDEV-13333 is fixed now, are you able to try 10.1.36 to confirm that the problem is solved?

            alice Alice Sherepa added a comment - azurit MDEV-13333 is fixed now, are you able to try 10.1.36 to confirm that the problem is solved?
            alice Alice Sherepa made changes -
            Labels need_feedback
            azurit azurit added a comment -

            I cannot tell for 100%, because the problem was appearing randomly, but it looks good so far. Thank you!

            azurit azurit added a comment - I cannot tell for 100%, because the problem was appearing randomly, but it looks good so far. Thank you!
            alice Alice Sherepa made changes -
            Labels need_feedback
            alice Alice Sherepa made changes -
            alice Alice Sherepa made changes -
            alice Alice Sherepa made changes -
            Fix Version/s N/A [ 14700 ]
            Resolution Duplicate [ 3 ]
            Status Open [ 1 ] Closed [ 6 ]
            serg Sergei Golubchik made changes -
            Workflow MariaDB v3 [ 86867 ] MariaDB v4 [ 154288 ]

            People

              alice Alice Sherepa
              azurit azurit
              Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Git Integration

                  Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.