Uploaded image for project: 'MariaDB Server'
  1. MariaDB Server
  2. MDEV-16791

mariabackup : allow consistent backup, in presence of concurrent DDL, also without --lock-ddl-per-table

Details

    Description

      There is a lot of problems with mariabackup, if DDL statements are executed in parallel to backup.
      Some of them show up even if --lock-ddl-per-table is used.

      A fundamental problem with lock-ddl-per-table is that only those tables, that exist at the start of backup, are MDL-locked. Another problem is that mariabackup will deadlock during backup's "FLUSH TABLE WITH READ LOCKS", if concurrent DDL statements run. To resolve this deadlock, we either KILL user's DDL query, or user has the option to omit FTWRL with unsafe --no-lock option (MDEV-15636)

      So, there are problems which we currently have with backup in presence of Innodb DDL.

      1. Tables created during backup are not there after prepare (with/without lock-ddl-per-table)
      2. Tables dropped during backup might show up after prepare (without lock-ddl-per-table), or backup fails if file that was at the start of backup, is not found during copy.
      3. If lock-ddl-per-table is used, acquiring MDL lock will fail, if the table is concurrently being dropped or renamed.
      4. Tables that are renamed during backup, do not show up after prepare, if rename happens after table was copied.
      5. Tables that are recreated (dropped, and created under the same name) during backup,after tablespace copy, break prepare due to different tablespace id.
      6. Backup fails with "ALTER TABLE or OPTIMIZE TABLE was executed during backup"
      7. Concurrent multi-rename is prone to race condition, for example table t1 can be missed from backup, if RENAME TABLE t1 to t2, t3 to t1 runs concurrently.

      A possible fix to some of those problems would be to

      • Tolerate missing files during innodb copy - missing files can happen when DROP or RENAME runs in parallel. Also allow MDL lock failures in --lock-ddl-per-table.
      • At the end of backup, under protection of FTWRL, during the stage when frm and MyISAM files are copied, rescan data directory, looking for Innodb tablespaces, and copy those that are not already in backup. This will pick up tables from parallel CREATE or RENAME.
      • We might need to remove orphan (frm-less) .ibd files in prepare - there can be some left due to RENAME, or DROP.
      • Recreated (DROP/CREATE) tablespaces, that have changed tablespace ids at the end of backup. We might need to copy them to backup second time. This is be tricky for streaming backup, since xbstream format does not support multiple copies of the same file , and does not have any "delete" command. One way to workaround it, it to give second copy special extension, e.g "ibd.new" , so that "prepare" would know it can replace old copy with the new one.

      Maybe some of the above can be taken care by Innodb recovery, I'm not really sure how this would work. marko, any idea on that?
      "ALTER TABLE or OPTIMIZE TABLE was executed during backup" must be taken care by Innodb however.

      Attachments

        Issue Links

          Activity

            wlad Vladislav Vaintroub created issue -
            wlad Vladislav Vaintroub made changes -
            Field Original Value New Value
            Summary Backup Innodb fails , or is inconsistent, if DDL statements run during backup mariabackup : allow DDL on Innodb tables, without --lock-ddl-per-table
            wlad Vladislav Vaintroub made changes -
            Affects Version/s 10.1 [ 16100 ]
            Affects Version/s 10.2 [ 14601 ]
            Affects Version/s 10.3 [ 22126 ]
            wlad Vladislav Vaintroub made changes -
            Affects Version/s 10.2 [ 14601 ]
            Affects Version/s 10.1 [ 16100 ]
            Affects Version/s 10.3 [ 22126 ]
            Issue Type Bug [ 1 ] Task [ 3 ]
            wlad Vladislav Vaintroub made changes -
            Fix Version/s 10.2 [ 14601 ]
            Fix Version/s 10.3 [ 22126 ]
            wlad Vladislav Vaintroub made changes -
            Description There is a lot of problems with mariabackup, if DDL statements are executed in parallel to backup.
            Some of them show up even if --lock-ddl-per-table is used.

            A fundamental problem with lock-ddl-per-table is that only tables that exist at the start of backup are locked. Another one is that it will deadlock during backup's "FLUSH TABLE WITH READ LOCKS", if concurrent DDL statements runs, and to resolve the deadlock, we either KILL user's DDL query, or omit FTWRL with *unsafe* --no-lock option (MDEV-15636)

            # Tables created during backup are not there after prepare (with/without lock-ddl-per-table)
            # Tables dropped during backup might show up after prepare (without lock-ddl-per-table), or backup fails if file that was at the start of backup, is not found during copy.
            # Tables that are renamed during backup, do not show up after prepare, if rename happens after table was renamed.
            # Tables that are recreated (dropped, and created under the same name) during backup, break prepare due to different tablespace id.
            # Backup fails with "ALTER TABLE or OPTIMIZE TABLE was executed during backup"


            wlad Vladislav Vaintroub made changes -
            Comment [ There is a lot of problems with mariabackup, if DDL statements are executed in parallel to backup.
            Some of them show up even if --lock-ddl-per-table is used.

            A fundamental problem with lock-ddl-per-table is that only tables that exist at the start of backup are locked. Another one is that it will deadlock during backup's "FLUSH TABLE WITH READ LOCKS", if concurrent DDL statements runs, and to resolve the deadlock, we either KILL user's DDL query, or omit FTWRL with *unsafe* --no-lock option (MDEV-15636)

            # Tables created during backup are not there after prepare (with/without lock-ddl-per-table)
            # Tables dropped during backup might show up after prepare (without lock-ddl-per-table), or backup fails if file that was at the start of backup, is not found during copy.
            # Tables that are renamed during backup, do not show up after prepare, if rename happens after table was renamed.
            # Tables that are recreated (dropped, and created under the same name) during backup, break prepare due to different tablespace id.
            # Backup fails with "ALTER TABLE or OPTIMIZE TABLE was executed during backup"


            ]
            wlad Vladislav Vaintroub made changes -
            wlad Vladislav Vaintroub made changes -
            Description There is a lot of problems with mariabackup, if DDL statements are executed in parallel to backup.
            Some of them show up even if --lock-ddl-per-table is used.

            A fundamental problem with lock-ddl-per-table is that only tables that exist at the start of backup are locked. Another one is that it will deadlock during backup's "FLUSH TABLE WITH READ LOCKS", if concurrent DDL statements runs, and to resolve the deadlock, we either KILL user's DDL query, or omit FTWRL with *unsafe* --no-lock option (MDEV-15636)

            # Tables created during backup are not there after prepare (with/without lock-ddl-per-table)
            # Tables dropped during backup might show up after prepare (without lock-ddl-per-table), or backup fails if file that was at the start of backup, is not found during copy.
            # Tables that are renamed during backup, do not show up after prepare, if rename happens after table was renamed.
            # Tables that are recreated (dropped, and created under the same name) during backup, break prepare due to different tablespace id.
            # Backup fails with "ALTER TABLE or OPTIMIZE TABLE was executed during backup"


            There is a lot of problems with mariabackup, if DDL statements are executed in parallel to backup.
            Some of them show up even if --lock-ddl-per-table is used.

            A fundamental problem with lock-ddl-per-table is that only tables that exist at the start of backup are locked. Another one is that it will deadlock during backup's "FLUSH TABLE WITH READ LOCKS", if concurrent DDL statements runs, and to resolve the deadlock, we either KILL user's DDL query, or omit FTWRL with *unsafe* --no-lock option (MDEV-15636)

            # Tables created during backup are not there after prepare (with/without lock-ddl-per-table)
            # Tables dropped during backup might show up after prepare (without lock-ddl-per-table), or backup fails if file that was at the start of backup, is not found during copy.
            # Tables that are renamed during backup, do not show up after prepare, if rename happens after table was renamed.
            # Tables that are recreated (dropped, and created under the same name) during backup,after tablespace copy, break prepare due to different tablespace id.
            # Backup fails with "ALTER TABLE or OPTIMIZE TABLE was executed during backup"


            wlad Vladislav Vaintroub made changes -
            Description There is a lot of problems with mariabackup, if DDL statements are executed in parallel to backup.
            Some of them show up even if --lock-ddl-per-table is used.

            A fundamental problem with lock-ddl-per-table is that only tables that exist at the start of backup are locked. Another one is that it will deadlock during backup's "FLUSH TABLE WITH READ LOCKS", if concurrent DDL statements runs, and to resolve the deadlock, we either KILL user's DDL query, or omit FTWRL with *unsafe* --no-lock option (MDEV-15636)

            # Tables created during backup are not there after prepare (with/without lock-ddl-per-table)
            # Tables dropped during backup might show up after prepare (without lock-ddl-per-table), or backup fails if file that was at the start of backup, is not found during copy.
            # Tables that are renamed during backup, do not show up after prepare, if rename happens after table was renamed.
            # Tables that are recreated (dropped, and created under the same name) during backup,after tablespace copy, break prepare due to different tablespace id.
            # Backup fails with "ALTER TABLE or OPTIMIZE TABLE was executed during backup"


            There is a lot of problems with mariabackup, if DDL statements are executed in parallel to backup.
            Some of them show up even if --lock-ddl-per-table is used.

            A fundamental problem with lock-ddl-per-table is that only those tables, that exist at the start of backup, are MDL-locked. Another problem is that mariabackup will deadlock during backup's "FLUSH TABLE WITH READ LOCKS", if concurrent DDL statements run. To resolve this deadlock, we either KILL user's DDL query, or user has the option to omit FTWRL with *unsafe* --no-lock option (MDEV-15636)

            So, there are problems which we currently have with backup in presence of Innodb DDL.

            # Tables created during backup are not there after prepare (with/without lock-ddl-per-table)
            # Tables dropped during backup might show up after prepare (without lock-ddl-per-table), or backup fails if file that was at the start of backup, is not found during copy.
            # If lock-ddl-per-table is used, acquiring MDL lock will fail, if the table is concurrently being dropped or renamed.
            # Tables that are renamed during backup, do not show up after prepare, if rename happens after table was renamed.
            # Tables that are recreated (dropped, and created under the same name) during backup,after tablespace copy, break prepare due to different tablespace id.
            # Backup fails with "ALTER TABLE or OPTIMIZE TABLE was executed during backup"
            # Concurrent multi-rename is prone to race condition, for example table t1 can be missed from backup, if RENAME TABLE t1 to t2, t3 to t1 runs concurrently.


            wlad Vladislav Vaintroub made changes -
            Description There is a lot of problems with mariabackup, if DDL statements are executed in parallel to backup.
            Some of them show up even if --lock-ddl-per-table is used.

            A fundamental problem with lock-ddl-per-table is that only those tables, that exist at the start of backup, are MDL-locked. Another problem is that mariabackup will deadlock during backup's "FLUSH TABLE WITH READ LOCKS", if concurrent DDL statements run. To resolve this deadlock, we either KILL user's DDL query, or user has the option to omit FTWRL with *unsafe* --no-lock option (MDEV-15636)

            So, there are problems which we currently have with backup in presence of Innodb DDL.

            # Tables created during backup are not there after prepare (with/without lock-ddl-per-table)
            # Tables dropped during backup might show up after prepare (without lock-ddl-per-table), or backup fails if file that was at the start of backup, is not found during copy.
            # If lock-ddl-per-table is used, acquiring MDL lock will fail, if the table is concurrently being dropped or renamed.
            # Tables that are renamed during backup, do not show up after prepare, if rename happens after table was renamed.
            # Tables that are recreated (dropped, and created under the same name) during backup,after tablespace copy, break prepare due to different tablespace id.
            # Backup fails with "ALTER TABLE or OPTIMIZE TABLE was executed during backup"
            # Concurrent multi-rename is prone to race condition, for example table t1 can be missed from backup, if RENAME TABLE t1 to t2, t3 to t1 runs concurrently.


            There is a lot of problems with mariabackup, if DDL statements are executed in parallel to backup.
            Some of them show up even if --lock-ddl-per-table is used.

            A fundamental problem with lock-ddl-per-table is that only those tables, that exist at the start of backup, are MDL-locked. Another problem is that mariabackup will deadlock during backup's "FLUSH TABLE WITH READ LOCKS", if concurrent DDL statements run. To resolve this deadlock, we either KILL user's DDL query, or user has the option to omit FTWRL with *unsafe* --no-lock option (MDEV-15636)

            So, there are problems which we currently have with backup in presence of Innodb DDL.

            # Tables created during backup are not there after prepare (with/without lock-ddl-per-table)
            # Tables dropped during backup might show up after prepare (without lock-ddl-per-table), or backup fails if file that was at the start of backup, is not found during copy.
            # If lock-ddl-per-table is used, acquiring MDL lock will fail, if the table is concurrently being dropped or renamed.
            # Tables that are renamed during backup, do not show up after prepare, if rename happens after table was renamed.
            # Tables that are recreated (dropped, and created under the same name) during backup,after tablespace copy, break prepare due to different tablespace id.
            # Backup fails with "ALTER TABLE or OPTIMIZE TABLE was executed during backup"
            # Concurrent multi-rename is prone to race condition, for example table t1 can be missed from backup, if RENAME TABLE t1 to t2, t3 to t1 runs concurrently.


            A possible fix to some of those problems would be to
            * allow files to be missing during innodb copy (this happens when drop or rename runs in parallel), and allow MDL lock failures.
            * At the end of backup, under protection of FTWRL (at the stage where frm and MyISAM files are copied). rescan data directory, looking for Innodb tablespaces that are not already in the backup, and copy them . This will pick up new tables, created during backup, or renamed tables.
            * I think, for the recreated tables, that have changed tablespace ids at the end of backup, we'd might need to copy them to again, at the end of backup.

            So of the above can be taken care by recovery, but I'm not really sure how this would work. [~marko], any idea on that-
             "ALTER TABLE or OPTIMIZE TABLE was executed during backup" must be taken care by recovery.


            wlad Vladislav Vaintroub made changes -
            Description There is a lot of problems with mariabackup, if DDL statements are executed in parallel to backup.
            Some of them show up even if --lock-ddl-per-table is used.

            A fundamental problem with lock-ddl-per-table is that only those tables, that exist at the start of backup, are MDL-locked. Another problem is that mariabackup will deadlock during backup's "FLUSH TABLE WITH READ LOCKS", if concurrent DDL statements run. To resolve this deadlock, we either KILL user's DDL query, or user has the option to omit FTWRL with *unsafe* --no-lock option (MDEV-15636)

            So, there are problems which we currently have with backup in presence of Innodb DDL.

            # Tables created during backup are not there after prepare (with/without lock-ddl-per-table)
            # Tables dropped during backup might show up after prepare (without lock-ddl-per-table), or backup fails if file that was at the start of backup, is not found during copy.
            # If lock-ddl-per-table is used, acquiring MDL lock will fail, if the table is concurrently being dropped or renamed.
            # Tables that are renamed during backup, do not show up after prepare, if rename happens after table was renamed.
            # Tables that are recreated (dropped, and created under the same name) during backup,after tablespace copy, break prepare due to different tablespace id.
            # Backup fails with "ALTER TABLE or OPTIMIZE TABLE was executed during backup"
            # Concurrent multi-rename is prone to race condition, for example table t1 can be missed from backup, if RENAME TABLE t1 to t2, t3 to t1 runs concurrently.


            A possible fix to some of those problems would be to
            * allow files to be missing during innodb copy (this happens when drop or rename runs in parallel), and allow MDL lock failures.
            * At the end of backup, under protection of FTWRL (at the stage where frm and MyISAM files are copied). rescan data directory, looking for Innodb tablespaces that are not already in the backup, and copy them . This will pick up new tables, created during backup, or renamed tables.
            * I think, for the recreated tables, that have changed tablespace ids at the end of backup, we'd might need to copy them to again, at the end of backup.

            So of the above can be taken care by recovery, but I'm not really sure how this would work. [~marko], any idea on that-
             "ALTER TABLE or OPTIMIZE TABLE was executed during backup" must be taken care by recovery.


            There is a lot of problems with mariabackup, if DDL statements are executed in parallel to backup.
            Some of them show up even if --lock-ddl-per-table is used.

            A fundamental problem with lock-ddl-per-table is that only those tables, that exist at the start of backup, are MDL-locked. Another problem is that mariabackup will deadlock during backup's "FLUSH TABLE WITH READ LOCKS", if concurrent DDL statements run. To resolve this deadlock, we either KILL user's DDL query, or user has the option to omit FTWRL with *unsafe* --no-lock option (MDEV-15636)

            So, there are problems which we currently have with backup in presence of Innodb DDL.

            # Tables created during backup are not there after prepare (with/without lock-ddl-per-table)
            # Tables dropped during backup might show up after prepare (without lock-ddl-per-table), or backup fails if file that was at the start of backup, is not found during copy.
            # If lock-ddl-per-table is used, acquiring MDL lock will fail, if the table is concurrently being dropped or renamed.
            # Tables that are renamed during backup, do not show up after prepare, if rename happens after table was renamed.
            # Tables that are recreated (dropped, and created under the same name) during backup,after tablespace copy, break prepare due to different tablespace id.
            # Backup fails with "ALTER TABLE or OPTIMIZE TABLE was executed during backup"
            # Concurrent multi-rename is prone to race condition, for example table t1 can be missed from backup, if RENAME TABLE t1 to t2, t3 to t1 runs concurrently.


            A possible fix to some of those problems would be to
            * allow files to be missing during innodb copy - missing files can happens when DROP or RENAME runs in parallel. Also allow MDL lock failures in --lock-ddl-per-table
            * At the end of backup, under protection of FTWRL, during the stage when frm and MyISAM files are copied, rescan data directory, looking for Innodb tablespaces, and copy those that are not already in backup. This will pick up tables from CREATE during backup, or RENAMEd tables.
            * We might need to remove orphan (frm-less) .ibd files in prepare - there can be some left due to RENAME.
            * Recreated (DROP/CREATE) tablespaces, that have changed tablespace ids at the end of backup. We might need to copy them to backup second time at the end of backup. This is be tricky for streaming backup, since xbstream format does not support multiple copies , or "delete" command. One way to workaround it, it to give second copy special extension, e.g "ibd.new" , so that "prepare" would know it can replace old copy with the new one.

            Maybe some of the above can be taken care by Innodb recovery, I'm not really sure how this would work. [~marko], any idea on that?
             "ALTER TABLE or OPTIMIZE TABLE was executed during backup" *must* be taken care by recovery however.


            wlad Vladislav Vaintroub made changes -
            Description There is a lot of problems with mariabackup, if DDL statements are executed in parallel to backup.
            Some of them show up even if --lock-ddl-per-table is used.

            A fundamental problem with lock-ddl-per-table is that only those tables, that exist at the start of backup, are MDL-locked. Another problem is that mariabackup will deadlock during backup's "FLUSH TABLE WITH READ LOCKS", if concurrent DDL statements run. To resolve this deadlock, we either KILL user's DDL query, or user has the option to omit FTWRL with *unsafe* --no-lock option (MDEV-15636)

            So, there are problems which we currently have with backup in presence of Innodb DDL.

            # Tables created during backup are not there after prepare (with/without lock-ddl-per-table)
            # Tables dropped during backup might show up after prepare (without lock-ddl-per-table), or backup fails if file that was at the start of backup, is not found during copy.
            # If lock-ddl-per-table is used, acquiring MDL lock will fail, if the table is concurrently being dropped or renamed.
            # Tables that are renamed during backup, do not show up after prepare, if rename happens after table was renamed.
            # Tables that are recreated (dropped, and created under the same name) during backup,after tablespace copy, break prepare due to different tablespace id.
            # Backup fails with "ALTER TABLE or OPTIMIZE TABLE was executed during backup"
            # Concurrent multi-rename is prone to race condition, for example table t1 can be missed from backup, if RENAME TABLE t1 to t2, t3 to t1 runs concurrently.


            A possible fix to some of those problems would be to
            * allow files to be missing during innodb copy - missing files can happens when DROP or RENAME runs in parallel. Also allow MDL lock failures in --lock-ddl-per-table
            * At the end of backup, under protection of FTWRL, during the stage when frm and MyISAM files are copied, rescan data directory, looking for Innodb tablespaces, and copy those that are not already in backup. This will pick up tables from CREATE during backup, or RENAMEd tables.
            * We might need to remove orphan (frm-less) .ibd files in prepare - there can be some left due to RENAME.
            * Recreated (DROP/CREATE) tablespaces, that have changed tablespace ids at the end of backup. We might need to copy them to backup second time at the end of backup. This is be tricky for streaming backup, since xbstream format does not support multiple copies , or "delete" command. One way to workaround it, it to give second copy special extension, e.g "ibd.new" , so that "prepare" would know it can replace old copy with the new one.

            Maybe some of the above can be taken care by Innodb recovery, I'm not really sure how this would work. [~marko], any idea on that?
             "ALTER TABLE or OPTIMIZE TABLE was executed during backup" *must* be taken care by recovery however.


            There is a lot of problems with mariabackup, if DDL statements are executed in parallel to backup.
            Some of them show up even if --lock-ddl-per-table is used.

            A fundamental problem with lock-ddl-per-table is that only those tables, that exist at the start of backup, are MDL-locked. Another problem is that mariabackup will deadlock during backup's "FLUSH TABLE WITH READ LOCKS", if concurrent DDL statements run. To resolve this deadlock, we either KILL user's DDL query, or user has the option to omit FTWRL with *unsafe* --no-lock option (MDEV-15636)

            So, there are problems which we currently have with backup in presence of Innodb DDL.

            # Tables created during backup are not there after prepare (with/without lock-ddl-per-table)
            # Tables dropped during backup might show up after prepare (without lock-ddl-per-table), or backup fails if file that was at the start of backup, is not found during copy.
            # If lock-ddl-per-table is used, acquiring MDL lock will fail, if the table is concurrently being dropped or renamed.
            # Tables that are renamed during backup, do not show up after prepare, if rename happens after table was renamed.
            # Tables that are recreated (dropped, and created under the same name) during backup,after tablespace copy, break prepare due to different tablespace id.
            # Backup fails with "ALTER TABLE or OPTIMIZE TABLE was executed during backup"
            # Concurrent multi-rename is prone to race condition, for example table t1 can be missed from backup, if RENAME TABLE t1 to t2, t3 to t1 runs concurrently.


            A possible fix to some of those problems would be to
            * allow files to be missing during innodb copy - missing files can happens when DROP or RENAME runs in parallel. Also allow MDL lock failures in --lock-ddl-per-table
            * At the end of backup, under protection of FTWRL, during the stage when frm and MyISAM files are copied, rescan data directory, looking for Innodb tablespaces, and copy those that are not already in backup. This will pick up tables from CREATE during backup, or RENAMEd tables.
            * We might need to remove orphan (frm-less) .ibd files in prepare - there can be some left due to RENAME, or DROP.
            * Recreated (DROP/CREATE) tablespaces, that have changed tablespace ids at the end of backup. We might need to copy them to backup second time at the end of backup. This is be tricky for streaming backup, since xbstream format does not support multiple copies , or "delete" command. One way to workaround it, it to give second copy special extension, e.g "ibd.new" , so that "prepare" would know it can replace old copy with the new one.

            Maybe some of the above can be taken care by Innodb recovery, I'm not really sure how this would work. [~marko], any idea on that?
             "ALTER TABLE or OPTIMIZE TABLE was executed during backup" *must* be taken care by recovery however.


            wlad Vladislav Vaintroub made changes -
            Description There is a lot of problems with mariabackup, if DDL statements are executed in parallel to backup.
            Some of them show up even if --lock-ddl-per-table is used.

            A fundamental problem with lock-ddl-per-table is that only those tables, that exist at the start of backup, are MDL-locked. Another problem is that mariabackup will deadlock during backup's "FLUSH TABLE WITH READ LOCKS", if concurrent DDL statements run. To resolve this deadlock, we either KILL user's DDL query, or user has the option to omit FTWRL with *unsafe* --no-lock option (MDEV-15636)

            So, there are problems which we currently have with backup in presence of Innodb DDL.

            # Tables created during backup are not there after prepare (with/without lock-ddl-per-table)
            # Tables dropped during backup might show up after prepare (without lock-ddl-per-table), or backup fails if file that was at the start of backup, is not found during copy.
            # If lock-ddl-per-table is used, acquiring MDL lock will fail, if the table is concurrently being dropped or renamed.
            # Tables that are renamed during backup, do not show up after prepare, if rename happens after table was renamed.
            # Tables that are recreated (dropped, and created under the same name) during backup,after tablespace copy, break prepare due to different tablespace id.
            # Backup fails with "ALTER TABLE or OPTIMIZE TABLE was executed during backup"
            # Concurrent multi-rename is prone to race condition, for example table t1 can be missed from backup, if RENAME TABLE t1 to t2, t3 to t1 runs concurrently.


            A possible fix to some of those problems would be to
            * allow files to be missing during innodb copy - missing files can happens when DROP or RENAME runs in parallel. Also allow MDL lock failures in --lock-ddl-per-table
            * At the end of backup, under protection of FTWRL, during the stage when frm and MyISAM files are copied, rescan data directory, looking for Innodb tablespaces, and copy those that are not already in backup. This will pick up tables from CREATE during backup, or RENAMEd tables.
            * We might need to remove orphan (frm-less) .ibd files in prepare - there can be some left due to RENAME, or DROP.
            * Recreated (DROP/CREATE) tablespaces, that have changed tablespace ids at the end of backup. We might need to copy them to backup second time at the end of backup. This is be tricky for streaming backup, since xbstream format does not support multiple copies , or "delete" command. One way to workaround it, it to give second copy special extension, e.g "ibd.new" , so that "prepare" would know it can replace old copy with the new one.

            Maybe some of the above can be taken care by Innodb recovery, I'm not really sure how this would work. [~marko], any idea on that?
             "ALTER TABLE or OPTIMIZE TABLE was executed during backup" *must* be taken care by recovery however.


            There is a lot of problems with mariabackup, if DDL statements are executed in parallel to backup.
            Some of them show up even if --lock-ddl-per-table is used.

            A fundamental problem with lock-ddl-per-table is that only those tables, that exist at the start of backup, are MDL-locked. Another problem is that mariabackup will deadlock during backup's "FLUSH TABLE WITH READ LOCKS", if concurrent DDL statements run. To resolve this deadlock, we either KILL user's DDL query, or user has the option to omit FTWRL with *unsafe* --no-lock option (MDEV-15636)

            So, there are problems which we currently have with backup in presence of Innodb DDL.

            # Tables created during backup are not there after prepare (with/without lock-ddl-per-table)
            # Tables dropped during backup might show up after prepare (without lock-ddl-per-table), or backup fails if file that was at the start of backup, is not found during copy.
            # If lock-ddl-per-table is used, acquiring MDL lock will fail, if the table is concurrently being dropped or renamed.
            # Tables that are renamed during backup, do not show up after prepare, if rename happens after table was copied.
            # Tables that are recreated (dropped, and created under the same name) during backup,after tablespace copy, break prepare due to different tablespace id.
            # Backup fails with "ALTER TABLE or OPTIMIZE TABLE was executed during backup"
            # Concurrent multi-rename is prone to race condition, for example table t1 can be missed from backup, if RENAME TABLE t1 to t2, t3 to t1 runs concurrently.


            A possible fix to some of those problems would be to
            * allow files to be missing during innodb copy - missing files can happens when DROP or RENAME runs in parallel. Also allow MDL lock failures in --lock-ddl-per-table
            * At the end of backup, under protection of FTWRL, during the stage when frm and MyISAM files are copied, rescan data directory, looking for Innodb tablespaces, and copy those that are not already in backup. This will pick up tables from CREATE during backup, or RENAMEd tables.
            * We might need to remove orphan (frm-less) .ibd files in prepare - there can be some left due to RENAME, or DROP.
            * Recreated (DROP/CREATE) tablespaces, that have changed tablespace ids at the end of backup. We might need to copy them to backup second time at the end of backup. This is be tricky for streaming backup, since xbstream format does not support multiple copies , or "delete" command. One way to workaround it, it to give second copy special extension, e.g "ibd.new" , so that "prepare" would know it can replace old copy with the new one.

            Maybe some of the above can be taken care by Innodb recovery, I'm not really sure how this would work. [~marko], any idea on that?
             "ALTER TABLE or OPTIMIZE TABLE was executed during backup" *must* be taken care by Innodb however.


            wlad Vladislav Vaintroub made changes -
            Description There is a lot of problems with mariabackup, if DDL statements are executed in parallel to backup.
            Some of them show up even if --lock-ddl-per-table is used.

            A fundamental problem with lock-ddl-per-table is that only those tables, that exist at the start of backup, are MDL-locked. Another problem is that mariabackup will deadlock during backup's "FLUSH TABLE WITH READ LOCKS", if concurrent DDL statements run. To resolve this deadlock, we either KILL user's DDL query, or user has the option to omit FTWRL with *unsafe* --no-lock option (MDEV-15636)

            So, there are problems which we currently have with backup in presence of Innodb DDL.

            # Tables created during backup are not there after prepare (with/without lock-ddl-per-table)
            # Tables dropped during backup might show up after prepare (without lock-ddl-per-table), or backup fails if file that was at the start of backup, is not found during copy.
            # If lock-ddl-per-table is used, acquiring MDL lock will fail, if the table is concurrently being dropped or renamed.
            # Tables that are renamed during backup, do not show up after prepare, if rename happens after table was copied.
            # Tables that are recreated (dropped, and created under the same name) during backup,after tablespace copy, break prepare due to different tablespace id.
            # Backup fails with "ALTER TABLE or OPTIMIZE TABLE was executed during backup"
            # Concurrent multi-rename is prone to race condition, for example table t1 can be missed from backup, if RENAME TABLE t1 to t2, t3 to t1 runs concurrently.


            A possible fix to some of those problems would be to
            * allow files to be missing during innodb copy - missing files can happens when DROP or RENAME runs in parallel. Also allow MDL lock failures in --lock-ddl-per-table
            * At the end of backup, under protection of FTWRL, during the stage when frm and MyISAM files are copied, rescan data directory, looking for Innodb tablespaces, and copy those that are not already in backup. This will pick up tables from CREATE during backup, or RENAMEd tables.
            * We might need to remove orphan (frm-less) .ibd files in prepare - there can be some left due to RENAME, or DROP.
            * Recreated (DROP/CREATE) tablespaces, that have changed tablespace ids at the end of backup. We might need to copy them to backup second time at the end of backup. This is be tricky for streaming backup, since xbstream format does not support multiple copies , or "delete" command. One way to workaround it, it to give second copy special extension, e.g "ibd.new" , so that "prepare" would know it can replace old copy with the new one.

            Maybe some of the above can be taken care by Innodb recovery, I'm not really sure how this would work. [~marko], any idea on that?
             "ALTER TABLE or OPTIMIZE TABLE was executed during backup" *must* be taken care by Innodb however.


            There is a lot of problems with mariabackup, if DDL statements are executed in parallel to backup.
            Some of them show up even if --lock-ddl-per-table is used.

            A fundamental problem with lock-ddl-per-table is that only those tables, that exist at the start of backup, are MDL-locked. Another problem is that mariabackup will deadlock during backup's "FLUSH TABLE WITH READ LOCKS", if concurrent DDL statements run. To resolve this deadlock, we either KILL user's DDL query, or user has the option to omit FTWRL with *unsafe* --no-lock option (MDEV-15636)

            So, there are problems which we currently have with backup in presence of Innodb DDL.

            # Tables created during backup are not there after prepare (with/without lock-ddl-per-table)
            # Tables dropped during backup might show up after prepare (without lock-ddl-per-table), or backup fails if file that was at the start of backup, is not found during copy.
            # If lock-ddl-per-table is used, acquiring MDL lock will fail, if the table is concurrently being dropped or renamed.
            # Tables that are renamed during backup, do not show up after prepare, if rename happens after table was copied.
            # Tables that are recreated (dropped, and created under the same name) during backup,after tablespace copy, break prepare due to different tablespace id.
            # Backup fails with "ALTER TABLE or OPTIMIZE TABLE was executed during backup"
            # Concurrent multi-rename is prone to race condition, for example table t1 can be missed from backup, if RENAME TABLE t1 to t2, t3 to t1 runs concurrently.


            A possible fix to some of those problems would be to
            * Tolerate missing files during innodb copy - missing files can happen when DROP or RENAME runs in parallel. Also allow MDL lock failures in --lock-ddl-per-table.
            * At the end of backup, under protection of FTWRL, during the stage when frm and MyISAM files are copied, rescan data directory, looking for Innodb tablespaces, and copy those that are not already in backup. This will pick up tables from CREATE during backup, or RENAMEd tables.
            * We might need to remove orphan (frm-less) .ibd files in prepare - there can be some left due to RENAME, or DROP.
            * Recreated (DROP/CREATE) tablespaces, that have changed tablespace ids at the end of backup. We might need to copy them to backup second time at the end of backup. This is be tricky for streaming backup, since xbstream format does not support multiple copies , or "delete" command. One way to workaround it, it to give second copy special extension, e.g "ibd.new" , so that "prepare" would know it can replace old copy with the new one.

            Maybe some of the above can be taken care by Innodb recovery, I'm not really sure how this would work. [~marko], any idea on that?
             "ALTER TABLE or OPTIMIZE TABLE was executed during backup" *must* be taken care by Innodb however.


            wlad Vladislav Vaintroub made changes -
            Description There is a lot of problems with mariabackup, if DDL statements are executed in parallel to backup.
            Some of them show up even if --lock-ddl-per-table is used.

            A fundamental problem with lock-ddl-per-table is that only those tables, that exist at the start of backup, are MDL-locked. Another problem is that mariabackup will deadlock during backup's "FLUSH TABLE WITH READ LOCKS", if concurrent DDL statements run. To resolve this deadlock, we either KILL user's DDL query, or user has the option to omit FTWRL with *unsafe* --no-lock option (MDEV-15636)

            So, there are problems which we currently have with backup in presence of Innodb DDL.

            # Tables created during backup are not there after prepare (with/without lock-ddl-per-table)
            # Tables dropped during backup might show up after prepare (without lock-ddl-per-table), or backup fails if file that was at the start of backup, is not found during copy.
            # If lock-ddl-per-table is used, acquiring MDL lock will fail, if the table is concurrently being dropped or renamed.
            # Tables that are renamed during backup, do not show up after prepare, if rename happens after table was copied.
            # Tables that are recreated (dropped, and created under the same name) during backup,after tablespace copy, break prepare due to different tablespace id.
            # Backup fails with "ALTER TABLE or OPTIMIZE TABLE was executed during backup"
            # Concurrent multi-rename is prone to race condition, for example table t1 can be missed from backup, if RENAME TABLE t1 to t2, t3 to t1 runs concurrently.


            A possible fix to some of those problems would be to
            * Tolerate missing files during innodb copy - missing files can happen when DROP or RENAME runs in parallel. Also allow MDL lock failures in --lock-ddl-per-table.
            * At the end of backup, under protection of FTWRL, during the stage when frm and MyISAM files are copied, rescan data directory, looking for Innodb tablespaces, and copy those that are not already in backup. This will pick up tables from CREATE during backup, or RENAMEd tables.
            * We might need to remove orphan (frm-less) .ibd files in prepare - there can be some left due to RENAME, or DROP.
            * Recreated (DROP/CREATE) tablespaces, that have changed tablespace ids at the end of backup. We might need to copy them to backup second time at the end of backup. This is be tricky for streaming backup, since xbstream format does not support multiple copies , or "delete" command. One way to workaround it, it to give second copy special extension, e.g "ibd.new" , so that "prepare" would know it can replace old copy with the new one.

            Maybe some of the above can be taken care by Innodb recovery, I'm not really sure how this would work. [~marko], any idea on that?
             "ALTER TABLE or OPTIMIZE TABLE was executed during backup" *must* be taken care by Innodb however.


            There is a lot of problems with mariabackup, if DDL statements are executed in parallel to backup.
            Some of them show up even if --lock-ddl-per-table is used.

            A fundamental problem with lock-ddl-per-table is that only those tables, that exist at the start of backup, are MDL-locked. Another problem is that mariabackup will deadlock during backup's "FLUSH TABLE WITH READ LOCKS", if concurrent DDL statements run. To resolve this deadlock, we either KILL user's DDL query, or user has the option to omit FTWRL with *unsafe* --no-lock option (MDEV-15636)

            So, there are problems which we currently have with backup in presence of Innodb DDL.

            # Tables created during backup are not there after prepare (with/without lock-ddl-per-table)
            # Tables dropped during backup might show up after prepare (without lock-ddl-per-table), or backup fails if file that was at the start of backup, is not found during copy.
            # If lock-ddl-per-table is used, acquiring MDL lock will fail, if the table is concurrently being dropped or renamed.
            # Tables that are renamed during backup, do not show up after prepare, if rename happens after table was copied.
            # Tables that are recreated (dropped, and created under the same name) during backup,after tablespace copy, break prepare due to different tablespace id.
            # Backup fails with "ALTER TABLE or OPTIMIZE TABLE was executed during backup"
            # Concurrent multi-rename is prone to race condition, for example table t1 can be missed from backup, if RENAME TABLE t1 to t2, t3 to t1 runs concurrently.


            A possible fix to some of those problems would be to
            * Tolerate missing files during innodb copy - missing files can happen when DROP or RENAME runs in parallel. Also allow MDL lock failures in --lock-ddl-per-table.
            * At the end of backup, under protection of FTWRL, during the stage when frm and MyISAM files are copied, rescan data directory, looking for Innodb tablespaces, and copy those that are not already in backup. This will pick up tables from CREATE during backup, or RENAMEd tables.
            * We might need to remove orphan (frm-less) .ibd files in prepare - there can be some left due to RENAME, or DROP.
            * Recreated (DROP/CREATE) tablespaces, that have changed tablespace ids at the end of backup. We might need to copy them to backup second time. This is be tricky for streaming backup, since xbstream format does not support multiple copies , or "delete" command. One way to workaround it, it to give second copy special extension, e.g "ibd.new" , so that "prepare" would know it can replace old copy with the new one.

            Maybe some of the above can be taken care by Innodb recovery, I'm not really sure how this would work. [~marko], any idea on that?
             "ALTER TABLE or OPTIMIZE TABLE was executed during backup" *must* be taken care by Innodb however.


            wlad Vladislav Vaintroub made changes -
            Summary mariabackup : allow DDL on Innodb tables, without --lock-ddl-per-table mariabackup : allow consistent backup, in presence of concurrent DDL, also without --lock-ddl-per-table
            wlad Vladislav Vaintroub made changes -
            Description There is a lot of problems with mariabackup, if DDL statements are executed in parallel to backup.
            Some of them show up even if --lock-ddl-per-table is used.

            A fundamental problem with lock-ddl-per-table is that only those tables, that exist at the start of backup, are MDL-locked. Another problem is that mariabackup will deadlock during backup's "FLUSH TABLE WITH READ LOCKS", if concurrent DDL statements run. To resolve this deadlock, we either KILL user's DDL query, or user has the option to omit FTWRL with *unsafe* --no-lock option (MDEV-15636)

            So, there are problems which we currently have with backup in presence of Innodb DDL.

            # Tables created during backup are not there after prepare (with/without lock-ddl-per-table)
            # Tables dropped during backup might show up after prepare (without lock-ddl-per-table), or backup fails if file that was at the start of backup, is not found during copy.
            # If lock-ddl-per-table is used, acquiring MDL lock will fail, if the table is concurrently being dropped or renamed.
            # Tables that are renamed during backup, do not show up after prepare, if rename happens after table was copied.
            # Tables that are recreated (dropped, and created under the same name) during backup,after tablespace copy, break prepare due to different tablespace id.
            # Backup fails with "ALTER TABLE or OPTIMIZE TABLE was executed during backup"
            # Concurrent multi-rename is prone to race condition, for example table t1 can be missed from backup, if RENAME TABLE t1 to t2, t3 to t1 runs concurrently.


            A possible fix to some of those problems would be to
            * Tolerate missing files during innodb copy - missing files can happen when DROP or RENAME runs in parallel. Also allow MDL lock failures in --lock-ddl-per-table.
            * At the end of backup, under protection of FTWRL, during the stage when frm and MyISAM files are copied, rescan data directory, looking for Innodb tablespaces, and copy those that are not already in backup. This will pick up tables from CREATE during backup, or RENAMEd tables.
            * We might need to remove orphan (frm-less) .ibd files in prepare - there can be some left due to RENAME, or DROP.
            * Recreated (DROP/CREATE) tablespaces, that have changed tablespace ids at the end of backup. We might need to copy them to backup second time. This is be tricky for streaming backup, since xbstream format does not support multiple copies , or "delete" command. One way to workaround it, it to give second copy special extension, e.g "ibd.new" , so that "prepare" would know it can replace old copy with the new one.

            Maybe some of the above can be taken care by Innodb recovery, I'm not really sure how this would work. [~marko], any idea on that?
             "ALTER TABLE or OPTIMIZE TABLE was executed during backup" *must* be taken care by Innodb however.


            There is a lot of problems with mariabackup, if DDL statements are executed in parallel to backup.
            Some of them show up even if --lock-ddl-per-table is used.

            A fundamental problem with lock-ddl-per-table is that only those tables, that exist at the start of backup, are MDL-locked. Another problem is that mariabackup will deadlock during backup's "FLUSH TABLE WITH READ LOCKS", if concurrent DDL statements run. To resolve this deadlock, we either KILL user's DDL query, or user has the option to omit FTWRL with *unsafe* --no-lock option (MDEV-15636)

            So, there are problems which we currently have with backup in presence of Innodb DDL.

            # Tables created during backup are not there after prepare (with/without lock-ddl-per-table)
            # Tables dropped during backup might show up after prepare (without lock-ddl-per-table), or backup fails if file that was at the start of backup, is not found during copy.
            # If lock-ddl-per-table is used, acquiring MDL lock will fail, if the table is concurrently being dropped or renamed.
            # Tables that are renamed during backup, do not show up after prepare, if rename happens after table was copied.
            # Tables that are recreated (dropped, and created under the same name) during backup,after tablespace copy, break prepare due to different tablespace id.
            # Backup fails with "ALTER TABLE or OPTIMIZE TABLE was executed during backup"
            # Concurrent multi-rename is prone to race condition, for example table t1 can be missed from backup, if RENAME TABLE t1 to t2, t3 to t1 runs concurrently.


            A possible fix to some of those problems would be to
            * Tolerate missing files during innodb copy - missing files can happen when DROP or RENAME runs in parallel. Also allow MDL lock failures in --lock-ddl-per-table.
            * At the end of backup, under protection of FTWRL, during the stage when frm and MyISAM files are copied, rescan data directory, looking for Innodb tablespaces, and copy those that are not already in backup. This will pick up tables from parallel CREATE or RENAME.
            * We might need to remove orphan (frm-less) .ibd files in prepare - there can be some left due to RENAME, or DROP.
            * Recreated (DROP/CREATE) tablespaces, that have changed tablespace ids at the end of backup. We might need to copy them to backup second time. This is be tricky for streaming backup, since xbstream format does not support multiple copies , or "delete" command. One way to workaround it, it to give second copy special extension, e.g "ibd.new" , so that "prepare" would know it can replace old copy with the new one.

            Maybe some of the above can be taken care by Innodb recovery, I'm not really sure how this would work. [~marko], any idea on that?
             "ALTER TABLE or OPTIMIZE TABLE was executed during backup" *must* be taken care by Innodb however.


            wlad Vladislav Vaintroub made changes -
            wlad Vladislav Vaintroub made changes -
            wlad Vladislav Vaintroub made changes -
            wlad Vladislav Vaintroub made changes -
            Description There is a lot of problems with mariabackup, if DDL statements are executed in parallel to backup.
            Some of them show up even if --lock-ddl-per-table is used.

            A fundamental problem with lock-ddl-per-table is that only those tables, that exist at the start of backup, are MDL-locked. Another problem is that mariabackup will deadlock during backup's "FLUSH TABLE WITH READ LOCKS", if concurrent DDL statements run. To resolve this deadlock, we either KILL user's DDL query, or user has the option to omit FTWRL with *unsafe* --no-lock option (MDEV-15636)

            So, there are problems which we currently have with backup in presence of Innodb DDL.

            # Tables created during backup are not there after prepare (with/without lock-ddl-per-table)
            # Tables dropped during backup might show up after prepare (without lock-ddl-per-table), or backup fails if file that was at the start of backup, is not found during copy.
            # If lock-ddl-per-table is used, acquiring MDL lock will fail, if the table is concurrently being dropped or renamed.
            # Tables that are renamed during backup, do not show up after prepare, if rename happens after table was copied.
            # Tables that are recreated (dropped, and created under the same name) during backup,after tablespace copy, break prepare due to different tablespace id.
            # Backup fails with "ALTER TABLE or OPTIMIZE TABLE was executed during backup"
            # Concurrent multi-rename is prone to race condition, for example table t1 can be missed from backup, if RENAME TABLE t1 to t2, t3 to t1 runs concurrently.


            A possible fix to some of those problems would be to
            * Tolerate missing files during innodb copy - missing files can happen when DROP or RENAME runs in parallel. Also allow MDL lock failures in --lock-ddl-per-table.
            * At the end of backup, under protection of FTWRL, during the stage when frm and MyISAM files are copied, rescan data directory, looking for Innodb tablespaces, and copy those that are not already in backup. This will pick up tables from parallel CREATE or RENAME.
            * We might need to remove orphan (frm-less) .ibd files in prepare - there can be some left due to RENAME, or DROP.
            * Recreated (DROP/CREATE) tablespaces, that have changed tablespace ids at the end of backup. We might need to copy them to backup second time. This is be tricky for streaming backup, since xbstream format does not support multiple copies , or "delete" command. One way to workaround it, it to give second copy special extension, e.g "ibd.new" , so that "prepare" would know it can replace old copy with the new one.

            Maybe some of the above can be taken care by Innodb recovery, I'm not really sure how this would work. [~marko], any idea on that?
             "ALTER TABLE or OPTIMIZE TABLE was executed during backup" *must* be taken care by Innodb however.


            There is a lot of problems with mariabackup, if DDL statements are executed in parallel to backup.
            Some of them show up even if --lock-ddl-per-table is used.

            A fundamental problem with lock-ddl-per-table is that only those tables, that exist at the start of backup, are MDL-locked. Another problem is that mariabackup will deadlock during backup's "FLUSH TABLE WITH READ LOCKS", if concurrent DDL statements run. To resolve this deadlock, we either KILL user's DDL query, or user has the option to omit FTWRL with *unsafe* --no-lock option (MDEV-15636)

            So, there are problems which we currently have with backup in presence of Innodb DDL.

            # Tables created during backup are not there after prepare (with/without lock-ddl-per-table)
            # Tables dropped during backup might show up after prepare (without lock-ddl-per-table), or backup fails if file that was at the start of backup, is not found during copy.
            # If lock-ddl-per-table is used, acquiring MDL lock will fail, if the table is concurrently being dropped or renamed.
            # Tables that are renamed during backup, do not show up after prepare, if rename happens after table was copied.
            # Tables that are recreated (dropped, and created under the same name) during backup,after tablespace copy, break prepare due to different tablespace id.
            # Backup fails with "ALTER TABLE or OPTIMIZE TABLE was executed during backup"
            # Concurrent multi-rename is prone to race condition, for example table t1 can be missed from backup, if RENAME TABLE t1 to t2, t3 to t1 runs concurrently.


            A possible fix to some of those problems would be to
            * Tolerate missing files during innodb copy - missing files can happen when DROP or RENAME runs in parallel. Also allow MDL lock failures in --lock-ddl-per-table.
            * At the end of backup, under protection of FTWRL, during the stage when frm and MyISAM files are copied, rescan data directory, looking for Innodb tablespaces, and copy those that are not already in backup. This will pick up tables from parallel CREATE or RENAME.
            * We might need to remove orphan (frm-less) .ibd files in prepare - there can be some left due to RENAME, or DROP.
            * Recreated (DROP/CREATE) tablespaces, that have changed tablespace ids at the end of backup. We might need to copy them to backup second time. This is be tricky for streaming backup, since xbstream format does not support multiple copies of the same file , and does not have any "delete" command. One way to workaround it, it to give second copy special extension, e.g "ibd.new" , so that "prepare" would know it can replace old copy with the new one.

            Maybe some of the above can be taken care by Innodb recovery, I'm not really sure how this would work. [~marko], any idea on that?
             "ALTER TABLE or OPTIMIZE TABLE was executed during backup" *must* be taken care by Innodb however.


            marko Marko Mäkelä made changes -
            marko Marko Mäkelä made changes -
            wlad Vladislav Vaintroub made changes -
            Fix Version/s 10.2.18 [ 23112 ]
            Fix Version/s 10.3.10 [ 23140 ]
            Fix Version/s 10.2 [ 14601 ]
            Fix Version/s 10.3 [ 22126 ]
            Resolution Fixed [ 1 ]
            Status Open [ 1 ] Closed [ 6 ]
            wlad Vladislav Vaintroub made changes -
            GeoffMontee Geoff Montee (Inactive) made changes -
            marko Marko Mäkelä made changes -
            marko Marko Mäkelä made changes -
            GeoffMontee Geoff Montee (Inactive) made changes -
            GeoffMontee Geoff Montee (Inactive) made changes -
            serg Sergei Golubchik made changes -
            Workflow MariaDB v3 [ 88504 ] MariaDB v4 [ 133632 ]
            mariadb-jira-automation Jira Automation (IT) made changes -
            Zendesk Related Tickets 198383

            People

              Unassigned Unassigned
              wlad Vladislav Vaintroub
              Votes:
              3 Vote for this issue
              Watchers:
              10 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Git Integration

                  Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.