Uploaded image for project: 'MariaDB Server'
  1. MariaDB Server
  2. MDEV-23514

SEGV storage/innobase/row/row0log.cc:863 in row_log_table_low

Details

    Description

      commit 309302a3dad5f06cb62b0846dcb8a3671d91ff29 (HEAD, origin/10.2)
      2020-08-19 16:36:05 122901187513344 [Note] /home/mleich/Server_bin/10.2_asan/bin/mysqld: ready for connections.
      Version: '10.2.34-MariaDB-debug-log'  socket: '/home/mleich/Server_bin/10.2_asan/mysql-test/var/tmp/4/mysqld.1.sock'  port: 16060  Source distribution
      ASAN:DEADLYSIGNAL
      =================================================================
      ==12019==ERROR: AddressSanitizer: SEGV on unknown address 0x000000000080 (pc 0x5644ffd7498d bp 0x640001607070 sp 0x640001606f40 T29)
      ==12019==The signal is caused by a READ memory access.
      ==12019==Hint: address points to the zero page.
          #0 0x5644ffd7498c in row_log_table_low /home/mleich/Server/10.2/storage/innobase/row/row0log.cc:863
          #1 0x5644ffd75840 in row_log_table_update(unsigned char const*, dict_index_t*, unsigned short const*, dtuple_t const*) /home/mleich/Server/10.2/storage/innobase/row/row0log.cc:967
          #2 0x56450028d7f1 in row_undo_mod_clust /home/mleich/Server/10.2/storage/innobase/row/row0umod.cc:339
          #3 0x564500293248 in row_undo_mod(undo_node_t*, que_thr_t*) /home/mleich/Server/10.2/storage/innobase/row/row0umod.cc:1244
          #4 0x5644ffe0999e in row_undo /home/mleich/Server/10.2/storage/innobase/row/row0undo.cc:291
          #5 0x5644ffe09cfd in row_undo_step(que_thr_t*) /home/mleich/Server/10.2/storage/innobase/row/row0undo.cc:334
          #6 0x5644ffc8f8d4 in que_thr_step /home/mleich/Server/10.2/storage/innobase/que/que0que.cc:1040
          #7 0x5644ffc8fc9f in que_run_threads_low /home/mleich/Server/10.2/storage/innobase/que/que0que.cc:1104
          #8 0x5644ffc90050 in que_run_threads(que_thr_t*) /home/mleich/Server/10.2/storage/innobase/que/que0que.cc:1144
          #9 0x5644ffeebf37 in trx_rollback_to_savepoint_low /home/mleich/Server/10.2/storage/innobase/trx/trx0roll.cc:107
          #10 0x5644ffeec586 in trx_rollback_for_mysql_low /home/mleich/Server/10.2/storage/innobase/trx/trx0roll.cc:169
          #11 0x5644ffeeccfe in trx_rollback_for_mysql(trx_t*) /home/mleich/Server/10.2/storage/innobase/trx/trx0roll.cc:200
          #12 0x5644ffa3a183 in innobase_rollback /home/mleich/Server/10.2/storage/innobase/handler/ha_innodb.cc:4764
          #13 0x5644ff4fa06d in ha_rollback_trans(THD*, bool) /home/mleich/Server/10.2/sql/handler.cc:1707
          #14 0x5644ff263cdd in trans_rollback(THD*) /home/mleich/Server/10.2/sql/transaction.cc:415
          #15 0x5644fee9fc21 in mysql_execute_command(THD*) /home/mleich/Server/10.2/sql/sql_parse.cc:5350
          #16 0x5644feeb02b2 in mysql_parse(THD*, char*, unsigned int, Parser_state*, bool, bool) /home/mleich/Server/10.2/sql/sql_parse.cc:7733
          #17 0x5644fee86f5d in dispatch_command(enum_server_command, THD*, char*, unsigned int, bool, bool) /home/mleich/Server/10.2/sql/sql_parse.cc:1823
          #18 0x5644fee83984 in do_command(THD*) /home/mleich/Server/10.2/sql/sql_parse.cc:1377
          #19 0x5644ff22cc43 in do_handle_one_connection(CONNECT*) /home/mleich/Server/10.2/sql/sql_connect.cc:1336
          #20 0x5644ff22c500 in handle_one_connection /home/mleich/Server/10.2/sql/sql_connect.cc:1241
          #21 0x5645006b5845 in pfs_spawn_thread /home/mleich/Server/10.2/storage/perfschema/pfs.cc:1869
          #22 0x39490f1cd7fb in start_thread (/lib/x86_64-linux-gnu/libpthread.so.0+0x77fb)
          #23 0x73ed0113ab5e in clone (/lib/x86_64-linux-gnu/libc.so.6+0x114b5e)
       
      AddressSanitizer can not provide additional info.
      SUMMARY: AddressSanitizer: SEGV /home/mleich/Server/10.2/storage/innobase/row/row0log.cc:863 in row_log_table_low
      Thread T29 created by T0 here:
          #0 0x5645028e0d2f in __interceptor_pthread_create (/usr/lib/x86_64-linux-gnu/libasan.so.4+0x37d2f)
          #1 0x5645006b5c7f in spawn_thread_v1 /home/mleich/Server/10.2/storage/perfschema/pfs.cc:1919
          #2 0x5644fec19130 in inline_mysql_thread_create /home/mleich/Server/10.2/include/mysql/psi/mysql_thread.h:1246
          #3 0x5644fec3101d in create_thread_to_handle_connection(CONNECT*) /home/mleich/Server/10.2/sql/mysqld.cc:6520
          #4 0x5644fec317ee in create_new_thread /home/mleich/Server/10.2/sql/mysqld.cc:6588
          #5 0x5644fec329a0 in handle_connections_sockets() /home/mleich/Server/10.2/sql/mysqld.cc:6846
          #6 0x5644fec303c9 in mysqld_main(int, char**) /home/mleich/Server/10.2/sql/mysqld.cc:6137
          #7 0x5644fec17a79 in main /home/mleich/Server/10.2/sql/main.cc:25
          #8 0x73ed010471c0 in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x211c0)
       
      ==12019==ABORTING
      200819 16:43:54 [ERROR] mysqld got signal 6 ;
      ...
      Query (0x62b00003f228): ROLLBACK
      Connection ID (thread ID): 11
      Status: NOT_KILLED
      

      Attachments

        Issue Links

          Activity

            mleich Matthias Leich added a comment - - edited

            How to replay
            1. cd <Top directory with MariaDB binaries>/mysql-test
            2. tar xvzf MDEV-23514.tgz
                This unpacks
                - shellscript mysqltest_background.sh
               - test               suite/innodb/t/ml_TBR-631.test
            3. ./mtr --mem --parallel=4 -rr -rr-arg='--chaos' --mysqld=--innodb_page_size=16K ml_TBR-631{,,,}
                 or similar
            4. After observing the test failure you might need to wait lets say 10 seconds and than run
                      killall -9 perl mysqld mariadbd mysqltest
                 because otherwise certain processes will make useless load for several minutes.
                 
            Some notes:
            The test and the shape of the MTR call are a brute force attempt to replay the problem.
            I assume that neither "rr" nor ASAN build nor parallel > 1 nor the flood of SQL inside of ml_TBR-631 are strict required.
            But its quite probable that their use made replays of the problem more likely.
             
            The grammar of the corresponding RQG test is
            __clone__3:
                COMMIT |
                ROLLBACK ;
             
            thread1:
                ALTER TABLE t2 ADD PRIMARY KEY ( col_text(9) , col2 ) |
                ALTER TABLE t2 DROP PRIMARY KEY ;
             
            thread1_connect:
                ;
             
            thread1_init:
                CREATE TABLE t2 ( col1 INT, col2 INT, col_int INTEGER , col_string INTEGER, col_text TEXT ) ENGINE = InnoDB ROW_FORMAT = Compressed ;
             
            thread2:
                UPDATE t2 SET col_int = { $my_int= $prng->int( 65, 512) } LIMIT 2 |
                INSERT INTO t2 (col2,col_int,col_string, col_text) VALUES ( { $my_int= $prng->int( 65, 512) } , $my_int, 1111111111, REPEAT(SUBSTR(CAST( $my_int AS CHAR),1,1), @fill_amount) ); __clone__3 ;
             
            thread2_connect:
                SET AUTOCOMMIT = 0; SET @fill_amount = (@@innodb_page_size / 2 ) + 1 ;
             
            thread2_init:
                ;
            

            mleich Matthias Leich added a comment - - edited How to replay 1. cd <Top directory with MariaDB binaries>/mysql-test 2. tar xvzf MDEV-23514.tgz This unpacks - shellscript mysqltest_background.sh - test suite/innodb/t/ml_TBR-631.test 3. ./mtr --mem --parallel=4 -rr -rr-arg='--chaos' --mysqld=--innodb_page_size=16K ml_TBR-631{,,,} or similar 4. After observing the test failure you might need to wait lets say 10 seconds and than run killall -9 perl mysqld mariadbd mysqltest because otherwise certain processes will make useless load for several minutes. Some notes: The test and the shape of the MTR call are a brute force attempt to replay the problem. I assume that neither "rr" nor ASAN build nor parallel > 1 nor the flood of SQL inside of ml_TBR-631 are strict required. But its quite probable that their use made replays of the problem more likely.   The grammar of the corresponding RQG test is __clone__3: COMMIT | ROLLBACK ;   thread1: ALTER TABLE t2 ADD PRIMARY KEY ( col_text(9) , col2 ) | ALTER TABLE t2 DROP PRIMARY KEY ;   thread1_connect: ;   thread1_init: CREATE TABLE t2 ( col1 INT, col2 INT, col_int INTEGER , col_string INTEGER, col_text TEXT ) ENGINE = InnoDB ROW_FORMAT = Compressed ;   thread2: UPDATE t2 SET col_int = { $my_int= $prng->int( 65, 512) } LIMIT 2 | INSERT INTO t2 (col2,col_int,col_string, col_text) VALUES ( { $my_int= $prng->int( 65, 512) } , $my_int, 1111111111, REPEAT(SUBSTR(CAST( $my_int AS CHAR),1,1), @fill_amount) ); __clone__3 ;   thread2_connect: SET AUTOCOMMIT = 0; SET @fill_amount = (@@innodb_page_size / 2 ) + 1 ;   thread2_init: ;

            This is a race condition between a DML operation and a rollback of the ALTER TABLE. The log was freed here:

            10.2 bfba2bce6a350e2893c76e42415ac733c39a3976

            Thread 30 hit Hardware watchpoint 1: -location $1->online_log
            Old value = (row_log_t *) 0x6e957008ac00
            New value = (row_log_t *) 0x0
            row_log_free (log=@0x6e95700215b8: 0x0)
                at /mariadb/10.2o/storage/innobase/row/row0log.cc:3050
            3050	}
            (rr) bt
            #0  row_log_free (log=@0x6e95700215b8: 0)
                at /mariadb/10.2o/storage/innobase/row/row0log.cc:3049
            #1  0x000055d70ca0a472 in innobase_online_rebuild_log_free (
                table=<optimized out>)
                at /mariadb/10.2o/storage/innobase/handler/handler0alter.cc:6391
            #2  0x000055d70ca0d0e2 in rollback_inplace_alter_table (
                ha_alter_info=0x79110e27f730, table=0x6e95700394c8, 
                prebuilt=0x6e95700c7e60)
                at /mariadb/10.2o/storage/innobase/handler/handler0alter.cc:6524
            #3  0x000055d70ca06c14 in ha_innobase::commit_inplace_alter_table (this=
                0x6e9570049730, altered_table=0x6e95700c8e78, 
                ha_alter_info=0x79110e27f730, commit=false)
            …
            (rr) thread 3
            (rr) bt
            …
            #19 mtr_t::s_lock (this=0x3f9a30b397c0, lock=0x6e95700216a8, 
                file=0x55d70d07c3b0 "/mariadb/10.2o/storage/innobase/row/row0umod.cc", 
                line=284) at /mariadb/10.2o/storage/innobase/include/mtr0mtr.ic:237
            #20 0x000055d70cd17765 in row_undo_mod_clust (node=0x6e956c03e850, 
                thr=0x6e956c039e18) at /mariadb/10.2o/storage/innobase/row/row0umod.cc:284
            …
            #33 0x000055d70c646e04 in mysql_parse (thd=0x6e956c000d28, rawbuf=0x6e956c0107e0 "ROLLBACK", length=<optimized out>, parser_state=0x3f9a30b3b6d0, is_com_multi=<optimized out>, is_next_command=false) at /mariadb/10.2o/sql/sql_parse.cc:7733
            

            At this point of time, the dict_operation_lock is held by Thread 30 (which is rolling back the online ALTER TABLE). Before MDEV-23484, any concurrent ROLLBACK on any table was blocked by that.

            In row_log_table_low(), we have the following debug assertion:

            	ut_ad(rw_lock_own_flagged(
            			&index->lock,
            			RW_LOCK_FLAG_S | RW_LOCK_FLAG_X | RW_LOCK_FLAG_SX));
            

            In innobase_online_rebuild_log_free() we are acquiring that latch in exclusive mode. This proves that the field should be protected by dict_index_t::lock.

            The function row_undo_ins_remove_clust_rec() is correctly re-checking the attribute while holding dict_index_t::lock, but row_undo_mod_clust() was missing that. It is also duplicating the above mentioned debug assertion. I believe that the following should adequately fix this bug:

            diff --git a/storage/innobase/row/row0umod.cc b/storage/innobase/row/row0umod.cc
            index b1cc994cdd8..0de0760834e 100644
            --- a/storage/innobase/row/row0umod.cc
            +++ b/storage/innobase/row/row0umod.cc
            @@ -319,17 +319,7 @@ row_undo_mod_clust(
             		ut_ad(err == DB_SUCCESS || err == DB_OUT_OF_FILE_SPACE);
             	}
             
            -	/* Online rebuild cannot be initiated while we are holding
            -	dict_operation_lock and index->lock. (It can be aborted.) */
            -	ut_ad(online || !dict_index_is_online_ddl(index));
            -
            -	if (err == DB_SUCCESS && online) {
            -
            -		ut_ad(rw_lock_own_flagged(
            -				&index->lock,
            -				RW_LOCK_FLAG_S | RW_LOCK_FLAG_X
            -				| RW_LOCK_FLAG_SX));
            -
            +	if (err == DB_SUCCESS && online && dict_index_is_online_ddl(index)) {
             		switch (node->rec_type) {
             		case TRX_UNDO_DEL_MARK_REC:
             			row_log_table_insert(
            

            Online creation of secondary indexes should be unaffected by this.

            marko Marko Mäkelä added a comment - This is a race condition between a DML operation and a rollback of the ALTER TABLE . The log was freed here: 10.2 bfba2bce6a350e2893c76e42415ac733c39a3976 Thread 30 hit Hardware watchpoint 1: -location $1->online_log Old value = (row_log_t *) 0x6e957008ac00 New value = (row_log_t *) 0x0 row_log_free (log=@0x6e95700215b8: 0x0) at /mariadb/10.2o/storage/innobase/row/row0log.cc:3050 3050 } (rr) bt #0 row_log_free (log=@0x6e95700215b8: 0) at /mariadb/10.2o/storage/innobase/row/row0log.cc:3049 #1 0x000055d70ca0a472 in innobase_online_rebuild_log_free ( table=<optimized out>) at /mariadb/10.2o/storage/innobase/handler/handler0alter.cc:6391 #2 0x000055d70ca0d0e2 in rollback_inplace_alter_table ( ha_alter_info=0x79110e27f730, table=0x6e95700394c8, prebuilt=0x6e95700c7e60) at /mariadb/10.2o/storage/innobase/handler/handler0alter.cc:6524 #3 0x000055d70ca06c14 in ha_innobase::commit_inplace_alter_table (this= 0x6e9570049730, altered_table=0x6e95700c8e78, ha_alter_info=0x79110e27f730, commit=false) … (rr) thread 3 (rr) bt … #19 mtr_t::s_lock (this=0x3f9a30b397c0, lock=0x6e95700216a8, file=0x55d70d07c3b0 "/mariadb/10.2o/storage/innobase/row/row0umod.cc", line=284) at /mariadb/10.2o/storage/innobase/include/mtr0mtr.ic:237 #20 0x000055d70cd17765 in row_undo_mod_clust (node=0x6e956c03e850, thr=0x6e956c039e18) at /mariadb/10.2o/storage/innobase/row/row0umod.cc:284 … #33 0x000055d70c646e04 in mysql_parse (thd=0x6e956c000d28, rawbuf=0x6e956c0107e0 "ROLLBACK", length=<optimized out>, parser_state=0x3f9a30b3b6d0, is_com_multi=<optimized out>, is_next_command=false) at /mariadb/10.2o/sql/sql_parse.cc:7733 At this point of time, the dict_operation_lock is held by Thread 30 (which is rolling back the online ALTER TABLE ). Before MDEV-23484 , any concurrent ROLLBACK on any table was blocked by that. In row_log_table_low() , we have the following debug assertion: ut_ad(rw_lock_own_flagged( &index->lock, RW_LOCK_FLAG_S | RW_LOCK_FLAG_X | RW_LOCK_FLAG_SX)); In innobase_online_rebuild_log_free() we are acquiring that latch in exclusive mode. This proves that the field should be protected by dict_index_t::lock . The function row_undo_ins_remove_clust_rec() is correctly re-checking the attribute while holding dict_index_t::lock , but row_undo_mod_clust() was missing that. It is also duplicating the above mentioned debug assertion. I believe that the following should adequately fix this bug: diff --git a/storage/innobase/row/row0umod.cc b/storage/innobase/row/row0umod.cc index b1cc994cdd8..0de0760834e 100644 --- a/storage/innobase/row/row0umod.cc +++ b/storage/innobase/row/row0umod.cc @@ -319,17 +319,7 @@ row_undo_mod_clust( ut_ad(err == DB_SUCCESS || err == DB_OUT_OF_FILE_SPACE); } - /* Online rebuild cannot be initiated while we are holding - dict_operation_lock and index->lock. (It can be aborted.) */ - ut_ad(online || !dict_index_is_online_ddl(index)); - - if (err == DB_SUCCESS && online) { - - ut_ad(rw_lock_own_flagged( - &index->lock, - RW_LOCK_FLAG_S | RW_LOCK_FLAG_X - | RW_LOCK_FLAG_SX)); - + if (err == DB_SUCCESS && online && dict_index_is_online_ddl(index)) { switch (node->rec_type) { case TRX_UNDO_DEL_MARK_REC: row_log_table_insert( Online creation of secondary indexes should be unaffected by this.

            I think that secondary index creation is affected by this in a way that is much harder to reproduce.

            While row_merge_drop_indexes() would not invoke dict_index_remove_from_cache() on normal indexes as long as transactions exist that hold locks on the table, it would invoke that on FULLTEXT INDEX objects. Luckily, fulltext index creation is not allowed ONLINE (LOCK=NONE), and the ALTER TABLE operation would be unable to start in prepare_inplace_alter_table_dict() as long as any transactions (recovered or not) hold locks on the table:

            	if (ctx->online) {
            		error = DB_SUCCESS;
            	} else {
            		error = row_merge_lock_table(
            			ctx->prebuilt->trx, ctx->new_table, LOCK_S);
             
            		if (error != DB_SUCCESS) {
             
            			goto error_handling;
            		}
            	}
            

            Because we are skipping the table lock acquisition for online operations, we seem to have a race condition between the rollback of recovered transactions and the online creation of a secondary index. The dict_table_t::indexes could be modified while the rollback is executing. I think that the easiest way to block this race condition is to ensure that both trx_rollback_active() and innobase_rollback_by_xid() will hold dict_operation_lock at least in shared mode when processing each undo log record of a recovered transaction. To incur a minimum overhead when we are rolling back large transactions, we’d better acquire and release the latch at the low level, where we used to do it for all transactions before MDEV-23484 removed the code:

            diff --git a/storage/innobase/row/row0undo.cc b/storage/innobase/row/row0undo.cc
            index 34b55f25527..185a71e670b 100644
            --- a/storage/innobase/row/row0undo.cc
            +++ b/storage/innobase/row/row0undo.cc
            @@ -279,6 +279,19 @@ row_undo(
             			? UNDO_NODE_INSERT : UNDO_NODE_MODIFY;
             	}
             
            +	/* Prevent prepare_inplace_alter_table_dict() from adding
            +	dict_table_t::indexes while we are processing the record.
            +	Recovered transactions are not protected by MDL, and the
            +	secondary index creation is not protected by table locks
            +	for online operation. (A table lock would only be acquired
            +	when committing the ALTER TABLE operation.) */
            +	const bool locked_data_dict = UNIV_UNLIKELY(trx->is_recovered)
            +		&& !trx->dict_operation_lock_mode;
            +
            +	if (UNIV_UNLIKELY(locked_data_dict)) {
            +		row_mysql_freeze_data_dictionary(trx);
            +	}
            +
             	dberr_t err;
             
             	if (node->state == UNDO_NODE_INSERT) {
            @@ -291,6 +304,10 @@ row_undo(
             		err = row_undo_mod(node, thr);
             	}
             
            +	if (UNIV_UNLIKELY(locked_data_dict)) {
            +		row_mysql_unfreeze_data_dictionary(trx);
            +	}
            +
             	/* Do some cleanup */
             	btr_pcur_close(&(node->pcur));
             
            

            It would be challenging to write a reasonably deterministic test case for this, because DEBUG_SYNC cannot be directly activated for background threads. For the failure that is revealed by MDEV-23514.tgz it should be technically possible to write a test that would invoke a new DEBUG_SYNC point in row_undo_mod_clust() right before the code fix, but I do not think that it is worth the effort.

            marko Marko Mäkelä added a comment - I think that secondary index creation is affected by this in a way that is much harder to reproduce. While row_merge_drop_indexes() would not invoke dict_index_remove_from_cache() on normal indexes as long as transactions exist that hold locks on the table, it would invoke that on FULLTEXT INDEX objects. Luckily, fulltext index creation is not allowed ONLINE ( LOCK=NONE ), and the ALTER TABLE operation would be unable to start in prepare_inplace_alter_table_dict() as long as any transactions (recovered or not) hold locks on the table: if (ctx->online) { error = DB_SUCCESS; } else { error = row_merge_lock_table( ctx->prebuilt->trx, ctx->new_table, LOCK_S);   if (error != DB_SUCCESS) {   goto error_handling; } } Because we are skipping the table lock acquisition for online operations, we seem to have a race condition between the rollback of recovered transactions and the online creation of a secondary index. The dict_table_t::indexes could be modified while the rollback is executing. I think that the easiest way to block this race condition is to ensure that both trx_rollback_active() and innobase_rollback_by_xid() will hold dict_operation_lock at least in shared mode when processing each undo log record of a recovered transaction. To incur a minimum overhead when we are rolling back large transactions, we’d better acquire and release the latch at the low level, where we used to do it for all transactions before MDEV-23484 removed the code: diff --git a/storage/innobase/row/row0undo.cc b/storage/innobase/row/row0undo.cc index 34b55f25527..185a71e670b 100644 --- a/storage/innobase/row/row0undo.cc +++ b/storage/innobase/row/row0undo.cc @@ -279,6 +279,19 @@ row_undo( ? UNDO_NODE_INSERT : UNDO_NODE_MODIFY; } + /* Prevent prepare_inplace_alter_table_dict() from adding + dict_table_t::indexes while we are processing the record. + Recovered transactions are not protected by MDL, and the + secondary index creation is not protected by table locks + for online operation. (A table lock would only be acquired + when committing the ALTER TABLE operation.) */ + const bool locked_data_dict = UNIV_UNLIKELY(trx->is_recovered) + && !trx->dict_operation_lock_mode; + + if (UNIV_UNLIKELY(locked_data_dict)) { + row_mysql_freeze_data_dictionary(trx); + } + dberr_t err; if (node->state == UNDO_NODE_INSERT) { @@ -291,6 +304,10 @@ row_undo( err = row_undo_mod(node, thr); } + if (UNIV_UNLIKELY(locked_data_dict)) { + row_mysql_unfreeze_data_dictionary(trx); + } + /* Do some cleanup */ btr_pcur_close(&(node->pcur)); It would be challenging to write a reasonably deterministic test case for this, because DEBUG_SYNC cannot be directly activated for background threads. For the failure that is revealed by MDEV-23514.tgz it should be technically possible to write a test that would invoke a new DEBUG_SYNC point in row_undo_mod_clust() right before the code fix, but I do not think that it is worth the effort.

            This regression was never merged to other branches and never part of any release; it was only present in 10.2.

            marko Marko Mäkelä added a comment - This regression was never merged to other branches and never part of any release; it was only present in 10.2.

            People

              marko Marko Mäkelä
              mleich Matthias Leich
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Git Integration

                  Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.