Uploaded image for project: 'MariaDB Server'
  1. MariaDB Server
  2. MDEV-29684

Testing and improving 'cluster conflict resolving' with 10.4 and later

Details

    Description

      There are a number of bug reports of cluster wide conflict resolving related crashes or hangs. To troubleshoot these, it would be good to run generic cluster conflict tests, just for easier reproducing of the issue(s).

      Run high intensity write-write conflict load in the cluster, with varying SQL access patterns.
      Also, some mtr tests exercising high priority victim aborting, appear to be disabled in 10.4. Enable these and fix the test if it appears to be non-deterministic.

      Analyze and troubleshoot issues surfacing from the testing.

      Attachments

        1. mysql.err
          1.02 MB
        2. oltp.zz
          2 kB
        3. oltp_and_ddl_v1.yy
          4 kB
        4. bt_all.txt
          115 kB
        5. new.err
          1.21 MB
        6. all_bt_28102022.txt
          116 kB
        7. bt_all_v2.txt
          117 kB
        8. oltp_and_ddl_v2.yy
          3 kB

        Issue Links

          Activity

            seppo Seppo Jaakola created issue -
            jplindst Jan Lindström (Inactive) made changes -
            Field Original Value New Value
            Priority Major [ 3 ] Critical [ 2 ]
            jplindst Jan Lindström (Inactive) made changes -
            Assignee Seppo Jaakola [ seppo ] Jan Lindström [ jplindst ]
            jplindst Jan Lindström (Inactive) made changes -
            Status Open [ 1 ] In Progress [ 3 ]
            jplindst Jan Lindström (Inactive) made changes -
            Due Date 2022-10-10 2022-10-28
            jplindst Jan Lindström (Inactive) made changes -
            Status In Progress [ 3 ] In Testing [ 10301 ]
            jplindst Jan Lindström (Inactive) made changes -
            Assignee Jan Lindström [ jplindst ] Ramesh Sivaraman [ JIRAUSER48189 ]

            ramesh branch : bb-10.4-MDEV-29684-galera

            This requires multi-master testing so that you have KILL [QUERY|CONNECTION] clauses there.

            jplindst Jan Lindström (Inactive) added a comment - ramesh branch : bb-10.4- MDEV-29684 -galera This requires multi-master testing so that you have KILL [QUERY|CONNECTION] clauses there.
            Roel Roel Van de Paar made changes -
            Summary Testing and improving cluster conflcit resolving with 10.4 and later Testing and improving 'cluster conflict resolving' with 10.4 and later
            ramesh Ramesh Sivaraman made changes -
            Attachment mysql.err [ 66031 ]
            ramesh Ramesh Sivaraman made changes -
            Attachment oltp_and_ddl_v1.yy [ 66032 ]
            Attachment oltp.zz [ 66033 ]
            ramesh Ramesh Sivaraman added a comment - - edited

            seppo jplindst RQG run shows server hang issue using KILL connection/query grammar. Seems the issue is related to MDL. Please find the attached RQG grammar, error log and stack trace. mysql.err oltp_and_ddl_v1.yy oltp.zz bt_all.txt

            MariaDB [(none)]> show processlist;
            +----+-------------+-----------------+------+---------+------+---------------------------------+---------------------------------------------------------------------------------------+----------+
            | Id | User        | Host            | db   | Command | Time | State                           | Info                                                                                  | Progress |
            +----+-------------+-----------------+------+---------+------+---------------------------------+---------------------------------------------------------------------------------------+----------+
            |  1 | system user |                 | NULL | Sleep   | 1488 | wsrep aborter idle              | NULL                                                                                  |    0.000 |
            |  2 | system user |                 | NULL | Sleep   | 1488 | Closing tables                  | NULL                                                                                  |    0.000 |
            |  3 | system user |                 | NULL | Daemon  | NULL | InnoDB purge worker             | NULL                                                                                  |    0.000 |
            |  4 | system user |                 | NULL | Daemon  | NULL | InnoDB purge coordinator        | NULL                                                                                  |    0.000 |
            |  5 | system user |                 | NULL | Daemon  | NULL | InnoDB purge worker             | NULL                                                                                  |    0.000 |
            |  6 | system user |                 | NULL | Daemon  | NULL | InnoDB purge worker             | NULL                                                                                  |    0.000 |
            |  7 | system user |                 | NULL | Daemon  | NULL | InnoDB shutdown handler         | NULL                                                                                  |    0.000 |
            | 11 | system user |                 | NULL | Sleep   | 1487 | Closing tables                  | NULL                                                                                  |    0.000 |
            | 12 | system user |                 | NULL | Sleep   | 1487 | wsrep applier idle              | NULL                                                                                  |    0.000 |
            | 14 | system user |                 | NULL | Sleep   | 1487 | wsrep applier idle              | NULL                                                                                  |    0.000 |
            | 22 | ramesh      | localhost:43962 | test | Query   | 1445 | acquiring total order isolation | KILL CONNECTION @conid /* QNO 981 CON_ID 22 */                                        |    0.000 |
            | 23 | ramesh      | localhost:43972 | test | Query   | 1445 | Commit                          | INSERT INTO `oltp5` ( `id` ) VALUES ( 35403 )  /* QNO 924 CON_ID 23 */                |    0.000 |
            | 24 | ramesh      | localhost:43974 | test | Query   | 1446 | acquiring total order isolation | TRUNCATE `oltp6` /* QNO 940 CON_ID 24 */                                              |    0.000 |
            | 27 | ramesh      | localhost:53284 | test | Query   | 1445 | acquiring total order isolation | KILL QUERY @conid /* QNO 923 CON_ID 27 */                                             |    0.000 |
            | 29 | ramesh      | localhost:53304 | test | Killed  | 1445 | acquiring total order isolation | KILL QUERY @conid /* QNO 956 CON_ID 29 */                                             |    0.000 |
            | 30 | ramesh      | localhost:53318 | test | Query   | 1446 | Commit                          | INSERT INTO `oltp4` ( `id`, `k`) VALUES ( NULL, 565051392 ) /* QNO 913 CON_ID 30 */   |    0.000 |
            | 32 | ramesh      | localhost:53334 | test | Query   | 1415 | acquiring total order isolation | ALTER ONLINE TABLE `oltp1` PARTITION BY KEY(`id`) /* QNO 926 CON_ID 32 */             |    0.000 |
            | 33 | ramesh      | localhost:53346 | test | Query   | 1205 | acquiring total order isolation | TRUNCATE `oltp4` /* QNO 923 CON_ID 33 */                                              |    0.000 |
            | 34 | ramesh      | localhost:53360 | test | Query   | 1446 | Waiting for table metadata lock | ALTER ONLINE TABLE `oltp6` ROW_FORMAT=COMPACT /* QNO 905 CON_ID 34 */                 |    0.000 |
            | 35 | ramesh      | localhost:53374 | test | Query   | 1295 | acquiring total order isolation | OPTIMIZE TABLE `oltp3` /* QNO 918 CON_ID 35 */                                        |    0.000 |
            | 36 | ramesh      | localhost:53376 | test | Query   | 1445 | Freeing items                   | INSERT INTO `oltp6` ( `id`, `k`) VALUES ( NULL, 1681063936 )  /* QNO 905 CON_ID 36 */ |    0.000 |
            | 38 | ramesh      | localhost:53392 | test | Query   | 1445 | Init                            | KILL CONNECTION @conid /* QNO 908 CON_ID 31 */                                        |    0.000 |
            | 40 | ramesh      | localhost:53406 | test | Query   | 1445 | Init                            | KILL CONNECTION @conid /* QNO 956 CON_ID 21 */                                        |    0.000 |
            | 41 | ramesh      | localhost:37020 | test | Query   | 1445 | Init                            | COMMIT /* QNO 943 CON_ID 25 */                                                        |    0.000 |
            | 42 | ramesh      | localhost:37022 | test | Query   | 1446 | Commit                          | INSERT INTO `oltp4` ( `id`, `pad` ) VALUES ( NULL, 'm' )  /* QNO 921 CON_ID 26 */     |    0.000 |
            | 43 | ramesh      | localhost:37036 | test | Query   |  995 | Init                            | KILL CONNECTION @conid /* QNO 930 CON_ID 28 */                                        |    0.000 |
            | 44 | root        | localhost       | NULL | Query   |    0 | Init                            | show processlist                                                                      |    0.000 |
            | 45 | root        | localhost       | NULL | Query   | 1319 | Checking permissions            | create database test_one                                                              |    0.000 |
            +----+-------------+-----------------+------+---------+------+---------------------------------+---------------------------------------------------------------------------------------+----------+
            28 rows in set (0.001 sec)
             
            MariaDB [(none)]> 
            

            ramesh Ramesh Sivaraman added a comment - - edited seppo jplindst RQG run shows server hang issue using KILL connection/query grammar. Seems the issue is related to MDL. Please find the attached RQG grammar, error log and stack trace. mysql.err oltp_and_ddl_v1.yy oltp.zz bt_all.txt MariaDB [(none)]> show processlist; +----+-------------+-----------------+------+---------+------+---------------------------------+---------------------------------------------------------------------------------------+----------+ | Id | User | Host | db | Command | Time | State | Info | Progress | +----+-------------+-----------------+------+---------+------+---------------------------------+---------------------------------------------------------------------------------------+----------+ | 1 | system user | | NULL | Sleep | 1488 | wsrep aborter idle | NULL | 0.000 | | 2 | system user | | NULL | Sleep | 1488 | Closing tables | NULL | 0.000 | | 3 | system user | | NULL | Daemon | NULL | InnoDB purge worker | NULL | 0.000 | | 4 | system user | | NULL | Daemon | NULL | InnoDB purge coordinator | NULL | 0.000 | | 5 | system user | | NULL | Daemon | NULL | InnoDB purge worker | NULL | 0.000 | | 6 | system user | | NULL | Daemon | NULL | InnoDB purge worker | NULL | 0.000 | | 7 | system user | | NULL | Daemon | NULL | InnoDB shutdown handler | NULL | 0.000 | | 11 | system user | | NULL | Sleep | 1487 | Closing tables | NULL | 0.000 | | 12 | system user | | NULL | Sleep | 1487 | wsrep applier idle | NULL | 0.000 | | 14 | system user | | NULL | Sleep | 1487 | wsrep applier idle | NULL | 0.000 | | 22 | ramesh | localhost:43962 | test | Query | 1445 | acquiring total order isolation | KILL CONNECTION @conid /* QNO 981 CON_ID 22 */ | 0.000 | | 23 | ramesh | localhost:43972 | test | Query | 1445 | Commit | INSERT INTO `oltp5` ( `id` ) VALUES ( 35403 ) /* QNO 924 CON_ID 23 */ | 0.000 | | 24 | ramesh | localhost:43974 | test | Query | 1446 | acquiring total order isolation | TRUNCATE `oltp6` /* QNO 940 CON_ID 24 */ | 0.000 | | 27 | ramesh | localhost:53284 | test | Query | 1445 | acquiring total order isolation | KILL QUERY @conid /* QNO 923 CON_ID 27 */ | 0.000 | | 29 | ramesh | localhost:53304 | test | Killed | 1445 | acquiring total order isolation | KILL QUERY @conid /* QNO 956 CON_ID 29 */ | 0.000 | | 30 | ramesh | localhost:53318 | test | Query | 1446 | Commit | INSERT INTO `oltp4` ( `id`, `k`) VALUES ( NULL, 565051392 ) /* QNO 913 CON_ID 30 */ | 0.000 | | 32 | ramesh | localhost:53334 | test | Query | 1415 | acquiring total order isolation | ALTER ONLINE TABLE `oltp1` PARTITION BY KEY(`id`) /* QNO 926 CON_ID 32 */ | 0.000 | | 33 | ramesh | localhost:53346 | test | Query | 1205 | acquiring total order isolation | TRUNCATE `oltp4` /* QNO 923 CON_ID 33 */ | 0.000 | | 34 | ramesh | localhost:53360 | test | Query | 1446 | Waiting for table metadata lock | ALTER ONLINE TABLE `oltp6` ROW_FORMAT=COMPACT /* QNO 905 CON_ID 34 */ | 0.000 | | 35 | ramesh | localhost:53374 | test | Query | 1295 | acquiring total order isolation | OPTIMIZE TABLE `oltp3` /* QNO 918 CON_ID 35 */ | 0.000 | | 36 | ramesh | localhost:53376 | test | Query | 1445 | Freeing items | INSERT INTO `oltp6` ( `id`, `k`) VALUES ( NULL, 1681063936 ) /* QNO 905 CON_ID 36 */ | 0.000 | | 38 | ramesh | localhost:53392 | test | Query | 1445 | Init | KILL CONNECTION @conid /* QNO 908 CON_ID 31 */ | 0.000 | | 40 | ramesh | localhost:53406 | test | Query | 1445 | Init | KILL CONNECTION @conid /* QNO 956 CON_ID 21 */ | 0.000 | | 41 | ramesh | localhost:37020 | test | Query | 1445 | Init | COMMIT /* QNO 943 CON_ID 25 */ | 0.000 | | 42 | ramesh | localhost:37022 | test | Query | 1446 | Commit | INSERT INTO `oltp4` ( `id`, `pad` ) VALUES ( NULL, 'm' ) /* QNO 921 CON_ID 26 */ | 0.000 | | 43 | ramesh | localhost:37036 | test | Query | 995 | Init | KILL CONNECTION @conid /* QNO 930 CON_ID 28 */ | 0.000 | | 44 | root | localhost | NULL | Query | 0 | Init | show processlist | 0.000 | | 45 | root | localhost | NULL | Query | 1319 | Checking permissions | create database test_one | 0.000 | +----+-------------+-----------------+------+---------+------+---------------------------------+---------------------------------------------------------------------------------------+----------+ 28 rows in set (0.001 sec)   MariaDB [(none)]>

            seppo To me it seems like KILL does not bf kill conflicting DDL if it owns MDL-lock. From error log I found following (see new.err):

            2022-10-25 18:01:52 24 [Note] WSREP: MDL conflict db=test table=oltp8 ticket=4 solved by abort
            2022-10-25 18:01:53 39 [Note] WSREP: MDL conflict 
            schema:  test
            request: (24 	seqno 490 	wsrep (toi, exec, aborted) cmd 3 3 	ALTER ONLINE TABLE `oltp8` ROW_FORMAT=DYNAMIC /* QNO 942 CON_ID 24 */)
            granted: (43 	seqno -1 	wsrep (local, exec, must_abort) cmd 3 50 	KILL QUERY @conid /* QNO 956 CON_ID 34 */)
            2022-10-25 18:01:53 39 [Note] WSREP: MDL ticket: type: MDL_SHARED_WRITE space: TABLE db: test name: oltp8 (Waiting for table metadata lock)
            2022-10-25 18:01:53 39 [Note] WSREP: MDL conflict-> BF abort
            schema:  test
            request: (24 	seqno 490 	wsrep (toi, exec, aborted) cmd 3 3 	ALTER ONLINE TABLE `oltp8` ROW_FORMAT=DYNAMIC /* QNO 942 CON_ID 24 */)
            granted: (43 	seqno -1 	wsrep (local, exec, must_abort) cmd 3 50 	KILL QUERY @conid /* QNO 956 CON_ID 34 */)
            2022-10-25 18:01:53 39 [Note] WSREP: MDL ticket: type: MDL_SHARED_WRITE space: TABLE db: test name: oltp8 (Waiting for table metadata lock)
            2022-10-25 18:01:53 39 [Note] WSREP: wsrep_abort_thd not effective: bf 22620232664832 victim 22620235196160 wsrep 1 wsrep_on 1 RSU 0 TOI 1 aborting 1
            2022-10-25 18:01:53 39 [Note] WSREP: MDL conflict db=test table=oltp8 ticket=4 solved by abort
            

            jplindst Jan Lindström (Inactive) added a comment - - edited seppo To me it seems like KILL does not bf kill conflicting DDL if it owns MDL-lock. From error log I found following (see new.err): 2022-10-25 18:01:52 24 [Note] WSREP: MDL conflict db=test table=oltp8 ticket=4 solved by abort 2022-10-25 18:01:53 39 [Note] WSREP: MDL conflict schema: test request: (24 seqno 490 wsrep (toi, exec, aborted) cmd 3 3 ALTER ONLINE TABLE `oltp8` ROW_FORMAT=DYNAMIC /* QNO 942 CON_ID 24 */) granted: (43 seqno -1 wsrep (local, exec, must_abort) cmd 3 50 KILL QUERY @conid /* QNO 956 CON_ID 34 */) 2022-10-25 18:01:53 39 [Note] WSREP: MDL ticket: type: MDL_SHARED_WRITE space: TABLE db: test name: oltp8 (Waiting for table metadata lock) 2022-10-25 18:01:53 39 [Note] WSREP: MDL conflict-> BF abort schema: test request: (24 seqno 490 wsrep (toi, exec, aborted) cmd 3 3 ALTER ONLINE TABLE `oltp8` ROW_FORMAT=DYNAMIC /* QNO 942 CON_ID 24 */) granted: (43 seqno -1 wsrep (local, exec, must_abort) cmd 3 50 KILL QUERY @conid /* QNO 956 CON_ID 34 */) 2022-10-25 18:01:53 39 [Note] WSREP: MDL ticket: type: MDL_SHARED_WRITE space: TABLE db: test name: oltp8 (Waiting for table metadata lock) 2022-10-25 18:01:53 39 [Note] WSREP: wsrep_abort_thd not effective: bf 22620232664832 victim 22620235196160 wsrep 1 wsrep_on 1 RSU 0 TOI 1 aborting 1 2022-10-25 18:01:53 39 [Note] WSREP: MDL conflict db=test table=oltp8 ticket=4 solved by abort
            ramesh Ramesh Sivaraman made changes -
            Attachment bt_all.txt [ 66034 ]
            ramesh Ramesh Sivaraman made changes -
            Assignee Ramesh Sivaraman [ JIRAUSER48189 ] Seppo Jaakola [ seppo ]
            Status In Testing [ 10301 ] Stalled [ 10000 ]

            I found that in wsrep_abort_thd we have:

              if ((WSREP(bf_thd) ||
                   ((WSREP_ON || bf_thd->variables.wsrep_OSU_method == WSREP_OSU_RSU) &&
            	 wsrep_thd_is_toi(bf_thd))) &&
                   !wsrep_thd_is_aborting(victim_thd))
             
            but wsrep_thd_is_aborting(victim_thd) == true
            

            Based on debug output OPTIMIZE TABLE is aborted and INSERT is aborting state.

            jplindst Jan Lindström (Inactive) added a comment - - edited I found that in wsrep_abort_thd we have: if ((WSREP(bf_thd) || ((WSREP_ON || bf_thd->variables.wsrep_OSU_method == WSREP_OSU_RSU) && wsrep_thd_is_toi(bf_thd))) && !wsrep_thd_is_aborting(victim_thd))   but wsrep_thd_is_aborting(victim_thd) == true Based on debug output OPTIMIZE TABLE is aborted and INSERT is aborting state.

            From same run

            MariaDB [(none)]> show processlist;
            +----+-------------+-----------------+------+---------+------+---------------------------------+------------------------------------------------------------------------------------------------------+----------+
            | Id | User        | Host            | db   | Command | Time | State                           | Info                                                                                                 | Progress |
            +----+-------------+-----------------+------+---------+------+---------------------------------+------------------------------------------------------------------------------------------------------+----------+
            |  1 | system user |                 | NULL | Sleep   |  376 | wsrep aborter idle              | NULL                                                                                                 |    0.000 |
            |  2 | system user |                 | NULL | Sleep   |  376 | Closing tables                  | NULL                                                                                                 |    0.000 |
            |  5 | system user |                 | NULL | Daemon  | NULL | InnoDB purge worker             | NULL                                                                                                 |    0.000 |
            |  6 | system user |                 | NULL | Daemon  | NULL | InnoDB purge worker             | NULL                                                                                                 |    0.000 |
            |  3 | system user |                 | NULL | Daemon  | NULL | InnoDB purge coordinator        | NULL                                                                                                 |    0.000 |
            |  4 | system user |                 | NULL | Daemon  | NULL | InnoDB purge worker             | NULL                                                                                                 |    0.000 |
            |  7 | system user |                 | NULL | Daemon  | NULL | InnoDB shutdown handler         | NULL                                                                                                 |    0.000 |
            | 11 | system user |                 | NULL | Sleep   |  375 | wsrep applier idle              | NULL                                                                                                 |    0.000 |
            | 12 | system user |                 | NULL | Sleep   |  375 | Closing tables                  | NULL                                                                                                 |    0.000 |
            | 14 | system user |                 | NULL | Sleep   |  375 | wsrep applier idle              | NULL                                                                                                 |    0.000 |
            | 22 | ramesh      | localhost:50776 | test | Query   |  295 | acquiring total order isolation | OPTIMIZE TABLE `oltp8` /* QNO 982 CON_ID 22 */                                                       |    0.000 |
            | 23 | ramesh      | localhost:50788 | test | Query   |  325 | Commit                          | INSERT INTO `oltp10` ( `id` ) VALUES ( NULL ),( NULL ),( NULL ),( NULL )  /* QNO 940 CON_ID 23 */    |    0.000 |
            | 24 | ramesh      | localhost:50802 | test | Query   |  325 | Waiting for table metadata lock | ALTER ONLINE TABLE `oltp8` ROW_FORMAT=DYNAMIC /* QNO 942 CON_ID 24 */                                |    0.000 |
            | 27 | ramesh      | localhost:50836 | test | Query   |  325 | Commit                          | INSERT INTO `oltp7` ( `id` ) VALUES ( NULL ),( NULL ),( NULL ),( NULL )  /* QNO 934 CON_ID 27 */     |    0.000 |
            | 28 | ramesh      | localhost:50852 | test | Query   |  295 | acquiring total order isolation | KILL CONNECTION @conid /* QNO 931 CON_ID 28 */                                                       |    0.000 |
            | 31 | ramesh      | localhost:50878 | test | Query   |  115 | acquiring total order isolation | KILL CONNECTION @conid /* QNO 929 CON_ID 31 */                                                       |    0.000 |
            | 35 | ramesh      | localhost:50900 | test | Query   |   85 | acquiring total order isolation | KILL CONNECTION @conid /* QNO 941 CON_ID 35 */                                                       |    0.000 |
            | 36 | ramesh      | localhost:50916 | test | Query   |  325 | acquiring total order isolation | KILL QUERY @conid /* QNO 927 CON_ID 36 */                                                            |    0.000 |
            | 37 | ramesh      | localhost:54008 | test | Query   |   25 | Init                            | INSERT INTO `oltp5` ( `id` ) VALUES ( NULL ),( NULL ),( NULL ),( NULL )  /* QNO 953 CON_ID 29 */     |    0.000 |
            | 39 | ramesh      | localhost:54032 | test | Query   |   25 | Init                            | INSERT INTO `oltp8` ( `id`, `id`) VALUES ( NULL, -699531264 )  /* QNO 959 CON_ID 33 */               |    0.000 |
            | 40 | ramesh      | localhost:54034 | test | Query   |  325 | acquiring total order isolation | OPTIMIZE TABLE `oltp8` /* QNO 943 CON_ID 26 */                                                       |    0.000 |
            | 42 | ramesh      | localhost:52848 | test | Query   |   25 | Init                            | SELECT CONCAT('SELECT * FROM INFORMATION_SCHEMA.',table_name,' ORDER BY RAND() LIMIT 10') FROM INFOR |    0.000 |
            | 43 | ramesh      | localhost:52850 | test | Killed  |  325 | acquiring total order isolation | KILL QUERY @conid /* QNO 956 CON_ID 34 */                                                            |    0.000 |
            | 44 | ramesh      | localhost:52856 | test | Query   |  325 | Commit                          | INSERT INTO `oltp1` ( `id` ) VALUES ( NULL ),( NULL ),( NULL ),( NULL )  /* QNO 983 CON_ID 21 */     |    0.000 |
            | 45 | ramesh      | localhost:52864 | test | Killed  |  325 | Commit                          | INSERT INTO `oltp7` ( `id`, `pad` ) VALUES ( NULL, 'v' ) /* QNO 952 CON_ID 25 */                     |    0.000 |
            | 46 | ramesh      | localhost:52876 | test | Query   |   25 | Init                            | INSERT INTO `oltp4` ( `c` ) VALUES ( 's' )  /* QNO 939 CON_ID 32 */                                  |    0.000 |
            | 47 | ramesh      | localhost:52890 | test | Query   |  145 | acquiring total order isolation | ALTER ONLINE TABLE `oltp9` FORCE, LOCK=SHARED, ALGORITHM=INPLACE /* QNO 961 CON_ID 25 */             |    0.000 |
            | 48 | root        | localhost       | NULL | Query   |  177 | acquiring total order isolation | create database test_one                                                                             |    0.000 |
            | 49 | root        | localhost       | NULL | Query   |    0 | Init                            | show processlist                                                                                     |    0.000 |
            +----+-------------+-----------------+------+---------+------+---------------------------------+------------------------------------------------------------------------------------------------------+----------+
            29 rows in set (0.001 sec)
             
            MariaDB [(none)]> 
            

            jplindst Jan Lindström (Inactive) added a comment - From same run MariaDB [(none)]> show processlist; +----+-------------+-----------------+------+---------+------+---------------------------------+------------------------------------------------------------------------------------------------------+----------+ | Id | User | Host | db | Command | Time | State | Info | Progress | +----+-------------+-----------------+------+---------+------+---------------------------------+------------------------------------------------------------------------------------------------------+----------+ | 1 | system user | | NULL | Sleep | 376 | wsrep aborter idle | NULL | 0.000 | | 2 | system user | | NULL | Sleep | 376 | Closing tables | NULL | 0.000 | | 5 | system user | | NULL | Daemon | NULL | InnoDB purge worker | NULL | 0.000 | | 6 | system user | | NULL | Daemon | NULL | InnoDB purge worker | NULL | 0.000 | | 3 | system user | | NULL | Daemon | NULL | InnoDB purge coordinator | NULL | 0.000 | | 4 | system user | | NULL | Daemon | NULL | InnoDB purge worker | NULL | 0.000 | | 7 | system user | | NULL | Daemon | NULL | InnoDB shutdown handler | NULL | 0.000 | | 11 | system user | | NULL | Sleep | 375 | wsrep applier idle | NULL | 0.000 | | 12 | system user | | NULL | Sleep | 375 | Closing tables | NULL | 0.000 | | 14 | system user | | NULL | Sleep | 375 | wsrep applier idle | NULL | 0.000 | | 22 | ramesh | localhost:50776 | test | Query | 295 | acquiring total order isolation | OPTIMIZE TABLE `oltp8` /* QNO 982 CON_ID 22 */ | 0.000 | | 23 | ramesh | localhost:50788 | test | Query | 325 | Commit | INSERT INTO `oltp10` ( `id` ) VALUES ( NULL ),( NULL ),( NULL ),( NULL ) /* QNO 940 CON_ID 23 */ | 0.000 | | 24 | ramesh | localhost:50802 | test | Query | 325 | Waiting for table metadata lock | ALTER ONLINE TABLE `oltp8` ROW_FORMAT=DYNAMIC /* QNO 942 CON_ID 24 */ | 0.000 | | 27 | ramesh | localhost:50836 | test | Query | 325 | Commit | INSERT INTO `oltp7` ( `id` ) VALUES ( NULL ),( NULL ),( NULL ),( NULL ) /* QNO 934 CON_ID 27 */ | 0.000 | | 28 | ramesh | localhost:50852 | test | Query | 295 | acquiring total order isolation | KILL CONNECTION @conid /* QNO 931 CON_ID 28 */ | 0.000 | | 31 | ramesh | localhost:50878 | test | Query | 115 | acquiring total order isolation | KILL CONNECTION @conid /* QNO 929 CON_ID 31 */ | 0.000 | | 35 | ramesh | localhost:50900 | test | Query | 85 | acquiring total order isolation | KILL CONNECTION @conid /* QNO 941 CON_ID 35 */ | 0.000 | | 36 | ramesh | localhost:50916 | test | Query | 325 | acquiring total order isolation | KILL QUERY @conid /* QNO 927 CON_ID 36 */ | 0.000 | | 37 | ramesh | localhost:54008 | test | Query | 25 | Init | INSERT INTO `oltp5` ( `id` ) VALUES ( NULL ),( NULL ),( NULL ),( NULL ) /* QNO 953 CON_ID 29 */ | 0.000 | | 39 | ramesh | localhost:54032 | test | Query | 25 | Init | INSERT INTO `oltp8` ( `id`, `id`) VALUES ( NULL, -699531264 ) /* QNO 959 CON_ID 33 */ | 0.000 | | 40 | ramesh | localhost:54034 | test | Query | 325 | acquiring total order isolation | OPTIMIZE TABLE `oltp8` /* QNO 943 CON_ID 26 */ | 0.000 | | 42 | ramesh | localhost:52848 | test | Query | 25 | Init | SELECT CONCAT('SELECT * FROM INFORMATION_SCHEMA.',table_name,' ORDER BY RAND() LIMIT 10') FROM INFOR | 0.000 | | 43 | ramesh | localhost:52850 | test | Killed | 325 | acquiring total order isolation | KILL QUERY @conid /* QNO 956 CON_ID 34 */ | 0.000 | | 44 | ramesh | localhost:52856 | test | Query | 325 | Commit | INSERT INTO `oltp1` ( `id` ) VALUES ( NULL ),( NULL ),( NULL ),( NULL ) /* QNO 983 CON_ID 21 */ | 0.000 | | 45 | ramesh | localhost:52864 | test | Killed | 325 | Commit | INSERT INTO `oltp7` ( `id`, `pad` ) VALUES ( NULL, 'v' ) /* QNO 952 CON_ID 25 */ | 0.000 | | 46 | ramesh | localhost:52876 | test | Query | 25 | Init | INSERT INTO `oltp4` ( `c` ) VALUES ( 's' ) /* QNO 939 CON_ID 32 */ | 0.000 | | 47 | ramesh | localhost:52890 | test | Query | 145 | acquiring total order isolation | ALTER ONLINE TABLE `oltp9` FORCE, LOCK=SHARED, ALGORITHM=INPLACE /* QNO 961 CON_ID 25 */ | 0.000 | | 48 | root | localhost | NULL | Query | 177 | acquiring total order isolation | create database test_one | 0.000 | | 49 | root | localhost | NULL | Query | 0 | Init | show processlist | 0.000 | +----+-------------+-----------------+------+---------+------+---------------------------------+------------------------------------------------------------------------------------------------------+----------+ 29 rows in set (0.001 sec)   MariaDB [(none)]>
            jplindst Jan Lindström (Inactive) made changes -
            Attachment new.err [ 66051 ]
            seppo Seppo Jaakola added a comment -

            The pull request should not affect MDL lock conflict resolving, and these tests are specifically for MDL conflicts. Do you have a reason to believe that the PR has caused this behavior, and is a regression?
            I have later work for MDL conflict resolving, expect a PR for that shortly.

            seppo Seppo Jaakola added a comment - The pull request should not affect MDL lock conflict resolving, and these tests are specifically for MDL conflicts. Do you have a reason to believe that the PR has caused this behavior, and is a regression? I have later work for MDL conflict resolving, expect a PR for that shortly.

            ramesh Can you verify using 10.4 branch is this MDL problem reproducible there also?

            jplindst Jan Lindström (Inactive) added a comment - ramesh Can you verify using 10.4 branch is this MDL problem reproducible there also?

            jplindst Could not reproduce the MDL issue on the latest 10.4 base build yet.

            ramesh Ramesh Sivaraman added a comment - jplindst Could not reproduce the MDL issue on the latest 10.4 base build yet.
            ramesh Ramesh Sivaraman made changes -
            Attachment all_bt_28102022.txt [ 66211 ]
            ramesh Ramesh Sivaraman added a comment - - edited

            seppo PFA stack all_bt_28102022.txt from hung server built with latest changes

            9da381796a49b74643808972bcc3028b15962dfd
            

            ramesh Ramesh Sivaraman added a comment - - edited seppo PFA stack all_bt_28102022.txt from hung server built with latest changes 9da381796a49b74643808972bcc3028b15962dfd
            jplindst Jan Lindström (Inactive) made changes -
            Due Date 2022-10-28 2023-10-28
            ramesh Ramesh Sivaraman made changes -
            Attachment bt_all_v2.txt [ 66857 ]

            seppo The latest commit still has a hung issue. PFA full stack bt_all_v2.txt

            10.4.28 665f81757b2a7b29099307f6c5a15e04964502a0

            MariaDB [test]> show processlist;
            +----+-------------+-----------------+------+---------+-------+---------------------------------+--------------------------------------------------------------------------------------------------+----------+
            | Id | User        | Host            | db   | Command | Time  | State                           | Info                                                                                             | Progress |
            +----+-------------+-----------------+------+---------+-------+---------------------------------+--------------------------------------------------------------------------------------------------+----------+
            |  1 | system user |                 | NULL | Sleep   | 54136 | wsrep aborter idle              | NULL                                                                                             |    0.000 |
            |  2 | system user |                 | NULL | Sleep   | 54136 | Closing tables                  | NULL                                                                                             |    0.000 |
            |  3 | system user |                 | NULL | Daemon  |  NULL | InnoDB purge coordinator        | NULL                                                                                             |    0.000 |
            |  6 | system user |                 | NULL | Daemon  |  NULL | InnoDB purge worker             | NULL                                                                                             |    0.000 |
            |  4 | system user |                 | NULL | Daemon  |  NULL | InnoDB purge worker             | NULL                                                                                             |    0.000 |
            |  5 | system user |                 | NULL | Daemon  |  NULL | InnoDB purge worker             | NULL                                                                                             |    0.000 |
            |  7 | system user |                 | NULL | Daemon  |  NULL | InnoDB shutdown handler         | NULL                                                                                             |    0.000 |
            | 12 | system user |                 | NULL | Sleep   | 54136 | Closing tables                  | NULL                                                                                             |    0.000 |
            | 11 | system user |                 | NULL | Sleep   | 54136 | wsrep applier idle              | NULL                                                                                             |    0.000 |
            | 14 | system user |                 | NULL | Sleep   | 54136 | wsrep applier idle              | NULL                                                                                             |    0.000 |
            | 25 | ramesh      | localhost:33900 | test | Query   | 54084 | Waiting for table metadata lock | OPTIMIZE TABLE `oltp8` /* QNO 956 CON_ID 25 */                                                   |    0.000 |
            | 27 | ramesh      | localhost:46682 | test | Query   | 54084 | Commit                          | INSERT INTO `oltp7` ( `id`, `k`) VALUES ( NULL, 2059206656 ) /* QNO 974 CON_ID 27 */             |    0.000 |
            | 28 | ramesh      | localhost:46696 | test | Killed  | 54084 | acquiring total order isolation | KILL QUERY @conid /* QNO 979 CON_ID 28 */                                                        |    0.000 |
            | 31 | ramesh      | localhost:46720 | test | Query   | 53934 | acquiring total order isolation | KILL CONNECTION @conid /* QNO 1001 CON_ID 31 */                                                  |    0.000 |
            | 32 | ramesh      | localhost:46728 | test | Query   | 54084 | acquiring total order isolation | KILL CONNECTION @conid /* QNO 965 CON_ID 32 */                                                   |    0.000 |
            | 33 | ramesh      | localhost:46732 | test | Query   | 54085 | Commit                          | INSERT INTO `oltp11` ( `pad` ) VALUES ( 't' )  /* QNO 963 CON_ID 33 */                           |    0.000 |
            | 34 | ramesh      | localhost:46742 | test | Query   | 54084 | Commit                          | INSERT INTO `oltp4` ( `id` ) VALUES ( NULL ),( NULL ),( NULL ),( NULL )  /* QNO 992 CON_ID 34 */ |    0.000 |
            | 38 | root        | localhost       | test | Query   | 53968 | acquiring total order isolation | create database test_one                                                                         |    0.000 |
            | 42 | root        | localhost       | test | Query   |     0 | Init                            | show processlist                                                                                 |    0.000 |
            +----+-------------+-----------------+------+---------+-------+---------------------------------+--------------------------------------------------------------------------------------------------+----------+
            19 rows in set (0.001 sec)
             
            MariaDB [test]> 
            

            ramesh Ramesh Sivaraman added a comment - seppo The latest commit still has a hung issue. PFA full stack bt_all_v2.txt 10.4.28 665f81757b2a7b29099307f6c5a15e04964502a0 MariaDB [test]> show processlist; +----+-------------+-----------------+------+---------+-------+---------------------------------+--------------------------------------------------------------------------------------------------+----------+ | Id | User | Host | db | Command | Time | State | Info | Progress | +----+-------------+-----------------+------+---------+-------+---------------------------------+--------------------------------------------------------------------------------------------------+----------+ | 1 | system user | | NULL | Sleep | 54136 | wsrep aborter idle | NULL | 0.000 | | 2 | system user | | NULL | Sleep | 54136 | Closing tables | NULL | 0.000 | | 3 | system user | | NULL | Daemon | NULL | InnoDB purge coordinator | NULL | 0.000 | | 6 | system user | | NULL | Daemon | NULL | InnoDB purge worker | NULL | 0.000 | | 4 | system user | | NULL | Daemon | NULL | InnoDB purge worker | NULL | 0.000 | | 5 | system user | | NULL | Daemon | NULL | InnoDB purge worker | NULL | 0.000 | | 7 | system user | | NULL | Daemon | NULL | InnoDB shutdown handler | NULL | 0.000 | | 12 | system user | | NULL | Sleep | 54136 | Closing tables | NULL | 0.000 | | 11 | system user | | NULL | Sleep | 54136 | wsrep applier idle | NULL | 0.000 | | 14 | system user | | NULL | Sleep | 54136 | wsrep applier idle | NULL | 0.000 | | 25 | ramesh | localhost:33900 | test | Query | 54084 | Waiting for table metadata lock | OPTIMIZE TABLE `oltp8` /* QNO 956 CON_ID 25 */ | 0.000 | | 27 | ramesh | localhost:46682 | test | Query | 54084 | Commit | INSERT INTO `oltp7` ( `id`, `k`) VALUES ( NULL, 2059206656 ) /* QNO 974 CON_ID 27 */ | 0.000 | | 28 | ramesh | localhost:46696 | test | Killed | 54084 | acquiring total order isolation | KILL QUERY @conid /* QNO 979 CON_ID 28 */ | 0.000 | | 31 | ramesh | localhost:46720 | test | Query | 53934 | acquiring total order isolation | KILL CONNECTION @conid /* QNO 1001 CON_ID 31 */ | 0.000 | | 32 | ramesh | localhost:46728 | test | Query | 54084 | acquiring total order isolation | KILL CONNECTION @conid /* QNO 965 CON_ID 32 */ | 0.000 | | 33 | ramesh | localhost:46732 | test | Query | 54085 | Commit | INSERT INTO `oltp11` ( `pad` ) VALUES ( 't' ) /* QNO 963 CON_ID 33 */ | 0.000 | | 34 | ramesh | localhost:46742 | test | Query | 54084 | Commit | INSERT INTO `oltp4` ( `id` ) VALUES ( NULL ),( NULL ),( NULL ),( NULL ) /* QNO 992 CON_ID 34 */ | 0.000 | | 38 | root | localhost | test | Query | 53968 | acquiring total order isolation | create database test_one | 0.000 | | 42 | root | localhost | test | Query | 0 | Init | show processlist | 0.000 | +----+-------------+-----------------+------+---------+-------+---------------------------------+--------------------------------------------------------------------------------------------------+----------+ 19 rows in set (0.001 sec)   MariaDB [test]>
            seppo Seppo Jaakola added a comment - - edited

            I completed a RQG run with 10 connections against one node in a cluster, with the attached oltp_and_ddl_v1.yy grammar. It appears that this test is generating a load with mixture of KILL commands, DML, and DDL both with TOI and RSU method. The loaded node will hang very soon after starting the test. The reason for this hang appears to be RSU execution, which desyncs and pauses the node, while there is ongoing TOI replication in execution. Because of the paused state, the earlier TOI execution cannot complete, and because of the executing TOI, the RSU cannot get necessary MDL locks => hence the hang.
            This is rather artificial situation, as DDL is supposed to happen either by TOI or RSU, not mixed. The fix for such hang should be to make the RSU to wait until all ongoing TOI has completed. This may lead to RSU starving with this test load, though.

            seppo Seppo Jaakola added a comment - - edited I completed a RQG run with 10 connections against one node in a cluster, with the attached oltp_and_ddl_v1.yy grammar. It appears that this test is generating a load with mixture of KILL commands, DML, and DDL both with TOI and RSU method. The loaded node will hang very soon after starting the test. The reason for this hang appears to be RSU execution, which desyncs and pauses the node, while there is ongoing TOI replication in execution. Because of the paused state, the earlier TOI execution cannot complete, and because of the executing TOI, the RSU cannot get necessary MDL locks => hence the hang. This is rather artificial situation, as DDL is supposed to happen either by TOI or RSU, not mixed. The fix for such hang should be to make the RSU to wait until all ongoing TOI has completed. This may lead to RSU starving with this test load, though.
            ramesh Ramesh Sivaraman added a comment - - edited

            seppo We can reproduce the problem even without the {{SET wsrep_OSU_method..}}statement. This is a sporadic issue, so start the RQG run multiple times to reproduce the problem.

            PFA grammar oltp_and_ddl_v2.yy to reproduce the issue.

            RQG command to reproduce the hang problem.

            perl runall-new.pl \
               --basedir=/test/mtest/mariadb-10.4.28-linux-x86_64 \
               --vardir=/home/ramesh/galera-rqg-test \
               --mysqld=--wsrep-provider=/test/mtest/mariadb-10.4.28-linux-x86_64/lib/libgalera_smm.so \
               --gendata=conf/mariadb/oltp.zz \
               --grammar=conf/mariadb/oltp_and_ddl_v2.yy \
               --threads=16 \
               --galera=mss \
               --mysqld=--wsrep_sst_method=rsync \
               --mysqld=--core \
               --mysqld=--general-log \
               --mysqld=--general-log-file=queries.log \
               --mysqld=--log-output=file \
               --mysqld=--wsrep-debug=0 \
               --mysqld=--wsrep-sync-wait=15 \
               --mysqld=--wsrep_retry_autocommit=0 \
               --mysqld=--wsrep_slave_threads=12 \
               --mysqld=--wsrep_log_conflicts=1 \
               --mysqld=--wsrep_on=1 \
               --mysqld=--default-storage-engine=innodb \
               --mysqld=--sort_buffer_size=200M \
               --mysqld=--innodb-lock-wait-timeout=1 \
               --mysqld=--gtid_domain_id=10 \
               --mysqld=--wsrep_gtid_domain_id=100 \
               --mysqld=--wsrep_gtid_mode=ON \
               --mysqld=--wsrep_slave_threads=4 \
               --mysqld=--slave_parallel_threads=4 \
               --mysqld=--server_id=11 \
               --mysqld=--gtid_strict_mode=1 \
               --mysqld=--log_slave_updates=ON \
               --mysqld=--log_bin=binlog \
               --mysqld=--binlog_format=ROW \
               --mysqld=--master_info_repository=TABLE \
               --mysqld=--relay_log_info_repository=TABLE 
            

            ramesh Ramesh Sivaraman added a comment - - edited seppo We can reproduce the problem even without the {{SET wsrep_OSU_method..}}statement. This is a sporadic issue, so start the RQG run multiple times to reproduce the problem. PFA grammar oltp_and_ddl_v2.yy to reproduce the issue. RQG command to reproduce the hang problem. perl runall-new.pl \ --basedir=/test/mtest/mariadb-10.4.28-linux-x86_64 \ --vardir=/home/ramesh/galera-rqg-test \ --mysqld=--wsrep-provider=/test/mtest/mariadb-10.4.28-linux-x86_64/lib/libgalera_smm.so \ --gendata=conf/mariadb/oltp.zz \ --grammar=conf/mariadb/oltp_and_ddl_v2.yy \ --threads=16 \ --galera=mss \ --mysqld=--wsrep_sst_method=rsync \ --mysqld=--core \ --mysqld=--general-log \ --mysqld=--general-log-file=queries.log \ --mysqld=--log-output=file \ --mysqld=--wsrep-debug=0 \ --mysqld=--wsrep-sync-wait=15 \ --mysqld=--wsrep_retry_autocommit=0 \ --mysqld=--wsrep_slave_threads=12 \ --mysqld=--wsrep_log_conflicts=1 \ --mysqld=--wsrep_on=1 \ --mysqld=--default-storage-engine=innodb \ --mysqld=--sort_buffer_size=200M \ --mysqld=--innodb-lock-wait-timeout=1 \ --mysqld=--gtid_domain_id=10 \ --mysqld=--wsrep_gtid_domain_id=100 \ --mysqld=--wsrep_gtid_mode=ON \ --mysqld=--wsrep_slave_threads=4 \ --mysqld=--slave_parallel_threads=4 \ --mysqld=--server_id=11 \ --mysqld=--gtid_strict_mode=1 \ --mysqld=--log_slave_updates=ON \ --mysqld=--log_bin=binlog \ --mysqld=--binlog_format=ROW \ --mysqld=--master_info_repository=TABLE \ --mysqld=--relay_log_info_repository=TABLE
            ramesh Ramesh Sivaraman made changes -
            Attachment oltp_and_ddl_v2.yy [ 67462 ]
            seppo Seppo Jaakola added a comment -

            This new RQG grammar makes two other scenarios to surface, I will submit separate MDEV issue for both, as they are not related to the fixes suggested in PR's for this MDEV. The surfacing issues are:

            • KILL command is issued from inside of a transaction. And further, this KILL execution can be a target of another KILL command. BF aborting a KILL inside a transaction appears to cause a crash.
            • SAVEPOINT command will be replicated as query log event. If transaction containing such SAVEPOINT event, has to replay, the replaying may fail for un-assigned character set information.
            seppo Seppo Jaakola added a comment - This new RQG grammar makes two other scenarios to surface, I will submit separate MDEV issue for both, as they are not related to the fixes suggested in PR's for this MDEV. The surfacing issues are: KILL command is issued from inside of a transaction. And further, this KILL execution can be a target of another KILL command. BF aborting a KILL inside a transaction appears to cause a crash. SAVEPOINT command will be replicated as query log event. If transaction containing such SAVEPOINT event, has to replay, the replaying may fail for un-assigned character set information.
            jplindst Jan Lindström (Inactive) made changes -
            Assignee Seppo Jaakola [ seppo ] Jan Lindström [ jplindst ]
            jplindst Jan Lindström (Inactive) made changes -
            Status Stalled [ 10000 ] In Progress [ 3 ]
            jplindst Jan Lindström (Inactive) added a comment - - edited bb-10.4- MDEV-29684 -galera https://github.com/MariaDB/server/commit/c85f18a37feae677a1593fd7bbebd19bb4a5fd94 (sql) https://github.com/MariaDB/server/commit/3e466c745ef3a97e1372a23f37a0cf04745ce25f (test case) https://github.com/MariaDB/server/commit/c76134bb305e49689380e271edd959f656be4de0 (sql) https://github.com/MariaDB/server/commit/83a450464806625e8d77e1aeb93aa0f5b972ec59 (InnoDB) https://github.com/MariaDB/server/commit/54abb014a3abe2a2f09e7bdeb799b5614d9d94a4 (sql + InnoDB) bb-10.5- MDEV-29684 -galera (about same as 10.4) https://github.com/MariaDB/server/commit/e0e06ed82a6d75a279a425cd362537c3d3476ee9 bb-10.6- MDEV-29684 -galera https://github.com/MariaDB/server/commit/bcf368f58883f7efc8676426c56853014d15857d https://github.com/MariaDB/server/commit/31a8b027422ad63e0b1bc91ed982a5ba309d7c51 https://github.com/MariaDB/server/commit/2a1e41727cb9f4b563fb5bbefd4047d3e5878519 https://github.com/MariaDB/server/commit/640b5d52a8d48934c980d7cb15c02661ad20e56d https://github.com/MariaDB/server/commit/2a92e74dd4611367c3581e1b6542c537863b4675
            jplindst Jan Lindström (Inactive) made changes -
            Assignee Jan Lindström [ jplindst ] Marko Mäkelä [ marko ]
            Status In Progress [ 3 ] In Review [ 10002 ]

            seppo, I provided some review comments to a 10.6 version of this: 1, 2, 3, 4. Please address it.

            I did not look at other versions yet.

            marko Marko Mäkelä added a comment - seppo , I provided some review comments to a 10.6 version of this: 1 , 2 , 3 , 4 . Please address it. I did not look at other versions yet.
            marko Marko Mäkelä made changes -
            Assignee Marko Mäkelä [ marko ] Seppo Jaakola [ seppo ]
            Status In Review [ 10002 ] Stalled [ 10000 ]
            marko Marko Mäkelä made changes -

            For the 10.6 version, a fix of MDEV-29860 could be useful to have.

            marko Marko Mäkelä added a comment - For the 10.6 version, a fix of MDEV-29860 could be useful to have.
            jplindst Jan Lindström (Inactive) made changes -
            Assignee Seppo Jaakola [ seppo ] Jan Lindström [ jplindst ]
            jplindst Jan Lindström (Inactive) made changes -
            Status Stalled [ 10000 ] In Progress [ 3 ]
            jplindst Jan Lindström (Inactive) made changes -
            Assignee Jan Lindström [ jplindst ] Marko Mäkelä [ marko ]
            Status In Progress [ 3 ] In Review [ 10002 ]
            marko Marko Mäkelä made changes -
            Assignee Marko Mäkelä [ marko ] Jan Lindström [ jplindst ]
            Status In Review [ 10002 ] Stalled [ 10000 ]
            jplindst Jan Lindström (Inactive) made changes -
            issue.field.resolutiondate 2023-01-16 05:37:35.0 2023-01-16 05:37:35.695
            jplindst Jan Lindström (Inactive) made changes -
            Fix Version/s 10.4.28 [ 28509 ]
            Fix Version/s 10.5.19 [ 28511 ]
            Fix Version/s 10.6.12 [ 28513 ]
            Fix Version/s 10.7.8 [ 28515 ]
            Fix Version/s 10.8.7 [ 28517 ]
            Fix Version/s 10.9.5 [ 28519 ]
            Fix Version/s 10.10.3 [ 28521 ]
            Fix Version/s 10.4 [ 22408 ]
            Resolution Fixed [ 1 ]
            Status Stalled [ 10000 ] Closed [ 6 ]

            People

              jplindst Jan Lindström (Inactive)
              seppo Seppo Jaakola
              Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Git Integration

                  Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.