[MDEV-10604] Create a list of unstable MTR tests to be disabled in distribution builds Created: 2016-08-20  Updated: 2021-05-30  Resolved: 2016-08-27

Status: Closed
Project: MariaDB Server
Component/s: Tests
Fix Version/s: 10.1.17, 10.0.27

Type: Task Priority: Blocker
Reporter: Elena Stepanova Assignee: Elena Stepanova
Resolution: Fixed Votes: 0
Labels: None

Attachments: PNG File screenshot-1.png    
Issue Links:
PartOf
includes MDEV-11947 InnoDB purge workers fail to shut down Closed
Relates
relates to MDBF-36 Remove skip-test=unstable-tests Closed
relates to MDBF-155 Test running (almost) all tests on al... Closed

 Description   

There are lots of MTR tests which fail sporadically. They create a problem for distribution builds, where a single test failure means a failed build.

Many of these failures have been identified in scope of the effort to make buildbot green (MDEV-7069), but this activity is still far from succeeding. For most of failures separate JIRA issues exist.

We don't want to discard these tests, eventually they need to be fixed (or the product needs to be fixed, if the failure is real); but meanwhile we don't want these tests to disrupt distribution builds.

Distribution builds already run with non-default options, e.g. --skip-test-list, but so far Debian has been maintaining these lists on their own, which is inefficient.
We need to create such a list in the main tree. Apart from distribution builds, it can be beneficial for anyone who wants to run a smaller but more reliable set of tests.

So the current plan is to take a pessimistic approach and create a rather generous list. It shouldn't matter much for distribution builds – default MTR test suite is excessive for testing a build, so skipping 10-15% of tests shouldn't hurt much. For now, the list (mysql-test/unstable-tests) will have the contents as below; it can be re-considered and tuned later.

  • all tests which have been identified as unstable by Debian, either through bug reports they filed, or through unstable lists they are currently maintaining;
  • all tests which have failed in buildbot on 10.0 main tree since the beginning of the year, unless those were clearly environmental failures, e.g. disk space problems and such, and have not been fixed after the last failure (in general, the observation period is roughly 6 months, based on lowest frequency for known sporadic failures been 2-3 times a year);
  • tests which have been modified since the <upcoming release>-2 (that is, for 10.0.27 it will contain tests which have been modified since 10.0.25). The idea here is is that if a test has been modified, we cannot guarantee its behavior to be stable until we have enough statistics for that;
    Note: An exception here is TokuDB tests – since they come through a merge, technically they all get marked as modified, and since there are lots of them, going through them one by one is too time-consuming. So, their modifications will not be taken into account (currently it's not important because they are disabled anyway due to MDEV-9891);
  • create a separate JIRA item for each disabled test, if such item does not exist.

Additionally, there will be a small change in mtr_cases.pm to allow wildcards in the skip list. It's just a convenience measure, because we need to disable big chunks of tests, such as TokuDB or Spider.



 Comments   
Comment by Elena Stepanova [ 2016-08-20 ]

Below is the list of (all) tests failed in buildbot on 10.0 main tree since the beginning of the year. It is an entry point, the failures, especially one-time, still need to be looked at separately, because they can be false positives. e.g. when tests fail massively due to an environment problem, disk space etc.

Also, some of them can already be fixed. 

MariaDB [buildbot]> select tf.test_name tname, max(tr.dt) as last_fail, count(*) fail_count from test_failure tf join test_run tr on tf.test_run_id=tr.id where branch = '10.0' and dt > '2016-01-01 00:00:00' group by tname order by test_name;


+---------------------------------------------------+---------------------+------------+
 
| tname                                             | last_fail           | fail_count |
 
+---------------------------------------------------+---------------------+------------+
 
| archive.archive                                   | 2016-04-27 00:59:10 |          3 | # --------- Broken build in April, MDEV-10613
 
| archive.archive-big                               | 2016-06-22 12:52:32 |          1 | # UNSTABLE: MDEV-10615
 
| archive.discover                                  | 2016-08-02 14:40:24 |          1 | # UNSTABLE: MDEV-10510
 
| binlog.binlog_checksum                            | 2016-03-18 00:44:55 |          1 | # ?-------- Startup problem on p8-rhel6-bintar-debug in March, logs not available, probably env
 
| binlog.binlog_commit_wait                         | 2016-08-13 17:13:47 |          2 | # UNSTABLE: MDEV-10150
 
| binlog.binlog_mysqlbinlog_row_innodb              | 2016-02-03 18:53:38 |          1 | # ?-------- Startup problem on p8-rhel6-bintar-debug in February, logs not available, probably env
 
| binlog.binlog_row_binlog                          | 2016-04-26 23:04:03 |          1 | # --------- Broken build in April, MDEV-10613
 
| binlog.binlog_stm_binlog                          | 2016-04-22 09:07:33 |          1 | # ?-------- Startup problem on p8-rhel6-bintar-debug in April, logs not available, probably env
 
| binlog.binlog_unsafe                              | 2016-04-27 00:59:10 |          1 | # --------- Broken build in April, MDEV-10613
 
| binlog.binlog_xa_recover                          | 2016-08-17 01:57:51 |          5 | # UNSTABLE: MDEV-8517
 
| connect.alter                                     | 2016-07-25 02:53:29 |         35 | # ?-------- Connect does not work on Sparc: MDEV-10616
 
| connect.alter_xml                                 | 2016-07-25 02:53:29 |         35 | # ?-------- Connect does not work on Sparc: MDEV-10616
 
| connect.bin                                       | 2016-07-25 02:53:29 |         35 | # ?-------- Connect does not work on Sparc: MDEV-10616
 
| connect.csv                                       | 2016-07-25 02:53:29 |         35 | # ?-------- Connect does not work on Sparc: MDEV-10616
 
| connect.datest                                    | 2016-07-25 02:53:29 |         35 | # ?-------- Connect does not work on Sparc: MDEV-10616
 
| connect.dbf                                       | 2016-07-25 02:53:29 |         35 | # ?-------- Connect does not work on Sparc: MDEV-10616
 
| connect.json                                      | 2016-07-25 02:53:29 |         35 | # ?-------- Connect does not work on Sparc: MDEV-10616
 
| connect.part_file                                 | 2016-07-25 02:53:29 |         35 | # ?-------- Connect does not work on Sparc: MDEV-10616
 
| connect.part_table                                | 2016-07-25 02:53:29 |         35 | # ?-------- Connect does not work on Sparc: MDEV-10616
 
| connect.secure_file_priv                          | 2016-07-25 02:53:29 |         35 | # ?-------- Connect does not work on Sparc: MDEV-10616
 
| connect.tbl                                       | 2016-08-16 23:49:43 |         22 | # UNSTABLE: MDEV-9844, MDEV-10179
 
| csv.csv                                           | 2016-04-26 16:33:01 |          1 | # --------- Broken build in April, MDEV-10613
 
| engines/funcs.crash_manytables_string             | 2016-02-12 08:52:33 |          1 | 
| engines/funcs.ld_all_number_string_calendar_types | 2016-05-28 02:40:24 |         11 |
 
| engines/funcs.se_join_cross                       | 2016-03-12 22:48:41 |          7 |
 
| engines/funcs.se_join_default                     | 2016-03-12 22:48:41 |          8 |
 
| engines/funcs.se_join_inner                       | 2016-03-12 22:48:41 |          7 |
 
| engines/funcs.se_join_left                        | 2016-03-10 16:02:22 |          4 |
 
| engines/funcs.se_join_left_outer                  | 2016-03-10 16:02:22 |          4 |
 
| engines/funcs.se_join_natural_left                | 2016-03-10 16:02:22 |          1 |
 
| engines/funcs.se_join_natural_left_outer          | 2016-03-12 22:48:41 |          1 |
 
| engines/funcs.se_join_natural_right               | 2016-03-11 18:28:29 |          1 |
 
| engines/funcs.se_join_natural_right_outer         | 2016-02-11 14:15:17 |          1 |
 
| engines/funcs.se_string_having                    | 2016-03-12 22:48:41 |          6 |
 
| engines/funcs.ta_add_column                       | 2016-05-29 05:05:22 |         12 |
 
| engines/funcs.ta_add_column2                      | 2016-02-12 08:52:33 |          1 |
 
| engines/funcs.ta_add_column_first                 | 2016-05-29 05:05:22 |          9 |
 
| engines/funcs.ta_add_column_middle                | 2016-05-29 05:05:22 |          6 |
 
| engines/funcs.tc_multicolumn_different            | 2016-05-29 05:05:22 |          5 |
 
| engines/funcs.tc_partition_key                    | 2016-05-29 05:05:22 |          4 |
 
| engines/funcs.tc_partition_linear_key             | 2016-05-29 05:05:22 |          4 |
 
| engines/iuds.type_bit_iuds                        | 2016-05-28 02:40:24 |         11 |
 
| engines/iuds.update_delete_number                 | 2016-05-28 02:40:24 |         11 |
 
| federated.federatedx                              | 2016-06-20 13:58:49 |          1 | # UNSTABLE: MDEV-10617
 
| federated.federated_innodb                        | 2016-06-21 06:24:19 |          1 | # UNSTABLE: MDEV-10617
 
| federated.federated_transactions                  | 2016-08-14 16:32:49 |         12 | # UNSTABLE: MDEV-10617
 
| funcs_1.processlist_priv_no_prot                  | 2016-06-22 00:12:51 |          1 | # --------- Fixed after the last failure: MDEV-10311
 
| innodb.binlog_consistent                          | 2016-06-20 11:32:10 |          1 | # UNSTABLE: MDEV-10618
 
| innodb.innodb-alter-table                         | 2016-08-14 16:32:49 |          6 | # UNSTABLE: MDEV-10619
 
| innodb.innodb-alter-tempfile                      | 2016-07-25 06:50:18 |          2 | # --------- Fixed after the last failure: MDEV-10469
 
| innodb.innodb-changed-pages                       | 2016-02-03 18:53:38 |          1 | # ?-------- Startup problem on p8-rhel6-bintar-debug in Feb, logs not available, probably env
 
| innodb.innodb-fk                                  | 2016-03-10 11:24:28 |          1 | # ?-------- Startup problem on p8-rhel6-bintar-debug in Feb, logs not available, probably env
 
| innodb.innodb-get-fk                              | 2016-03-01 12:27:56 |          1 | # ?-------- Apparently timeout on valgrind, one-time in March, logs not available
 
| innodb.innodb-wl5522-debug-zip                    | 2016-07-18 16:31:31 |          1 | # UNSTABLE: MDEV-10427
 
| innodb.innodb_bug30423                            | 2016-06-20 11:50:11 |          3 | # UNSTABLE: MDEV-7311
 
| innodb.innodb_bug39438                            | 2016-02-11 12:54:12 |          1 | # ?-------- Startup problem on p8-rhel6-bintar-debug in Feb, logs not available, probably env
 
| innodb.innodb_bug42101-nonzero                    | 2016-03-16 11:53:05 |          1 | # ?-------- Startup problem on p8-rhel6-bintar-debug in March, logs not available, probably env
 
| innodb.innodb_bug53290                            | 2016-06-21 01:10:41 |         51 | # --------- Fixed after the last failure: MDEV-9356
 
| innodb.innodb_simulate_comp_failures_small        | 2016-04-22 09:07:33 |          1 | # ?-------- Startup problem on p8-rhel6-bintar-debug in March, logs not available, probably env
 
| main.contributors                                 | 2016-08-02 17:42:50 |        115 | # --------- Fixed after the last failure by b5fb2a685b6
 
| main.create_delayed                               | 2016-04-26 18:40:16 |          1 | # UNSTABLE: MDEV-10605 (one-time failure)
 
| main.create_or_replace                            | 2016-06-22 02:49:50 |         42 | # --------- apparently fixed by 1f761c5615c7 (MDEV-9728)
 
| main.delayed                                      | 2016-04-26 17:24:05 |         84 | # --------- Fixed after the last failure (MDEV-9679)
 
| main.events_restart                               | 2016-06-16 12:21:02 |          2 | # --------- Fixed after the last failure (MDEV-10043)
 
| main.host_cache_size_functionality                | 2016-02-04 14:06:47 |          1 | # UNSTABLE: MDEV-10606 (2 failures, counting dev branches)
 
| main.index_merge_innodb                           | 2016-06-09 01:07:10 |          2 | # UNSTABLE: MDEV-7142
 
| main.information_schema_stats                     | 2016-07-18 17:58:36 |          2 | # --------- Fixed after the last failure (MDEV-10428)
 
| main.innodb_mysql_lock                            | 2016-08-16 23:08:47 |          5 | # UNSTABLE: MDEV-7861
 
| main.kill_processlist-6619                        | 2016-04-23 09:27:01 |          5 | # --------- Fixed after the last failure (MDEV-9945)
 
| main.mdev-504                                     | 2016-08-15 18:38:19 |          4 | # UNSTABLE: MDEV-10607
 
| main.mdev375                                      | 2016-08-15 18:38:19 |          2 | # UNSTABLE: MDEV-10607
 
| main.merge                                        | 2016-08-15 18:38:19 |          1 | # UNSTABLE: MDEV-10607 (one-time failure)
 
| main.myisam                                       | 2016-03-10 12:34:07 |          1 | # ?-------- One-time timeout in March on valgrind, logs not available, possibly environmental
 
| main.mysqlbinlog-innodb                           | 2016-03-18 00:44:55 |          2 | # ?-------- Two server startup problems on p8-rhel6-bintar-debug in March, logs not available, probably environmental
 
| main.mysqldump                                    | 2016-06-24 19:11:47 |         49 | # UNSTABLE: MDEV-10512
 
| main.mysqld_option_err                            | 2016-06-16 15:18:22 |         10 | # --------- Fixed after the last failure (MDEV-10257)
 
| main.mysql_client_test_nonblock                   | 2016-08-15 17:21:02 |         88 | # --------- Fixed after the last failure (MDEV-10559)
 
| main.parser_bug21114_innodb                       | 2016-03-16 19:25:39 |          1 | # ?-------- MDEV-10610 Not marked as unstable yet (last failure in March, total 2, previous on dev branch in 2015)
 
| main.partition_column                             | 2016-01-27 02:38:59 |          6 | # --------- Fixed after the last failure (MDEV-9358)
 
| main.partition_innodb_plugin                      | 2016-04-23 09:05:02 |          1 | # --------- Fixed after the last failure (MDEV-9975)
 
| main.pool_of_threads                              | 2016-07-25 05:35:05 |          3 | # UNSTABLE: MDEV-10100
 
| main.query_cache                                  | 2016-04-26 17:58:11 |          1 | # UNSTABLE: MDEV-10611 (one-time failure)
 
| main.shutdown                                     | 2016-08-16 23:16:38 |          2 | # UNSTABLE: MDEV-10612
 
| main.sp-security                                  | 2016-05-25 00:46:01 |          1 | # UNSTABLE: MDEV-10607
 
| main.sp-threads                                   | 2016-04-21 19:57:06 |          2 | # --------- Fixed after the last failure (MDEV-9970)
 
| main.ssl                                          | 2016-07-12 14:17:58 |          3 | # UNSTABLE: MDEV-10211
 
| main.subselect                                    | 2016-04-26 15:40:50 |          2 | # --------- Broken build in April, MDEV-10613
 
| main.subselect_innodb                             | 2016-03-13 05:51:21 |          1 | # UNSTABLE: MDEV-10614
 
| main.subselect_no_exists_to_in                    | 2016-04-26 17:24:05 |          3 | # --------- Broken build in April, MDEV-10613
 
| main.subselect_no_mat                             | 2016-04-27 00:59:10 |          5 | # --------- Broken build in April, MDEV-10613
 
| main.subselect_no_opts                            | 2016-04-26 16:33:01 |          1 | # --------- Broken build in April, MDEV-10613
 
| main.subselect_no_scache                          | 2016-04-26 15:40:50 |          1 | # --------- Broken build in April, MDEV-10613
 
| main.subselect_sj2                                | 2016-03-10 11:38:19 |          1 | # --------- Broken build in April, MDEV-10613
 
| main.type_date                                    | 2016-06-29 02:44:59 |          1 | # --------- Fixed after the last failure (MDEV-10316)
 
| main.xtradb_mrr                                   | 2016-08-02 14:30:41 |          2 | # --------- Fixed after the last failure (MDEV-9946)
 
| metadata_lock_info.global_read_lock               | 2016-06-22 02:49:50 |         16 | # --------- Fixed after the last failure (MDEV-9728)
 
| metadata_lock_info.table_metadata_lock            | 2016-06-22 02:49:50 |         16 | # --------- Fixed after the last failure (MDEV-9728)
 
| metadata_lock_info.user_lock                      | 2016-06-22 02:49:50 |         14 | # --------- Fixed after the last failure (MDEV-9728)
 
| multi_source.gtid                                 | 2016-06-22 04:38:18 |          6 | # UNSTABLE: MDEV-10620
 
| multi_source.gtid_ignore_duplicates               | 2016-04-26 17:29:15 |          1 | # ?-------- Startup problem on p8-rhel6-bintar-debug in Feb, logs not available, probably env
 
| multi_source.simple                               | 2016-07-18 12:36:20 |          2 | # UNSTABLE: MDEV-4633
 
| multi_source.status_vars                          | 2016-08-14 16:32:49 |          2 | # UNSTABLE: MDEV-4632
 
| parts.partition_debug_innodb                      | 2016-02-11 11:40:20 |          2 | # --------- Fixed after the last failure (MDEV-9355)
 
| parts.partition_float_myisam                      | 2016-06-23 13:28:12 |          1 | # UNSTABLE: MDEV-10621
 
| parts.partition_int_myisam                        | 2016-06-21 20:50:39 |          1 | # UNSTABLE: MDEV-10621
 
| perfschema.socket_summary_by_event_name_func      | 2016-08-17 04:58:57 |          2 | # UNSTABLE: MDEV-10622
 
| perfschema.trigger_table_io                       | 2016-04-26 13:26:18 |          1 | # --------- Broken build in April, MDEV-10613
 
| plugins.feedback_plugin_send                      | 2016-06-18 16:38:55 |         43 | # UNSTABLE: MDEV-7932
 
| plugins.server_audit                              | 2016-02-17 07:25:17 |         14 | # UNSTABLE: MDEV-9562
 
| plugins.show_all_plugins                          | 2016-02-17 07:25:17 |         14 | # ?-------- Was failing in Feb on Azure, along with other plugin tests, probably environmental
 
| plugins.thread_pool_server_audit                  | 2016-02-17 07:25:17 |         48 | # UNSTABLE: MDEV-9562
 
| rpl.last_insert_id                                | 2016-06-22 18:56:04 |          2 | # UNSTABLE: MDEV-10625
 
| rpl.rpl_auto_increment                            | 2016-04-27 14:48:58 |          1 | # ?-------- Startup problem on p8-rhel6-bintar-debug in Apr, logs not available, probably env
 
| rpl.rpl_auto_increment_bug45679                   | 2016-04-23 00:28:07 |          2 | # ?-------- Startup problem on p8-rhel6-bintar-debug in Apr, logs not available, probably env
 
| rpl.rpl_auto_increment_update_failure             | 2016-06-18 11:17:39 |          1 | # UNSTABLE: MDEV-10625
 
| rpl.rpl_binlog_errors                             | 2016-04-26 17:29:15 |          2 | # ?-------- Startup problem on p8-rhel6-bintar-debug in Apr, logs not available, probably env
 
| rpl.rpl_binlog_grant                              | 2016-02-03 18:53:38 |          1 | # ?-------- Startup problem on p8-rhel6-bintar-debug in Feb, logs not available, probably env
 
| rpl.rpl_binlog_index                              | 2016-04-22 14:21:05 |          6 | # UNSTABLE: MDEV-9501
 
| rpl.rpl_checksum_cache                            | 2016-06-22 18:56:04 |          2 | # UNSTABLE: MDEV-10626
 
| rpl.rpl_circular_for_4_hosts                      | 2016-06-23 22:36:13 |         14 | # UNSTABLE: MDEV-10627
 
| rpl.rpl_deadlock_innodb                           | 2016-03-01 08:56:18 |          1 | # ?-------- Startup problem on p8-rhel6-bintar-debug in March, logs not available, probably env
 
| rpl.rpl_err_ignoredtable                          | 2016-03-01 08:56:18 |          1 | # ?-------- Startup problem on p8-rhel6-bintar-debug in March, logs not available, probably env
 
| rpl.rpl_gtid_basic                                | 2016-02-17 06:16:14 |          1 | # ?-------- Startup problem on p8-rhel6-bintar-debug in March, logs not available, probably env
 
| rpl.rpl_gtid_crash                                | 2016-04-27 00:59:10 |          2 | # UNSTABLE: MDEV-9501
 
| rpl.rpl_gtid_master_promote                       | 2016-06-23 01:09:50 |          2 | # UNSTABLE: MDEV-10628
 
| rpl.rpl_gtid_stop_start                           | 2016-04-27 14:48:58 |          2 | # UNSTABLE: MDEV-10629
 
| rpl.rpl_gtid_until                                | 2016-06-16 12:40:24 |          1 | # UNSTABLE: MDEV-10625
 
| rpl.rpl_innodb_bug30888                           | 2016-04-23 00:28:07 |          1 | # ?-------- Startup problem on p8-rhel6-bintar-debug in April, logs not available, probably env
 
| rpl.rpl_mdev6020                                  | 2016-08-17 10:05:10 |        160 | # UNSTABLE: MDEV-1630
 
| rpl.rpl_mdev6386                                  | 2016-08-14 11:01:27 |          5 | # UNSTABLE: MDEV-10631
 
| rpl.rpl_mixed_ddl_dml                             | 2016-03-10 11:24:28 |          1 | # ?-------- Startup problem on p8-rhel6-bintar-debug in March, logs not available, probably env
 
| rpl.rpl_parallel                                  | 2016-08-14 11:16:42 |         28 | # UNSTABLE: MDEV-10632
 
| rpl.rpl_parallel_multilevel2                      | 2016-04-21 18:29:38 |          1 | # ?-------- Startup problem on p8-rhel6-bintar-debug in April, logs not available, probably env
 
| rpl.rpl_parallel_no_log_slave_updates             | 2016-04-26 12:34:22 |          1 | # ?-------- Startup problem on p8-rhel6-bintar-debug in April, logs not available, probably env
 
| rpl.rpl_parallel_partition                        | 2016-02-16 11:01:33 |          1 | # ?-------- Startup problem on p8-rhel6-bintar-debug in February, logs not available, probably env
 
| rpl.rpl_parallel_temptable                        | 2016-03-21 21:12:59 |          1 | # UNSTABLE: MDEV-10356
 
| rpl.rpl_rotate_purge_deadlock                     | 2016-04-23 00:28:07 |          1 | # ?-------- Startup problem on p8-rhel6-bintar-debug in February, logs not available, probably env
 
| rpl.rpl_row_basic_3innodb                         | 2016-04-23 00:28:07 |          1 | # ?-------- Startup problem on p8-rhel6-bintar-debug in February, logs not available, probably env
 
| rpl.rpl_row_drop_create_temp_table                | 2016-08-14 16:32:49 |         37 | # UNSTABLE: MDEV-10626
 
| rpl.rpl_row_find_row                              | 2016-03-16 11:53:05 |          1 | # ?-------- Startup problem on p8-rhel6-bintar-debug in March, logs not available, probably env
 
| rpl.rpl_semi_sync                                 | 2016-04-26 17:29:15 |          1 | # ?-------- Startup problem on p8-rhel6-bintar-debug in April, logs not available, probably env
 
| rpl.rpl_semi_sync_uninstall_plugin                | 2016-04-21 11:40:17 |          1 | # UNSTABLE: MDEV-7140
 
| rpl.rpl_server_id2                                | 2016-03-21 21:12:59 |          1 | # ?-------- Startup problem on p8-rhel6-bintar-debug in March, logs not available, probably env
 
| rpl.rpl_slave_grp_exec                            | 2016-04-26 16:36:38 |          4 | # UNSTABLE: MDEV-10514
 
| rpl.rpl_sp_effects                                | 2016-02-11 11:40:20 |          1 | # ?-------- Startup problem on p8-rhel6-bintar-debug in February, logs not available, probably env
 
| rpl.rpl_stm_maria                                 | 2016-02-16 11:01:33 |          1 | # ?-------- Startup problem on p8-rhel6-bintar-debug in February, logs not available, probably env
 
| rpl.rpl_stm_start_stop_slave                      | 2016-04-26 17:29:15 |          2 | # ?-------- Startup problem on p8-rhel6-bintar-debug in April, logs not available, probably env
 
| rpl.rpl_stm_user_variables                        | 2016-04-26 22:37:21 |          1 | # ?-------- Startup problem on p8-rhel6-bintar-debug in April, logs not available, probably env
 
| rpl.rpl_switch_stm_row_mixed                      | 2016-04-26 15:40:50 |          2 | # UNSTABLE: MDEV-10611
 
| rpl.rpl_sync                                      | 2016-03-24 13:59:57 |          4 | # UNSTABLE: MDEV-10633
 
| rpl.rpl_temporary_error2                          | 2016-06-22 04:38:18 |          1 | # UNSTABLE: MDEV-10634
 
| rpl.rpl_temp_table_mix_row                        | 2016-03-16 11:53:05 |          1 | # ?-------- Startup problem on p8-rhel6-bintar-debug in March, logs not available, probably env
 
| rpl.sec_behind_master-5114                        | 2016-08-17 06:28:30 |         15 | # UNSTABLE: MDEV-8518
 
| spider/bg.direct_aggregate                        | 2016-04-27 21:10:43 |          1 | # UNSTABLE: MDEV-7098
 
| spider/bg.direct_aggregate_part                   | 2016-03-25 15:28:37 |          1 | # UNSTABLE: MDEV-7098
 
| spider/bg.ha                                      | 2016-04-22 01:22:33 |          4 | # UNSTABLE: MDEV-7914
 
| spider/bg.ha_part                                 | 2016-04-27 15:36:09 |          2 | # UNSTABLE: MDEV-9329
 
| spider/bg.spider_fixes                            | 2016-06-23 16:38:01 |          3 | # UNSTABLE: MDEV-7098
 
| storage_engine-myisam.alter_table_online          | 2016-05-27 22:18:24 |         10 | # --------- Fixed after the last failure (f9d453e893c)
 
| stress.ddl_innodb                                 | 2016-08-14 21:46:38 |         50 | # UNSTABLE: MDEV-10635
 
| sys_vars.innodb_buffer_pool_load_now_basic        | 2016-03-24 15:36:12 |          5 | # --------- Fixed after the last failure (MDEV-9713)
 
| tokudb.cluster_filter_unpack_varchar              | 2016-04-23 09:27:01 |          1 | # UNSTABLE: MDEV-10636
 
| tokudb_bugs.checkpoint_lock                       | 2016-03-11 21:44:42 |         16 | # UNSTABLE: MDEV-10637
 
| tokudb_bugs.checkpoint_lock_3                     | 2016-03-11 21:44:42 |         16 | # UNSTABLE: MDEV-10637
 
| unit.ma_test_loghandler                           | 2016-02-19 18:06:00 |          1 | # UNSTABLE: MDEV-10638
 
| vcol.not_supported                                | 2016-04-26 16:02:19 |          2 | # UNSTABLE: MDEV-10639
 
| vcol.vcol_keys_innodb                             | 2016-04-26 15:40:50 |          1 | # UNSTABLE: MDEV-10639
 
+---------------------------------------------------+---------------------+------------+

Comment by Elena Stepanova [ 2016-08-21 ]

Tests from Debian "unstable" files (https://anonscm.debian.org/cgit/pkg-mysql/mariadb-10.0.git/tree/debian)

rpl.rpl_skip_replication MDEV-9268 marked as unstable
main.mysqltest MDEV-9269 marked as unstable
mroonga/storage.column_datetime_32bit_2038 MDEV-9270 marked as unstable
mroonga/storage.column_datetime_32bit_before_unix_epoch MDEV-9270 marked as unstable
mroonga/storage.column_datetime_32bit_max MDEV-9270 marked as unstable
mroonga/storage.column_datetime_32bit_out_of_range MDEV-9270 marked as unstable
mroonga/storage.index_multiple_column_unique_date_32bit_equal MDEV-9270 marked as unstable
mroonga/storage.index_multiple_column_unique_date_order_32bit_desc MDEV-9270 marked as unstable
main.events_2 MDEV-9299 Fixed in 10.0.25, not marking as unstable
main.delayed MDEV-9679 Fixed in 10.0.25, not marking as unstable
sys_vars.innodb_buffer_pool_load_now_basic MDEV-9713 Fixed in 10.0.25, not marking as unstable
federated.federated_partition MDEV-8691 marked as unstable
federated.federated_transactions MDEV-8691 marked as unstable
federated.federated_innodb MDEV-8691 marked as unstable
multi_source.gtid MDEV-8691 marked as unstable
multi_source.multisource MDEV-8691 marked as unstable
rpl.rpl_auto_increment MDEV-8691 marked as unstable
rpl.rpl_invoked_features MDEV-8691 marked as unstable
rpl_auto_increment_bug45679 MDEV-8691 marked as unstable
spider.basic_sql MDEV-9329 marked as unstable
spider.basic_sql_part MDEV-9329 marked as unstable
spider/bg.ha MDEV-9329 marked as unstable
spider/bg.ha_part MDEV-9329 marked as unstable
spider/bg.vp_fixes MDEV-9329 marked as unstable
spider.spider_fixes MDEV-9329 marked as unstable
spider.spider3_fixes MDEV-9329 marked as unstable
spider.spider3_fixes_part MDEV-9329 marked as unstable
spider.direct_update MDEV-9329 marked as unstable
spider.function MDEV-9329 marked as unstable
spider.direct_aggregate MDEV-9329 marked as unstable
spider.direct_aggregate_part MDEV-9329 marked as unstable
sys_vars.autocommit_func2 MDEV-9329 marked as unstable
rpl.rpl_row_sp001 MDEV-9329 marked as unstable
rpl.rpl_insert MDEV-9329 marked as unstable
rpl.rpl_insert_delayed MDEV-9329 marked as unstable
Comment by Elena Stepanova [ 2016-08-21 ]

all TokuDB tests marked as unstable due to MDEV-9891 (massive crashes on shutdown).

Comment by Elena Stepanova [ 2016-08-21 ]

all Spider tests marked as unstable due to MDEV-9329 (not enough memory to run them). I'm not sure what we can do about it, except for wait till distributions upgrade the build machines.

Comment by Elena Stepanova [ 2016-08-21 ]

MDEV-8691, MDEV-9152 – cannot do anything if all tests fail.

Comment by Elena Stepanova [ 2016-08-22 ]

perfschema.func_file_io and perfschema.func_mutex - marked as unstable due to MDEV-5708.

Comment by Elena Stepanova [ 2016-08-24 ]

Test counts and timing for disabled vs the normal run on my local machine:

Normal run

 
perl ./mtr --parallel=4 --mem  --force --max-test-fail=0
 
...
 
The servers were restarted 1235 times
 
Spent 9257.469 of 3441 seconds executing testcases
 
 
 
Completed: All 5316 tests were successful.
 
 
 
Errors/warnings were found in logfiles during server shutdown after running the
 
following sequence(s) of tests:
 
    tokudb.background_job_manager
 
257 tests were skipped, 168 by the test itself.

With the skip list

 
perl ./mtr --parallel=4 --mem --skip-test-list=./unstable-tests --force --max-test-fail=0
 
...
 
The servers were restarted 996 times
 
Spent 6180.735 of 2408 seconds executing testcases
 
 
 
Completed: All 4490 tests were successful.
 
 
 
201 tests were skipped, 147 by the test itself.

By far the most part of the disabled tests are tokudb.* – as percomment above, they all are currently disabled due to MDEV-9891. It needs to be investigated and fixed a.s.a.p.

Comment by Elena Stepanova [ 2016-08-24 ]

https://github.com/MariaDB/server/commit/5bbe929d706e26cb3f9b291da6009526a17b1545

Note: For 10.1 the list needs to be updated. I won't spawn a separate MDEV item for that, because it will have to become a part of the release process anyway.

Comment by Elena Stepanova [ 2016-08-27 ]

Re-opening to (initially) update the list for 10.1

Failures from buildbot starting from May 01, 2016:

main.sp-threads 2016-05-01 20:33:04 1 Fixed by MDEV-9970
main.partition_innodb_plugin 2016-05-03 20:16:30 1 Fixed by MDEV-9975
sys_vars.innodb_buffer_pool_load_now_basic 2016-05-04 02:37:27 3 Fixed by MDEV-9713
connect.dir 2016-05-05 10:06:04 5 Fixed by MDEV-10034
binlog.binlog_mdev717 2016-05-05 14:05:47 48 Fixed by 19c4d22a1e
connect.dbf 2016-05-06 02:00:19 6 Fixed by MDEV-10034
sys_vars.autocommit_func2 2016-05-07 01:38:56 17 Fixed by MDEV-10034
sys_vars.autocommit_func3 2016-05-07 01:38:56 25 Fixed by MDEV-10034
maria.small_blocksize 2016-05-07 01:38:56 20 Fixed by MDEV-10034
main.show_explain 2016-05-08 12:30:28 1 Added to unstable (MDEV-10674)
main.join_outer_jcl6 2016-05-12 07:06:13 40 Fixed by MDEV-10072
maria.encrypt-wrong-key 2016-05-18 12:03:25 7 "bytes lost", disappeared after system upgrade
main.ctype_utf16le 2016-05-18 12:12:19 1 Added to unstable (MDEV-10675)
sys_vars.keep_files_on_create_basic 2016-05-18 12:12:19 1 Added to untsable (MDEV-10676)
encryption.innodb-page_encryption_compression 2016-05-18 14:45:00 42 "bytes lost", disappeared after system upgrade
encryption.innodb-discard-import 2016-05-18 14:45:00 42 "bytes lost", disappeared after system upgrade
encryption.innochecksum 2016-05-18 14:45:00 42 "bytes lost", disappeared after system upgrade
encryption.innodb_page_encryption_key_change 2016-05-18 14:45:00 42 "bytes lost", disappeared after system upgrade
encryption.innodb-page_encryption_log_encryption 2016-05-18 14:45:00 42 "bytes lost", disappeared after system upgrade
encryption.innodb-bad-key-change 2016-05-18 14:45:00 42 "bytes lost", disappeared after system upgrade
sys_vars.log_slow_admin_statements_func 2016-05-30 18:30:54 1 disk space problem
perfschema.setup_objects 2016-05-30 18:30:54 1 disk space problem
main.range_vs_index_merge 2016-05-30 18:30:54 1 disk space problem
perfschema.socket_summary_by_instance_func 2016-05-30 18:30:54 1 disk space problem
maria.maria-recovery-rtree-ft 2016-05-30 18:30:54 1 disk space problem
main.alter_table_mdev539_maria 2016-05-30 18:30:54 1 disk space problem
encryption.innodb-log-encrypt-crash 2016-05-31 18:22:42 19 Fixed by MDEV-9865
rpl.rpl_binlog_index 2016-06-01 01:42:45 1 Already unstable for 10.0
main.events_restart 2016-06-01 11:09:30 2 Fixed by MDEV-10043
perfschema.threads_mysql 2016-06-02 15:38:34 1 Added to unstable (MDEV-10677)
tokudb.cluster_filter_hidden 2016-06-03 05:21:48 1 Added to unstable (MDEV-10678)
spider/bg.direct_aggregate_part 2016-06-03 12:30:43 2 Already unstable for 10.0
main.set_statement 2016-06-03 15:00:22 6 Fixed by 9ff93656a
main.status 2016-06-03 15:01:36 1 Added to unstable (MDEV-8510)
rpl.rpl_init_slave 2016-06-19 16:02:59 1 Server failed to start on old p8 box, not happens after switching slaves
plugins.feedback_plugin_send 2016-06-19 22:33:45 28 Already unstable for 10.0
perfschema.setup_actors 2016-06-19 22:33:45 1 Added to unstable (MDEV-10679)
innodb.innodb_corrupt_bit 2016-06-22 01:41:49 2 Fixed by MDEV-7139
rpl.rpl_gtid_mdev9033 2016-06-23 07:44:16 1 Added to unstable (MDEV-10680)
innodb.innodb_ctype_ldml 2016-06-24 00:34:52 50 Fixed by 6f6692008
funcs_2.memory_charset 2016-06-24 17:34:47 1 Added to unstable (MDEV-10290)
tokudb.cluster_filter 2016-06-27 11:23:46 1 Added to unstable (MDEV-10678)
rpl.rpl_show_slave_hosts 2016-06-27 15:32:17 1 Added to unstable (MDEV-10681)
encryption.innodb_onlinealter_encryption 2016-06-28 06:47:50 3 Added to unstable (MDEV-10099)
innodb.innodb_stats 2016-06-28 07:15:53 1 Added to unstable (MDEV-10682)
encryption.innodb_encryption_discard_import 2016-06-28 20:50:11 2 Added to unstable (MDEV-9099)
sys_vars.innodb_monitor_enable_basic 2016-06-28 21:57:03 1 disk space problem
sys_vars.innodb_mirrored_log_groups_basic 2016-06-28 21:57:03 1 disk space problem
sys_vars.innodb_max_purge_lag_basic 2016-06-28 21:57:03 1 disk space problem
sys_vars.innodb_max_dirty_pages_pct_func 2016-06-28 21:57:03 1 disk space problem
sys_vars.innodb_monitor_reset_all_basic 2016-06-28 21:57:03 1 disk space problem
sys_vars.innodb_monitor_disable_basic 2016-06-28 21:57:03 1 disk space problem
sys_vars.innodb_max_purge_lag_delay_basic 2016-06-28 21:57:03 1 disk space problem
sys_vars.innodb_max_dirty_pages_pct_lwm_basic 2016-06-28 21:57:03 1 disk space problem
main.multi_update2 2016-06-28 21:57:03 1 disk space problem
rpl.rpl_sporadic_master 2016-06-29 07:51:01 1 problem with valgrind suppression syntax
federated.error_on_close-8313 2016-06-29 07:51:01 1 problem with valgrind suppression syntax
multi_source.simple 2016-06-29 07:51:01 1 problem with valgrind suppression syntax
federated.federated_transactions 2016-06-29 07:51:01 2 Already unstable for 10.0
federated.federated_maybe_16324629 2016-06-29 07:51:01 1 problem with valgrind suppression syntax
federated.federated_debug 2016-06-29 07:51:01 1 problem with valgrind suppression syntax
federated.federated_bug_35333 2016-06-29 07:51:01 1 problem with valgrind suppression syntax
federated.federated_bug_25714 2016-06-29 07:51:01 1 problem with valgrind suppression syntax
rpl.rpl_loaddata_m 2016-06-29 07:51:01 1 problem with valgrind suppression syntax
federated.federated_archive 2016-06-29 07:51:01 1 problem with valgrind suppression syntax
rpl.rpl_sync 2016-06-29 07:51:01 2 Already unstable for 10.0
federated.assisted_discovery 2016-06-29 07:51:01 1 problem with valgrind suppression syntax
rpl.rpl_create_database 2016-06-29 07:51:01 3 problem with valgrind suppression syntax
federated.federated_partition 2016-06-29 07:51:01 2 Already unstable for 10.0
federated.federated_innodb 2016-06-29 07:51:01 2 Already unstable for 10.0
rpl.rpl_checksum 2016-06-29 07:51:01 1 problem with valgrind suppression syntax
federated.federated_bug_585688 2016-06-29 07:51:01 1 problem with valgrind suppression syntax
federated.federated_bug_32426 2016-06-29 07:51:01 1 problem with valgrind suppression syntax
federated.federated_bug_13118 2016-06-29 07:51:01 1 problem with valgrind suppression syntax
rpl.rpl_timezone 2016-06-29 07:51:01 1 problem with valgrind suppression syntax
rpl.rpl_row_img_eng_noblob 2016-06-30 03:31:42 1 disk space problem
rpl.rpl_gtid_basic 2016-06-30 05:49:51 1 Added to unstable (MDEV-10681)
rpl.rpl_circular_for_4_hosts 2016-06-30 05:49:51 4 Already unstable for 10.0
rpl.rpl_row_img_eng_min 2016-07-01 16:36:29 9 Timeouts on old P8, fixed by upgrade
rpl.rpl_heartbeat_basic 2016-07-01 16:36:29 1 Timeouts on old P8, fixed by upgrade
rpl.rpl_row_img_blobs 2016-07-01 16:36:29 11 Timeouts on old P8, fixed by upgrade
rpl.rpl_multi_update4 2016-07-03 00:50:28 1 disk space problem
rpl.rpl_parallel_retry 2016-07-03 00:50:28 1 disk space problem
main.order_by_optimizer_innodb 2016-07-03 10:29:06 1 Added to unstable (MDEV-10683)
rpl.rpl_parallel 2016-07-03 10:48:52 14 Already unstable for 10.0
main.analyze_stmt_slow_query_log 2016-07-03 11:06:25 1 Added to unstable (MDEV-7558)
stress.ddl_innodb 2016-07-03 14:20:44 29 Already unstable for 10.0
rpl.rpl_row_drop_create_temp_table 2016-07-03 16:32:25 25 Already unstable for 10.0
rpl.rpl_domain_id_filter_restart 2016-07-05 14:46:36 1 Added to unstable (MDEV-10684)
mroonga/storage.index_multiple_column_unique_date_order_64bit_asc 2016-07-05 19:25:19 27 Fixed in MDEV-9363
mroonga/storage.index_multiple_column_unique_datetime_order_desc 2016-07-05 19:25:19 27 Fixed in MDEV-9363
mroonga/storage.index_multiple_column_unique_date_order_64bit_desc 2016-07-05 19:25:19 27 Fixed in MDEV-9363
mroonga/storage.index_multiple_column_unique_date_64bit_equal 2016-07-05 19:25:19 27 Fixed in MDEV-9363
mroonga/storage.index_multiple_column_unique_datetime_order_asc 2016-07-05 19:25:19 27 Fixed in MDEV-9363
mroonga/storage.column_datetime_64bit_before_unix_epoch 2016-07-05 19:25:19 27 Fixed in MDEV-9363
perfschema.cnf_option 2016-07-06 04:11:34 2 disk space problem
rpl.rpl_multi_engine 2016-07-06 04:11:34 1 disk space problem
rpl.rpl_gtid_crash 2016-07-06 04:11:34 3 disk space problem
rpl.rpl_domain_id_filter_parallel 2016-07-06 04:11:34 3 disk space problem
rpl.rpl_drop_db_fail 2016-07-06 04:11:34 2 disk space problem
rpl.rpl_skip_error 2016-07-06 04:11:34 2 disk space problem
rpl.rpl_loaddata_s 2016-07-06 04:11:34 1 disk space problem
rpl.rpl_ddl 2016-07-06 04:11:34 2 disk space problem
rpl.rpl_domain_id_filter_io_crash 2016-07-06 04:11:34 2 disk space problem
rpl.rpl_semi_sync_event 2016-07-06 04:11:34 3 disk space problem
rpl.rpl_insert_id_pk 2016-07-06 04:11:34 1 disk space problem
rpl.rpl_corruption 2016-07-06 04:11:34 5 disk space problem
rpl.kill_hard-6290 2016-07-06 04:11:34 1 disk space problem
rpl.rpl_trigger 2016-07-06 04:11:34 1 disk space problem
rpl.rpl_mdev6020 2016-07-06 04:11:34 56 disk space problem, unstable for 10.0
rpl.rpl_gtid_until 2016-07-06 04:11:34 1 disk space problem
rpl.rpl_checksum_cache 2016-07-06 04:11:34 4 disk space problem
rpl.rpl_stop_slave 2016-07-06 04:11:34 1 disk space problem
rpl.rpl_parallel_partition 2016-07-06 04:11:34 4 disk space problem
rpl.rpl_gtid_sort 2016-07-06 04:11:34 1 disk space problem
rpl.rpl_bug26395 2016-07-06 04:11:34 2 disk space problem
rpl.rpl_start_stop_slave 2016-07-06 04:11:34 1 disk space problem
rpl.rpl_mixed_binlog_max_cache_size 2016-07-06 04:11:34 2 disk space problem
rpl.rpl_gtid_mdev4484 2016-07-06 04:11:34 3 disk space problem
rpl.rpl_binlog_errors 2016-07-06 04:11:34 3 disk space problem
rpl.rpl_slave_load_tmpdir_not_exist 2016-07-06 04:11:34 1 disk space problem
rpl.rpl_gtid_errorhandling 2016-07-06 04:11:34 2 disk space problem
rpl.rpl_auto_increment_bug45679 2016-07-06 04:11:34 3 disk space problem
rpl.rpl_server_id_ignore 2016-07-06 04:11:34 1 disk space problem
rpl.rpl_innodb_mixed_dml 2016-07-06 04:11:34 2 disk space problem
rpl.rpl_failed_optimize 2016-07-06 04:11:34 2 disk space problem
rpl.create_or_replace2 2016-07-06 04:11:34 4 disk space problem
rpl.rpl_relay_space_innodb 2016-07-06 04:11:34 1 disk space problem
federated.federated 2016-07-06 04:11:34 2 disk space problem
rpl.rpl_innodb_bug30888 2016-07-06 04:11:34 4 disk space problem
rpl.rpl_drop_temp 2016-07-06 04:11:34 1 disk space problem
rpl.rpl_special_charset 2016-07-06 04:11:34 2 disk space problem
rpl.rpl_mdev359 2016-07-06 04:11:34 1 disk space problem
rpl.rpl_deadlock_innodb 2016-07-06 04:11:34 3 disk space problem
rpl.rpl_domain_id_filter_master_crash 2016-07-06 04:11:34 1 disk space problem
rpl.rpl_semi_sync_event_after_sync 2016-07-06 04:11:34 5 disk space problem
rpl.rpl_insert_ignore 2016-07-06 04:11:34 1 disk space problem
rpl.rpl_domain_id_filter 2016-07-06 04:11:34 1 disk space problem
rpl.rpl_unsafe_statements 2016-07-06 04:11:34 1 disk space problem
rpl.rpl_semi_sync 2016-07-06 04:11:34 4 disk space problem
rpl.rpl_insert_id 2016-07-06 04:11:34 1 disk space problem
rpl.rpl_commit_after_flush 2016-07-06 04:11:34 3 disk space problem
rpl.rpl_temp_table_mix_row 2016-07-06 04:11:34 1 disk space problem
rpl.rpl_partition_innodb 2016-07-06 04:11:34 3 disk space problem
rpl.rpl_gtid_startpos 2016-07-06 04:11:34 1 disk space problem
rpl.rpl_stm_innodb 2016-07-06 04:11:34 1 disk space problem
rpl.rpl_mixed_implicit_commit_binlog 2016-07-06 04:11:34 2 disk space problem
rpl.rpl_gtid_reconnect 2016-07-06 04:11:34 1 disk space problem
rpl.rpl_binlog_grant 2016-07-06 04:11:34 2 disk space problem
rpl.rpl_slave_skip 2016-07-06 04:11:34 1 disk space problem
rpl.rpl_mdev6386 2016-07-06 04:11:34 3 disk space problem
rpl.rpl_gtid_ignored 2016-07-06 04:11:34 1 disk space problem
rpl.rpl_auto_increment_update_failure 2016-07-06 04:11:34 3 disk space problem
rpl.rpl_set_null_innodb 2016-07-06 04:11:34 1 disk space problem
rpl.rpl_invoked_features 2016-07-06 04:11:34 3 disk space problem
rpl.rpl_foreign_key_innodb 2016-07-06 04:11:34 2 disk space problem
rpl.rpl_auto_increment 2016-07-06 04:11:34 9 disk space problem
rpl.rpl_rewrt_db 2016-07-06 04:11:34 1 disk space problem
federated.federatedx 2016-07-06 04:11:34 2 disk space problem
rpl.rpl_innodb_mixed_ddl 2016-07-06 04:11:34 2 disk space problem
rpl.rpl_dual_pos_advance 2016-07-06 04:11:34 2 disk space problem
rpl.rpl_parallel_temptable 2016-07-09 06:35:40 2 Already unstable for 10.0
encryption.innodb_encryption_filekeys 2016-07-12 22:19:41 5 Added to unstable (MDEV-9062)
main.innodb_ext_key 2016-07-13 19:49:06 47 Fixed in 12ac3ee11
multi_source.status_vars 2016-07-13 20:15:15 5 Already unstable for 10.0
sys_vars.sysvars_innodb 2016-07-14 04:32:16 50 Added to unstable (MDEV-6958)
sys_vars.sysvars_server_embedded 2016-07-14 04:32:16 12 Added to unstable (MDEV-6958)
spider/bg.direct_aggregate 2016-07-18 19:05:43 1 Already unstable for 10.0
innodb.xa_recovery 2016-07-18 19:26:24 1 Added to unstable (MDEV-10685)
main.pool_of_threads 2016-07-18 19:29:14 1 Already unstable for 10.0
encryption.innodb-bad-key-shutdown 2016-07-19 23:58:32 85 Added to unstable (MDEV-9105)
perfschema.socket_summary_by_event_name_func 2016-07-20 00:07:03 1 Already unstable for 10.0
rpl.rpl_slave_grp_exec 2016-07-20 00:23:03 7 Already unstable for 10.0
spider/bg.ha_part 2016-07-20 03:21:34 3 Already unstable for 10.0
sys_vars.sysvars_wsrep 2016-07-27 22:30:25 3 Fixed in MDEV-10429
innodb_zip.innodb_prefix_index_liftedlimit 2016-07-27 22:30:25 1 Added to unstable (MDEV-10686)
main.derived 2016-07-29 16:34:15 93 Fixed in MDEV-10470
innodb.innodb_blob_truncate 2016-08-01 12:39:15 3 Added to unstable (MDEV-10377)
binlog.binlog_xa_recover 2016-08-01 12:42:12 2 Already unstable for 10.0
main.index_merge_innodb 2016-08-01 12:57:46 3 Already unstable for 10.0
connect.alter 2016-08-01 13:19:34 44 MDEV-10616 - Connect does not load on Sparc
connect.secure_file_priv 2016-08-01 13:19:34 44 MDEV-10616 - Connect does not load on Sparc
connect.part_file 2016-08-01 13:19:34 44 MDEV-10616 - Connect does not load on Sparc
connect.drop-open-error 2016-08-01 13:19:34 32 MDEV-10616 - Connect does not load on Sparc
connect.csv 2016-08-01 13:19:34 36 MDEV-10616 - Connect does not load on Sparc
connect.alter_xml 2016-08-01 13:19:34 44 MDEV-10616 - Connect does not load on Sparc
connect.part_table 2016-08-01 13:19:34 24 MDEV-10616 - Connect does not load on Sparc
connect.json 2016-08-01 13:19:34 24 MDEV-10616 - Connect does not load on Sparc
connect.datest 2016-08-01 13:19:34 31 MDEV-10616 - Connect does not load on Sparc
connect.bin 2016-08-01 13:19:34 44 MDEV-10616 - Connect does not load on Sparc
innodb.innodb_sys_semaphore_waits 2016-08-01 16:24:00 1 Added to unstable (MDEV-10331)
rpl.rpl_semi_sync_uninstall_plugin 2016-08-02 00:21:54 1 Already unstable for 10.0
main.innodb_mysql_lock 2016-08-02 00:49:32 2 Already unstable for 10.0
spider/bg.ha 2016-08-02 19:46:02 6 Already unstable for 10.0
main.mysqldump 2016-08-02 23:19:48 8 Already unstable for 10.0
rpl.sec_behind_master-5114 2016-08-03 04:24:47 10 Already unstable for 10.0
sys_vars.innodb_fatal_semaphore_wait_threshold 2016-08-03 05:08:51 3 Added to unstable (MDEV-10513)
spider/bg.spider_fixes 2016-08-04 06:30:30 1 Already unstable for 10.0
rpl.rpl_parallel_optimistic 2016-08-05 00:18:51 2 Added to unstable (MDEV-10511)
main.type_datetime_hires 2016-08-16 16:59:25 1 Added to unstable (MDEV-10687)
rpl.rpl_row_log_innodb 2016-08-17 13:26:16 1 Added to unstable (MDEV-10688)
main.subselect_innodb 2016-08-17 17:33:11 1 Already unstable for 10.0
main.stat_tables_par_innodb 2016-08-18 11:13:46 1 Already unstable for 10.0
innodb.innodb-wl5522-1 2016-08-18 11:18:23 20 Added to unstable (MDEV-9105)
innodb.innodb-bug-14084530 2016-08-18 11:18:23 20 Added to unstable (MDEV-9105)
innodb.innodb-wl5522-zip 2016-08-18 11:18:23 18 Added to unstable (MDEV-9105)
mroonga/storage.index_multiple_column_unique_datetime_index_read 2016-08-18 11:18:23 36 Added to unstable (MDEV-8643)
innodb.innodb-page_compression_zip 2016-08-18 11:18:23 1 Added to unstable (MDEV-10641)
encryption.innodb-page_encryption 2016-08-18 11:18:23 45 Added to unstable (MDEV-10641)
innodb.innodb-bug-14068765 2016-08-18 11:18:23 58 Added to unstable (MDEV-9105)
innodb.innodb-wl5522 2016-08-18 11:18:23 20 Changed reason in unstable from 'modified' to MDEV-9105
main.information_schema_stats 2016-08-25 05:01:35 6 Fixed in MDEV-10428
connect.tbl 2016-08-26 00:53:43 44 Already unstable for 10.0
multi_source.gtid 2016-08-26 01:23:55 9 Already unstable for 10.0
multi_source.multisource 2016-08-26 01:23:55 6 Already unstable for 10.0
encryption.innodb_first_page 2016-08-26 01:34:46 47 Added to untsable (MDEV-10689)
wsrep.pool_of_threads 2016-08-26 01:37:36 31 All wsrep tests added to unstable (MDEV-10041)
wsrep.mdev_7798 2016-08-26 01:37:36 12 All wsrep tests added to unstable (MDEV-10041)
wsrep.binlog_format 2016-08-26 01:37:36 18 All wsrep tests added to unstable (MDEV-10041)
rpl.rpl_relayrotate 2016-08-26 12:32:54 5 Environmental (enforced open-files-limit from command line)
multi_source.info_logs 2016-08-26 20:24:42 8 Added to unstable (MDEV-10042)
multi_source.relaylog_events 2016-08-26 20:24:42 5 Environmental (enforced open-files-limit from command line)
multi_source.load_data 2016-08-26 20:24:42 6 Environmental (enforced open-files-limit from command line)
multi_source.gtid_ignore_duplicates 2016-08-26 20:24:42 10 Environmental (enforced open-files-limit from command line)
multi_source.skip_counter 2016-08-26 20:24:42 3 Environmental (enforced open-files-limit from command line)
multi_source.reset_slave 2016-08-26 20:24:42 5 Added to unstable (MDEV-10690)
multi_source.syntax 2016-08-26 20:24:42 18 Environmental (enforced open-files-limit from command line)
wsrep.mdev_6832 2016-08-26 21:42:46 20 All wsrep tests added to unstable (MDEV-10041)
mroonga/storage.repair_table_no_index_file 2016-08-26 21:43:31 47 Added to unstable (MDEV-9364)
encryption.create_or_replace 2016-08-26 21:56:37 19 Added to unstable (MDEV-9359)

231 rows in set (0.11 sec)

Comment by Elena Stepanova [ 2016-08-27 ]

I don't see a Debian-maintained 10.1 tree similar to 10.0 (https://anonscm.debian.org/cgit/pkg-mysql/mariadb-10.0.git/tree/debian), so there are no 10.1-specific unstable lists from Debian to process.

Comment by Elena Stepanova [ 2016-08-27 ]

10.1 part:
https://github.com/MariaDB/server/commit/7d115e73b8a0a9f5bc8160a0bcf966ea2283a829

Comment by Otto Kekäläinen [ 2020-10-11 ]

I removed many patches and test skiplists in https://salsa.debian.org/mariadb-team/mariadb-10.5/-/commit/bde2cf481fa48a0dd85b9ad40e27ad5005ad1122 after testing that many of them are not needed anymore.

Validation:

For a couple of failing tests I had to introduce new skiplists for alpha, hppa and powerpc: https://salsa.debian.org/mariadb-team/mariadb-10.5/-/tree/master/debian

Comment by Otto Kekäläinen [ 2021-05-30 ]

About the design and custom scripts: The CMake build already makes a Makefile that has targets to run `make test` and `make test-force`. What about using this mechanism instead of creating a new fully custom script? The purpose of the tests is to validate that the build was successful, thus the Makefile will be around immediately after the run, and one can run `make test` etc at that point right after the `make package` or `make all` has run.

Generated at Thu Feb 08 07:43:30 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.