Details
-
Bug
-
Status: Open (View Workflow)
-
Major
-
Resolution: Unresolved
-
None
-
None
-
None
-
BuildBot runs on any PR in github.com/MariaDB/server
Description
In the PRs in the past two weeks, I've encountered several unstable test (fails on one run, pass on another)
Since there is no way for external contributors to trigger specific tests re-runs, the only way to do the re-run is 'git commit --amend' + 'git push --force' to re-run all of the tests.
I suggest broader investigation, analyzing all most commonly failing tests in BB during last months, and fixing those tests; to save both BB resources and contributor nerves 
—
I tired to analyse all 'failed test debug-ps (failure)' statuses on 'amd64-ubuntu-2204-debug-ps' builder in the last two weeks.
Builder 534: amd64-ubuntu-2204-debug-ps |
Builds analyzed: 166 with "failed test debug-ps (failure)" out of 600 |
|
|
# | Failing Test | Count | % of 166
|
--|---------------------------------------|-------|--------
|
1 | innodb.skip_locked_nowait | 40 | 24.1%
|
2 | main.xa | 15 | 9.0%
|
3 | binlog.global_temporary_table | 11 | 6.6%
|
4 | main.global_temporary_table | 11 | 6.6%
|
5 | rpl.global_temporary_table | 11 | 6.6%
|
6 | events.events_restart | 10 | 6.0%
|
7 | federated.federatedx | 9 | 5.4%
|
8 | innodb.log_file_overwrite | 9 | 5.4%
|
9 | rpl.rpl_parallel_multi_domain_xa | 8 | 4.8%
|
10 | sys_vars.sysvars_server_notembedded | 7 | 4.2%
|
11 | perfschema.lowercase_fs_off | 6 | 3.6%
|
12 | main.auth_bounded_parsing | 4 | 2.4%
|
13 | main.events_processlist | 4 | 2.4%
|
14 | main.socket_conflict | 4 | 2.4%
|
15 | main.change_master_default | 3 | 1.8%
|
16 | main.deprecated_features | 3 | 1.8%
|
17 | main.resolveip | 3 | 1.8%
|
18 | main.vector2 | 3 | 1.8%
|
19 | federated.federated | 3 | 1.8%
|
20 | federated.federatedx_mdev39196 | 3 | 1.8%
|
21 | plugins.mdev38550 | 3 | 1.8%
|
22 | innodb.foreign_key_errors | 2 | 1.2%
|
23 | main.debug_sync | 2 | 1.2%
|
24 | main.deprecated_sql_mode | 2 | 1.2%
|
25 | main.func_json | 2 | 1.2%
|
26 | main.information_schema | 2 | 1.2%
|
27 | main.mdev_38752 | 2 | 1.2%
|
28 | main.mysql_secure_installation | 2 | 1.2%
|
29 | main.mysqld--help | 2 | 1.2%
|
30 | main.opt_trace | 2 | 1.2%
|
31 | main.sql_mode | 2 | 1.2%
|
32 | main.ansi | 2 | 1.2%
|
33 | main.events_bugs | 2 | 1.2%
|
34 | parts.alter_table | 2 | 1.2%
|
35 | rpl.rpl_row_foreign_key_mdl | 2 | 1.2%
|
36 | spider/bugfix.sql_mode_mariadb | 2 | 1.2% |
37 | spider/bugfix.sql_mode_mysql | 2 | 1.2% |
38 | sys_vars.sql_mode_basic | 2 | 1.2%
|
|
|
99 additional tests failed once each (0.6%).
|
Builder 'amd64-ubuntu-2204-debug'
Builder 949: amd64-ubuntu-2204-debug
|
Builds analyzed: 126 with "failed test debug (failure)" out of 600 |
(1 build had no parseable "Failing test(s):" line) |
|
|
# | Failing Test | Count | % of 126 |
---|-----------------------------------------------|-------|--------
|
1 | main.xa | 18 | 14.3%
|
2 | binlog.global_temporary_table | 10 | 7.9%
|
3 | main.global_temporary_table | 10 | 7.9%
|
4 | rpl.global_temporary_table | 10 | 7.9%
|
5 | federated.federatedx | 9 | 7.1%
|
6 | sys_vars.sysvars_server_notembedded | 7 | 5.6%
|
7 | innodb.log_file_overwrite | 6 | 4.8%
|
8 | events.events_restart | 5 | 4.0%
|
9 | rpl.rpl_parallel_multi_domain_xa | 5 | 4.0%
|
10 | main.auth_bounded_parsing | 4 | 3.2%
|
11 | main.events_processlist | 4 | 3.2%
|
12 | main.socket_conflict | 4 | 3.2%
|
13 | main.debug_sync | 3 | 2.4%
|
14 | main.deprecated_features | 3 | 2.4%
|
15 | main.func_json | 3 | 2.4%
|
16 | main.resolveip | 3 | 2.4%
|
17 | federated.federated | 3 | 2.4%
|
18 | federated.federatedx_mdev39196 | 3 | 2.4%
|
19 | plugins.mdev38550 | 3 | 2.4%
|
20 | rpl.rpl_semisync_ali_issues | 3 | 2.4%
|
21 | binlog_in_engine.rpl_row_basic_3innodb_mfsi | 2 | 1.6%
|
22 | binlog_in_engine.rpl_row_basic_3innodb_misf | 2 | 1.6%
|
23 | innodb.foreign_key_errors | 2 | 1.6%
|
24 | innodb.log_file_size_online | 2 | 1.6%
|
25 | innodb.skip_locked_nowait | 2 | 1.6%
|
26 | main.ansi | 2 | 1.6%
|
27 | main.deprecated_sql_mode | 2 | 1.6%
|
28 | main.events_bugs | 2 | 1.6%
|
29 | main.information_schema | 2 | 1.6%
|
30 | main.lock_sync | 2 | 1.6%
|
31 | main.mdev_38752 | 2 | 1.6%
|
32 | main.mysql_secure_installation | 2 | 1.6%
|
33 | main.mysql_upgrade-28915 | 2 | 1.6%
|
34 | main.mysqld--help | 2 | 1.6%
|
35 | main.opt_trace | 2 | 1.6%
|
36 | main.sp | 2 | 1.6%
|
37 | main.sp-cursor-slow-log | 2 | 1.6%
|
38 | main.sql_mode | 2 | 1.6%
|
39 | mariabackup.incremental_backup | 2 | 1.6%
|
40 | parts.alter_table | 2 | 1.6%
|
41 | perfschema.mdl_func | 2 | 1.6%
|
42 | rpl.rpl_extra_col_slave_rebinlog | 2 | 1.6%
|
43 | rpl.rpl_row_foreign_key_mdl | 2 | 1.6%
|
44 | spider/bugfix.sql_mode_mariadb | 2 | 1.6% |
45 | spider/bugfix.sql_mode_mysql | 2 | 1.6% |
46 | sys_vars.sql_mode_basic | 2 | 1.6%
|
|
|
97 additional tests failed once each (0.8%).
|
Builder 'amd64-msan-clang-20'
Builder 867: amd64-msan-clang-20
|
Builds analyzed: 151 with "failed test msan (failure)" out of 600 |
|
|
# | Failing Test | Count | % of 151 |
---|-------------------------------------------------------|-------|--------
|
1 | main.tmp_space_usage | 24 | 15.9%
|
2 | binlog.global_temporary_table | 11 | 7.3%
|
3 | main.global_temporary_table | 11 | 7.3%
|
4 | rpl.global_temporary_table | 11 | 7.3%
|
5 | main.information_schema | 10 | 6.6%
|
6 | main.sp | 10 | 6.6%
|
7 | main.view | 9 | 6.0%
|
8 | federated.federatedx | 9 | 6.0%
|
9 | sysschema.* (28 tests, all appearing together) | 8 | 5.3%
|
10 | rpl.rpl_row_view01 | 8 | 5.3%
|
11 | main.subselect4 | 8 | 5.3%
|
12 | main.ps_ddl | 8 | 5.3%
|
13 | main.mysqladmin | 8 | 5.3%
|
14 | main.lock_view | 8 | 5.3%
|
15 | main.insert_select | 8 | 5.3%
|
16 | main.information_schema_db | 8 | 5.3%
|
17 | main.group_by | 8 | 5.3%
|
18 | main.func_like | 8 | 5.3%
|
19 | main.explain | 8 | 5.3%
|
20 | main.derived_view | 8 | 5.3%
|
21 | main.ctype_utf8mb4_uca | 8 | 5.3%
|
22 | main.cte_nonrecursive | 8 | 5.3%
|
23 | main.brackets | 8 | 5.3%
|
24 | funcs_1.myisam_views-big | 8 | 5.3%
|
25 | funcs_1.memory_views | 8 | 5.3%
|
26 | funcs_1.innodb_views | 8 | 5.3%
|
27 | compat/oracle.func_trim | 8 | 5.3% |
28 | compat/oracle.func_pad | 8 | 5.3% |
29 | compat/oracle.func_concat | 8 | 5.3% |
30 | compat/oracle.empty_string_literal | 8 | 5.3% |
31 | sys_vars.sysvars_server_notembedded | 7 | 4.6%
|
32 | innodb.skip_locked_nowait | 7 | 4.6%
|
33 | main.xa | 6 | 4.0%
|
34 | pkg_dbms_sql.dbms_sql | 5 | 3.3%
|
35 | innodb.recovery_memory | 4 | 2.6%
|
36 | main.auth_bounded_parsing | 4 | 2.6%
|
37 | main.socket_conflict | 4 | 2.6%
|
38 | plugins.mdev38550 | 4 | 2.6%
|
39 | events.events_restart | 3 | 2.0%
|
40 | federated.federated | 3 | 2.0%
|
41 | federated.federated_server | 3 | 2.0%
|
42 | federated.federatedx_mdev39196 | 3 | 2.0%
|
43 | innodb.bug | 3 | 2.0%
|
44 | main.deprecated_features | 3 | 2.0%
|
45 | main.resolveip | 3 | 2.0%
|
46 | maria.maria-purge | 3 | 2.0%
|
47 | innodb.autoinc_persist | 2 | 1.3%
|
48 | main.analyze_format_json | 2 | 1.3%
|
49 | main.ansi | 2 | 1.3%
|
50 | main.change_master_default | 2 | 1.3%
|
51 | main.deprecated_sql_mode | 2 | 1.3%
|
52 | main.events_bugs | 2 | 1.3%
|
53 | main.func_json | 2 | 1.3%
|
54 | main.mdev_38752 | 2 | 1.3%
|
55 | main.mysql_secure_installation | 2 | 1.3%
|
56 | main.mysqld--help | 2 | 1.3%
|
57 | main.opt_trace | 2 | 1.3%
|
58 | main.sql_mode | 2 | 1.3%
|
59 | parts.alter_table | 2 | 1.3%
|
60 | plugins.server_audit | 2 | 1.3%
|
61 | rpl.rpl_fragment_row_event_main | 2 | 1.3%
|
62 | sys_vars.ft_boolean_syntax_basic | 2 | 1.3%
|
63 | sys_vars.sql_mode_basic | 2 | 1.3%
|
64 | type_xmltype.type_xmltype | 2 | 1.3%
|
65 | type_xmltype.type_xmltype_validation | 2 | 1.3%
|
66 | unit.conc_connection | 2 | 1.3%
|
|
|
85 additional tests failed once each (0.7%).
|
|
Builder 'amd64-fedora-42'
Builder 917: amd64-fedora-42
|
Builds analyzed: 66 with test failure out of 600 |
|
|
# | Failing Test | Count | % of 66 |
---|-------------------------------------------------------|-------|--------
|
1 | main.user_limits | 6 | 9.1%
|
2 | main.events_processlist | 6 | 9.1%
|
3 | main.socket_conflict | 4 | 6.1%
|
4 | main.auth_bounded_parsing | 4 | 6.1%
|
5 | main.vector2 | 3 | 4.5%
|
6 | main.sp | 3 | 4.5%
|
7 | main.resolveip | 3 | 4.5%
|
8 | main.func_json | 3 | 4.5%
|
9 | main.deprecated_features | 3 | 4.5%
|
10 | spider/bugfix.sql_mode_mysql | 2 | 3.0% |
11 | spider/bugfix.sql_mode_mariadb | 2 | 3.0% |
12 | main.vector | 2 | 3.0%
|
13 | main.sql_mode | 2 | 3.0%
|
14 | main.query_cache | 2 | 3.0%
|
15 | main.opt_trace | 2 | 3.0%
|
16 | main.mysql_secure_installation | 2 | 3.0%
|
17 | main.mysql-interactive | 2 | 3.0%
|
18 | main.mysqld--help | 2 | 3.0%
|
19 | main.mdev_38752 | 2 | 3.0%
|
20 | main.information_schema | 2 | 3.0%
|
21 | main.events_bugs | 2 | 3.0%
|
22 | main.deprecated_sql_mode | 2 | 3.0%
|
23 | main.change_master_default | 2 | 3.0%
|
24 | main.ansi | 2 | 3.0%
|
25 | main.analyze_format_json | 2 | 3.0%
|
|
|
64 additional tests failed once each (1.5%).
|
Builder 'aarch64-debian-11'
Builder 313: aarch64-debian-11
|
Builds analyzed: 42 with test failure out of 600 |
|
|
# | Failing Test | Count | % of 42 |
---|-------------------------------------------------------|-------|--------
|
1 | main.vector2 | 4 | 9.5%
|
2 | main.socket_conflict | 4 | 9.5%
|
3 | main.auth_bounded_parsing | 4 | 9.5%
|
4 | main.mysql-interactive | 3 | 7.1%
|
5 | main.func_math | 3 | 7.1%
|
6 | main.func_json | 3 | 7.1%
|
7 | spider/bugfix.sql_mode_mysql | 2 | 4.8% |
8 | spider/bugfix.sql_mode_mariadb | 2 | 4.8% |
9 | main.vector | 2 | 4.8%
|
10 | main.sql_mode | 2 | 4.8%
|
11 | main.sp | 2 | 4.8%
|
12 | main.opt_trace | 2 | 4.8%
|
13 | main.mysql_secure_installation | 2 | 4.8%
|
14 | main.mysqld--help | 2 | 4.8%
|
15 | main.information_schema | 2 | 4.8%
|
16 | main.events_processlist | 2 | 4.8%
|
17 | main.events_bugs | 2 | 4.8%
|
18 | main.deprecated_sql_mode | 2 | 4.8%
|
19 | main.ansi | 2 | 4.8%
|
20 | main.analyze_format_json | 2 | 4.8%
|
|
|
40 additional tests failed once each (2.4%).
|
—
The failure analysis script is available here: https://github.com/FaramosCZ/mariadb-bb-failure-analysis
Attachments
Issue Links
- includes
-
-
- Stalled
-
-
-
- Confirmed
-
-
-
- Open
-
-
-
- In Review
-
-
-
- Stalled
-
-
-
- Confirmed
-
- relates to
-
MDEV-33073 always green buildbot
-
- Stalled
-
(1 includes, 1 relates to)