[MDEV-25820] CUnit: 20 tests failed out of 86 when running 'make test' Created: 2021-05-30  Updated: 2023-04-27

Status: Open
Project: MariaDB Server
Component/s: Tests
Affects Version/s: 10.6.1
Fix Version/s: 10.4, 10.5

Type: Bug Priority: Minor
Reporter: Otto Kekäläinen Assignee: Georg Richter
Resolution: Unresolved Votes: 0
Labels: None

Issue Links:
Duplicate
is duplicated by MDEV-12470 'make test' failing Closed

 Description   

I noticed that when running 'make tests' directly after a build (any build, on all platforms I tested) it does not pass.

The CUnit tests should pass. If there are tests that are permanently broken or require something special, then they should not be part of the standard CUnit test run, but separated. It does not make sense for a 'make test' to fail on a vanilla build.

Full output:

# make test
 
Running tests...
 
Test project /build/builddir
 
      Start  1: strings
 
 1/86 Test  #1: strings ..........................   Passed    0.01 sec
 
      Start  2: json
 
 2/86 Test  #2: json .............................   Passed    0.00 sec
 
      Start  3: simple
 
 3/86 Test  #3: simple ...........................   Passed    0.21 sec
 
      Start  4: skip
 
 4/86 Test  #4: skip .............................   Passed    0.00 sec
 
      Start  5: todo
 
 5/86 Test  #5: todo .............................   Passed    0.00 sec
 
      Start  6: skip_all
 
 6/86 Test  #6: skip_all .........................   Passed    0.00 sec
 
      Start  7: no_plan
 
 7/86 Test  #7: no_plan ..........................   Passed    0.00 sec
 
      Start  8: bitmap
 
 8/86 Test  #8: bitmap ...........................   Passed    0.16 sec
 
      Start  9: base64
 
 9/86 Test  #9: base64 ...........................   Passed    0.02 sec
 
      Start 10: my_atomic
 
10/86 Test #10: my_atomic ........................   Passed    0.02 sec
 
      Start 11: my_rdtsc
 
11/86 Test #11: my_rdtsc .........................   Passed    0.02 sec
 
      Start 12: lf
 
12/86 Test #12: lf ...............................   Passed    0.39 sec
 
      Start 13: my_malloc
 
13/86 Test #13: my_malloc ........................   Passed    0.00 sec
 
      Start 14: my_getopt
 
14/86 Test #14: my_getopt ........................   Passed    0.00 sec
 
      Start 15: dynstring
 
15/86 Test #15: dynstring ........................   Passed    0.00 sec
 
      Start 16: byte_order
 
16/86 Test #16: byte_order .......................   Passed    0.00 sec
 
      Start 17: queues
 
17/86 Test #17: queues ...........................   Passed    0.01 sec
 
      Start 18: stacktrace
 
18/86 Test #18: stacktrace .......................   Passed    0.10 sec
 
      Start 19: crc32
 
19/86 Test #19: crc32 ............................   Passed    0.00 sec
 
      Start 20: my_vsnprintf
 
20/86 Test #20: my_vsnprintf .....................   Passed    0.00 sec
 
      Start 21: aes
 
21/86 Test #21: aes ..............................   Passed    0.00 sec
 
      Start 22: ma_dyncol
 
22/86 Test #22: ma_dyncol ........................   Passed    0.01 sec
 
      Start 23: my_decimal
 
23/86 Test #23: my_decimal .......................   Passed    0.00 sec
 
      Start 24: json_lib
 
24/86 Test #24: json_lib .........................   Passed    0.00 sec
 
      Start 25: my_apc
 
25/86 Test #25: my_apc ...........................   Passed    7.54 sec
 
      Start 26: explain_filename
 
26/86 Test #26: explain_filename .................   Passed    0.05 sec
 
      Start 27: mf_iocache
 
27/86 Test #27: mf_iocache .......................   Passed    0.03 sec
 
      Start 28: conc336
 
28/86 Test #28: conc336 ..........................***Failed    0.01 sec
 
      Start 29: bulk1
 
29/86 Test #29: bulk1 ............................***Failed    0.01 sec
 
      Start 30: performance
 
30/86 Test #30: performance ......................***Failed    0.01 sec
 
      Start 31: basic-t
 
31/86 Test #31: basic-t ..........................***Failed    0.01 sec
 
      Start 32: fetch
 
32/86 Test #32: fetch ............................***Failed    0.01 sec
 
      Start 33: charset
 
33/86 Test #33: charset ..........................***Failed    0.01 sec
 
      Start 34: logs
 
34/86 Test #34: logs .............................***Failed    0.01 sec
 
      Start 35: cursor
 
35/86 Test #35: cursor ...........................***Failed    0.01 sec
 
      Start 36: errors
 
36/86 Test #36: errors ...........................***Failed    0.01 sec
 
      Start 37: view
 
37/86 Test #37: view .............................***Failed    0.01 sec
 
      Start 38: ps
 
38/86 Test #38: ps ...............................***Failed    0.01 sec
 
      Start 39: ps_bugs
 
39/86 Test #39: ps_bugs ..........................***Failed    0.01 sec
 
      Start 40: sp
 
40/86 Test #40: sp ...............................***Failed    0.01 sec
 
      Start 41: result
 
41/86 Test #41: result ...........................***Failed    0.01 sec
 
      Start 42: connection
 
42/86 Test #42: connection .......................***Failed    0.01 sec
 
      Start 43: misc
 
43/86 Test #43: misc .............................***Failed    0.01 sec
 
      Start 44: ps_new
 
44/86 Test #44: ps_new ...........................***Failed    0.01 sec
 
      Start 45: thread
 
45/86 Test #45: thread ...........................***Failed    0.01 sec
 
      Start 46: features-10_2
 
46/86 Test #46: features-10_2 ....................***Failed    0.01 sec
 
      Start 47: async
 
47/86 Test #47: async ............................***Failed    0.01 sec
 
      Start 48: innodb_sync
 
48/86 Test #48: innodb_sync ......................   Passed    1.43 sec
 
      Start 49: ma_control_file
 
49/86 Test #49: ma_control_file ..................   Passed    0.02 sec
 
      Start 50: trnman
 
50/86 Test #50: trnman ...........................   Passed    0.23 sec
 
      Start 51: ma_test_loghandler
 
51/86 Test #51: ma_test_loghandler ...............   Passed    0.50 sec
 
      Start 52: ma_test_loghandler_multigroup
 
52/86 Test #52: ma_test_loghandler_multigroup ....   Passed    0.01 sec
 
      Start 53: ma_test_loghandler_multithread
 
53/86 Test #53: ma_test_loghandler_multithread ...   Passed    0.48 sec
 
      Start 54: ma_test_loghandler_pagecache
 
54/86 Test #54: ma_test_loghandler_pagecache .....   Passed    0.02 sec
 
      Start 55: ma_test_loghandler_long
 
55/86 Test #55: ma_test_loghandler_long ..........   Passed    0.17 sec
 
      Start 56: ma_test_loghandler_noflush
 
56/86 Test #56: ma_test_loghandler_noflush .......   Passed    0.02 sec
 
      Start 57: ma_test_loghandler_first_lsn
 
57/86 Test #57: ma_test_loghandler_first_lsn .....   Passed    0.03 sec
 
      Start 58: ma_test_loghandler_max_lsn
 
58/86 Test #58: ma_test_loghandler_max_lsn .......   Passed    0.24 sec
 
      Start 59: ma_test_loghandler_purge
 
59/86 Test #59: ma_test_loghandler_purge .........   Passed    0.20 sec
 
      Start 60: ma_test_loghandler_readonly
 
60/86 Test #60: ma_test_loghandler_readonly ......   Passed    0.02 sec
 
      Start 61: ma_test_loghandler_nologs
 
61/86 Test #61: ma_test_loghandler_nologs ........   Passed    0.20 sec
 
      Start 62: ma_pagecache_single_1k
 
62/86 Test #62: ma_pagecache_single_1k ...........   Passed    0.16 sec
 
      Start 63: ma_pagecache_single_8k
 
63/86 Test #63: ma_pagecache_single_8k ...........   Passed    0.01 sec
 
      Start 64: ma_pagecache_single_64k
 
64/86 Test #64: ma_pagecache_single_64k ..........   Passed    0.01 sec
 
      Start 65: ma_pagecache_consist_1k
 
65/86 Test #65: ma_pagecache_consist_1k ..........   Passed    0.01 sec
 
      Start 66: ma_pagecache_consist_64k
 
66/86 Test #66: ma_pagecache_consist_64k .........   Passed    0.01 sec
 
      Start 67: ma_pagecache_consist_1kHC
 
67/86 Test #67: ma_pagecache_consist_1kHC ........   Passed    0.01 sec
 
      Start 68: ma_pagecache_consist_64kHC
 
68/86 Test #68: ma_pagecache_consist_64kHC .......   Passed    0.01 sec
 
      Start 69: ma_pagecache_consist_1kRD
 
69/86 Test #69: ma_pagecache_consist_1kRD ........   Passed    0.28 sec
 
      Start 70: ma_pagecache_consist_64kRD
 
70/86 Test #70: ma_pagecache_consist_64kRD .......   Passed    5.33 sec
 
      Start 71: ma_pagecache_consist_1kWR
 
71/86 Test #71: ma_pagecache_consist_1kWR ........   Passed    0.52 sec
 
      Start 72: ma_pagecache_consist_64kWR
 
72/86 Test #72: ma_pagecache_consist_64kWR .......   Passed    4.81 sec
 
      Start 73: ma_pagecache_rwconsist_1k
 
73/86 Test #73: ma_pagecache_rwconsist_1k ........   Passed    0.00 sec
 
      Start 74: ma_pagecache_rwconsist2_1k
 
74/86 Test #74: ma_pagecache_rwconsist2_1k .......   Passed    0.00 sec
 
      Start 75: pfs_instr_class
 
75/86 Test #75: pfs_instr_class ..................   Passed    0.01 sec
 
      Start 76: pfs_instr_class-oom
 
76/86 Test #76: pfs_instr_class-oom ..............   Passed    0.00 sec
 
      Start 77: pfs_instr
 
77/86 Test #77: pfs_instr ........................   Passed    0.00 sec
 
      Start 78: pfs_instr-oom
 
78/86 Test #78: pfs_instr-oom ....................   Passed    0.01 sec
 
      Start 79: pfs_account-oom
 
79/86 Test #79: pfs_account-oom ..................   Passed    0.00 sec
 
      Start 80: pfs_host-oom
 
80/86 Test #80: pfs_host-oom .....................   Passed    0.01 sec
 
      Start 81: pfs_timer
 
81/86 Test #81: pfs_timer ........................   Passed   25.04 sec
 
      Start 82: pfs_user-oom
 
82/86 Test #82: pfs_user-oom .....................   Passed    0.03 sec
 
      Start 83: pfs_noop
 
83/86 Test #83: pfs_noop .........................   Passed    0.01 sec
 
      Start 84: pfs
 
84/86 Test #84: pfs ..............................   Passed    0.03 sec
 
      Start 85: pfs_misc
 
85/86 Test #85: pfs_misc .........................   Passed    0.01 sec
 
      Start 86: ed25519
 
86/86 Test #86: ed25519 ..........................   Passed    0.01 sec
 
 
 
77% tests passed, 20 tests failed out of 86
 
 
 
Total Test time (real) =  48.89 sec
 
 
 
The following tests FAILED:
 
	 28 - conc336 (Failed)
 
	 29 - bulk1 (Failed)
 
	 30 - performance (Failed)
 
	 31 - basic-t (Failed)
 
	 32 - fetch (Failed)
 
	 33 - charset (Failed)
 
	 34 - logs (Failed)
 
	 35 - cursor (Failed)
 
	 36 - errors (Failed)
 
	 37 - view (Failed)
 
	 38 - ps (Failed)
 
	 39 - ps_bugs (Failed)
 
	 40 - sp (Failed)
 
	 41 - result (Failed)
 
	 42 - connection (Failed)
 
	 43 - misc (Failed)
 
	 44 - ps_new (Failed)
 
	 45 - thread (Failed)
 
	 46 - features-10_2 (Failed)
 
	 47 - async (Failed)
 
Errors while running CTest
 
make: *** [Makefile:129: test] Error 8



 Comments   
Comment by Sergei Golubchik [ 2021-06-05 ]

This is a duplicate of MDEV-12470, the same answer applies. Unit tests of libmariadb need a running server, if you don't meet the precondition of the test, it will fail.

Comment by Otto Kekäläinen [ 2021-06-06 ]

This is not a bug report about CUnit tests failing. This is a bug report about 'make tests' failing. Users expect by convetion that running commands like 'make', 'make build', 'make test', 'make install' etc work. And they do work when CMake generates the Makefile and CMake is correctly configured.

Either you decrease the scope of CUnit tests run in 'make test' so it actually does run them, or you extend 'make test' to start a server so that the full set of CUnit passes.

Comment by Otto Kekäläinen [ 2021-06-14 ]

For the record, if built with `cmake -G Ninja` and when running `ninja test` cunit also fails. The mtr is starting/stopping the server, I hope there was some way to make cunit start the server as well so that the last 20 tests could properly run and cunit exit with success so that CI systems and new contributors running cunit would not see errors..

Comment by Sergei Golubchik [ 2021-06-21 ]

georg, what do you think about all C/C unit tests doing skip_all unless some MTR variable is set in the environment? Like

if (!getenv("MYSQLTEST_VARDIR")) {
 
  skip_all();
 
  exit(0);
 
}

Something like that?

Comment by Georg Richter [ 2021-06-21 ]

serg What about skipping all tests if we can't connect via specified credentials?

Comment by Sergei Golubchik [ 2021-06-21 ]

georg, no, please, not that. You had this in the first version of unit tests, and they were silently skipped in mtr for quite a while and nobody noticed, buildbot was green. Tests should fail hard if they are expected to connect, but cannot.

Comment by Georg Richter [ 2021-06-24 ]

Fixed in Connector/C:

Tests will be skipped unless the following environment variables
were specified:

  • MYSQLTEST_VARDIR (will be automatically set when running inside
    mtr)
  • MARIADB_CC_TEST
Comment by Georg Richter [ 2021-06-27 ]

sanja: this bug can be closed after updating libmariadb subproject.

Comment by Oleksandr Byelkin [ 2021-06-28 ]

IMHO it is enough to be sure that server is reachable and it does not matter if it is mysql-test-run or something else, i.e. I propose to just check all server related variables and do not make a new one.

Generated at Thu Feb 08 09:40:39 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.