[MDEV-20694] Tests on GCOV build fail with Merge mismatch for function Created: 2019-09-28  Updated: 2022-10-18

Status: Open
Project: MariaDB Server
Component/s: Tests
Affects Version/s: 5.5, 10.1, 10.2, 10.3, 10.4, 10.6
Fix Version/s: 10.4, 10.6

Type: Bug Priority: Major
Reporter: Elena Stepanova Assignee: Sergei Golubchik
Resolution: Unresolved Votes: 0
Labels: None
Environment:

gcc (Debian 6.3.0-18+deb9u1) 6.3.0 20170516 ; also observed on Xenial with gcc 5.4.0 and 7.4.0

Attachments: File mdev20694.tar.gz    
Issue Links:
Relates
relates to MDEV-28394 Tests do not work with clang -fsaniti... Open

 Description   

To reproduce:

cmake . -DCMAKE_BUILD_TYPE=Debug -DENABLE_GCOV=ON
 
make -j6
 
lcov --directory `pwd` --zerocounters
 
cd mysql-test
 
wget https://jira.mariadb.org/secure/attachment/49027/mdev20694.tar.gz
 
tar zxf mdev20694.tar.gz
 
perl ./mtr `echo "bug.gcov1 bug.gcov2"{,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,}` --noreorder --verbose-restart

Soon enough, the test run fails with

...
 
bug.gcov2                                [ pass ]     45
 
worker[1] > Restart [mysqld.1 - pid: 9243, winpid: 9243] - using different config file
 
bug.gcov1                                [ pass ]       
worker[1] > Restart [mysqld.1 - pid: 9262, winpid: 9262] - using different config file
 
worker[1] > Restart [mysqld.2 - pid: 9264, winpid: 9264] - using different config file
 
bug.gcov2                                [ fail ]
 
        Test ended at 2019-09-28 19:16:07
 
 
 
CURRENT_TEST: bug.gcov2
 
--- /data/src/10.4-gcov/mysql-test/suite/bug/gcov2.result       2019-09-28 19:10:25.000000000 +0300
 
+++ /data/src/10.4-gcov/mysql-test/suite/bug/gcov2.reject       2019-09-28 19:16:06.996515133 +0300
 
@@ -1,2 +1,3 @@
 
 mysqld would have been started with the following arguments:
 
 
+profiling:/data/src/10.4-gcov/sql/CMakeFiles/sql.dir/sql_type.cc.gcda:Merge mismatch for function 1201
 
 
 
mysqltest: Result length mismatch

The archive contents is very simple, it's just added as a tarball for convenience. There are two test files in it, a cnf file for the first test, and result files for both:

gcov1.test

 
select 1;

gcov1.cnf

 
# Use default setting for mysqld processes
 
!include include/default_mysqld.cnf
 
!include include/default_client.cnf
 
 
 
[mysqld.1]
 
 
 
[mysqld.2]
 
 
 
[ENV]
 
MASTER_MYPORT=           @mysqld.1.port
 
MASTER_MYSOCK=           @mysqld.1.socket
 
 
 
SLAVE_MYPORT=            @mysqld.2.port
 
SLAVE_MYSOCK=            @mysqld.2.socket

gcov2.test

 
--replace_regex /.*mysqld/mysqld/
 
exec $MYSQLD --print-defaults 2>&1;

The second test is derived from main.mysqld--defaults-file. The first one can be anything, as long as it requires two servers.

It is not the only unique way to reproduce the problem. Here is another one, for example. gcov1 test is the same one as above:

perl ./mtr `echo "bug.gcov1 main.1st"{,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,}` --noreorder --verbose-restart
 
perl ./mtr main.1st

The first test run passes, there are no errors. But the second one produces the errors already on the bootstrap:

Logging: ./mtr  main.1st
 
vardir: /data/src/10.4-gcov/mysql-test/var
 
Checking leftover processes...
 
Removing old var directory...
 
Creating var directory '/data/src/10.4-gcov/mysql-test/var'...
 
Checking supported features...
 
profiling:/data/src/10.4-gcov/storage/innobase/CMakeFiles/innobase.dir/lock/lock0lock.cc.gcda:Merge mismatch for function 460
 
profiling:/data/src/10.4-gcov/sql/CMakeFiles/sql.dir/item_timefunc.cc.gcda:Merge mismatch for function 454
 
profiling:/data/src/10.4-gcov/sql/CMakeFiles/sql.dir/item_create.cc.gcda:Merge mismatch for function 333
 
profiling:/data/src/10.4-gcov/sql/CMakeFiles/sql.dir/item.cc.gcda:Merge mismatch for function 1238
 
profiling:/data/src/10.4-gcov/sql/CMakeFiles/sql.dir/handler.cc.gcda:Merge mismatch for function 868
 
profiling:/data/src/10.4-gcov/sql/CMakeFiles/sql.dir/sql_yacc_ora.cc.gcda:Merge mismatch for function 164
 
profiling:/data/src/10.4-gcov/sql/CMakeFiles/sql.dir/sql_type.cc.gcda:Merge mismatch for function 798
 
profiling:/data/src/10.4-gcov/sql/CMakeFiles/sql.dir/opt_subselect.cc.gcda:Merge mismatch for function 100
 
profiling:/data/src/10.4-gcov/sql/CMakeFiles/sql.dir/sql_acl.cc.gcda:Merge mismatch for function 159
 
MariaDB Version 10.4.9-MariaDB-debug
 
 - SSL connections supported
 
 - binaries are debug compiled
 
 - binaries built with wsrep patch
 
Collecting tests...
 
Installing system database...

Reproducible on all of current 5.5-10.4. I didn't try 10.5, but I don't expect any difference.
Couldn't check old 5.5 versions, as even 5.5.40 no longer builds on my machine.

 Comments   
Comment by Marko Mäkelä [ 2022-04-19 ]

10.6 7da351d8047087fc589f2f6cdd49b0b3647043f5

 
Failing test(s): main.mysqld--defaults-file main.mysqld--help main.mysqld--help-aria

The errors are of the following type (using GCC 12):

CURRENT_TEST: main.mysqld--help-aria
 
--- /mariadb/10.6/mysql-test/main/mysqld--help-aria.result	2022-02-24 14:34:23.321421271 +0200
 
+++ /mariadb/10.6/mysql-test/main/mysqld--help-aria.reject	2022-04-19 13:21:28.684660796 +0300
 
@@ -1,9 +1,12 @@
 
 flush tables;
 
+libgcov profiling error:/dev/shm/10.6/sql/CMakeFiles/sql.dir/sql_show.cc.gcda:Merge mismatch for function 203
 
 #
 
 # Check that we don't write any data to wrong or not existing datadir
 
 #
 
 [Warning] Could not open mysql.plugin table: "Table 'mysql.plugin' doesn't exist". Some options may be missing from the help text
 
+libgcov profiling error:/dev/shm/10.6/sql/CMakeFiles/sql.dir/sql_show.cc.gcda:Merge mismatch for function 203
 
 #
 
 # Check with existing directory
 
 #
 
 [Warning] Could not open mysql.plugin table: "Table 'mysql.plugin' doesn't exist". Some options may be missing from the help text
 
+libgcov profiling error:/dev/shm/10.6/sql/CMakeFiles/sql.dir/sql_show.cc.gcda:Merge mismatch for function 203

Comment by Anel Husakovic [ 2022-04-19 ]

Hi marko can you test with this patch https://jira.mariadb.org/browse/MDEV-26102 ?

Comment by Marko Mäkelä [ 2022-04-19 ]

anel, as far as I can tell, MDEV-26102 is about fixing a script dgcov.pl to make it accept the output (.gcov files) of newer versions of gcov. This bug here has nothing to do with that tool, but only to tests that are being run to generate the input files for gcov (.gcda and .gcno).

Something in the code that is being exercised by those 3 tests seems to be incompatible with the code generated or linked by the GCC option --coverage.

I do not know if it is related, but I also got wrongly claimed non-coverage of an error message that is being suppressed and was being emitted during the execution of the test innodb.instant_alter_import. After the message was output, the server was shut down and restarted according to the logs.

10.6 7da351d8047087fc589f2f6cdd49b0b3647043f5

 
2022-04-19 13:20:12 4 [ERROR] InnoDB: Table `test`.`t2` contains unrecognizable instant ALTER metadata

I hadn’t used gcov for years, and I am not familiar with its implementation, so I am only a dummy messenger here.

Comment by Marko Mäkelä [ 2022-04-23 ]

My claim about a bogus non-coverage report may be incorrect. There are two code snippets where the message that innodb.instant_alter_import would trigger is output, and only one of them is covered by the test.

As reported in MDEV-28394, I made an experiment with the coverage tool built in clang. The sancov tool to aggregate data from per-process trace files turned out to be unusable. It would run single-threaded. When invoked on too many files, it would allocate about 1 gigabyte of RAM per minute, until finally running out of RAM.

I am rather convinced that with any profiling tool, profiling data will normally be lost for executions that involve killing a process. Should this be a problem with the few mtr tests that involve killing the database server process, this could be worked around by duplicating some of the test logic so that the test steps would be followed by a normal process termination.

Comment by Marko Mäkelä [ 2022-10-18 ]

Note: When compiling with GCC 11 or later, some changes to the code will be needed:

diff --git a/dbug/dbug.c b/dbug/dbug.c
 
index a6d893e3f49..51c41021ded 100644
 
--- a/dbug/dbug.c
 
+++ b/dbug/dbug.c
 
@@ -86,7 +86,8 @@
 
 #include <m_string.h>
 
 #include <errno.h>
 
 #ifdef HAVE_gcov
 
-extern void __gcov_flush();
 
+extern void __gcov_dump();
 
+# define __gcov_flush __gcov_dump
 
 #endif
 
 
 #ifndef DBUG_OFF
 
diff --git a/mysys/stacktrace.c b/mysys/stacktrace.c
 
index 844d8a0b28f..cf097be054b 100644
 
--- a/mysys/stacktrace.c
 
+++ b/mysys/stacktrace.c
 
@@ -410,7 +410,8 @@ void my_print_stacktrace(uchar* stack_bottom, ulong thread_stack,
 
 void my_write_core(int sig)
 
 {
 
 #ifdef HAVE_gcov
 
-  extern void __gcov_flush(void);
 
+  extern void __gcov_dump(void);
 
+# define __gcov_flush __gcov_dump
 
 #endif
 
   signal(sig, SIG_DFL);
 
 #ifdef HAVE_gcov

Generated at Thu Feb 08 09:01:28 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.