Uploaded image for project: 'MariaDB Server'
  1. MariaDB Server
  2. MDEV-20694

Tests on GCOV build fail with Merge mismatch for function

Details

    • Bug
    • Status: Open (View Workflow)
    • Major
    • Resolution: Unresolved
    • 5.5(EOL), 10.1(EOL), 10.2(EOL), 10.3(EOL), 10.4(EOL), 10.6
    • 10.5, 10.6
    • Tests
    • None
    • gcc (Debian 6.3.0-18+deb9u1) 6.3.0 20170516 ; also observed on Xenial with gcc 5.4.0 and 7.4.0

    Description

      To reproduce:

      cmake . -DCMAKE_BUILD_TYPE=Debug -DENABLE_GCOV=ON
      make -j6
      lcov --directory `pwd` --zerocounters
      cd mysql-test
      wget https://jira.mariadb.org/secure/attachment/49027/mdev20694.tar.gz
      tar zxf mdev20694.tar.gz
      perl ./mtr `echo "bug.gcov1 bug.gcov2"{,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,}` --noreorder --verbose-restart
      

      Soon enough, the test run fails with

      ...
      bug.gcov2                                [ pass ]     45
      worker[1] > Restart [mysqld.1 - pid: 9243, winpid: 9243] - using different config file
      bug.gcov1                                [ pass ]       
      worker[1] > Restart [mysqld.1 - pid: 9262, winpid: 9262] - using different config file
      worker[1] > Restart [mysqld.2 - pid: 9264, winpid: 9264] - using different config file
      bug.gcov2                                [ fail ]
              Test ended at 2019-09-28 19:16:07
       
      CURRENT_TEST: bug.gcov2
      --- /data/src/10.4-gcov/mysql-test/suite/bug/gcov2.result       2019-09-28 19:10:25.000000000 +0300
      +++ /data/src/10.4-gcov/mysql-test/suite/bug/gcov2.reject       2019-09-28 19:16:06.996515133 +0300
      @@ -1,2 +1,3 @@
       mysqld would have been started with the following arguments:
       
      +profiling:/data/src/10.4-gcov/sql/CMakeFiles/sql.dir/sql_type.cc.gcda:Merge mismatch for function 1201
       
      mysqltest: Result length mismatch
      

      The archive contents is very simple, it's just added as a tarball for convenience. There are two test files in it, a cnf file for the first test, and result files for both:

      gcov1.test

      select 1;
      

      gcov1.cnf

      # Use default setting for mysqld processes
      !include include/default_mysqld.cnf
      !include include/default_client.cnf
       
      [mysqld.1]
       
      [mysqld.2]
       
      [ENV]
      MASTER_MYPORT=           @mysqld.1.port
      MASTER_MYSOCK=           @mysqld.1.socket
       
      SLAVE_MYPORT=            @mysqld.2.port
      SLAVE_MYSOCK=            @mysqld.2.socket
      

      gcov2.test

      --replace_regex /.*mysqld/mysqld/
      exec $MYSQLD --print-defaults 2>&1;
      

      The second test is derived from main.mysqld--defaults-file. The first one can be anything, as long as it requires two servers.

      It is not the only unique way to reproduce the problem. Here is another one, for example. gcov1 test is the same one as above:

      perl ./mtr `echo "bug.gcov1 main.1st"{,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,}` --noreorder --verbose-restart
      perl ./mtr main.1st
      

      The first test run passes, there are no errors. But the second one produces the errors already on the bootstrap:

      Logging: ./mtr  main.1st
      vardir: /data/src/10.4-gcov/mysql-test/var
      Checking leftover processes...
      Removing old var directory...
      Creating var directory '/data/src/10.4-gcov/mysql-test/var'...
      Checking supported features...
      profiling:/data/src/10.4-gcov/storage/innobase/CMakeFiles/innobase.dir/lock/lock0lock.cc.gcda:Merge mismatch for function 460
      profiling:/data/src/10.4-gcov/sql/CMakeFiles/sql.dir/item_timefunc.cc.gcda:Merge mismatch for function 454
      profiling:/data/src/10.4-gcov/sql/CMakeFiles/sql.dir/item_create.cc.gcda:Merge mismatch for function 333
      profiling:/data/src/10.4-gcov/sql/CMakeFiles/sql.dir/item.cc.gcda:Merge mismatch for function 1238
      profiling:/data/src/10.4-gcov/sql/CMakeFiles/sql.dir/handler.cc.gcda:Merge mismatch for function 868
      profiling:/data/src/10.4-gcov/sql/CMakeFiles/sql.dir/sql_yacc_ora.cc.gcda:Merge mismatch for function 164
      profiling:/data/src/10.4-gcov/sql/CMakeFiles/sql.dir/sql_type.cc.gcda:Merge mismatch for function 798
      profiling:/data/src/10.4-gcov/sql/CMakeFiles/sql.dir/opt_subselect.cc.gcda:Merge mismatch for function 100
      profiling:/data/src/10.4-gcov/sql/CMakeFiles/sql.dir/sql_acl.cc.gcda:Merge mismatch for function 159
      MariaDB Version 10.4.9-MariaDB-debug
       - SSL connections supported
       - binaries are debug compiled
       - binaries built with wsrep patch
      Collecting tests...
      Installing system database...
      

      Reproducible on all of current 5.5-10.4. I didn't try 10.5, but I don't expect any difference.
      Couldn't check old 5.5 versions, as even 5.5.40 no longer builds on my machine.

      Attachments

        Issue Links

          Activity

            10.6 7da351d8047087fc589f2f6cdd49b0b3647043f5

            Failing test(s): main.mysqld--defaults-file main.mysqld--help main.mysqld--help-aria
            

            The errors are of the following type (using GCC 12):

            CURRENT_TEST: main.mysqld--help-aria
            --- /mariadb/10.6/mysql-test/main/mysqld--help-aria.result	2022-02-24 14:34:23.321421271 +0200
            +++ /mariadb/10.6/mysql-test/main/mysqld--help-aria.reject	2022-04-19 13:21:28.684660796 +0300
            @@ -1,9 +1,12 @@
             flush tables;
            +libgcov profiling error:/dev/shm/10.6/sql/CMakeFiles/sql.dir/sql_show.cc.gcda:Merge mismatch for function 203
             #
             # Check that we don't write any data to wrong or not existing datadir
             #
             [Warning] Could not open mysql.plugin table: "Table 'mysql.plugin' doesn't exist". Some options may be missing from the help text
            +libgcov profiling error:/dev/shm/10.6/sql/CMakeFiles/sql.dir/sql_show.cc.gcda:Merge mismatch for function 203
             #
             # Check with existing directory
             #
             [Warning] Could not open mysql.plugin table: "Table 'mysql.plugin' doesn't exist". Some options may be missing from the help text
            +libgcov profiling error:/dev/shm/10.6/sql/CMakeFiles/sql.dir/sql_show.cc.gcda:Merge mismatch for function 203
            

            marko Marko Mäkelä added a comment - 10.6 7da351d8047087fc589f2f6cdd49b0b3647043f5 Failing test(s): main.mysqld--defaults-file main.mysqld--help main.mysqld--help-aria The errors are of the following type (using GCC 12): CURRENT_TEST: main.mysqld--help-aria --- /mariadb/10.6/mysql-test/main/mysqld--help-aria.result 2022-02-24 14:34:23.321421271 +0200 +++ /mariadb/10.6/mysql-test/main/mysqld--help-aria.reject 2022-04-19 13:21:28.684660796 +0300 @@ -1,9 +1,12 @@ flush tables; +libgcov profiling error:/dev/shm/10.6/sql/CMakeFiles/sql.dir/sql_show.cc.gcda:Merge mismatch for function 203 # # Check that we don't write any data to wrong or not existing datadir # [Warning] Could not open mysql.plugin table: "Table 'mysql.plugin' doesn't exist". Some options may be missing from the help text +libgcov profiling error:/dev/shm/10.6/sql/CMakeFiles/sql.dir/sql_show.cc.gcda:Merge mismatch for function 203 # # Check with existing directory # [Warning] Could not open mysql.plugin table: "Table 'mysql.plugin' doesn't exist". Some options may be missing from the help text +libgcov profiling error:/dev/shm/10.6/sql/CMakeFiles/sql.dir/sql_show.cc.gcda:Merge mismatch for function 203

            Hi marko can you test with this patch https://jira.mariadb.org/browse/MDEV-26102 ?

            anel Anel Husakovic added a comment - Hi marko can you test with this patch https://jira.mariadb.org/browse/MDEV-26102 ?

            anel, as far as I can tell, MDEV-26102 is about fixing a script dgcov.pl to make it accept the output (.gcov files) of newer versions of gcov. This bug here has nothing to do with that tool, but only to tests that are being run to generate the input files for gcov (.gcda and .gcno).

            Something in the code that is being exercised by those 3 tests seems to be incompatible with the code generated or linked by the GCC option --coverage.

            I do not know if it is related, but I also got wrongly claimed non-coverage of an error message that is being suppressed and was being emitted during the execution of the test innodb.instant_alter_import. After the message was output, the server was shut down and restarted according to the logs.

            10.6 7da351d8047087fc589f2f6cdd49b0b3647043f5

            2022-04-19 13:20:12 4 [ERROR] InnoDB: Table `test`.`t2` contains unrecognizable instant ALTER metadata
            

            I hadn’t used gcov for years, and I am not familiar with its implementation, so I am only a dummy messenger here.

            marko Marko Mäkelä added a comment - anel , as far as I can tell, MDEV-26102 is about fixing a script dgcov.pl to make it accept the output ( .gcov files) of newer versions of gcov . This bug here has nothing to do with that tool, but only to tests that are being run to generate the input files for gcov ( .gcda and .gcno ). Something in the code that is being exercised by those 3 tests seems to be incompatible with the code generated or linked by the GCC option --coverage . I do not know if it is related, but I also got wrongly claimed non-coverage of an error message that is being suppressed and was being emitted during the execution of the test innodb.instant_alter_import . After the message was output, the server was shut down and restarted according to the logs. 10.6 7da351d8047087fc589f2f6cdd49b0b3647043f5 2022-04-19 13:20:12 4 [ERROR] InnoDB: Table `test`.`t2` contains unrecognizable instant ALTER metadata I hadn’t used gcov for years, and I am not familiar with its implementation, so I am only a dummy messenger here.

            My claim about a bogus non-coverage report may be incorrect. There are two code snippets where the message that innodb.instant_alter_import would trigger is output, and only one of them is covered by the test.

            As reported in MDEV-28394, I made an experiment with the coverage tool built in clang. The sancov tool to aggregate data from per-process trace files turned out to be unusable. It would run single-threaded. When invoked on too many files, it would allocate about 1 gigabyte of RAM per minute, until finally running out of RAM.

            I am rather convinced that with any profiling tool, profiling data will normally be lost for executions that involve killing a process. Should this be a problem with the few mtr tests that involve killing the database server process, this could be worked around by duplicating some of the test logic so that the test steps would be followed by a normal process termination.

            marko Marko Mäkelä added a comment - My claim about a bogus non-coverage report may be incorrect. There are two code snippets where the message that innodb.instant_alter_import would trigger is output, and only one of them is covered by the test. As reported in MDEV-28394 , I made an experiment with the coverage tool built in clang . The sancov tool to aggregate data from per-process trace files turned out to be unusable. It would run single-threaded. When invoked on too many files, it would allocate about 1 gigabyte of RAM per minute, until finally running out of RAM. I am rather convinced that with any profiling tool, profiling data will normally be lost for executions that involve killing a process. Should this be a problem with the few mtr tests that involve killing the database server process, this could be worked around by duplicating some of the test logic so that the test steps would be followed by a normal process termination.

            Note: When compiling with GCC 11 or later, some changes to the code will be needed:

            diff --git a/dbug/dbug.c b/dbug/dbug.c
            index a6d893e3f49..51c41021ded 100644
            --- a/dbug/dbug.c
            +++ b/dbug/dbug.c
            @@ -86,7 +86,8 @@
             #include <m_string.h>
             #include <errno.h>
             #ifdef HAVE_gcov
            -extern void __gcov_flush();
            +extern void __gcov_dump();
            +# define __gcov_flush __gcov_dump
             #endif
             
             #ifndef DBUG_OFF
            diff --git a/mysys/stacktrace.c b/mysys/stacktrace.c
            index 844d8a0b28f..cf097be054b 100644
            --- a/mysys/stacktrace.c
            +++ b/mysys/stacktrace.c
            @@ -410,7 +410,8 @@ void my_print_stacktrace(uchar* stack_bottom, ulong thread_stack,
             void my_write_core(int sig)
             {
             #ifdef HAVE_gcov
            -  extern void __gcov_flush(void);
            +  extern void __gcov_dump(void);
            +# define __gcov_flush __gcov_dump
             #endif
               signal(sig, SIG_DFL);
             #ifdef HAVE_gcov
            

            marko Marko Mäkelä added a comment - Note: When compiling with GCC 11 or later, some changes to the code will be needed: diff --git a/dbug/dbug.c b/dbug/dbug.c index a6d893e3f49..51c41021ded 100644 --- a/dbug/dbug.c +++ b/dbug/dbug.c @@ -86,7 +86,8 @@ #include <m_string.h> #include <errno.h> #ifdef HAVE_gcov -extern void __gcov_flush(); +extern void __gcov_dump(); +# define __gcov_flush __gcov_dump #endif #ifndef DBUG_OFF diff --git a/mysys/stacktrace.c b/mysys/stacktrace.c index 844d8a0b28f..cf097be054b 100644 --- a/mysys/stacktrace.c +++ b/mysys/stacktrace.c @@ -410,7 +410,8 @@ void my_print_stacktrace(uchar* stack_bottom, ulong thread_stack, void my_write_core(int sig) { #ifdef HAVE_gcov - extern void __gcov_flush(void); + extern void __gcov_dump(void); +# define __gcov_flush __gcov_dump #endif signal(sig, SIG_DFL); #ifdef HAVE_gcov

            People

              serg Sergei Golubchik
              elenst Elena Stepanova
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

                Created:
                Updated:

                Git Integration

                  Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.