Uploaded image for project: 'MariaDB Server'
  1. MariaDB Server
  2. MDEV-25288

Create a list of tests for distributions to run upon building/packaging MariaDB server

    XMLWordPrintable

    Details

      Description

      Replace the black list of unstable tests introduced in MDEV-10604 and maintained since then with a list of tests which distributions can run in their build/packaging process.

      Since the goal of running MTR tests during 3-rd party build process is not to test MariaDB code but only to ensure that the build is functional, there is no need to run thousands of MariaDB tests and combinations. So, the new collection will be much shorter and more static than the previous unstable-tests list.

      Criteria:

      • separate components (plugins, clients, scripts) should be represented by at least one test when possible;
      • every suite should be represented by at least one test, except for those which aren't run regularly and thus aren't maintained properly;
      • the tests shouldn't have any sporadic failures on main maintenance branches in the last half of a year (*);
      • the tests can't require debug build, windows, or a special configuration not normally present on build machines;
      • out of the remaining candidates, the priority is given to those with higher line coverage and/or more generic ones (judging by name) and/or more those which closer relate to the system/environment (judging by name).

      (*) Once the list is created, this condition won't be explicitly re-checked before the release, as it was done for unstable-tests. Instead, we should have an buildbot step which would run these tests, and its failure should be "fatal" and require an immediate action.

      Draft list for 10.2 (first edition)

      archive.archive
      archive.archive_gis
      archive.partition_archive
      binlog.binlog_base64_flag
      binlog.binlog_database
      binlog.binlog_innodb
      binlog.binlog_parallel_replication_marks_stm_mix
      binlog.binlog_row_mix_innodb_myisam
      binlog.flashback
      binlog_encryption.encrypted_slave
      connect.bin
      connect.bson
      connect.dbf
      connect.dir
      connect.endian
      connect.general
      connect.json
      connect.mysql
      connect.secure_file_priv
      connect.tbl
      csv.csv
      disks.disks
      encryption.encryption_force
      encryption.innodb_encryption_tables
      encryption.tempfiles_encrypted
      federated.federated
      federated.federatedx
      gcol.gcol_select_innodb
      handler.interface
      heap.heap
      innodb.innodb
      innodb.autoinc_persist
      innodb.innodb_defrag_binlog
      innodb.innodb_mysql
      innodb.monitor
      innodb.purge
      innodb.table_flags
      innodb.xa_recovery
      innodb_fts.fulltext
      innodb_gis.geometry
      innodb_gis.rtree
      innodb_zip.innodb-zip
      innodb_zip.page_size
      json.json_no_table
      main-test_sql_discovery.create
      main.blackhole
      main.bootstrap
      main.compress
      main.connect
      main.ctype_collate
      main.ctype_utf8
      main.default
      main.dyncol
      main.fulltext
      main.function_defaults
      main.gis
      main.grant
      main.handlersocket
      main.information_schema
      main.innodb_ext_key
      main.log_tables
      main.lowercase_fs_off
      main.myisam
      main.mysql_client_test
      main.mysql_protocols
      main.mysql_upgrade
      main.mysqladmin
      main.mysqlbinlog
      main.mysqlcheck
      main.mysqld--defaults-file
      main.mysqldump
      main.mysqlhotcopy_myisam
      main.mysqlshow
      main.mysqlslap
      main.mysqltest
      main.parser
      main.partition
      main.perror
      main.plugin
      main.plugin_auth
      main.pool_of_threads
      main.ps
      main.repair
      main.shutdown
      main.sp
      main.ssl
      main.ssl_compress
      main.stat_tables
      main.statistics
      main.subselect
      main.symlink
      main.temp_table
      main.timezone
      main.type_timestamp_hires
      main.user_var
      main.userstat
      main.variables
      main.view
      main.win
      main.xa
      maria.maria
      mariabackup.full_backup
      metadata_lock_info.table_metadata_lock
      mroonga/storage.variable_version
      mroonga/wrapper.count_star
      multi_source.multisource
      oqgraph.general-innodb
      parts.rpl_partition
      perfschema.selects
      plugins.auth_ed25519
      plugins.cracklib_password_check
      plugins.dialog
      plugins.fulltext_plugin
      plugins.locales
      plugins.pam_cleartext
      plugins.processlist
      plugins.qc_info
      plugins.server_audit
      plugins.simple_password_check
      plugins.sql_error_log
      plugins.two_password_validations
      plugins.unix_socket
      query_response_time.basic
      rocksdb.rocksdb
      roles.definer
      rpl.rpl_gtid_basic
      rpl.rpl_relayrotate
      rpl.rpl_row_blob_innodb
      rpl.rpl_semi_sync_event
      rpl.rpl_sp
      rpl.rpl_stm_binlog_max_cache_size
      rpl.rpl_switch_stm_row_mixed
      sequence.simple
      spider.basic_sql
      spider.ha
      sql_discovery.simple
      sys_vars.sysvars_aria
      sys_vars.sysvars_server_notembedded
      vcol.vcol_syntax
      wsrep.variables
      

      On my machine it takes ~3 min on a non-debug 10.2 build, with MTR running in shm with --parallel=4.

      10.3 change over 10.2 list

      --- a/mysql-test/collections/smoke_test
      +++ b/mysql-test/collections/smoke_test
      @@ -48,7 +48,8 @@ innodb_gis.rtree
       innodb_zip.innodb-zip
       innodb_zip.page_size
       json.json_no_table
      -main-test_sql_discovery.create
      +# Doesn't work in 10.3+ due to MDEV-25384
      +# main-test_sql_discovery.create
       main.blackhole
       main.bootstrap
       main.compress
      @@ -145,13 +146,22 @@ sys_vars.sysvars_server_notembedded
       wsrep.variables
       "
       
      +tests_103="
      +compat/oracle.binlog_ptr_mysqlbinlog
      +compat/oracle.sp-package
      +compat/maxdb.rpl_mariadb_timestamp
      +sql_sequence.mysqldump
      +versioning.simple
      +versioning.trx_id
      +"
      +
       tests_to_run=""
       
       while read -r test
       do
         test=`echo $test | sed -e s/\#.*//`
         tests_to_run="$tests_to_run $test"
      -done <<< $(echo "$tests_102")
      +done <<< $(echo "$tests_102 $tests_103")
       
       echo $tests_to_run
       

      10.4 additions to 10.3 list

      diff --git a/mysql-test/collections/smoke_test b/mysql-test/collections/smoke_test
      index 445e0f6d529..986d9856ac2 100755
      --- a/mysql-test/collections/smoke_test
      +++ b/mysql-test/collections/smoke_test
      @@ -154,6 +154,10 @@ sql_sequence.mysqldump
       versioning.simple
       versioning.trx_id
       "
      +tests_104="
      +period.versioning
      +plugins.multiauth
      +"
       
       tests_to_run=""
       
      @@ -161,7 +165,7 @@ while read -r test
       do
         test=`echo $test | sed -e s/\#.*//`
         tests_to_run="$tests_to_run $test"
      -done <<< $(echo "$tests_102 $tests_103")
      +done <<< $(echo "$tests_102 $tests_103 $tests_104")
       
       echo $tests_to_run
      

      10.5 additions to 10.4 list

      diff --git a/mysql-test/collections/smoke_test b/mysql-test/collections/smoke_test
      index 986d9856ac2..2391f4f6057 100755
      --- a/mysql-test/collections/smoke_test
      +++ b/mysql-test/collections/smoke_test
      @@ -159,13 +159,18 @@ period.versioning
       plugins.multiauth
       "
       
      +tests_105="
      +client.mariadb-conv
      +innodb_i_s.innodb_sys_tables
      +"
      +
       tests_to_run=""
       
       while read -r test
       do
         test=`echo $test | sed -e s/\#.*//`
         tests_to_run="$tests_to_run $test"
      -done <<< $(echo "$tests_102 $tests_103 $tests_104")
      +done <<< $(echo "$tests_102 $tests_103 $tests_104 $tests_105")
       
       echo $tests_to_run
      

      In the current edition, the list is implemented as a shell script under mysql-test/collections/
      It runs MTR with the pre-defined list of tests and optionally with other MTR parameters passed from the command line.

      One disadvantage of having it this way (and same would be if it was a plain text list) is that i out-of-source builds it will be a bit awkward to call, because collections remain in the sourcedir, together with MTR tests.
      For example, in the old buildbot deb package builds, it would be something like

      cd builddir/mysql-test
      ../../mysql-test/collections/smoke_test <other options>
      

        Attachments

          Issue Links

            Activity

              People

              Assignee:
              elenst Elena Stepanova
              Reporter:
              elenst Elena Stepanova
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved: