[MCOL-4659] Server MTR failures on ColumnStore tests Created: 2021-04-02  Updated: 2023-07-02

Status: Open
Project: MariaDB ColumnStore
Component/s: mtr
Affects Version/s: N/A
Fix Version/s: Icebox

Type: Bug Priority: Major
Reporter: Alexander Barkov Assignee: Unassigned
Resolution: Unresolved Votes: 0
Labels: None

Issue Links:
Blocks
is blocked by MCOL-4676 ColumnStore MTR tests: missing have_i... Closed
is blocked by MCOL-4678 MTR tests fail because of /tmp and ./... Closed
is blocked by MCOL-4693 ColumnStore MTR tests: FUNCTION mcs19... Closed
is blocked by MCOL-4748 mtr mcs229_data_compression_type fail... Closed
Relates
relates to MCOL-4674 Fix ColumnStore to run MTR tests in a... Closed
relates to MCOL-4736 Fix cross-engine mtr tests to run wit... Closed
relates to MCOL-4749 mcs247_from_unixtime_funtion fails if... Closed

 Description   

If I run "mtr" from the mysql-test directory of the server build directory without --extern (to have MTR start and use its own mariadbd server), many tests fail.

Problems

The majority of failures happen due to the following reasons:

1. Missing InnoDB plugin (MCOL-4676)

Some tests miss have_innodb.inc and fail with

Unknown storage engine 'INNODB'

2. Unpredictable default storage engine (MCOL-4676)

Some tests miss:

set @@default_storage_engine=InnoDB

and fail with this symptom:

-) ENGINE=InnoDB DEFAULT CHARSET=koi8r
 
+) ENGINE=MyISAM DEFAULT CHARSET=koi8r

Note, many tests can actually be performed with MyISAM instead of InnoDB. This should be slightly faster. So perhaps in many tests we can use:

set @@default_storage_engine=MyISAM

and re-record tests results (e.g. SHOW CREATE TABLE output) accordingly.

3. Wrong locations for LOAD DATA files (MCOL-4678)

Some tests fail because of --secure-file-priv on OUTFILE and INFILE:

CURRENT_TEST: columnstore/basic.mcs102_ldi_transform_csv
 
mysqltest: At line 32: query 'LOAD DATA INFILE '/tmp/mcs102.csv' INTO TABLE discounts FIELDS TERMINATED BY ','' failed with wrong errno 1290: 'The MariaDB server is running with the --secure-file-priv option so it cannot execute this statement', instead of 1292...

These tests should be fixed to create temporary files (for LOAD) in the MariaDB data directory instead of /tmp.

4. Post-check failures

Some tests do not drop all objects they create so the post check phase fails:

--- /home/bar/maria-git/s10.5.cs6.pow10s01/BUILD-DEB/mysql-test/var/tmp/check-mysqld_1.result	2021-04-02 09:42:15.641402755 +0400
 
+++ /home/bar/maria-git/s10.5.cs6.pow10s01/BUILD-DEB/mysql-test/var/tmp/check-mysqld_1.reject	2021-04-02 09:42:16.897407543 +0400
 
@@ -572,12 +572,14 @@
 
 WSREP_TRX_FRAGMENT_UNIT	bytes
 
 CATALOG_NAME	SCHEMA_NAME	DEFAULT_CHARACTER_SET_NAME	DEFAULT_COLLATION_NAME	SQL_PATH	SCHEMA_COMMENT
 
 def	information_schema	utf8	utf8_general_ci	NULL	
+def	mcs13_db2	latin1	latin1_swedish_ci	NULL	
 def	mtr	latin1	latin1_swedish_ci	NULL	
 def	mysql	latin1	latin1_swedish_ci	NULL	
 def	performance_schema	utf8	utf8_general_ci	NULL	
9:50

5. Missing UDFs (MCOL-4693)

mysqltest: At line 17: query 'SELECT CORR(y, x) FROM t1' failed: 1305: FUNCTION mcs192_db.CORR does not exist

When mtr is started without --extern, it installs only the storage engine plugin, but it does not install UDFs. UDFs must be installed manually in the test file.

How to repeat

Compiling and installing

Suppose I have MariaDB sources in this directory:

/home/bar/maria-git/s10.5.cs6.pow10

I compile MariaDB+MCS using the following commands:

cd /home/bar/maria-git/s10.5.cs6.pow10
 
mkdir BUILD-DEB
 
cd BUILD-DEB
 
cmake .. -DCMAKE_BUILD_TYPE=Debug \
 
 -DCMAKE_INSTALL_PREFIX=/opt/mariadb-10.5-cs \
 
 -DPLUGIN_COLUMNSTORE=DYNAMIC \
 
 -DSECURITY_HARDENED=0
 
make -j16
 
sudo make install

So in this example:

  • source tree root is /home/bar/maria-git/s10.5.cs6.pow10
  • compiled tree root is /home/bar/maria-git/s10.5.cs6.pow10/BUILD-DEB

Post-installation

Then I run usual MCS post-installation procedures and shutdown MariaDB server. Shutdown is not absolutely necessary but I just want to make sure MTR will start and use its own server instance (instead of the currently running one).

Preparation for an MTR run

THIS STEP IS NOT NEEDED ANY MORE (AFTER MCOL-4674 WAS FIXED)
Removed (see history for details)

Running tests

cd $BUILD/mysql-test/
 
./mtr --force --max-test-fail=0 --suite=columnstore/basic/



 Comments   
Comment by Alexander Barkov [ 2021-04-02 ]

I had the following output during the last MTR run:

Completed: Failed 59/328 tests, 82.01% were successful.

Failing test(s): columnstore/basic.mcol_2000 columnstore/basic.mcol_4222 columnstore/basic.mcs102_ldi_transform_csv columnstore/basic.mcs103_ldi_fields_enclosed_by columnstore/basic.mcs104_ldi_fields_escaped_by columnstore/basic.mcs105_ldi_lines_terminated_by columnstore/basic.mcs106_ldi_ignore_rows columnstore/basic.mcs10_drop_db_table columnstore/basic.mcs11_show_db_table columnstore/basic.mcs12_alter_table columnstore/basic.mcs14_truncate_table columnstore/basic.mcs18_create_drop_users columnstore/basic.mcs192_corr_function columnstore/basic.mcs193_covar_pop_function columnstore/basic.mcs194_covar_samp_function columnstore/basic.mcs195_regr_avgx_avgy_function columnstore/basic.mcs196_regr_sxx_sxy_syy_functions columnstore/basic.mcs197_regr_count_function columnstore/basic.mcs198_regr_intercept_function columnstore/basic.mcs199_regr_r2_function columnstore/basic.mcs19_grant_revoke columnstore/basic.mcs200_regr_slope_function columnstore/basic.mcs201_calshowpartitions_function columnstore/basic.mcs202_caldisablepartitions_function columnstore/basic.mcs203_calenablepartitions_function columnstore/basic.mcs204_caldroppartitions_function columnstore/basic.mcs207_utility_functions columnstore/basic.mcs208_idbPartition_function columnstore/basic.mcs209_idbDBRoot_function columnstore/basic.mcs210_idbPm_function columnstore/basic.mcs211_idbExtentId_function columnstore/basic.mcs212_idbExtentMax_function columnstore/basic.mcs213_idbExtentMin_function columnstore/basic.mcs214_idbExtentRelativeRid_function columnstore/basic.mcs215_idbSegment_function columnstore/basic.mcs216_idbSegmentDir_function columnstore/basic.mcs229_data_compression_type columnstore/basic.mcs247_from_unixtime_funtion columnstore/basic.mcs24_insert_from_another_table columnstore/basic.mcs251_time_to_sec_function columnstore/basic.mcs44_select_crossengine_join columnstore/basic.mcs45_write_crossengine_join columnstore/basic.mcs47_cpimport_central_loc_sin_source columnstore/basic.mcs49_cpimport_parallel_dist columnstore/basic.mcs50_cpimport_stdin columnstore/basic.mcs51_cpimport_select_from columnstore/basic.mcs55_cpimport_binary_source columnstore/basic.mcs56_cpimport_negative columnstore/basic.mcs5_create_table_from_other_tables columnstore/basic.mcs63_crossengine_views columnstore/basic.mcs64_crossengine_group_by columnstore/basic.mcs65_crossengine_order_by columnstore/basic.mcs68_cpimport_datafile_separators columnstore/basic.mcs71_crossengine_where_groupby_orderby_limit columnstore/basic.mcs79_exists columnstore/basic.mcs80_set_operations columnstore/basic.mcs85_derived_table columnstore/basic.mcs88_import_export_csv columnstore/basic.mcs90_aggregate_functions

Comment by Alexander Barkov [ 2021-04-02 ]

I was able to repeat the failures also in an installed tree, after "make install".
This revealed a new problem - "make install" does not install MCS suites. So I had to symlink them from the source tree:

export SOURCE=/home/bar/maria-git/s10.5.cs6.pow10
 
export TARGET=/opt/mariadb-10.5-cs
 
ln -s $SOURCE/storage/columnstore/columnstore/mtr/basic $TARGET/mysql-test/plugin/columnstore/columnstore
 
cd $TARGET/mysql-test
 
./mtr --force --max-test-fail=0 --suite=columnstore/basic/

It should be also repeatable with a binary distribution after changing the above script with proper directories. But I have not checked.

Generated at Thu Feb 08 02:52:01 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.