[MDEV-7210] mroonga tests fail Created: 2014-11-26  Updated: 2015-12-21  Resolved: 2015-12-21

Status: Closed
Project: MariaDB Server
Component/s: Platform Debian, Storage Engine - Mroonga
Affects Version/s: 10.0.15
Fix Version/s: 10.0.23

Type: Bug Priority: Major
Reporter: Sergei Golubchik Assignee: Kentoku Shiba (Inactive)
Resolution: Fixed Votes: 0
Labels: None


 Description   

Mroonga tests fails in debian builds:

http://labs.seravo.fi/~otto/mariadb-repo/logs/mariadb-10.0_10.0.15-1_i386_sid.build-fail

It was a 32-bit build inside a pbuilder chroot on 64-bit host.

Note that mroonga 32-bit tests are skipped and 64-bit tests are run. Perhaps that's why they fail. Supposedly, 32/64-bit detection for tests needs to be adjusted to detect how mroonga was actually built, not what the host cpu is capable of.

 Comments   
Comment by Otto Kekäläinen [ 2014-11-26 ]

It might be difficult to detect the target architecture inside CMakeLists.txt in cross-compilation situations and therefore e.g. the TokuDB arch dependent on/off is done in the Debian rules file and passed along to cmake as an option, see http://anonscm.debian.org/cgit/pkg-mysql/mariadb-10.0.git/tree/debian/rules#n24

Upload of 10.0.15 to Debian will wait until this is fixed or alternatively until there is some temporary patch that disables suitable parts of mroonga to get around this or disables all of mroonga.

Mroonga is a new plugin and it seems the cross-compilation need of it hasn't been on the table before. Everything else of MariaDB cross-compiles (or at least to the degree needed on Debian build systems) nicely.

Comment by Otto Kekäläinen [ 2014-11-26 ]

Looking at the code I wonder it these lines really both want to check for '64'?

}}
 
storage/mroonga/mysql-test/mroonga/include/mroonga/have_32bit.inc:  `SELECT IF(@@version_compile_machine like '%64%', 1, 0)`;
 
storage/mroonga/mysql-test/mroonga/include/mroonga/have_64bit.inc:  `SELECT IF(@@version_compile_machine like '%64%', 1, 0)`;
 
{{

Comment by Daniel Black [ 2014-11-26 ]

version_compile_machine comes from MACHINE_TYPE (sql/sys_vars.cc)

MACHINE_TYPE comes from ./config.h.cmake:#define MACHINE_TYPE "@CMAKE_SYSTEM_PROCESSOR@"

CMAKE_SYSTEM_PROCESSOR comes from uname -p ( http://www.cmake.org/cmake/help/v3.0/variable/CMAKE_SYSTEM_PROCESSOR.html )

Comment by Daniel Black [ 2014-11-26 ]

fyi, tokudb, innodb,xtradb all use CMAKE_SYSTEM_PROCESSOR

storage/xtradb/CMakeLists.txt:  IF(CMAKE_SYSTEM_PROCESSOR STREQUAL "i686" AND CMAKE_COMPILER_IS_GNUCC AND
 
storage/innobase/CMakeLists.txt:  IF(CMAKE_SYSTEM_PROCESSOR STREQUAL "i686" AND CMAKE_COMPILER_IS_GNUCC AND
 
storage/tokudb/CMakeLists.txt:IF(CMAKE_SYSTEM_PROCESSOR STREQUAL "x86_64" AND

Comment by Otto Kekäläinen [ 2014-12-03 ]

Kentoku I think this issue is urgent. Could you please take a look at it soon?

Comment by Otto Kekäläinen [ 2014-12-09 ]

I solved this now by disabling Mroonga for non-amd64 platforms, see in commit http://anonscm.debian.org/cgit/pkg-mysql/mariadb-10.0.git/commit/?id=579282f22e6ff3293993651a0232e8c26ea3e75c

Comment by Kouhei Sutou [ 2015-08-16 ]

Could you try to add "-DCMAKE_SYSTEM_PROCESSOR=i386" to "cmake" options?
It may fix this problem.

Comment by Otto Kekäläinen [ 2015-08-16 ]

Here are the environment variables I could pass along to the cmake command in the debian/rules file:

DEB_BUILD_ARCH=amd64
DEB_BUILD_ARCH_OS=linux
DEB_BUILD_ARCH_CPU=amd64
DEB_BUILD_ARCH_BITS=64
DEB_BUILD_ARCH_ENDIAN=little
DEB_BUILD_GNU_CPU=x86_64
DEB_BUILD_GNU_SYSTEM=linux-gnu
DEB_BUILD_GNU_TYPE=x86_64-linux-gnu
DEB_BUILD_MULTIARCH=x86_64-linux-gnu

I need to do some testing before choosing ARCH or CPU, I am not sure which one will have i386 and which will have i686

Comment by Kouhei Sutou [ 2015-08-16 ]

Can we use "DEB_HOST_*" variables?

I can get the following variables with "dpkg-architecture -a i386":

% dpkg-architecture -a i386
DEB_BUILD_ARCH=amd64
DEB_BUILD_ARCH_BITS=64
DEB_BUILD_ARCH_CPU=amd64
DEB_BUILD_ARCH_ENDIAN=little
DEB_BUILD_ARCH_OS=linux
DEB_BUILD_GNU_CPU=x86_64
DEB_BUILD_GNU_SYSTEM=linux-gnu
DEB_BUILD_GNU_TYPE=x86_64-linux-gnu
DEB_BUILD_MULTIARCH=x86_64-linux-gnu
DEB_HOST_ARCH=i386
DEB_HOST_ARCH_BITS=32
DEB_HOST_ARCH_CPU=i386
DEB_HOST_ARCH_ENDIAN=little
DEB_HOST_ARCH_OS=linux
DEB_HOST_GNU_CPU=i586
DEB_HOST_GNU_SYSTEM=linux-gnu
DEB_HOST_GNU_TYPE=i586-linux-gnu
DEB_HOST_MULTIARCH=i386-linux-gnu
DEB_TARGET_ARCH=i386
DEB_TARGET_ARCH_BITS=32
DEB_TARGET_ARCH_CPU=i386
DEB_TARGET_ARCH_ENDIAN=little
DEB_TARGET_ARCH_OS=linux
DEB_TARGET_GNU_CPU=i586
DEB_TARGET_GNU_SYSTEM=linux-gnu
DEB_TARGET_GNU_TYPE=i586-linux-gnu
DEB_TARGET_MULTIARCH=i386-linux-gnu

Comment by Otto Kekäläinen [ 2015-08-16 ]

Yes we can use those and we already do use, see e.g. https://github.com/ottok/mariadb-10.0/blob/master/debian/rules#L101

Comment by Kouhei Sutou [ 2015-08-17 ]

Thanks for the information.
Could you try with "-DCMAKE_SYSTEM_PROCESSOR=$(DEB_HOST_ARCH)" CMake option?

Comment by Otto Kekäläinen [ 2015-08-27 ]

Log Mroonga enabled and CMAKE_SYSTEM_PROCESSOR defined: https://launchpadlibrarian.net/215657064/buildlog_ubuntu-vivid-i386.mariadb-10.0_10.0.21-2~vivid1~1440623752.15e94d8_BUILDING.txt.gz

Failing test parts.partition_exch_qa_10 is probably not Mroonga related? Please check the log, the Mroonga parts do emit some warnings.

Comment by Otto Kekäläinen [ 2015-08-27 ]

kou The test failure seemed random, with the exact same codebase a new build was successful: https://launchpadlibrarian.net/215684241/buildlog_ubuntu-vivid-i386.mariadb-10.0_10.0.21-2~vivid1~1440655723.15e94d8_BUILDING.txt.gz

The relevant Mroonga changes before this was:
https://github.com/ottok/mariadb-10.0/commit/7634d3d0fbae2ec114fb035f9ea0df4e88d95220
https://github.com/ottok/mariadb-10.0/commit/15e94d85f938990470c40c6effebbe1ac4b20f21

Does everything seem OK to you? Do you think Mroonga will build on most Debian platforms (https://buildd.debian.org/status/package.php?p=mariadb-10.0) or just LInux i386 and amd64? There is the option that we limit the platforms that try to build Mroonga using code similar to what was removed in https://github.com/ottok/mariadb-10.0/commit/7634d3d0fbae2ec114fb035f9ea0df4e88d95220

Comment by Kouhei Sutou [ 2015-08-27 ]

Thanks for trying it.

> Failing test parts.partition_exch_qa_10 is probably not Mroonga related?

It's not related to Mroonga.

> Please check the log, the Mroonga parts do emit some warnings.

These warnings aren't problem. They just say that optimization is disabled.

> The relevant Mroonga changes before this was:

It looks good to me.

> Does everything seem OK to you?

Yes.

> Do you think Mroonga will build on most Debian platforms (https://buildd.debian.org/status/package.php?p=mariadb-10.0) or just LInux i386 and amd64?

Mroonga can't be built on some platforms. But I don't know these platforms because I don't have all platforms.

I'm developing Mroonga on Debian GNU/Linux sid amd64. Mroonga must work on the platform well.

How about trying to build on all platforms and then we remove failed platforms from Mroonga support platforms?

Comment by Otto Kekäläinen [ 2015-08-28 ]

kou Ok, I uploaded 10.0.21-2 to Debian experimental. See the build logs at https://buildd.debian.org/status/package.php?p=mariadb-10.0&suite=experimental Click on the text "Build-Attempted" to view full logs. Mroonga issues now seemed to have broken arm64, armel, armhf, mips and mipsel. Please also click on the text "Old" to view previous build logs from 10.0.21-1 for comparison.

1. The ideal is that you would fix any build errors in Mroonga so that it would build on all platforms. If you do so, please send me link to commits so that I can backport them as patches on 10.0.21 and do a new upload to Debian experimental.

2. If you cannot fix something, then please tell me what platforms you want to skip, and we'll copy back code from https://github.com/ottok/mariadb-10.0/commit/7634d3d0fbae2ec114fb035f9ea0df4e88d95220 and disable Mroonga builds and test on the platforms you declare as impossible to fix.

Thanks for you help!

Comment by Kouhei Sutou [ 2015-08-28 ]

Thanks!

The following platforms don't have any Mroonga problems:

  • arm64
    • Failed tests are all perfschema related tests:
      • perfschema.func_file_io
      • perfschema.func_mutex
      • perfschema.global_read_lock
      • perfschema.setup_objects
  • armel
    • Failed tests are all perfschema related tests:
      • perfschema.func_file_io
      • perfschema.func_mutex
      • perfschema.global_read_lock
      • perfschema.setup_objects
  • armhf
    • Failed tests are connect related tests and perfschema related tests:
      • connect.bin
      • connect.endian
      • perfschema.func_file_io
      • perfschema.func_mutex
      • perfschema.global_read_lock
      • perfschema.setup_objects

Mroonga is already disabled (*1) on the following platforms because they are big-endian:

  • mips
  • mipsel
  • s390x
  • ppc64

(*1) https://github.com/MariaDB/server/blob/10.1/storage/mroonga/CMakeLists.txt#L31-L40

(I think that Mroonga can work on big-endian but I didn't tries yet. Because I don't have big-endian environment.)

Comment by Otto Kekäläinen [ 2015-08-30 ]

kou Interestingly, the same codebase fails when built on my own system via pbuilder, see full log at http://labs.seravo.fi/~otto/mariadb-repo/logs/mroonga-fails/mariadb-10.0_10.0.21-3_amd64.build-7954da1-pbuilder.log

Too many failed: Failed 10/2772 tests, 99.64% were successful.
 
 
 
Failing test(s): mroonga/storage.column_datetime_64bit_2038 mroonga/storage.column_datetime_64bit_before_unix_epoch mroonga/storage.column_datetime_64bit_max mroonga/storage.column_datetime_64bit_version_56_or_later_out_of_range mroonga/storage.index_btree_equal_datetime mroonga/storage.index_multiple_column_unique_date_64bit_equal mroonga/storage.index_multiple_column_unique_date_index_read mroonga/storage.index_multiple_column_unique_date_order_64bit_asc mroonga/storage.index_multiple_column_unique_date_order_64bit_desc mroonga/storage.index_multiple_column_unique_datetime_index_read

Some of the fail parts:

mroonga/storage.column_datetime_64bit_2038 [ fail ]
 
        Test ended at 2015-08-29 23:35:12
 
 
 
CURRENT_TEST: mroonga/storage.column_datetime_64bit_2038
 
--- /tmp/buildd/mariadb-10.0-10.0.21/storage/mroonga/mysql-test/mroonga/storage/r/column_datetime_64bit_2038.result	2015-08-05 18:11:32.000000000 +0000
 
+++ /tmp/buildd/mariadb-10.0-10.0.21/storage/mroonga/mysql-test/mroonga/storage/r/column_datetime_64bit_2038.reject	2015-08-29 23:35:11.356223645 +0000
 
@@ -16,8 +16,10 @@
 
 VALUES ('2038-01-19 03:14:07', '2038-01-19 03:14:07');
 
 INSERT INTO diaries (title, created_at)
 
 VALUES ('2038-01-19 03:14:08', '2038-01-19 03:14:08');
 
+Warnings:
 
+Warning	1265	Data truncated for column 'created_at' at row 1
 
 SELECT * FROM diaries;
 
 id	title	created_at
 
 1	2038-01-19 03:14:07	2038-01-19 03:14:07
 
-2	2038-01-19 03:14:08	2038-01-19 03:14:08
 
+2	2038-01-19 03:14:08	1970-01-01 00:00:00
 
 DROP TABLE diaries;
 
 
 
mysqltest: Result length mismatch
 
 
 
 - saving '/tmp/buildd/mariadb-10.0-10.0.21/builddir/mysql-test/var/log/mroonga/storage.column_datetime_64bit_2038/' to '/tmp/buildd/mariadb-10.0-10.0.21/builddir/mysql-test/var/log/storage.column_datetime_64bit_2038/'
 
mroonga/storage.column_datetime_64bit_before_unix_epoch [ fail ]
 
        Test ended at 2015-08-29 23:35:12
 
 
 
CURRENT_TEST: mroonga/storage.column_datetime_64bit_before_unix_epoch
 
--- /tmp/buildd/mariadb-10.0-10.0.21/storage/mroonga/mysql-test/mroonga/storage/r/column_datetime_64bit_before_unix_epoch.result	2015-08-05 18:11:32.000000000 +0000
 
+++ /tmp/buildd/mariadb-10.0-10.0.21/storage/mroonga/mysql-test/mroonga/storage/r/column_datetime_64bit_before_unix_epoch.reject	2015-08-29 23:35:12.500223622 +0000
 
@@ -14,7 +14,9 @@
 
 ) ENGINE=Mroonga DEFAULT CHARSET=utf8
 
 INSERT INTO diaries (title, created_at)
 
 VALUES ('1000-01-01 00:00:00', '1000-01-01 00:00:00');
 
+Warnings:
 
+Warning	1265	Data truncated for column 'created_at' at row 1
 
 SELECT * FROM diaries;
 
 id	title	created_at
 
-1	1000-01-01 00:00:00	1000-01-01 00:00:00
 
+1	1000-01-01 00:00:00	1970-01-01 00:00:00
 
 DROP TABLE diaries;
 
 
 
mysqltest: Result length mismatch
 
 
 
 - saving '/tmp/buildd/mariadb-10.0-10.0.21/builddir/mysql-test/var/log/mroonga/storage.column_datetime_64bit_before_unix_epoch/' to '/tmp/buildd/mariadb-10.0-10.0.21/builddir/mysql-test/var/log/storage.column_datetime_64bit_before_unix_epoch/'
 
mroonga/storage.column_datetime_64bit_max [ fail ]
 
        Test ended at 2015-08-29 23:35:13
 
 
 
CURRENT_TEST: mroonga/storage.column_datetime_64bit_max
 
--- /tmp/buildd/mariadb-10.0-10.0.21/storage/mroonga/mysql-test/mroonga/storage/r/column_datetime_64bit_max.result	2015-08-05 18:11:32.000000000 +0000
 
+++ /tmp/buildd/mariadb-10.0-10.0.21/storage/mroonga/mysql-test/mroonga/storage/r/column_datetime_64bit_max.reject	2015-08-29 23:35:12.992223613 +0000
 
@@ -14,7 +14,9 @@
 
 ) ENGINE=Mroonga DEFAULT CHARSET=utf8
 
 INSERT INTO diaries (title, created_at)
 
 VALUES ('9999-12-31 23:59:59', '9999-12-31 23:59:59');
 
+Warnings:
 
+Warning	1265	Data truncated for column 'created_at' at row 1
 
 SELECT * FROM diaries;
 
 id	title	created_at
 
-1	9999-12-31 23:59:59	9999-12-31 23:59:59
 
+1	9999-12-31 23:59:59	1970-01-01 00:00:00
 
 DROP TABLE diaries;
 
 
 
mysqltest: Result length mismatch
 
 
 
 - saving '/tmp/buildd/mariadb-10.0-10.0.21/builddir/mysql-test/var/log/mroonga/storage.column_datetime_64bit_max/' to '/tmp/buildd/mariadb-10.0-10.0.21/builddir/mysql-test/var/log/storage.column_datetime_64bit_max/'
 
mroonga/storage.column_datetime_64bit_version_55_out_of_range [ skipped ]  This test is for MySQL version 5.5.x
 
mroonga/storage.column_datetime_64bit_version_56_or_later_out_of_range [ fail ]
 
        Test ended at 2015-08-29 23:35:13
 
 
 
CURRENT_TEST: mroonga/storage.column_datetime_64bit_version_56_or_later_out_of_range
 
--- /tmp/buildd/mariadb-10.0-10.0.21/storage/mroonga/mysql-test/mroonga/storage/r/column_datetime_64bit_version_56_or_later_out_of_range.result	2015-08-05 18:11:32.000000000 +0000
 
+++ /tmp/buildd/mariadb-10.0-10.0.21/storage/mroonga/mysql-test/mroonga/storage/r/column_datetime_64bit_version_56_or_later_out_of_range.reject	2015-08-29 23:35:13.512223602 +0000
 
@@ -19,5 +19,5 @@
 
 Warning	1265	Data truncated for column 'created_at' at row 1
 
 SELECT * FROM diaries;
 
 id	title	created_at
 
-1	2012	0000-01-01 00:00:00
 
+1	2012	1970-01-01 00:00:00
 
 DROP TABLE diaries;
 
 
 
mysqltest: Result content mismatch
 
 
 
 - saving '/tmp/buildd/mariadb-10.0-10.0.21/builddir/mysql-test/var/log/mroonga/storage.column_datetime_64bit_version_56_or_later_out_of_range/' to '/tmp/buildd/mariadb-10.0-10.0.21/builddir/mysql-test/var/log/storage.column_datetime_64bit_version_56_or_later_out_of_range/'

Comment by Kouhei Sutou [ 2015-09-04 ]

It's strange result. Because these tests are for 64bit environment. (You can find "64" word in test name.)

Could you confirm the followings in pbuilder?

1. grep MACHINE_TYPE config.h

config.h exists in the build directory.

2. mysql -u root -e "show variables like 'version_compile_machine'"

Comment by Otto Kekäläinen [ 2015-09-04 ]

kou Pbuilder is basically a chroot. I will not start testing it because there is too much work for little gain, but I am pretty sure all test that test directly the kernel will see the host machine which is amd64. Builds are supposed to adhere to the flags CMAKE has about the build target. When building 32-bit binaries on 64-bit host machine, CMAKE will be run with the following line:

sh -c  'PATH=${MYSQL_BUILD_PATH:-"/usr/local/bin:/usr/bin:/bin"} \
 
    	CC=i586-linux-gnu-gcc \
 
	CFLAGS=${MYSQL_BUILD_CFLAGS:-"-O3 -fno-strict-aliasing"} \
 
    	CXX=i586-linux-gnu-g++ \
 
    	CXXFLAGS=${MYSQL_BUILD_CXXFLAGS:-"-O3 -felide-constructors -fno-exceptions -fno-rtti -fno-strict-aliasing "-DTAOCRYPT_DISABLE_X86ASM""} \
 
    cmake -DCMAKE_INSTALL_PREFIX=/usr \
 
    -DWITHOUT_TOKUDB=true \
 
    -DWITH_SSL=bundled \
 
    -DCOMPILATION_COMMENT="Debian unstable" \
 
    -DMYSQL_SERVER_SUFFIX="-3" \
 
    -DSYSTEM_TYPE="debian-linux-gnu" \
 
    -DCMAKE_SYSTEM_PROCESSOR=i386 \
 
    -DBUILD_CONFIG=mysql_release \
 
    -DINSTALL_LIBDIR=lib/i386-linux-gnu \
 
    -DINSTALL_PLUGINDIR=lib/mysql/plugin \
 
    -DINSTALL_MYSQLTESTDIR=share/mysql/mysql-test \
 
    -DDEB=Debian ..'

See log at http://labs.seravo.fi/~otto/mariadb-repo/logs/mroonga-fails/mariadb-10.0_10.0.21-3_sid-i386.build-7954da1-pbuilder.log

Comment by Kouhei Sutou [ 2015-09-04 ]

I wanted to confirm these CMake options are really applied in MariaDB layer...

Comment by Otto Kekäläinen [ 2015-12-21 ]

As of 10.0.22 these 64-bit Mroonga tests are correctly skipped on i386 builds:

mroonga/storage.column_datetime_64bit_2038 w2 [ skipped ]  Need a 64 binary
 
mroonga/storage.column_datetime_64bit_before_unix_epoch w2 [ skipped ]  Need a 64 binary
 
parts.part_supported_sql_func_myisam     w3 [ pass ]    893
 
mroonga/storage.column_datetime_64bit_max w2 [ skipped ]  Need a 64 binary
 
mroonga/storage.column_datetime_64bit_version_55_out_of_range w2 [ skipped ]  Need a 64 binary
 
mroonga/storage.column_datetime_64bit_version_56_or_later_out_of_range w2 [ skipped ]  Need a 64 binary

Full log at http://labs.seravo.fi/~otto/mariadb-repo/mariadb-10.0-sid-i386/mariadb-10.0_10.0.23-1_amd64.build-8cdd791-pbuilder.log

Closing issue.

Generated at Thu Feb 08 07:17:51 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.