[MDEV-9891] Official Ubuntu 16.04 builds unsuccessful due to TokuDB test suite fails Created: 2016-04-09  Updated: 2016-11-28  Resolved: 2016-11-28

Status: Closed
Project: MariaDB Server
Component/s: Platform Debian, Storage Engine - TokuDB
Affects Version/s: 10.0
Fix Version/s: 10.0.28

Type: Bug Priority: Minor
Reporter: Otto Kekäläinen Assignee: VicenČ›iu Ciorbaru
Resolution: Fixed Votes: 0
Labels: foundation

Issue Links:
Duplicate
is duplicated by MDEV-9091 mysqld crashes on shutdown after runn... Closed

 Description   

It was discovered, that MariaDB 10.0 builds done by official Ubuntu infrastructure on Launchpad randomly has failures in TokuDB tests:

Failing test(s): tokudb.rows-32m-0 tokudb.rows-32m-1 tokudb.savepoint-5 tokudb_bugs.frm_store2 tokudb_bugs.frm_store3 plugins.show_all_plugins

Example of one failure:

tokudb.rows-32m-0                        w1 [ fail ]  timeout after 1800 seconds
 
        Test ended at 2016-03-05 22:12:06
 
 
 
Test case timeout after 1800 seconds
 
 
 
== /«PKGBUILDDIR»/builddir/mysql-test/var/1/log/rows-32m-0.log == 
SET DEFAULT_STORAGE_ENGINE='tokudb';
 
call mtr.add_suppression("TokuDB.*");
 
call mtr.add_suppression(".*returned handler error 22");
 
drop table if exists t;
 
create table t (id int not null primary key, v longblob not null);
 
select @@max_allowed_packet into @my_max_allowed_packet;
 
set global max_allowed_packet=100000000;
 
insert into t values (1,repeat('a',32*1024*1024));
 
ERROR HY000: Got error 22 "Invalid argument" from storage engine TokuDB
 
insert into t values (1,repeat('a',32*1024*1024-1));
 
ERROR HY000: Got error 22 "Invalid argument" from storage engine TokuDB
 
insert into t values (1,repeat('a',32*1024*1024-2));
 
ERROR HY000: Got error 22 "Invalid argument" from storage engine TokuDB
 
insert into t values (1,repeat('a',32*1024*1024-3));
 
ERROR HY000: Got error 22 "Invalid argument" from storage engine TokuDB
 
insert into t values (1,repeat('a',32*1024*1024-4));

See recent log at https://launchpadlibrarian.net/252003566/buildlog_ubuntu-xenial-amd64.mariadb-10.0_10.0.24-5ubuntu2_BUILDING.txt.gz
Overview of mariadb-10.0 builds on their way into the Ubuntu 16.04 repository: https://launchpad.net/ubuntu/+source/mariadb-10.0/+publishinghistory

This test failure leads to the whole build exiting as unsuccessful, which in turn stops the new MariaDB package revision to enter the Ubuntu 16.04 archive.

The issue is hard to reproduce. I've seen it a few times in our own MariaDB test PPA, e.g
https://launchpadlibrarian.net/246287701/buildlog_ubuntu-xenial-amd64.mariadb-10.0_10.0.24-2~xenial1~1457210767.8871a3b_BUILDING.txt.gz
https://launchpadlibrarian.net/245709429/buildlog_ubuntu-xenial-amd64.mariadb-10.0_10.0.24-2~xenial1~1457073991.8871a3b_BUILDING.txt.gz
https://launchpadlibrarian.net/240974021/buildlog_ubuntu-xenial-amd64.mariadb-10.0_10.0.24-1~xenial1~1456004579.d20a814_BUILDING.txt.gz

Overview: https://launchpad.net/~mysql-ubuntu/+archive/ubuntu/mariadb-10.0/+builds?build_text=&build_state=all

I just triggered multiple builds in a row but they are all successful.

I have not seen this issue surface on official Debian build systems.

I managed though to reproduce this partially in my own build system in a xenial environment: http://labs.seravo.fi/~otto/debian/mariadb-10.0-xenial-amd64/mariadb-10.0_10.0.24-5_amd64.build-d7c9399-pbuilder.log Interestingly though the test suite does not exit properly despite failing TokuDB tests and the build ends up being successful.

I have no idea how to proceed with this. Please somebody pick this up.

 Comments   
Comment by Otto Kekäläinen [ 2016-04-09 ]

I just kept on retriggering the build and bingo, here it fails: https://launchpadlibrarian.net/253044430/buildlog_ubuntu-xenial-amd64.mariadb-10.0_10.0.24-6~xenial1~1460220068.67008ea_BUILDING.txt.gz

The summary only says Failing test(s): plugins.show_all_plugins but looking at the full log, there are multiple failing TokuDB tests.

When looking at the previous build log https://launchpadlibrarian.net/253030088/buildlog_ubuntu-xenial-amd64.mariadb-10.0_10.0.24-6~xenial1~1460212706.67008ea_BUILDING.txt.gz it does seem to have the same TokuDB test failures, but the test suite summary does not complain and the build ends up being successful.

Comment by Otto Kekäläinen [ 2016-04-13 ]

Potentially related:
https://jira.mariadb.org/browse/MDEV-9091
https://github.com/ottok/mariadb-10.0/pull/34

Comment by Otto Kekäläinen [ 2016-04-16 ]

bar Can you please assign this to yourself and investigate?

Comment by Elena Stepanova [ 2016-04-17 ]

otto, do you have server error logs from the run when you were able to reproduce it in your own system? (Or from other runs, for that matter). It can be useful.
So, there are either timeouts, or crashes. Timeouts happen on tests which are normally quite fast, so they don't have an excuse to happen, even on a slow machine.

For MTR not reporting the failed TokuDB tests in the summary, I believe that's because MTR attempted to re-run the failed tests as it usually does, but these retries did not finish, so MTR had no conclusion about the results. The output confirms it – we can see how MTR starts retries, but can't see them finish. It's questionable behavior of MTR, but it's unrelated to the core problem.

Comment by Otto Kekäläinen [ 2016-05-31 ]

For the record: Sporadic, massive TokuDB failure is still present, just noticed it in a 10.0.25 build at https://launchpadlibrarian.net/262465469/buildlog_ubuntu-wily-amd64.mariadb-10.0_10.0.25-1~wily1~1464639104.e12da90_BUILDING.txt.gz

This does not seem urgent for the time beeing, but at some point we need to get to the root of this. Maybe cvicentiu can take a look some time during the summer. He has set up a PPA nowadays too where he could run builds until this issue surfaces and then use that info to track down the failure.

Comment by Otto Kekäläinen [ 2016-07-08 ]

This is still present and Ubuntu folks are complaining about it..

Comment by Otto Kekäläinen [ 2016-10-04 ]

This seems to be fixed now as of https://github.com/ottok/mariadb-10.0/commit/e204d18ea12f137d10082c0d077c54755dd35b9e

Comment by Otto Kekäläinen [ 2016-11-25 ]

Can you please close this issue and link to the commit where you fixed it in upstream cvicentiu

Comment by Vicențiu Ciorbaru [ 2016-11-28 ]

Fixed with:

https://github.com/MariaDB/server/commit/83d5b963bd38e327a673c5d4f4d70c8223f45dd2

Generated at Thu Feb 08 07:38:06 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.