Details
-
Bug
-
Status: Closed (View Workflow)
-
Minor
-
Resolution: Fixed
-
10.0(EOL)
Description
It was discovered, that MariaDB 10.0 builds done by official Ubuntu infrastructure on Launchpad randomly has failures in TokuDB tests:
Failing test(s): tokudb.rows-32m-0 tokudb.rows-32m-1 tokudb.savepoint-5 tokudb_bugs.frm_store2 tokudb_bugs.frm_store3 plugins.show_all_plugins
|
Example of one failure:
tokudb.rows-32m-0 w1 [ fail ] timeout after 1800 seconds
|
Test ended at 2016-03-05 22:12:06
|
|
|
Test case timeout after 1800 seconds
|
|
|
== /«PKGBUILDDIR»/builddir/mysql-test/var/1/log/rows-32m-0.log ==
|
SET DEFAULT_STORAGE_ENGINE='tokudb';
|
call mtr.add_suppression("TokuDB.*");
|
call mtr.add_suppression(".*returned handler error 22");
|
drop table if exists t;
|
create table t (id int not null primary key, v longblob not null);
|
select @@max_allowed_packet into @my_max_allowed_packet;
|
set global max_allowed_packet=100000000;
|
insert into t values (1,repeat('a',32*1024*1024));
|
ERROR HY000: Got error 22 "Invalid argument" from storage engine TokuDB
|
insert into t values (1,repeat('a',32*1024*1024-1));
|
ERROR HY000: Got error 22 "Invalid argument" from storage engine TokuDB
|
insert into t values (1,repeat('a',32*1024*1024-2));
|
ERROR HY000: Got error 22 "Invalid argument" from storage engine TokuDB
|
insert into t values (1,repeat('a',32*1024*1024-3));
|
ERROR HY000: Got error 22 "Invalid argument" from storage engine TokuDB
|
insert into t values (1,repeat('a',32*1024*1024-4));
|
|
See recent log at https://launchpadlibrarian.net/252003566/buildlog_ubuntu-xenial-amd64.mariadb-10.0_10.0.24-5ubuntu2_BUILDING.txt.gz
Overview of mariadb-10.0 builds on their way into the Ubuntu 16.04 repository: https://launchpad.net/ubuntu/+source/mariadb-10.0/+publishinghistory
This test failure leads to the whole build exiting as unsuccessful, which in turn stops the new MariaDB package revision to enter the Ubuntu 16.04 archive.
The issue is hard to reproduce. I've seen it a few times in our own MariaDB test PPA, e.g
https://launchpadlibrarian.net/246287701/buildlog_ubuntu-xenial-amd64.mariadb-10.0_10.0.24-2~xenial1~1457210767.8871a3b_BUILDING.txt.gz
https://launchpadlibrarian.net/245709429/buildlog_ubuntu-xenial-amd64.mariadb-10.0_10.0.24-2~xenial1~1457073991.8871a3b_BUILDING.txt.gz
https://launchpadlibrarian.net/240974021/buildlog_ubuntu-xenial-amd64.mariadb-10.0_10.0.24-1~xenial1~1456004579.d20a814_BUILDING.txt.gz
Overview: https://launchpad.net/~mysql-ubuntu/+archive/ubuntu/mariadb-10.0/+builds?build_text=&build_state=all
I just triggered multiple builds in a row but they are all successful.
I have not seen this issue surface on official Debian build systems.
I managed though to reproduce this partially in my own build system in a xenial environment: http://labs.seravo.fi/~otto/debian/mariadb-10.0-xenial-amd64/mariadb-10.0_10.0.24-5_amd64.build-d7c9399-pbuilder.log Interestingly though the test suite does not exit properly despite failing TokuDB tests and the build ends up being successful.
I have no idea how to proceed with this. Please somebody pick this up.
Attachments
Issue Links
- is duplicated by
-
-
- Closed
-
Activity
Potentially related:
https://jira.mariadb.org/browse/MDEV-9091
https://github.com/ottok/mariadb-10.0/pull/34
| Field | Original Value | New Value |
|---|---|---|
| Affects Version/s | 10.0 [ 16000 ] |
otto, do you have server error logs from the run when you were able to reproduce it in your own system? (Or from other runs, for that matter). It can be useful.
So, there are either timeouts, or crashes. Timeouts happen on tests which are normally quite fast, so they don't have an excuse to happen, even on a slow machine.
For MTR not reporting the failed TokuDB tests in the summary, I believe that's because MTR attempted to re-run the failed tests as it usually does, but these retries did not finish, so MTR had no conclusion about the results. The output confirms it – we can see how MTR starts retries, but can't see them finish. It's questionable behavior of MTR, but it's unrelated to the core problem.
| Fix Version/s | 10.0 [ 16000 ] | |
| Assignee | Alexander Barkov [ bar ] |
For the record: Sporadic, massive TokuDB failure is still present, just noticed it in a 10.0.25 build at https://launchpadlibrarian.net/262465469/buildlog_ubuntu-wily-amd64.mariadb-10.0_10.0.25-1~wily1~1464639104.e12da90_BUILDING.txt.gz
This does not seem urgent for the time beeing, but at some point we need to get to the root of this. Maybe cvicentiu can take a look some time during the summer. He has set up a PPA nowadays too where he could run builds until this issue surfaces and then use that info to track down the failure.
| Labels | foundation |
| Assignee | Alexander Barkov [ bar ] | Vicentiu Ciorbaru [ cvicentiu ] |
This seems to be fixed now as of https://github.com/ottok/mariadb-10.0/commit/e204d18ea12f137d10082c0d077c54755dd35b9e
| Priority | Major [ 3 ] | Minor [ 4 ] |
Can you please close this issue and link to the commit where you fixed it in upstream cvicentiu
| Fix Version/s | 10.0.28 [ 22107 ] | |
| Fix Version/s | 10.0 [ 16000 ] | |
| Resolution | Fixed [ 1 ] | |
| Status | Open [ 1 ] | Closed [ 6 ] |
| Workflow | MariaDB v3 [ 74788 ] | MariaDB v4 [ 150313 ] |
I just kept on retriggering the build and bingo, here it fails: https://launchpadlibrarian.net/253044430/buildlog_ubuntu-xenial-amd64.mariadb-10.0_10.0.24-6~xenial1~1460220068.67008ea_BUILDING.txt.gz
The summary only says Failing test(s): plugins.show_all_plugins but looking at the full log, there are multiple failing TokuDB tests.
When looking at the previous build log https://launchpadlibrarian.net/253030088/buildlog_ubuntu-xenial-amd64.mariadb-10.0_10.0.24-6~xenial1~1460212706.67008ea_BUILDING.txt.gz it does seem to have the same TokuDB test failures, but the test suite summary does not complain and the build ends up being successful.