[MDEV-10651] sys_vars.innodb_buffer_pool_dump_pct_basic failed in buildbot Created: 2016-08-23  Updated: 2017-11-05  Resolved: 2017-11-05

Status: Closed
Project: MariaDB Server
Component/s: Tests
Affects Version/s: 10.0, 10.1
Fix Version/s: 10.0.34, 10.1.29, 10.2.11

Type: Bug Priority: Minor
Reporter: Elena Stepanova Assignee: Elena Stepanova
Resolution: Fixed Votes: 0
Labels: None

Issue Links:
Relates
relates to MDEV-9713 Sporadic test failure: sys_vars.innod... Closed
relates to MDEV-14196 sys_vars.innodb_buffer_pool_load_now_... Closed
relates to MDEV-7069 Fix buildbot failures in main server ... Stalled
Sprint: 10.1.29

 Description   

http://buildbot.askmonty.org/buildbot/builders/kvm-bintar-quantal-amd64/builds/3377/steps/test/logs/stdio

sys_vars.innodb_buffer_pool_dump_pct_basic 'xtradb' w2 [ fail ]
        Test ended at 2016-08-23 23:43:20
 
CURRENT_TEST: sys_vars.innodb_buffer_pool_dump_pct_basic
mysqltest: At line 23: command "file_exists" failed with error: 1  my_errno: 0  errno: 2
 
The result from queries just before the failure was:
SET @orig = @@global.innodb_buffer_pool_dump_pct;
SELECT @orig;
@orig
100
SET GLOBAL innodb_buffer_pool_dump_pct=3, GLOBAL innodb_buffer_pool_dump_now = ON;

By now there have been 5 recorded occurrences - on 10.0, 10.2, and 3 times on bb-10.1-serg.

Running the same sequence of tests does not cause the failure. There might be a race condition similar to the one recently fixed in another buffer_pool test.



 Comments   
Comment by Daniel Black [ 2016-12-05 ]

10.2 no longer includes a file_exist test. Attached PR re-enables test for 10.2

Comment by Daniel Black [ 2016-12-05 ]

Looking at the logs from the above build, http://buildbot.askmonty.org/buildbot/builders/kvm-bintar-quantal-amd64/builds/3377/steps/test/logs/mysqld.1.err.4 and http://buildbot.askmonty.org/buildbot/builders/kvm-bintar-quantal-amd64/builds/3377/steps/test/logs/mysqld.1.err.2, it appears as though the sys_vars.innodb_buffer_pool_dump_now_basic test was run immediately before the sys_vars.innodb_buffer_pool_dump_now_pct test.. As both set Innodb_buffer_pool_dump_status to the same value without a restart this unreliable result occurred.

For 10.0, 10.1, as MDEV-8923 in 10.0.23 introduced innodb_buffer_pool_dump_pct, and this test uses innodb_buffer_pool_dump_now as a mechanism, so the innodb_buffer_pool_dump_now test could be disabled.

As flush status doesn't reset Innodb_buffer_pool_dump_status. Alternately set a different file name and wait for that to occur in the error log before the wait loop on Innodb_buffer_pool_dump_status. Alternately ./include/restart_mysqld.inc. I might come up with a solution in MDEV-11454 / MDEV-11455 that can be backported.

Comment by Marko Mäkelä [ 2016-12-07 ]

https://github.com/MariaDB/server/pull/268 was merged to fix this bug in MariaDB Server 10.2.3.

Comment by Elena Stepanova [ 2016-12-07 ]

unstable-tests list is not about disabling tests, for that we have disabled.def lists. unstable-tests is a list of tests which proved to be unstable, or didn't prove to be stable. If you modify the test, it cannot be "stable" right away, we don't know about it; thus, it shouldn't be removed from the list. The list is updated separately before releases, based on buildbot history.

Comment by Elena Stepanova [ 2016-12-11 ]

Okay, so, the patch was only pushed in 10.2. Saving the default value is redundant (because the test sets it at the end to Default anyway), but innocent, so let it be. unstable-tests list in 10.2 is meaningless, it shouldn't even be there, just got merged along with everything else. When the time comes, it will be created from scratch, it doesn't matter what's there now, so I won't roll back the change.

The real problem is that the fix has nothing to do with the bug, I wish the commit message didn't mention this JIRA item. The problem for 10.0 and 10.1 remains (and no, we can't just disable the test as long as the variable exists). I'll keep the bug open, even though it will create a mess in the history, hopefully it won't matter much since it's just a test.

Please in future create a different JIRA item.

Comment by Elena Stepanova [ 2016-12-11 ]

Recent occurrence on 10.1:
http://buildbot.askmonty.org/buildbot/builders/kvm-deb-sid-amd64/builds/2425/steps/test_5/logs/stdio

Comment by Elena Stepanova [ 2017-11-03 ]

It's the same issue as MDEV-9713 (with a different test), reproducible in the same way:

diff --git a/storage/xtradb/buf/buf0dump.cc b/storage/xtradb/buf/buf0dump.cc
index 51c41cc..54c3336 100644
--- a/storage/xtradb/buf/buf0dump.cc
+++ b/storage/xtradb/buf/buf0dump.cc
@@ -215,6 +215,8 @@ buf_dump(
 
        ut_snprintf(tmp_filename, sizeof(tmp_filename),
                    "%s.incomplete", full_filename);
+        
+        my_sleep(1000000);
 
        buf_dump_status(STATUS_NOTICE, "Dumping buffer pool(s) to %s",
                        full_filename);

$ perl ./mtr sys_vars.innodb_buffer_pool_filename_basic,xtradb sys_vars.innodb_buffer_pool_dump_pct_basic,xtradb --mem --noreorder
...
sys_vars.innodb_buffer_pool_dump_pct_basic 'xtradb' [ fail ]
        Test ended at 2017-11-03 16:22:29
 
CURRENT_TEST: sys_vars.innodb_buffer_pool_dump_pct_basic
mysqltest: At line 23: command "file_exists" failed with error: 1  my_errno: 0  errno: 2
 
The result from queries just before the failure was:
SET @orig = @@global.innodb_buffer_pool_dump_pct;
SELECT @orig;
@orig
100
SET GLOBAL innodb_buffer_pool_dump_pct=3, GLOBAL innodb_buffer_pool_dump_now = ON;

Comment by Elena Stepanova [ 2017-11-03 ]

Another test which still fails with a similar error is

CURRENT_TEST: sys_vars.innodb_buffer_pool_dump_at_shutdown_basic
mysqltest: At line 42: command "file_exists" failed with error: 1  my_errno: 175  errno: 2

However, on the second look, it doesn't seem to be the same problem. I've filed it separately as MDEV-14280.

sys_vars.innodb_buffer_pool_load_now_basic doesn't fail this way after the fix MDEV-9713, but it fails with the timeout MDEV-14196 (especially on embedded), so the fix must be revised rather than repeated.

Comment by Elena Stepanova [ 2017-11-03 ]

sys_vars.innodb_buffer_pool_dump_now_basic is prone to the same error and reproducible in the same way as the original MDEV-9713, but it has never happened in buildbot, apparently just due to the lucky ordering of tests – innodb_buffer_pool_filename_basic never goes before innodb_buffer_pool_dump_now_basic.

Generated at Thu Feb 08 07:43:52 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.