[MDEV-11802] innodb.innodb_bug14676111 fails in buildbot due to InnoDB purge failing to start when there is work to do - Jira

Details

Type: Bug
Status: Closed (View Workflow)
Priority: Major
Resolution: Fixed
Affects Version/s: 10.1(EOL)
Fix Version/s: 10.2.7, 10.3.1
Component/s: Storage Engine - InnoDB, Tests
Labels:
None

Sprint:
10.2.4-1, 10.2.4-2

Description

http://buildbot.askmonty.org/buildbot/builders/p8-rhel7-bintar-debug/builds/2018/steps/test/logs/stdio

innodb.innodb_bug14676111 'innodb_plugin' w1 [ fail ]

        Test ended at 2017-01-05 22:06:59

CURRENT_TEST: innodb.innodb_bug14676111

--- /home/buildbot/maria-slave/power8-vlp03-bintar-debug/build/mysql-test/suite/innodb/r/innodb_bug14676111.result	2017-01-05 19:54:53.473261302 -0500

+++ /home/buildbot/maria-slave/power8-vlp03-bintar-debug/build/mysql-test/suite/innodb/r/innodb_bug14676111.reject	2017-01-05 22:06:58.634010067 -0500

@@ -22,7 +22,7 @@

 test.t1	analyze	status	OK

 select CLUST_INDEX_SIZE from information_schema.INNODB_SYS_TABLESTATS where NAME = 'test/t1';

 CLUST_INDEX_SIZE

-8

+10

 delete from t1 where a=5;

 set global innodb_purge_stop_now=ON;

 set global innodb_purge_run_now=ON;

mysqltest: Result length mismatch

Started happening quite regularly in buildbot on 10.1 tree since January 5, 2017.
Observed on win32-debug, winx64-debug, p8-rhel7-bintar-debug.

Attachments

Issue Links

causes

MDEV-12708 innodb.truncate_purge_debug fails sporadically in buildbot

Closed

MDEV-13603 innodb_fast_shutdown=0 may fail to purge all history

Closed

is caused by

MDEV-4396 Fix innodb.innodb_bug14676111 test

Closed

relates to

MDEV-11947 InnoDB purge workers fail to shut down

Closed

MDEV-12708 innodb.truncate_purge_debug fails sporadically in buildbot

Closed

MDEV-13603 innodb_fast_shutdown=0 may fail to purge all history

Closed

MDEV-13779 InnoDB fails to shut down purge workers, causing hang

Closed

MDEV-16159 Use atomic memory access for purge_sys

Closed

MDEV-17745 innodb.innodb_stats_persistent failed in buildbot with wrong result

Closed

MDEV-18878 Purge is slow due to repeatedly looking up missing or discarded table

Closed

MDEV-24402 CHECK TABLE may miss some cases of index inconsistencies

Closed

MDEV-26193 Skip check to invoke purge if the transaction is read-only

Closed

MDEV-30671 innodb_undo_log_truncate=ON fails to wait for purge of transaction history

Closed

MDEV-7069 Fix buildbot failures in main server trees

Stalled

MDEV-8139 Fix scrubbing

Closed

MDEV-11024 Make number of purge threads variable dynamic

Closed

MDEV-12463 Unable to find a record to delete-mark just after upgrading from 10.2.4 to 10.2.5

Closed

MDEV-13603 innodb_fast_shutdown=0 may fail to purge all history

Closed

MDEV-14080 InnoDB shutdown sometimes hangs

Closed

MDEV-14445 innodb.table_flags failed in buildbot with wrong result

Closed

MDEV-15826 Purge attempts to free BLOB page after BEGIN;INSERT;UPDATE;ROLLBACK

Closed

MDEV-16260 Scale the purge effort according to the workload

Open

MDEV-16264 Implement a common work queue for InnoDB background tasks

Closed

(18 relates to)

Activity

Ascending order - Click to sort in descending order

View 21 older comments

Marko Mäkelä added a comment - 2018-04-09 12:46

svoj, please note that I rewrote the test so that it does not rely on purge any more, but on rollback instead.

I also worked around possible lost signals by making SHOW ENGINE INNODB STATUS trigger purge. To better analyze this problem, these changes should be reverted.

Marko Mäkelä added a comment - 2018-04-09 12:46 svoj , please note that I rewrote the test so that it does not rely on purge any more, but on rollback instead. I also worked around possible lost signals by making SHOW ENGINE INNODB STATUS trigger purge . To better analyze this problem, these changes should be reverted.

Marko Mäkelä added a comment - 2018-05-17 08:28

As noted in ~~MDEV-13779~~, srv_release_threads() can fail to wake up some purge worker threads. ~~MDEV-13779~~ introduced an extra check to ensure that all purge workers will be woken up at shutdown, but they might occasionally remain sleeping during normal operation.

The fix of ~~MDEV-12708~~ in MariaDB 10.2.6 introduced the work-around that SHOW ENGINE INNODB STATUS will wake up purge threads. This seems to work fine in wait_all_purged.inc.

With ~~MDEV-13603~~ in MariaDB 10.3.6, innodb_fast_shutdown=0 should always run full purge. Older versions could fail to purge some of the last active transactions.

Marko Mäkelä added a comment - 2018-05-17 08:28 As noted in MDEV-13779 , srv_release_threads() can fail to wake up some purge worker threads. MDEV-13779 introduced an extra check to ensure that all purge workers will be woken up at shutdown, but they might occasionally remain sleeping during normal operation. The fix of MDEV-12708 in MariaDB 10.2.6 introduced the work-around that SHOW ENGINE INNODB STATUS will wake up purge threads . This seems to work fine in wait_all_purged.inc . With MDEV-13603 in MariaDB 10.3.6, innodb_fast_shutdown=0 should always run full purge. Older versions could fail to purge some of the last active transactions.

Marko Mäkelä added a comment - 2018-11-01 10:37

With the current work-around (SHOW ENGINE INNODB STATUS will initiate a purge), this is not a practical problem any more.
The mysql-test/suite/innodb/include/wait_all_purged.inc appears to be working reliably both in non-debug and debug builds of the server.

Marko Mäkelä added a comment - 2018-11-01 10:37 With the current work-around ( SHOW ENGINE INNODB STATUS will initiate a purge), this is not a practical problem any more. The mysql-test/suite/innodb/include/wait_all_purged.inc appears to be working reliably both in non-debug and debug builds of the server.

Marko Mäkelä added a comment - 2018-11-01 10:41

The wait_all_purged.inc was introduced in ~~MDEV-12698~~ (10.2.7, 10.3.1).
I do not remember seeing failures related to this lately.
The actual problem still exists (purge may be idling when there is work to do), but it does not show up in tests, because wait_all_purged.inc will trigger purge by issuing SHOW ENGINE INNODB STATUS. MDEV-16260 should fix the problem and remove the work-around of triggering the purge when executing that statement.

Marko Mäkelä added a comment - 2018-11-01 10:41 The wait_all_purged.inc was introduced in MDEV-12698 (10.2.7, 10.3.1). I do not remember seeing failures related to this lately. The actual problem still exists (purge may be idling when there is work to do), but it does not show up in tests, because wait_all_purged.inc will trigger purge by issuing SHOW ENGINE INNODB STATUS . MDEV-16260 should fix the problem and remove the work-around of triggering the purge when executing that statement.

Marko Mäkelä added a comment - 2023-02-24 10:37

I think that the original problem (that purge fails to remove some history of committed transactions) might be fixed by ~~MDEV-30671~~. There appears to be a correctness problem. When discarding rollback segments, purge only considers currently active transactions, and does not check if all history contained in the rollback segment actually has been purged.

The CHECK TABLE…EXTENDED implemented in ~~MDEV-24402~~ will warn about history that should have been purged. The history would be purged when rebuilding the table, for example, by OPTIMIZE TABLE.

Marko Mäkelä added a comment - 2023-02-24 10:37 I think that the original problem (that purge fails to remove some history of committed transactions) might be fixed by MDEV-30671 . There appears to be a correctness problem. When discarding rollback segments, purge only considers currently active transactions, and does not check if all history contained in the rollback segment actually has been purged. The CHECK TABLE…EXTENDED implemented in MDEV-24402 will warn about history that should have been purged. The history would be purged when rebuilding the table, for example, by OPTIMIZE TABLE .

People

Assignee:: Marko Mäkelä

Reporter:: Elena Stepanova

Votes:: 0 Vote for this issue

Watchers:: 7 Start watching this issue

Dates

Created:: 2017-01-15 22:39

Updated:: 2023-02-24 10:37

Resolved:: 2018-11-01 10:41

Git Integration

Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.

MariaDB Server

Details

Description

Attachments

Issue Links

Activity

People

Dates

Git Integration