[MDEV-33285] Assertion `m_table' failed in ha_perfschema::rnd_end on CHECKSUM TABLE - Jira

Details

Type: Bug
Status: Closed (View Workflow)
Priority: Major
Resolution: Fixed
Affects Version/s: 11.4, 11.8
Fix Version/s: 10.5.28, 10.6.21, 10.11.11, 11.4.5, 11.7.2
Component/s: Server
Labels:
- sporadic

Description

SET @@max_statement_time=0.0001;

CHECKSUM TABLE performance_schema.rwlock_instances;

CHECKSUM TABLE performance_schema.table_io_waits_summary_by_table;

CHECKSUM TABLE performance_schema.users;

CHECKSUM TABLE performance_schema.cond_instances;

Leads to:

11.4.0 57618265ad914824ce78108729829df13c75e224
mariadbd: /test/bb-11.4-MDEV-7850_opt/storage/perfschema/ha_perfschema.cc:341: virtual int ha_perfschema::rnd_end(): Assertion `m_table' failed.

Regrettably no full stack is available:

11.4.0 57618265ad914824ce78108729829df13c75e224 (Optimized)
Core was generated by `/test/MDEV-7850_MD161223-mariadb-11.4.0-linux-x86_64-opt/bin/mariadbd --no-defa'.
Program terminated with signal SIGABRT, Aborted.
#0 0x00001529ba6969fc in __pthread_mutex_lock_full (mutex=0x3541d3)
at ./nptl/pthread_mutex_lock.c:577
[Current thread is 1 (LWP 3490259)]
(gdb) bt
#0 0x00001529ba6969fc in __pthread_mutex_lock_full (mutex=0x3541d3) at ./nptl/pthread_mutex_lock.c:577
#1 0x0000000000000000 in ?? ()

However an earlier run captured the following frames:

11.4.0 57618265ad914824ce78108729829df13c75e224 (Optimized)
#1 ha_perfschema::rnd_end
#2 ha_perfschema::rnd_end
#3 handler::ha_rnd_end
#4 mysql_checksum_table

Attachments

Issue Links

relates to

MDEV-34860 Implement MAX_EXECUTION_TIME hint

Stalled

Activity

Ascending order - Click to sort in descending order

Roel Van de Paar created issue - 2024-01-20 00:16

Roel Van de Paar made changes - 2024-01-20 00:18

Field	Original Value	New Value
Description	Non-important issue at all, just logging so it can be added to filters. Extremely sporadic. Expect run times of 1h+ even with 200 threads repeating the SQL. Only one of the CHECKSUM TABLE statements is required, they can all fail with the said assertion. {code:sql} SET @@max_statement_time=0.0001; CHECKSUM TABLE performance_schema.rwlock_instances; CHECKSUM TABLE performance_schema.table_io_waits_summary_by_table; CHECKSUM TABLE performance_schema.users; {code} Leads to: {noformat:title=11.4.0 57618265ad914824ce78108729829df13c75e224} mariadbd: /test/bb-11.4-~~MDEV-7850~~_opt/storage/perfschema/ha_perfschema.cc:341: virtual int ha_perfschema::rnd_end(): Assertion `m_table' failed. {noformat} Regrettably no full stack is available: {noformat:title=11.4.0 57618265ad914824ce78108729829df13c75e224 (Optimized)} Core was generated by `/test/~~MDEV-7850~~_MD161223-mariadb-11.4.0-linux-x86_64-opt/bin/mariadbd --no-defa'. Program terminated with signal SIGABRT, Aborted. #0 0x00001529ba6969fc in __pthread_mutex_lock_full (mutex=0x3541d3) at ./nptl/pthread_mutex_lock.c:577 [Current thread is 1 (LWP 3490259)] (gdb) bt #0 0x00001529ba6969fc in __pthread_mutex_lock_full (mutex=0x3541d3) at ./nptl/pthread_mutex_lock.c:577 #1 0x0000000000000000 in ?? () {noformat} However an earlier run captured the following frames: {noformat:title=11.4.0 57618265ad914824ce78108729829df13c75e224 (Optimized)} #1 ha_perfschema::rnd_end #2 ha_perfschema::rnd_end #3 handler::ha_rnd_end #4 mysql_checksum_table {noformat}	Non-important issue at all, just logging so it can be added to filters. Extremely sporadic. Expect run times of 1h+ even with 200 threads repeating the SQL. Only one of the CHECKSUM TABLE statements is required, they can all fail with the given assertion. {code:sql} SET @@max_statement_time=0.0001; CHECKSUM TABLE performance_schema.rwlock_instances; CHECKSUM TABLE performance_schema.table_io_waits_summary_by_table; CHECKSUM TABLE performance_schema.users; {code} Leads to: {noformat:title=11.4.0 57618265ad914824ce78108729829df13c75e224} mariadbd: /test/bb-11.4-~~MDEV-7850~~_opt/storage/perfschema/ha_perfschema.cc:341: virtual int ha_perfschema::rnd_end(): Assertion `m_table' failed. {noformat} Regrettably no full stack is available: {noformat:title=11.4.0 57618265ad914824ce78108729829df13c75e224 (Optimized)} Core was generated by `/test/~~MDEV-7850~~_MD161223-mariadb-11.4.0-linux-x86_64-opt/bin/mariadbd --no-defa'. Program terminated with signal SIGABRT, Aborted. #0 0x00001529ba6969fc in __pthread_mutex_lock_full (mutex=0x3541d3) at ./nptl/pthread_mutex_lock.c:577 [Current thread is 1 (LWP 3490259)] (gdb) bt #0 0x00001529ba6969fc in __pthread_mutex_lock_full (mutex=0x3541d3) at ./nptl/pthread_mutex_lock.c:577 #1 0x0000000000000000 in ?? () {noformat} However an earlier run captured the following frames: {noformat:title=11.4.0 57618265ad914824ce78108729829df13c75e224 (Optimized)} #1 ha_perfschema::rnd_end #2 ha_perfschema::rnd_end #3 handler::ha_rnd_end #4 mysql_checksum_table {noformat}

Sergey Vojtovich made changes - 2024-11-08 15:15

Assignee

Sergey Vojtovich [ svoj ]

Sergey Vojtovich made changes - 2024-11-08 15:15

Status

Open [ 1 ]

In Progress [ 3 ]

Sergey Vojtovich added a comment - 2024-11-08 15:16

Note that this issue is not PFS specific.

Sergey Vojtovich added a comment - 2024-11-08 15:16 Note that this issue is not PFS specific.

Sergey Vojtovich made changes - 2024-11-08 15:16

Assignee	Sergey Vojtovich [ svoj ]	Sergei Golubchik [ serg ]
Status	In Progress [ 3 ]	In Review [ 10002 ]

Roel Van de Paar added a comment - 2025-01-15 03:52 - edited

MDEV-34860 (or any other related feature pushed to bb-11.8-MDEV-35504-opt-hints) Seems to make this issue more pronounced (occur more often), as well as creating a stack where mysql_checksum_table is the third frame. This time I was able to capture the full stack:

bb-11.8-MDEV-35504-opt-hints CS 11.8.0 6d92aa1cf0a042e506189b6739e03a8b45e20ee7 (Optimized)
mariadbd: /test/bb-11.8-MDEV-35504-opt-hints_opt/storage/perfschema/ha_perfschema.cc:339: virtual int ha_perfschema::rnd_end(): Assertion `m_table' failed.

bb-11.8-MDEV-35504-opt-hints CS 11.8.0 6d92aa1cf0a042e506189b6739e03a8b45e20ee7 (Optimized)
Core was generated by `/test/MDEV-35504_MD100125-mariadb-11.8.0-linux-x86_64-opt/bin/mariadbd --no-def'.
Program terminated with signal SIGABRT, Aborted.
#0 __pthread_kill_implementation (no_tid=0, signo=6, threadid=<optimized out>)at ./nptl/pthread_kill.c:44

[Current thread is 1 (LWP 3589503)]
(gdb) bt
#0 __pthread_kill_implementation (no_tid=0, signo=6, threadid=<optimized out>)at ./nptl/pthread_kill.c:44
#1 __pthread_kill_internal (signo=6, threadid=<optimized out>)at ./nptl/pthread_kill.c:78
#2 __GI___pthread_kill (threadid=<optimized out>, signo=signo@entry=6)at ./nptl/pthread_kill.c:89
#3 0x0000145b7584526e in __GI_raise (sig=sig@entry=6)at ../sysdeps/posix/raise.c:26
#4 0x0000145b758288ff in __GI_abort () at ./stdlib/abort.c:79
#5 0x0000145b7582881b in __assert_fail_base (fmt=0x145b759d01e8 "%s%s%s:%u: %s%sAssertion `%s' failed.\n%n", assertion=assertion@entry=0x561229f176a2 "m_table", file=file@entry=0x561229ecb9c2 "/test/bb-11.8-MDEV-35504-opt-hints_opt/storage/perfschema/ha_perfschema.cc", line=line@entry=339, function=function@entry=0x561229ecba5b "virtual int ha_perfschema::rnd_end()") at ./assert/assert.c:94
#6 0x0000145b7583b507 in __assert_fail (assertion=0x561229f176a2 "m_table", file=0x561229ecb9c2 "/test/bb-11.8-MDEV-35504-opt-hints_opt/storage/perfschema/ha_perfschema.cc", line=339, function=0x561229ecba5b "virtual int ha_perfschema::rnd_end()")at ./assert/assert.c:103
#7 0x0000561229991e0e in ha_perfschema::rnd_end (this=0x145abcbe57a0)at /test/bb-11.8-MDEV-35504-opt-hints_opt/storage/perfschema/ha_perfschema.cc:339
#8 0x00005612295a324e in handler::ha_rnd_end (this=0x36afd2)at /test/bb-11.8-MDEV-35504-opt-hints_opt/sql/handler.h:3562
#9 mysql_checksum_table (thd=thd@entry=0x145b14000c68, tables=tables@entry=0x145b140183d0, check_opt=check_opt@entry=0x145b14006330)at /test/bb-11.8-MDEV-35504-opt-hints_opt/sql/sql_table.cc:12983
#10 0x00005612294e3b44 in mysql_execute_command (thd=thd@entry=0x145b14000c68, is_called_from_prepared_stmt=false)at /test/bb-11.8-MDEV-35504-opt-hints_opt/sql/sql_parse.cc:4404
#11 0x00005612294dd321 in mysql_parse (thd=thd@entry=0x145b14000c68, rawbuf=<optimized out>, length=<optimized out>, parser_state=parser_state@entry=0x145b7427c420)at /test/bb-11.8-MDEV-35504-opt-hints_opt/sql/sql_parse.cc:7901
#12 0x00005612294db7b0 in dispatch_command (command=command@entry=COM_QUERY, thd=thd@entry=0x145b14000c68, packet=packet@entry=0x145abcd12b79 "", packet_length=packet_length@entry=66, blocking=true)at /test/bb-11.8-MDEV-35504-opt-hints_opt/sql/sql_parse.cc:1903
#13 0x00005612294dd731 in do_command (thd=thd@entry=0x145b14000c68, blocking=true)at /test/bb-11.8-MDEV-35504-opt-hints_opt/sql/sql_parse.cc:1416
#14 0x000056122960868d in do_handle_one_connection (connect=<optimized out>, connect@entry=0x56122c02b308, put_in_cache=true)at /test/bb-11.8-MDEV-35504-opt-hints_opt/sql/sql_connect.cc:1438
#15 0x000056122960844f in handle_one_connection (arg=arg@entry=0x56122c02b308)at /test/bb-11.8-MDEV-35504-opt-hints_opt/sql/sql_connect.cc:1350
#16 0x00005612299942d9 in pfs_spawn_thread (arg=0x56122bfd33d8)at /test/bb-11.8-MDEV-35504-opt-hints_opt/storage/perfschema/pfs.cc:2198
#17 0x0000145b7589ca94 in start_thread (arg=<optimized out>)at ./nptl/pthread_create.c:447
#18 0x0000145b75929c3c in clone3 ()at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:78

The semi-reduced (and again sporadic) testcase was in this case:

INSERT INTO t1  VALUES (CONVERT(_ucs2 0x0645062C06440633 USING utf8));

UNLOCK TABLES;

SET MAX_STATEMENT_TIME=0.000001;

XA RECOVER FORMAT='SQL';

INSERT DELAYED INTO t1  VALUES(33);

drop table mt1,mt2;

SELECT 1;

CHECKSUM TABLE performance_schema.table_io_waits_summary_by_table; ;

; ;

SELECT 1;

CHECKSUM TABLE performance_schema.table_io_waits_summary_by_table;#

SELECT SLEEP(2);

Which clearly is the same bug/issue.

Roel Van de Paar added a comment - 2025-01-15 03:52 - edited MDEV-34860 (or any other related feature pushed to bb-11.8- MDEV-35504 -opt-hints ) Seems to make this issue more pronounced (occur more often), as well as creating a stack where mysql_checksum_table is the third frame. This time I was able to capture the full stack: bb-11.8-MDEV-35504-opt-hints CS 11.8.0 6d92aa1cf0a042e506189b6739e03a8b45e20ee7 (Optimized) mariadbd: /test/bb-11.8-MDEV-35504-opt-hints_opt/storage/perfschema/ha_perfschema.cc:339: virtual int ha_perfschema::rnd_end(): Assertion `m_table' failed. bb-11.8-MDEV-35504-opt-hints CS 11.8.0 6d92aa1cf0a042e506189b6739e03a8b45e20ee7 (Optimized) Core was generated by `/test/MDEV-35504_MD100125-mariadb-11.8.0-linux-x86_64-opt/bin/mariadbd --no-def'. Program terminated with signal SIGABRT, Aborted. #0 __pthread_kill_implementation (no_tid=0, signo=6, threadid=<optimized out>)at ./nptl/pthread_kill.c:44 [Current thread is 1 (LWP 3589503)] (gdb) bt #0 __pthread_kill_implementation (no_tid=0, signo=6, threadid=<optimized out>)at ./nptl/pthread_kill.c:44 #1 __pthread_kill_internal (signo=6, threadid=<optimized out>)at ./nptl/pthread_kill.c:78 #2 __GI___pthread_kill (threadid=<optimized out>, signo=signo@entry=6)at ./nptl/pthread_kill.c:89 #3 0x0000145b7584526e in __GI_raise (sig=sig@entry=6)at ../sysdeps/posix/raise.c:26 #4 0x0000145b758288ff in __GI_abort () at ./stdlib/abort.c:79 #5 0x0000145b7582881b in __assert_fail_base (fmt=0x145b759d01e8 "%s%s%s:%u: %s%sAssertion `%s' failed.\n%n", assertion=assertion@entry=0x561229f176a2 "m_table", file=file@entry=0x561229ecb9c2 "/test/bb-11.8-MDEV-35504-opt-hints_opt/storage/perfschema/ha_perfschema.cc", line=line@entry=339, function=function@entry=0x561229ecba5b "virtual int ha_perfschema::rnd_end()") at ./assert/assert.c:94 #6 0x0000145b7583b507 in __assert_fail (assertion=0x561229f176a2 "m_table", file=0x561229ecb9c2 "/test/bb-11.8-MDEV-35504-opt-hints_opt/storage/perfschema/ha_perfschema.cc", line=339, function=0x561229ecba5b "virtual int ha_perfschema::rnd_end()")at ./assert/assert.c:103 #7 0x0000561229991e0e in ha_perfschema::rnd_end (this=0x145abcbe57a0)at /test/bb-11.8-MDEV-35504-opt-hints_opt/storage/perfschema/ha_perfschema.cc:339 #8 0x00005612295a324e in handler::ha_rnd_end (this=0x36afd2)at /test/bb-11.8-MDEV-35504-opt-hints_opt/sql/handler.h:3562 #9 mysql_checksum_table (thd=thd@entry=0x145b14000c68, tables=tables@entry=0x145b140183d0, check_opt=check_opt@entry=0x145b14006330)at /test/bb-11.8-MDEV-35504-opt-hints_opt/sql/sql_table.cc:12983 #10 0x00005612294e3b44 in mysql_execute_command (thd=thd@entry=0x145b14000c68, is_called_from_prepared_stmt=false)at /test/bb-11.8-MDEV-35504-opt-hints_opt/sql/sql_parse.cc:4404 #11 0x00005612294dd321 in mysql_parse (thd=thd@entry=0x145b14000c68, rawbuf=<optimized out>, length=<optimized out>, parser_state=parser_state@entry=0x145b7427c420)at /test/bb-11.8-MDEV-35504-opt-hints_opt/sql/sql_parse.cc:7901 #12 0x00005612294db7b0 in dispatch_command (command=command@entry=COM_QUERY, thd=thd@entry=0x145b14000c68, packet=packet@entry=0x145abcd12b79 "", packet_length=packet_length@entry=66, blocking=true)at /test/bb-11.8-MDEV-35504-opt-hints_opt/sql/sql_parse.cc:1903 #13 0x00005612294dd731 in do_command (thd=thd@entry=0x145b14000c68, blocking=true)at /test/bb-11.8-MDEV-35504-opt-hints_opt/sql/sql_parse.cc:1416 #14 0x000056122960868d in do_handle_one_connection (connect=<optimized out>, connect@entry=0x56122c02b308, put_in_cache=true)at /test/bb-11.8-MDEV-35504-opt-hints_opt/sql/sql_connect.cc:1438 #15 0x000056122960844f in handle_one_connection (arg=arg@entry=0x56122c02b308)at /test/bb-11.8-MDEV-35504-opt-hints_opt/sql/sql_connect.cc:1350 #16 0x00005612299942d9 in pfs_spawn_thread (arg=0x56122bfd33d8)at /test/bb-11.8-MDEV-35504-opt-hints_opt/storage/perfschema/pfs.cc:2198 #17 0x0000145b7589ca94 in start_thread (arg=<optimized out>)at ./nptl/pthread_create.c:447 #18 0x0000145b75929c3c in clone3 ()at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:78 The semi-reduced (and again sporadic) testcase was in this case: INSERT INTO t1 VALUES ( CONVERT (_ucs2 0x0645062C06440633 USING utf8)); UNLOCK TABLES; SET MAX_STATEMENT_TIME=0.000001; XA RECOVER FORMAT= 'SQL' ; INSERT DELAYED INTO t1 VALUES (33); drop table mt1,mt2; SELECT 1; CHECKSUM TABLE performance_schema.table_io_waits_summary_by_table; ; ; ; SELECT 1; CHECKSUM TABLE performance_schema.table_io_waits_summary_by_table;# SELECT SLEEP(2); Which clearly is the same bug/issue.

Roel Van de Paar made changes - 2025-01-15 03:54

Link

This issue relates to MDEV-34860 [ MDEV-34860 ]

Roel Van de Paar made changes - 2025-01-15 03:59

Description

Non-important issue at all, just logging so it can be added to filters. Extremely sporadic. Expect run times of 1h+ even with 200 threads repeating the SQL. Only one of the CHECKSUM TABLE statements is required, they can all fail with the given assertion.
{code:sql}
SET @@max_statement_time=0.0001;
CHECKSUM TABLE performance_schema.rwlock_instances;
CHECKSUM TABLE performance_schema.table_io_waits_summary_by_table;
CHECKSUM TABLE performance_schema.users;
{code}
Leads to:
{noformat:title=11.4.0 57618265ad914824ce78108729829df13c75e224}
mariadbd: /test/bb-11.4-~~MDEV-7850~~_opt/storage/perfschema/ha_perfschema.cc:341: virtual int ha_perfschema::rnd_end(): Assertion `m_table' failed.
{noformat}
Regrettably no full stack is available:
{noformat:title=11.4.0 57618265ad914824ce78108729829df13c75e224 (Optimized)}
Core was generated by `/test/~~MDEV-7850~~_MD161223-mariadb-11.4.0-linux-x86_64-opt/bin/mariadbd --no-defa'.
Program terminated with signal SIGABRT, Aborted.
#0 0x00001529ba6969fc in __pthread_mutex_lock_full (mutex=0x3541d3)
at ./nptl/pthread_mutex_lock.c:577
[Current thread is 1 (LWP 3490259)]
(gdb) bt
#0 0x00001529ba6969fc in __pthread_mutex_lock_full (mutex=0x3541d3) at ./nptl/pthread_mutex_lock.c:577
#1 0x0000000000000000 in ?? ()
{noformat}
However an earlier run captured the following frames:
{noformat:title=11.4.0 57618265ad914824ce78108729829df13c75e224 (Optimized)}
#1 ha_perfschema::rnd_end
#2 ha_perfschema::rnd_end
#3 handler::ha_rnd_end
#4 mysql_checksum_table
{noformat}

Non-important issue at all, just logging so it can be added to filters. Extremely sporadic. Expect run times of 1h+ even with 200 threads repeating the SQL. Only one of the CHECKSUM TABLE statements is required, they can all fail with the given assertion.
{code:sql}
SET @@max_statement_time=0.0001;
CHECKSUM TABLE performance_schema.rwlock_instances;
CHECKSUM TABLE performance_schema.table_io_waits_summary_by_table;
CHECKSUM TABLE performance_schema.users;
CHECKSUM TABLE performance_schema.cond_instances;
{code}
Leads to:
{noformat:title=11.4.0 57618265ad914824ce78108729829df13c75e224}
mariadbd: /test/bb-11.4-~~MDEV-7850~~_opt/storage/perfschema/ha_perfschema.cc:341: virtual int ha_perfschema::rnd_end(): Assertion `m_table' failed.
{noformat}
Regrettably no full stack is available:
{noformat:title=11.4.0 57618265ad914824ce78108729829df13c75e224 (Optimized)}
Core was generated by `/test/~~MDEV-7850~~_MD161223-mariadb-11.4.0-linux-x86_64-opt/bin/mariadbd --no-defa'.
Program terminated with signal SIGABRT, Aborted.
#0 0x00001529ba6969fc in __pthread_mutex_lock_full (mutex=0x3541d3)
at ./nptl/pthread_mutex_lock.c:577
[Current thread is 1 (LWP 3490259)]
(gdb) bt
#0 0x00001529ba6969fc in __pthread_mutex_lock_full (mutex=0x3541d3) at ./nptl/pthread_mutex_lock.c:577
#1 0x0000000000000000 in ?? ()
{noformat}
However an earlier run captured the following frames:
{noformat:title=11.4.0 57618265ad914824ce78108729829df13c75e224 (Optimized)}
#1 ha_perfschema::rnd_end
#2 ha_perfschema::rnd_end
#3 handler::ha_rnd_end
#4 mysql_checksum_table
{noformat}

Roel Van de Paar made changes - 2025-01-15 03:59

Priority

Minor [ 4 ]

Major [ 3 ]

Roel Van de Paar made changes - 2025-01-15 03:59

Affects Version/s

11.8 [ 29921 ]

Sergei Golubchik made changes - 2025-01-22 14:04

Status

In Review [ 10002 ]

In Testing [ 10301 ]

Sergei Golubchik added a comment - 2025-01-22 14:04

ok to push

Sergei Golubchik added a comment - 2025-01-22 14:04 ok to push

Sergei Golubchik made changes - 2025-01-22 14:32

Fix Version/s		10.5.28 [ 29952 ]
Fix Version/s		10.6.21 [ 29953 ]
Fix Version/s		10.11.11 [ 29954 ]
Fix Version/s		11.4.5 [ 29956 ]
Fix Version/s		11.7.2 [ 29914 ]
Fix Version/s	11.4 [ 29301 ]
Assignee	Sergei Golubchik [ serg ]	Sergey Vojtovich [ svoj ]
Resolution		Fixed [ 1 ]
Status	In Testing [ 10301 ]	Closed [ 6 ]

Sergei Golubchik made changes - 2025-01-22 14:33

Component/s		Server [ 13907 ]
Component/s	Performance Schema [ 15627 ]

People

Assignee:: Sergey Vojtovich

Reporter:: Roel Van de Paar

Votes:: 0 Vote for this issue

Watchers:: 3 Start watching this issue

Dates

Created:: 2024-01-20 00:16

Updated:: 2025-01-28 18:31

Resolved:: 2025-01-22 14:32

Git Integration

Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.

MariaDB Server

Details

Description

Attachments

Issue Links

Activity

People

Dates

Git Integration