[MDEV-22761] InnoDB: Assertion failure row0sel.cc line 3966 exception 0x80000003 Created: 2020-05-31  Updated: 2020-12-17  Resolved: 2020-10-13

Status: Closed
Project: MariaDB Server
Component/s: Server, Storage Engine - InnoDB
Affects Version/s: 10.4.13, 10.5.3, 10.5.5
Fix Version/s: 10.4.16, 10.5.7

Type: Bug Priority: Critical
Reporter: Rigoberto Munguia Assignee: Daniel Black
Resolution: Fixed Votes: 0
Labels: crash, innodb, not-10.1, not-10.2, not-10.3, regression, regression-10.4
Environment:

Windows Server 2016 VM, 16 core 40GB RAM.


Issue Links:
Duplicate
is duplicated by MDEV-23834 InnoDB: Assertion failure Closed
is duplicated by MDEV-24180 InnoDB: Assertion failure in row0sel.... Closed
is duplicated by MDEV-24428 mysqld got exception 0x80000003 Closed
Problem/Incident
is caused by MDEV-21794 Optimizer flag rowid_filter leads to ... Closed
Relates
relates to MDEV-16188 Use in-memory PK filters built from r... Closed
relates to MDEV-23938 innodb aborted by user in ICP fails t... Closed

 Description   

MariaDB Server crashes during select

2020-05-31 12:27:44 0x8ec  InnoDB: Assertion failure in file D:\winx64-packages\build\src\storage\innobase\row\row0sel.cc line 3966
InnoDB: We intentionally generate a memory trap.
InnoDB: Submit a detailed bug report to https://jira.mariadb.org/
InnoDB: If you get repeated assertion failures or crashes, even
InnoDB: immediately after the mysqld startup, there may be
InnoDB: corruption in the InnoDB tablespace. Please refer to
InnoDB: https://mariadb.com/kb/en/library/innodb-recovery-modes/
InnoDB: about forcing recovery.
200531 12:27:44 [ERROR] mysqld got exception 0x80000003 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
 
To report this bug, see https://mariadb.com/kb/en/reporting-bugs
 
We will try our best to scrape up some info that will hopefully help
diagnose the problem, but since we have already crashed, 
something is definitely wrong and this may fail.
 
Server version: 10.4.13-MariaDB-log
key_buffer_size=134217728
read_buffer_size=131072
max_used_connections=12
max_threads=65537
thread_count=9
It is possible that mysqld could use up to 
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 141521 K  bytes of memory
Hope that's ok; if not, decrease some variables in the equation.
 
Thread pointer: 0x1f23fd98a18
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
mysqld.exe!my_sigabrt_handler()[my_thr_init.c:485]
ucrtbase.dll!raise()
ucrtbase.dll!abort()
mysqld.exe!ut_dbg_assertion_failed()[ut0dbg.cc:60]
mysqld.exe!row_search_idx_cond_check()[row0sel.cc:3966]
mysqld.exe!row_search_mvcc()[row0sel.cc:5266]
mysqld.exe!ha_innobase::general_fetch()[ha_innodb.cc:9546]
mysqld.exe!handler::ha_index_prev()[handler.cc:2967]
mysqld.exe!join_read_prev_same()[sql_select.cc:21213]
mysqld.exe!sub_select()[sql_select.cc:20385]
mysqld.exe!do_select()[sql_select.cc:19904]
mysqld.exe!JOIN::exec_inner()[sql_select.cc:4459]
mysqld.exe!JOIN::exec()[sql_select.cc:4242]
mysqld.exe!mysql_select()[sql_select.cc:4675]
mysqld.exe!handle_select()[sql_select.cc:410]
mysqld.exe!execute_sqlcom_select()[sql_parse.cc:6359]
mysqld.exe!mysql_execute_command()[sql_parse.cc:3898]
mysqld.exe!mysql_parse()[sql_parse.cc:7904]
mysqld.exe!dispatch_command()[sql_parse.cc:1844]
mysqld.exe!do_command()[sql_parse.cc:1359]
mysqld.exe!threadpool_process_request()[threadpool_common.cc:365]
mysqld.exe!tp_callback()[threadpool_common.cc:192]
ntdll.dll!RtlReleaseSRWLockExclusive()
ntdll.dll!RtlReleaseSRWLockExclusive()
KERNEL32.DLL!BaseThreadInitThunk()
ntdll.dll!RtlUserThreadStart()
 
Trying to get some variables.
Some pointers may be invalid and cause the dump to abort.
Query (0x1f240537180): select a.session_id, a.user
				from redcap_log_view a
				inner join redcap_user_rights u ON a.user = u.username AND u.project_id = a.project_id
				inner join redcap_user_information i ON i.username = u.username
				left join redcap_user_roles ur ON u.role_id = ur.role_id
				where a.project_id = 6108
				and a.log_view_id >= 13349060
				and a.user != 'exxxxxx'
				and a.event_id = 31089
				and a.record = '878096793'
				and a.form_name = 'sedation'
				and a.page in ('DataEntry/index.php', 'ProjectGeneral/keep_alive.php', 'DataEntry/check_unique_ajax.php', 
							   'DataEntry/file_download.php', 'DataEntry/file_upload.php', 'DataEntry/file_delete.php')
				and (((u.data_entry like '%[sedation,1]%' OR u.data_entry like '%[sedation,3]%') AND ur.role_id IS NULL) 
					OR ((ur.data_entry LIKE '%[sedation,1]%' OR ur.data_entry LIKE '%[sedation,3]%') AND ur.role_id IS NOT NULL)
					OR i.super_user = 1)
				and a.log_view_id = (select b.log_view_id from redcap_log_view b where b.user = a.user order by b.log_view_id desc limit 1)
				
				order by a.log_view_id desc limit 1
Connection ID (thread ID): 21184
Status: KILL_CONNECTION
 
Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=on,condition_pushdown_from_having=on
 
The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
information that should help you find out what is causing the crash.
Writing a core file at G:\MariaDB\MariaDB 10.4\data\

----------
contents of my.ini
----------

[mysqld]
datadir=G:/MariaDB/MariaDB 10.4/data
port=3306
innodb_buffer_pool_size=20480M
character-set-server=utf8
performance_schema=on
max_connections=400
 
# The Pipe the MySQL Server will use
socket=MariaDB104
#shared_memory_base_name=MariaDB104
general-log=0
general_log_file="D:/MariaDB/MariaDB104/logs/general.log"
slow-query-log=1
slow_query_log_file="D:/MariaDB/MariaDB104/logs/slow-query.log"
long_query_time=20
 
# Error Logging.
log-error="D:/MariaDB/MariaDB104/logs/error.log"
log-warnings = 2
 
# Binary Logging.
# >> Removed the last sud-dir (binlog) <<
#log-bin = "D:/MySQL/MySQL56/binlog/binlog"
log-bin = "D:/MariaDB/MariaDB104/binlog/binlog"
expire_logs_days = 28
#binlog-ignore-db = tch
binlog_format = row
 
# Temporary location (multiple locations cas be set ';' is separator)
tmpdir=D:/MariaDB/MariaDB104/temp
 
# Query cache is used to cache SELECT results and later return them
# without actual executing the same query once again. Having the query
# cache enabled may result in significant speed improvements, if your
# have a lot of identical queries and rarely changing tables. See the
# "Qcache_lowmem_prunes" status variable to check if the current value
# is high enough for your load.
# Note: In case your tables change very often or if your queries are
# textually different every time, the query cache may result in a
# slowdown instead of a performance improvement.
# query_cache_size=192M
 
# Size of each log file in a log group. You should set the combined size
# of log files to about 25%-100% of your buffer pool size to avoid
# unneeded buffer pool flush activity on log file overwrite. However,
# note that a larger logfile size will increase the time needed for the
# recovery process.
innodb_log_file_size=512M
innodb_log_group_home_dir = "F:/MariaDB/MariaDB104/redo"
innodb_log_files_in_group = 4
 
# Number of threads allowed inside the InnoDB kernel. The optimal value
# depends highly on the application, hardware as well as the OS
# scheduler properties. A too high value may lead to thread thrashing.
# innodb_read_io_threads = 16
# innodb_write_io_threads = 16
 
# Set the query cache type. 0 for OFF, 1 for ON and 2 for DEMAND.
# query_cache_type=0
 
# The number of table definitions (from .frm files) that can be stored in the definition cache.
# If you use a large number of tables, you can create a large table definition cache to speed up opening of tables.
# The table definition cache takes less space and does not use file descriptors, unlike the normal table cache.
# The minimum and default values are both 400.
table_definition_cache=1400
 
# Enable the even-scheduler
# event-scheduler = ON
 
# Set server-id in order to read binlog with remote options
server-id = 1 # or something else unique
 
# Secure File Priv.
#secure-file-priv="C:\ProgramData\MySQL\MySQL Server 5.7\Uploads"
secure-file-priv="G:/MariaDB/MariaDB 10.4/spool"
 
[client]
port=3306
plugin-dir=D:/MariaDB/MariaDB 10.4/lib/plugin



 Comments   
Comment by Silver Asu [ 2020-07-06 ]

We had same buffer overflow on linux.

2020-07-06 10:34:13 0x7fb8861c6700  InnoDB: Assertion failure in file /build/amd64-usr/var/tmp/portage/dev-db/mariadb-10.4.13/work/mariadb-10.4.13/storage/innobase/row/row0sel.cc line 3966
InnoDB: We intentionally generate a memory trap.
InnoDB: Submit a detailed bug report to https://jira.mariadb.org/
InnoDB: If you get repeated assertion failures or crashes, even
InnoDB: immediately after the mysqld startup, there may be
InnoDB: corruption in the InnoDB tablespace. Please refer to
InnoDB: https://mariadb.com/kb/en/library/innodb-recovery-modes/
InnoDB: about forcing recovery.
200706 10:34:13 [ERROR] mysqld got signal 6 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
To report this bug, see https://mariadb.com/kb/en/reporting-bugs
We will try our best to scrape up some info that will hopefully help
diagnose the problem, but since we have already crashed,
something is definitely wrong and this may fail.
Server version: 10.4.13-MariaDB-log
key_buffer_size=4294967296
read_buffer_size=2097152
max_used_connections=135
max_threads=1502
thread_count=87
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 19611875 K  bytes of memory
Hope that's ok; if not, decrease some variables in the equation.
Thread pointer: 0x7fb7e402d538
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 0x7fb8861c5d98 thread_stack 0x49000
*** buffer overflow detected ***: /opt/zse/sbin/mysqld terminated

Comment by Mykhaylo Poboynyy [ 2020-07-20 ]

We have this Problem too. In our CI, will be every 2 Hours one database new created and over 1 hour is intensive in use. After 6 or 7 days we have the same crash.
(Windows 10 Prof. x64)

2020-07-17 14:48:05 0xf10 InnoDB: Assertion failure in file D:\winx64-packages\build\src\storage\innobase\row\row0sel.cc line 3966
InnoDB: We intentionally generate a memory trap.
InnoDB: Submit a detailed bug report to https://jira.mariadb.org/
InnoDB: If you get repeated assertion failures or crashes, even
InnoDB: immediately after the mysqld startup, there may be
InnoDB: corruption in the InnoDB tablespace. Please refer to
InnoDB: https://mariadb.com/kb/en/library/innodb-recovery-modes/
InnoDB: about forcing recovery.
200717 14:48:05 [ERROR] mysqld got exception 0x80000003 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.

To report this bug, see https://mariadb.com/kb/en/reporting-bugs

We will try our best to scrape up some info that will hopefully help
diagnose the problem, but since we have already crashed,
something is definitely wrong and this may fail.

Server version: 10.4.13-MariaDB
key_buffer_size=134217728
read_buffer_size=131072
max_used_connections=5
max_threads=65537
thread_count=8
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 136384 K bytes of memory
Hope that's ok; if not, decrease some variables in the equation.

Thread pointer: 0x1e925bae748
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
mysqld.exe!my_sigabrt_handler()[my_thr_init.c:485]
ucrtbase.dll!raise()
ucrtbase.dll!abort()
mysqld.exe!ut_dbg_assertion_failed()[ut0dbg.cc:60]
mysqld.exe!row_search_idx_cond_check()[row0sel.cc:3966]
mysqld.exe!row_search_mvcc()[row0sel.cc:5266]
mysqld.exe!ha_innobase::index_read()[ha_innodb.cc:9278]
mysqld.exe!handler::ha_index_read_map()[handler.cc:2896]
mysqld.exe!join_read_always_key()[sql_select.cc:21124]
mysqld.exe!sub_select()[sql_select.cc:20363]
mysqld.exe!evaluate_join_record()[sql_select.cc:20587]
mysqld.exe!sub_select()[sql_select.cc:20366]
mysqld.exe!JOIN_CACHE::generate_full_extensions()[sql_join_cache.cc:2403]
mysqld.exe!JOIN_CACHE::join_matching_records()[sql_join_cache.cc:2295]
mysqld.exe!JOIN_CACHE::join_records()[sql_join_cache.cc:2088]
mysqld.exe!sub_select_cache()[sql_select.cc:20145]
mysqld.exe!do_select()[sql_select.cc:19906]
mysqld.exe!JOIN::exec_inner()[sql_select.cc:4459]
mysqld.exe!JOIN::exec()[sql_select.cc:4242]
mysqld.exe!mysql_select()[sql_select.cc:4675]
mysqld.exe!handle_select()[sql_select.cc:410]
mysqld.exe!execute_sqlcom_select()[sql_parse.cc:6359]
mysqld.exe!mysql_execute_command()[sql_parse.cc:3898]
mysqld.exe!mysql_parse()[sql_parse.cc:7904]
mysqld.exe!dispatch_command()[sql_parse.cc:1844]
mysqld.exe!do_command()[sql_parse.cc:1359]
mysqld.exe!threadpool_process_request()[threadpool_common.cc:365]
mysqld.exe!tp_callback()[threadpool_common.cc:192]
ntdll.dll!TpReleaseWait()
ntdll.dll!RtlInitializeResource()
KERNEL32.DLL!BaseThreadInitThunk()
ntdll.dll!RtlUserThreadStart()

Comment by Silver Asu [ 2020-09-25 ]

Can we have more priority for this?
We had this crash again, that leaded to dataloss because many databases had to be restored from backups.
(dump crashed server even with innodb_force_recovery 6).

2020-09-24 16:50:55 0x7f0716c32700  InnoDB: Assertion failure in file /build/amd64-usr/var/tmp/portage/dev-db/mariadb-10.4.13/work/mariadb-10.4.13/storage/innobase/row/row0sel.cc line 3966
InnoDB: We intentionally generate a memory trap.
InnoDB: Submit a detailed bug report to https://jira.mariadb.org/
InnoDB: If you get repeated assertion failures or crashes, even
InnoDB: immediately after the mysqld startup, there may be
InnoDB: corruption in the InnoDB tablespace. Please refer to
InnoDB: https://mariadb.com/kb/en/library/innodb-recovery-modes/
InnoDB: about forcing recovery.
200924 16:50:55 [ERROR] mysqld got signal 6 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
To report this bug, see https://mariadb.com/kb/en/reporting-bugs
We will try our best to scrape up some info that will hopefully help
diagnose the problem, but since we have already crashed,
something is definitely wrong and this may fail.
Server version: 10.4.13-MariaDB-log
key_buffer_size=4294967296
read_buffer_size=2097152
max_used_connections=103
max_threads=1502
thread_count=110
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 19611875 K  bytes of memory
Hope that's ok; if not, decrease some variables in the equation.
Thread pointer: 0x7f05e8000c48
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 0x7f0716c31d98 thread_stack 0x49000
*** buffer overflow detected ***: /opt/zse/sbin/mysqld terminated
 
[Note] /opt/zse/sbin/mysqld (mysqld 10.4.13-MariaDB-log) starting as process 5900 ...
[Warning] Could not increase number of max_open_files to more than 65536 (request: 67131)
[Note] Plugin 'partition' is disabled.
[Note] InnoDB: Using Linux native AIO
[Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
[Note] InnoDB: Uses event mutexes
[Note] InnoDB: Compressed tables use zlib 1.2.11
[Note] InnoDB: Number of pools: 1
[Note] InnoDB: Using SSE2 crc32 instructions
[Note] mysqld: O_TMPFILE is not supported on /srvdata/mysql/tmp (disabling future attempts)
[Note] InnoDB: Initializing buffer pool, total size = 8G, instances = 8, chunk size = 128M
[Note] InnoDB: Completed initialization of buffer pool
[Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority().
[Note] InnoDB: Starting crash recovery from checkpoint LSN=9324336704026
[Note] InnoDB: Ignoring data file './XXX/catalog_category_product_index_store1_replica.ibd' with space ID 2971995, since the redo log references ./XXX/catalog_category_product_index_store1_replica.ibd with space ID 2971883.
[Note] InnoDB: Ignoring data file './XXX/catalog_category_product_index_store2_replica.ibd' with space ID 2971996, since the redo log references ./XXX/catalog_category_product_index_store2_replica.ibd with space ID 2971884.
[Note] InnoDB: Ignoring data file './XXX/catalog_category_product_index_store3_replica.ibd' with space ID 2971997, since the redo log references ./XXX/catalog_category_product_index_store3_replica.ibd with space ID 2971885.
[Note] InnoDB: Ignoring data file './XXX/catalog_category_product_index_store4_replica.ibd' with space ID 2971998, since the redo log references ./XXX/catalog_category_product_index_store4_replica.ibd with space ID 2971886.
[Note] InnoDB: Ignoring data file './XXX/catalog_product_index_price_replica.ibd' with space ID 2972002, since the redo log references ./XXX/catalog_product_index_price_replica.ibd with space ID 2971890.
[Note] InnoDB: Ignoring data file './XXX/catalog_category_product_index_store1.ibd' with space ID 2972051, since the redo log references ./XXX/catalog_category_product_index_store1.ibd with space ID 2971939.
[Note] InnoDB: Ignoring data file './XXX/catalog_category_product_index_store2.ibd' with space ID 2972052, since the redo log references ./XXX/catalog_category_product_index_store2.ibd with space ID 2971940.
[Note] InnoDB: Ignoring data file './XXX/catalog_category_product_index_store3.ibd' with space ID 2972053, since the redo log references ./XXX/catalog_category_product_index_store3.ibd with space ID 2971941.
[Note] InnoDB: Ignoring data file './XXX/catalog_category_product_index_store4.ibd' with space ID 2972054, since the redo log references ./XXX/catalog_category_product_index_store4.ibd with space ID 2971942.
[Note] InnoDB: Ignoring data file './XXX/catalogrule_product.ibd' with space ID 2972055, since the redo log references ./XXX/catalogrule_product.ibd with space ID 2971943.
[Note] InnoDB: Ignoring data file './XXX/catalogrule_product_price.ibd' with space ID 2972056, since the redo log references ./XXX/catalogrule_product_price.ibd with space ID 2971944.
[Note] InnoDB: Ignoring data file './XXX/catalogrule_group_website.ibd' with space ID 2972057, since the redo log references ./XXX/catalogrule_group_website.ibd with space ID 2971945.
[Note] InnoDB: Ignoring data file './XXX/catalog_product_index_price.ibd' with space ID 2972058, since the redo log references ./XXX/catalog_product_index_price.ibd with space ID 2971946.
[Note] InnoDB: Ignoring data file './XXX/catalogsearch_fulltext_scope1.ibd' with space ID 2972059, since the redo log references ./XXX/catalogsearch_fulltext_scope1.ibd with space ID 2971947.
[Note] InnoDB: Ignoring data file './XXX/catalogsearch_fulltext_scope2.ibd' with space ID 2972071, since the redo log references ./XXX/catalogsearch_fulltext_scope2.ibd with space ID 2971959.
[Note] InnoDB: Ignoring data file './XXX/catalogsearch_fulltext_scope3.ibd' with space ID 2972083, since the redo log references ./XXX/catalogsearch_fulltext_scope3.ibd with space ID 2971971.
[Note] InnoDB: Ignoring data file './XXX/catalogsearch_fulltext_scope4.ibd' with space ID 2972095, since the redo log references ./XXX/catalogsearch_fulltext_scope4.ibd with space ID 2971983.
[Note] InnoDB: Ignoring data file './XXX/catalog_category_product_index_store1.ibd' with space ID 2971995. Another data file called ./XXX/catalog_category_product_index_store1_replica.ibd exists with the same space ID.
[ERROR] InnoDB: Cannot rename './XXX/catalog_category_product_index_store1_replica.ibd' to './XXX/catalog_category_product_index_store1.ibd' because the target file exists. Remove the target file and try again.
[ERROR] InnoDB:  Cannot replay file rename. Remove either file and try again.
[ERROR] InnoDB: Plugin initialization aborted with error Generic error
[Note] InnoDB: Starting shutdown...
[ERROR] Plugin 'InnoDB' init function returned error.
[ERROR] Plugin 'InnoDB' registration as a STORAGE ENGINE failed.
[Note] Plugin 'FEEDBACK' is disabled.
[ERROR] Unknown/unsupported storage engine: InnoDB
[ERROR] Aborting

Comment by Daniel Black [ 2020-09-30 ]

marko I've pushed the bb-10.4-danielblack-MDEV-22761 0ab23132136. Introduced in 10.4 by 8d85715d507d

It seems CHECK_ABORTED_BY_USER wasn't handled correctly.

Comment by Daniel Black [ 2020-09-30 ]

Work around for the mean time - don't kill threads with 'optimizer_switch=index_condition_pushdown=on'. index condition pushdown overview is per our knowledge base. This might cause some performance degradation of queries (better than a crash however). show variables like 'Handler_icp%' for an estimate of its current use.

Alternately 10.4.12 is unaffected.

Comment by Sergei Petrunia [ 2020-09-30 ]

Work around for the mean time - don't kill threads with 'optimizer_switch=index_condition_pushdown=on'

danblack, is this really optimizer_switch=index_condition_pushdown=on, or rowid_filter=ON ?

I see that the patch makes the change here:

  result = handler_rowid_filter_check(prebuilt->pk_filter);
  switch (result) {
     
     // the change is here
  }

which imples that rowid filter check was done while ICP check wasn't?

Comment by Sergei Petrunia [ 2020-09-30 ]

danblack One thing that's obviously missing in the patch is test coverage. AFAIU it should be easy to add using debug_sync ? (EDIT: sorry, missed the exchange about it. Could it be that the difficulty in constructing a testcase is due to the query needing to use rowid filter, not ICP?)

Comment by Daniel Black [ 2020-09-30 ]

Quite right psergey its obviously rowid_filter and for @users disabling this will avoid the bug for now.

After taking some existing rowid_filter test I've come up with
https://github.com/MariaDB/server/compare/10.4...grooverdan:10.4-MDEV-22761-innodb-rowid_filter-error-testcase_WIP?expand=1

Its currently not entering the '!h->pushed_idx_cond' section of handler_rowid_filter_check. Can you advice as how to trigger this condition in the test?

Comment by Daniel Black [ 2020-10-03 ]

psergey bb-10.4-danielblack-MDEV-22761 (aka https://github.com/MariaDB/server/commit/e8b9afa4a132e8e32a7e8d0cee97b82f52f739de ) updated to include debug_sync test case.

Comment by Sergei Petrunia [ 2020-10-08 ]

Ok looking

Comment by Sergei Petrunia [ 2020-10-09 ]

kill query @id;
set debug_sync='now SIGNAL go';
 
connection default;
reap;
set debug_sync='RESET';

This shows that the killed query does not return any errors. This is not good.

Comment by Sergei Petrunia [ 2020-10-09 ]

I've tried to construct a bigger example, and there, it does return "Query execution was interrupted" error...

Comment by Sergei Petrunia [ 2020-10-09 ]

Ok this happened, because the bigger example was a join of two tables

MariaDB [test]> explain
    -> select * from t2, t3
    -> where 
    ->   t3.key1=t2.a and t3.key2 in (2,3);
+------+-------------+-------+------------+---------------+-----------+---------+-----------+-----------+---------------------------------+
| id   | select_type | table | type       | possible_keys | key       | key_len | ref       | rows      | Extra                           |
+------+-------------+-------+------------+---------------+-----------+---------+-----------+-----------+---------------------------------+
|    1 | SIMPLE      | t2    | ALL        | NULL          | NULL      | NULL    | NULL      | 100       | Using where                     |
|    1 | SIMPLE      | t3    | ref|filter | key1,key2     | key1|key2 | 5|5     | test.t2.a | 499 (20%) | Using where; Using rowid filter |
+------+-------------+-------+------------+---------------+-----------+---------+-----------+-----------+---------------------------------+

The KILL arrives into rowid filter check on table t3. It causes the read on t3 to return HA_ERR_KEY_NOT_FOUND.

So it continues the full table scan on table t2, and execution arrives here in row_search_mvcc():

	DEBUG_SYNC_C("row_search_rec_loop");
	if (trx_is_interrupted(trx)) {
		if (!spatial_search) {
			btr_pcur_store_position(pcur, &mtr);
		}
		err = DB_INTERRUPTED;
		goto normal_return;
	}

where the thread-killed condition is detected.

Comment by Sergei Petrunia [ 2020-10-09 ]

Pushed another commit into https://github.com/MariaDB/server/commits/bb-10.4-danielblack-MDEV-22761
https://github.com/MariaDB/server/commit/e206c0c71596052cc3f0cc6804ee43048086bbd4
I think these two commits together are ok to push.

danblack or marko can you check the latest commit?

Comment by Daniel Black [ 2020-10-09 ]

pushed dbba393023344d0943e76da87372f12f9274054e to the end of the branch as there where two more calls to row_search_idx_cond_check that needed to handle the CHECK_ABORTED_BY_USER => DB_INTERRUPTED in the same way.

Isn't

+++ b/mysql-test/main/aria_icp_debug.test
@@ -0,0 +1,5 @@
+
+set default_storage_engine=aria;
+--source include/icp_debug_kill.inc

Going to leave the storage engine in aria for non-debug tests with default_storage_engine=aria?

Nice use of `evalp kill query $target_id`

Comment by Daniel Black [ 2020-10-10 ]

The handling of killing the query also applies to 10.1 under ICP_ABORTED_BY_USER conditions.

Comment by Daniel Black [ 2020-10-12 ]

Ported ICP patches fixes and tests (excluding rowid_filter) to 10.1 - MDEV-23938 on ensuring ICP_ABORTED_BY_USER -> DB_INTERRUPTED error.

Comment by Sergei Petrunia [ 2020-10-13 ]

danblack

Isn't ... Going to leave the storage engine in aria for non-debug tests with default_storage_engine=aria?

but it sets the storage engine back:
https://github.com/MariaDB/server/blob/e206c0c71596052cc3f0cc6804ee43048086bbd4/mysql-test/main/aria_icp_debug.test

Comment by Sergei Petrunia [ 2020-10-13 ]

The rest of the `dbba393023344d0943e76da87372f12f9274054e` looks good to me.
I see BB shows failures https://buildbot.askmonty.org/buildbot/grid?category=main&branch=bb-10.4-danielblack-MDEV-22761
but they dont seem to be related?

This one looks odd https://buildbot.askmonty.org/buildbot/builders/kvm-fulltest2/builds/25001/steps/mtr_ps_emb/logs/stdio ... but should be counted as un-related?

Comment by Sergei Petrunia [ 2020-10-13 ]

Re-based the tree on the latest 10.4 and pushed.

Generated at Thu Feb 08 09:17:16 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.