Type:
Bug
Priority:
Major
Resolution:
Unresolved
Affects Version/s:
10.6
I have been chasing this "complex to reproduce and reduce" bug for months. Let me list everything I have thus far. See this search for potentially related bugs.
1. I regularly see this crash (stack from test run sample, not from reduced testcase):
10.6.0 786bc312b85e58857cb26a24ab6e997ba0fdfc32
mysqld: /test/10.6_dbg/storage/innobase/dict/dict0dict.cc:1451: ulint dict_make_room_in_cache(ulint, ulint): Assertion `!table->fts' failed.
10.6.0 786bc312b85e58857cb26a24ab6e997ba0fdfc32
Core was generated by `/test/MD100221-mariadb-10.6.0-linux-x86_64-dbg/bin/mysqld --no-defaults --max_a'.
Program terminated with signal SIGABRT, Aborted.
#0 __pthread_kill (threadid=<optimized out>, signo=signo@entry=6)
at ../sysdeps/unix/sysv/linux/pthread_kill.c:56
[Current thread is 1 (Thread 0x14b548df6700 (LWP 4100654))]
(gdb) bt
#0 __pthread_kill (threadid=<optimized out>, signo=signo@entry=6) at ../sysdeps/unix/sysv/linux/pthread_kill.c:56
#1 0x0000562d5f52d55e in my_write_core (sig=sig@entry=6) at /test/10.6_dbg/mysys/stacktrace.c:424
#2 0x0000562d5ecc54de in handle_fatal_signal (sig=6) at /test/10.6_dbg/sql/signal_handler.cc:330
#3 <signal handler called>
#4 __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50
#5 0x000014b57d893859 in __GI_abort () at abort.c:79
#6 0x000014b57d893729 in __assert_fail_base (fmt=0x14b57da29588 "%s%s%s:%u: %s%sAssertion `%s' failed.\n%n", assertion=0x562d5f9573b2 "!table->fts", file=0x562d5f9574b0 "/test/10.6_dbg/storage/innobase/dict/dict0dict.cc", line=1451, function=<optimized out>) at assert.c:92
#7 0x000014b57d8a4f36 in __GI___assert_fail (assertion=assertion@entry=0x562d5f9573b2 "!table->fts", file=file@entry=0x562d5f9574b0 "/test/10.6_dbg/storage/innobase/dict/dict0dict.cc", line=line@entry=1451, function=function@entry=0x562d5f9588e0 "ulint dict_make_room_in_cache(ulint, ulint)") at assert.c:101
#8 0x0000562d5f3e3d4a in dict_make_room_in_cache (max_tables=512, pct_check=pct_check@entry=100) at /test/10.6_dbg/storage/innobase/dict/dict0dict.cc:1451
#9 0x0000562d5f2b6b47 in srv_master_evict_from_table_cache (pct_check=pct_check@entry=100) at /test/10.6_dbg/storage/innobase/srv/srv0srv.cc:1496
#10 0x0000562d5f2b98d6 in srv_master_do_idle_tasks () at /test/10.6_dbg/storage/innobase/srv/srv0srv.cc:1690
#11 srv_master_callback () at /test/10.6_dbg/storage/innobase/srv/srv0srv.cc:1753
#12 0x0000562d5f4b9200 in tpool::thread_pool_generic::timer_generic::run (this=0x562d616870e0) at /test/10.6_dbg/tpool/tpool_generic.cc:309
#13 tpool::thread_pool_generic::timer_generic::execute (arg=0x562d616870e0) at /test/10.6_dbg/tpool/tpool_generic.cc:329
#14 0x0000562d5f4ba16b in tpool::task::execute (this=0x562d61687120) at /test/10.6_dbg/tpool/task.cc:52
#15 0x0000562d5f4b8d1b in tpool::thread_pool_generic::worker_main (this=0x562d60f08d60, thread_var=0x562d60f18400) at /test/10.6_dbg/tpool/tpool_generic.cc:546
#16 0x0000562d5f4b9052 in std::__invoke_impl<void, void (tpool::thread_pool_generic::*)(tpool::worker_data*), tpool::thread_pool_generic*, tpool::worker_data*> (__t=<optimized out>, __f=<optimized out>) at /usr/include/c++/9/bits/invoke.h:89
#17 std::__invoke<void (tpool::thread_pool_generic::*)(tpool::worker_data*), tpool::thread_pool_generic*, tpool::worker_data*> (__fn=<optimized out>) at /usr/include/c++/9/bits/invoke.h:95
#18 std::thread::_Invoker<std::tuple<void (tpool::thread_pool_generic::*)(tpool::worker_data*), tpool::thread_pool_generic*, tpool::worker_data*> >::_M_invoke<0ul, 1ul, 2ul> (this=<optimized out>) at /usr/include/c++/9/thread:244
#19 std::thread::_Invoker<std::tuple<void (tpool::thread_pool_generic::*)(tpool::worker_data*), tpool::thread_pool_generic*, tpool::worker_data*> >::operator() (this=<optimized out>) at /usr/include/c++/9/thread:251
#20 std::thread::_State_impl<std::thread::_Invoker<std::tuple<void (tpool::thread_pool_generic::*)(tpool::worker_data*), tpool::thread_pool_generic*, tpool::worker_data*> > >::_M_run (this=<optimized out>) at /usr/include/c++/9/thread:195
#21 0x000014b57dc85d84 in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#22 0x000014b57dda1609 in start_thread (arg=<optimized out>) at pthread_create.c:477
#23 0x000014b57d990293 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
2. I have attached at testcase, which, when the following unreduced set of options:
--max_allowed_packet=33554432 --maximum-bulk_insert_buffer_size=1M --maximum-join_buffer_size=1M --maximum-max_heap_table_size=1M --maximum-max_join_size=1M --maximum-myisam_max_sort_file_size=1M --maximum-myisam_mmap_size=1M --maximum-myisam_sort_buffer_size=1M --maximum-optimizer_trace_max_mem_size=1M --maximum-preload_buffer_size=1M --maximum-query_alloc_block_size=1M --maximum-query_prealloc_size=1M --maximum-range_alloc_block_size=1M --maximum-read_buffer_size=1M --maximum-read_rnd_buffer_size=1M --maximum-sort_buffer_size=1M --maximum-tmp_table_size=1M --maximum-transaction_alloc_block_size=1M --maximum-transaction_prealloc_size=1M --log-output=none --sql_mode=ONLY_FULL_GROUP_BY --innodb_file_per_table=1 --innodb_flush_method=O_DIRECT --innodb_stats_persistent=off --loose-idle_write_transaction_timeout=0 --loose-idle_transaction_timeout=0 --loose-idle_readonly_transaction_timeout=0 --connect_timeout=60 --interactive_timeout=28800 --slave_net_timeout=60 --net_read_timeout=30 --net_write_timeout=60 --loose-table_lock_wait_timeout=50 --wait_timeout=28800 --lock-wait-timeout=86400 --innodb-lock-wait-timeout=50 --log_output=FILE --log-bin --log_bin_trust_function_creators=1 --loose-max-statement-time=30 --loose-debug_assert_on_not_freed_memory=0 --innodb-buffer-pool-size=300M
And using the pquery binary is able to reproduce (and previously reduce to this) the issue under reducer.sh, with the same uniqueID.
3. This bug closely relates to MDEV-25072 as I see a similar "closing tables" issue there, even though the testcase and setup is very different, with the exception of partitions being involved.
see that innnodb file close issue in log. in.sql running for this one.
{"report":{"fcp":1229.1999998092651,"ttfb":311.5999994277954,"pageVisibility":"visible","entityId":97646,"key":"jira.project.issue.view-issue","isInitial":true,"threshold":1000,"elementTimings":{},"userDeviceMemory":8,"userDeviceProcessors":64,"apdex":0.5,"journeyId":"418c17a0-6359-4b3e-8b38-003fe6077b1a","navigationType":0,"readyForUser":1318.3999996185303,"redirectCount":0,"resourceLoadedEnd":963.5999994277954,"resourceLoadedStart":317.0999994277954,"resourceTiming":[{"duration":293.4000005722046,"initiatorType":"link","name":"https://jira.mariadb.org/s/2c21342762a6a02add1c328bed317ffd-CDN/lu2cib/820016/12ta74/0a8bac35585be7fc6c9cc5a0464cd4cf/_/download/contextbatch/css/_super/batch.css","startTime":317.0999994277954,"connectEnd":0,"connectStart":0,"domainLookupEnd":0,"domainLookupStart":0,"fetchStart":317.0999994277954,"redirectEnd":0,"redirectStart":0,"requestStart":0,"responseEnd":610.5,"responseStart":0,"secureConnectionStart":0},{"duration":293.3999996185303,"initiatorType":"link","name":"https://jira.mariadb.org/s/7ebd35e77e471bc30ff0eba799ebc151-CDN/lu2cib/820016/12ta74/494e4c556ecbb29f90a3d3b4f09cb99c/_/download/contextbatch/css/jira.browse.project,project.issue.navigator,jira.view.issue,jira.general,jira.global,atl.general,-_super/batch.css?agile_global_admin_condition=true&jag=true&jira.create.linked.issue=true&slack-enabled=true&whisper-enabled=true","startTime":317.5,"connectEnd":0,"connectStart":0,"domainLookupEnd":0,"domainLookupStart":0,"fetchStart":317.5,"redirectEnd":0,"redirectStart":0,"requestStart":0,"responseEnd":610.8999996185303,"responseStart":0,"secureConnectionStart":0},{"duration":303,"initiatorType":"script","name":"https://jira.mariadb.org/s/0917945aaa57108d00c5076fea35e069-CDN/lu2cib/820016/12ta74/0a8bac35585be7fc6c9cc5a0464cd4cf/_/download/contextbatch/js/_super/batch.js?locale=en","startTime":317.5999994277954,"connectEnd":317.5999994277954,"connectStart":317.5999994277954,"domainLookupEnd":317.5999994277954,"domainLookupStart":317.5999994277954,"fetchStart":317.5999994277954,"redirectEnd":0,"redirectStart":0,"requestStart":317.5999994277954,"responseEnd":620.5999994277954,"responseStart":620.5999994277954,"secureConnectionStart":317.5999994277954},{"duration":365.30000019073486,"initiatorType":"script","name":"https://jira.mariadb.org/s/2d8175ec2fa4c816e8023260bd8c1786-CDN/lu2cib/820016/12ta74/494e4c556ecbb29f90a3d3b4f09cb99c/_/download/contextbatch/js/jira.browse.project,project.issue.navigator,jira.view.issue,jira.general,jira.global,atl.general,-_super/batch.js?agile_global_admin_condition=true&jag=true&jira.create.linked.issue=true&locale=en&slack-enabled=true&whisper-enabled=true","startTime":317.79999923706055,"connectEnd":317.79999923706055,"connectStart":317.79999923706055,"domainLookupEnd":317.79999923706055,"domainLookupStart":317.79999923706055,"fetchStart":317.79999923706055,"redirectEnd":0,"redirectStart":0,"requestStart":317.79999923706055,"responseEnd":683.0999994277954,"responseStart":683.0999994277954,"secureConnectionStart":317.79999923706055},{"duration":369.5,"initiatorType":"script","name":"https://jira.mariadb.org/s/a9324d6758d385eb45c462685ad88f1d-CDN/lu2cib/820016/12ta74/c92c0caa9a024ae85b0ebdbed7fb4bd7/_/download/contextbatch/js/atl.global,-_super/batch.js?locale=en","startTime":318,"connectEnd":318,"connectStart":318,"domainLookupEnd":318,"domainLookupStart":318,"fetchStart":318,"redirectEnd":0,"redirectStart":0,"requestStart":318,"responseEnd":687.5,"responseStart":687.5,"secureConnectionStart":318},{"duration":369.80000019073486,"initiatorType":"script","name":"https://jira.mariadb.org/s/d41d8cd98f00b204e9800998ecf8427e-CDN/lu2cib/820016/12ta74/1.0/_/download/batch/jira.webresources:calendar-en/jira.webresources:calendar-en.js","startTime":318.19999980926514,"connectEnd":318.19999980926514,"connectStart":318.19999980926514,"domainLookupEnd":318.19999980926514,"domainLookupStart":318.19999980926514,"fetchStart":318.19999980926514,"redirectEnd":0,"redirectStart":0,"requestStart":318.19999980926514,"responseEnd":688,"responseStart":688,"secureConnectionStart":318.19999980926514},{"duration":370,"initiatorType":"script","name":"https://jira.mariadb.org/s/d41d8cd98f00b204e9800998ecf8427e-CDN/lu2cib/820016/12ta74/1.0/_/download/batch/jira.webresources:calendar-localisation-moment/jira.webresources:calendar-localisation-moment.js","startTime":318.3999996185303,"connectEnd":318.3999996185303,"connectStart":318.3999996185303,"domainLookupEnd":318.3999996185303,"domainLookupStart":318.3999996185303,"fetchStart":318.3999996185303,"redirectEnd":0,"redirectStart":0,"requestStart":318.3999996185303,"responseEnd":688.3999996185303,"responseStart":688.3999996185303,"secureConnectionStart":318.3999996185303},{"duration":464.0999994277954,"initiatorType":"link","name":"https://jira.mariadb.org/s/b04b06a02d1959df322d9cded3aeecc1-CDN/lu2cib/820016/12ta74/a2ff6aa845ffc9a1d22fe23d9ee791fc/_/download/contextbatch/css/jira.global.look-and-feel,-_super/batch.css","startTime":318.5,"connectEnd":0,"connectStart":0,"domainLookupEnd":0,"domainLookupStart":0,"fetchStart":318.5,"redirectEnd":0,"redirectStart":0,"requestStart":0,"responseEnd":782.5999994277954,"responseStart":0,"secureConnectionStart":0},{"duration":370.30000019073486,"initiatorType":"script","name":"https://jira.mariadb.org/rest/api/1.0/shortcuts/820016/47140b6e0a9bc2e4913da06536125810/shortcuts.js?context=issuenavigation&context=issueaction","startTime":318.69999980926514,"connectEnd":318.69999980926514,"connectStart":318.69999980926514,"domainLookupEnd":318.69999980926514,"domainLookupStart":318.69999980926514,"fetchStart":318.69999980926514,"redirectEnd":0,"redirectStart":0,"requestStart":318.69999980926514,"responseEnd":689,"responseStart":689,"secureConnectionStart":318.69999980926514},{"duration":463.8999996185303,"initiatorType":"link","name":"https://jira.mariadb.org/s/3ac36323ba5e4eb0af2aa7ac7211b4bb-CDN/lu2cib/820016/12ta74/d176f0986478cc64f24226b3d20c140d/_/download/contextbatch/css/com.atlassian.jira.projects.sidebar.init,-_super,-project.issue.navigator,-jira.view.issue/batch.css?jira.create.linked.issue=true","startTime":318.8999996185303,"connectEnd":0,"connectStart":0,"domainLookupEnd":0,"domainLookupStart":0,"fetchStart":318.8999996185303,"redirectEnd":0,"redirectStart":0,"requestStart":0,"responseEnd":782.7999992370605,"responseStart":0,"secureConnectionStart":0},{"duration":370.5,"initiatorType":"script","name":"https://jira.mariadb.org/s/5d5e8fe91fbc506585e83ea3b62ccc4b-CDN/lu2cib/820016/12ta74/d176f0986478cc64f24226b3d20c140d/_/download/contextbatch/js/com.atlassian.jira.projects.sidebar.init,-_super,-project.issue.navigator,-jira.view.issue/batch.js?jira.create.linked.issue=true&locale=en","startTime":319.0999994277954,"connectEnd":319.0999994277954,"connectStart":319.0999994277954,"domainLookupEnd":319.0999994277954,"domainLookupStart":319.0999994277954,"fetchStart":319.0999994277954,"redirectEnd":0,"redirectStart":0,"requestStart":319.0999994277954,"responseEnd":689.5999994277954,"responseStart":689.5999994277954,"secureConnectionStart":319.0999994277954},{"duration":541.4000005722046,"initiatorType":"script","name":"https://jira.mariadb.org/s/d41d8cd98f00b204e9800998ecf8427e-CDN/lu2cib/820016/12ta74/1.0/_/download/batch/jira.webresources:bigpipe-js/jira.webresources:bigpipe-js.js","startTime":319.79999923706055,"connectEnd":319.79999923706055,"connectStart":319.79999923706055,"domainLookupEnd":319.79999923706055,"domainLookupStart":319.79999923706055,"fetchStart":319.79999923706055,"redirectEnd":0,"redirectStart":0,"requestStart":319.79999923706055,"responseEnd":861.1999998092651,"responseStart":861.1999998092651,"secureConnectionStart":319.79999923706055},{"duration":643.6999998092651,"initiatorType":"script","name":"https://jira.mariadb.org/s/d41d8cd98f00b204e9800998ecf8427e-CDN/lu2cib/820016/12ta74/1.0/_/download/batch/jira.webresources:bigpipe-init/jira.webresources:bigpipe-init.js","startTime":319.8999996185303,"connectEnd":319.8999996185303,"connectStart":319.8999996185303,"domainLookupEnd":319.8999996185303,"domainLookupStart":319.8999996185303,"fetchStart":319.8999996185303,"redirectEnd":0,"redirectStart":0,"requestStart":319.8999996185303,"responseEnd":963.5999994277954,"responseStart":963.5999994277954,"secureConnectionStart":319.8999996185303},{"duration":36.10000038146973,"initiatorType":"xmlhttprequest","name":"https://jira.mariadb.org/rest/webResources/1.0/resources","startTime":825.7999992370605,"connectEnd":825.7999992370605,"connectStart":825.7999992370605,"domainLookupEnd":825.7999992370605,"domainLookupStart":825.7999992370605,"fetchStart":825.7999992370605,"redirectEnd":0,"redirectStart":0,"requestStart":825.7999992370605,"responseEnd":861.8999996185303,"responseStart":861.8999996185303,"secureConnectionStart":825.7999992370605}],"fetchStart":0,"domainLookupStart":0,"domainLookupEnd":0,"connectStart":0,"connectEnd":0,"requestStart":130,"responseStart":311,"responseEnd":313,"domLoading":315,"domInteractive":1382,"domContentLoadedEventStart":1387,"domContentLoadedEventEnd":1440,"domComplete":2410,"loadEventStart":2410,"loadEventEnd":2412,"userAgent":"Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; ClaudeBot/1.0; +claudebot@anthropic.com)","marks":[{"name":"bigPipe.sidebar-id.start","time":1357.1999998092651},{"name":"bigPipe.sidebar-id.end","time":1357.8999996185303},{"name":"bigPipe.activity-panel-pipe-id.start","time":1358.0999994277954},{"name":"bigPipe.activity-panel-pipe-id.end","time":1361.3999996185303},{"name":"activityTabFullyLoaded","time":1448}],"measures":[],"correlationId":"59899bd390a56b","effectiveType":"4g","downlink":9.5,"rtt":0,"serverDuration":116,"dbReadsTimeInMs":12,"dbConnsTimeInMs":21,"applicationHash":"9d11dbea5f4be3d4cc21f03a88dd11d8c8687422","experiments":[]}}