[MDEV-21400] encryption.innochecksum fails during shutdown Created: 2019-12-28  Updated: 2020-01-07  Resolved: 2020-01-07

Status: Closed
Project: MariaDB Server
Component/s: Storage Engine - InnoDB
Affects Version/s: 10.5.1
Fix Version/s: 10.5.1

Type: Bug Priority: Major
Reporter: Michael Widenius Assignee: Thirunarayanan Balathandayuthapani
Resolution: Duplicate Votes: 0
Labels: None

Issue Links:
Relates
relates to MDEV-21344 Valgrind uninitialised value warnings... Closed
relates to MDEV-16678 Use MDL for innodb background threads... Closed

 Description   

Running full test suite on a relative late 10.5.1 tree gave me the following
(was not able to repeat this)

encryption.innochecksum '8k,cbc,innodb,strict_full_crc32' w7 [ fail ]
Program terminated with signal SIGABRT, Aborted.
#0 0x00007f28357cdf14 in pthread_kill () from /lib64/libpthread.so.0
[Current thread is 1 (Thread 0x7f2835bc7880 (LWP 26235))]
#0 0x00007f28357cdf14 in pthread_kill () from /lib64/libpthread.so.0
#1 0x00000000015fccc8 in my_write_core (sig=6) at /my/maria-10.5/mysys/stacktrace.c:518
#2 0x0000000000bed102 in handle_fatal_signal (sig=6) at /my/maria-10.5/sql/signal_handler.cc:343
#3 <signal handler called>
#4 0x00007f28357cc90d in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#5 0x00007f2834052b4c in std::condition_variable::wait(std::unique_lock<std::mutex>&) () from /usr/lib64/libstdc++.so.6
#6 0x0000000001468c46 in tpool::waitable_task::wait (this=0x25c4320 <purge_coordinator_task>) at /my/maria-10.5/tpool/task.cc:64
#7 0x000000000122757f in srv_shutdown_purge_tasks () at /my/maria-10.5/storage/innobase/srv/srv0srv.cc:2232
#8 0x0000000001227a00 in srv_purge_shutdown () at /my/maria-10.5/storage/innobase/srv/srv0srv.cc:2302
#9 0x0000000001230903 in innodb_preshutdown () at /my/maria-10.5/storage/innobase/srv/srv0start.cc:2304
#10 0x0000000000bf147a in plugin_pre_shutdown (plugin=0x4d495a8) at /my/maria-10.5/sql/handler.cc:885
#11 0x00000000008cd1b4 in plugin_foreach_with_mask (thd=0x0, func=0xbf1439 <plugin_pre_shutdown(THD*, plugin_ref, void*)>, type=1, state_mask=10, arg=0x0) at /my/maria-10.5/sql/sql_plugin.cc:2452
#12 0x0000000000bf14a6 in ha_pre_shutdown () at /my/maria-10.5/sql/handler.cc:89

 Comments   
Comment by Marko Mäkelä [ 2019-12-29 ]

We are aware of occasional InnoDB shutdown hangs in 10.5 ever since MDEV-16678 was pushed. thiru should try to repeat and fix this, and perhaps start by looking at the MemorySanitizer reports (MDEV-20377) for the memcmp() calls in dict_acquire_mdl_shared().

Comment by Marko Mäkelä [ 2020-01-03 ]

As noted in MDEV-21344, both Valgrind and MemorySanitizer report that memcmp() is being invoked on uninitialized data in dict_acquire_mdl_shared().

Comment by Marko Mäkelä [ 2020-01-07 ]

I am convinced that the MDEV-21344 fix also fixed the shutdown hangs. I have been running ./mtr --big-test on a 10.5-branch at least 5 times, maybe more than 10 times, on a branch that includes the fix, and did not see the hang once. Before the fix, I would observe a shutdown hang at least once in 5 or 10 runs.

Generated at Thu Feb 08 09:06:51 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.