The problem is apparently that a waiting thread is not woken up when an InnoDB mutes is released:
while (!event->is_set && event->signal_count == reset_sig_count)
|
os_cond_wait(&(event->cond_var), &(event->os_mutex));
|
|
|
(gdb) p event->is_set
|
$3 = 0
|
(gdb) p event->signal_count
|
$4 = 2
|
(gdb) p reset_signal_count
|
No symbol "reset_signal_count" in current context.
|
(gdb) p reset_sig_count
|
$5 = 2
|
(gdb) p !event->is_set && event->signal_count == reset_sig_count
|
$6 = 1
|
The wakeup condition is not true...
(gdb) p *mutex
|
$8 = {event = 0x10029bf30a0, lock_word = 0 '\000', waiters = 1, list = {prev = 0x10029bf2f70, next = 0x11953928 <srv_misc_tmpfile_mutex>},
|
cfile_name = 0x10d01150 "/home/skysql/bzr/knielsen/5.5/storage/xtradb/fil/fil0fil.c", cline = 1673, thread_id = 18446744073709551615, magic_n = 979585, count_os_wait = 1, count_using = 1611,
|
count_spin_loop = 6, count_spin_rounds = 175, count_os_yield = 3, lspent_time = 0, lmax_spent_time = 0, mutex_type = 0, cmutex_name = 0x10d02228 "&fil_system->mutex", pfs_psi = 0x3fff81ff2330}
|
(gdb) p mutex->lock_word
|
$9 = 0 '\000'
|
But the mutex is not locked (lock_word == 0).
Looks like some issue with the InnoDB mutex implementation...
There is a comment in mutex_exit_func():
/* A problem: we assume that mutex_reset_lock word
|
is a memory barrier, that is when we read the waiters
|
field next, the read must be serialized in memory
|
after the reset. A speculative processor might
|
perform the read first, which could leave a waiting
|
thread hanging indefinitely.
|
|
|
Our current solution call every second
|
sync_arr_wake_threads_if_sema_free()
|
to wake up possible hanging threads if
|
they are missed in mutex_signal_object. */
|
Even though we added "os_isync" between "lock_word store" and "waiters load", it is not enough. Only full memory barrier guarantees StoreLoad order.
So we still rely on the error monitor thread to wake up such stale threads. But error monitor thread is using mutexes itself and may get stuck similarly to other threads. To workaround this Monty added log_get_lsn_nowait(), which basically does trylock instead of lock. This is added in revision 4271 (monty@mariadb.org-20140819162835-sorv0ogd39f7mui8) of 5.5.
In this case I suspect error monitor thread is either not started or is stuck on different mutex (other than log_get_lsn mutex).
I wrote up a detailed analysis on maria-developers@ of what appears to be the
cause of the bug.
https://lists.launchpad.net/maria-developers/msg07860.html
Incidentally, it appears that this same bug is also what is
causing various strange hangs for a number of users lately.
I agree that in this case, the error monitor thread is not yet started (it was
not seen in the full stack trace). And in the user hang cases, that the error
monitor thread itself gets stuck on a mutex (I saw this in the case I
investigated).
However, I strongly disagree that the whole idea of using error monitor
thread, or trylock, is in any way an acceptable solution. If the mutex
implementation is broken, it should be fixed, not worked around with horrible
hacks.
Kristian, please review fix for this bug.
...second patch:
Date: Wed, 12 Nov 2014 11:49:27 +0000 (UTC)
|
From: Sergey Vojtovich <svoj@mariadb.org>
|
To: commits@mariadb.org
|
Subject: [Commits] Rev 4346: MDEV-7026 - Occasional hang during startup on Power8 in lp:maria/5.5
|
Review sent to maria-developers@:
Kristian, please review updated patch.
New review sent to maria-developers@:
Patch to clarify suggestions in previously made review:
http://lists.askmonty.org/pipermail/commits/2014-November/007050.html
lp:~maria-captains/maria/5.5-mdev7026
Fixed in 10.0.15, 5.5.41:
revno: 4367
|
revision-id: knielsen@knielsen-hq.org-20141119125646-njq02h7irmu9s71b
|
parent: sergii@pisem.net-20141118231952-fk3o69075dn3ravv
|
committer: Kristian Nielsen <knielsen@knielsen-hq.org>
|
branch nick: mariadb-5.5
|
timestamp: Wed 2014-11-19 13:56:46 +0100
|
message:
|
MDEV-7026: Race in InnoDB/XtraDB mutex implementation can stall or hang the server.
|
|
|
The bug was that full memory barrier was missing in the code that ensures that
|
a waiter on an InnoDB mutex will not go to sleep unless it is guaranteed to be
|
woken up again by another thread currently holding the mutex. This made
|
possible a race where a thread could get stuck waiting for a mutex that is in
|
fact no longer locked. If that thread was also holding other critical locks,
|
this could stall the entire server. There is an error monitor thread than can
|
break the stall, it runs about once per second. But if the error monitor
|
thread itself got stuck or was not running, then the entire server could hang
|
infinitely.
|
|
|
This was introduced on i386/amd64 platforms in 5.5.40 and 10.0.13 by an
|
incorrect patch that tried to fix the similar problem for PowerPC.
|
|
|
This commit reverts the incorrect PowerPC patch, and instead implements a fix
|
for PowerPC that does not change i386/amd64 behaviour, making PowerPC work
|
similarly to i386/amd64.
|
The use of __sync_synchronize() here is incorrect as it gererates a sync() instruction which is a heavyweight barrier which is only needed if you're writing device drivers.
__sync_lock_release() is sufficient
realistically, __atomic_test_and_set() and __atomic_clear() should be used these days.
stewart-ibm, right. Good mutex implementation should never need to issue heavyweight sync.
In 5.5 and 10.0 old mutex implementation was kept intact as much as possible, for which this sync is a must. 
In 10.1 there was an aim (MDEV-6654) to reimplement InnoDB mutexes. This was suspended because InnoDB also reimplemented mutexes in MySQL-5.7. We'll probably get back to this when we have InnoDB from 5.7.
You should be able to remove the __sync_synchronize() and everything work. There's no reason for the sync instruction to be there.
I do not plan to re-visit this issue myself. But be sure to read the mailing list discussion on this in the archive, there was a very detailed discussion on what kind of barriers are needed in the current implementation, and why.
stewart-ibm, we added __sync_synchronize() to make sure "lock_word" is stored before "waiters" are loaded. That is to ensure StoreLoad order.
You say it's not needed. If so, how can we ensure StoreLoad order in this case?