Details
-
Bug
-
Status: Closed (View Workflow)
-
Critical
-
Resolution: Fixed
-
10.1.22
-
CentOS 7 x86_64
Description
After upgrading to MariaDB 10.1.22 (from 10.1.21) the process started crashing randomly (didn't notice a pattern until now):
Mar 15 18:28:50 Updated: MariaDB-server-10.1.22-1.el7.centos.x86_64
Mar 15 18:58:21 HOST mysqld: *** buffer overflow detected ***: /usr/sbin/mysqld terminated
Mar 15 23:05:43 HOST mysqld: *** buffer overflow detected ***: /usr/sbin/mysqld terminated
Mar 16 06:49:15 HOST mysqld: *** buffer overflow detected ***: /usr/sbin/mysqld terminated
Mar 16 07:32:15 HOST mysqld: *** buffer overflow detected ***: /usr/sbin/mysqld terminated
Mar 16 09:17:21 HOST mysqld: *** buffer overflow detected ***: /usr/sbin/mysqld terminated
Mar 16 09:19:51 HOST mysqld: *** buffer overflow detected ***: /usr/sbin/mysqld terminated
Mar 16 09:22:18 HOST mysqld: *** buffer overflow detected ***: /usr/sbin/mysqld terminated
Mar 16 09:24:51 HOST mysqld: *** buffer overflow detected ***: /usr/sbin/mysqld terminated
Mar 16 09:26:00 HOST mysqld: *** buffer overflow detected ***: /usr/sbin/mysqld terminated
Server was stable until the upgrade.
I've attached the full crash report here.
Seems to be similar to what's reported here on 5.6.35-80.0:
https://bugs.launchpad.net/percona-server/+bug/1664519
https://bugs.mysql.com/bug.php?id=84940
We use this server as a multi-master slave, mainly for backups (done pretty often, almost constant read on this server). Seems to be happening on certain kind of slave updates coming from master.
We have an, almost, identical slave which doesn't experience this problem, the difference is that this server isn't used that much, no backups, mainly as a readonly access to data for development team, infrequent.
We downgraded to 10.1.21 to see if we still experience this problem. So far, so good.
Please let me know if any additional data is required.
Thank you.
Attachments
Issue Links
- is part of
-
MDEV-12282 10.1.23 merge
-
- Closed
-
-
-
- Closed
-
- relates to
-
-
- Closed
-
- links to
Activity
MariaDB 10.1.23
terminate called after throwing an instance of 'std::out_of_range' |
what(): vector::_M_range_check
|
170524 16:03:37 [ERROR] mysqld got signal 6 ; |
This could be because you hit a bug. It is also possible that this binary |
or one of the libraries it was linked against is corrupt, improperly built,
|
or misconfigured. This error can also be caused by malfunctioning hardware.
|
|
|
To report this bug, see https://mariadb.com/kb/en/reporting-bugs |
|
|
We will try our best to scrape up some info that will hopefully help |
diagnose the problem, but since we have already crashed,
|
something is definitely wrong and this may fail. |
|
|
Server version: 10.1.23-MariaDB |
key_buffer_size=32212254720 |
read_buffer_size=131072 |
max_used_connections=4 |
max_threads=153 |
thread_count=4 |
It is possible that mysqld could use up to
|
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 31793329 K bytes of memory |
Hope that's ok; if not, decrease some variables in the equation. |
|
|
Thread pointer: 0x0 |
Attempting backtrace. You can use the following information to find out
|
where mysqld died. If you see no messages after this, something went |
terribly wrong...
|
stack_bottom = 0x0 thread_stack 0x48400 |
(my_addr_resolve failure: fork)
|
/usr/sbin/mysqld(my_print_stacktrace+0x2e) [0x7f3d2ed3b4ce] |
/usr/sbin/mysqld(handle_fatal_signal+0x305) [0x7f3d2e85e345] |
/lib64/libpthread.so.0(+0xf370) [0x7f3d2de78370] |
/lib64/libc.so.6(gsignal+0x37) [0x7f3d2c1cd1d7] |
/lib64/libc.so.6(abort+0x148) [0x7f3d2c1ce8c8] |
/lib64/libstdc++.so.6(__gnu_cxx::__verbose_terminate_handler()+0x165) [0x7f3d2c8bb9d5] |
/lib64/libstdc++.so.6(+0x5e946) [0x7f3d2c8b9946] |
/lib64/libstdc++.so.6(+0x5e973) [0x7f3d2c8b9973] |
/lib64/libstdc++.so.6(+0x5eb93) [0x7f3d2c8b9b93] |
/lib64/libstdc++.so.6(std::__throw_out_of_range(char const*)+0x77) [0x7f3d2c90ea17] |
/usr/sbin/mysqld(+0x8d97a0) [0x7f3d2eb807a0] |
/usr/sbin/mysqld(+0x8de3a3) [0x7f3d2eb853a3] |
/usr/sbin/mysqld(+0x8e0805) [0x7f3d2eb87805] |
/usr/sbin/mysqld(+0x8e2573) [0x7f3d2eb89573] |
/lib64/libpthread.so.0(+0x7dc5) [0x7f3d2de70dc5] |
/lib64/libc.so.6(clone+0x6d) [0x7f3d2c28f73d] |
The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains |
information that should help you find out what is causing the crash.
|
This is unofficial patch for 10.1.23 that works for us
--- storage/xtradb/dict/dict0stats.cc.orig 2017-05-02 08:13:52.000000000 +0300
|
+++ storage/xtradb/dict/dict0stats.cc 2017-05-11 18:49:35.000000000 +0300
|
@@ -1168,10 +1168,10 @@
|
leaf-level delete marks because delete marks on
|
non-leaf level do not make sense. */
|
|
- if (level == 0 && srv_stats_include_delete_marked? 0:
|
+ if (level == 0 && (srv_stats_include_delete_marked? 0:
|
rec_get_deleted_flag(
|
rec,
|
- page_is_comp(btr_pcur_get_page(&pcur)))) {
|
+ page_is_comp(btr_pcur_get_page(&pcur))))) {
|
|
if (rec_is_last_on_page
|
&& !prev_rec_is_copied
|
Are there any updates on this issue? Our servers crash every day, we had to revert all lab/production systems to 10.1.18.
Thanks.
We've got InnoDB-5.6.36 in 10.0.31 and 10.1.24, so this issue should be fixed now.
Thanks Patrick. After patching XtraDB as you mentioned, mysqld is quite stable.