[MDEV-12281] MariaDB crashing regularly with: terminate called after throwing an instance of 'std::out_of_range' Created: 2017-03-16  Updated: 2019-03-01  Resolved: 2017-07-10

Status: Closed
Project: MariaDB Server
Component/s: Storage Engine - InnoDB
Affects Version/s: 10.1.22
Fix Version/s: 10.1.24, 10.0.31

Type: Bug Priority: Critical
Reporter: Ovidiu Stanila Assignee: Vicențiu Ciorbaru
Resolution: Fixed Votes: 4
Labels: upstream-fixed
Environment:

CentOS 7 x86_64

Attachments: Text File mariadb-crash.txt     File my.cnf    
Issue Links:
PartOf
is part of MDEV-12282 10.1.23 merge Closed
is part of MDEV-12283 10.0.31 merge Closed
Relates
relates to MDEV-18574 terminate called after throwing an in... Closed

 Description   

After upgrading to MariaDB 10.1.22 (from 10.1.21) the process started crashing randomly (didn't notice a pattern until now):

Mar 15 18:28:50 Updated: MariaDB-server-10.1.22-1.el7.centos.x86_64
Mar 15 18:58:21 HOST mysqld: *** buffer overflow detected ***: /usr/sbin/mysqld terminated
Mar 15 23:05:43 HOST mysqld: *** buffer overflow detected ***: /usr/sbin/mysqld terminated
Mar 16 06:49:15 HOST mysqld: *** buffer overflow detected ***: /usr/sbin/mysqld terminated
Mar 16 07:32:15 HOST mysqld: *** buffer overflow detected ***: /usr/sbin/mysqld terminated
Mar 16 09:17:21 HOST mysqld: *** buffer overflow detected ***: /usr/sbin/mysqld terminated
Mar 16 09:19:51 HOST mysqld: *** buffer overflow detected ***: /usr/sbin/mysqld terminated
Mar 16 09:22:18 HOST mysqld: *** buffer overflow detected ***: /usr/sbin/mysqld terminated
Mar 16 09:24:51 HOST mysqld: *** buffer overflow detected ***: /usr/sbin/mysqld terminated
Mar 16 09:26:00 HOST mysqld: *** buffer overflow detected ***: /usr/sbin/mysqld terminated

Server was stable until the upgrade.
I've attached the full crash report here.
Seems to be similar to what's reported here on 5.6.35-80.0:
https://bugs.launchpad.net/percona-server/+bug/1664519
https://bugs.mysql.com/bug.php?id=84940

We use this server as a multi-master slave, mainly for backups (done pretty often, almost constant read on this server). Seems to be happening on certain kind of slave updates coming from master.

We have an, almost, identical slave which doesn't experience this problem, the difference is that this server isn't used that much, no backups, mainly as a readonly access to data for development team, infrequent.

We downgraded to 10.1.21 to see if we still experience this problem. So far, so good.

Please let me know if any additional data is required.

Thank you.

 Comments   
Comment by Sergei Golubchik [ 2017-03-16 ]

If it is, indeed, bug#84940, then it's supposedly fixed in 5.6.36, and will be merged and released in the 10.1.23 release.

Comment by Ovidiu Stanila [ 2017-03-16 ]

Thanks, any idea when will version 10.1.23 be released ?

Comment by Sergei Golubchik [ 2017-03-16 ]

Of course! It is always on the start page jira.mariadb.org (unless you've created your own dashboard).

You'll need to be logged in to see it (until Atlassian fixes https://jira.atlassian.com/browse/JRA-59335).

Comment by Ovidiu Stanila [ 2017-03-16 ]

Silly me, didn't notice that, first time accessing this. Thanks for the quick reply.

Comment by Silver Asu [ 2017-04-06 ]

Is it possible to get patch for this issue before waiting for new official release?

Comment by Sergei Golubchik [ 2017-04-06 ]

MySQL-5.6.36 is still not released, we cannot merge the fix yet

Comment by Daniel Black [ 2017-04-11 ]

FYI it was released today.

Comment by Daniel Black [ 2017-04-11 ]

From ivan.tyagov in MDEV-11778:

Hi all,
I tend to believe that I got hit by same or similar issue. It happened for 3 times today but after that no issues.
Here's my log (mariadb_error.log) maybe this can help you out?

terminate called after throwing an instance of 'std::out_of_range'
 
  what():  vector::_M_range_check: __n (which is 4294967295) >= this->size() (which is 0)
 
170410 13:29:06 [ERROR] mysqld got signal 6 ;
 
This could be because you hit a bug. It is also possible that this binary
 
or one of the libraries it was linked against is corrupt, improperly built,
 
or misconfigured. This error can also be caused by malfunctioning hardware.
 
 
 
To report this bug, see https://mariadb.com/kb/en/reporting-bugs
 
 
 
We will try our best to scrape up some info that will hopefully help
 
diagnose the problem, but since we have already crashed,
 
something is definitely wrong and this may fail.
 
 
 
Server version: 10.1.22-MariaDB
 
key_buffer_size=134217728
 
read_buffer_size=131072
 
max_used_connections=96
 
max_threads=1002
 
thread_count=85
 
It is possible that mysqld could use up to
 
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 2331983 K  bytes of memory
 
Hope that's ok; if not, decrease some variables in the equation.
 
 
 
Thread pointer: 0x0
 
Attempting backtrace. You can use the following information to find out
 
where mysqld died. If you see no messages after this, something went
 
terribly wrong...
 
stack_bottom = 0x0 thread_stack 0x48400
 
/srv/slapgrid/slappart7/srv/runner/software/380bb82d455aefe8fec3d427143c9601/parts/mariadb/bin/mysqld(my_print_stacktrace+0x2e)[0x7f567f85a74e]
 
/srv/slapgrid/slappart7/srv/runner/software/380bb82d455aefe8fec3d427143c9601/parts/mariadb/bin/mysqld(handle_fatal_signal+0x2fd)[0x7f567f39480d]
 
/lib/x86_64-linux-gnu/libpthread.so.0(+0xf8d0)[0x7f567e9bf8d0]
 
/lib/x86_64-linux-gnu/libc.so.6(gsignal+0x37)[0x7f567ca64107]
 
/lib/x86_64-linux-gnu/libc.so.6(abort+0x148)[0x7f567ca654e8]
 
/usr/lib/x86_64-linux-gnu/libstdc++.so.6(_ZN9__gnu_cxx27__verbose_terminate_handlerEv+0x15d)[0x7f567d139b3d]
 
/usr/lib/x86_64-linux-gnu/libstdc++.so.6(+0x5ebb6)[0x7f567d137bb6]
 
/usr/lib/x86_64-linux-gnu/libstdc++.so.6(+0x5ec01)[0x7f567d137c01]
 
/usr/lib/x86_64-linux-gnu/libstdc++.so.6(+0x5ee19)[0x7f567d137e19]
 
/usr/lib/x86_64-linux-gnu/libstdc++.so.6(_ZSt24__throw_out_of_range_fmtPKcz+0x11f)[0x7f567d18dcdf]
 
/srv/slapgrid/slappart7/srv/runner/software/380bb82d455aefe8fec3d427143c9601/parts/mariadb/bin/mysqld(+0x9c32bb)[0x7f567f7b32bb]
 
/srv/slapgrid/slappart7/srv/runner/software/380bb82d455aefe8fec3d427143c9601/parts/mariadb/bin/mysqld(+0x9c7629)[0x7f567f7b7629]
 
/srv/slapgrid/slappart7/srv/runner/software/380bb82d455aefe8fec3d427143c9601/parts/mariadb/bin/mysqld(+0x9ca959)[0x7f567f7ba959]
 
/srv/slapgrid/slappart7/srv/runner/software/380bb82d455aefe8fec3d427143c9601/parts/mariadb/bin/mysqld(+0x9cc72b)[0x7f567f7bc72b]
 
/lib/x86_64-linux-gnu/libpthread.so.0(+0x80a4)[0x7f567e9b80a4]
 
/lib/x86_64-linux-gnu/libc.so.6(clone+0x6d)[0x7f567cb1504d]
 
The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
 
information that should help you find out what is causing the crash.
 
handlersocket: initialized

Comment by Kazuhiko Shiozaki [ 2017-04-14 ]

Hi !

Oracle has just released new versions of MySQL and now we see their fixes. For the 5.7 branch:

https://github.com/mysql/mysql-server/commit/0241e1c1b216d87050cdfcb4531ffc17f9eb1dc7

The same fix should work on the MariaDB:

https://github.com/MariaDB/server/blob/10.1/storage/innobase/dict/dict0stats.cc#L1165

And the 10.2 branch does not seem affected:

https://github.com/MariaDB/server/blob/10.2/storage/innobase/dict/dict0stats.cc#L1154

Comment by Silver Asu [ 2017-04-25 ]

For us, this fix does not work

Comment by Marko Mäkelä [ 2017-04-27 ]

BB, why does the fix not work for you?
As noted in MDEV-11751, today I pushed to our upcoming 10.2.6 release the merge of a number of upstream patches, including this port of the MySQL 5.7 patch mentioned by kazuhiko.fdiary. The MySQL 5.6 patch indeed first appeared in MySQL 5.6.36, whose InnoDB we have not merged yet.

Comment by Pavel Kaidalov [ 2017-04-30 ]

I just faced this problem at replica and can't recover it - this is CentOS 7, MariaDB 10.1.22, and here is logs, first is the initial crash:

2017-04-29  0:45:19 140630256495360 [Note] InnoDB: Online DDL : End of applying row log
 
2017-04-29  0:45:19 140630256495360 [Note] InnoDB: Online DDL : Completed
 
terminate called after throwing an instance of 'std::out_of_range'
 
  what():  vector::_M_range_check
 
170429 15:49:11 [ERROR] mysqld got signal 6 ;
 
This could be because you hit a bug. It is also possible that this binary
 
or one of the libraries it was linked against is corrupt, improperly built,
 
or misconfigured. This error can also be caused by malfunctioning hardware.
 
 
 
To report this bug, see https://mariadb.com/kb/en/reporting-bugs
 
 
 
We will try our best to scrape up some info that will hopefully help
 
diagnose the problem, but since we have already crashed, 
something is definitely wrong and this may fail.
 
 
 
Server version: 10.1.22-MariaDB
 
key_buffer_size=134217728
 
read_buffer_size=131072
 
max_used_connections=6
 
max_threads=3002
 
thread_count=1
 
It is possible that mysqld could use up to 
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 6725061 K  bytes of memory
 
Hope that's ok; if not, decrease some variables in the equation.
 
 
 
Thread pointer: 0x0
 
Attempting backtrace. You can use the following information to find out
 
where mysqld died. If you see no messages after this, something went
 
terribly wrong...
 
stack_bottom = 0x0 thread_stack 0x48400
 
(my_addr_resolve failure: fork)
 
/usr/sbin/mysqld(my_print_stacktrace+0x2e) [0x7fe70939210e]
 
/usr/sbin/mysqld(handle_fatal_signal+0x305) [0x7fe708eb59d5]
 
/lib64/libpthread.so.0(+0xf370) [0x7fe7084d0370]
 
/lib64/libc.so.6(gsignal+0x37) [0x7fe7068251d7]
 
/lib64/libc.so.6(abort+0x148) [0x7fe7068268c8]
 
/lib64/libstdc++.so.6(__gnu_cxx::__verbose_terminate_handler()+0x165) [0x7fe706f139d5]
 
/lib64/libstdc++.so.6(+0x5e946) [0x7fe706f11946]
 
/lib64/libstdc++.so.6(+0x5e973) [0x7fe706f11973]
 
/lib64/libstdc++.so.6(+0x5eb93) [0x7fe706f11b93]
 
/lib64/libstdc++.so.6(std::__throw_out_of_range(char const*)+0x77) [0x7fe706f66a17]
 
/usr/sbin/mysqld(+0x8d7710) [0x7fe7091d6710]
 
/usr/sbin/mysqld(+0x8dc383) [0x7fe7091db383]
 
/usr/sbin/mysqld(+0x8de854) [0x7fe7091dd854]
 
/usr/sbin/mysqld(+0x8e05b3) [0x7fe7091df5b3]
 
/lib64/libpthread.so.0(+0x7dc5) [0x7fe7084c8dc5]
 
/lib64/libc.so.6(clone+0x6d) [0x7fe7068e773d]
 
The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
 
information that should help you find out what is causing the crash.
 
170429 15:49:12 mysqld_safe Number of processes running now: 0
 
170429 15:49:12 mysqld_safe WSREP: not restarting wsrep node automatically
 
170429 15:49:12 mysqld_safe mysqld from pid file /var/lib/mysql/host.pid ended

and second is crashing with the same error on recovery and connecting to replication master:

...
 
2017-04-30  6:16:45 140561292105984 [Note] InnoDB: 128 rollback segment(s) are active.
 
2017-04-30  6:16:45 140561292105984 [Note] InnoDB: Waiting for purge to start
 
2017-04-30  6:16:45 140561292105984 [Note] InnoDB:  Percona XtraDB (http://www.percona.com) 5.6.35-80.0 started; log sequence number 10888962626082
 
2017-04-30  6:16:45 140556940343040 [Note] InnoDB: Dumping buffer pool(s) not yet started
 
2017-04-30  6:16:45 140561292105984 [Note] Plugin 'FEEDBACK' is disabled.
 
2017-04-30  6:16:45 140561292105984 [Note] Recovering after a crash using tc.log
 
2017-04-30  6:16:45 140561292105984 [Note] Starting crash recovery...
 
2017-04-30  6:16:45 140561292105984 [Note] Crash recovery finished.
 
2017-04-30  6:16:45 140561292105984 [Note] Server socket created on IP: '::'.
 
2017-04-30  6:16:45 140561292105984 [Note] Server socket created on IP: '::'.
 
2017-04-30  6:17:05 140561292105984 [Note] /usr/sbin/mysqld: ready for connections.
 
Version: '10.1.22-MariaDB'  socket: '/var/lib/mysql/mysql.sock'  port: 3306  MariaDB Server
 
2017-04-30  6:17:05 140561291070208 [Note] Slave SQL thread initialized, starting replication in log 'mysql-bin.001539' at position 4, relay log './mysql-relay-bin.000001' position: 4; GTID position '0-101132-703327796,1555-101157-1519563402'
 
2017-04-30  6:17:05 140561291373312 [Note] Slave I/O thread: connected to master 'replica@10.10.11.32:3306',replication starts at GTID position '1555-101157-1519563402,0-101132-703327796'
 
terminate called after throwing an instance of 'std::out_of_range'
 
  what():  vector::_M_range_check
 
170430  6:17:07 [ERROR] mysqld got signal 6 ;
 
This could be because you hit a bug. It is also possible that this binary
 
or one of the libraries it was linked against is corrupt, improperly built,
 
or misconfigured. This error can also be caused by malfunctioning hardware.
 
 
 
To report this bug, see https://mariadb.com/kb/en/reporting-bugs
 
 
 
We will try our best to scrape up some info that will hopefully help
 
diagnose the problem, but since we have already crashed, 
something is definitely wrong and this may fail.
 
 
 
Server version: 10.1.22-MariaDB
 
key_buffer_size=134217728
 
read_buffer_size=131072
 
max_used_connections=1
 
max_threads=3002
 
thread_count=0
 
It is possible that mysqld could use up to 
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 6725061 K  bytes of memory
 
Hope that's ok; if not, decrease some variables in the equation.
 
 
 
Thread pointer: 0x0
 
Attempting backtrace. You can use the following information to find out
 
where mysqld died. If you see no messages after this, something went
 
terribly wrong...
 
stack_bottom = 0x0 thread_stack 0x48400
 
*** buffer overflow detected ***: /usr/sbin/mysqld terminated
 
======= Backtrace: =========
 
/lib64/libc.so.6(__fortify_fail+0x37)[0x7fd6f7e71047]
 
/lib64/libc.so.6(+0x10d200)[0x7fd6f7e6f200]
 
/lib64/libc.so.6(+0x10efb7)[0x7fd6f7e70fb7]
 
/usr/sbin/mysqld(my_addr_resolve+0x48)[0x7fd6fa917c28]
 
/usr/sbin/mysqld(my_print_stacktrace+0x1c2)[0x7fd6fa9042a2]
 
/usr/sbin/mysqld(handle_fatal_signal+0x305)[0x7fd6fa4279d5]
 
/lib64/libpthread.so.0(+0xf370)[0x7fd6f9a42370]
 
/lib64/libc.so.6(gsignal+0x37)[0x7fd6f7d971d7]
 
/lib64/libc.so.6(abort+0x148)[0x7fd6f7d988c8]
 
/lib64/libstdc++.so.6(_ZN9__gnu_cxx27__verbose_terminate_handlerEv+0x165)[0x7fd6f84859d5]
 
/lib64/libstdc++.so.6(+0x5e946)[0x7fd6f8483946]
 
/lib64/libstdc++.so.6(+0x5e973)[0x7fd6f8483973]
 
/lib64/libstdc++.so.6(+0x5eb93)[0x7fd6f8483b93]
 
/lib64/libstdc++.so.6(_ZSt20__throw_out_of_rangePKc+0x77)[0x7fd6f84d8a17]
 
/usr/sbin/mysqld(+0x8d7710)[0x7fd6fa748710]
 
/usr/sbin/mysqld(+0x8dc383)[0x7fd6fa74d383]
 
/usr/sbin/mysqld(+0x8de854)[0x7fd6fa74f854]
 
/usr/sbin/mysqld(+0x8e05b3)[0x7fd6fa7515b3]
 
/lib64/libpthread.so.0(+0x7dc5)[0x7fd6f9a3adc5]
 
/lib64/libc.so.6(clone+0x6d)[0x7fd6f7e5973d]
 
======= Memory map: ========
 
7fd5f2000000-7fd5f3400000 rw-p 00000000 00:00 0 
7fd5f37ff000-7fd5f3800000 ---p 00000000 00:00 0 
7fd5f3800000-7fd5f4800000 rw-p 00000000 00:00 0                          [stack:6452]
 
7fd5f4800000-7fd5f4c00000 rw-p 00000000 00:00 0 
7fd5f4ffd000-7fd5f4ffe000 ---p 00000000 00:00 0 
7fd5f4ffe000-7fd5f57fe000 rw-p 00000000 00:00 0                          [stack:6451]
 
7fd5f57fe000-7fd5f57ff000 ---p 00000000 00:00 0 
7fd5f57ff000-7fd5f5fff000 rw-p 00000000 00:00 0                          [stack:6450]
 
7fd5f5fff000-7fd5f6000000 ---p 00000000 00:00 0 
7fd5f6000000-7fd5f7000000 rw-p 00000000 00:00 0                          [stack:6449]
 
7fd5f73fb000-7fd5f73fc000 ---p 00000000 00:00 0 
7fd5f73fc000-7fd5f7bfc000 rw-p 00000000 00:00 0                          [stack:6448]
 
7fd5f7bfc000-7fd5f7bfd000 ---p 00000000 00:00 0 
7fd5f7bfd000-7fd5f83fd000 rw-p 00000000 00:00 0                          [stack:6447]
 
7fd5f83fd000-7fd5f83fe000 ---p 00000000 00:00 0 
7fd5f83fe000-7fd5f8bfe000 rw-p 00000000 00:00 0                          [stack:6446]
 
7fd5f8bfe000-7fd5f8bff000 ---p 00000000 00:00 0 
7fd5f8bff000-7fd5f93ff000 rw-p 00000000 00:00 0                          [stack:6445]
 
7fd5f93ff000-7fd5f9400000 ---p 00000000 00:00 0 
7fd5f9400000-7fd605800000 rw-p 00000000 00:00 0                          [stack:6444]
 
7fd605bff000-7fd605c00000 ---p 00000000 00:00 0 
7fd605c00000-7fd607c00000 rw-p 00000000 00:00 0                          [stack:6443]
 
7fd607fff000-7fd608000000 ---p 00000000 00:00 0 
7fd608000000-7fd60c400000 rw-p 00000000 00:00 0                          [stack:1921]
 
7fd60c7f6000-7fd60c7f7000 ---p 00000000 00:00 0 
7fd60c7f7000-7fd60cff7000 rw-p 00000000 00:00 0                          [stack:1892]
 
7fd60cff7000-7fd60cff8000 ---p 00000000 00:00 0 
7fd60cff8000-7fd60d7f8000 rw-p 00000000 00:00 0                          [stack:1891]
 
7fd60d7f8000-7fd60d7f9000 ---p 00000000 00:00 0 
7fd60d7f9000-7fd60dff9000 rw-p 00000000 00:00 0                          [stack:1890]
 
7fd60dff9000-7fd60dffa000 ---p 00000000 00:00 0 
7fd60dffa000-7fd60e7fa000 rw-p 00000000 00:00 0                          [stack:1889]
 
7fd60e7fa000-7fd60e7fb000 ---p 00000000 00:00 0 
7fd60e7fb000-7fd60effb000 rw-p 00000000 00:00 0                          [stack:1888]
 
7fd60effb000-7fd60effc000 ---p 00000000 00:00 0 
7fd60effc000-7fd60f7fc000 rw-p 00000000 00:00 0                          [stack:1887]
 
7fd60f7fc000-7fd60f7fd000 ---p 00000000 00:00 0 
7fd60f7fd000-7fd60fffd000 rw-p 00000000 00:00 0                          [stack:1886]
 
7fd60fffd000-7fd60fffe000 ---p 00000000 00:00 0 
7fd60fffe000-7fd6107fe000 rw-p 00000000 00:00 0                          [stack:1885]
 
7fd6107fe000-7fd6107ff000 ---p 00000000 00:00 0 
7fd6107ff000-7fd610fff000 rw-p 00000000 00:00 0                          [stack:1884]
 
7fd610fff000-7fd611000000 ---p 00000000 00:00 0 
7fd611000000-7fd617400000 rw-p 00000000 00:00 0                          [stack:1883]
 
7fd617780000-7fd65ac00000 rw-p 00000000 00:00 0 
7fd65af80000-7fd69e400000 rw-p 00000000 00:00 0 
7fd69e780000-7fd6e1c00000 rw-p 00000000 00:00 0 
7fd6e1df9000-7fd6e1dfe000 r-xp 00000000 ca:01 539555                     /usr/lib64/libnss_dns-2.17.so
 
7fd6e1dfe000-7fd6e1ffd000 ---p 00005000 ca:01 539555                     /usr/lib64/libnss_dns-2.17.so
 
7fd6e1ffd000-7fd6e1ffe000 r--p 00004000 ca:01 539555                     /usr/lib64/libnss_dns-2.17.so
 
7fd6e1ffe000-7fd6e1fff000 rw-p 00005000 ca:01 539555                     /usr/lib64/libnss_dns-2.17.so
 
7fd6e1fff000-7fd6e2000000 ---p 00000000 00:00 0 
7fd6e2000000-7fd6eb400000 rw-p 00000000 00:00 0                          [stack:1882]
 
7fd6eb5e8000-7fd6eb5fe000 r-xp 00000000 ca:01 10138504                   /usr/lib64/mysql/plugin/ha_federatedx.so
 
7fd6eb5fe000-7fd6eb7fe000 ---p 00016000 ca:01 10138504                   /usr/lib64/mysql/plugin/ha_federatedx.so
 
7fd6eb7fe000-7fd6eb7ff000 r--p 00016000 ca:01 10138504                   /usr/lib64/mysql/plugin/ha_federatedx.so
 
7fd6eb7ff000-7fd6eb800000 rw-p 00017000 ca:01 10138504                   /usr/lib64/mysql/plugin/ha_federatedx.so
 
7fd6eb800000-7fd6f4000000 rw-p 00000000 00:00 0 
7fd6f41ec000-7fd6f41ed000 ---p 00000000 00:00 0 
7fd6f41ed000-7fd6f49ed000 rw-p 00000000 00:00 0                          [stack:1873]
 
7fd6f49ed000-7fd6f49f9000 r-xp 00000000 ca:01 539557                     /usr/lib64/libnss_files-2.17.so
 
7fd6f49f9000-7fd6f4bf8000 ---p 0000c000 ca:01 539557                     /usr/lib64/libnss_files-2.17.so
 
7fd6f4bf8000-7fd6f4bf9000 r--p 0000b000 ca:01 539557                     /usr/lib64/libnss_files-2.17.so
 
7fd6f4bf9000-7fd6f4bfa000 rw-p 0000c000 ca:01 539557                     /usr/lib64/libnss_files-2.17.so
 
7fd6f4bfa000-7fd6f5400000 rw-p 00000000 00:00 0 
7fd6f5421000-7fd6f5430000 r-xp 00000000 ca:01 125287                     /usr/lib64/libbz2.so.1.0.6
 
7fd6f5430000-7fd6f562f000 ---p 0000f000 ca:01 125287                     /usr/lib64/libbz2.so.1.0.6
 
7fd6f562f000-7fd6f5630000 r--p 0000e000 ca:01 125287                     /usr/lib64/libbz2.so.1.0.6
 
7fd6f5630000-7fd6f5631000 rw-p 0000f000 ca:01 125287                     /usr/lib64/libbz2.so.1.0.6
 
7fd6f5631000-7fd6f5648000 r-xp 00000000 ca:01 539582                     /usr/lib64/libelf-0.166.so
 
7fd6f5648000-7fd6f5847000 ---p 00017000 ca:01 539582                     /usr/lib64/libelf-0.166.so
 
7fd6f5847000-7fd6f5848000 r--p 00016000 ca:01 539582                     /usr/lib64/libelf-0.166.so
 
7fd6f5848000-7fd6f5849000 rw-p 00017000 ca:01 539582                     /usr/lib64/libelf-0.166.so
 
7fd6f5849000-7fd6f58a9000 r-xp 00000000 ca:01 3479                       /usr/lib64/libpcre.so.1.2.0
 
7fd6f58a9000-7fd6f5aa8000 ---p 00060000 ca:01 3479                       /usr/lib64/libpcre.so.1.2.0
 
7fd6f5aa8000-7fd6f5aa9000 r--p 0005f000 ca:01 3479                       /usr/lib64/libpcre.so.1.2.0
 
7fd6f5aa9000-7fd6f5aaa000 rw-p 00060000 ca:01 3479                       /usr/lib64/libpcre.so.1.2.0
 
7fd6f5aaa000-7fd6f5aae000 r-xp 00000000 ca:01 125337                     /usr/lib64/libattr.so.1.1.0
 
7fd6f5aae000-7fd6f5cad000 ---p 00004000 ca:01 125337                     /usr/lib64/libattr.so.1.1.0
 
7fd6f5cad000-7fd6f5cae000 r--p 00003000 ca:01 125337                     /usr/lib64/libattr.so.1.1.0
 
7fd6f5cae000-7fd6f5caf000 rw-p 00004000 ca:01 125337                     /usr/lib64/libattr.so.1.1.0
 
7fd6f5caf000-7fd6f5cb2000 r-xp 00000000 ca:01 125778                     /usr/lib64/libkeyutils.so.1.5
 
7fd6f5cb2000-7fd6f5eb1000 ---p 00003000 ca:01 125778                     /usr/lib64/libkeyutils.so.1.5
 
7fd6f5eb1000-7fd6f5eb2000 r--p 00002000 ca:01 125778                     /usr/lib64/libkeyutils.so.1.5
 
7fd6f5eb2000-7fd6f5eb3000 rw-p 00003000 ca:01 125778                     /usr/lib64/libkeyutils.so.1.5
 
7fd6f5eb3000-7fd6f5ec0000 r-xp 00000000 ca:01 126272                     /usr/lib64/libkrb5support.so.0.1
 
7fd6f5ec0000-7fd6f60c0000 ---p 0000d000 ca:01 126272                     /usr/lib64/libkrb5support.so.0.1
 
7fd6f60c0000-7fd6f60c1000 r--p 0000d000 ca:01 126272                     /usr/lib64/libkrb5support.so.0.1
 
7fd6f60c1000-7fd6f60c2000 rw-p 0000e000 ca:01 126272                     /usr/lib64/libkrb5support.so.0.1
 
7fd6f60c2000-7fd6f60d7000 r-xp 00000000 ca:01 125123                     /usr/lib64/libgcc_s-4.8.5-20150702.so.1
 
7fd6f60d7000-7fd6f62d6000 ---p 00015000 ca:01 125123                     /usr/lib64/libgcc_s-4.8.5-20150702.so.1
 
7fd6f62d6000-7fd6f62d7000 r--p 00014000 ca:01 125123                     /usr/lib64/libgcc_s-4.8.5-20150702.so.1
 
7fd6f62d7000-7fd6f62d8000 rw-p 00015000 ca:01 125123                     /usr/lib64/libgcc_s-4.8.5-20150702.so.1
 
7fd6f62d8000-7fd6f631d000 r-xp 00000000 ca:01 780                        /usr/lib64/libdw-0.166.so
 
7fd6f631d000-7fd6f651d000 ---p 00045000 ca:01 780                        /usr/lib64/libdw-0.166.so
 
7fd6f651d000-7fd6f651f000 r--p 00045000 ca:01 780                        /usr/lib64/libdw-0.166.so
 
7fd6f651f000-7fd6f6520000 rw-p 00047000 ca:01 780                        /usr/lib64/libdw-0.166.so
 
7fd6f6520000-7fd6f6536000 r-xp 00000000 ca:01 539569                     /usr/lib64/libresolv-2.17.so
 
7fd6f6536000-7fd6f6736000 ---p 00016000 ca:01 539569                     /usr/lib64/libresolv-2.17.so
 
7fd6f6736000-7fd6f6737000 r--p 00016000 ca:01 539569                     /usr/lib64/libresolv-2.17.so
 
7fd6f6737000-7fd6f6738000 rw-p 00017000 ca:01 539569                     /usr/lib64/libresolv-2.17.so
 
7fd6f6738000-7fd6f673a000 rw-p 00000000 00:00 0 
7fd6f673a000-7fd6f673e000 r-xp 00000000 ca:01 125343                     /usr/lib64/libgpg-error.so.0.10.0
 
7fd6f673e000-7fd6f693d000 ---p 00004000 ca:01 125343                     /usr/lib64/libgpg-error.so.0.10.0
 
7fd6f693d000-7fd6f693e000 r--p 00003000 ca:01 125343                     /usr/lib64/libgpg-error.so.0.10.0
 
7fd6f693e000-7fd6f693f000 rw-p 00004000 ca:01 125343                     /usr/lib64/libgpg-error.so.0.10.0
 
7fd6f693f000-7fd6f69bb000 r-xp 00000000 ca:01 777                        /usr/lib64/libgcrypt.so.11.8.2
 
7fd6f69bb000-7fd6f6bbb000 ---p 0007c000 ca:01 777                        /usr/lib64/libgcrypt.so.11.8.2
 
7fd6f6bbb000-7fd6f6bbc000 r--p 0007c000 ca:01 777                        /usr/lib64/libgcrypt.so.11.8.2
 
7fd6f6bbc000-7fd6f6bbf000 rw-p 0007d000 ca:01 777                        /usr/lib64/libgcrypt.so.11.8.2
 
7fd6f6bbf000-7fd6f6bc0000 rw-p 00000000 00:00 0 
7fd6f6bc0000-7fd6f6be4000 r-xp 00000000 ca:01 153297                     /usr/lib64/libselinux.so.1
 
7fd6f6be4000-7fd6f6de3000 ---p 00024000 ca:01 153297                     /usr/lib64/libselinux.so.1
 
7fd6f6de3000-7fd6f6de4000 r--p 00023000 ca:01 153297                     /usr/lib64/libselinux.so.1
 
7fd6f6de4000-7fd6f6de5000 rw-p 00024000 ca:01 153297                     /usr/lib64/libselinux.so.1
 
7fd6f6de5000-7fd6f6de7000 rw-p 00000000 00:00 0 
7fd6f6de7000-7fd6f6dee000 r-xp 00000000 ca:01 539571                     /usr/lib64/librt-2.17.so
 
7fd6f6dee000-7fd6f6fed000 ---p 00007000 ca:01 539571                     /usr/lib64/librt-2.17.so
 
7fd6f6fed000-7fd6f6fee000 r--p 00006000 ca:01 539571                     /usr/lib64/librt-2.17.so
 
7fd6f6fee000-7fd6f6fef000 rw-p 00007000 ca:01 539571                     /usr/lib64/librt-2.17.so
 
7fd6f6fef000-7fd6f6ff3000 r-xp 00000000 ca:01 125390                     /usr/lib64/libcap.so.2.22
 
7fd6f6ff3000-7fd6f71f2000 ---p 00004000 ca:01 125390                     /usr/lib64/libcap.so.2.22
 
7fd6f71f2000-7fd6f71f3000 r--p 00003000 ca:01 125390                     /usr/lib64/libcap.so.2.22
 
7fd6f71f3000-7fd6f71f4000 rw-p 00004000 ca:01 125390                     /usr/lib64/libcap.so.2.22
 
7fd6f71f4000-7fd6f7223000 r-xp 00000000 ca:01 103261                     /usr/lib64/libk5crypto.so.3.1
 
7fd6f7223000-7fd6f7422000 ---p 0002f000 ca:01 103261                     /usr/lib64/libk5crypto.so.3.1
 
7fd6f7422000-7fd6f7424000 r--p 0002e000 ca:01 103261                     /usr/lib64/libk5crypto.so.3.1
 
7fd6f7424000-7fd6f7425000 rw-p 00030000 ca:01 103261                     /usr/lib64/libk5crypto.so.3.1
 
7fd6f7425000-7fd6f7426000 rw-p 00000000 00:00 0 
7fd6f7426000-7fd6f7429000 r-xp 00000000 ca:01 153306                     /usr/lib64/libcom_err.so.2.1
 
7fd6f7429000-7fd6f7628000 ---p 00003000 ca:01 153306                     /usr/lib64/libcom_err.so.2.1
 
7fd6f7628000-7fd6f7629000 r--p 00002000 ca:01 153306                     /usr/lib64/libcom_err.so.2.1
 
7fd6f7629000-7fd6f762a000 rw-p 00003000 ca:01 153306                     /usr/lib64/libcom_err.so.2.1
 
7fd6f762a000-7fd6f7700000 r-xp 00000000 ca:01 126270                     /usr/lib64/libkrb5.so.3.3
 
7fd6f7700000-7fd6f7900000 ---p 000d6000 ca:01 126270                     /usr/lib64/libkrb5.so.3.3
 
7fd6f7900000-7fd6f790e000 r--p 000d6000 ca:01 126270                     /usr/lib64/libkrb5.so.3.3
 
7fd6f790e000-7fd6f7911000 rw-p 000e4000 ca:01 126270                     /usr/lib64/libkrb5.so.3.3
 
7fd6f7911000-7fd6f795c000 r-xp 00000000 ca:01 103256                     /usr/lib64/libgssapi_krb5.so.2.2
 
7fd6f795c000-7fd6f7b5c000 ---p 0004b000 ca:01 103256                     /usr/lib64/libgssapi_krb5.so.2.2
 
7fd6f7b5c000-7fd6f7b5d000 r--p 0004b000 ca:01 103256                     /usr/lib64/libgssapi_krb5.so.2.2
 
7fd6f7b5d000-7fd6f7b5f000 rw-p 0004c000 ca:01 103256                     /usr/lib64/libgssapi_krb5.so.2.2
 
7fd6f7b5f000-7fd6f7b61000 r-xp 00000000 ca:01 1469                       /usr/lib64/libfreebl3.so
 
7fd6f7b61000-7fd6f7d60000 ---p 00002000 ca:01 1469                       /usr/lib64/libfreebl3.so
 
7fd6f7d60000-7fd6f7d61000 r--p 00001000 ca:01 1469                       /usr/lib64/libfreebl3.so
 
7fd6f7d61000-7fd6f7d62000 rw-p 00002000 ca:01 1469                       /usr/lib64/libfreebl3.so
 
7fd6f7d62000-7fd6f7f18000 r-xp 00000000 ca:01 1545                       /usr/lib64/libc-2.17.so
 
7fd6f7f18000-7fd6f8118000 ---p 001b6000 ca:01 1545                       /usr/lib64/libc-2.17.so
 
7fd6f8118000-7fd6f811c000 r--p 001b6000 ca:01 1545                       /usr/lib64/libc-2.17.so
 
7fd6f811c000-7fd6f811e000 rw-p 001ba000 ca:01 1545                       /usr/lib64/libc-2.17.so
 
7fd6f811e000-7fd6f8123000 rw-p 00000000 00:00 0 
7fd6f8123000-7fd6f8223000 r-xp 00000000 ca:01 539547                     /usr/lib64/libm-2.17.so
 
7fd6f8223000-7fd6f8423000 ---p 00100000 ca:01 539547                     /usr/lib64/libm-2.17.so
 
7fd6f8423000-7fd6f8424000 r--p 00100000 ca:01 539547                     /usr/lib64/libm-2.17.so
 
7fd6f8424000-7fd6f8425000 rw-p 00101000 ca:01 539547                     /usr/lib64/libm-2.17.so
 
7fd6f8425000-7fd6f850e000 r-xp 00000000 ca:01 51703                      /usr/lib64/libstdc++.so.6.0.19
 
7fd6f850e000-7fd6f870e000 ---p 000e9000 ca:01 51703                      /usr/lib64/libstdc++.so.6.0.19
 
7fd6f870e000-7fd6f8717000 r--p 000e9000 ca:01 51703                      /usr/lib64/libstdc++.so.6.0.19
 
7fd6f8717000-7fd6f8719000 rw-p 000f2000 ca:01 51703                      /usr/lib64/libstdc++.so.6.0.19
 
7fd6f8719000-7fd6f872e000 rw-p 00000000 00:00 0 
7fd6f872e000-7fd6f8730000 r-xp 00000000 ca:01 539545                     /usr/lib64/libdl-2.17.so
 
7fd6f8730000-7fd6f8930000 ---p 00002000 ca:01 539545                     /usr/lib64/libdl-2.17.so
 
7fd6f8930000-7fd6f8931000 r--p 00002000 ca:01 539545                     /usr/lib64/libdl-2.17.so
 
7fd6f8931000-7fd6f8932000 rw-p 00003000 ca:01 539545                     /usr/lib64/libdl-2.17.so
 
7fd6f8932000-7fd6f8af2000 r-xp 00000000 ca:01 126279                     /usr/lib64/libcrypto.so.1.0.1e
 
7fd6f8af2000-7fd6f8cf2000 ---p 001c0000 ca:01 126279                     /usr/lib64/libcrypto.so.1.0.1e
 
7fd6f8cf2000-7fd6f8d0c000 r--p 001c0000 ca:01 126279                     /usr/lib64/libcrypto.so.1.0.1e
 
7fd6f8d0c000-7fd6f8d18000 rw-p 001da000 ca:01 126279                     /usr/lib64/libcrypto.so.1.0.1e
 
7fd6f8d18000-7fd6f8d1c000 rw-p 00000000 00:00 0 
7fd6f8d1c000-7fd6f8d80000 r-xp 00000000 ca:01 463274                     /usr/lib64/libssl.so.1.0.1e
 
7fd6f8d80000-7fd6f8f7f000 ---p 00064000 ca:01 463274                     /usr/lib64/libssl.so.1.0.1e
 
7fd6f8f7f000-7fd6f8f83000 r--p 00063000 ca:01 463274                     /usr/lib64/libssl.so.1.0.1e
 
7fd6f8f83000-7fd6f8f8a000 rw-p 00067000 ca:01 463274                     /usr/lib64/libssl.so.1.0.1e
 
7fd6f8f8a000-7fd6f8f92000 r-xp 00000000 ca:01 539543                     /usr/lib64/libcrypt-2.17.so
 
7fd6f8f92000-7fd6f9191000 ---p 00008000 ca:01 539543                     /usr/lib64/libcrypt-2.17.so
 
7fd6f9191000-7fd6f9192000 r--p 00007000 ca:01 539543                     /usr/lib64/libcrypt-2.17.so
 
7fd6f9192000-7fd6f9193000 rw-p 00008000 ca:01 539543                     /usr/lib64/libcrypt-2.17.so
 
7fd6f9193000-7fd6f91c1000 rw-p 00000000 00:00 0 
7fd6f91c1000-7fd6f91f2000 r-xp 00000000 ca:01 103251                     /usr/lib64/libjemalloc.so.1
 
7fd6f91f2000-7fd6f93f1000 ---p 00031000 ca:01 103251                     /usr/lib64/libjemalloc.so.1
 
7fd6f93f1000-7fd6f93f3000 r--p 00030000 ca:01 103251                     /usr/lib64/libjemalloc.so.1
 
7fd6f93f3000-7fd6f93f4000 rw-p 00032000 ca:01 103251                     /usr/lib64/libjemalloc.so.1
 
7fd6f93f4000-7fd6f93f5000 rw-p 00000000 00:00 0 
7fd6f93f5000-7fd6f940a000 r-xp 00000000 ca:01 153304                     /usr/lib64/libz.so.1.2.7
 
7fd6f940a000-7fd6f9609000 ---p 00015000 ca:01 153304                     /usr/lib64/libz.so.1.2.7
 
7fd6f9609000-7fd6f960a000 r--p 00014000 ca:01 153304                     /usr/lib64/libz.so.1.2.7
 
7fd6f960a000-7fd6f960b000 rw-p 00015000 ca:01 153304                     /usr/lib64/libz.so.1.2.7
 
7fd6f960b000-7fd6f960c000 r-xp 00000000 ca:01 197872                     /usr/lib64/libaio.so.1.0.1
 
7fd6f960c000-7fd6f980b000 ---p 00001000 ca:01 197872                     /usr/lib64/libaio.so.1.0.1
 
7fd6f980b000-7fd6f980c000 r--p 00000000 ca:01 197872                     /usr/lib64/libaio.so.1.0.1
 
7fd6f980c000-7fd6f980d000 rw-p 00001000 ca:01 197872                     /usr/lib64/libaio.so.1.0.1
 
7fd6f980d000-7fd6f9832000 r-xp 00000000 ca:01 153308                     /usr/lib64/liblzma.so.5.2.2
 
7fd6f9832000-7fd6f9a31000 ---p 00025000 ca:01 153308                     /usr/lib64/liblzma.so.5.2.2
 
7fd6f9a31000-7fd6f9a32000 r--p 00024000 ca:01 153308                     /usr/lib64/liblzma.so.5.2.2
 
7fd6f9a32000-7fd6f9a33000 rw-p 00025000 ca:01 153308                     /usr/lib64/liblzma.so.5.2.2
 
7fd6f9a33000-7fd6f9a4a000 r-xp 00000000 ca:01 539565                     /usr/lib64/libpthread-2.17.so
 
7fd6f9a4a000-7fd6f9c49000 ---p 00017000 ca:01 539565                     /usr/lib64/libpthread-2.17.so
 
7fd6f9c49000-7fd6f9c4a000 r--p 00016000 ca:01 539565                     /usr/lib64/libpthread-2.17.so
 
7fd6f9c4a000-7fd6f9c4b000 rw-p 00017000 ca:01 539565                     /usr/lib64/libpthread-2.17.so
 
7fd6f9c4b000-7fd6f9c4f000 rw-p 00000000 00:00 0 
7fd6f9c4f000-7fd6f9c6f000 r-xp 00000000 ca:01 1538                       /usr/lib64/ld-2.17.so
 
7fd6f9c99000-7fd6f9c9a000 rw-p 00000000 00:00 0 
7fd6f9c9a000-7fd6f9c9b000 ---p 00000000 00:00 0 
7fd6f9c9b000-7fd6f9ce4000 rw-p 00000000 00:00 0                          [stack:6582]
 
7fd6f9ce4000-7fd6f9ce5000 ---p 00000000 00:00 0 
7fd6f9ce5000-7fd6f9d2e000 rw-p 00000000 00:00 0                          [stack:6581]
 
7fd6f9d2e000-7fd6f9d2f000 ---p 00000000 00:00 0 
7fd6f9d2f000-7fd6f9d78000 rw-p 00000000 00:00 0                          [stack:6580]
 
7fd6f9d78000-7fd6f9d79000 ---p 00000000 00:00 0 
7fd6f9d79000-7fd6f9dc2000 rw-p 00000000 00:00 0                          [stack:6454]
 
7fd6f9dc2000-7fd6f9dc3000 ---p 00000000 00:00 0 
7fd6f9dc3000-7fd6f9e0c000 rw-p 00000000 00:00 0                          [stack:6453]
 
7fd6f9e0c000-7fd6f9e12000 rw-s 00000000 fd:02 28                         /var/lib/mysql/tc.log
 
7fd6f9e12000-7fd6f9e15000 rw-s 00000000 00:0a 20250                      /[aio] (deleted)
 
7fd6f9e15000-7fd6f9e18000 rw-s 00000000 00:0a 20249                      /[aio] (deleted)
 
7fd6f9e18000-7fd6f9e1b000 rw-s 00000000 00:0a 20248                      /[aio] (deleted)
 
7fd6f9e1b000-7fd6f9e1e000 rw-s 00000000 00:0a 20247                      /[aio] (deleted)
 
7fd6f9e1e000-7fd6f9e21000 rw-s 00000000 00:0a 20246                      /[aio] (deleted)
 
7fd6f9e21000-7fd6f9e24000 rw-s 00000000 00:0a 20245                      /[aio] (deleted)
 
7fd6f9e24000-7fd6f9e27000 rw-s 00000000 00:0a 20244                      /[aio] (deleted)
 
7fd6f9e27000-7fd6f9e37000 rw-p 00000000 00:00 0 
7fd6f9e37000-7fd6f9e5d000 r-xp 00000000 ca:01 197667                     /usr/lib64/libsystemd.so.0.6.0
 
7fd6f9e5d000-7fd6f9e5e000 r--p 00025000 ca:01 197667                     /usr/lib64/libsystemd.so.0.6.0
 
7fd6f9e5e000-7fd6f9e5f000 rw-p 00026000 ca:01 197667                     /usr/lib64/libsystemd.so.0.6.0
 
7fd6f9e5f000-7fd6f9e62000 rw-p 00000000 00:00 0 
7fd6f9e62000-7fd6f9e63000 rw-s 00000000 00:0a 20251                      /[aio] (deleted)
 
7fd6f9e63000-7fd6f9e66000 rw-s 00000000 00:0a 20243                      /[aio] (deleted)
 
7fd6f9e66000-7fd6f9e69000 rw-s 00000000 00:0a 20242                      /[aio] (deleted)
 
7fd6f9e69000-7fd6f9e6c000 rw-s 00000000 00:0a 20241                      /[aio] (deleted)
 
7fd6f9e6c000-7fd6f9e6d000 rw-s 00000000 00:0a 20240                      /[aio] (deleted)
 
7fd6f9e6d000-7fd6f9e6e000 rw-p 00000000 00:00 0 
7fd6f9e6e000-7fd6f9e6f000 r--p 0001f000 ca:01 1538                       /usr/lib64/ld-2.17.so
 
7fd6f9e6f000-7fd6f9e70000 rw-p 00020000 ca:01 1538                       /usr/lib64/ld-2.17.so
 
7fd6f9e70000-7fd6f9e71000 rw-p 00000000 00:00 0 
7fd6f9e71000-7fd6fadf2000 r-xp 00000000 ca:01 3253217                    /usr/sbin/mysqld
 
7fd6faff2000-7fd6fb0a3000 r--p 00f81000 ca:01 3253217                    /usr/sbin/mysqld
 
7fd6fb0a3000-7fd6fb155000 rw-p 01032000 ca:01 3253217                    /usr/sbin/mysqld
 
7fd6fb155000-7fd6fb9cc000 rw-p 00000000 00:00 0 
7ffe9a68d000-7ffe9a6ae000 rw-p 00000000 00:00 0                          [stack]
 
7ffe9a779000-7ffe9a77b000 r-xp 00000000 00:00 0                          [vdso]
 
ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0                  [vsyscall]
 
170430 06:17:07 mysqld_safe Number of processes running now: 0
 
170430 06:17:07 mysqld_safe WSREP: not restarting wsrep node automatically
 
170430 06:17:07 mysqld_safe mysqld from pid file /var/lib/mysql/host.pid ended

Comment by Kazuhiko Shiozaki [ 2017-05-03 ]

No change related to this issue seems included in MariaDB 10.1.23 that is just released.

The MySQL 5.6 patch indeed first appeared in MySQL 5.6.36, whose InnoDB we have not merged yet.

This problematic change already exists in 10.1 branch thus its fix is required in 10.1 branch as well, I believe.

Comment by Silver Asu [ 2017-05-04 ]

2.5 hours after upgrading to 10.1.23, MariaDB crashed.

2017-05-04  7:06:32 139924442187536 [Note] /opt/zone/sbin/mysqld: ready for connections.
 
Version: '10.1.23-MariaDB'  socket: '/tmp/mysql.sock'  port: 3306  Source distribution
 
terminate called after throwing an instance of 'std::out_of_range'
 
  what():  vector::_M_range_check
 
170504  9:32:29 [ERROR] mysqld got signal 6 ;
 
This could be because you hit a bug. It is also possible that this binary
 
or one of the libraries it was linked against is corrupt, improperly built,
 
or misconfigured. This error can also be caused by malfunctioning hardware.
 
 
 
To report this bug, see https://mariadb.com/kb/en/reporting-bugs
 
 
 
We will try our best to scrape up some info that will hopefully help
 
diagnose the problem, but since we have already crashed, 
something is definitely wrong and this may fail.
 
 
 
Server version: 10.1.23-MariaDB
 
key_buffer_size=4294967296
 
read_buffer_size=2097152
 
max_used_connections=37
 
max_threads=1502
 
thread_count=10
 
It is possible that mysqld could use up to 
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 19605613 K  bytes of memory
 
Hope that's ok; if not, decrease some variables in the equation.
 
 
 
Thread pointer: 0x0
 
Attempting backtrace. You can use the following information to find out
 
where mysqld died. If you see no messages after this, something went
 
terribly wrong...
 
stack_bottom = 0x0 thread_stack 0x48400
 
/opt/zone/sbin/mysqld(my_print_stacktrace+0x24)[0xbe6d44]
 
/opt/zone/sbin/mysqld(handle_fatal_signal+0x57a)[0x75753a]
 
/lib64/libpthread.so.0[0x7f42b2458ca0]
 
addr2line: /opt/zone/sbin/mysqld: Permission denied
 
/lib64/libc.so.6(gsignal+0x35)[0x7f42b06a6fc5]
 
/lib64/libc.so.6(abort+0x110)[0x7f42b06a8a70]
 
/usr/lib64/libstdc++.so.6(_ZN9__gnu_cxx27__verbose_terminate_handlerEv+0x114)[0x7f42b0f1fd94]
 
/usr/lib64/libstdc++.so.6[0x7f42b0f1de46]
 
/usr/lib64/libstdc++.so.6[0x7f42b0f1de73]
 
/usr/lib64/libstdc++.so.6[0x7f42b0f1df71]
 
/usr/lib64/libstdc++.so.6(_ZSt20__throw_out_of_rangePKc+0x67)[0x7f42b0ebaaa7]
 
/opt/zone/sbin/mysqld[0xa4c8aa]
 
/opt/zone/sbin/mysqld[0xa51938]
 
/opt/zone/sbin/mysqld[0xa52805]
 
/opt/zone/sbin/mysqld[0xa54620]
 
/lib64/libpthread.so.0[0x7f42b245083d]
 
/lib64/libc.so.6(clone+0x6d)[0x7f42b074bfcd]
 
The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
 
information that should help you find out what is causing the crash.
 
170504 09:32:30 mysqld_safe Number of processes running now: 0
 
170504 09:32:30 mysqld_safe mysqld restarted

Comment by Daisuke Morita [ 2017-05-04 ]

Hi,

I also met the same issue with our MariaDB (10.1.22) Galera cluster.

May  4 10:08:06 ****************: terminate called after throwing an instance of 'std::out_of_range'
 
May  4 10:08:06 ****************:   what():  vector::_M_range_check: __n (which is 4294967295) >= this->size() (which is 0)
 
May  4 10:08:06 ****************: 170504 10:08:06 [ERROR] mysqld got signal 6 ;
 
May  4 10:08:06 ****************: This could be because you hit a bug. It is also possible that this binary
 
May  4 10:08:06 ****************: or one of the libraries it was linked against is corrupt, improperly built,
 
May  4 10:08:06 ****************: or misconfigured. This error can also be caused by malfunctioning hardware.
 
May  4 10:08:06 ****************:
 
May  4 10:08:06 ****************: To report this bug, see https://mariadb.com/kb/en/reporting-bugs
 
May  4 10:08:06 ****************:
 
May  4 10:08:06 ****************: We will try our best to scrape up some info that will hopefully help
 
May  4 10:08:06 ****************: diagnose the problem, but since we have already crashed,
 
May  4 10:08:06 ****************: something is definitely wrong and this may fail.
 
May  4 10:08:06 ****************:
 
May  4 10:08:06 ****************: Server version: 10.1.22-MariaDB-1~xenial
 
May  4 10:08:06 ****************: key_buffer_size=16777216
 
May  4 10:08:06 ****************: read_buffer_size=131072
 
May  4 10:08:06 ****************: max_used_connections=301
 
May  4 10:08:06 ****************: max_threads=302
 
May  4 10:08:06 ****************: thread_count=299
 
May  4 10:08:06 ****************: It is possible that mysqld could use up to
 
May  4 10:08:06 ****************: key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 679718 K  bytes of memory
 
May  4 10:08:06 ****************: Hope that's ok; if not, decrease some variables in the equation.
 
May  4 10:08:06 ****************:
 
May  4 10:08:06 ****************: Thread pointer: 0x0
 
May  4 10:08:06 ****************: Attempting backtrace. You can use the following information to find out
 
May  4 10:08:06 ****************: where mysqld died. If you see no messages after this, something went
 
May  4 10:08:06 ****************: terribly wrong...
 
May  4 10:08:06 ****************: stack_bottom = 0x0 thread_stack 0x30000

I also believe this critical issue fix should be included in ver 10.1.x release series.

Comment by Kazuhiko Shiozaki [ 2017-05-05 ]

FYI, I tried to patch MariaDB 10.1.23 like MySQL repository's commit, but still the same crash happened.

terminate called after throwing an instance of 'std::out_of_range'
 
  what():  vector::_M_range_check: __n (which is 4294967295) >= this->size() (which is 0)
 
170504 11:57:29 [ERROR] mysqld got signal 6 ;

Comment by Patrick Gaus [ 2017-05-08 ]

Maybe this has to be addressed in XtraDB as well. We have patched it like the MySQL commit named.

https://github.com/MariaDB/server/blob/10.1/storage/xtradb/dict/dict0stats.cc#L1171

Comment by Marko Mäkelä [ 2017-05-09 ]

d-panja, as far as I understand, serg normally merges XtraDB straight from the Percona release, which means that the XtraDB release would be one minor release behind InnoDB.

Comment by Kazuhiko Shiozaki [ 2017-05-10 ]

Thanks Patrick. After patching XtraDB as you mentioned, mysqld is quite stable.

Comment by Daniil Muidinov [ 2017-05-24 ]

MariaDB 10.1.23 

terminate called after throwing an instance of 'std::out_of_range'
 
  what():  vector::_M_range_check
 
170524 16:03:37 [ERROR] mysqld got signal 6 ;
 
This could be because you hit a bug. It is also possible that this binary
 
or one of the libraries it was linked against is corrupt, improperly built,
 
or misconfigured. This error can also be caused by malfunctioning hardware.
 
 
 
To report this bug, see https://mariadb.com/kb/en/reporting-bugs
 
 
 
We will try our best to scrape up some info that will hopefully help
 
diagnose the problem, but since we have already crashed, 
something is definitely wrong and this may fail.
 
 
 
Server version: 10.1.23-MariaDB
 
key_buffer_size=32212254720
 
read_buffer_size=131072
 
max_used_connections=4
 
max_threads=153
 
thread_count=4
 
It is possible that mysqld could use up to 
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 31793329 K  bytes of memory
 
Hope that's ok; if not, decrease some variables in the equation.
 
 
 
Thread pointer: 0x0
 
Attempting backtrace. You can use the following information to find out
 
where mysqld died. If you see no messages after this, something went
 
terribly wrong...
 
stack_bottom = 0x0 thread_stack 0x48400
 
(my_addr_resolve failure: fork)
 
/usr/sbin/mysqld(my_print_stacktrace+0x2e) [0x7f3d2ed3b4ce]
 
/usr/sbin/mysqld(handle_fatal_signal+0x305) [0x7f3d2e85e345]
 
/lib64/libpthread.so.0(+0xf370) [0x7f3d2de78370]
 
/lib64/libc.so.6(gsignal+0x37) [0x7f3d2c1cd1d7]
 
/lib64/libc.so.6(abort+0x148) [0x7f3d2c1ce8c8]
 
/lib64/libstdc++.so.6(__gnu_cxx::__verbose_terminate_handler()+0x165) [0x7f3d2c8bb9d5]
 
/lib64/libstdc++.so.6(+0x5e946) [0x7f3d2c8b9946]
 
/lib64/libstdc++.so.6(+0x5e973) [0x7f3d2c8b9973]
 
/lib64/libstdc++.so.6(+0x5eb93) [0x7f3d2c8b9b93]
 
/lib64/libstdc++.so.6(std::__throw_out_of_range(char const*)+0x77) [0x7f3d2c90ea17]
 
/usr/sbin/mysqld(+0x8d97a0) [0x7f3d2eb807a0]
 
/usr/sbin/mysqld(+0x8de3a3) [0x7f3d2eb853a3]
 
/usr/sbin/mysqld(+0x8e0805) [0x7f3d2eb87805]
 
/usr/sbin/mysqld(+0x8e2573) [0x7f3d2eb89573]
 
/lib64/libpthread.so.0(+0x7dc5) [0x7f3d2de70dc5]
 
/lib64/libc.so.6(clone+0x6d) [0x7f3d2c28f73d]
 
The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
 
information that should help you find out what is causing the crash.

Comment by Silver Asu [ 2017-05-24 ]

This is unofficial patch for 10.1.23 that works for us

--- storage/xtradb/dict/dict0stats.cc.orig      2017-05-02 08:13:52.000000000 +0300
 
+++ storage/xtradb/dict/dict0stats.cc   2017-05-11 18:49:35.000000000 +0300
 
@@ -1168,10 +1168,10 @@
 
                leaf-level delete marks because delete marks on
 
                non-leaf level do not make sense. */
 
 
-               if (level == 0 && srv_stats_include_delete_marked? 0:
 
+               if (level == 0 && (srv_stats_include_delete_marked? 0:
 
                    rec_get_deleted_flag(
 
                            rec,
 
-                           page_is_comp(btr_pcur_get_page(&pcur)))) {
 
+                           page_is_comp(btr_pcur_get_page(&pcur))))) {
 
 
                        if (rec_is_last_on_page
 
                            && !prev_rec_is_copied

Comment by Adrian Tarau [ 2017-07-10 ]

Are there any updates on this issue? Our servers crash every day, we had to revert all lab/production systems to 10.1.18.

Thanks.

Comment by Sergei Golubchik [ 2017-07-10 ]

We've got InnoDB-5.6.36 in 10.0.31 and 10.1.24, so this issue should be fixed now.

Generated at Thu Feb 08 07:56:30 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.