|
we have 3 galera cluster nodes, 2 nodes where crashed.
node #1
2017-04-26 12:56:12 140166675031808 [Note] WSREP: Created page /var/lib/mysql/gcache.page.000000 of size 283818317 bytes
|
2017-04-26 12:57:46 140166675031808 [Note] WSREP: Created page /var/lib/mysql/gcache.page.000001 of size 146617936 bytes
|
2017-04-26 12:59:06 140166675031808 [Note] WSREP: Created page /var/lib/mysql/gcache.page.000002 of size 144435710 bytes
|
170426 13:07:47 [ERROR] mysqld got signal 11 ;
|
This could be because you hit a bug. It is also possible that this binary
|
or one of the libraries it was linked against is corrupt, improperly built,
|
or misconfigured. This error can also be caused by malfunctioning hardware.
|
|
To report this bug, see https://mariadb.com/kb/en/reporting-bugs
|
|
We will try our best to scrape up some info that will hopefully help
|
diagnose the problem, but since we have already crashed,
|
something is definitely wrong and this may fail.
|
|
Server version: 10.1.21-MariaDB
|
key_buffer_size=134217728
|
read_buffer_size=131072
|
max_used_connections=69
|
max_threads=10002
|
thread_count=133
|
It is possible that mysqld could use up to
|
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 22100835 K bytes of memory
|
Hope that's ok; if not, decrease some variables in the equation.
|
|
Thread pointer: 0x0
|
Attempting backtrace. You can use the following information to find out
|
where mysqld died. If you see no messages after this, something went
|
terribly wrong...
|
stack_bottom = 0x0 thread_stack 0x48400
|
/usr/sbin/mysqld(my_print_stacktrace+0x2b)[0x7f7b2af60a9b]
|
/usr/sbin/mysqld(handle_fatal_signal+0x4d5)[0x7f7b2aabc8f5]
|
/lib64/libpthread.so.0(+0x3edea0f7e0)[0x7f7b2a0bf7e0]
|
/usr/lib64/libjemalloc.so.1(free+0x63)[0x7f7b29866ca3]
|
/usr/lib64/galera/libgalera_smm.so(_ZN6gcache6GCache13discard_seqnoEl+0xdc)[0x7f7b25038e3c]
|
/usr/lib64/galera/libgalera_smm.so(_ZN6gcache6GCache13seqno_releaseEl+0x172)[0x7f7b25024792]
|
/usr/lib64/galera/libgalera_smm.so(_ZN6galera10ServiceThd8thd_funcEPv+0x100)[0x7f7b25122dd0]
|
/lib64/libpthread.so.0(+0x3edea07aa1)[0x7f7b2a0b7aa1]
|
/lib64/libc.so.6(clone+0x6d)[0x7f7b2859baad]
|
The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
|
information that should help you find out what is causing the crash.
|
170426 13:07:48 mysqld_safe Number of processes running now: 0
|
170426 13:07:48 mysqld_safe WSREP: not restarting wsrep node automatically
|
170426 13:07:48 mysqld_safe mysqld from pid file /var/lib/mysql/MVOIP-17.pid ended
|
170426 13:07:51 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql
|
170426 13:07:51 mysqld_safe WSREP: Running position recovery with --log_error='/var/lib/mysql/wsrep_recovery.DNQJ4D' --pid-file='/var/lib/mysql/MVOIP-17-recover.pid'
|
2017-04-26 13:07:51 140667503646752 [Note] /usr/sbin/mysqld (mysqld 10.1.21-MariaDB) starting as process 13048 ...
|
170426 13:08:35 mysqld_safe WSREP: Recovered position 1dbbee43-0d82-11e7-be6c-eae935054295:2576299
|
node #2
2017-04-26 12:56:12 139905764157184 [Note] WSREP: Created page /var/lib/mysql/gcache.page.000000 of size 283818317 bytes
|
2017-04-26 12:57:46 139905764157184 [Note] WSREP: Created page /var/lib/mysql/gcache.page.000001 of size 146617936 bytes
|
2017-04-26 12:59:06 139905764157184 [Note] WSREP: Created page /var/lib/mysql/gcache.page.000002 of size 144435710 bytes
|
170426 13:07:47 [ERROR] mysqld got signal 11 ;
|
This could be because you hit a bug. It is also possible that this binary
|
or one of the libraries it was linked against is corrupt, improperly built,
|
or misconfigured. This error can also be caused by malfunctioning hardware.
|
|
To report this bug, see https://mariadb.com/kb/en/reporting-bugs
|
|
We will try our best to scrape up some info that will hopefully help
|
diagnose the problem, but since we have already crashed,
|
something is definitely wrong and this may fail.
|
|
Server version: 10.1.22-MariaDB
|
key_buffer_size=134217728
|
read_buffer_size=131072
|
max_used_connections=50
|
max_threads=10002
|
thread_count=129
|
It is possible that mysqld could use up to
|
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 22100835 K bytes of memory
|
Hope that's ok; if not, decrease some variables in the equation.
|
|
Thread pointer: 0x0
|
Attempting backtrace. You can use the following information to find out
|
where mysqld died. If you see no messages after this, something went
|
terribly wrong...
|
stack_bottom = 0x0 thread_stack 0x48400
|
/usr/sbin/mysqld(my_print_stacktrace+0x2b)[0x7f3e6b70cfbb]
|
/usr/sbin/mysqld(handle_fatal_signal+0x4d5)[0x7f3e6b267be5]
|
/lib64/libpthread.so.0(+0x31a8c0f7e0)[0x7f3e6a8697e0]
|
/usr/lib64/libjemalloc.so.1(free+0x63)[0x7f3e6a010ca3]
|
/usr/lib64/galera/libgalera_smm.so(_ZN6gcache6GCache13discard_seqnoEl+0xdc)[0x7f3e65838e3c]
|
/usr/lib64/galera/libgalera_smm.so(_ZN6gcache6GCache13seqno_releaseEl+0x172)[0x7f3e65824792]
|
/usr/lib64/galera/libgalera_smm.so(_ZN6galera10ServiceThd8thd_funcEPv+0x100)[0x7f3e65922dd0]
|
/lib64/libpthread.so.0(+0x31a8c07aa1)[0x7f3e6a861aa1]
|
/lib64/libc.so.6(clone+0x6d)[0x7f3e68d45aad]
|
The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
|
information that should help you find out what is causing the crash.
|
170426 13:07:48 mysqld_safe Number of processes running now: 0
|
170426 13:07:48 mysqld_safe WSREP: not restarting wsrep node automatically
|
170426 13:07:48 mysqld_safe mysqld from pid file /var/lib/mysql/MVOIP-18.pid ended
|
170426 13:08:36 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql
|
170426 13:08:36 mysqld_safe WSREP: Running position recovery with --log_error='/var/lib/mysql/wsrep_recovery.6mptIG' --pid-file='/var/lib/mysql/MVOIP-18-recover.pid'
|
2017-04-26 13:08:36 140432795473952 [Note] /usr/sbin/mysqld (mysqld 10.1.22-MariaDB) starting as process 3761 ...
|
170426 13:09:29 mysqld_safe WSREP: Recovered position 1dbbee43-0d82-11e7-be6c-eae935054295:2576299
|
node #3 ( is not crushed )
2017-04-26 12:56:12 139627748910848 [Note] WSREP: Created page /var/lib/mysql/gcache.page.000000 of size 283818317 bytes
|
2017-04-26 12:57:46 139627748910848 [Note] WSREP: Created page /var/lib/mysql/gcache.page.000001 of size 146617936 bytes
|
2017-04-26 12:59:06 139627748910848 [Note] WSREP: Created page /var/lib/mysql/gcache.page.000002 of size 144435710 bytes
|
2017-04-26 13:07:47 139580051224320 [Note] WSREP: Deleted page /var/lib/mysql/gcache.page.000000
|
2017-04-26 13:07:48 139580051224320 [Note] WSREP: Deleted page /var/lib/mysql/gcache.page.000001
|
2017-04-26 13:07:48 139580051224320 [Note] WSREP: Deleted page /var/lib/mysql/gcache.page.000002
|
2017-04-26 13:07:48 139627759400704 [Note] WSREP: (25c09e1d, 'tcp://0.0.0.0:4567') turning message relay requesting on, nonlive peers: tcp://203.240.153.17:4567
|
tcp://203.240.153.18:4567
|
2017-04-26 13:07:49 139627759400704 [Note] WSREP: (25c09e1d, 'tcp://0.0.0.0:4567') reconnecting to faf30d43 (tcp://203.240.153.17:4567), attempt 0
|
2017-04-26 13:07:49 139627759400704 [Note] WSREP: (25c09e1d, 'tcp://0.0.0.0:4567') reconnecting to 9680f3fb (tcp://203.240.153.18:4567), attempt 0
|
2017-04-26 13:07:52 139627759400704 [Note] WSREP: evs::proto(25c09e1d, OPERATIONAL, view_id(REG,25c09e1d,124)) suspecting node: 9680f3fb
|
2017-04-26 13:07:52 139627759400704 [Note] WSREP: evs::proto(25c09e1d, OPERATIONAL, view_id(REG,25c09e1d,124)) suspected node without join message, declaring in
|
active
|
2017-04-26 13:07:52 139627759400704 [Note] WSREP: evs::proto(25c09e1d, OPERATIONAL, view_id(REG,25c09e1d,124)) suspecting node: faf30d43
|
2017-04-26 13:07:52 139627759400704 [Note] WSREP: evs::proto(25c09e1d, OPERATIONAL, view_id(REG,25c09e1d,124)) suspected node without join message, declaring in
|
active
|
2017-04-26 13:07:53 139627759400704 [Note] WSREP: view(view_id(NON_PRIM,25c09e1d,124) memb {
|
25c09e1d,0
|
} joined {
|
} left {
|
} partitioned {
|
9680f3fb,0
|
faf30d43,0
|
})
|
2017-04-26 13:07:53 139627748910848 [Note] WSREP: New COMPONENT: primary = no, bootstrap = no, my_idx = 0, memb_num = 1
|
2017-04-26 13:07:53 139627759400704 [Note] WSREP: view(view_id(NON_PRIM,25c09e1d,125) memb {
|
25c09e1d,0
|
} joined {
|
} left {
|
} partitioned {
|
9680f3fb,0
|
faf30d43,0
|
})
|
2017-04-26 13:07:53 139627748910848 [Note] WSREP: Flow-control interval: [16, 16]
|
2017-04-26 13:07:53 139627748910848 [Note] WSREP: Received NON-PRIMARY.
|
2017-04-26 13:07:53 139627748910848 [Note] WSREP: Shifting SYNCED -> OPEN (TO: 2576299)
|
2017-04-26 13:07:53 139627748910848 [Note] WSREP: New COMPONENT: primary = no, bootstrap = no, my_idx = 0, memb_num = 1
|
2017-04-26 13:07:53 139627748910848 [Note] WSREP: Flow-control interval: [16, 16]
|
2017-04-26 13:07:53 139627748910848 [Note] WSREP: Received NON-PRIMARY.
|
2017-04-26 13:07:53 139627727305472 [Note] WSREP: New cluster view: global state: 1dbbee43-0d82-11e7-be6c-eae935054295:2576299, view# -1: non-Primary, number of
|
nodes: 1, my index: 0, protocol version 3
|
2017-04-26 13:07:53 139627727305472 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
|
2017-04-26 13:07:53 139627727305472 [Note] WSREP: New cluster view: global state: 1dbbee43-0d82-11e7-be6c-eae935054295:2576299, view# -1: non-Primary, number of
|
nodes: 1, my index: 0, protocol version 3
|
2017-04-26 13:07:53 139627727305472 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
|
2017-04-26 13:08:33 139627759400704 [Note] WSREP: (25c09e1d, 'tcp://0.0.0.0:4567') reconnecting to faf30d43 (tcp://203.240.153.17:4567), attempt 30
|
2017-04-26 13:08:33 139627759400704 [Note] WSREP: (25c09e1d, 'tcp://0.0.0.0:4567') reconnecting to 9680f3fb (tcp://203.240.153.18:4567), attempt 30
|
2017-04-26 13:08:35 139627759400704 [Note] WSREP: (25c09e1d, 'tcp://0.0.0.0:4567') connection established to faf30d43 tcp://203.240.153.17:4567
|
2017-04-26 13:08:35 139627759400704 [Note] WSREP: declaring faf30d43 at tcp://203.240.153.17:4567 stable
|
|