Details
-
Bug
-
Status: Closed (View Workflow)
-
Major
-
Resolution: Fixed
-
None
-
None
-
None
-
jheiss@maria3:~$ cat /etc/debian_version
7.1
jheiss@maria3:~$ uname -a
Linux maria3 3.2.0-4-amd64 #1 SMP Debian 3.2.46-1+deb7u1 x86_64 GNU/Linux
Description
Three node maria+galera cluster. Add iptables rules on two of the nodes blocking all traffic. The third node then crashes with the following message. I would expect the node to realize that it has been orphaned and probably exit gracefully. Just reporting this since it seems to exit ungracefully and suggests reporting it as a bug.
130928 18:39:57 [Note] WSREP: (4e46a530-2887-11e3-a27f-17a796765c58, 'tcp://0.0.0.0:4567') turning message relay requesting on, nonlive peers: tcp://192.168.106.140:4567 tcp://192.168.106.141:4567
|
130928 18:40:46 [Note] WSREP: (4e46a530-2887-11e3-a27f-17a796765c58, 'tcp://0.0.0.0:4567') reconnecting to aa15359a-2545-11e3-a663-d6c347d67c0e (tcp://192.168.106.140:4567), attempt 0
|
130928 18:40:46 [Note] WSREP: (4e46a530-2887-11e3-a27f-17a796765c58, 'tcp://0.0.0.0:4567') reconnecting to 3d96e1c7-2546-11e3-995e-023b987c1a66 (tcp://192.168.106.141:4567), attempt 0
|
130928 18:40:46 [Warning] WSREP: last inactive check more than PT1.5S ago (PT49.1046S), skipping check
|
130928 18:40:46 [Note] WSREP: evs::proto(4e46a530-2887-11e3-a27f-17a796765c58, OPERATIONAL, view_id(REG,3d96e1c7-2546-11e3-995e-023b987c1a66,98)) detected inactive node: 3d96e1c7-2546-11e3-995e-023b987c1a66
|
130928 18:40:46 [Note] WSREP: evs::proto(4e46a530-2887-11e3-a27f-17a796765c58, OPERATIONAL, view_id(REG,3d96e1c7-2546-11e3-995e-023b987c1a66,98)) detected inactive node: aa15359a-2545-11e3-a663-d6c347d67c0e
|
130928 18:40:47 [Note] WSREP: view(view_id(NON_PRIM,3d96e1c7-2546-11e3-995e-023b987c1a66,98) memb {
|
4e46a530-2887-11e3-a27f-17a796765c58,
|
} joined {
|
} left {
|
} partitioned {
|
3d96e1c7-2546-11e3-995e-023b987c1a66,
|
aa15359a-2545-11e3-a663-d6c347d67c0e,
|
})
|
130928 18:40:47 [Note] WSREP: New COMPONENT: primary = no, bootstrap = no, my_idx = 0, memb_num = 1
|
130928 18:40:47 [Note] WSREP: Flow-control interval: [16, 16]
|
130928 18:40:47 [Note] WSREP: Received NON-PRIMARY.
|
130928 18:40:47 [Note] WSREP: Shifting SYNCED -> OPEN (TO: 1)
|
130928 18:40:47 [Note] WSREP: New cluster view: global state: 5c12c088-24d9-11e3-aaa7-ff8e1b069ebd:1, view# -1: non-Primary, number of nodes: 1, my index: 0, protocol version 2
|
130928 18:40:47 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
|
130928 18:40:47 [Note] WSREP: view(view_id(NON_PRIM,4e46a530-2887-11e3-a27f-17a796765c58,99) memb {
|
4e46a530-2887-11e3-a27f-17a796765c58,
|
} joined {
|
} left {
|
} partitioned {
|
3d96e1c7-2546-11e3-995e-023b987c1a66,
|
aa15359a-2545-11e3-a663-d6c347d67c0e,
|
})
|
130928 18:40:47 [Note] WSREP: New COMPONENT: primary = no, bootstrap = no, my_idx = 0, memb_num = 1
|
130928 18:40:47 [Note] WSREP: Flow-control interval: [16, 16]
|
130928 18:40:47 [Note] WSREP: Received NON-PRIMARY.
|
130928 18:40:47 [Note] WSREP: New cluster view: global state: 5c12c088-24d9-11e3-aaa7-ff8e1b069ebd:1, view# -1: non-Primary, number of nodes: 1, my index: 0, protocol version 2
|
130928 18:40:47 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
|
130928 18:40:49 [Note] WSREP: (4e46a530-2887-11e3-a27f-17a796765c58, 'tcp://0.0.0.0:4567') address 'tcp://192.168.106.142:4567' pointing to uuid 4e46a530-2887-11e3-a27f-17a796765c58 is blacklisted, skipping
|
130928 18:40:49 [Note] WSREP: (4e46a530-2887-11e3-a27f-17a796765c58, 'tcp://0.0.0.0:4567') address 'tcp://192.168.106.142:4567' pointing to uuid 4e46a530-2887-11e3-a27f-17a796765c58 is blacklisted, skipping
|
130928 18:40:49 [Note] WSREP: declaring aa15359a-2545-11e3-a663-d6c347d67c0e stable
|
130928 18:40:49 [Note] WSREP: view(view_id(NON_PRIM,4e46a530-2887-11e3-a27f-17a796765c58,100) memb {
|
4e46a530-2887-11e3-a27f-17a796765c58,
|
aa15359a-2545-11e3-a663-d6c347d67c0e,
|
} joined {
|
} left {
|
} partitioned {
|
3d96e1c7-2546-11e3-995e-023b987c1a66,
|
})
|
130928 18:40:49 [Note] WSREP: New COMPONENT: primary = no, bootstrap = no, my_idx = 0, memb_num = 2
|
130928 18:40:49 [Note] WSREP: Flow-control interval: [23, 23]
|
130928 18:40:49 [Note] WSREP: Received NON-PRIMARY.
|
130928 18:40:49 [Note] WSREP: New cluster view: global state: 5c12c088-24d9-11e3-aaa7-ff8e1b069ebd:1, view# -1: non-Primary, number of nodes: 2, my index: 0, protocol version 2
|
130928 18:40:49 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
|
130928 18:40:53 [Note] WSREP: (4e46a530-2887-11e3-a27f-17a796765c58, 'tcp://0.0.0.0:4567') reconnecting to aa15359a-2545-11e3-a663-d6c347d67c0e (tcp://192.168.106.140:4567), attempt 0
|
130928 18:40:54 [Note] WSREP: evs::proto(4e46a530-2887-11e3-a27f-17a796765c58, OPERATIONAL, view_id(REG,4e46a530-2887-11e3-a27f-17a796765c58,100)) suspecting node: aa15359a-2545-11e3-a663-d6c347d67c0e
|
130928 18:40:55 [Note] WSREP: evs::proto(4e46a530-2887-11e3-a27f-17a796765c58, GATHER, view_id(REG,4e46a530-2887-11e3-a27f-17a796765c58,100)) suspecting node: aa15359a-2545-11e3-a663-d6c347d67c0e
|
130928 18:40:55 [Note] WSREP: evs::proto(4e46a530-2887-11e3-a27f-17a796765c58, GATHER, view_id(REG,4e46a530-2887-11e3-a27f-17a796765c58,100)) suspecting node: aa15359a-2545-11e3-a663-d6c347d67c0e
|
130928 18:40:56 [Note] WSREP: evs::proto(4e46a530-2887-11e3-a27f-17a796765c58, GATHER, view_id(REG,4e46a530-2887-11e3-a27f-17a796765c58,100)) suspecting node: aa15359a-2545-11e3-a663-d6c347d67c0e
|
130928 18:40:56 [Note] WSREP: evs::proto(4e46a530-2887-11e3-a27f-17a796765c58, GATHER, view_id(REG,4e46a530-2887-11e3-a27f-17a796765c58,100)) suspecting node: aa15359a-2545-11e3-a663-d6c347d67c0e
|
130928 18:40:57 [Note] WSREP: (4e46a530-2887-11e3-a27f-17a796765c58, 'tcp://0.0.0.0:4567') address 'tcp://192.168.106.142:4567' pointing to uuid 4e46a530-2887-11e3-a27f-17a796765c58 is blacklisted, skipping
|
130928 18:40:57 [Note] WSREP: (4e46a530-2887-11e3-a27f-17a796765c58, 'tcp://0.0.0.0:4567') address 'tcp://192.168.106.142:4567' pointing to uuid 4e46a530-2887-11e3-a27f-17a796765c58 is blacklisted, skipping
|
130928 18:40:57 [Note] WSREP: evs::proto(4e46a530-2887-11e3-a27f-17a796765c58, GATHER, view_id(REG,4e46a530-2887-11e3-a27f-17a796765c58,100)) suspecting node: aa15359a-2545-11e3-a663-d6c347d67c0e
|
130928 18:40:57 [Note] WSREP: (4e46a530-2887-11e3-a27f-17a796765c58, 'tcp://0.0.0.0:4567') address 'tcp://192.168.106.142:4567' pointing to uuid 4e46a530-2887-11e3-a27f-17a796765c58 is blacklisted, skipping
|
130928 18:40:57 [Note] WSREP: (4e46a530-2887-11e3-a27f-17a796765c58, 'tcp://0.0.0.0:4567') address 'tcp://192.168.106.142:4567' pointing to uuid 4e46a530-2887-11e3-a27f-17a796765c58 is blacklisted, skipping
|
130928 18:40:57 [Note] WSREP: evs::proto(4e46a530-2887-11e3-a27f-17a796765c58, GATHER, view_id(REG,4e46a530-2887-11e3-a27f-17a796765c58,100)) suspecting node: aa15359a-2545-11e3-a663-d6c347d67c0e
|
130928 18:40:58 [Note] WSREP: (4e46a530-2887-11e3-a27f-17a796765c58, 'tcp://0.0.0.0:4567') address 'tcp://192.168.106.142:4567' pointing to uuid 4e46a530-2887-11e3-a27f-17a796765c58 is blacklisted, skipping
|
130928 18:40:58 [Note] WSREP: (4e46a530-2887-11e3-a27f-17a796765c58, 'tcp://0.0.0.0:4567') address 'tcp://192.168.106.142:4567' pointing to uuid 4e46a530-2887-11e3-a27f-17a796765c58 is blacklisted, skipping
|
130928 18:40:58 [Note] WSREP: (4e46a530-2887-11e3-a27f-17a796765c58, 'tcp://0.0.0.0:4567') address 'tcp://192.168.106.142:4567' pointing to uuid 4e46a530-2887-11e3-a27f-17a796765c58 is blacklisted, skipping
|
130928 18:40:58 [Note] WSREP: (4e46a530-2887-11e3-a27f-17a796765c58, 'tcp://0.0.0.0:4567') turning message relay requesting off
|
130928 18:41:02 [Note] WSREP: (4e46a530-2887-11e3-a27f-17a796765c58, 'tcp://0.0.0.0:4567') address 'tcp://192.168.106.142:4567' pointing to uuid 4e46a530-2887-11e3-a27f-17a796765c58 is blacklisted, skipping
|
130928 18:41:02 [Note] WSREP: (4e46a530-2887-11e3-a27f-17a796765c58, 'tcp://0.0.0.0:4567') turning message relay requesting on, nonlive peers: tcp://192.168.106.140:4567 tcp://192.168.106.141:4567
|
130928 18:41:03 [Note] WSREP: (4e46a530-2887-11e3-a27f-17a796765c58, 'tcp://0.0.0.0:4567') reconnecting to aa15359a-2545-11e3-a663-d6c347d67c0e (tcp://192.168.106.140:4567), attempt 0
|
130928 18:41:03 [Note] WSREP: (4e46a530-2887-11e3-a27f-17a796765c58, 'tcp://0.0.0.0:4567') reconnecting to 3d96e1c7-2546-11e3-995e-023b987c1a66 (tcp://192.168.106.141:4567), attempt 0
|
130928 18:41:04 [Note] WSREP: (4e46a530-2887-11e3-a27f-17a796765c58, 'tcp://0.0.0.0:4567') address 'tcp://192.168.106.142:4567' pointing to uuid 4e46a530-2887-11e3-a27f-17a796765c58 is blacklisted, skipping
|
130928 18:41:04 [Note] WSREP: (4e46a530-2887-11e3-a27f-17a796765c58, 'tcp://0.0.0.0:4567') cleaning up duplicate 0x7fa7d067c7c0 after established 0x7fa7d067cc30
|
130928 18:41:04 [Note] WSREP: (4e46a530-2887-11e3-a27f-17a796765c58, 'tcp://0.0.0.0:4567') address 'tcp://192.168.106.142:4567' pointing to uuid 4e46a530-2887-11e3-a27f-17a796765c58 is blacklisted, skipping
|
130928 18:41:04 [Note] WSREP: evs::proto(4e46a530-2887-11e3-a27f-17a796765c58, INSTALL, view_id(REG,4e46a530-2887-11e3-a27f-17a796765c58,100)) suspecting node: 3d96e1c7-2546-11e3-995e-023b987c1a66
|
130928 18:41:04 [Note] WSREP: evs::proto(4e46a530-2887-11e3-a27f-17a796765c58, INSTALL, view_id(REG,4e46a530-2887-11e3-a27f-17a796765c58,100)) suspecting node: aa15359a-2545-11e3-a663-d6c347d67c0e
|
130928 18:41:05 [Note] WSREP: view(view_id(NON_PRIM,4e46a530-2887-11e3-a27f-17a796765c58,100) memb {
|
4e46a530-2887-11e3-a27f-17a796765c58,
|
} joined {
|
} left {
|
} partitioned {
|
3d96e1c7-2546-11e3-995e-023b987c1a66,
|
aa15359a-2545-11e3-a663-d6c347d67c0e,
|
})
|
130928 18:41:05 [Note] WSREP: New COMPONENT: primary = no, bootstrap = no, my_idx = 0, memb_num = 1
|
130928 18:41:05 [Note] WSREP: Flow-control interval: [16, 16]
|
130928 18:41:05 [Note] WSREP: Received NON-PRIMARY.
|
130928 18:41:05 [Note] WSREP: New cluster view: global state: 5c12c088-24d9-11e3-aaa7-ff8e1b069ebd:1, view# -1: non-Primary, number of nodes: 1, my index: 0, protocol version 2
|
130928 18:41:05 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
|
130928 18:41:05 [Note] WSREP: view(view_id(NON_PRIM,3d96e1c7-2546-11e3-995e-023b987c1a66,101) memb {
|
4e46a530-2887-11e3-a27f-17a796765c58,
|
} joined {
|
} left {
|
} partitioned {
|
3d96e1c7-2546-11e3-995e-023b987c1a66,
|
aa15359a-2545-11e3-a663-d6c347d67c0e,
|
})
|
130928 18:41:05 [Note] WSREP: New COMPONENT: primary = no, bootstrap = no, my_idx = 0, memb_num = 1
|
130928 18:41:05 [Note] WSREP: Flow-control interval: [16, 16]
|
130928 18:41:05 [Note] WSREP: Received NON-PRIMARY.
|
130928 18:41:05 [Note] WSREP: New cluster view: global state: 5c12c088-24d9-11e3-aaa7-ff8e1b069ebd:1, view# -1: non-Primary, number of nodes: 1, my index: 0, protocol version 2
|
130928 18:41:05 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
|
130928 18:41:05 [Note] WSREP: (4e46a530-2887-11e3-a27f-17a796765c58, 'tcp://0.0.0.0:4567') address 'tcp://192.168.106.142:4567' pointing to uuid 4e46a530-2887-11e3-a27f-17a796765c58 is blacklisted, skipping
|
130928 18:41:05 [Note] WSREP: (4e46a530-2887-11e3-a27f-17a796765c58, 'tcp://0.0.0.0:4567') address 'tcp://192.168.106.142:4567' pointing to uuid 4e46a530-2887-11e3-a27f-17a796765c58 is blacklisted, skipping
|
130928 18:41:05 [Note] WSREP: (4e46a530-2887-11e3-a27f-17a796765c58, 'tcp://0.0.0.0:4567') address 'tcp://192.168.106.142:4567' pointing to uuid 4e46a530-2887-11e3-a27f-17a796765c58 is blacklisted, skipping
|
130928 18:41:05 [Note] WSREP: (4e46a530-2887-11e3-a27f-17a796765c58, 'tcp://0.0.0.0:4567') turning message relay requesting off
|
terminate called after throwing an instance of 'std::out_of_range'
|
what(): vector::_M_range_check
|
130928 18:41:06 [ERROR] mysqld got signal 6 ;
|
This could be because you hit a bug. It is also possible that this binary
|
or one of the libraries it was linked against is corrupt, improperly built,
|
or misconfigured. This error can also be caused by malfunctioning hardware.
|
|
|
To report this bug, see http://kb.askmonty.org/en/reporting-bugs
|
|
|
We will try our best to scrape up some info that will hopefully help
|
diagnose the problem, but since we have already crashed,
|
something is definitely wrong and this may fail.
|
|
|
Server version: 5.5.32-MariaDB-1~wheezy-log
|
key_buffer_size=134217728
|
read_buffer_size=2097152
|
max_used_connections=0
|
max_threads=102
|
thread_count=2
|
It is possible that mysqld could use up to
|
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 759606 K bytes of memory
|
Hope that's ok; if not, decrease some variables in the equation.
|
|
|
Thread pointer: 0x0x0
|
Attempting backtrace. You can use the following information to find out
|
where mysqld died. If you see no messages after this, something went
|
terribly wrong...
|
stack_bottom = 0x0 thread_stack 0x48000
|
mysqld(my_print_stacktrace+0x2b)[0x7fa7ce163e8b]
|
mysqld(handle_fatal_signal+0x422)[0x7fa7cdd96202]
|
/lib/x86_64-linux-gnu/libpthread.so.0(+0xf030)[0x7fa7cd47a030]
|
/lib/x86_64-linux-gnu/libc.so.6(gsignal+0x35)[0x7fa7cbced475]
|
/lib/x86_64-linux-gnu/libc.so.6(abort+0x180)[0x7fa7cbcf06f0]
|
/usr/lib/x86_64-linux-gnu/libstdc++.so.6(_ZN9__gnu_cxx27__verbose_terminate_handlerEv+0x11d)[0x7fa7cc32c89d]
|
/usr/lib/x86_64-linux-gnu/libstdc++.so.6(+0x63996)[0x7fa7cc32a996]
|
/usr/lib/x86_64-linux-gnu/libstdc++.so.6(+0x639c3)[0x7fa7cc32a9c3]
|
/usr/lib/x86_64-linux-gnu/libstdc++.so.6(+0x63bee)[0x7fa7cc32abee]
|
/usr/lib/x86_64-linux-gnu/libstdc++.so.6(_ZSt20__throw_out_of_rangePKc+0x5d)[0x7fa7cc37c6fd]
|
/usr/lib/galera/libgalera_smm.so(+0xe5816)[0x7fa7ca72b816]
|
/usr/lib/galera/libgalera_smm.so(_ZN5gcomm3evs5Proto10handle_gapERKNS0_10GapMessageESt17_Rb_tree_iteratorISt4pairIKNS_4UUIDENS0_4NodeEEE+0x7d4)[0x7fa7ca73cb64]
|
/usr/lib/galera/libgalera_smm.so(_ZN5gcomm3evs5Proto10handle_msgERKNS0_7MessageERKNS_8DatagramE+0x39a)[0x7fa7ca74448a]
|
/usr/lib/galera/libgalera_smm.so(_ZN5gcomm3evs5Proto9handle_upEPKvRKNS_8DatagramERKNS_11ProtoUpMetaE+0x279)[0x7fa7ca744e19]
|
/usr/lib/galera/libgalera_smm.so(_ZN5gcomm8Protolay7send_upERKNS_8DatagramERKNS_11ProtoUpMetaE+0x36)[0x7fa7ca7461a6]
|
/usr/lib/galera/libgalera_smm.so(_ZN5gcomm6GMCast9handle_upEPKvRKNS_8DatagramERKNS_11ProtoUpMetaE+0x200)[0x7fa7ca759090]
|
/usr/lib/galera/libgalera_smm.so(_ZN5gcomm10Protostack8dispatchEPKvRKNS_8DatagramERKNS_11ProtoUpMetaE+0x58)[0x7fa7ca77e408]
|
/usr/lib/galera/libgalera_smm.so(_ZN5gcomm12AsioProtonet8dispatchERKPKvRKNS_8DatagramERKNS_11ProtoUpMetaE+0x4b)[0x7fa7ca7a5dbb]
|
/usr/lib/galera/libgalera_smm.so(_ZN5gcomm13AsioTcpSocket12read_handlerERKN4asio10error_codeEm+0x77e)[0x7fa7ca78957e]
|
/usr/lib/galera/libgalera_smm.so(_ZN4asio6detail7read_opINS_19basic_stream_socketINS_2ip3tcpENS_21stream_socket_serviceIS4_EEEEN5boost5arrayINS_14mutable_bufferELm1EEENS8_3_bi6bind_tImNS8_4_mfi3mf2ImN5gcomm13AsioTcpSocketERKNS_10error_codeEmEENSC_5list3INSC_5valueINS8_10shared_ptrISH_EEEEPFNS8_3argILi1EEEvEPFNSR_ILi2EEEvEEEEENSD_IvNSF_IvSH_SK_mEESY_EEEclESK_mi+0x93)[0x7fa7ca797283]
|
/usr/lib/galera/libgalera_smm.so(_ZN4asio6detail23reactive_socket_recv_opINS0_17consuming_buffersINS_14mutable_bufferEN5boost5arrayIS3_Lm1EEEEENS0_7read_opINS_19basic_stream_socketINS_2ip3tcpENS_21stream_socket_serviceISB_EEEES6_NS4_3_bi6bind_tImNS4_4_mfi3mf2ImN5gcomm13AsioTcpSocketERKNS_10error_codeEmEENSF_5list3INSF_5valueINS4_10shared_ptrISK_EEEEPFNS4_3argILi1EEEvEPFNSU_ILi2EEEvEEEEENSG_IvNSI_IvSK_SN_mEES11_EEEEE11do_completeEPNS0_15task_io_serviceEPNS0_25task_io_service_operationESL_m+0xdd)[0x7fa7ca7975bd]
|
/usr/lib/galera/libgalera_smm.so(_ZN4asio6detail15task_io_service3runERNS_10error_codeE+0x3d7)[0x7fa7ca7a93a7]
|
/usr/lib/galera/libgalera_smm.so(_ZN5gcomm12AsioProtonet10event_loopERKN2gu8datetime6PeriodE+0x1a2)[0x7fa7ca7a71e2]
|
/usr/lib/galera/libgalera_smm.so(_ZN9GCommConn3runEv+0x68)[0x7fa7ca7bdd38]
|
/usr/lib/galera/libgalera_smm.so(_ZN9GCommConn6run_fnEPv+0x9)[0x7fa7ca7c1219]
|
/lib/x86_64-linux-gnu/libpthread.so.0(+0x6b50)[0x7fa7cd471b50]
|
/lib/x86_64-linux-gnu/libc.so.6(clone+0x6d)[0x7fa7cbd95a7d]
|
The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
|
information that should help you find out what is causing the crash.
|