Details
Description
3 Node Setup, all 3 nodes crashed. Bootstrap required.
Node 1 crash output
Mar 5 17:48:58 mysqld[116636]: terminate called after throwing an instance of 'boost::exception_detail::clone_impl<boost::exception_detail::error_info_injector<std::system_error> >' |
Mar 5 17:48:58 mysqld[116636]: what(): remote_endpoint: Transport endpoint is not connected |
Mar 5 17:48:58 mysqld[116636]: 210305 17:48:58 [ERROR] mysqld got signal 6 ; |
Mar 5 17:48:58 mysqld[116636]: This could be because you hit a bug. It is also possible that this binary |
Mar 5 17:48:58 mysqld[116636]: or one of the libraries it was linked against is corrupt, improperly built, |
Mar 5 17:48:58 mysqld[116636]: or misconfigured. This error can also be caused by malfunctioning hardware. |
Mar 5 17:48:58 mysqld[116636]: To report this bug, see https://mariadb.com/kb/en/reporting-bugs |
Mar 5 17:48:58 mysqld[116636]: We will try our best to scrape up some info that will hopefully help |
Mar 5 17:48:58 mysqld[116636]: diagnose the problem, but since we have already crashed, |
Mar 5 17:48:58 mysqld[116636]: something is definitely wrong and this may fail. |
Mar 5 17:48:58 mysqld[116636]: Server version: 10.4.18-MariaDB-log |
Mar 5 17:48:58 mysqld[116636]: key_buffer_size=134217728 |
Mar 5 17:48:58 mysqld[116636]: read_buffer_size=131072 |
Mar 5 17:48:58 mysqld[116636]: max_used_connections=76 |
Mar 5 17:48:58 mysqld[116636]: max_threads=153 |
Mar 5 17:48:58 mysqld[116636]: thread_count=75 |
Mar 5 17:48:58 mysqld[116636]: It is possible that mysqld could use up to |
Mar 5 17:48:58 mysqld[116636]: key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 781104 K bytes of memory |
Mar 5 17:48:58 mysqld[116636]: Hope that's ok; if not, decrease some variables in the equation. |
Mar 5 17:48:58 mysqld[116636]: Thread pointer: 0x0 |
Mar 5 17:48:58 mysqld[116636]: Attempting backtrace. You can use the following information to find out |
Mar 5 17:48:58 mysqld[116636]: where mysqld died. If you see no messages after this, something went |
Mar 5 17:48:58 mysqld[116636]: terribly wrong... |
Mar 5 17:48:59 mysqld[116636]: stack_bottom = 0x0 thread_stack 0x49000 |
Mar 5 17:48:59 mysqld[116636]: /usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x55a14151e47e] |
Mar 5 17:48:59 mysqld[116636]: /usr/sbin/mysqld(handle_fatal_signal+0x30f)[0x55a140face8f] |
Mar 5 17:48:59 mysqld[116636]: /lib64/libpthread.so.0(+0xf630)[0x7f5879976630] |
Mar 5 17:48:59 mysqld[116636]: /lib64/libc.so.6(gsignal+0x37)[0x7f5877c463d7] |
Mar 5 17:48:59 mysqld[116636]: /lib64/libc.so.6(abort+0x148)[0x7f5877c47ac8] |
Mar 5 17:48:59 mysqld[116636]: /lib64/libstdc++.so.6(_ZN9__gnu_cxx27__verbose_terminate_handlerEv+0x165)[0x7f5878340a95] |
Mar 5 17:48:59 mysqld[116636]: /lib64/libstdc++.so.6(+0x5ea06)[0x7f587833ea06] |
Mar 5 17:48:59 mysqld[116636]: /lib64/libstdc++.so.6(+0x5ea33)[0x7f587833ea33] |
Mar 5 17:48:59 mysqld[116636]: /lib64/libstdc++.so.6(+0x5ec53)[0x7f587833ec53] |
Mar 5 17:48:59 mysqld[116636]: /usr/lib64/galera-4/libgalera_smm.so(+0x1997a4)[0x7f586ddc07a4] |
Mar 5 17:48:59 mysqld[116636]: /usr/lib64/galera-4/libgalera_smm.so(+0x19984c)[0x7f586ddc084c] |
Mar 5 17:48:59 mysqld[116636]: /usr/lib64/galera-4/libgalera_smm.so(+0x1a57bf)[0x7f586ddcc7bf] |
Mar 5 17:48:59 mysqld[116636]: /usr/lib64/galera-4/libgalera_smm.so(+0x1a78b8)[0x7f586ddce8b8] |
Mar 5 17:48:59 mysqld[116636]: /usr/lib64/galera-4/libgalera_smm.so(+0x1aaf13)[0x7f586ddd1f13] |
Mar 5 17:48:59 mysqld[116636]: /usr/lib64/galera-4/libgalera_smm.so(+0x19897b)[0x7f586ddbf97b] |
Mar 5 17:48:59 mysqld[116636]: /usr/lib64/galera-4/libgalera_smm.so(+0x194b91)[0x7f586ddbbb91] |
Mar 5 17:48:59 mysqld[116636]: /usr/lib64/galera-4/libgalera_smm.so(+0xc08c9)[0x7f586dce78c9] |
Mar 5 17:48:59 mysqld[116636]: /usr/lib64/galera-4/libgalera_smm.so(+0xa6bcf)[0x7f586dccdbcf] |
Mar 5 17:48:59 mysqld[116636]: /usr/lib64/galera-4/libgalera_smm.so(+0xad859)[0x7f586dcd4859] |
Mar 5 17:48:59 mysqld[116636]: /lib64/libpthread.so.0(+0x7ea5)[0x7f587996eea5] |
Mar 5 17:48:59 mysqld[116636]: /lib64/libc.so.6(clone+0x6d)[0x7f5877d0e9fd] |
|
Attachments
Issue Links
- is duplicated by
-
MDEV-30336 Galera: Handshake failed: peer did not return a certificate
-
- Closed
-
- relates to
-
-
- Closed
-
-
-
- Closed
-
- links to
Activity
| Field | Original Value | New Value |
|---|---|---|
| Fix Version/s | 10.4 [ 22408 ] | |
| Assignee | Jan Lindström [ jplindst ] |
I had the same issue on a MariaDB Galera node and also the Galera Arbitrator.
Version: MariaDB 10.5.12 | Galera 26.4.9
To reproduce this issue, I did a vulnerability scan using GVM (Greenbone Vulnerability Management, ex OpenVAS) and the brute force category scan caused a DOS to the MariaDB service on port 4567.
The replication on port 4567 is done by certificate authentication.
Log:
terminate called after throwing an instance of 'boost::exception_detail::clone_impl<boost::exception_detail::error_info_injector<std::system_error> >'
|
what(): remote_endpoint: Transport endpoint is not connected
|
210916 8:48:03 [ERROR] mysqld got signal 6 ;
|
This could be because you hit a bug. It is also possible that this binary
|
or one of the libraries it was linked against is corrupt, improperly built,
|
or misconfigured. This error can also be caused by malfunctioning hardware.
|
|
|
To report this bug, see https://mariadb.com/kb/en/reporting-bugs
|
|
|
We will try our best to scrape up some info that will hopefully help
|
diagnose the problem, but since we have already crashed,
|
something is definitely wrong and this may fail.
|
|
|
Server version: 10.5.12-MariaDB-1:10.5.12+maria~buster
|
key_buffer_size=134217728
|
read_buffer_size=131072
|
max_used_connections=40
|
max_threads=202
|
thread_count=142
|
It is possible that mysqld could use up to
|
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 575743 K bytes of memory
|
Hope that's ok; if not, decrease some variables in the equation.
|
|
|
Thread pointer: 0x0
|
Attempting backtrace. You can use the following information to find out
|
where mysqld died. If you see no messages after this, something went
|
terribly wrong...
|
stack_bottom = 0x0 thread_stack 0x49000
|
??:0(my_print_stacktrace)[0x56199a545e4e]
|
??:0(handle_fatal_signal)[0x561999fd04f5]
|
??:0(__restore_rt)[0x7f0528fdd730]
|
??:0(gsignal)[0x7f0528b337bb]
|
??:0(abort)[0x7f0528b1e535]
|
/lib/x86_64-linux-gnu/libstdc++.so.6(+0x8c983)[0x7f0528ece983]
|
??:0(std::rethrow_exception(std::__exception_ptr::exception_ptr))[0x7f0528ed48c6]
|
??:0(std::terminate())[0x7f0528ed4901]
|
??:0(__cxa_throw)[0x7f0528ed4b34]
|
boost/throw_exception.hpp:69(void boost::throw_exception<std::system_error>(std::system_error const&))[0x7f0523174ef6]
|
impl/throw_error.ipp:49(asio::detail::do_throw_error(std::error_code const&, char const*) [clone .isra.265])[0x7f0523181224]
|
detail/throw_error.hpp:41(gu::AsioStreamReact::assign_addresses())[0x7f052318434b]
|
bits/shared_ptr_base.h:1019(gu::AsioAcceptorReact::accept_handler(std::shared_ptr<gu::AsioStreamReact> const&, std::shared_ptr<gu::AsioAcceptorHandler> const&, std::error_code const&))[0x7f0523184697]
|
detail/gcc_x86_fenced_block.hpp:80(asio::detail::reactive_socket_accept_op<asio::basic_socket<asio::ip::tcp, asio::stream_socket_service<asio::ip::tcp> >, asio::ip::tcp, boost::_bi::bind_t<void, boost::_mfi::mf3<void, gu::AsioAcceptorReact, std::shared_ptr<gu::AsioStreamReact> const&, std::shared_ptr<gu::AsioAcceptorHandler> const&, std::error_code const&>, boost::_bi::list4<boost::_bi::value<std::shared_ptr<gu::AsioAcceptorReact> >, boost::_bi::value<std::shared_ptr<gu::AsioStreamReact> >, boost::_bi::valimpl/epoll_reactor.ipp:653(asio::detail::epoll_reactor::descriptor_state::do_complete(asio::detail::task_io_service*, asio::detail::task_io_service_operation*, std::error_code const&, unsigned long))[0x7f052317c795]
|
impl/task_io_service.ipp:62(gu::AsioIoService::run())[0x7f052316daf8]
|
src/asio_protonet.cpp:104(gcomm::AsioProtonet::event_loop(gu::datetime::Period const&))[0x7f05230a1ab8]
|
src/gu_mutex.hpp:50(GCommConn::run())[0x7f05230873b4]
|
src/gcs_gcomm.cpp:305(run_fn)[0x7f0523087a49]
|
??:0(start_thread)[0x7f0528fd2fa3]
|
??:0(clone)[0x7f0528bf54cf]
|
The manual page at https://mariadb.com/kb/en/how-to-produce-a-full-stack-trace-for-mysqld/ contains
|
information that should help you find out what is causing the crash.
|
Writing a core file...
|
Working directory at /var/lib/mysql
|
Resource Limits:
|
Fatal signal 11 while backtracing
|
2021-09-16 8:48:16 0 [Note] WSREP: Loading provider /usr/lib/galera/libgalera_smm.so initial position: db2fdb34-801c-11eb-9f55-623bc6603e05:8205359
|
2021-09-16 8:48:16 0 [Note] WSREP: wsrep_load(): loading provider library '/usr/lib/galera/libgalera_smm.so'
|
2021-09-16 8:48:16 0 [Note] WSREP: wsrep_load(): Galera 26.4.9(r819f29cb) by Codership Oy <info@codership.com> loaded successfully.
|
2021-09-16 8:48:16 0 [Note] WSREP: CRC-32C: using 64-bit x86 acceleration.
|
2021-09-16 8:48:16 0 [Note] WSREP: SSL cipher list set to 'ECDHE-ECDSA-AES256-GCM-SHA384'
|
2021-09-16 8:48:16 0 [Note] WSREP: Found saved state: db2fdb34-801c-11eb-9f55-623bc6603e05:-1, safe_to_bootstrap: 0
|
2021-09-16 8:48:16 0 [Note] WSREP: GCache DEBUG: opened preamble:
|
For the persons that had this issue, did your port 4567 was exposed to the Internet or a potential scan could be done on it?
| Attachment | log-10.4.2 [ 59783 ] | |
| Attachment | log-10.5.11 [ 59784 ] |
| Attachment | gdb.txt [ 59785 ] |
| Comment |
[ I've also seen had this crash happen numerous times on 2 separate 3-node clusters.
The 2 clusters are running different versions, one MariaDB 10.4.20 cluster & another 10.5.11 cluster. This has taken one of the clusters down - requiring a manual bootstrap (this happened when 2 nodes in the cluster both suffered from this crash within 20 seconds of each other). Otherwise, the clusters are able to recover when just one node suffers this crash. Included a backtrace for the cluster running 10.5.11 [^log-10.5.11] [^gdb.txt] [^log-10.4.2] ] |
Hello @Nicolas
I have inspected our scanning logs and have found that we did also run a vulnerabity scan at the time of crash. We use a cloud provider which has an OpenVAS backend scanning engine, it would have been an active scan, so 'crafted packets' of some sort. The scan we ran would have scanned all ports.
Hi @Stuart,
Thanks for your feedback, GVM/OpenVAS use the same engine so we can confirm this is a bug on MariaDB and should be fixed.
A brute-force should not be able to crash the entire cluster on port 4567 (we use TLS also so this shouldn't happen).
Hi all,
I thought I would verify/update as we have upgraded a cluster to 10.4.22.
@Nicolas I have found slightly different conditions are required to what you suggested
If I scan port 3306 I get denial messages from OpenVAS trying to logon, but the node stays up
If I scan port 4567 I get a lot of internal error messages, but the node stays up.
If I scan port 3306 and port 4567 in the same scan, the node crashes.
It seems that it is required to scan both of these ports at the same time to get the node to crash.
It would make sense that using OpenVAS defaults as a preset would crash the node as sequentially 4567 would follow 3306 as there would be no ports found inbetween.
Output follows:
OpenVAS Scan Port 3306 targeted.
Nov 19 16:55:03 NODE-01 mysqld[78300]: 2021-11-19 16:55:03 72 [Warning] Aborted connection 72 to db: 'unconnected' user: 'unauthenticated' host: 'the.scanner.ip.address' (This connection closed normally without authentication) |
Nov 19 16:55:03 NODE-01 mysqld[78300]: 2021-11-19 16:55:03 73 [Warning] Aborted connection 73 to db: 'unconnected' user: 'unauthenticated' host: 'the.scanner.ip.address' (This connection closed normally without authentication) |
Nov 19 16:55:03 NODE-01 mysqld[78300]: 2021-11-19 16:55:03 74 [Warning] Aborted connection 74 to db: 'unconnected' user: 'unauthenticated' host: 'the.scanner.ip.address' (This connection closed normally without authentication) |
Nov 19 16:55:03 NODE-01 mysqld[78300]: 2021-11-19 16:55:03 75 [Warning] Aborted connection 75 to db: 'unconnected' user: 'unauthenticated' host: 'the.scanner.ip.address' (This connection closed normally without authentication) |
Nov 19 16:55:03 NODE-01 mysqld[78300]: 2021-11-19 16:55:03 76 [Warning] Aborted connection 76 to db: 'unconnected' user: 'unauthenticated' host: 'the.scanner.ip.address' (This connection closed normally without authentication) |
Nov 19 16:55:04 NODE-01 mysqld[78300]: 2021-11-19 16:55:04 77 [Warning] Aborted connection 77 to db: 'unconnected' user: 'unauthenticated' host: 'the.scanner.ip.address' (This connection closed normally without authentication) |
Nov 19 16:55:04 NODE-01 mysqld[78300]: 2021-11-19 16:55:04 78 [Warning] Aborted connection 78 to db: 'unconnected' user: 'unauthenticated' host: 'the.scanner.ip.address' (This connection closed normally without authentication) |
Nov 19 16:55:04 NODE-01 mysqld[78300]: 2021-11-19 16:55:04 79 [Warning] Aborted connection 79 to db: 'unconnected' user: 'unauthenticated' host: 'the.scanner.ip.address' (This connection closed normally without authentication) |
Nov 19 16:55:04 NODE-01 mysqld[78300]: 2021-11-19 16:55:04 80 [Warning] Aborted connection 80 to db: 'unconnected' user: 'unauthenticated' host: 'the.scanner.ip.address' (This connection closed normally without authentication) |
Nov 19 16:55:04 NODE-01 mysqld[78300]: 2021-11-19 16:55:04 81 [Warning] Aborted connection 81 to db: 'unconnected' user: 'unauthenticated' host: 'the.scanner.ip.address' (This connection closed normally without authentication) |
Nov 19 16:55:04 NODE-01 mysqld[78300]: 2021-11-19 16:55:04 82 [Warning] Aborted connection 82 to db: 'unconnected' user: 'unauthenticated' host: 'the.scanner.ip.address' (This connection closed normally without authentication) |
Nov 19 16:55:04 NODE-01 mysqld[78300]: 2021-11-19 16:55:04 83 [Warning] Aborted connection 83 to db: 'unconnected' user: 'unauthenticated' host: 'the.scanner.ip.address' (This connection closed normally without authentication) |
Nov 19 16:55:04 NODE-01 mysqld[78300]: 2021-11-19 16:55:04 84 [Warning] Aborted connection 84 to db: 'unconnected' user: 'unauthenticated' host: 'the.scanner.ip.address' (This connection closed normally without authentication) |
Nov 19 16:55:45 NODE-01 mysqld[78300]: 2021-11-19 16:55:45 87 [Warning] Access denied for user 'htr'@'the.scanner.ip.address' (using password: YES) |
Nov 19 16:55:45 NODE-01 mysqld[78300]: 2021-11-19 16:55:45 88 [Warning] Access denied for user 'root'@'the.scanner.ip.address' (using password: YES) |
Nov 19 16:55:45 NODE-01 mysqld[78300]: 2021-11-19 16:55:45 91 [Warning] Access denied for user 'root'@'the.scanner.ip.address' (using password: YES) |
Nov 19 16:55:45 NODE-01 mysqld[78300]: 2021-11-19 16:55:45 92 [Warning] Access denied for user 'root'@'the.scanner.ip.address' (using password: YES) |
Nov 19 16:55:45 NODE-01 mysqld[78300]: 2021-11-19 16:55:45 93 [Warning] Access denied for user 'root'@'the.scanner.ip.address' (using password: YES) |
Nov 19 16:55:45 NODE-01 mysqld[78300]: 2021-11-19 16:55:45 94 [Warning] Access denied for user 'root'@'the.scanner.ip.address' (using password: YES) |
Nov 19 16:55:45 NODE-01 mysqld[78300]: 2021-11-19 16:55:45 95 [Warning] Access denied for user 'root'@'the.scanner.ip.address' (using password: YES) |
Nov 19 16:55:45 NODE-01 mysqld[78300]: 2021-11-19 16:55:45 96 [Warning] Access denied for user 'root'@'the.scanner.ip.address' (using password: YES) |
Nov 19 16:55:45 NODE-01 mysqld[78300]: 2021-11-19 16:55:45 89 [Warning] Aborted connection 89 to db: 'unconnected' user: 'unauthenticated' host: 'the.scanner.ip.address' (This connection closed normally without authentication) |
Nov 19 16:55:46 NODE-01 mysqld[78300]: 2021-11-19 16:55:46 97 [Warning] Access denied for user 'root'@'the.scanner.ip.address' (using password: YES) |
Nov 19 16:55:46 NODE-01 mysqld[78300]: 2021-11-19 16:55:46 98 [Warning] Access denied for user 'root'@'the.scanner.ip.address' (using password: YES) |
Nov 19 16:55:46 NODE-01 mysqld[78300]: 2021-11-19 16:55:46 99 [Warning] Access denied for user 'root'@'the.scanner.ip.address' (using password: YES) |
Nov 19 16:55:46 NODE-01 mysqld[78300]: 2021-11-19 16:55:46 100 [Warning] Access denied for user 'root'@'the.scanner.ip.address' (using password: YES) |
Nov 19 16:55:46 NODE-01 mysqld[78300]: 2021-11-19 16:55:46 101 [Warning] Access denied for user 'root'@'the.scanner.ip.address' (using password: NO) |
Nov 19 16:55:47 NODE-01 mysqld[78300]: 2021-11-19 16:55:47 90 [Warning] Aborted connection 90 to db: 'unconnected' user: 'unauthenticated' host: 'the.scanner.ip.address' (This connection closed normally without authentication) |
Nov 19 16:55:47 NODE-01 mysqld[78300]: 2021-11-19 16:55:47 102 [Warning] Aborted connection 102 to db: 'unconnected' user: 'unauthenticated' host: 'the.scanner.ip.address' (This connection closed normally without authentication) |
OpenVAS Scan Port 4567 targeted.
Nov 19 17:00:41 NODE-01 mysqld[78300]: 2021-11-19 17:00:41 0 [Warning] WSREP: unserialize error invalid protocol version 2: 71 (Protocol error) |
Nov 19 17:00:41 NODE-01 mysqld[78300]: at /home/buildbot/buildbot/build/gcomm/src/gcomm/datagram.hpp:unserialize():133 |
Nov 19 17:00:42 NODE-01 mysqld[78300]: 2021-11-19 17:00:42 0 [Warning] WSREP: unserialize error invalid protocol version 7: 71 (Protocol error) |
Nov 19 17:00:42 NODE-01 mysqld[78300]: at /home/buildbot/buildbot/build/gcomm/src/gcomm/datagram.hpp:unserialize():133 |
Nov 19 17:00:45 NODE-01 mysqld[78300]: 2021-11-19 17:00:45 0 [Note] WSREP: (1132bd81-a3fd, 'tcp://0.0.0.0:4567') connection to peer 00000000-0000 with addr timed out, no messages seen in PT3S, socket stats: rtt: 2976 rttvar: 1171 rto: 203000 lost: 0 last_data_recv: 110334802 cwnd: 10 last_queued_since: 3065247393 last_delivered_since: 110634801991969 send_queue_length: 0 send_queue_bytes: 0 segment: 0 messages: 0 |
Nov 19 17:00:45 NODE-01 mysqld[78300]: 2021-11-19 17:00:45 0 [Warning] WSREP: unserialize error invalid protocol version 2: 71 (Protocol error) |
Nov 19 17:00:45 NODE-01 mysqld[78300]: at /home/buildbot/buildbot/build/gcomm/src/gcomm/datagram.hpp:unserialize():133 |
Nov 19 17:00:45 NODE-01 mysqld[78300]: 2021-11-19 17:00:45 0 [Warning] WSREP: unserialize error invalid protocol version 2: 71 (Protocol error) |
Nov 19 17:00:45 NODE-01 mysqld[78300]: at /home/buildbot/buildbot/build/gcomm/src/gcomm/datagram.hpp:unserialize():133 |
Nov 19 17:00:45 NODE-01 mysqld[78300]: 2021-11-19 17:00:45 0 [Warning] WSREP: unserialize error invalid flags 11: 71 (Protocol error) |
Nov 19 17:00:45 NODE-01 mysqld[78300]: at /home/buildbot/buildbot/build/gcomm/src/gcomm/datagram.hpp:unserialize():126 |
Nov 19 17:00:47 NODE-01 mysqld[78300]: 2021-11-19 17:00:47 0 [Warning] WSREP: unserialize error invalid protocol version 6: 71 (Protocol error) |
Nov 19 17:00:47 NODE-01 mysqld[78300]: at /home/buildbot/buildbot/build/gcomm/src/gcomm/datagram.hpp:unserialize():133 |
Nov 19 17:00:47 NODE-01 mysqld[78300]: 2021-11-19 17:00:47 0 [Warning] WSREP: unserialize error invalid protocol version 4: 71 (Protocol error) |
Nov 19 17:00:47 NODE-01 mysqld[78300]: at /home/buildbot/buildbot/build/gcomm/src/gcomm/datagram.hpp:unserialize():133 |
Nov 19 17:00:47 NODE-01 mysqld[78300]: 2021-11-19 17:00:47 0 [Warning] WSREP: unserialize error invalid protocol version 2: 71 (Protocol error) |
Nov 19 17:00:47 NODE-01 mysqld[78300]: at /home/buildbot/buildbot/build/gcomm/src/gcomm/datagram.hpp:unserialize():133 |
Nov 19 17:00:47 NODE-01 mysqld[78300]: 2021-11-19 17:00:47 0 [Warning] WSREP: unserialize error invalid protocol version 6: 71 (Protocol error) |
Nov 19 17:00:47 NODE-01 mysqld[78300]: at /home/buildbot/buildbot/build/gcomm/src/gcomm/datagram.hpp:unserialize():133 |
Nov 19 17:00:47 NODE-01 mysqld[78300]: 2021-11-19 17:00:47 0 [Warning] WSREP: unserialize error invalid protocol version 4: 71 (Protocol error) |
Nov 19 17:00:47 NODE-01 mysqld[78300]: at /home/buildbot/buildbot/build/gcomm/src/gcomm/datagram.hpp:unserialize():133 |
Nov 19 17:01:04 NODE-01 mysqld[78300]: 2021-11-19 17:01:04 0 [Warning] WSREP: checksum failed, hdr: len=790 has_crc32=1 has_crc32c=1 crc32=50332065 |
Nov 19 17:01:04 NODE-01 mysqld[78300]: 2021-11-19 17:01:04 0 [Warning] WSREP: checksum failed, hdr: len=2 has_crc32=0 has_crc32c=0 crc32=16780288 |
Nov 19 17:01:13 NODE-01 mysqld[78300]: 2021-11-19 17:01:13 0 [Warning] WSREP: unserialize error invalid protocol version 4: 71 (Protocol error) |
Nov 19 17:01:13 NODE-01 mysqld[78300]: at /home/buildbot/buildbot/build/gcomm/src/gcomm/datagram.hpp:unserialize():133 |
Nov 19 17:01:13 NODE-01 mysqld[78300]: 2021-11-19 17:01:13 0 [Warning] WSREP: unserialize error invalid protocol version 2: 71 (Protocol error) |
Nov 19 17:01:13 NODE-01 mysqld[78300]: at /home/buildbot/buildbot/build/gcomm/src/gcomm/datagram.hpp:unserialize():133 |
Nov 19 17:01:14 NODE-01 mysqld[78300]: 2021-11-19 17:01:14 0 [Warning] WSREP: unserialize error invalid protocol version 5: 71 (Protocol error) |
Nov 19 17:01:14 NODE-01 mysqld[78300]: at /home/buildbot/buildbot/build/gcomm/src/gcomm/datagram.hpp:unserialize():133 |
Nov 19 17:01:14 NODE-01 mysqld[78300]: 2021-11-19 17:01:14 0 [Warning] WSREP: unserialize error invalid protocol version 7: 71 (Protocol error) |
Nov 19 17:01:14 NODE-01 mysqld[78300]: at /home/buildbot/buildbot/build/gcomm/src/gcomm/datagram.hpp:unserialize():133 |
Nov 19 17:01:14 NODE-01 mysqld[78300]: 2021-11-19 17:01:14 0 [Warning] WSREP: unserialize error invalid protocol version 5: 71 (Protocol error) |
Nov 19 17:01:14 NODE-01 mysqld[78300]: at /home/buildbot/buildbot/build/gcomm/src/gcomm/datagram.hpp:unserialize():133 |
Nov 19 17:01:14 NODE-01 mysqld[78300]: 2021-11-19 17:01:14 0 [Warning] WSREP: checksum failed, hdr: len=0 has_crc32=0 has_crc32c=0 crc32=721421568 |
Nov 19 17:01:14 NODE-01 mysqld[78300]: 2021-11-19 17:01:14 0 [Warning] WSREP: unserialize error invalid protocol version 4: 71 (Protocol error) |
Nov 19 17:01:14 NODE-01 mysqld[78300]: at /home/buildbot/buildbot/build/gcomm/src/gcomm/datagram.hpp:unserialize():133 |
Nov 19 17:01:26 NODE-01 mysqld[78300]: 2021-11-19 17:01:26 0 [Warning] WSREP: unserialize error invalid protocol version 5: 71 (Protocol error) |
Nov 19 17:01:26 NODE-01 mysqld[78300]: at /home/buildbot/buildbot/build/gcomm/src/gcomm/datagram.hpp:unserialize():133 |
Nov 19 17:01:27 NODE-01 mysqld[78300]: 2021-11-19 17:01:27 0 [Warning] WSREP: unserialize error invalid protocol version 1: 71 (Protocol error) |
Nov 19 17:01:27 NODE-01 mysqld[78300]: at /home/buildbot/buildbot/build/gcomm/src/gcomm/datagram.hpp:unserialize():133 |
Nov 19 17:01:32 NODE-01 mysqld[78300]: 2021-11-19 17:01:32 0 [Warning] WSREP: unserialize error invalid protocol version 5: 71 (Protocol error) |
Nov 19 17:01:32 NODE-01 mysqld[78300]: at /home/buildbot/buildbot/build/gcomm/src/gcomm/datagram.hpp:unserialize():133 |
Nov 19 17:01:32 NODE-01 mysqld[78300]: 2021-11-19 17:01:32 0 [Warning] WSREP: unserialize error invalid flags 6: 71 (Protocol error) |
Nov 19 17:01:32 NODE-01 mysqld[78300]: at /home/buildbot/buildbot/build/gcomm/src/gcomm/datagram.hpp:unserialize():126 |
Nov 19 17:01:48 NODE-01 mysqld[78300]: 2021-11-19 17:01:48 0 [Warning] WSREP: unserialize error invalid protocol version 6: 71 (Protocol error) |
Nov 19 17:01:48 NODE-01 mysqld[78300]: at /home/buildbot/buildbot/build/gcomm/src/gcomm/datagram.hpp:unserialize():133 |
Nov 19 17:01:48 NODE-01 mysqld[78300]: 2021-11-19 17:01:48 0 [Warning] WSREP: unserialize error invalid protocol version 6: 71 (Protocol error) |
Nov 19 17:01:48 NODE-01 mysqld[78300]: at /home/buildbot/buildbot/build/gcomm/src/gcomm/datagram.hpp:unserialize():133 |
Nov 19 17:01:48 NODE-01 mysqld[78300]: 2021-11-19 17:01:48 0 [Warning] WSREP: unserialize error invalid protocol version 15: 71 (Protocol error) |
Nov 19 17:01:48 NODE-01 mysqld[78300]: at /home/buildbot/buildbot/build/gcomm/src/gcomm/datagram.hpp:unserialize():133 |
Nov 19 17:01:48 NODE-01 mysqld[78300]: 2021-11-19 17:01:48 0 [Warning] WSREP: checksum failed, hdr: len=1 has_crc32=0 has_crc32c=0 crc32=1 |
Nov 19 17:01:48 NODE-01 mysqld[78300]: 2021-11-19 17:01:48 0 [Warning] WSREP: checksum failed, hdr: len=0 has_crc32=1 has_crc32c=0 crc32=318767104 |
Nov 19 17:01:48 NODE-01 mysqld[78300]: 2021-11-19 17:01:48 0 [Warning] WSREP: checksum failed, hdr: len=0 has_crc32=1 has_crc32c=0 crc32=318767104 |
Nov 19 17:01:48 NODE-01 mysqld[78300]: 2021-11-19 17:01:48 0 [Warning] WSREP: unserialize error invalid protocol version 6: 71 (Protocol error) |
Nov 19 17:01:48 NODE-01 mysqld[78300]: at /home/buildbot/buildbot/build/gcomm/src/gcomm/datagram.hpp:unserialize():133 |
Nov 19 17:01:48 NODE-01 mysqld[78300]: 2021-11-19 17:01:48 0 [Warning] WSREP: unserialize error invalid protocol version 5: 71 (Protocol error) |
Nov 19 17:01:48 NODE-01 mysqld[78300]: at /home/buildbot/buildbot/build/gcomm/src/gcomm/datagram.hpp:unserialize():133 |
Nov 19 17:01:50 NODE-01 mysqld[78300]: 2021-11-19 17:01:50 0 [Warning] WSREP: unserialize error invalid flags 6: 71 (Protocol error) |
Nov 19 17:01:50 NODE-01 mysqld[78300]: at /home/buildbot/buildbot/build/gcomm/src/gcomm/datagram.hpp:unserialize():126 |
Nov 19 17:01:50 NODE-01 mysqld[78300]: 2021-11-19 17:01:50 0 [Warning] WSREP: unserialize error invalid protocol version 5: 71 (Protocol error) |
Nov 19 17:01:50 NODE-01 mysqld[78300]: at /home/buildbot/buildbot/build/gcomm/src/gcomm/datagram.hpp:unserialize():133 |
Nov 19 17:01:50 NODE-01 mysqld[78300]: 2021-11-19 17:01:50 0 [Warning] WSREP: unserialize error invalid protocol version 4: 71 (Protocol error) |
Nov 19 17:01:50 NODE-01 mysqld[78300]: at /home/buildbot/buildbot/build/gcomm/src/gcomm/datagram.hpp:unserialize():133 |
Nov 19 17:01:51 NODE-01 mysqld[78300]: 2021-11-19 17:01:51 0 [Warning] WSREP: unserialize error invalid flags 4: 71 (Protocol error) |
Nov 19 17:01:51 NODE-01 mysqld[78300]: at /home/buildbot/buildbot/build/gcomm/src/gcomm/datagram.hpp:unserialize():126 |
Nov 19 17:01:51 NODE-01 mysqld[78300]: 2021-11-19 17:01:51 0 [Warning] WSREP: unserialize error invalid protocol version 5: 71 (Protocol error) |
Nov 19 17:01:51 NODE-01 mysqld[78300]: at /home/buildbot/buildbot/build/gcomm/src/gcomm/datagram.hpp:unserialize():133 |
Nov 19 17:01:51 NODE-01 mysqld[78300]: 2021-11-19 17:01:51 0 [Warning] WSREP: unserialize error invalid protocol version 12: 71 (Protocol error) |
Nov 19 17:01:51 NODE-01 mysqld[78300]: at /home/buildbot/buildbot/build/gcomm/src/gcomm/datagram.hpp:unserialize():133 |
Nov 19 17:01:52 NODE-01 mysqld[78300]: 2021-11-19 17:01:52 0 [Warning] WSREP: unserialize error invalid protocol version 1: 71 (Protocol error) |
Nov 19 17:01:52 NODE-01 mysqld[78300]: at /home/buildbot/buildbot/build/gcomm/src/gcomm/datagram.hpp:unserialize():133 |
Nov 19 17:01:52 NODE-01 mysqld[78300]: 2021-11-19 17:01:52 0 [Warning] WSREP: unserialize error invalid protocol version 5: 71 (Protocol error) |
Nov 19 17:01:52 NODE-01 mysqld[78300]: at /home/buildbot/buildbot/build/gcomm/src/gcomm/datagram.hpp:unserialize():133 |
Nov 19 17:01:59 NODE-01 mysqld[78300]: 2021-11-19 17:01:59 0 [Warning] WSREP: unserialize error invalid protocol version 5: 71 (Protocol error) |
Nov 19 17:01:59 NODE-01 mysqld[78300]: at /home/buildbot/buildbot/build/gcomm/src/gcomm/datagram.hpp:unserialize():133 |
Nov 19 17:01:59 NODE-01 mysqld[78300]: 2021-11-19 17:01:59 0 [Warning] WSREP: unserialize error invalid protocol version 5: 71 (Protocol error) |
Nov 19 17:01:59 NODE-01 mysqld[78300]: at /home/buildbot/buildbot/build/gcomm/src/gcomm/datagram.hpp:unserialize():133 |
Nov 19 17:02:00 NODE-01 mysqld[78300]: 2021-11-19 17:02:00 0 [Warning] WSREP: unserialize error invalid protocol version 5: 71 (Protocol error) |
Nov 19 17:02:00 NODE-01 mysqld[78300]: at /home/buildbot/buildbot/build/gcomm/src/gcomm/datagram.hpp:unserialize():133 |
Nov 19 17:02:00 NODE-01 mysqld[78300]: 2021-11-19 17:02:00 0 [Warning] WSREP: unserialize error invalid protocol version 5: 71 (Protocol error) |
Nov 19 17:02:00 NODE-01 mysqld[78300]: at /home/buildbot/buildbot/build/gcomm/src/gcomm/datagram.hpp:unserialize():133 |
Nov 19 17:02:03 NODE-01 mysqld[78300]: 2021-11-19 17:02:03 0 [Note] WSREP: (1132bd81-a3fd, 'tcp://0.0.0.0:4567') connection to peer 00000000-0000 with addr timed out, no messages seen in PT3S, socket stats: rtt: 2949 rttvar: 1226 rto: 203000 lost: 0 last_data_recv: 110413322 cwnd: 10 last_queued_since: 3024963641 last_delivered_since: 110713321447043 send_queue_length: 0 send_queue_bytes: 0 segment: 0 messages: 0 |
Nov 19 17:02:03 NODE-01 mysqld[78300]: 2021-11-19 17:02:03 0 [Warning] WSREP: unserialize error invalid protocol version 2: 71 (Protocol error) |
Nov 19 17:02:03 NODE-01 mysqld[78300]: at /home/buildbot/buildbot/build/gcomm/src/gcomm/datagram.hpp:unserialize():133 |
Nov 19 17:02:07 NODE-01 mysqld[78300]: 2021-11-19 17:02:07 0 [Note] WSREP: (1132bd81-a3fd, 'tcp://0.0.0.0:4567') connection to peer 00000000-0000 with addr timed out, no messages seen in PT3S, socket stats: rtt: 3250 rttvar: 1625 rto: 204000 lost: 0 last_data_recv: 3431 cwnd: 10 last_queued_since: 3433237748 last_delivered_since: 110716822135452 send_queue_length: 0 send_queue_bytes: 0 segment: 0 messages: 0 |
Nov 19 17:02:07 NODE-01 mysqld[78300]: 2021-11-19 17:02:07 0 [Warning] WSREP: unserialize error invalid protocol version 4: 71 (Protocol error) |
Nov 19 17:02:07 NODE-01 mysqld[78300]: at /home/buildbot/buildbot/build/gcomm/src/gcomm/datagram.hpp:unserialize():133 |
Nov 19 17:02:07 NODE-01 mysqld[78300]: 2021-11-19 17:02:07 0 [Warning] WSREP: unserialize error invalid protocol version 4: 71 (Protocol error) |
Nov 19 17:02:07 NODE-01 mysqld[78300]: at /home/buildbot/buildbot/build/gcomm/src/gcomm/datagram.hpp:unserialize():133 |
Nov 19 17:02:07 NODE-01 mysqld[78300]: 2021-11-19 17:02:07 0 [Warning] WSREP: unserialize error invalid protocol version 2: 71 (Protocol error) |
Nov 19 17:02:07 NODE-01 mysqld[78300]: at /home/buildbot/buildbot/build/gcomm/src/gcomm/datagram.hpp:unserialize():133 |
Nov 19 17:02:07 NODE-01 mysqld[78300]: 2021-11-19 17:02:07 0 [Warning] WSREP: unserialize error invalid flags 6: 71 (Protocol error) |
Nov 19 17:02:07 NODE-01 mysqld[78300]: at /home/buildbot/buildbot/build/gcomm/src/gcomm/datagram.hpp:unserialize():126 |
Nov 19 17:02:09 NODE-01 mysqld[78300]: 2021-11-19 17:02:09 0 [Warning] WSREP: unserialize error invalid protocol version 3: 71 (Protocol error) |
Nov 19 17:02:09 NODE-01 mysqld[78300]: at /home/buildbot/buildbot/build/gcomm/src/gcomm/datagram.hpp:unserialize():133 |
Nov 19 17:02:10 NODE-01 mysqld[78300]: 2021-11-19 17:02:10 0 [Note] WSREP: (1132bd81-a3fd, 'tcp://0.0.0.0:4567') connection to peer 00000000-0000 with addr timed out, no messages seen in PT3S, socket stats: rtt: 3258 rttvar: 1642 rto: 204000 lost: 0 last_data_recv: 3388 cwnd: 10 last_queued_since: 3391322802 last_delivered_since: 110720322902424 send_queue_length: 0 send_queue_bytes: 0 segment: 0 messages: 0 |
Nov 19 17:02:14 NODE-01 mysqld[78300]: 2021-11-19 17:02:14 0 [Note] WSREP: (1132bd81-a3fd, 'tcp://0.0.0.0:4567') connection to peer 00000000-0000 with addr timed out, no messages seen in PT3S, socket stats: rtt: 2412 rttvar: 1574 rto: 203000 lost: 0 last_data_recv: 3489 cwnd: 10 last_queued_since: 3491386280 last_delivered_since: 110723823637520 send_queue_length: 0 send_queue_bytes: 0 segment: 0 messages: 0 |
Nov 19 17:02:17 NODE-01 mysqld[78300]: 2021-11-19 17:02:17 0 [Note] WSREP: (1132bd81-a3fd, 'tcp://0.0.0.0:4567') connection to peer 00000000-0000 with addr timed out, no messages seen in PT3S, socket stats: rtt: 46870 rttvar: 17884 rto: 247000 lost: 0 last_data_recv: 3447 cwnd: 10 last_queued_since: 3448829822 last_delivered_since: 110727324411113 send_queue_length: 0 send_queue_bytes: 0 segment: 0 messages: 0 |
Nov 19 17:02:17 NODE-01 mysqld[78300]: 2021-11-19 17:02:17 0 [Warning] WSREP: unserialize error invalid protocol version 2: 71 (Protocol error) |
Nov 19 17:02:17 NODE-01 mysqld[78300]: at /home/buildbot/buildbot/build/gcomm/src/gcomm/datagram.hpp:unserialize():133 |
Nov 19 17:02:21 NODE-01 mysqld[78300]: 2021-11-19 17:02:21 0 [Note] WSREP: (1132bd81-a3fd, 'tcp://0.0.0.0:4567') connection to peer 00000000-0000 with addr timed out, no messages seen in PT3S, socket stats: rtt: 2380 rttvar: 1510 rto: 203000 lost: 0 last_data_recv: 3387 cwnd: 10 last_queued_since: 3389095114 last_delivered_since: 110730825168063 send_queue_length: 0 send_queue_bytes: 0 segment: 0 messages: 0 |
Nov 19 17:02:21 NODE-01 mysqld[78300]: 2021-11-19 17:02:21 0 [Warning] WSREP: unserialize error invalid protocol version 7: 71 (Protocol error) |
Nov 19 17:02:21 NODE-01 mysqld[78300]: at /home/buildbot/buildbot/build/gcomm/src/gcomm/datagram.hpp:unserialize():133 |
Nov 19 17:02:21 NODE-01 mysqld[78300]: 2021-11-19 17:02:21 0 [Warning] WSREP: unserialize error invalid protocol version 10: 71 (Protocol error) |
Nov 19 17:02:21 NODE-01 mysqld[78300]: at /home/buildbot/buildbot/build/gcomm/src/gcomm/datagram.hpp:unserialize():133 |
Nov 19 17:02:24 NODE-01 mysqld[78300]: 2021-11-19 17:02:24 0 [Note] WSREP: (1132bd81-a3fd, 'tcp://0.0.0.0:4567') connection to peer 00000000-0000 with addr timed out, no messages seen in PT3S, socket stats: rtt: 2362 rttvar: 1475 rto: 203000 lost: 0 last_data_recv: 3338 cwnd: 10 last_queued_since: 3341175489 last_delivered_since: 110734325955033 send_queue_length: 0 send_queue_bytes: 0 segment: 0 messages: 0 |
Nov 19 17:02:24 NODE-01 mysqld[78300]: 2021-11-19 17:02:24 0 [Warning] WSREP: unserialize error invalid protocol version 2: 71 (Protocol error) |
Nov 19 17:02:24 NODE-01 mysqld[78300]: at /home/buildbot/buildbot/build/gcomm/src/gcomm/datagram.hpp:unserialize():133 |
Nov 19 17:02:24 NODE-01 mysqld[78300]: 2021-11-19 17:02:24 0 [Warning] WSREP: unserialize error invalid protocol version 6: 71 (Protocol error) |
Nov 19 17:02:24 NODE-01 mysqld[78300]: at /home/buildbot/buildbot/build/gcomm/src/gcomm/datagram.hpp:unserialize():133 |
Nov 19 17:02:28 NODE-01 mysqld[78300]: 2021-11-19 17:02:28 0 [Note] WSREP: (1132bd81-a3fd, 'tcp://0.0.0.0:4567') connection to peer 00000000-0000 with addr timed out, no messages seen in PT3S, socket stats: rtt: 2357 rttvar: 1464 rto: 203000 lost: 0 last_data_recv: 3332 cwnd: 10 last_queued_since: 3334558479 last_delivered_since: 110737826645820 send_queue_length: 0 send_queue_bytes: 0 segment: 0 messages: 0 |
Nov 19 17:02:31 NODE-01 mysqld[78300]: 2021-11-19 17:02:31 0 [Note] WSREP: (1132bd81-a3fd, 'tcp://0.0.0.0:4567') connection to peer 00000000-0000 with addr timed out, no messages seen in PT3S, socket stats: rtt: 46817 rttvar: 17990 rto: 247000 lost: 0 last_data_recv: 3446 cwnd: 10 last_queued_since: 3447757772 last_delivered_since: 110741327415873 send_queue_length: 0 send_queue_bytes: 0 segment: 0 messages: 0 |
Nov 19 17:02:31 NODE-01 mysqld[78300]: 2021-11-19 17:02:31 0 [Warning] WSREP: unserialize error invalid protocol version 4: 71 (Protocol error) |
Nov 19 17:02:31 NODE-01 mysqld[78300]: at /home/buildbot/buildbot/build/gcomm/src/gcomm/datagram.hpp:unserialize():133 |
Nov 19 17:02:31 NODE-01 mysqld[78300]: 2021-11-19 17:02:31 0 [Warning] WSREP: unserialize error invalid protocol version 5: 71 (Protocol error) |
Nov 19 17:02:31 NODE-01 mysqld[78300]: at /home/buildbot/buildbot/build/gcomm/src/gcomm/datagram.hpp:unserialize():133 |
Nov 19 17:02:31 NODE-01 mysqld[78300]: 2021-11-19 17:02:31 0 [Warning] WSREP: unserialize error invalid flags 11: 71 (Protocol error) |
Nov 19 17:02:31 NODE-01 mysqld[78300]: at /home/buildbot/buildbot/build/gcomm/src/gcomm/datagram.hpp:unserialize():126 |
Nov 19 17:02:31 NODE-01 mysqld[78300]: 2021-11-19 17:02:31 0 [Warning] WSREP: unserialize error invalid protocol version 5: 71 (Protocol error) |
Nov 19 17:02:31 NODE-01 mysqld[78300]: at /home/buildbot/buildbot/build/gcomm/src/gcomm/datagram.hpp:unserialize():133 |
Nov 19 17:02:35 NODE-01 mysqld[78300]: 2021-11-19 17:02:35 0 [Note] WSREP: (1132bd81-a3fd, 'tcp://0.0.0.0:4567') connection to peer 00000000-0000 with addr timed out, no messages seen in PT3S, socket stats: rtt: 3234 rttvar: 1594 rto: 204000 lost: 0 last_data_recv: 3229 cwnd: 10 last_queued_since: 3230754498 last_delivered_since: 110744828097513 send_queue_length: 0 send_queue_bytes: 0 segment: 0 messages: 0 |
Nov 19 17:02:38 NODE-01 mysqld[78300]: 2021-11-19 17:02:38 0 [Note] WSREP: (1132bd81-a3fd, 'tcp://0.0.0.0:4567') connection to peer 00000000-0000 with addr timed out, no messages seen in PT3S, socket stats: rtt: 45089 rttvar: 17054 rto: 246000 lost: 0 last_data_recv: 3448 cwnd: 10 last_queued_since: 3449277638 last_delivered_since: 110748328836025 send_queue_length: 0 send_queue_bytes: 0 segment: 0 messages: 0 |
Nov 19 17:02:38 NODE-01 mysqld[78300]: 2021-11-19 17:02:38 0 [Warning] WSREP: unserialize error invalid protocol version 15: 71 (Protocol error) |
Nov 19 17:02:38 NODE-01 mysqld[78300]: at /home/buildbot/buildbot/build/gcomm/src/gcomm/datagram.hpp:unserialize():133 |
Nov 19 17:02:42 NODE-01 mysqld[78300]: 2021-11-19 17:02:42 0 [Note] WSREP: (1132bd81-a3fd, 'tcp://0.0.0.0:4567') connection to peer 00000000-0000 with addr timed out, no messages seen in PT3S, socket stats: rtt: 42631 rttvar: 24592 rto: 243000 lost: 0 last_data_recv: 3351 cwnd: 10 last_queued_since: 3353602435 last_delivered_since: 110751829586149 send_queue_length: 0 send_queue_bytes: 0 segment: 0 messages: 0 |
Nov 19 17:02:42 NODE-01 mysqld[78300]: 2021-11-19 17:02:42 0 [Warning] WSREP: unserialize error invalid protocol version 3: 71 (Protocol error) |
Nov 19 17:02:42 NODE-01 mysqld[78300]: at /home/buildbot/buildbot/build/gcomm/src/gcomm/datagram.hpp:unserialize():133 |
Nov 19 17:02:45 NODE-01 mysqld[78300]: 2021-11-19 17:02:45 0 [Note] WSREP: (1132bd81-a3fd, 'tcp://0.0.0.0:4567') connection to peer 00000000-0000 with addr timed out, no messages seen in PT3S, socket stats: rtt: 42619 rttvar: 24595 rto: 243000 lost: 0 last_data_recv: 3352 cwnd: 10 last_queued_since: 3354069244 last_delivered_since: 110755330277426 send_queue_length: 0 send_queue_bytes: 0 segment: 0 messages: 0 |
Nov 19 17:02:45 NODE-01 mysqld[78300]: 2021-11-19 17:02:45 0 [Warning] WSREP: unserialize error invalid protocol version 5: 71 (Protocol error) |
Nov 19 17:02:45 NODE-01 mysqld[78300]: at /home/buildbot/buildbot/build/gcomm/src/gcomm/datagram.hpp:unserialize():133 |
Nov 19 17:02:45 NODE-01 mysqld[78300]: 2021-11-19 17:02:45 0 [Warning] WSREP: unserialize error invalid protocol version 2: 71 (Protocol error) |
Nov 19 17:02:45 NODE-01 mysqld[78300]: at /home/buildbot/buildbot/build/gcomm/src/gcomm/datagram.hpp:unserialize():133 |
Nov 19 17:02:48 NODE-01 mysqld[78300]: 2021-11-19 17:02:48 0 [Warning] WSREP: unserialize error invalid protocol version 5: 71 (Protocol error) |
Nov 19 17:02:48 NODE-01 mysqld[78300]: at /home/buildbot/buildbot/build/gcomm/src/gcomm/datagram.hpp:unserialize():133 |
Nov 19 17:03:21 NODE-01 mysqld[78300]: 2021-11-19 17:03:21 0 [Warning] WSREP: checksum failed, hdr: len=0 has_crc32=0 has_crc32c=0 crc32=16777216 |
Nov 19 17:03:21 NODE-01 mysqld[78300]: 2021-11-19 17:03:21 0 [Warning] WSREP: checksum failed, hdr: len=0 has_crc32=0 has_crc32c=0 crc32=16777216 |
Nov 19 17:03:21 NODE-01 mysqld[78300]: 2021-11-19 17:03:21 0 [Warning] WSREP: unserialize error invalid protocol version 8: 71 (Protocol error) |
Nov 19 17:03:21 NODE-01 mysqld[78300]: at /home/buildbot/buildbot/build/gcomm/src/gcomm/datagram.hpp:unserialize():133 |
Nov 19 17:03:24 NODE-01 mysqld[78300]: 2021-11-19 17:03:24 0 [Note] WSREP: (1132bd81-a3fd, 'tcp://0.0.0.0:4567') connection to peer 00000000-0000 with addr timed out, no messages seen in PT3S, socket stats: rtt: 4207 rttvar: 1915 rto: 205000 lost: 0 last_data_recv: 1065 cwnd: 10 last_queued_since: 3077417409 last_delivered_since: 110793838298892 send_queue_length: 0 send_queue_bytes: 0 segment: 0 messages: 0 |
OpenVAS Scan Port 3306+4567 Targeted
Nov 19 16:47:42 NODE-01 mysqld[75077]: 2021-11-19 16:47:42 118 [Warning] Aborted connection 118 to db: 'unconnected' user: 'unauthenticated' host: 'connecting host' (This connection closed normally without authentication) |
Nov 19 16:47:42 NODE-01 mysqld[75077]: terminate called after throwing an instance of 'boost::exception_detail::clone_impl<boost::exception_detail::error_info_injector<std::system_error> >' |
Nov 19 16:47:42 NODE-01 mysqld[75077]: what(): remote_endpoint: Transport endpoint is not connected |
Nov 19 16:47:42 NODE-01 mysqld[75077]: 211119 16:47:42 [ERROR] mysqld got signal 6 ; |
Nov 19 16:47:42 NODE-01 mysqld[75077]: This could be because you hit a bug. It is also possible that this binary |
Nov 19 16:47:42 NODE-01 mysqld[75077]: or one of the libraries it was linked against is corrupt, improperly built, |
Nov 19 16:47:42 NODE-01 mysqld[75077]: or misconfigured. This error can also be caused by malfunctioning hardware. |
Nov 19 16:47:42 NODE-01 mysqld[75077]: To report this bug, see https://mariadb.com/kb/en/reporting-bugs |
Nov 19 16:47:42 NODE-01 mysqld[75077]: We will try our best to scrape up some info that will hopefully help |
Nov 19 16:47:42 NODE-01 mysqld[75077]: diagnose the problem, but since we have already crashed, |
Nov 19 16:47:42 NODE-01 mysqld[75077]: something is definitely wrong and this may fail. |
Nov 19 16:47:42 NODE-01 mysqld[75077]: Server version: 10.4.22-MariaDB-log |
Nov 19 16:47:42 NODE-01 mysqld[75077]: key_buffer_size=134217728 |
Nov 19 16:47:42 NODE-01 mysqld[75077]: read_buffer_size=131072 |
Nov 19 16:47:42 NODE-01 mysqld[75077]: max_used_connections=32 |
Nov 19 16:47:42 NODE-01 mysqld[75077]: max_threads=153 |
Nov 19 16:47:42 NODE-01 mysqld[75077]: thread_count=49 |
Nov 19 16:47:42 NODE-01 mysqld[75077]: It is possible that mysqld could use up to |
Nov 19 16:47:42 NODE-01 mysqld[75077]: key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 781113 K bytes of memory |
Nov 19 16:47:42 NODE-01 mysqld[75077]: Hope that's ok; if not, decrease some variables in the equation. |
Nov 19 16:47:42 NODE-01 mysqld[75077]: Thread pointer: 0x0 |
Nov 19 16:47:42 NODE-01 mysqld[75077]: Attempting backtrace. You can use the following information to find out |
Nov 19 16:47:42 NODE-01 mysqld[75077]: where mysqld died. If you see no messages after this, something went |
Nov 19 16:47:42 NODE-01 mysqld[75077]: terribly wrong... |
Nov 19 16:47:42 NODE-01 mysqld[75077]: stack_bottom = 0x0 thread_stack 0x49000 |
Nov 19 16:47:42 NODE-01 mysqld[75077]: /usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x55ce5f1dd2be] |
Nov 19 16:47:42 NODE-01 mysqld[75077]: /usr/sbin/mysqld(handle_fatal_signal+0x30f)[0x55ce5ec64dbf] |
Nov 19 16:47:42 NODE-01 mysqld[75077]: /lib64/libpthread.so.0(+0xf630)[0x7f682458c630] |
Nov 19 16:47:42 NODE-01 mysqld[75077]: /lib64/libc.so.6(gsignal+0x37)[0x7f68239d7387] |
Nov 19 16:47:42 NODE-01 mysqld[75077]: /lib64/libc.so.6(abort+0x148)[0x7f68239d8a78] |
Nov 19 16:47:42 NODE-01 mysqld[75077]: /lib64/libstdc++.so.6(_ZN9__gnu_cxx27__verbose_terminate_handlerEv+0x165)[0x7f68240d1a95] |
Nov 19 16:47:42 NODE-01 mysqld[75077]: /lib64/libstdc++.so.6(+0x5ea06)[0x7f68240cfa06] |
Nov 19 16:47:42 NODE-01 mysqld[75077]: /lib64/libstdc++.so.6(+0x5ea33)[0x7f68240cfa33] |
Nov 19 16:47:42 NODE-01 mysqld[75077]: /lib64/libstdc++.so.6(+0x5ec53)[0x7f68240cfc53] |
Nov 19 16:47:42 NODE-01 mysqld[75077]: /usr/lib64/galera-4/libgalera_smm.so(+0x197ab4)[0x7f6819b0bab4] |
Nov 19 16:47:42 NODE-01 mysqld[75077]: /usr/lib64/galera-4/libgalera_smm.so(+0x197b5c)[0x7f6819b0bb5c] |
Nov 19 16:47:42 NODE-01 mysqld[75077]: /usr/lib64/galera-4/libgalera_smm.so(+0x1a3cbf)[0x7f6819b17cbf] |
Nov 19 16:47:42 NODE-01 mysqld[75077]: /usr/lib64/galera-4/libgalera_smm.so(+0x1a62e2)[0x7f6819b1a2e2] |
Nov 19 16:47:42 NODE-01 mysqld[75077]: /usr/lib64/galera-4/libgalera_smm.so(+0x1aa533)[0x7f6819b1e533] |
Nov 19 16:47:42 NODE-01 mysqld[75077]: /usr/lib64/galera-4/libgalera_smm.so(+0x196eeb)[0x7f6819b0aeeb] |
Nov 19 16:47:42 NODE-01 mysqld[75077]: /usr/lib64/galera-4/libgalera_smm.so(+0x192d81)[0x7f6819b06d81] |
Nov 19 16:47:42 NODE-01 mysqld[75077]: /usr/lib64/galera-4/libgalera_smm.so(+0xbf599)[0x7f6819a33599] |
Nov 19 16:47:42 NODE-01 mysqld[75077]: /usr/lib64/galera-4/libgalera_smm.so(+0xa65d7)[0x7f6819a1a5d7] |
Nov 19 16:47:42 NODE-01 mysqld[75077]: /usr/lib64/galera-4/libgalera_smm.so(+0xa6989)[0x7f6819a1a989] |
Nov 19 16:47:42 NODE-01 mysqld[75077]: /lib64/libpthread.so.0(+0x7ea5)[0x7f6824584ea5] |
Nov 19 16:47:42 NODE-01 mysqld[75077]: /lib64/libc.so.6(clone+0x6d)[0x7f6823a9f9fd] |
Nov 19 16:47:42 NODE-01 mysqld[75077]: The manual page at https://mariadb.com/kb/en/how-to-produce-a-full-stack-trace-for-mysqld/ contains |
Nov 19 16:47:42 NODE-01 mysqld[75077]: information that should help you find out what is causing the crash. |
Nov 19 16:47:42 NODE-01 mysqld[75077]: Writing a core file... |
Nov 19 16:47:42 NODE-01 mysqld[75077]: Working directory at /var/lib/mysql |
Nov 19 16:47:42 NODE-01 mysqld[75077]: Resource Limits: |
Nov 19 16:47:42 NODE-01 mysqld[75077]: Fatal signal 11 while backtracing |
Nov 19 16:47:43 NODE-01 systemd[1]: mariadb.service: main process exited, code=killed, status=11/SEGV |
Nov 19 16:47:43 NODE-01 systemd[1]: Unit mariadb.service entered failed state. |
Nov 19 16:47:43 NODE-01 systemd[1]: mariadb.service failed. |
Nov 19 16:47:48 NODE-01 systemd[1]: mariadb.service holdoff time over, scheduling restart. |
Nov 19 16:47:48 NODE-01 systemd[1]: Stopped MariaDB 10.4.22 database server. |
Nov 19 16:47:48 NODE-01 systemd[1]: Starting MariaDB 10.4.22 database server... |
Additionally I guess it makes sense that I managed to crash our cluster entirely with our scheduled scan which is my OP here, we can run up to 5 machine scans in parallel so it is highly likely the 3 machines in the cluster being scanned at the same time caused a split-brain/impossible automated recovery.
| Affects Version/s | 10.4.22 [ 26031 ] |
| Summary | 10.4.18 crashes with Transport endpoint is not connected mysqld got signal 6 ; | Node crashes with Transport endpoint is not connected mysqld got signal 6 ; |
| Affects Version/s | 10.4.21 [ 26030 ] | |
| Affects Version/s | 10.4.20 [ 25733 ] | |
| Affects Version/s | 10.4.19 [ 25205 ] |
| Affects Version/s | 10.4 [ 22408 ] |
| Affects Version/s | 10.4 [ 22408 ] |
| Workflow | MariaDB v3 [ 119845 ] | MariaDB v4 [ 142658 ] |
| Assignee | Jan Lindström [ jplindst ] | Seppo Jaakola [ seppo ] |
Watching.
We have also experienced this exact behaviour on a customer's lab system, and it appears to be due to a scan from Nessus which from wikipedia I understand is the same/similar code base to OpenVAS. Luckily, the DB cluster was being sporadically scanned and the crashed node brought back on line before the scan took out a second node and hence split-brain of the 3 node cluster.
| Affects Version/s | 10.5.9 [ 25109 ] |
| Assignee | Seppo Jaakola [ seppo ] | Teemu Ollakka [ teemu.ollakka ] |
| Assignee | Teemu Ollakka [ teemu.ollakka ] | Mario Karuza [ mkaruza ] |
| Assignee | Mario Karuza [ mkaruza ] | Pramod Mahto [ pramod.mahto@mariadb.com ] |
| Status | Open [ 1 ] | Needs Feedback [ 10501 ] |
| Attachment | _InVuln__PCI-DSS_ORG023150_OneCloudINFRA_day3__jpw1a_2_of_2__09nptb.zip [ 62301 ] |
| Assignee | Pramod Mahto [ pramod.mahto@mariadb.com ] | Mario Karuza [ mkaruza ] |
| Link | This issue is blocked by MENT-1419 [ MENT-1419 ] |
| Status | Needs Feedback [ 10501 ] | Open [ 1 ] |
| Assignee | Mario Karuza [ mkaruza ] | Jan Lindström [ jplindst ] |
| Status | Open [ 1 ] | Confirmed [ 10101 ] |
| Status | Confirmed [ 10101 ] | In Progress [ 3 ] |
| Assignee | Jan Lindström [ jplindst ] | Julien Fritsch [ julien.fritsch ] |
| Assignee | Julien Fritsch [ julien.fritsch ] | Jan Lindström [ jplindst ] |
| Status | In Progress [ 3 ] | In Review [ 10002 ] |
| Link |
This issue relates to |
| Link |
This issue relates to |
| Priority | Major [ 3 ] | Critical [ 2 ] |
I have the same issue, but on MariaDB 10.6.7. Maria crashes when OpenVas scans the host.
| issue.field.resolutiondate | 2022-06-03 05:31:13.0 | 2022-06-03 05:31:13.396 |
| Component/s | Galera [ 10124 ] | |
| Component/s | wsrep [ 11500 ] | |
| Fix Version/s | 10.4.26 [ 27511 ] | |
| Fix Version/s | 10.5.17 [ 27509 ] | |
| Fix Version/s | 10.8.4 [ 27503 ] | |
| Fix Version/s | 10.6.9, 10.7.5 [ 28203 ] | |
| Fix Version/s | 10.4 [ 22408 ] | |
| Resolution | Fixed [ 1 ] | |
| Status | In Review [ 10002 ] | Closed [ 6 ] |
| Fix Version/s | 10.6.9 [ 27507 ] | |
| Fix Version/s | 10.7.5 [ 27505 ] | |
| Fix Version/s | 10.6.9, 10.7.5 [ 28203 ] |
| Link |
This issue is duplicated by |
| Remote Link | This issue links to "CVE-2023-5157 (Web Link)" [ 36165 ] |
| Fix Version/s | 10.3.36 [ 27513 ] |
| Zendesk Related Tickets | 136631 |
Just got the same error. Also 3 node setup. Only one node crashed.
Log:
Mar 28 22:07:16 mysqld[5932]: terminate called after throwing an instance of 'boost::exception_detail::clone_impl<boost::exception_detail::error_info_injector<std::system_error> >'Mar 28 22:07:16 mysqld[5932]: what(): remote_endpoint: Transport endpoint is not connectedMar 28 22:07:16 mysqld[5932]: 210328 22:07:16 [ERROR] mysqld got signal 6 ;Mar 28 22:07:16 mysqld[5932]: This could be because you hit a bug. It is also possible that this binaryMar 28 22:07:16 mysqld[5932]: or one of the libraries it was linked against is corrupt, improperly built,Mar 28 22:07:16 mysqld[5932]: or misconfigured. This error can also be caused by malfunctioning hardware.Mar 28 22:07:16 mysqld[5932]: To report this bug, see https://mariadb.com/kb/en/reporting-bugsMar 28 22:07:16 mysqld[5932]: We will try our best to scrape up some info that will hopefully helpMar 28 22:07:16 mysqld[5932]: diagnose the problem, but since we have already crashed,Mar 28 22:07:16 mysqld[5932]: something is definitely wrong and this may fail.Mar 28 22:07:16 mysqld[5932]: Server version: 10.4.18-MariaDB-1:10.4.18+maria~busterMar 28 22:07:16 mysqld[5932]: key_buffer_size=134217728Mar 28 22:07:16 mysqld[5932]: read_buffer_size=131072Mar 28 22:07:16 mysqld[5932]: max_used_connections=20Mar 28 22:07:16 mysqld[5932]: max_threads=402Mar 28 22:07:16 mysqld[5932]: thread_count=27Mar 28 22:07:16 mysqld[5932]: It is possible that mysqld could use up toMar 28 22:07:16 mysqld[5932]: key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 1015745 K bytes of memoryMar 28 22:07:16 mysqld[5932]: Hope that's ok; if not, decrease some variables in the equation.Mar 28 22:07:16 mysqld[5932]: Thread pointer: 0x0Mar 28 22:07:16 mysqld[5932]: Attempting backtrace. You can use the following information to find outMar 28 22:07:16 mysqld[5932]: where mysqld died. If you see no messages after this, something wentMar 28 22:07:16 mysqld[5932]: terribly wrong...Mar 28 22:07:16 mysqld[5932]: stack_bottom = 0x0 thread_stack 0x49000Mar 28 22:07:16 mysqld[5932]: /usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x55f1f23727fe]Mar 28 22:07:16 mysqld[5932]: /usr/sbin/mysqld(handle_fatal_signal+0x54d)[0x55f1f1e65b0d]Mar 28 22:07:16 mysqld[5932]: /lib/x86_64-linux-gnu/libpthread.so.0(+0x12730)[0x7fb30c287730]Mar 28 22:07:16 mysqld[5932]: /lib/x86_64-linux-gnu/libc.so.6(gsignal+0x10b)[0x7fb30b6cf7bb]Mar 28 22:07:16 mysqld[5932]: /lib/x86_64-linux-gnu/libc.so.6(abort+0x121)[0x7fb30b6ba535]Mar 28 22:07:16 mysqld[5932]: /lib/x86_64-linux-gnu/libstdc++.so.6(+0x8c983)[0x7fb30ba6a983]Mar 28 22:07:16 mysqld[5932]: /lib/x86_64-linux-gnu/libstdc++.so.6(+0x928c6)[0x7fb30ba708c6]Mar 28 22:07:16 mysqld[5932]: /lib/x86_64-linux-gnu/libstdc++.so.6(+0x92901)[0x7fb30ba70901]Mar 28 22:07:16 mysqld[5932]: /lib/x86_64-linux-gnu/libstdc++.so.6(+0x92b34)[0x7fb30ba70b34]Mar 28 22:07:16 mysqld[5932]: /usr/lib/galera/libgalera_smm.so(+0x1b7cb6)[0x7fb30ac38cb6]Mar 28 22:07:16 mysqld[5932]: /usr/lib/galera/libgalera_smm.so(+0x1c4274)[0x7fb30ac45274]Mar 28 22:07:16 mysqld[5932]: /usr/lib/galera/libgalera_smm.so(+0x1c70bd)[0x7fb30ac480bd]Mar 28 22:07:16 mysqld[5932]: /usr/lib/galera/libgalera_smm.so(+0x1c73d7)[0x7fb30ac483d7]Mar 28 22:07:16 mysqld[5932]: /usr/lib/galera/libgalera_smm.so(+0x1cdc60)[0x7fb30ac4ec60]Mar 28 22:07:16 mysqld[5932]: /usr/lib/galera/libgalera_smm.so(+0x1bf6e5)[0x7fb30ac406e5]Mar 28 22:07:16 mysqld[5932]: /usr/lib/galera/libgalera_smm.so(+0x1b10e7)[0x7fb30ac320e7]Mar 28 22:07:16 mysqld[5932]: /usr/lib/galera/libgalera_smm.so(+0xe47e8)[0x7fb30ab657e8]Mar 28 22:07:16 mysqld[5932]: /usr/lib/galera/libgalera_smm.so(+0xc9114)[0x7fb30ab4a114]Mar 28 22:07:16 mysqld[5932]: /usr/lib/galera/libgalera_smm.so(+0xcfbf9)[0x7fb30ab50bf9]Mar 28 22:07:16 mysqld[5932]: /lib/x86_64-linux-gnu/libpthread.so.0(+0x7fa3)[0x7fb30c27cfa3]Mar 28 22:07:16 mysqld[5932]: /lib/x86_64-linux-gnu/libc.so.6(clone+0x3f)[0x7fb30b7914cf]Mar 28 22:07:16 mysqld[5932]: The manual page at https://mariadb.com/kb/en/how-to-produce-a-full-stack-trace-for-mysqld/ containsMar 28 22:07:16 mysqld[5932]: information that should help you find out what is causing the crash.Mar 28 22:07:16 mysqld[5932]: Writing a core file...Mar 28 22:07:16 mysqld[5932]: Working directory at /srv/mysqlMar 28 22:07:16 mysqld[5932]: Resource Limits:Mar 28 22:07:16 mysqld[5932]: Fatal signal 11 while backtracingMar 28 22:07:16 systemd[1]: mariadb.service: Main process exited, code=killed, status=11/SEGVMar 28 22:07:16 systemd[1]: mariadb.service: Failed with result 'signal'.