[MDEV-25068] Node crashes with Transport endpoint is not connected mysqld got signal 6 ; Created: 2021-03-05  Updated: 2023-12-29  Resolved: 2022-06-03

Status: Closed
Project: MariaDB Server
Component/s: Galera
Affects Version/s: 10.4.18, 10.4.19, 10.5.9, 10.4.20, 10.4.21, 10.4.22
Fix Version/s: 10.3.36, 10.4.26, 10.5.17, 10.6.9, 10.7.5, 10.8.4

Type: Bug Priority: Critical
Reporter: Stuart Green Assignee: Jan Lindström (Inactive)
Resolution: Fixed Votes: 4
Labels: None
Environment:

CentOS

Attachments: Zip Archive _InVuln__PCI-DSS_ORG023150_OneCloudINFRA_day3__jpw1a_2_of_2__09nptb.zip     Text File gdb.txt     File log-10.4.2     File log-10.5.11    
Issue Links:
Blocks
Duplicate
is duplicated by MDEV-30336 Galera: Handshake failed: peer did no... Closed
Relates
relates to MDEV-27004 McAfee database vulnerability scan ca... Closed
relates to MDEV-26861 Galera Crashing - what(): remote_end... Closed

 Description   

3 Node Setup, all 3 nodes crashed. Bootstrap required.

Node 1 crash output

Mar  5 17:48:58  mysqld[116636]: terminate called after throwing an instance of 'boost::exception_detail::clone_impl<boost::exception_detail::error_info_injector<std::system_error> >'
 
Mar  5 17:48:58  mysqld[116636]: what():  remote_endpoint: Transport endpoint is not connected
 
Mar  5 17:48:58  mysqld[116636]: 210305 17:48:58 [ERROR] mysqld got signal 6 ;
 
Mar  5 17:48:58  mysqld[116636]: This could be because you hit a bug. It is also possible that this binary
 
Mar  5 17:48:58  mysqld[116636]: or one of the libraries it was linked against is corrupt, improperly built,
 
Mar  5 17:48:58  mysqld[116636]: or misconfigured. This error can also be caused by malfunctioning hardware.
 
Mar  5 17:48:58  mysqld[116636]: To report this bug, see https://mariadb.com/kb/en/reporting-bugs
 
Mar  5 17:48:58  mysqld[116636]: We will try our best to scrape up some info that will hopefully help
 
Mar  5 17:48:58  mysqld[116636]: diagnose the problem, but since we have already crashed,
 
Mar  5 17:48:58  mysqld[116636]: something is definitely wrong and this may fail.
 
Mar  5 17:48:58  mysqld[116636]: Server version: 10.4.18-MariaDB-log
 
Mar  5 17:48:58  mysqld[116636]: key_buffer_size=134217728
 
Mar  5 17:48:58  mysqld[116636]: read_buffer_size=131072
 
Mar  5 17:48:58  mysqld[116636]: max_used_connections=76
 
Mar  5 17:48:58  mysqld[116636]: max_threads=153
 
Mar  5 17:48:58  mysqld[116636]: thread_count=75
 
Mar  5 17:48:58  mysqld[116636]: It is possible that mysqld could use up to
 
Mar  5 17:48:58  mysqld[116636]: key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 781104 K  bytes of memory
 
Mar  5 17:48:58  mysqld[116636]: Hope that's ok; if not, decrease some variables in the equation.
 
Mar  5 17:48:58  mysqld[116636]: Thread pointer: 0x0
 
Mar  5 17:48:58  mysqld[116636]: Attempting backtrace. You can use the following information to find out
 
Mar  5 17:48:58  mysqld[116636]: where mysqld died. If you see no messages after this, something went
 
Mar  5 17:48:58  mysqld[116636]: terribly wrong...
 
Mar  5 17:48:59  mysqld[116636]: stack_bottom = 0x0 thread_stack 0x49000
 
Mar  5 17:48:59  mysqld[116636]: /usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x55a14151e47e]
 
Mar  5 17:48:59  mysqld[116636]: /usr/sbin/mysqld(handle_fatal_signal+0x30f)[0x55a140face8f]
 
Mar  5 17:48:59  mysqld[116636]: /lib64/libpthread.so.0(+0xf630)[0x7f5879976630]
 
Mar  5 17:48:59  mysqld[116636]: /lib64/libc.so.6(gsignal+0x37)[0x7f5877c463d7]
 
Mar  5 17:48:59  mysqld[116636]: /lib64/libc.so.6(abort+0x148)[0x7f5877c47ac8]
 
Mar  5 17:48:59  mysqld[116636]: /lib64/libstdc++.so.6(_ZN9__gnu_cxx27__verbose_terminate_handlerEv+0x165)[0x7f5878340a95]
 
Mar  5 17:48:59  mysqld[116636]: /lib64/libstdc++.so.6(+0x5ea06)[0x7f587833ea06]
 
Mar  5 17:48:59  mysqld[116636]: /lib64/libstdc++.so.6(+0x5ea33)[0x7f587833ea33]
 
Mar  5 17:48:59  mysqld[116636]: /lib64/libstdc++.so.6(+0x5ec53)[0x7f587833ec53]
 
Mar  5 17:48:59  mysqld[116636]: /usr/lib64/galera-4/libgalera_smm.so(+0x1997a4)[0x7f586ddc07a4]
 
Mar  5 17:48:59  mysqld[116636]: /usr/lib64/galera-4/libgalera_smm.so(+0x19984c)[0x7f586ddc084c]
 
Mar  5 17:48:59  mysqld[116636]: /usr/lib64/galera-4/libgalera_smm.so(+0x1a57bf)[0x7f586ddcc7bf]
 
Mar  5 17:48:59  mysqld[116636]: /usr/lib64/galera-4/libgalera_smm.so(+0x1a78b8)[0x7f586ddce8b8]
 
Mar  5 17:48:59  mysqld[116636]: /usr/lib64/galera-4/libgalera_smm.so(+0x1aaf13)[0x7f586ddd1f13]
 
Mar  5 17:48:59  mysqld[116636]: /usr/lib64/galera-4/libgalera_smm.so(+0x19897b)[0x7f586ddbf97b]
 
Mar  5 17:48:59  mysqld[116636]: /usr/lib64/galera-4/libgalera_smm.so(+0x194b91)[0x7f586ddbbb91]
 
Mar  5 17:48:59  mysqld[116636]: /usr/lib64/galera-4/libgalera_smm.so(+0xc08c9)[0x7f586dce78c9]
 
Mar  5 17:48:59  mysqld[116636]: /usr/lib64/galera-4/libgalera_smm.so(+0xa6bcf)[0x7f586dccdbcf]
 
Mar  5 17:48:59  mysqld[116636]: /usr/lib64/galera-4/libgalera_smm.so(+0xad859)[0x7f586dcd4859]
 
Mar  5 17:48:59  mysqld[116636]: /lib64/libpthread.so.0(+0x7ea5)[0x7f587996eea5]
 
Mar  5 17:48:59  mysqld[116636]: /lib64/libc.so.6(clone+0x6d)[0x7f5877d0e9fd]



 Comments   
Comment by Steffen Persch [ 2021-03-28 ]

Just got the same error. Also 3 node setup. Only one node crashed.

Log:

Mar 28 22:07:16 mysqld[5932]: terminate called after throwing an instance of 'boost::exception_detail::clone_impl<boost::exception_detail::error_info_injector<std::system_error> >'
 
Mar 28 22:07:16 mysqld[5932]:   what():  remote_endpoint: Transport endpoint is not connected
 
Mar 28 22:07:16 mysqld[5932]: 210328 22:07:16 [ERROR] mysqld got signal 6 ;
 
Mar 28 22:07:16 mysqld[5932]: This could be because you hit a bug. It is also possible that this binary
 
Mar 28 22:07:16 mysqld[5932]: or one of the libraries it was linked against is corrupt, improperly built,
 
Mar 28 22:07:16 mysqld[5932]: or misconfigured. This error can also be caused by malfunctioning hardware.
 
Mar 28 22:07:16 mysqld[5932]: To report this bug, see https://mariadb.com/kb/en/reporting-bugs
 
Mar 28 22:07:16 mysqld[5932]: We will try our best to scrape up some info that will hopefully help
 
Mar 28 22:07:16 mysqld[5932]: diagnose the problem, but since we have already crashed,
 
Mar 28 22:07:16 mysqld[5932]: something is definitely wrong and this may fail.
 
Mar 28 22:07:16 mysqld[5932]: Server version: 10.4.18-MariaDB-1:10.4.18+maria~buster
 
Mar 28 22:07:16 mysqld[5932]: key_buffer_size=134217728
 
Mar 28 22:07:16 mysqld[5932]: read_buffer_size=131072
 
Mar 28 22:07:16 mysqld[5932]: max_used_connections=20
 
Mar 28 22:07:16 mysqld[5932]: max_threads=402
 
Mar 28 22:07:16 mysqld[5932]: thread_count=27
 
Mar 28 22:07:16 mysqld[5932]: It is possible that mysqld could use up to
 
Mar 28 22:07:16 mysqld[5932]: key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 1015745 K  bytes of memory
 
Mar 28 22:07:16 mysqld[5932]: Hope that's ok; if not, decrease some variables in the equation.
 
Mar 28 22:07:16 mysqld[5932]: Thread pointer: 0x0
 
Mar 28 22:07:16 mysqld[5932]: Attempting backtrace. You can use the following information to find out
 
Mar 28 22:07:16 mysqld[5932]: where mysqld died. If you see no messages after this, something went
 
Mar 28 22:07:16 mysqld[5932]: terribly wrong...
 
Mar 28 22:07:16 mysqld[5932]: stack_bottom = 0x0 thread_stack 0x49000
 
Mar 28 22:07:16 mysqld[5932]: /usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x55f1f23727fe]
 
Mar 28 22:07:16 mysqld[5932]: /usr/sbin/mysqld(handle_fatal_signal+0x54d)[0x55f1f1e65b0d]
 
Mar 28 22:07:16 mysqld[5932]: /lib/x86_64-linux-gnu/libpthread.so.0(+0x12730)[0x7fb30c287730]
 
Mar 28 22:07:16 mysqld[5932]: /lib/x86_64-linux-gnu/libc.so.6(gsignal+0x10b)[0x7fb30b6cf7bb]
 
Mar 28 22:07:16 mysqld[5932]: /lib/x86_64-linux-gnu/libc.so.6(abort+0x121)[0x7fb30b6ba535]
 
Mar 28 22:07:16 mysqld[5932]: /lib/x86_64-linux-gnu/libstdc++.so.6(+0x8c983)[0x7fb30ba6a983]
 
Mar 28 22:07:16 mysqld[5932]: /lib/x86_64-linux-gnu/libstdc++.so.6(+0x928c6)[0x7fb30ba708c6]
 
Mar 28 22:07:16 mysqld[5932]: /lib/x86_64-linux-gnu/libstdc++.so.6(+0x92901)[0x7fb30ba70901]
 
Mar 28 22:07:16 mysqld[5932]: /lib/x86_64-linux-gnu/libstdc++.so.6(+0x92b34)[0x7fb30ba70b34]
 
Mar 28 22:07:16 mysqld[5932]: /usr/lib/galera/libgalera_smm.so(+0x1b7cb6)[0x7fb30ac38cb6]
 
Mar 28 22:07:16 mysqld[5932]: /usr/lib/galera/libgalera_smm.so(+0x1c4274)[0x7fb30ac45274]
 
Mar 28 22:07:16 mysqld[5932]: /usr/lib/galera/libgalera_smm.so(+0x1c70bd)[0x7fb30ac480bd]
 
Mar 28 22:07:16 mysqld[5932]: /usr/lib/galera/libgalera_smm.so(+0x1c73d7)[0x7fb30ac483d7]
 
Mar 28 22:07:16 mysqld[5932]: /usr/lib/galera/libgalera_smm.so(+0x1cdc60)[0x7fb30ac4ec60]
 
Mar 28 22:07:16 mysqld[5932]: /usr/lib/galera/libgalera_smm.so(+0x1bf6e5)[0x7fb30ac406e5]
 
Mar 28 22:07:16 mysqld[5932]: /usr/lib/galera/libgalera_smm.so(+0x1b10e7)[0x7fb30ac320e7]
 
Mar 28 22:07:16 mysqld[5932]: /usr/lib/galera/libgalera_smm.so(+0xe47e8)[0x7fb30ab657e8]
 
Mar 28 22:07:16 mysqld[5932]: /usr/lib/galera/libgalera_smm.so(+0xc9114)[0x7fb30ab4a114]
 
Mar 28 22:07:16 mysqld[5932]: /usr/lib/galera/libgalera_smm.so(+0xcfbf9)[0x7fb30ab50bf9]
 
Mar 28 22:07:16 mysqld[5932]: /lib/x86_64-linux-gnu/libpthread.so.0(+0x7fa3)[0x7fb30c27cfa3]
 
Mar 28 22:07:16 mysqld[5932]: /lib/x86_64-linux-gnu/libc.so.6(clone+0x3f)[0x7fb30b7914cf]
 
Mar 28 22:07:16 mysqld[5932]: The manual page at https://mariadb.com/kb/en/how-to-produce-a-full-stack-trace-for-mysqld/ contains
 
Mar 28 22:07:16 mysqld[5932]: information that should help you find out what is causing the crash.
 
Mar 28 22:07:16 mysqld[5932]: Writing a core file...
 
Mar 28 22:07:16 mysqld[5932]: Working directory at /srv/mysql
 
Mar 28 22:07:16 mysqld[5932]: Resource Limits:
 
Mar 28 22:07:16 mysqld[5932]: Fatal signal 11 while backtracing
 
Mar 28 22:07:16 systemd[1]: mariadb.service: Main process exited, code=killed, status=11/SEGV
 
Mar 28 22:07:16 systemd[1]: mariadb.service: Failed with result 'signal'.

Comment by Alain Bourgeois [ 2021-05-04 ]

I have a very similar issue with 10.5.9. 2 nodes out of 3 broken.

Comment by Nicolas [ 2021-09-16 ]

I had the same issue on a MariaDB Galera node and also the Galera Arbitrator.
Version: MariaDB 10.5.12 | Galera 26.4.9

To reproduce this issue, I did a vulnerability scan using GVM (Greenbone Vulnerability Management, ex OpenVAS) and the brute force category scan caused a DOS to the MariaDB service on port 4567.
The replication on port 4567 is done by certificate authentication.

Log:

terminate called after throwing an instance of 'boost::exception_detail::clone_impl<boost::exception_detail::error_info_injector<std::system_error> >'
 
  what():  remote_endpoint: Transport endpoint is not connected
 
210916  8:48:03 [ERROR] mysqld got signal 6 ;
 
This could be because you hit a bug. It is also possible that this binary
 
or one of the libraries it was linked against is corrupt, improperly built,
 
or misconfigured. This error can also be caused by malfunctioning hardware.
 
 
 
To report this bug, see https://mariadb.com/kb/en/reporting-bugs
 
 
 
We will try our best to scrape up some info that will hopefully help
 
diagnose the problem, but since we have already crashed, 
something is definitely wrong and this may fail.
 
 
 
Server version: 10.5.12-MariaDB-1:10.5.12+maria~buster
 
key_buffer_size=134217728
 
read_buffer_size=131072
 
max_used_connections=40
 
max_threads=202
 
thread_count=142
 
It is possible that mysqld could use up to 
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 575743 K  bytes of memory
 
Hope that's ok; if not, decrease some variables in the equation.
 
 
 
Thread pointer: 0x0
 
Attempting backtrace. You can use the following information to find out
 
where mysqld died. If you see no messages after this, something went
 
terribly wrong...
 
stack_bottom = 0x0 thread_stack 0x49000
 
??:0(my_print_stacktrace)[0x56199a545e4e]
 
??:0(handle_fatal_signal)[0x561999fd04f5]
 
??:0(__restore_rt)[0x7f0528fdd730]
 
??:0(gsignal)[0x7f0528b337bb]
 
??:0(abort)[0x7f0528b1e535]
 
/lib/x86_64-linux-gnu/libstdc++.so.6(+0x8c983)[0x7f0528ece983]
 
??:0(std::rethrow_exception(std::__exception_ptr::exception_ptr))[0x7f0528ed48c6]
 
??:0(std::terminate())[0x7f0528ed4901]
 
??:0(__cxa_throw)[0x7f0528ed4b34]
 
boost/throw_exception.hpp:69(void boost::throw_exception<std::system_error>(std::system_error const&))[0x7f0523174ef6]
 
impl/throw_error.ipp:49(asio::detail::do_throw_error(std::error_code const&, char const*) [clone .isra.265])[0x7f0523181224]
 
detail/throw_error.hpp:41(gu::AsioStreamReact::assign_addresses())[0x7f052318434b]
 
bits/shared_ptr_base.h:1019(gu::AsioAcceptorReact::accept_handler(std::shared_ptr<gu::AsioStreamReact> const&, std::shared_ptr<gu::AsioAcceptorHandler> const&, std::error_code const&))[0x7f0523184697]
 
detail/gcc_x86_fenced_block.hpp:80(asio::detail::reactive_socket_accept_op<asio::basic_socket<asio::ip::tcp, asio::stream_socket_service<asio::ip::tcp> >, asio::ip::tcp, boost::_bi::bind_t<void, boost::_mfi::mf3<void, gu::AsioAcceptorReact, std::shared_ptr<gu::AsioStreamReact> const&, std::shared_ptr<gu::AsioAcceptorHandler> const&, std::error_code const&>, boost::_bi::list4<boost::_bi::value<std::shared_ptr<gu::AsioAcceptorReact> >, boost::_bi::value<std::shared_ptr<gu::AsioStreamReact> >, boost::_bi::valimpl/epoll_reactor.ipp:653(asio::detail::epoll_reactor::descriptor_state::do_complete(asio::detail::task_io_service*, asio::detail::task_io_service_operation*, std::error_code const&, unsigned long))[0x7f052317c795]
 
impl/task_io_service.ipp:62(gu::AsioIoService::run())[0x7f052316daf8]
 
src/asio_protonet.cpp:104(gcomm::AsioProtonet::event_loop(gu::datetime::Period const&))[0x7f05230a1ab8]
 
src/gu_mutex.hpp:50(GCommConn::run())[0x7f05230873b4]
 
src/gcs_gcomm.cpp:305(run_fn)[0x7f0523087a49]
 
??:0(start_thread)[0x7f0528fd2fa3]
 
??:0(clone)[0x7f0528bf54cf]
 
The manual page at https://mariadb.com/kb/en/how-to-produce-a-full-stack-trace-for-mysqld/ contains
 
information that should help you find out what is causing the crash.
 
Writing a core file...
 
Working directory at /var/lib/mysql
 
Resource Limits:
 
Fatal signal 11 while backtracing
 
2021-09-16  8:48:16 0 [Note] WSREP: Loading provider /usr/lib/galera/libgalera_smm.so initial position: db2fdb34-801c-11eb-9f55-623bc6603e05:8205359
 
2021-09-16  8:48:16 0 [Note] WSREP: wsrep_load(): loading provider library '/usr/lib/galera/libgalera_smm.so'
 
2021-09-16  8:48:16 0 [Note] WSREP: wsrep_load(): Galera 26.4.9(r819f29cb) by Codership Oy <info@codership.com> loaded successfully.
 
2021-09-16  8:48:16 0 [Note] WSREP: CRC-32C: using 64-bit x86 acceleration.
 
2021-09-16  8:48:16 0 [Note] WSREP: SSL cipher list set to 'ECDHE-ECDSA-AES256-GCM-SHA384'
 
2021-09-16  8:48:16 0 [Note] WSREP: Found saved state: db2fdb34-801c-11eb-9f55-623bc6603e05:-1, safe_to_bootstrap: 0
 
2021-09-16  8:48:16 0 [Note] WSREP: GCache DEBUG: opened preamble:

For the persons that had this issue, did your port 4567 was exposed to the Internet or a potential scan could be done on it?

Comment by Stuart Green [ 2021-10-21 ]

Hello @Nicolas

I have inspected our scanning logs and have found that we did also run a vulnerabity scan at the time of crash. We use a cloud provider which has an OpenVAS backend scanning engine, it would have been an active scan, so 'crafted packets' of some sort. The scan we ran would have scanned all ports.

Comment by Nicolas [ 2021-10-22 ]

Hi @Stuart,

Thanks for your feedback, GVM/OpenVAS use the same engine so we can confirm this is a bug on MariaDB and should be fixed.
A brute-force should not be able to crash the entire cluster on port 4567 (we use TLS also so this shouldn't happen).

Comment by Stuart Green [ 2021-11-19 ]

Hi all,

I thought I would verify/update as we have upgraded a cluster to 10.4.22.

@Nicolas I have found slightly different conditions are required to what you suggested

If I scan port 3306 I get denial messages from OpenVAS trying to logon, but the node stays up
If I scan port 4567 I get a lot of internal error messages, but the node stays up.
If I scan port 3306 and port 4567 in the same scan, the node crashes.

It seems that it is required to scan both of these ports at the same time to get the node to crash.

It would make sense that using OpenVAS defaults as a preset would crash the node as sequentially 4567 would follow 3306 as there would be no ports found inbetween.

Output follows:

OpenVAS Scan Port 3306 targeted.

Nov 19 16:55:03 NODE-01 mysqld[78300]: 2021-11-19 16:55:03 72 [Warning] Aborted connection 72 to db: 'unconnected' user: 'unauthenticated' host: 'the.scanner.ip.address' (This connection closed normally without authentication)
 
Nov 19 16:55:03 NODE-01 mysqld[78300]: 2021-11-19 16:55:03 73 [Warning] Aborted connection 73 to db: 'unconnected' user: 'unauthenticated' host: 'the.scanner.ip.address' (This connection closed normally without authentication)
 
Nov 19 16:55:03 NODE-01 mysqld[78300]: 2021-11-19 16:55:03 74 [Warning] Aborted connection 74 to db: 'unconnected' user: 'unauthenticated' host: 'the.scanner.ip.address' (This connection closed normally without authentication)
 
Nov 19 16:55:03 NODE-01 mysqld[78300]: 2021-11-19 16:55:03 75 [Warning] Aborted connection 75 to db: 'unconnected' user: 'unauthenticated' host: 'the.scanner.ip.address' (This connection closed normally without authentication)
 
Nov 19 16:55:03 NODE-01 mysqld[78300]: 2021-11-19 16:55:03 76 [Warning] Aborted connection 76 to db: 'unconnected' user: 'unauthenticated' host: 'the.scanner.ip.address' (This connection closed normally without authentication)
 
Nov 19 16:55:04 NODE-01 mysqld[78300]: 2021-11-19 16:55:04 77 [Warning] Aborted connection 77 to db: 'unconnected' user: 'unauthenticated' host: 'the.scanner.ip.address' (This connection closed normally without authentication)
 
Nov 19 16:55:04 NODE-01 mysqld[78300]: 2021-11-19 16:55:04 78 [Warning] Aborted connection 78 to db: 'unconnected' user: 'unauthenticated' host: 'the.scanner.ip.address' (This connection closed normally without authentication)
 
Nov 19 16:55:04 NODE-01 mysqld[78300]: 2021-11-19 16:55:04 79 [Warning] Aborted connection 79 to db: 'unconnected' user: 'unauthenticated' host: 'the.scanner.ip.address' (This connection closed normally without authentication)
 
Nov 19 16:55:04 NODE-01 mysqld[78300]: 2021-11-19 16:55:04 80 [Warning] Aborted connection 80 to db: 'unconnected' user: 'unauthenticated' host: 'the.scanner.ip.address' (This connection closed normally without authentication)
 
Nov 19 16:55:04 NODE-01 mysqld[78300]: 2021-11-19 16:55:04 81 [Warning] Aborted connection 81 to db: 'unconnected' user: 'unauthenticated' host: 'the.scanner.ip.address' (This connection closed normally without authentication)
 
Nov 19 16:55:04 NODE-01 mysqld[78300]: 2021-11-19 16:55:04 82 [Warning] Aborted connection 82 to db: 'unconnected' user: 'unauthenticated' host: 'the.scanner.ip.address' (This connection closed normally without authentication)
 
Nov 19 16:55:04 NODE-01 mysqld[78300]: 2021-11-19 16:55:04 83 [Warning] Aborted connection 83 to db: 'unconnected' user: 'unauthenticated' host: 'the.scanner.ip.address' (This connection closed normally without authentication)
 
Nov 19 16:55:04 NODE-01 mysqld[78300]: 2021-11-19 16:55:04 84 [Warning] Aborted connection 84 to db: 'unconnected' user: 'unauthenticated' host: 'the.scanner.ip.address' (This connection closed normally without authentication)
 
Nov 19 16:55:45 NODE-01 mysqld[78300]: 2021-11-19 16:55:45 87 [Warning] Access denied for user 'htr'@'the.scanner.ip.address' (using password: YES)
 
Nov 19 16:55:45 NODE-01 mysqld[78300]: 2021-11-19 16:55:45 88 [Warning] Access denied for user 'root'@'the.scanner.ip.address' (using password: YES)
 
Nov 19 16:55:45 NODE-01 mysqld[78300]: 2021-11-19 16:55:45 91 [Warning] Access denied for user 'root'@'the.scanner.ip.address' (using password: YES)
 
Nov 19 16:55:45 NODE-01 mysqld[78300]: 2021-11-19 16:55:45 92 [Warning] Access denied for user 'root'@'the.scanner.ip.address' (using password: YES)
 
Nov 19 16:55:45 NODE-01 mysqld[78300]: 2021-11-19 16:55:45 93 [Warning] Access denied for user 'root'@'the.scanner.ip.address' (using password: YES)
 
Nov 19 16:55:45 NODE-01 mysqld[78300]: 2021-11-19 16:55:45 94 [Warning] Access denied for user 'root'@'the.scanner.ip.address' (using password: YES)
 
Nov 19 16:55:45 NODE-01 mysqld[78300]: 2021-11-19 16:55:45 95 [Warning] Access denied for user 'root'@'the.scanner.ip.address' (using password: YES)
 
Nov 19 16:55:45 NODE-01 mysqld[78300]: 2021-11-19 16:55:45 96 [Warning] Access denied for user 'root'@'the.scanner.ip.address' (using password: YES)
 
Nov 19 16:55:45 NODE-01 mysqld[78300]: 2021-11-19 16:55:45 89 [Warning] Aborted connection 89 to db: 'unconnected' user: 'unauthenticated' host: 'the.scanner.ip.address' (This connection closed normally without authentication)
 
Nov 19 16:55:46 NODE-01 mysqld[78300]: 2021-11-19 16:55:46 97 [Warning] Access denied for user 'root'@'the.scanner.ip.address' (using password: YES)
 
Nov 19 16:55:46 NODE-01 mysqld[78300]: 2021-11-19 16:55:46 98 [Warning] Access denied for user 'root'@'the.scanner.ip.address' (using password: YES)
 
Nov 19 16:55:46 NODE-01 mysqld[78300]: 2021-11-19 16:55:46 99 [Warning] Access denied for user 'root'@'the.scanner.ip.address' (using password: YES)
 
Nov 19 16:55:46 NODE-01 mysqld[78300]: 2021-11-19 16:55:46 100 [Warning] Access denied for user 'root'@'the.scanner.ip.address' (using password: YES)
 
Nov 19 16:55:46 NODE-01 mysqld[78300]: 2021-11-19 16:55:46 101 [Warning] Access denied for user 'root'@'the.scanner.ip.address' (using password: NO)
 
Nov 19 16:55:47 NODE-01 mysqld[78300]: 2021-11-19 16:55:47 90 [Warning] Aborted connection 90 to db: 'unconnected' user: 'unauthenticated' host: 'the.scanner.ip.address' (This connection closed normally without authentication)
 
Nov 19 16:55:47 NODE-01 mysqld[78300]: 2021-11-19 16:55:47 102 [Warning] Aborted connection 102 to db: 'unconnected' user: 'unauthenticated' host: 'the.scanner.ip.address' (This connection closed normally without authentication)

OpenVAS Scan Port 4567 targeted.

Nov 19 17:00:41 NODE-01 mysqld[78300]: 2021-11-19 17:00:41 0 [Warning] WSREP: unserialize error invalid protocol version 2: 71 (Protocol error)
 
Nov 19 17:00:41 NODE-01 mysqld[78300]: at /home/buildbot/buildbot/build/gcomm/src/gcomm/datagram.hpp:unserialize():133
 
Nov 19 17:00:42 NODE-01 mysqld[78300]: 2021-11-19 17:00:42 0 [Warning] WSREP: unserialize error invalid protocol version 7: 71 (Protocol error)
 
Nov 19 17:00:42 NODE-01 mysqld[78300]: at /home/buildbot/buildbot/build/gcomm/src/gcomm/datagram.hpp:unserialize():133
 
Nov 19 17:00:45 NODE-01 mysqld[78300]: 2021-11-19 17:00:45 0 [Note] WSREP: (1132bd81-a3fd, 'tcp://0.0.0.0:4567') connection to peer 00000000-0000 with addr  timed out, no messages seen in PT3S, socket stats: rtt: 2976 rttvar: 1171 rto: 203000 lost: 0 last_data_recv: 110334802 cwnd: 10 last_queued_since: 3065247393 last_delivered_since: 110634801991969 send_queue_length: 0 send_queue_bytes: 0 segment: 0 messages: 0
 
Nov 19 17:00:45 NODE-01 mysqld[78300]: 2021-11-19 17:00:45 0 [Warning] WSREP: unserialize error invalid protocol version 2: 71 (Protocol error)
 
Nov 19 17:00:45 NODE-01 mysqld[78300]: at /home/buildbot/buildbot/build/gcomm/src/gcomm/datagram.hpp:unserialize():133
 
Nov 19 17:00:45 NODE-01 mysqld[78300]: 2021-11-19 17:00:45 0 [Warning] WSREP: unserialize error invalid protocol version 2: 71 (Protocol error)
 
Nov 19 17:00:45 NODE-01 mysqld[78300]: at /home/buildbot/buildbot/build/gcomm/src/gcomm/datagram.hpp:unserialize():133
 
Nov 19 17:00:45 NODE-01 mysqld[78300]: 2021-11-19 17:00:45 0 [Warning] WSREP: unserialize error invalid flags 11: 71 (Protocol error)
 
Nov 19 17:00:45 NODE-01 mysqld[78300]: at /home/buildbot/buildbot/build/gcomm/src/gcomm/datagram.hpp:unserialize():126
 
Nov 19 17:00:47 NODE-01 mysqld[78300]: 2021-11-19 17:00:47 0 [Warning] WSREP: unserialize error invalid protocol version 6: 71 (Protocol error)
 
Nov 19 17:00:47 NODE-01 mysqld[78300]: at /home/buildbot/buildbot/build/gcomm/src/gcomm/datagram.hpp:unserialize():133
 
Nov 19 17:00:47 NODE-01 mysqld[78300]: 2021-11-19 17:00:47 0 [Warning] WSREP: unserialize error invalid protocol version 4: 71 (Protocol error)
 
Nov 19 17:00:47 NODE-01 mysqld[78300]: at /home/buildbot/buildbot/build/gcomm/src/gcomm/datagram.hpp:unserialize():133
 
Nov 19 17:00:47 NODE-01 mysqld[78300]: 2021-11-19 17:00:47 0 [Warning] WSREP: unserialize error invalid protocol version 2: 71 (Protocol error)
 
Nov 19 17:00:47 NODE-01 mysqld[78300]: at /home/buildbot/buildbot/build/gcomm/src/gcomm/datagram.hpp:unserialize():133
 
Nov 19 17:00:47 NODE-01 mysqld[78300]: 2021-11-19 17:00:47 0 [Warning] WSREP: unserialize error invalid protocol version 6: 71 (Protocol error)
 
Nov 19 17:00:47 NODE-01 mysqld[78300]: at /home/buildbot/buildbot/build/gcomm/src/gcomm/datagram.hpp:unserialize():133
 
Nov 19 17:00:47 NODE-01 mysqld[78300]: 2021-11-19 17:00:47 0 [Warning] WSREP: unserialize error invalid protocol version 4: 71 (Protocol error)
 
Nov 19 17:00:47 NODE-01 mysqld[78300]: at /home/buildbot/buildbot/build/gcomm/src/gcomm/datagram.hpp:unserialize():133
 
Nov 19 17:01:04 NODE-01 mysqld[78300]: 2021-11-19 17:01:04 0 [Warning] WSREP: checksum failed, hdr: len=790 has_crc32=1 has_crc32c=1 crc32=50332065
 
Nov 19 17:01:04 NODE-01 mysqld[78300]: 2021-11-19 17:01:04 0 [Warning] WSREP: checksum failed, hdr: len=2 has_crc32=0 has_crc32c=0 crc32=16780288
 
Nov 19 17:01:13 NODE-01 mysqld[78300]: 2021-11-19 17:01:13 0 [Warning] WSREP: unserialize error invalid protocol version 4: 71 (Protocol error)
 
Nov 19 17:01:13 NODE-01 mysqld[78300]: at /home/buildbot/buildbot/build/gcomm/src/gcomm/datagram.hpp:unserialize():133
 
Nov 19 17:01:13 NODE-01 mysqld[78300]: 2021-11-19 17:01:13 0 [Warning] WSREP: unserialize error invalid protocol version 2: 71 (Protocol error)
 
Nov 19 17:01:13 NODE-01 mysqld[78300]: at /home/buildbot/buildbot/build/gcomm/src/gcomm/datagram.hpp:unserialize():133
 
Nov 19 17:01:14 NODE-01 mysqld[78300]: 2021-11-19 17:01:14 0 [Warning] WSREP: unserialize error invalid protocol version 5: 71 (Protocol error)
 
Nov 19 17:01:14 NODE-01 mysqld[78300]: at /home/buildbot/buildbot/build/gcomm/src/gcomm/datagram.hpp:unserialize():133
 
Nov 19 17:01:14 NODE-01 mysqld[78300]: 2021-11-19 17:01:14 0 [Warning] WSREP: unserialize error invalid protocol version 7: 71 (Protocol error)
 
Nov 19 17:01:14 NODE-01 mysqld[78300]: at /home/buildbot/buildbot/build/gcomm/src/gcomm/datagram.hpp:unserialize():133
 
Nov 19 17:01:14 NODE-01 mysqld[78300]: 2021-11-19 17:01:14 0 [Warning] WSREP: unserialize error invalid protocol version 5: 71 (Protocol error)
 
Nov 19 17:01:14 NODE-01 mysqld[78300]: at /home/buildbot/buildbot/build/gcomm/src/gcomm/datagram.hpp:unserialize():133
 
Nov 19 17:01:14 NODE-01 mysqld[78300]: 2021-11-19 17:01:14 0 [Warning] WSREP: checksum failed, hdr: len=0 has_crc32=0 has_crc32c=0 crc32=721421568
 
Nov 19 17:01:14 NODE-01 mysqld[78300]: 2021-11-19 17:01:14 0 [Warning] WSREP: unserialize error invalid protocol version 4: 71 (Protocol error)
 
Nov 19 17:01:14 NODE-01 mysqld[78300]: at /home/buildbot/buildbot/build/gcomm/src/gcomm/datagram.hpp:unserialize():133
 
Nov 19 17:01:26 NODE-01 mysqld[78300]: 2021-11-19 17:01:26 0 [Warning] WSREP: unserialize error invalid protocol version 5: 71 (Protocol error)
 
Nov 19 17:01:26 NODE-01 mysqld[78300]: at /home/buildbot/buildbot/build/gcomm/src/gcomm/datagram.hpp:unserialize():133
 
Nov 19 17:01:27 NODE-01 mysqld[78300]: 2021-11-19 17:01:27 0 [Warning] WSREP: unserialize error invalid protocol version 1: 71 (Protocol error)
 
Nov 19 17:01:27 NODE-01 mysqld[78300]: at /home/buildbot/buildbot/build/gcomm/src/gcomm/datagram.hpp:unserialize():133
 
Nov 19 17:01:32 NODE-01 mysqld[78300]: 2021-11-19 17:01:32 0 [Warning] WSREP: unserialize error invalid protocol version 5: 71 (Protocol error)
 
Nov 19 17:01:32 NODE-01 mysqld[78300]: at /home/buildbot/buildbot/build/gcomm/src/gcomm/datagram.hpp:unserialize():133
 
Nov 19 17:01:32 NODE-01 mysqld[78300]: 2021-11-19 17:01:32 0 [Warning] WSREP: unserialize error invalid flags 6: 71 (Protocol error)
 
Nov 19 17:01:32 NODE-01 mysqld[78300]: at /home/buildbot/buildbot/build/gcomm/src/gcomm/datagram.hpp:unserialize():126
 
Nov 19 17:01:48 NODE-01 mysqld[78300]: 2021-11-19 17:01:48 0 [Warning] WSREP: unserialize error invalid protocol version 6: 71 (Protocol error)
 
Nov 19 17:01:48 NODE-01 mysqld[78300]: at /home/buildbot/buildbot/build/gcomm/src/gcomm/datagram.hpp:unserialize():133
 
Nov 19 17:01:48 NODE-01 mysqld[78300]: 2021-11-19 17:01:48 0 [Warning] WSREP: unserialize error invalid protocol version 6: 71 (Protocol error)
 
Nov 19 17:01:48 NODE-01 mysqld[78300]: at /home/buildbot/buildbot/build/gcomm/src/gcomm/datagram.hpp:unserialize():133
 
Nov 19 17:01:48 NODE-01 mysqld[78300]: 2021-11-19 17:01:48 0 [Warning] WSREP: unserialize error invalid protocol version 15: 71 (Protocol error)
 
Nov 19 17:01:48 NODE-01 mysqld[78300]: at /home/buildbot/buildbot/build/gcomm/src/gcomm/datagram.hpp:unserialize():133
 
Nov 19 17:01:48 NODE-01 mysqld[78300]: 2021-11-19 17:01:48 0 [Warning] WSREP: checksum failed, hdr: len=1 has_crc32=0 has_crc32c=0 crc32=1
 
Nov 19 17:01:48 NODE-01 mysqld[78300]: 2021-11-19 17:01:48 0 [Warning] WSREP: checksum failed, hdr: len=0 has_crc32=1 has_crc32c=0 crc32=318767104
 
Nov 19 17:01:48 NODE-01 mysqld[78300]: 2021-11-19 17:01:48 0 [Warning] WSREP: checksum failed, hdr: len=0 has_crc32=1 has_crc32c=0 crc32=318767104
 
Nov 19 17:01:48 NODE-01 mysqld[78300]: 2021-11-19 17:01:48 0 [Warning] WSREP: unserialize error invalid protocol version 6: 71 (Protocol error)
 
Nov 19 17:01:48 NODE-01 mysqld[78300]: at /home/buildbot/buildbot/build/gcomm/src/gcomm/datagram.hpp:unserialize():133
 
Nov 19 17:01:48 NODE-01 mysqld[78300]: 2021-11-19 17:01:48 0 [Warning] WSREP: unserialize error invalid protocol version 5: 71 (Protocol error)
 
Nov 19 17:01:48 NODE-01 mysqld[78300]: at /home/buildbot/buildbot/build/gcomm/src/gcomm/datagram.hpp:unserialize():133
 
Nov 19 17:01:50 NODE-01 mysqld[78300]: 2021-11-19 17:01:50 0 [Warning] WSREP: unserialize error invalid flags 6: 71 (Protocol error)
 
Nov 19 17:01:50 NODE-01 mysqld[78300]: at /home/buildbot/buildbot/build/gcomm/src/gcomm/datagram.hpp:unserialize():126
 
Nov 19 17:01:50 NODE-01 mysqld[78300]: 2021-11-19 17:01:50 0 [Warning] WSREP: unserialize error invalid protocol version 5: 71 (Protocol error)
 
Nov 19 17:01:50 NODE-01 mysqld[78300]: at /home/buildbot/buildbot/build/gcomm/src/gcomm/datagram.hpp:unserialize():133
 
Nov 19 17:01:50 NODE-01 mysqld[78300]: 2021-11-19 17:01:50 0 [Warning] WSREP: unserialize error invalid protocol version 4: 71 (Protocol error)
 
Nov 19 17:01:50 NODE-01 mysqld[78300]: at /home/buildbot/buildbot/build/gcomm/src/gcomm/datagram.hpp:unserialize():133
 
Nov 19 17:01:51 NODE-01 mysqld[78300]: 2021-11-19 17:01:51 0 [Warning] WSREP: unserialize error invalid flags 4: 71 (Protocol error)
 
Nov 19 17:01:51 NODE-01 mysqld[78300]: at /home/buildbot/buildbot/build/gcomm/src/gcomm/datagram.hpp:unserialize():126
 
Nov 19 17:01:51 NODE-01 mysqld[78300]: 2021-11-19 17:01:51 0 [Warning] WSREP: unserialize error invalid protocol version 5: 71 (Protocol error)
 
Nov 19 17:01:51 NODE-01 mysqld[78300]: at /home/buildbot/buildbot/build/gcomm/src/gcomm/datagram.hpp:unserialize():133
 
Nov 19 17:01:51 NODE-01 mysqld[78300]: 2021-11-19 17:01:51 0 [Warning] WSREP: unserialize error invalid protocol version 12: 71 (Protocol error)
 
Nov 19 17:01:51 NODE-01 mysqld[78300]: at /home/buildbot/buildbot/build/gcomm/src/gcomm/datagram.hpp:unserialize():133
 
Nov 19 17:01:52 NODE-01 mysqld[78300]: 2021-11-19 17:01:52 0 [Warning] WSREP: unserialize error invalid protocol version 1: 71 (Protocol error)
 
Nov 19 17:01:52 NODE-01 mysqld[78300]: at /home/buildbot/buildbot/build/gcomm/src/gcomm/datagram.hpp:unserialize():133
 
Nov 19 17:01:52 NODE-01 mysqld[78300]: 2021-11-19 17:01:52 0 [Warning] WSREP: unserialize error invalid protocol version 5: 71 (Protocol error)
 
Nov 19 17:01:52 NODE-01 mysqld[78300]: at /home/buildbot/buildbot/build/gcomm/src/gcomm/datagram.hpp:unserialize():133
 
Nov 19 17:01:59 NODE-01 mysqld[78300]: 2021-11-19 17:01:59 0 [Warning] WSREP: unserialize error invalid protocol version 5: 71 (Protocol error)
 
Nov 19 17:01:59 NODE-01 mysqld[78300]: at /home/buildbot/buildbot/build/gcomm/src/gcomm/datagram.hpp:unserialize():133
 
Nov 19 17:01:59 NODE-01 mysqld[78300]: 2021-11-19 17:01:59 0 [Warning] WSREP: unserialize error invalid protocol version 5: 71 (Protocol error)
 
Nov 19 17:01:59 NODE-01 mysqld[78300]: at /home/buildbot/buildbot/build/gcomm/src/gcomm/datagram.hpp:unserialize():133
 
Nov 19 17:02:00 NODE-01 mysqld[78300]: 2021-11-19 17:02:00 0 [Warning] WSREP: unserialize error invalid protocol version 5: 71 (Protocol error)
 
Nov 19 17:02:00 NODE-01 mysqld[78300]: at /home/buildbot/buildbot/build/gcomm/src/gcomm/datagram.hpp:unserialize():133
 
Nov 19 17:02:00 NODE-01 mysqld[78300]: 2021-11-19 17:02:00 0 [Warning] WSREP: unserialize error invalid protocol version 5: 71 (Protocol error)
 
Nov 19 17:02:00 NODE-01 mysqld[78300]: at /home/buildbot/buildbot/build/gcomm/src/gcomm/datagram.hpp:unserialize():133
 
Nov 19 17:02:03 NODE-01 mysqld[78300]: 2021-11-19 17:02:03 0 [Note] WSREP: (1132bd81-a3fd, 'tcp://0.0.0.0:4567') connection to peer 00000000-0000 with addr  timed out, no messages seen in PT3S, socket stats: rtt: 2949 rttvar: 1226 rto: 203000 lost: 0 last_data_recv: 110413322 cwnd: 10 last_queued_since: 3024963641 last_delivered_since: 110713321447043 send_queue_length: 0 send_queue_bytes: 0 segment: 0 messages: 0
 
Nov 19 17:02:03 NODE-01 mysqld[78300]: 2021-11-19 17:02:03 0 [Warning] WSREP: unserialize error invalid protocol version 2: 71 (Protocol error)
 
Nov 19 17:02:03 NODE-01 mysqld[78300]: at /home/buildbot/buildbot/build/gcomm/src/gcomm/datagram.hpp:unserialize():133
 
Nov 19 17:02:07 NODE-01 mysqld[78300]: 2021-11-19 17:02:07 0 [Note] WSREP: (1132bd81-a3fd, 'tcp://0.0.0.0:4567') connection to peer 00000000-0000 with addr  timed out, no messages seen in PT3S, socket stats: rtt: 3250 rttvar: 1625 rto: 204000 lost: 0 last_data_recv: 3431 cwnd: 10 last_queued_since: 3433237748 last_delivered_since: 110716822135452 send_queue_length: 0 send_queue_bytes: 0 segment: 0 messages: 0
 
Nov 19 17:02:07 NODE-01 mysqld[78300]: 2021-11-19 17:02:07 0 [Warning] WSREP: unserialize error invalid protocol version 4: 71 (Protocol error)
 
Nov 19 17:02:07 NODE-01 mysqld[78300]: at /home/buildbot/buildbot/build/gcomm/src/gcomm/datagram.hpp:unserialize():133
 
Nov 19 17:02:07 NODE-01 mysqld[78300]: 2021-11-19 17:02:07 0 [Warning] WSREP: unserialize error invalid protocol version 4: 71 (Protocol error)
 
Nov 19 17:02:07 NODE-01 mysqld[78300]: at /home/buildbot/buildbot/build/gcomm/src/gcomm/datagram.hpp:unserialize():133
 
Nov 19 17:02:07 NODE-01 mysqld[78300]: 2021-11-19 17:02:07 0 [Warning] WSREP: unserialize error invalid protocol version 2: 71 (Protocol error)
 
Nov 19 17:02:07 NODE-01 mysqld[78300]: at /home/buildbot/buildbot/build/gcomm/src/gcomm/datagram.hpp:unserialize():133
 
Nov 19 17:02:07 NODE-01 mysqld[78300]: 2021-11-19 17:02:07 0 [Warning] WSREP: unserialize error invalid flags 6: 71 (Protocol error)
 
Nov 19 17:02:07 NODE-01 mysqld[78300]: at /home/buildbot/buildbot/build/gcomm/src/gcomm/datagram.hpp:unserialize():126
 
Nov 19 17:02:09 NODE-01 mysqld[78300]: 2021-11-19 17:02:09 0 [Warning] WSREP: unserialize error invalid protocol version 3: 71 (Protocol error)
 
Nov 19 17:02:09 NODE-01 mysqld[78300]: at /home/buildbot/buildbot/build/gcomm/src/gcomm/datagram.hpp:unserialize():133
 
Nov 19 17:02:10 NODE-01 mysqld[78300]: 2021-11-19 17:02:10 0 [Note] WSREP: (1132bd81-a3fd, 'tcp://0.0.0.0:4567') connection to peer 00000000-0000 with addr  timed out, no messages seen in PT3S, socket stats: rtt: 3258 rttvar: 1642 rto: 204000 lost: 0 last_data_recv: 3388 cwnd: 10 last_queued_since: 3391322802 last_delivered_since: 110720322902424 send_queue_length: 0 send_queue_bytes: 0 segment: 0 messages: 0
 
Nov 19 17:02:14 NODE-01 mysqld[78300]: 2021-11-19 17:02:14 0 [Note] WSREP: (1132bd81-a3fd, 'tcp://0.0.0.0:4567') connection to peer 00000000-0000 with addr  timed out, no messages seen in PT3S, socket stats: rtt: 2412 rttvar: 1574 rto: 203000 lost: 0 last_data_recv: 3489 cwnd: 10 last_queued_since: 3491386280 last_delivered_since: 110723823637520 send_queue_length: 0 send_queue_bytes: 0 segment: 0 messages: 0
 
Nov 19 17:02:17 NODE-01 mysqld[78300]: 2021-11-19 17:02:17 0 [Note] WSREP: (1132bd81-a3fd, 'tcp://0.0.0.0:4567') connection to peer 00000000-0000 with addr  timed out, no messages seen in PT3S, socket stats: rtt: 46870 rttvar: 17884 rto: 247000 lost: 0 last_data_recv: 3447 cwnd: 10 last_queued_since: 3448829822 last_delivered_since: 110727324411113 send_queue_length: 0 send_queue_bytes: 0 segment: 0 messages: 0
 
Nov 19 17:02:17 NODE-01 mysqld[78300]: 2021-11-19 17:02:17 0 [Warning] WSREP: unserialize error invalid protocol version 2: 71 (Protocol error)
 
Nov 19 17:02:17 NODE-01 mysqld[78300]: at /home/buildbot/buildbot/build/gcomm/src/gcomm/datagram.hpp:unserialize():133
 
Nov 19 17:02:21 NODE-01 mysqld[78300]: 2021-11-19 17:02:21 0 [Note] WSREP: (1132bd81-a3fd, 'tcp://0.0.0.0:4567') connection to peer 00000000-0000 with addr  timed out, no messages seen in PT3S, socket stats: rtt: 2380 rttvar: 1510 rto: 203000 lost: 0 last_data_recv: 3387 cwnd: 10 last_queued_since: 3389095114 last_delivered_since: 110730825168063 send_queue_length: 0 send_queue_bytes: 0 segment: 0 messages: 0
 
Nov 19 17:02:21 NODE-01 mysqld[78300]: 2021-11-19 17:02:21 0 [Warning] WSREP: unserialize error invalid protocol version 7: 71 (Protocol error)
 
Nov 19 17:02:21 NODE-01 mysqld[78300]: at /home/buildbot/buildbot/build/gcomm/src/gcomm/datagram.hpp:unserialize():133
 
Nov 19 17:02:21 NODE-01 mysqld[78300]: 2021-11-19 17:02:21 0 [Warning] WSREP: unserialize error invalid protocol version 10: 71 (Protocol error)
 
Nov 19 17:02:21 NODE-01 mysqld[78300]: at /home/buildbot/buildbot/build/gcomm/src/gcomm/datagram.hpp:unserialize():133
 
Nov 19 17:02:24 NODE-01 mysqld[78300]: 2021-11-19 17:02:24 0 [Note] WSREP: (1132bd81-a3fd, 'tcp://0.0.0.0:4567') connection to peer 00000000-0000 with addr  timed out, no messages seen in PT3S, socket stats: rtt: 2362 rttvar: 1475 rto: 203000 lost: 0 last_data_recv: 3338 cwnd: 10 last_queued_since: 3341175489 last_delivered_since: 110734325955033 send_queue_length: 0 send_queue_bytes: 0 segment: 0 messages: 0
 
Nov 19 17:02:24 NODE-01 mysqld[78300]: 2021-11-19 17:02:24 0 [Warning] WSREP: unserialize error invalid protocol version 2: 71 (Protocol error)
 
Nov 19 17:02:24 NODE-01 mysqld[78300]: at /home/buildbot/buildbot/build/gcomm/src/gcomm/datagram.hpp:unserialize():133
 
Nov 19 17:02:24 NODE-01 mysqld[78300]: 2021-11-19 17:02:24 0 [Warning] WSREP: unserialize error invalid protocol version 6: 71 (Protocol error)
 
Nov 19 17:02:24 NODE-01 mysqld[78300]: at /home/buildbot/buildbot/build/gcomm/src/gcomm/datagram.hpp:unserialize():133
 
Nov 19 17:02:28 NODE-01 mysqld[78300]: 2021-11-19 17:02:28 0 [Note] WSREP: (1132bd81-a3fd, 'tcp://0.0.0.0:4567') connection to peer 00000000-0000 with addr  timed out, no messages seen in PT3S, socket stats: rtt: 2357 rttvar: 1464 rto: 203000 lost: 0 last_data_recv: 3332 cwnd: 10 last_queued_since: 3334558479 last_delivered_since: 110737826645820 send_queue_length: 0 send_queue_bytes: 0 segment: 0 messages: 0
 
Nov 19 17:02:31 NODE-01 mysqld[78300]: 2021-11-19 17:02:31 0 [Note] WSREP: (1132bd81-a3fd, 'tcp://0.0.0.0:4567') connection to peer 00000000-0000 with addr  timed out, no messages seen in PT3S, socket stats: rtt: 46817 rttvar: 17990 rto: 247000 lost: 0 last_data_recv: 3446 cwnd: 10 last_queued_since: 3447757772 last_delivered_since: 110741327415873 send_queue_length: 0 send_queue_bytes: 0 segment: 0 messages: 0
 
Nov 19 17:02:31 NODE-01 mysqld[78300]: 2021-11-19 17:02:31 0 [Warning] WSREP: unserialize error invalid protocol version 4: 71 (Protocol error)
 
Nov 19 17:02:31 NODE-01 mysqld[78300]: at /home/buildbot/buildbot/build/gcomm/src/gcomm/datagram.hpp:unserialize():133
 
Nov 19 17:02:31 NODE-01 mysqld[78300]: 2021-11-19 17:02:31 0 [Warning] WSREP: unserialize error invalid protocol version 5: 71 (Protocol error)
 
Nov 19 17:02:31 NODE-01 mysqld[78300]: at /home/buildbot/buildbot/build/gcomm/src/gcomm/datagram.hpp:unserialize():133
 
Nov 19 17:02:31 NODE-01 mysqld[78300]: 2021-11-19 17:02:31 0 [Warning] WSREP: unserialize error invalid flags 11: 71 (Protocol error)
 
Nov 19 17:02:31 NODE-01 mysqld[78300]: at /home/buildbot/buildbot/build/gcomm/src/gcomm/datagram.hpp:unserialize():126
 
Nov 19 17:02:31 NODE-01 mysqld[78300]: 2021-11-19 17:02:31 0 [Warning] WSREP: unserialize error invalid protocol version 5: 71 (Protocol error)
 
Nov 19 17:02:31 NODE-01 mysqld[78300]: at /home/buildbot/buildbot/build/gcomm/src/gcomm/datagram.hpp:unserialize():133
 
Nov 19 17:02:35 NODE-01 mysqld[78300]: 2021-11-19 17:02:35 0 [Note] WSREP: (1132bd81-a3fd, 'tcp://0.0.0.0:4567') connection to peer 00000000-0000 with addr  timed out, no messages seen in PT3S, socket stats: rtt: 3234 rttvar: 1594 rto: 204000 lost: 0 last_data_recv: 3229 cwnd: 10 last_queued_since: 3230754498 last_delivered_since: 110744828097513 send_queue_length: 0 send_queue_bytes: 0 segment: 0 messages: 0
 
Nov 19 17:02:38 NODE-01 mysqld[78300]: 2021-11-19 17:02:38 0 [Note] WSREP: (1132bd81-a3fd, 'tcp://0.0.0.0:4567') connection to peer 00000000-0000 with addr  timed out, no messages seen in PT3S, socket stats: rtt: 45089 rttvar: 17054 rto: 246000 lost: 0 last_data_recv: 3448 cwnd: 10 last_queued_since: 3449277638 last_delivered_since: 110748328836025 send_queue_length: 0 send_queue_bytes: 0 segment: 0 messages: 0
 
Nov 19 17:02:38 NODE-01 mysqld[78300]: 2021-11-19 17:02:38 0 [Warning] WSREP: unserialize error invalid protocol version 15: 71 (Protocol error)
 
Nov 19 17:02:38 NODE-01 mysqld[78300]: at /home/buildbot/buildbot/build/gcomm/src/gcomm/datagram.hpp:unserialize():133
 
Nov 19 17:02:42 NODE-01 mysqld[78300]: 2021-11-19 17:02:42 0 [Note] WSREP: (1132bd81-a3fd, 'tcp://0.0.0.0:4567') connection to peer 00000000-0000 with addr  timed out, no messages seen in PT3S, socket stats: rtt: 42631 rttvar: 24592 rto: 243000 lost: 0 last_data_recv: 3351 cwnd: 10 last_queued_since: 3353602435 last_delivered_since: 110751829586149 send_queue_length: 0 send_queue_bytes: 0 segment: 0 messages: 0
 
Nov 19 17:02:42 NODE-01 mysqld[78300]: 2021-11-19 17:02:42 0 [Warning] WSREP: unserialize error invalid protocol version 3: 71 (Protocol error)
 
Nov 19 17:02:42 NODE-01 mysqld[78300]: at /home/buildbot/buildbot/build/gcomm/src/gcomm/datagram.hpp:unserialize():133
 
Nov 19 17:02:45 NODE-01 mysqld[78300]: 2021-11-19 17:02:45 0 [Note] WSREP: (1132bd81-a3fd, 'tcp://0.0.0.0:4567') connection to peer 00000000-0000 with addr  timed out, no messages seen in PT3S, socket stats: rtt: 42619 rttvar: 24595 rto: 243000 lost: 0 last_data_recv: 3352 cwnd: 10 last_queued_since: 3354069244 last_delivered_since: 110755330277426 send_queue_length: 0 send_queue_bytes: 0 segment: 0 messages: 0
 
Nov 19 17:02:45 NODE-01 mysqld[78300]: 2021-11-19 17:02:45 0 [Warning] WSREP: unserialize error invalid protocol version 5: 71 (Protocol error)
 
Nov 19 17:02:45 NODE-01 mysqld[78300]: at /home/buildbot/buildbot/build/gcomm/src/gcomm/datagram.hpp:unserialize():133
 
Nov 19 17:02:45 NODE-01 mysqld[78300]: 2021-11-19 17:02:45 0 [Warning] WSREP: unserialize error invalid protocol version 2: 71 (Protocol error)
 
Nov 19 17:02:45 NODE-01 mysqld[78300]: at /home/buildbot/buildbot/build/gcomm/src/gcomm/datagram.hpp:unserialize():133
 
Nov 19 17:02:48 NODE-01 mysqld[78300]: 2021-11-19 17:02:48 0 [Warning] WSREP: unserialize error invalid protocol version 5: 71 (Protocol error)
 
Nov 19 17:02:48 NODE-01 mysqld[78300]: at /home/buildbot/buildbot/build/gcomm/src/gcomm/datagram.hpp:unserialize():133
 
Nov 19 17:03:21 NODE-01 mysqld[78300]: 2021-11-19 17:03:21 0 [Warning] WSREP: checksum failed, hdr: len=0 has_crc32=0 has_crc32c=0 crc32=16777216
 
Nov 19 17:03:21 NODE-01 mysqld[78300]: 2021-11-19 17:03:21 0 [Warning] WSREP: checksum failed, hdr: len=0 has_crc32=0 has_crc32c=0 crc32=16777216
 
Nov 19 17:03:21 NODE-01 mysqld[78300]: 2021-11-19 17:03:21 0 [Warning] WSREP: unserialize error invalid protocol version 8: 71 (Protocol error)
 
Nov 19 17:03:21 NODE-01 mysqld[78300]: at /home/buildbot/buildbot/build/gcomm/src/gcomm/datagram.hpp:unserialize():133
 
Nov 19 17:03:24 NODE-01 mysqld[78300]: 2021-11-19 17:03:24 0 [Note] WSREP: (1132bd81-a3fd, 'tcp://0.0.0.0:4567') connection to peer 00000000-0000 with addr  timed out, no messages seen in PT3S, socket stats: rtt: 4207 rttvar: 1915 rto: 205000 lost: 0 last_data_recv: 1065 cwnd: 10 last_queued_since: 3077417409 last_delivered_since: 110793838298892 send_queue_length: 0 send_queue_bytes: 0 segment: 0 messages: 0

OpenVAS Scan Port 3306+4567 Targeted

Nov 19 16:47:42 NODE-01 mysqld[75077]: 2021-11-19 16:47:42 118 [Warning] Aborted connection 118 to db: 'unconnected' user: 'unauthenticated' host: 'connecting host' (This connection closed normally without authentication)
 
Nov 19 16:47:42 NODE-01 mysqld[75077]: terminate called after throwing an instance of 'boost::exception_detail::clone_impl<boost::exception_detail::error_info_injector<std::system_error> >'
 
Nov 19 16:47:42 NODE-01 mysqld[75077]: what():  remote_endpoint: Transport endpoint is not connected
 
Nov 19 16:47:42 NODE-01 mysqld[75077]: 211119 16:47:42 [ERROR] mysqld got signal 6 ;
 
Nov 19 16:47:42 NODE-01 mysqld[75077]: This could be because you hit a bug. It is also possible that this binary
 
Nov 19 16:47:42 NODE-01 mysqld[75077]: or one of the libraries it was linked against is corrupt, improperly built,
 
Nov 19 16:47:42 NODE-01 mysqld[75077]: or misconfigured. This error can also be caused by malfunctioning hardware.
 
Nov 19 16:47:42 NODE-01 mysqld[75077]: To report this bug, see https://mariadb.com/kb/en/reporting-bugs
 
Nov 19 16:47:42 NODE-01 mysqld[75077]: We will try our best to scrape up some info that will hopefully help
 
Nov 19 16:47:42 NODE-01 mysqld[75077]: diagnose the problem, but since we have already crashed,
 
Nov 19 16:47:42 NODE-01 mysqld[75077]: something is definitely wrong and this may fail.
 
Nov 19 16:47:42 NODE-01 mysqld[75077]: Server version: 10.4.22-MariaDB-log
 
Nov 19 16:47:42 NODE-01 mysqld[75077]: key_buffer_size=134217728
 
Nov 19 16:47:42 NODE-01 mysqld[75077]: read_buffer_size=131072
 
Nov 19 16:47:42 NODE-01 mysqld[75077]: max_used_connections=32
 
Nov 19 16:47:42 NODE-01 mysqld[75077]: max_threads=153
 
Nov 19 16:47:42 NODE-01 mysqld[75077]: thread_count=49
 
Nov 19 16:47:42 NODE-01 mysqld[75077]: It is possible that mysqld could use up to
 
Nov 19 16:47:42 NODE-01 mysqld[75077]: key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 781113 K  bytes of memory
 
Nov 19 16:47:42 NODE-01 mysqld[75077]: Hope that's ok; if not, decrease some variables in the equation.
 
Nov 19 16:47:42 NODE-01 mysqld[75077]: Thread pointer: 0x0
 
Nov 19 16:47:42 NODE-01 mysqld[75077]: Attempting backtrace. You can use the following information to find out
 
Nov 19 16:47:42 NODE-01 mysqld[75077]: where mysqld died. If you see no messages after this, something went
 
Nov 19 16:47:42 NODE-01 mysqld[75077]: terribly wrong...
 
Nov 19 16:47:42 NODE-01 mysqld[75077]: stack_bottom = 0x0 thread_stack 0x49000
 
Nov 19 16:47:42 NODE-01 mysqld[75077]: /usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x55ce5f1dd2be]
 
Nov 19 16:47:42 NODE-01 mysqld[75077]: /usr/sbin/mysqld(handle_fatal_signal+0x30f)[0x55ce5ec64dbf]
 
Nov 19 16:47:42 NODE-01 mysqld[75077]: /lib64/libpthread.so.0(+0xf630)[0x7f682458c630]
 
Nov 19 16:47:42 NODE-01 mysqld[75077]: /lib64/libc.so.6(gsignal+0x37)[0x7f68239d7387]
 
Nov 19 16:47:42 NODE-01 mysqld[75077]: /lib64/libc.so.6(abort+0x148)[0x7f68239d8a78]
 
Nov 19 16:47:42 NODE-01 mysqld[75077]: /lib64/libstdc++.so.6(_ZN9__gnu_cxx27__verbose_terminate_handlerEv+0x165)[0x7f68240d1a95]
 
Nov 19 16:47:42 NODE-01 mysqld[75077]: /lib64/libstdc++.so.6(+0x5ea06)[0x7f68240cfa06]
 
Nov 19 16:47:42 NODE-01 mysqld[75077]: /lib64/libstdc++.so.6(+0x5ea33)[0x7f68240cfa33]
 
Nov 19 16:47:42 NODE-01 mysqld[75077]: /lib64/libstdc++.so.6(+0x5ec53)[0x7f68240cfc53]
 
Nov 19 16:47:42 NODE-01 mysqld[75077]: /usr/lib64/galera-4/libgalera_smm.so(+0x197ab4)[0x7f6819b0bab4]
 
Nov 19 16:47:42 NODE-01 mysqld[75077]: /usr/lib64/galera-4/libgalera_smm.so(+0x197b5c)[0x7f6819b0bb5c]
 
Nov 19 16:47:42 NODE-01 mysqld[75077]: /usr/lib64/galera-4/libgalera_smm.so(+0x1a3cbf)[0x7f6819b17cbf]
 
Nov 19 16:47:42 NODE-01 mysqld[75077]: /usr/lib64/galera-4/libgalera_smm.so(+0x1a62e2)[0x7f6819b1a2e2]
 
Nov 19 16:47:42 NODE-01 mysqld[75077]: /usr/lib64/galera-4/libgalera_smm.so(+0x1aa533)[0x7f6819b1e533]
 
Nov 19 16:47:42 NODE-01 mysqld[75077]: /usr/lib64/galera-4/libgalera_smm.so(+0x196eeb)[0x7f6819b0aeeb]
 
Nov 19 16:47:42 NODE-01 mysqld[75077]: /usr/lib64/galera-4/libgalera_smm.so(+0x192d81)[0x7f6819b06d81]
 
Nov 19 16:47:42 NODE-01 mysqld[75077]: /usr/lib64/galera-4/libgalera_smm.so(+0xbf599)[0x7f6819a33599]
 
Nov 19 16:47:42 NODE-01 mysqld[75077]: /usr/lib64/galera-4/libgalera_smm.so(+0xa65d7)[0x7f6819a1a5d7]
 
Nov 19 16:47:42 NODE-01 mysqld[75077]: /usr/lib64/galera-4/libgalera_smm.so(+0xa6989)[0x7f6819a1a989]
 
Nov 19 16:47:42 NODE-01 mysqld[75077]: /lib64/libpthread.so.0(+0x7ea5)[0x7f6824584ea5]
 
Nov 19 16:47:42 NODE-01 mysqld[75077]: /lib64/libc.so.6(clone+0x6d)[0x7f6823a9f9fd]
 
Nov 19 16:47:42 NODE-01 mysqld[75077]: The manual page at https://mariadb.com/kb/en/how-to-produce-a-full-stack-trace-for-mysqld/ contains
 
Nov 19 16:47:42 NODE-01 mysqld[75077]: information that should help you find out what is causing the crash.
 
Nov 19 16:47:42 NODE-01 mysqld[75077]: Writing a core file...
 
Nov 19 16:47:42 NODE-01 mysqld[75077]: Working directory at /var/lib/mysql
 
Nov 19 16:47:42 NODE-01 mysqld[75077]: Resource Limits:
 
Nov 19 16:47:42 NODE-01 mysqld[75077]: Fatal signal 11 while backtracing
 
Nov 19 16:47:43 NODE-01 systemd[1]: mariadb.service: main process exited, code=killed, status=11/SEGV
 
Nov 19 16:47:43 NODE-01 systemd[1]: Unit mariadb.service entered failed state.
 
Nov 19 16:47:43 NODE-01 systemd[1]: mariadb.service failed.
 
Nov 19 16:47:48 NODE-01 systemd[1]: mariadb.service holdoff time over, scheduling restart.
 
Nov 19 16:47:48 NODE-01 systemd[1]: Stopped MariaDB 10.4.22 database server.
 
Nov 19 16:47:48 NODE-01 systemd[1]: Starting MariaDB 10.4.22 database server...

Additionally I guess it makes sense that I managed to crash our cluster entirely with our scheduled scan which is my OP here, we can run up to 5 machine scans in parallel so it is highly likely the 3 machines in the cluster being scanned at the same time caused a split-brain/impossible automated recovery.

Comment by Brendon Abbott [ 2022-01-05 ]

Watching.
We have also experienced this exact behaviour on a customer's lab system, and it appears to be due to a scan from Nessus which from wikipedia I understand is the same/similar code base to OpenVAS. Luckily, the DB cluster was being sporadically scanned and the crashed node brought back on line before the scan took out a second node and hence split-brain of the 3 node cluster.

Comment by Henrik Wejdmark [ 2022-06-02 ]

I have the same issue, but on MariaDB 10.6.7. Maria crashes when OpenVas scans the host.

Comment by Jan Lindström (Inactive) [ 2022-06-03 ]

Fix is in Galera library 26.4.12.

Comment by Sergei Golubchik [ 2023-12-29 ]

Also fixed in 25.3.37

Generated at Thu Feb 08 09:34:53 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.