[MDEV-10490] MariaDB crashes - [ERROR] mysqld got signal 11 Created: 2016-08-03  Updated: 2019-12-12  Resolved: 2019-12-12

Status: Closed
Project: MariaDB Server
Component/s: Galera
Affects Version/s: 10.0.26-galera
Fix Version/s: N/A

Type: Bug Priority: Major
Reporter: Christian Affolter Assignee: Jan Lindström (Inactive)
Resolution: Won't Fix Votes: 1
Labels: galera
Environment:

Linux distribution: Gentoo Linux
Kernel: 3.10.101
MariaDB Galera package: dev-db/mariadb-galera-10.0.26 / sys-cluster/galera-25.3.15
OpenSSL package: dev-libs/openssl- 1.0.2h-r2
glibc package: sys-libs/glibc-2.22-r4


 Description   

MariaDB crashes at least once per day on one of the three node of a Galera Cluster, running within KVM based Gentoo Linux VMs with enabled SSL/TLS connections.

160803 17:02:37 [ERROR] mysqld got signal 11 ;
 
This could be because you hit a bug. It is also possible that this binary
 
or one of the libraries it was linked against is corrupt, improperly built,
 
or misconfigured. This error can also be caused by malfunctioning hardware.
 
 
 
To report this bug, see https://mariadb.com/kb/en/reporting-bugs
 
 
 
We will try our best to scrape up some info that will hopefully help
 
diagnose the problem, but since we have already crashed, 
something is definitely wrong and this may fail.
 
 
 
Server version: 10.0.26-MariaDB-wsrep
 
key_buffer_size=16777216
 
read_buffer_size=262144
 
max_used_connections=261
 
max_threads=1026
 
thread_count=97
 
It is possible that mysqld could use up to 
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 825068 K  bytes of memory
 
Hope that's ok; if not, decrease some variables in the equation.
 
 
 
Thread pointer: 0x0x0
 
Attempting backtrace. You can use the following information to find out
 
where mysqld died. If you see no messages after this, something went
 
terribly wrong...
 
stack_bottom = 0x0 thread_stack 0x48000
 
/usr/sbin/mysqld(my_print_stacktrace+0x49)[0x7f95fe6f4599]
 
/usr/sbin/mysqld(handle_fatal_signal+0x329)[0x7f95fe18a5f9]
 
/lib64/libpthread.so.0(+0x11880)[0x7f95fcf1d880]
 
/lib64/libc.so.6(+0x987c0)[0x7f95fbaaa7c0]
 
/usr/lib64/libcrypto.so.1.0.0(+0x125565)[0x7f95fc930565]
 
/usr/lib64/libcrypto.so.1.0.0(BIO_write+0x8f)[0x7f95fc92687f]
 
/usr/lib64/libssl.so.1.0.0(ssl3_write_pending+0x91)[0x7f95fccbe1f1]
 
/usr/lib64/libssl.so.1.0.0(ssl3_write_bytes+0xe5)[0x7f95fccbe965]
 
/usr/lib/galera/libgalera_smm.so(_ZN4asio3ssl6detail5io_opINS_19basic_stream_socketINS_2ip3tcpENS_21stream_socket_serviceIS5_EEEENS1_8write_opIN5boost5arrayINS_12const_bufferELm2EEEEENS_6detail8write_opINS0_6streamIS8_EESD_NSF_14transfer_all_tENSA_3_bi6bind_tIvNSA_4_mfi3mf2IvN5gcomm13AsioTcpSocketERKNS_10error_codeEmEENSK_5list3INSK_5valueINSA_10shared_ptrISP_EEEEPFNSA_3argILi1EEEvEPFNSZ_ILi2EEEvEEEEEEEEclESQ_mi+0x20e)[0x7f95faee5c2e]
 
/usr/lib/galera/libgalera_smm.so(_ZN4asio6detail8write_opINS_3ssl6streamINS_19basic_stream_socketINS_2ip3tcpENS_21stream_socket_serviceIS6_EEEEEEN5boost5arrayINS_12const_bufferELm2EEENS0_14transfer_all_tENSB_3_bi6bind_tIvNSB_4_mfi3mf2IvN5gcomm13AsioTcpSocketERKNS_10error_codeEmEENSG_5list3INSG_5valueINSB_10shared_ptrISL_EEEEPFNSB_3argILi1EEEvEPFNSV_ILi2EEEvEEEEEEclESO_mi+0x212)[0x7f95faee6242]
 
/usr/lib/galera/libgalera_smm.so(_ZN5gcomm13AsioTcpSocket9write_oneERKN5boost5arrayIN4asio12const_bufferELm2EEE+0x14c)[0x7f95faecc5ec]
 
/usr/lib/galera/libgalera_smm.so(_ZN5gcomm13AsioTcpSocket4sendERKNS_8DatagramE+0x2e0)[0x7f95faecd5e0]
 
/usr/lib/galera/libgalera_smm.so(_Z4sendPN5gcomm6SocketERNS_8DatagramE+0x38)[0x7f95fae82548]
 
/usr/lib/galera/libgalera_smm.so(_ZN5gcomm6GMCast11handle_downERNS_8DatagramERKNS_13ProtoDownMetaE+0x2f3)[0x7f95fae835a3]
 
/usr/lib/galera/libgalera_smm.so(_ZN5gcomm8Protolay9send_downERNS_8DatagramERKNS_13ProtoDownMetaE+0x6c)[0x7f95fae7fcdc]
 
/usr/lib/galera/libgalera_smm.so(_ZN5gcomm3evs5Proto8send_gapERKNS0_6CallerERKNS_4UUIDERKNS_6ViewIdENS0_5RangeEbb+0x65d)[0x7f95fae6b23d]
 
/usr/lib/galera/libgalera_smm.so(_ZN5gcomm3evs5Proto11handle_userERKNS0_11UserMessageESt17_Rb_tree_iteratorISt4pairIKNS_4UUIDENS0_4NodeEEERKNS_8DatagramE+0xd65)[0x7f95fae700e5]
 
/usr/lib/galera/libgalera_smm.so(_ZN5gcomm3evs5Proto10handle_msgERKNS0_7MessageERKNS_8DatagramEb+0x6ca)[0x7f95fae7347a]
 
/usr/lib/galera/libgalera_smm.so(_ZN5gcomm3evs5Proto9handle_upEPKvRKNS_8DatagramERKNS_11ProtoUpMetaE+0x4a4)[0x7f95fae74084]
 
/usr/lib/galera/libgalera_smm.so(_ZN5gcomm6GMCast9handle_upEPKvRKNS_8DatagramERKNS_11ProtoUpMetaE+0x329)[0x7f95fae8f399]
 
/usr/lib/galera/libgalera_smm.so(_ZN5gcomm10Protostack8dispatchEPKvRKNS_8DatagramERKNS_11ProtoUpMetaE+0x6e)[0x7f95faebc62e]
 
/usr/lib/galera/libgalera_smm.so(_ZN5gcomm12AsioProtonet8dispatchERKPKvRKNS_8DatagramERKNS_11ProtoUpMetaE+0x6f)[0x7f95faeef3df]
 
/usr/lib/galera/libgalera_smm.so(_ZN5gcomm13AsioTcpSocket12read_handlerERKN4asio10error_codeEm+0x612)[0x7f95faecbeb2]
 
/usr/lib/galera/libgalera_smm.so(_ZN4asio6detail7read_opINS_3ssl6streamINS_19basic_stream_socketINS_2ip3tcpENS_21stream_socket_serviceIS6_EEEEEEN5boost5arrayINS_14mutable_bufferELm1EEENSB_3_bi6bind_tImNSB_4_mfi3mf2ImN5gcomm13AsioTcpSocketERKNS_10error_codeEmEENSF_5list3INSF_5valueINSB_10shared_ptrISK_EEEEPFNSB_3argILi1EEEvEPFNSU_ILi2EEEvEEEEENSG_IvNSI_IvSK_SN_mEES11_EEEclESN_mi+0x4b1)[0x7f95faee45c1]
 
/usr/lib/galera/libgalera_smm.so(_ZN4asio3ssl6detail5io_opINS_19basic_stream_socketINS_2ip3tcpENS_21stream_socket_serviceIS5_EEEENS1_7read_opINS_6detail17consuming_buffersINS_14mutable_bufferEN5boost5arrayISC_Lm1EEEEEEENSA_7read_opINS0_6streamIS8_EESF_NSD_3_bi6bind_tImNSD_4_mfi3mf2ImN5gcomm13AsioTcpSocketERKNS_10error_codeEmEENSL_5list3INSL_5valueINSD_10shared_ptrISQ_EEEEPFNSD_3argILi1EEEvEPFNS10_ILi2EEEvEEEEENSM_IvNSO_IvSQ_ST_mEES17_EEEEEclESR_mi+0xbc)[0x7f95faee36cc]
 
/usr/lib/galera/libgalera_smm.so(_ZN4asio6detail23reactive_socket_recv_opINS_17mutable_buffers_1ENS_3ssl6detail5io_opINS_19basic_stream_socketINS_2ip3tcpENS_21stream_socket_serviceIS8_EEEENS4_7read_opINS0_17consuming_buffersINS_14mutable_bufferEN5boost5arrayISE_Lm1EEEEEEENS0_7read_opINS3_6streamISB_EESH_NSF_3_bi6bind_tImNSF_4_mfi3mf2ImN5gcomm13AsioTcpSocketERKNS_10error_codeEmEENSN_5list3INSN_5valueINSF_10shared_ptrISS_EEEEPFNSF_3argILi1EEEvEPFNS12_ILi2EEEvEEEEENSO_IvNSQ_IvSS_SV_mEES19_EEEEEEE11do_completeEPNS0_15task_io_serviceEPNS0_25task_io_service_operationESV_m+0xd2)[0x7f95faee4812]
 
/usr/lib/galera/libgalera_smm.so(_ZN4asio6detail15task_io_service3runERNS_10error_codeE+0x453)[0x7f95faed5e13]
 
/usr/lib/galera/libgalera_smm.so(_ZN5gcomm12AsioProtonet10event_loopERKN2gu8datetime6PeriodE+0x25f)[0x7f95faef071f]
 
/usr/lib/galera/libgalera_smm.so(_ZN9GCommConn3runEv+0x76)[0x7f95faf06fd6]
 
/usr/lib/galera/libgalera_smm.so(_ZN9GCommConn6run_fnEPv+0x2b)[0x7f95faf0f9ab]
 
/lib64/libpthread.so.0(+0x764c)[0x7f95fcf1364c]
 
/lib64/libc.so.6(clone+0x6d)[0x7f95fbb0820d]
 
The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
 
information that should help you find out what is causing the crash.



 Comments   
Comment by Daniel Black [ 2016-08-03 ]

which sys-cluster/galera version? 25.3.15 appears to be the latest in gentoo.

You appear to have some form of ssl transport for galera- what is your configuration for this?

Comment by Christian Affolter [ 2016-08-04 ]

sys-cluster/galera is installed in version 25.3.15

Below are the (hopefully) relevant my.cnf settings:

# These settings are required for Galera clustering
 
#mysql settings
 
binlog_format=ROW
 
default-storage-engine=innodb
 
innodb_autoinc_lock_mode=2
 
query_cache_size=0
 
query_cache_type=0
 
bind-address=192.0.2.249
 
 
 
### Galera Cluster Configuration
 
# Path to wsrep provider.
 
wsrep_provider=/usr/lib/galera/libgalera_smm.so
 
# The logical cluster name.
 
wsrep_cluster_name="example_cluster_01"
 
# Galera parameters.
 
wsrep_provider_options="socket.ssl_cert=/etc/ssl/galera/vm-01.example.com.cert.pem;socket.ssl_key=/etc/ssl/galera/vm-01.example.com.key.pem;socket.ssl_ca=/usr/local/share/ca-certificates/Example_Certificate_Authority.crt"
 
# Number of threads to use for applying slave writesets.
 
wsrep_slave_threads=8
 
# List of all primary nodes, excluding the actual node and arbitrator nodes.
 
wsrep_cluster_address="gcomm://192.0.2.250,192.0.2.251"
 
# The method to use for state snapshot transfers.
 
wsrep_sst_method=xtrabackup-v2
 
# Authentication information for state snapshot transfers.
 
wsrep_sst_auth=sstuser:obfuscated1234
 
 
 
### Galera Node Configuration
 
# Network address of the actual node.
 
wsrep_node_address="192.0.2.249"
 
# The address at which this node expects to receive state transfers.
 
# Explicitly set so the correct host name, matching with the certificate, is used by xtrabackup/socat.
 
wsrep_sst_receive_address="vm-01.example.com"
 
 
 
### State Snapshot Transfer configuration
 
[sst]
 
# Use OpenSSL based encryption.
 
encrypt=2
 
# Certificate and key concatenated in one file.
 
tcert=/etc/ssl/galera/vm-01.example.com.bundle.pem
 
# CA certificate.
 
tca=/usr/local/share/ca-certificates/Example_Certificate_Authority.crt

Comment by Hendrik Frenzel [ 2018-07-12 ]

We got the same error on CentOS 7 with the following packages/versions:

mariadb-10.1.20-2.el7.x86_64
 
mariadb-config-10.1.20-2.el7.x86_64
 
mariadb-server-10.1.20-2.el7.x86_64
 
mariadb-libs-10.1.20-2.el7.x86_64
 
mariadb-errmsg-10.1.20-2.el7.x86_64
 
mariadb-common-10.1.20-2.el7.x86_64
 
mariadb-server-galera-10.1.20-2.el7.x86_64
 
galera-25.3.16-3.el7.x86_64


# yum info mariadb-10.1.20-2.el7.x86_64
 
Loading mirror speeds from cached hostfile
 
Installed Packages
 
Name        : mariadb
 
Arch        : x86_64
 
Epoch       : 3
 
Version     : 10.1.20
 
Release     : 2.el7
 
Size        : 39 M
 
Repo        : installed
 
From repo   : centos-openstack-queens
 
Summary     : A community developed branch of MySQL
 
URL         : http://mariadb.org
 
License     : GPLv2 with exceptions and LGPLv2 and BSD
 
Description : MariaDB is a community developed branch of MySQL.
 
            : MariaDB is a multi-user, multi-threaded SQL database server.
 
            : It is a client/server implementation consisting of a server daemon (mysqld)
 
            : and many different client programs and libraries. The base package
 
            : contains the standard MariaDB/MySQL client programs and generic MySQL files.

This occured few times per day. Sometimes the process just freezes and a kill -9 is required.
We also use SSL with certificates created with FreeIPA.

Comment by Jan Lindström (Inactive) [ 2019-12-12 ]

Support for 10.0-galera has ended.

Generated at Thu Feb 08 07:42:36 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.