[MDEV-14584] mysqld got signal 11 Created: 2017-12-05  Updated: 2019-12-12  Resolved: 2019-12-12

Status: Closed
Project: MariaDB Server
Component/s: Galera, Server
Affects Version/s: 10.2.10
Fix Version/s: N/A

Type: Bug Priority: Major
Reporter: Valeriy Kazantsev Assignee: Jan Lindström (Inactive)
Resolution: Incomplete Votes: 0
Labels: Galera, MariaDB
Environment:

CentOS Linux release 7.4.1708 x86_64


Attachments: File core.8660.tar.gz.00     File core.8660.tar.gz.01     File core.8660.tar.gz.02    

 Description   

Hello, guys! Every night in my Galera cluster of 2 servers occurs alternately crash servers. I have report from mysql_error.log:

2017-12-05  9:27:13 140599126464256 [Note] WSREP: (8f3cb7d9, 'tcp://0.0.0.0:4567') turning message relay requesting off
171205 11:38:01 [ERROR] mysqld got signal 11 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
 
To report this bug, see https://mariadb.com/kb/en/reporting-bugs
 
We will try our best to scrape up some info that will hopefully help
diagnose the problem, but since we have already crashed,
something is definitely wrong and this may fail.
 
Server version: 10.2.10-MariaDB-log
key_buffer_size=16777216
read_buffer_size=131072
max_used_connections=15
max_threads=102
thread_count=23
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 240475 K  bytes of memory
Hope that's ok; if not, decrease some variables in the equation.
 
Thread pointer: 0x7fddf40009a8
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...

/etc/my.cnf.d/server.cnf:

# this is read by the standalone daemon and embedded servers
[server]
 
# this is only for the mysqld standalone daemon
[mysqld]
bind-address = 0.0.0.0
transaction-isolation = READ-COMMITTED
max_allowed_packet = 16M
myisam_recover_options = BACKUP
max_connections = 100
sql_mode =
 
# Logs
log_error = /var/lib/mysql/logs/mysql_error.log
slow_query_log = 1
slow_query_log_file = /var/lib/mysql/logs/mysql_slow.log
long_query_time = 2
 
 
# Cache parameters
query_cache_size = 64M
query_cache_limit = 2M
table_open_cache = 4096
thread_cache_size = 200
key_buffer_size = 16M
thread_stack = 128K
join_buffer_size = 2M
sort_buffer_size = 2M
 
# Parameters for temporary tables
tmpdir = /tmp
max_heap_table_size = 512M
tmp_table_size = 512M
query_cache_type=1
 
# InnoDB parameters
innodb_file_per_table
innodb_buffer_pool_size = 4G
innodb_flush_log_at_trx_commit = 2
innodb_log_file_size = 128M
innodb_flush_method = O_DIRECT
 
# Database charset parameters
character-set-server = utf8
collation-server = utf8_unicode_ci
init-connect = "SET NAMES utf8 COLLATE utf8_unicode_ci"
skip-character-set-client-handshake
#skip-name-resolve
 
#
# * Galera-related settings
#
[galera]
# Mandatory settings
wsrep_on = ON
wsrep_provider = /usr/lib64/galera/libgalera_smm.so
wsrep_cluster_address = "gcomm://192.168.59.78,172.16.140.29"
wsrep_cluster_name = "site-appdb"
wsrep_node_address = "192.168.59.78"
wsrep_node_name = "site-appdb01"
wsrep_sst_method = rsync
wsrep_sst_auth=repl_user:+vQvI9Fe8Nr6
binlog_format = row
default_storage_engine = InnoDB
innodb_autoinc_lock_mode = 2
#innodb_locks_unsafe_for_binlog = 1
#
# Allow server to accept connections on all interfaces.
#
#bind-address=0.0.0.0
#
# Optional setting
wsrep_slave_threads=1
innodb_file_per_table
innodb_buffer_pool_size = 4G
innodb_flush_log_at_trx_commit = 2
innodb_log_file_size = 128M
innodb_flush_method = O_DIRECT
 
# this is only for embedded server
[embedded]
 
# This group is only read by MariaDB servers, not by MySQL.
# If you use the same .cnf file for MySQL and MariaDB,
# you can put MariaDB-only options here
[mariadb]
 
# This group is only read by MariaDB-10.1 servers.
# If you use the same .cnf file for MariaDB of different versions,
# use this group for options that older servers don't understand
[mariadb-10.1]

What is installed:

# rpm -qa | grep "MariaDB\|galera"
galera-25.3.12-2.el7.x86_64
MariaDB-common-10.2.10-1.el7.centos.x86_64
MariaDB-client-10.2.10-1.el7.centos.x86_64
MariaDB-server-10.2.10-1.el7.centos.x86_64
MariaDB-compat-10.2.10-1.el7.centos.x86_64



 Comments   
Comment by Elena Stepanova [ 2017-12-06 ]

trider,

is it all that you've got in the server error log? There isn't much to work with. Do you have a coredump, maybe?

Comment by Valeriy Kazantsev [ 2017-12-06 ]

Hello, Elena!
I turned on the recording coredump files at MariaDB level:

[mysqld]
core-file
.......
[mysqld_safe]
core-file-size=unlimted
.......

Tried kill -11 `pidof mysqld`, coredump file is written to /var/lib/mysql. Waiting for MariaDB crash tonight.

Comment by Daniel Black [ 2017-12-07 ]

Note if using systemd, the systemd LimitCORE= directive is needed - ref: https://mariadb.com/kb/en/library/systemd/#customization

Comment by Valeriy Kazantsev [ 2017-12-07 ]

I have collected over the night core dump files of MariaDB. Archive in attachment.

Comment by Elena Stepanova [ 2017-12-07 ]

Are you using RPM packages or binary tarball, and in each case – which one(s)?

Comment by Valeriy Kazantsev [ 2017-12-07 ]

Yes, I'm using RPM packages https://downloads.mariadb.com/MariaDB/mariadb-10.2.10/yum/centos/7/x86_64/rpms/

Comment by Valeriy Kazantsev [ 2017-12-15 ]

Hello! Have you some information about solving this problem?

Comment by Valeriy Kazantsev [ 2017-12-19 ]

I come to the conclusion that the Galera is a complete shit, and raw bullshit, which will brought to mind through 4 years. Therefore, hardly anyone uses multi-master Galera on high loads. For example problem https://jira.mariadb.org/browse/MDEV-12837 an epic fail of a decision MariaDB+Galera.

Comment by Jan Lindström (Inactive) [ 2019-12-12 ]

No real info to work on. I can't open the core files, I would have needed ready opened core file with real symbols.

Generated at Thu Feb 08 08:14:43 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.