[MDEV-16362] MariaDB crashing after update to version 10.2.15-1 Created: 2018-05-31  Updated: 2023-06-06  Resolved: 2023-06-06

Status: Closed
Project: MariaDB Server
Component/s: Galera, Server, Storage Engine - InnoDB
Affects Version/s: 10.2.15
Fix Version/s: N/A

Type: Bug Priority: Major
Reporter: Oswaldo Garcia Assignee: Elena Stepanova
Resolution: Won't Fix Votes: 0
Labels: backlog
Environment:

OS: CentOS Linux release 7.5.1804 (Core)
Vmware host: 4GB RAM


 Description   

I have 3 node cluster that started to crash very often (two times in the last 24 hours) after upgrade mariadb version from 10.2.14 to 10.2.15.

Logs are reporting the following:

May 29 20:14:03 sqlserver mysqld: 180529 20:14:03 [ERROR] mysqld got signal 11 ;
 
May 29 20:14:03 sqlserver mysqld: This could be because you hit a bug. It is also possible that this binary
 
May 29 20:14:03 sqlserver mysqld: or one of the libraries it was linked against is corrupt, improperly built,
 
May 29 20:14:03 sqlserver mysqld: or misconfigured. This error can also be caused by malfunctioning hardware.
 
May 29 20:14:03 sqlserver mysqld: To report this bug, see https://mariadb.com/kb/en/reporting-bugs
 
May 29 20:14:03 sqlserver mysqld: We will try our best to scrape up some info that will hopefully help
 
May 29 20:14:03 sqlserver mysqld: diagnose the problem, but since we have already crashed,
 
May 29 20:14:03 sqlserver mysqld: something is definitely wrong and this may fail.
 
May 29 20:14:03 sqlserver mysqld: Server version: 10.2.15-MariaDB
 
May 29 20:14:03 sqlserver mysqld: key_buffer_size=134217728
 
May 29 20:14:03 sqlserver mysqld: read_buffer_size=131072
 
May 29 20:14:03 sqlserver mysqld: max_used_connections=9
 
May 29 20:14:03 sqlserver mysqld: max_threads=153
 
May 29 20:14:03 sqlserver mysqld: thread_count=17
 
May 29 20:14:03 sqlserver mysqld: It is possible that mysqld could use up to
 
May 29 20:14:03 sqlserver mysqld: key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 467238 K  bytes of memory
 
May 29 20:14:03 sqlserver mysqld: Hope that's ok; if not, decrease some variables in the equation.
 
May 29 20:14:03 sqlserver mysqld: Thread pointer: 0x7fc044002bb8
 
May 29 20:14:03 sqlserver mysqld: Attempting backtrace. You can use the following information to find out
 
May 29 20:14:03 sqlserver mysqld: where mysqld died. If you see no messages after this, something went
 
May 29 20:14:03 sqlserver mysqld: terribly wrong...
 
May 29 20:14:03 sqlserver mysqld: stack_bottom = 0x7fc058096d30 thread_stack 0x49000
 
May 29 20:14:03 sqlserver mysqld: *** buffer overflow detected ***: /usr/sbin/mysqld terminated
 
May 29 20:14:03 sqlserver mysqld: ======= Backtrace: =========
 
May 29 20:14:03 sqlserver mysqld: /lib64/libc.so.6(__fortify_fail+0x37)[0x7fc05b97c6e7]
 
May 29 20:14:03 sqlserver mysqld: /lib64/libc.so.6(+0x116862)[0x7fc05b97a862]
 
May 29 20:14:03 sqlserver mysqld: /lib64/libc.so.6(+0x118647)[0x7fc05b97c647]
 
May 29 20:14:03 sqlserver mysqld: /usr/sbin/mysqld(my_addr_resolve+0xda)[0x5637c83aca6a]
 
May 29 20:14:03 sqlserver mysqld: /usr/sbin/mysqld(my_print_stacktrace+0x1c2)[0x5637c8396132]
 
May 29 20:14:03 sqlserver mysqld: /usr/sbin/mysqld(handle_fatal_signal+0x355)[0x5637c7e21a05]
 
May 29 20:14:03 sqlserver mysqld: /lib64/libpthread.so.0(+0xf6d0)[0x7fc05d5c66d0]
 
May 29 20:14:03 sqlserver mysqld: /usr/sbin/mysqld(_Z11ull_get_keyPKhPmc+0x14)[0x5637c7e881f4]
 
May 29 20:14:03 sqlserver mysqld: /usr/sbin/mysqld(my_hash_first_from_hash_value+0x6b)[0x5637c837359b]
 
May 29 20:14:03 sqlserver mysqld: /usr/sbin/mysqld(my_hash_search+0x11)[0x5637c8373701]
 
May 29 20:14:03 sqlserver mysqld: /usr/sbin/mysqld(_ZN22Item_func_release_lock7val_intEv+0x199)[0x5637c7e8a449]
 
May 29 20:14:03 sqlserver mysqld: /usr/sbin/mysqld(_ZN4Item4sendEP8ProtocolP6String+0x20c)[0x5637c7e323dc]
 
May 29 20:14:03 sqlserver mysqld: /usr/sbin/mysqld(_ZN8Protocol19send_result_set_rowEP4ListI4ItemE+0xdb)[0x5637c7c1012b]
 
May 29 20:14:03 sqlserver mysqld: /usr/sbin/mysqld(_ZN11select_send9send_dataER4ListI4ItemE+0x53)[0x5637c7c67e33]
 
May 29 20:14:03 sqlserver mysqld: /usr/sbin/mysqld(_ZN4JOIN10exec_innerEv+0xaa4)[0x5637c7ce7474]
 
May 29 20:14:03 sqlserver mysqld: /usr/sbin/mysqld(_ZN4JOIN4execEv+0x33)[0x5637c7ce7623]
 
May 29 20:14:03 sqlserver mysqld: /usr/sbin/mysqld(_Z12mysql_selectP3THDP10TABLE_LISTjR4ListI4ItemEPS4_jP8st_orderS9_S7_S9_yP13select_resultP18st_select_lex_unitP13st_select_lex+0x11a)[0x5637c7ce777a]
 
May 29 20:14:03 sqlserver mysqld: /usr/sbin/mysqld(_Z13handle_selectP3THDP3LEXP13select_resultm+0x254)[0x5637c7ce82d4]
 
May 29 20:14:03 sqlserver mysqld: /usr/sbin/mysqld(+0x4197f9)[0x5637c7bcd7f9]
 
May 29 20:14:03 sqlserver mysqld: /usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x6864)[0x5637c7c99a24]
 
May 29 20:14:03 sqlserver mysqld: /usr/sbin/mysqld(_Z11mysql_parseP3THDPcjP12Parser_statebb+0x2de)[0x5637c7c9c3ae]
 
May 29 20:14:03 sqlserver mysqld: /usr/sbin/mysqld(+0x4e8c7f)[0x5637c7c9cc7f]
 
May 29 20:14:03 sqlserver mysqld: /usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcjbb+0xf0b)[0x5637c7c9e21b]
 
May 29 20:14:03 sqlserver mysqld: /usr/sbin/mysqld(_Z10do_commandP3THD+0x165)[0x5637c7ca0075]
 
May 29 20:14:03 sqlserver mysqld: /usr/sbin/mysqld(_Z24do_handle_one_connectionP7CONNECT+0x1aa)[0x5637c7d6230a]
 
May 29 20:14:03 sqlserver mysqld: /usr/sbin/mysqld(handle_one_connection+0x3d)[0x5637c7d6242d]
 
May 29 20:14:03 sqlserver mysqld: /lib64/libpthread.so.0(+0x7e25)[0x7fc05d5bee25]
 
May 29 20:14:03 sqlserver mysqld: /lib64/libc.so.6(clone+0x6d)[0x7fc05b962bad]

Can you guys please let me know if this is a bug with the new version or something else?

Thanks in advance for your support.

 Comments   
Comment by Elena Stepanova [ 2018-05-31 ]

We don't have any open bug reports which would obviously match this (although of course with buffer overflow it's not easy to tell for sure).
If it happens often, do you know the query which causes it? Normally it would be written in the error log, but again, with buffer overflow the crash report may remain unfinished; however, it looks like you don't have many connections in your node, so if you temporarily enable the general log, it will probably help to identify the query.

Also, do you have a coredump, and if not yet, can you configure your servers to produce one next time either of them crashes?

Comment by Oswaldo Garcia [ 2018-05-31 ]

Thanks @Elena Stepanova for your quick response. I've inspected the logs with more detail from the latest crash and found the query that initiated the problem. The issue started on the first node with the following error:

May 30 15:22:20 sqlserversql01 mysqld: 2018-05-30 15:22:20 140021285766912 [ERROR] Slave SQL: Error 'Table 'wp_sndr_mail_users_info' already exists' on query. Default database: 'ftpocvlxa_1130'. Query: 'CREATE TABLE `wp_sndr_mail_users_info` (  `mail_users_info_id` int(10) unsigned NOT NULL AUTO_INCREMENT,  `id_user` int(11) NOT NULL,  `user_email` varchar(255) NOT NULL,  `user_display_name` varchar(255) NOT NULL,  `subscribe` int(1) NOT NULL DEFAULT '1',  PRIMARY KEY (`mail_users_info_id`)) ENGINE=InnoDB AUTO_INCREMENT=9 DEFAULT CHARSET=utf8', Internal MariaDB error code: 1050
 
May 30 15:22:20 sqlserversql01 mysqld: 2018-05-30 15:22:20 140021285766912 [Warning] WSREP: RBR event 1 Query apply warning: 1, 72058305
 
May 30 15:22:20 sqlserversql01 mysqld: 2018-05-30 15:22:20 140021285766912 [Warning] WSREP: Ignoring error for TO isolated action: source: 77b17a1d-5f91-11e8-9ac2-2bd3cdd7a48c version: 3 local: 0 state: APPLYING flags: 65 conn_id: 83955 trx_id: -1 seqnos (l: 3950305, g: 72058305, s: 72058303, d: 72058304, ts: 985075589277288)
 
May 30 15:22:20 sqlserversql01 mysqld: 2018-05-30 15:22:20 140021285766912 [ERROR] Slave SQL: Error 'Table 'wp_sndr_mail_users_info' already exists' on query. Default database: 'ftpocvlxa_1130'. Query: 'CREATE TABLE `wp_sndr_mail_users_info` (  `mail_users_info_id` int(10) unsigned NOT NULL AUTO_INCREMENT,  `id_user` int(11) NOT NULL,  `user_email` varchar(255) NOT NULL,  `user_display_name` varchar(255) NOT NULL,  `subscribe` int(1) NOT NULL DEFAULT '1',  PRIMARY KEY (`mail_users_info_id`)) ENGINE=InnoDB AUTO_INCREMENT=9 DEFAULT CHARSET=utf8', Internal MariaDB error code: 1050
 
May 30 15:22:20 sqlserversql01 mysqld: 2018-05-30 15:22:20 140021285766912 [Warning] WSREP: RBR event 1 Query apply warning: 1, 72058307
 
May 30 15:22:20 sqlserversql01 mysqld: 2018-05-30 15:22:20 140021285766912 [Warning] WSREP: Ignoring error for TO isolated action: source: 77b17a1d-5f91-11e8-9ac2-2bd3cdd7a48c version: 3 local: 0 state: APPLYING flags: 65 conn_id: 83955 trx_id: -1 seqnos (l: 3950307, g: 72058307, s: 72058305, d: 72058306, ts: 985075594915715)
 
May 30 15:22:20 sqlserversql01 mysqld: 2018-05-30 15:22:20 140021285766912 [ERROR] Slave SQL: Error 'Table 'wp_sndr_mail_users_info' already exists' on query. Default database: 'ftpocvlxa_1130'. Query: 'CREATE TABLE `wp_sndr_mail_users_info` (  `mail_users_info_id` int(10) unsigned NOT NULL AUTO_INCREMENT,  `id_user` int(11) NOT NULL,  `user_email` varchar(255) NOT NULL,  `user_display_name` varchar(255) NOT NULL,  `subscribe` int(1) NOT NULL DEFAULT '1',  PRIMARY KEY (`mail_users_info_id`)) ENGINE=InnoDB AUTO_INCREMENT=9 DEFAULT CHARSET=utf8', Internal MariaDB error code: 1050
 
May 30 15:22:20 sqlserversql01 mysqld: 2018-05-30 15:22:20 140021285766912 [Warning] WSREP: RBR event 1 Query apply warning: 1, 72058309
 
May 30 15:22:20 sqlserversql01 mysqld: 2018-05-30 15:22:20 140021285766912 [Warning] WSREP: Ignoring error for TO isolated action: source: 77b17a1d-5f91-11e8-9ac2-2bd3cdd7a48c version: 3 local: 0 state: APPLYING flags: 65 conn_id: 83955 trx_id: -1 seqnos (l: 3950309, g: 72058309, s: 72058307, d: 72058308, ts: 985075599623261)
 
May 30 15:27:23 sqlserversql01 mysqld: 180530 15:27:23 [ERROR] mysqld got signal 11 ;
 
May 30 15:27:23 sqlserversql01 mysqld: This could be because you hit a bug. It is also possible that this binary
 
May 30 15:27:23 sqlserversql01 mysqld: or one of the libraries it was linked against is corrupt, improperly built,
 
May 30 15:27:23 sqlserversql01 mysqld: or misconfigured. This error can also be caused by malfunctioning hardware.
 
May 30 15:27:23 sqlserversql01 mysqld: To report this bug, see https://mariadb.com/kb/en/reporting-bugs
 
May 30 15:27:23 sqlserversql01 mysqld: We will try our best to scrape up some info that will hopefully help
 
May 30 15:27:23 sqlserversql01 mysqld: diagnose the problem, but since we have already crashed,
 
May 30 15:27:23 sqlserversql01 mysqld: something is definitely wrong and this may fail.
 
May 30 15:27:23 sqlserversql01 mysqld: Server version: 10.2.15-MariaDB
 
May 30 15:27:23 sqlserversql01 mysqld: key_buffer_size=134217728
 
May 30 15:27:23 sqlserversql01 mysqld: read_buffer_size=131072
 
May 30 15:27:23 sqlserversql01 mysqld: max_used_connections=14
 
May 30 15:27:23 sqlserversql01 mysqld: max_threads=153
 
May 30 15:27:23 sqlserversql01 mysqld: thread_count=22
 
May 30 15:27:23 sqlserversql01 mysqld: It is possible that mysqld could use up to
 
May 30 15:27:23 sqlserversql01 mysqld: key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 467238 K  bytes of memory
 
May 30 15:27:23 sqlserversql01 mysqld: Hope that's ok; if not, decrease some variables in the equation.
 
May 30 15:27:23 sqlserversql01 mysqld: Thread pointer: 0x7f5920000b08
 
May 30 15:27:23 sqlserversql01 mysqld: Attempting backtrace. You can use the following information to find out
 
May 30 15:27:23 sqlserversql01 mysqld: where mysqld died. If you see no messages after this, something went
 
May 30 15:27:23 sqlserversql01 mysqld: terribly wrong...
 
May 30 15:27:23 sqlserversql01 kernel: mysqld[19405]: segfault at 51 ip 00007f59507d90b8 sp 00007f592dbadc40 error 4 in libgcc_s-4.8.5-20150702.so.1[7f59507ca000+15000]
 
May 30 15:29:58 sqlserversql01 mysqld: 2018-05-30 15:29:58 140015258056896 [Note] /usr/sbin/mysqld (mysqld 10.2.15-MariaDB) starting as process 20738 ...
 
May 30 15:29:58 sqlserversql01 mysqld: 2018-05-30 15:29:58 140015258056896 [Note] WSREP: Read nil XID from storage engines, skipping position init
 
May 30 15:29:58 sqlserversql01 mysqld: 2018-05-30 15:29:58 140015258056896 [Note] WSREP: wsrep_load(): loading provider library '/usr/lib64/galera/libgalera_smm.so'
 
May 30 15:29:58 sqlserversql01 mysqld: 2018-05-30 15:29:58 140015258056896 [Note] WSREP: wsrep_load(): Galera 25.3.23(r3789) by Codership Oy <info@codership.com> loaded successfully.

Comment by Oswaldo Garcia [ 2018-05-31 ]

WSREP forgot the first node and updated the cluster status and then it crashed

May 30 15:27:25 sqlserversql02 mysqld: 2018-05-30 15:27:25 140074981250816 [Note] WSREP: (77b17a1d, 'tcp://0.0.0.0:4567') reconnecting to 1e0f5050 (tcp://10.30.83.49:4567), attempt 0
 
May 30 15:27:29 sqlserversql02 mysqld: 2018-05-30 15:27:29 140074981250816 [Note] WSREP: evs::proto(77b17a1d, OPERATIONAL, view_id(REG,1e0f5050,346)) suspecting node: 1e0f5050
 
May 30 15:27:29 sqlserversql02 mysqld: 2018-05-30 15:27:29 140074981250816 [Note] WSREP: evs::proto(77b17a1d, OPERATIONAL, view_id(REG,1e0f5050,346)) suspected node without join message, declaring inactive
 
May 30 15:27:30 sqlserversql02 mysqld: 2018-05-30 15:27:30 140074981250816 [Note] WSREP: declaring c551023b at tcp://10.30.83.59:4567 stable
 
May 30 15:27:30 sqlserversql02 mysqld: 2018-05-30 15:27:30 140074981250816 [Note] WSREP: Node 77b17a1d state prim
 
May 30 15:27:30 sqlserversql02 mysqld: 2018-05-30 15:27:30 140074981250816 [Note] WSREP: view(view_id(PRIM,77b17a1d,347) memb {
 
May 30 15:27:30 sqlserversql02 mysqld: 77b17a1d,0
 
May 30 15:27:30 sqlserversql02 mysqld: c551023b,0
 
May 30 15:27:30 sqlserversql02 mysqld: } joined {
 
May 30 15:27:30 sqlserversql02 mysqld: } left {
 
May 30 15:27:30 sqlserversql02 mysqld: } partitioned {
 
May 30 15:27:30 sqlserversql02 mysqld: 1e0f5050,0
 
May 30 15:27:30 sqlserversql02 mysqld: })
 
May 30 15:27:30 sqlserversql02 mysqld: 2018-05-30 15:27:30 140074981250816 [Note] WSREP: save pc into disk
 
May 30 15:27:30 sqlserversql02 mysqld: 2018-05-30 15:27:30 140074981250816 [Note] WSREP: forgetting 1e0f5050 (tcp://10.30.83.49:4567)
 
May 30 15:27:30 sqlserversql02 mysqld: 2018-05-30 15:27:30 140074981250816 [Note] WSREP: (77b17a1d, 'tcp://0.0.0.0:4567') turning message relay requesting off
 
May 30 15:27:30 sqlserversql02 mysqld: 2018-05-30 15:27:30 140074970732288 [Note] WSREP: New COMPONENT: primary = yes, bootstrap = no, my_idx = 0, memb_num = 2
 
May 30 15:27:30 sqlserversql02 mysqld: 2018-05-30 15:27:30 140074970732288 [Note] WSREP: STATE_EXCHANGE: sent state UUID: 7e55f01b-643f-11e8-85d0-6658cbe43fd4
 
May 30 15:27:30 sqlserversql02 mysqld: 2018-05-30 15:27:30 140074970732288 [Note] WSREP: STATE EXCHANGE: sent state msg: 7e55f01b-643f-11e8-85d0-6658cbe43fd4
 
May 30 15:27:30 sqlserversql02 mysqld: 2018-05-30 15:27:30 140074970732288 [Note] WSREP: STATE EXCHANGE: got state msg: 7e55f01b-643f-11e8-85d0-6658cbe43fd4 from 0 (sqlserversql02)
 
May 30 15:27:30 sqlserversql02 mysqld: 2018-05-30 15:27:30 140074970732288 [Note] WSREP: STATE EXCHANGE: got state msg: 7e55f01b-643f-11e8-85d0-6658cbe43fd4 from 1 (sqlserversql03)
 
May 30 15:27:30 sqlserversql02 mysqld: 2018-05-30 15:27:30 140074970732288 [Note] WSREP: Quorum results:
 
May 30 15:27:30 sqlserversql02 mysqld: version    = 4,
 
May 30 15:27:30 sqlserversql02 mysqld: component  = PRIMARY,
 
May 30 15:27:30 sqlserversql02 mysqld: conf_id    = 14,
 
May 30 15:27:30 sqlserversql02 mysqld: members    = 2/2 (joined/total),
 
May 30 15:27:30 sqlserversql02 mysqld: act_id     = 72092897,
 
May 30 15:27:30 sqlserversql02 mysqld: last_appl. = 72092873,
 
May 30 15:27:30 sqlserversql02 mysqld: protocols  = 0/8/3 (gcs/repl/appl),
 
May 30 15:27:30 sqlserversql02 mysqld: group UUID = 65cc5487-c983-11e7-b097-eb0d92101a94
 
May 30 15:27:30 sqlserversql02 mysqld: 2018-05-30 15:27:30 140074970732288 [Note] WSREP: Flow-control interval: [23, 23]
 
May 30 15:27:30 sqlserversql02 mysqld: 2018-05-30 15:27:30 140074970732288 [Note] WSREP: Trying to continue unpaused monitor
 
May 30 15:27:30 sqlserversql02 mysqld: 2018-05-30 15:27:30 140075192067840 [Note] WSREP: New cluster view: global state: 65cc5487-c983-11e7-b097-eb0d92101a94:72092897, view# 15: Primary, number of nodes: 2, my index: 0, protocol version 3
 
May 30 15:27:30 sqlserversql02 mysqld: 2018-05-30 15:27:30 140075192067840 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
 
May 30 15:27:30 sqlserversql02 mysqld: 2018-05-30 15:27:30 140075192067840 [Note] WSREP: REPL Protocols: 8 (3, 2)
 
May 30 15:27:30 sqlserversql02 mysqld: 2018-05-30 15:27:30 140075192067840 [Note] WSREP: Assign initial position for certification: 72092897, protocol version: 3
 
May 30 15:27:30 sqlserversql02 mysqld: 2018-05-30 15:27:30 140075217245952 [Note] WSREP: Service thread queue flushed.
 
May 30 15:27:32 sqlserversql02 mysqld: 2018-05-30 15:27:32 140074981250816 [Note] WSREP:  cleaning up 1e0f5050 (tcp://10.30.83.49:4567)
 
May 30 15:27:35 sqlserversql02 mysqld: 180530 15:27:35 [ERROR] mysqld got signal 11 ;
 
May 30 15:27:35 sqlserversql02 mysqld: This could be because you hit a bug. It is also possible that this binary
 
May 30 15:27:35 sqlserversql02 mysqld: or one of the libraries it was linked against is corrupt, improperly built,
 
May 30 15:27:35 sqlserversql02 mysqld: or misconfigured. This error can also be caused by malfunctioning hardware.
 
May 30 15:27:35 sqlserversql02 mysqld: To report this bug, see https://mariadb.com/kb/en/reporting-bugs
 
May 30 15:27:35 sqlserversql02 mysqld: We will try our best to scrape up some info that will hopefully help
 
May 30 15:27:35 sqlserversql02 mysqld: diagnose the problem, but since we have already crashed,
 
May 30 15:27:35 sqlserversql02 mysqld: something is definitely wrong and this may fail.
 
May 30 15:27:35 sqlserversql02 mysqld: Server version: 10.2.15-MariaDB
 
May 30 15:27:35 sqlserversql02 mysqld: key_buffer_size=134217728
 
May 30 15:27:35 sqlserversql02 mysqld: read_buffer_size=131072
 
May 30 15:27:35 sqlserversql02 mysqld: max_used_connections=13
 
May 30 15:27:35 sqlserversql02 mysqld: max_threads=153
 
May 30 15:27:35 sqlserversql02 mysqld: thread_count=21
 
May 30 15:27:35 sqlserversql02 mysqld: It is possible that mysqld could use up to
 
May 30 15:27:35 sqlserversql02 mysqld: key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 467238 K  bytes of memory
 
May 30 15:27:35 sqlserversql02 mysqld: Hope that's ok; if not, decrease some variables in the equation.
 
May 30 15:27:35 sqlserversql02 mysqld: Thread pointer: 0x7f65a800f238
 
May 30 15:27:35 sqlserversql02 mysqld: Attempting backtrace. You can use the following information to find out
 
May 30 15:27:35 sqlserversql02 mysqld: where mysqld died. If you see no messages after this, something went
 
May 30 15:27:35 sqlserversql02 mysqld: terribly wrong...

Comment by Oswaldo Garcia [ 2018-05-31 ]

Third node did the same as 02. At that point we had all three servers down.
Regarding the number of connections, you are right, we don't have too many, this is a brand new application which does not have too much traffic yet.

I don't have a coredump, I will configure the servers to get one next time they crash.

Thanks,

Comment by Oswaldo Garcia [ 2018-05-31 ]

Just adding more information here. I saw the same error with the query in the third node

May 30 15:22:20 sqlserversql03 mysqld: 2018-05-30 15:22:20 140086264608512 [ERROR] Slave SQL: Error 'Table 'wp_sndr_mail_users_info' already exists' on query. Default database: 'ftpocvlxa_1130'. Query: 'CREATE TABLE `wp_sndr_mail_users_info` (  `mail_users_info_id` int(10) unsigned NOT NULL AUTO_INCREMENT,  `id_user` int(11) NOT NULL,  `user_email` varchar(255) NOT NULL,  `user_display_name` varchar(255) NOT NULL,  `subscribe` int(1) NOT NULL DEFAULT '1',  PRIMARY KEY (`mail_users_info_id`)) ENGINE=InnoDB AUTO_INCREMENT=9 DEFAULT CHARSET=utf8', Internal MariaDB error code: 1050
 
May 30 15:22:20 sqlserversql03 mysqld: 2018-05-30 15:22:20 140086264608512 [Warning] WSREP: RBR event 1 Query apply warning: 1, 72058305
 
May 30 15:22:20 sqlserversql03 mysqld: 2018-05-30 15:22:20 140086264608512 [Warning] WSREP: Ignoring error for TO isolated action: source: 77b17a1d-5f91-11e8-9ac2-2bd3cdd7a48c version: 3 local: 0 state: APPLYING flags: 65 conn_id: 83955 trx_id: -1 seqnos (l: 3961828, g: 72058305, s: 72058303, d: 72058304, ts: 985075589277288)
 
May 30 15:22:20 sqlserversql03 mysqld: 2018-05-30 15:22:20 140086264608512 [ERROR] Slave SQL: Error 'Table 'wp_sndr_mail_users_info' already exists' on query. Default database: 'ftpocvlxa_1130'. Query: 'CREATE TABLE `wp_sndr_mail_users_info` (  `mail_users_info_id` int(10) unsigned NOT NULL AUTO_INCREMENT,  `id_user` int(11) NOT NULL,  `user_email` varchar(255) NOT NULL,  `user_display_name` varchar(255) NOT NULL,  `subscribe` int(1) NOT NULL DEFAULT '1',  PRIMARY KEY (`mail_users_info_id`)) ENGINE=InnoDB AUTO_INCREMENT=9 DEFAULT CHARSET=utf8', Internal MariaDB error code: 1050
 
May 30 15:22:20 sqlserversql03 mysqld: 2018-05-30 15:22:20 140086264608512 [Warning] WSREP: RBR event 1 Query apply warning: 1, 72058307
 
May 30 15:22:20 sqlserversql03 mysqld: 2018-05-30 15:22:20 140086264608512 [Warning] WSREP: Ignoring error for TO isolated action: source: 77b17a1d-5f91-11e8-9ac2-2bd3cdd7a48c version: 3 local: 0 state: APPLYING flags: 65 conn_id: 83955 trx_id: -1 seqnos (l: 3961830, g: 72058307, s: 72058305, d: 72058306, ts: 985075594915715)
 
May 30 15:22:20 sqlserversql03 mysqld: 2018-05-30 15:22:20 140086264608512 [ERROR] Slave SQL: Error 'Table 'wp_sndr_mail_users_info' already exists' on query. Default database: 'ftpocvlxa_1130'. Query: 'CREATE TABLE `wp_sndr_mail_users_info` (  `mail_users_info_id` int(10) unsigned NOT NULL AUTO_INCREMENT,  `id_user` int(11) NOT NULL,  `user_email` varchar(255) NOT NULL,  `user_display_name` varchar(255) NOT NULL,  `subscribe` int(1) NOT NULL DEFAULT '1',  PRIMARY KEY (`mail_users_info_id`)) ENGINE=InnoDB AUTO_INCREMENT=9 DEFAULT CHARSET=utf8', Internal MariaDB error code: 1050
 
May 30 15:22:20 sqlserversql03 mysqld: 2018-05-30 15:22:20 140086264608512 [Warning] WSREP: RBR event 1 Query apply warning: 1, 72058309
 
May 30 15:22:20 sqlserversql03 mysqld: 2018-05-30 15:22:20 140086264608512 [Warning] WSREP: Ignoring error for TO isolated action: source: 77b17a1d-5f91-11e8-9ac2-2bd3cdd7a48c version: 3 local: 0 state: APPLYING flags: 65 conn_id: 83955 trx_id: -1 seqnos (l: 3961832, g: 72058309, s: 72058307, d: 72058308, ts: 985075599623261)

The error was about the same time it happened in the first server. Could this be a replication issue?

Comment by Oswaldo Garcia [ 2018-05-31 ]

server.cnf conf in all servers is as follow:

#
 
# These groups are read by MariaDB server.
 
# Use it for options that only the server (but not clients) should see
 
#
 
# See the examples of server my.cnf files in /usr/share/mysql/
 
#
 
 
 
# this is read by the standalone daemon and embedded servers
 
[server]
 
 
 
# this is only for the mysqld standalone daemon
 
[mysqld]
 
 
 
#
 
# * Galera-related settings
 
#
 
[galera]
 
wsrep_on=ON
 
wsrep_provider=/usr/lib64/galera/libgalera_smm.so
 
wsrep_cluster_address='gcomm://sqlserversql01,sqlserversql02,sqlserversql03'
 
wsrep_cluster_name='WP0'
 
wsrep_node_address='sqlserversql03'
 
wsrep_node_name='sqlserversql03'
 
wsrep_sst_method=rsync
 
wsrep_sst_auth=galera:
 
binlog_format=row
 
default_storage_engine=InnoDB
 
innodb_autoinc_lock_mode=2
 
bind-address=0.0.0.0

Comment by Oswaldo Garcia [ 2018-06-07 ]

This one can be closed. There no bugs with the new version. My database needed some tunning and it hasn't crashed again.

Comment by Elena Stepanova [ 2018-06-07 ]

Would you maybe like to share with other users who might experience similar problems what exactly you have tuned to make it go away?

Comment by Oswaldo Garcia [ 2018-06-11 ]

Sure. We discovered that the buffer pool size was too small and we increased it from 1GB to 5GB (our server has 8GB of RAM). Cluster has been running without issues since the change.

Comment by Elena Stepanova [ 2018-07-18 ]

Thanks, good to hear that it helped.
However, there is still a problem, the server certainly shouldn't crash due to too small buffer pool. At worst, it should complain that the buffer pool too small and abort (there is a mechanism for that, and it's known to work). So, I'll keep the report open for now, to see if we can eventually get to the bottom of it.

Comment by Jan Lindström [ 2023-06-06 ]

10.2 is EOL.

Generated at Thu Feb 08 08:28:20 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.