Details
-
Bug
-
Status: Stalled (View Workflow)
-
Critical
-
Resolution: Unresolved
-
10.3.9, 10.3.21, 10.4.12, 10.3.23
Description
A three nodes Galera cluster behind MaxScale routing gets all three nodes crashed on a prepared statement (SELECT).
The application is in Java and uses mysql-connector-java-5.1.45.jar.
First node crashes at `11:57:51` and the other two at `11:57:58`
What I still don't understand is why Node 1 and 3 crash at the same time, I'd expect a bit of delay between the two.
To be noted that MaxScale (afaik) pre-sends prepared statements to all nodes.
Server version: 10.3.9-MariaDB-1:10.3.9+maria~xenial-log
_THE SELECT spotted to be causing the crash_
SELECT
|
t.id,
|
t.org_id,
|
t.int_id,
|
t.ext_id,
|
t.ext_name,
|
t.type,
|
t.flags,
|
t.aut,
|
t.ins
|
FROM
|
myTableName t
|
WHERE
|
t.org_id = 9 |
AND
|
type = ?
|
AND
|
int_id = ?
|
MAXSCALE (first routes the query to Node2, then it tries Node1 and Node3 apparently)
_NODE 2 CRASH_
2018-09-28 11:57:53 error : Monitor was unable to connect to server [192.168.12.-92-]:3306 : "Can't connect to MySQL server on '192.168.12.-92-' (115)"
|
2018-09-28 11:57:53 notice : Server changed state: devdb2[192.168.12.92:3306]: slave_down. [Slave, Synced, Running] -> [Down]
|
|
_NODE 1 & 3 CRASH_
2018-09-28 11:57:59 error : Monitor was unable to connect to server [192.168.12.-91-]:3306 : "Lost connection to MySQL server at 'handshake: reading initial communication packet', system error: 107"
|
2018-09-28 11:57:59 error : Monitor was unable to connect to server [192.168.12.-93-]:3306 : "Can't connect to MySQL server on '192.168.12.-93-' (115)"
|
2018-09-28 11:57:59 error : [galeramon] There are no cluster members
|
2018-09-28 11:57:59 notice : Server changed state: devdb1[192.168.12.-91-:3306]: master_down. [Master, Synced, Running] -> [Down]
|
2018-09-28 11:57:59 notice : Server changed state: devdb3[192.168.12.-93-:3306]: slave_down. [Slave, Synced, Running] -> [Down]
|
_ONE STACK TRACE (from Node 2):_
Sep 28 11:57:51 galera2 mysqld[21934]: terribly wrong...
|
Sep 28 11:57:51 galera2 mysqld[21934]: stack_bottom = 0x7f7b0ab46cf8 thread_stack 0x49000
|
Sep 28 11:57:51 galera2 mysqld[21934]: /usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x561af1385ace]
|
Sep 28 11:57:51 galera2 mysqld[21934]: /usr/sbin/mysqld(handle_fatal_signal+0x347)[0x561af0e2b8f7]
|
Sep 28 11:57:51 galera2 mysqld[21934]: /lib/x86_64-linux-gnu/libpthread.so.0(+0x11390)[0x7f7c628d5390]
|
Sep 28 11:57:51 galera2 mysqld[21934]: /usr/sbin/mysqld(append_query_string(charset_info_st const*, String*, char const*, unsigned long, bool)+0x5d)[0x561af0f0d6fd]
|
Sep 28 11:57:51 galera2 mysqld[21934]: /usr/sbin/mysqld(Item_param::value_query_val_str(THD*, String*) const+0xac)[0x561af0e4a8bc]
|
Sep 28 11:57:51 galera2 mysqld[21934]: /usr/sbin/mysqld(Item_param::append_for_log(THD*, String*)+0x63)[0x561af0e4ac43]
|
Sep 28 11:57:51 galera2 mysqld[21934]: /usr/sbin/mysqld(+0x58c7d1)[0x561af0c647d1]
|
Sep 28 11:57:51 galera2 mysqld[21934]: /usr/sbin/mysqld(Prepared_statement::set_parameters(String*, unsigned char*, unsigned char*)+0x6d)[0x561af0c67e9d]
|
Sep 28 11:57:51 galera2 mysqld[21934]: /usr/sbin/mysqld(Prepared_statement::execute_loop(String*, bool, unsigned char*, unsigned char*)+0x36)[0x561af0c6a376]
|
Sep 28 11:57:51 galera2 mysqld[21934]: /usr/sbin/mysqld(+0x592f48)[0x561af0c6af48]
|
Sep 28 11:57:51 galera2 mysqld[21934]: /usr/sbin/mysqld(mysqld_stmt_execute(THD*, char*, unsigned int)+0x25)[0x561af0c6b085]
|
Sep 28 11:57:51 galera2 mysqld[21934]: /usr/sbin/mysqld(dispatch_command(enum_server_command, THD*, char*, unsigned int, bool, bool)+0x14a0)[0x561af0c581a0]
|
Sep 28 11:57:51 galera2 mysqld[21934]: /usr/sbin/mysqld(do_command(THD*)+0x13c)[0x561af0c5944c]
|
Sep 28 11:57:51 galera2 mysqld[21934]: /usr/sbin/mysqld(do_handle_one_connection(CONNECT*)+0x252)[0x561af0d29fd2]
|
Sep 28 11:57:51 galera2 mysqld[21934]: /usr/sbin/mysqld(handle_one_connection+0x3d)[0x561af0d2a14d]
|
Sep 28 11:57:51 galera2 mysqld[21934]: /lib/x86_64-linux-gnu/libpthread.so.0(+0x76ba)[0x7f7c628cb6ba]
|
Sep 28 11:57:51 galera2 mysqld[21934]: /lib/x86_64-linux-gnu/libc.so.6(clone+0x6d)[0x7f7c61f7641d]
|
_THE THREE STACK TRACES ORDERED BY POSITION:_
Sep 28 11:57:58 galera3 mysqld[]: /usr/sbin/mysqld(+0x58c7d1)[0x555e956dc7d1]
|
Sep 28 11:57:58 galera1 mysqld[]: /usr/sbin/mysqld(+0x58c7d1)[0x55aa91c7b7d1]
|
Sep 28 11:57:51 galera2 mysqld[]: /usr/sbin/mysqld(+0x58c7d1)[0x561af0c647d1]
|
Sep 28 11:57:51 galera2 mysqld[]: /usr/sbin/mysqld(append_query_string(charset_info_st const*, String*, char const*, unsigned long, bool)+0x5d)[0
|
Sep 28 11:57:58 galera1 mysqld[]: /usr/sbin/mysqld(append_query_string(charset_info_st const*, String*, char const*, unsigned long, bool)+0x5d)[0
|
Sep 28 11:57:58 galera3 mysqld[]: /usr/sbin/mysqld(append_query_string(charset_info_st const*, String*, char const*, unsigned long, bool)+0x5d)[0x
|
Sep 28 11:57:58 galera3 mysqld[]: /usr/sbin/mysqld(Item_param::append_for_log(THD*, String*)+0x63)[0x555e958c2c43]
|
Sep 28 11:57:58 galera1 mysqld[]: /usr/sbin/mysqld(Item_param::append_for_log(THD*, String*)+0x63)[0x55aa91e61c43]
|
Sep 28 11:57:51 galera2 mysqld[]: /usr/sbin/mysqld(Item_param::append_for_log(THD*, String*)+0x63)[0x561af0e4ac43]
|
Sep 28 11:57:58 galera3 mysqld[]: /usr/sbin/mysqld(Item_param::value_query_val_str(THD*, String*) const+0xac)[0x555e958c28bc]
|
Sep 28 11:57:58 galera1 mysqld[]: /usr/sbin/mysqld(Item_param::value_query_val_str(THD*, String*) const+0xac)[0x55aa91e618bc]
|
Sep 28 11:57:51 galera2 mysqld[]: /usr/sbin/mysqld(Item_param::value_query_val_str(THD*, String*) const+0xac)[0x561af0e4a8bc]
|
Sep 28 11:57:58 galera3 mysqld[]: /usr/sbin/mysqld(Prepared_statement::execute_loop(String*, bool, unsigned char*, unsigned char*)+0x36)[0x555e956
|
Sep 28 11:57:58 galera1 mysqld[]: /usr/sbin/mysqld(Prepared_statement::execute_loop(String*, bool, unsigned char*, unsigned char*)+0x36)[0x55aa91
|
Sep 28 11:57:58 galera3 mysqld[]: /usr/sbin/mysqld(Prepared_statement::set_parameters(String*, unsigned char*, unsigned char*)+0x6d)[0x555e956dfe9
|
Sep 28 11:57:58 galera1 mysqld[]: /usr/sbin/mysqld(Prepared_statement::set_parameters(String*, unsigned char*, unsigned char*)+0x6d)[0x55aa91c7ee
|
Sep 28 11:57:51 galera2 mysqld[]: /usr/sbin/mysqld(Prepared_statement::set_parameters(String*, unsigned char*, unsigned char*)+0x6d)[0x561af0c67e
|
|
Attachments
Issue Links
- relates to
-
-
- Closed
-
-
-
- Closed
-