Details
Description
Server starts crashing after upgrade to Mariadb 10.4.11. We have 3 node galera cluster. Error is probably related to dropping and recreating table, which is used by other node.
Logs from few last crashes:
led 24 22:00:19 db2 mysqld[18625]: 2020-01-24 22:00:19 16 [Note] WSREP: MDL BF-BF conflict
|
led 24 22:00:19 db2 mysqld[18625]: schema: s_demo
|
led 24 22:00:19 db2 mysqld[18625]: [157B blob data]
|
led 24 22:00:19 db2 mysqld[18625]: granted: (11 seqno 804791504 wsrep (toi, exec, committed) cmd 0 1 CREATE TABLE IF NOT EXISTS `shopio_vouchers_products` (
|
led 24 22:00:19 db2 mysqld[18625]: `id` int(11) NOT NULL AUTO_INCREMENT,
|
led 24 22:00:19 db2 mysqld[18625]: `vouchers_id` int(11) NOT NULL,
|
led 24 22:00:19 db2 mysqld[18625]: `products_id` int(11) NOT NULL,
|
led 24 22:00:19 db2 mysqld[18625]: PRIMARY KEY (`id`),
|
led 24 22:00:19 db2 mysqld[18625]: UNIQUE KEY `unique_ids` (`vouchers_id`,`products_id`),
|
led 24 22:00:19 db2 mysqld[18625]: KEY `products_id` (`products_id`),
|
led 24 22:00:19 db2 mysqld[18625]: CONSTRAINT FOREIGN KEY (`products_id`) REFERENCES `shopio_products` (`id`) ON DELETE CASCADE ON UPDATE CASCADE,
|
led 24 22:00:19 db2 mysqld[18625]: CONSTRAINT FOREIGN KEY (`vouchers_id`) REFERENCES `shopio_vouchers` (`id`) ON DELETE CASCADE ON UPDATE CASCADE
|
led 24 22:00:19 db2 mysqld[18625]: ) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4)
|
led 24 22:00:19 db2 mysqld[18625]: 2020-01-24 22:00:19 16 [ERROR] Aborting
|
led 24 22:08:32 db2 systemd[1]: Stopping MariaDB 10.4.11 database server...
|
led 24 22:08:32 db2 mysqld[18625]: 2020-01-24 22:08:32 0 [Warning] WSREP: server: db2 unallowed state transition: disconnecting -> disconnecting
|
led 24 22:08:32 db2 mysqld[18625]: mysqld: /home/buildbot/buildbot/build/mariadb-10.4.11/wsrep-lib/src/server_state.cpp:1340: void wsrep::server_state::state(wsrep::unique_lock<wsrep::mutex>&, wsrep::server_state::state): Assertion `0' failed.
|
led 24 22:08:32 db2 mysqld[18625]: %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%200124 22:08:32 [ERROR] mysqld got signal 6 ;
|
led 24 22:08:32 db2 mysqld[18625]: This could be because you hit a bug. It is also possible that this binary
|
led 24 22:08:32 db2 mysqld[18625]: or one of the libraries it was linked against is corrupt, improperly built,
|
led 24 22:08:32 db2 mysqld[18625]: or misconfigured. This error can also be caused by malfunctioning hardware.
|
led 24 22:08:32 db2 mysqld[18625]: To report this bug, see https://mariadb.com/kb/en/reporting-bugs
|
led 24 22:08:32 db2 mysqld[18625]: We will try our best to scrape up some info that will hopefully help
|
led 24 22:08:32 db2 mysqld[18625]: diagnose the problem, but since we have already crashed,
|
led 24 22:08:32 db2 mysqld[18625]: something is definitely wrong and this may fail.
|
led 24 22:08:32 db2 mysqld[18625]: Server version: 10.4.11-MariaDB-1:10.4.11+maria~stretch-log
|
led 24 22:08:32 db2 mysqld[18625]: key_buffer_size=1048576
|
led 24 22:08:32 db2 mysqld[18625]: read_buffer_size=262144
|
led 24 22:08:32 db2 mysqld[18625]: max_used_connections=44
|
led 24 22:08:32 db2 mysqld[18625]: max_threads=2327
|
led 24 22:08:32 db2 mysqld[18625]: thread_count=21
|
led 24 22:08:32 db2 mysqld[18625]: It is possible that mysqld could use up to
|
led 24 22:08:32 db2 mysqld[18625]: key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 5419764 K bytes of memory
|
led 24 22:08:32 db2 mysqld[18625]: Hope that's ok; if not, decrease some variables in the equation.
|
led 24 22:08:32 db2 mysqld[18625]: Thread pointer: 0x0
|
led 24 22:08:32 db2 mysqld[18625]: Attempting backtrace. You can use the following information to find out
|
led 24 22:08:32 db2 mysqld[18625]: where mysqld died. If you see no messages after this, something went
|
led 24 22:08:32 db2 mysqld[18625]: terribly wrong...
|
led 24 22:08:32 db2 mysqld[18625]: stack_bottom = 0x0 thread_stack 0x31000
|
led 24 22:08:32 db2 mysqld[18625]: *** buffer overflow detected ***: /usr/sbin/mysqld terminated
|
led 24 22:08:32 db2 mysqld[18625]: ======= Backtrace: =========
|
led 24 22:08:32 db2 mysqld[18625]: /lib/x86_64-linux-gnu/libc.so.6(+0x70bfb)[0x7f39ad77cbfb]
|
led 24 22:08:32 db2 mysqld[18625]: /lib/x86_64-linux-gnu/libc.so.6(__fortify_fail+0x37)[0x7f39ad805437]
|
led 24 22:08:32 db2 mysqld[18625]: /lib/x86_64-linux-gnu/libc.so.6(+0xf7570)[0x7f39ad803570]
|
led 24 22:08:32 db2 mysqld[18625]: /lib/x86_64-linux-gnu/libc.so.6(+0xf93aa)[0x7f39ad8053aa]
|
led 24 22:08:32 db2 mysqld[18625]: /usr/sbin/mysqld(my_addr_resolve+0xe2)[0x559a16f0f492]
|
led 25 02:00:21 db2 mysqld[11890]: 2020-01-25 2:00:21 1 [Warning] WSREP: BF applier failed to open_and_lock_tables: 1146, fatal: 0 wsrep = (exec_mode: 2 conflict_state: 0 seqno: 805439671)
|
led 25 02:00:21 db2 mysqld[11890]: 2020-01-25 2:00:21 1 [ERROR] Slave SQL: Error executing row event: 'Table 's_demo.shopio_vouchers_products' doesn't exist', Internal MariaDB error code: 1146
|
led 25 02:00:21 db2 mysqld[11890]: 2020-01-25 2:00:21 1 [Warning] WSREP: Event 23 Update_rows_v1 apply failed: 1146, seqno 805439671
|
led 25 02:00:21 db2 mysqld[11890]: 2020-01-25 2:00:21 1 [ERROR] WSREP: Failed to apply write set: gtid: 54078096-9d91-11e9-8472-a7c937a2aaa8:805439671 server_id: 34aa6dfd-3d7d-11ea-9f02-1e9305220e61 client_id: 1
|
led 25 02:00:21 db2 mysqld[11890]: 2020-01-25 2:00:21 1 [Note] WSREP: Closing send monitor...
|
led 25 02:00:21 db2 mysqld[11890]: 2020-01-25 2:00:21 1 [Note] WSREP: Closed send monitor.
|
led 25 02:00:21 db2 mysqld[11890]: 2020-01-25 2:00:21 1 [Note] WSREP: gcomm: terminating thread
|
led 25 02:00:21 db2 mysqld[11890]: 2020-01-25 2:00:21 1 [Note] WSREP: gcomm: joining thread
|
led 25 02:00:21 db2 mysqld[11890]: 2020-01-25 2:00:21 1 [Note] WSREP: gcomm: closing backend
|
led 25 07:00:19 db2 mysqld[17264]: 2020-01-25 7:00:19 16 [Note] WSREP: MDL BF-BF conflict
|
led 25 07:00:19 db2 mysqld[17264]: schema: s_demo
|
led 25 07:00:19 db2 mysqld[17264]: request: (16 seqno 806396466 wsrep (toi, exec, committed) cmd 0 3 /*!40000 ALTER TABLE `shopio_vouchers_products` DISABLE KEYS */)
|
led 25 07:00:19 db2 mysqld[17264]: [157B blob data]
|
led 25 07:00:19 db2 mysqld[17264]: 2020-01-25 7:00:19 16 [ERROR] Aborting
|
led 25 07:10:24 db2 systemd[1]: Stopping MariaDB 10.4.11 database server...
|
led 25 07:10:24 db2 mysqld[17264]: 2020-01-25 7:10:24 0 [Warning] WSREP: server: db2 unallowed state transition: disconnecting -> disconnecting
|
led 25 07:10:24 db2 mysqld[17264]: mysqld: /home/buildbot/buildbot/build/mariadb-10.4.11/wsrep-lib/src/server_state.cpp:1340: void wsrep::server_state::state(wsrep::unique_lock<wsrep::mutex>&, wsrep::server_state
|
led 25 07:10:24 db2 mysqld[17264]: 200125 7:10:24 [ERROR] mysqld got signal 6 ;
|
led 25 07:10:24 db2 mysqld[17264]: This could be because you hit a bug. It is also possible that this binary
|
led 25 07:10:24 db2 mysqld[17264]: or one of the libraries it was linked against is corrupt, improperly built,
|
led 25 07:10:24 db2 mysqld[17264]: or misconfigured. This error can also be caused by malfunctioning hardware.
|
led 25 07:10:24 db2 mysqld[17264]: To report this bug, see https://mariadb.com/kb/en/reporting-bugs
|
led 25 07:10:24 db2 mysqld[17264]: We will try our best to scrape up some info that will hopefully help
|
led 25 07:10:24 db2 mysqld[17264]: diagnose the problem, but since we have already crashed,
|
led 25 07:10:24 db2 mysqld[17264]: something is definitely wrong and this may fail.
|
led 25 07:10:24 db2 mysqld[17264]: Server version: 10.4.11-MariaDB-1:10.4.11+maria~stretch-log
|
led 25 07:10:24 db2 mysqld[17264]: key_buffer_size=1048576
|
led 25 07:10:24 db2 mysqld[17264]: read_buffer_size=262144
|
led 25 07:10:24 db2 mysqld[17264]: max_used_connections=39
|
led 25 07:10:24 db2 mysqld[17264]: max_threads=2327
|
led 25 07:10:24 db2 mysqld[17264]: thread_count=26
|
led 25 07:10:24 db2 mysqld[17264]: It is possible that mysqld could use up to
|
led 25 07:10:24 db2 mysqld[17264]: key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 5419764 K bytes of memory
|
Attachments
Issue Links
- relates to
-
MDEV-24119 MDL BF-BF Conflict caused by TRUNCATE TABLE
- Closed
-
MDEV-23997 truncate is crashing mariadb galera cluster slave nodes
- Closed