Uploaded image for project: 'MariaDB Server'
  1. MariaDB Server
  2. MDEV-21577

Galera crash after MDL BF-BF conflict

    XMLWordPrintable

Details

    Description

      Server starts crashing after upgrade to Mariadb 10.4.11. We have 3 node galera cluster. Error is probably related to dropping and recreating table, which is used by other node.
      Logs from few last crashes:

      led 24 22:00:19 db2 mysqld[18625]: 2020-01-24 22:00:19 16 [Note] WSREP: MDL BF-BF conflict
      led 24 22:00:19 db2 mysqld[18625]: schema:  s_demo
      led 24 22:00:19 db2 mysqld[18625]: [157B blob data]
      led 24 22:00:19 db2 mysqld[18625]: granted: (11         seqno 804791504         wsrep (toi, exec, committed) cmd 0 1         CREATE TABLE IF NOT EXISTS `shopio_vouchers_products` (
      led 24 22:00:19 db2 mysqld[18625]:   `id` int(11) NOT NULL AUTO_INCREMENT,
      led 24 22:00:19 db2 mysqld[18625]:   `vouchers_id` int(11) NOT NULL,
      led 24 22:00:19 db2 mysqld[18625]:   `products_id` int(11) NOT NULL,
      led 24 22:00:19 db2 mysqld[18625]:   PRIMARY KEY (`id`),
      led 24 22:00:19 db2 mysqld[18625]:   UNIQUE KEY `unique_ids` (`vouchers_id`,`products_id`),
      led 24 22:00:19 db2 mysqld[18625]:   KEY `products_id` (`products_id`),
      led 24 22:00:19 db2 mysqld[18625]:   CONSTRAINT FOREIGN KEY (`products_id`) REFERENCES `shopio_products` (`id`) ON DELETE CASCADE ON UPDATE CASCADE,
      led 24 22:00:19 db2 mysqld[18625]:   CONSTRAINT FOREIGN KEY (`vouchers_id`) REFERENCES `shopio_vouchers` (`id`) ON DELETE CASCADE ON UPDATE CASCADE
      led 24 22:00:19 db2 mysqld[18625]: ) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4)
      led 24 22:00:19 db2 mysqld[18625]: 2020-01-24 22:00:19 16 [ERROR] Aborting
      led 24 22:08:32 db2 systemd[1]: Stopping MariaDB 10.4.11 database server...
      led 24 22:08:32 db2 mysqld[18625]: 2020-01-24 22:08:32 0 [Warning] WSREP: server: db2 unallowed state transition: disconnecting -> disconnecting
      led 24 22:08:32 db2 mysqld[18625]: mysqld: /home/buildbot/buildbot/build/mariadb-10.4.11/wsrep-lib/src/server_state.cpp:1340: void wsrep::server_state::state(wsrep::unique_lock<wsrep::mutex>&, wsrep::server_state::state): Assertion `0' failed.
      led 24 22:08:32 db2 mysqld[18625]: %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%200124 22:08:32 [ERROR] mysqld got signal 6 ;
      led 24 22:08:32 db2 mysqld[18625]: This could be because you hit a bug. It is also possible that this binary
      led 24 22:08:32 db2 mysqld[18625]: or one of the libraries it was linked against is corrupt, improperly built,
      led 24 22:08:32 db2 mysqld[18625]: or misconfigured. This error can also be caused by malfunctioning hardware.
      led 24 22:08:32 db2 mysqld[18625]: To report this bug, see https://mariadb.com/kb/en/reporting-bugs
      led 24 22:08:32 db2 mysqld[18625]: We will try our best to scrape up some info that will hopefully help
      led 24 22:08:32 db2 mysqld[18625]: diagnose the problem, but since we have already crashed,
      led 24 22:08:32 db2 mysqld[18625]: something is definitely wrong and this may fail.
      led 24 22:08:32 db2 mysqld[18625]: Server version: 10.4.11-MariaDB-1:10.4.11+maria~stretch-log
      led 24 22:08:32 db2 mysqld[18625]: key_buffer_size=1048576
      led 24 22:08:32 db2 mysqld[18625]: read_buffer_size=262144
      led 24 22:08:32 db2 mysqld[18625]: max_used_connections=44
      led 24 22:08:32 db2 mysqld[18625]: max_threads=2327
      led 24 22:08:32 db2 mysqld[18625]: thread_count=21
      led 24 22:08:32 db2 mysqld[18625]: It is possible that mysqld could use up to
      led 24 22:08:32 db2 mysqld[18625]: key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 5419764 K  bytes of memory
      led 24 22:08:32 db2 mysqld[18625]: Hope that's ok; if not, decrease some variables in the equation.
      led 24 22:08:32 db2 mysqld[18625]: Thread pointer: 0x0
      led 24 22:08:32 db2 mysqld[18625]: Attempting backtrace. You can use the following information to find out
      led 24 22:08:32 db2 mysqld[18625]: where mysqld died. If you see no messages after this, something went
      led 24 22:08:32 db2 mysqld[18625]: terribly wrong...
      led 24 22:08:32 db2 mysqld[18625]: stack_bottom = 0x0 thread_stack 0x31000
      led 24 22:08:32 db2 mysqld[18625]: *** buffer overflow detected ***: /usr/sbin/mysqld terminated
      led 24 22:08:32 db2 mysqld[18625]: ======= Backtrace: =========
      led 24 22:08:32 db2 mysqld[18625]: /lib/x86_64-linux-gnu/libc.so.6(+0x70bfb)[0x7f39ad77cbfb]
      led 24 22:08:32 db2 mysqld[18625]: /lib/x86_64-linux-gnu/libc.so.6(__fortify_fail+0x37)[0x7f39ad805437]
      led 24 22:08:32 db2 mysqld[18625]: /lib/x86_64-linux-gnu/libc.so.6(+0xf7570)[0x7f39ad803570]
      led 24 22:08:32 db2 mysqld[18625]: /lib/x86_64-linux-gnu/libc.so.6(+0xf93aa)[0x7f39ad8053aa]
      led 24 22:08:32 db2 mysqld[18625]: /usr/sbin/mysqld(my_addr_resolve+0xe2)[0x559a16f0f492]
      

      led 25 02:00:21 db2 mysqld[11890]: 2020-01-25  2:00:21 1 [Warning] WSREP: BF applier failed to open_and_lock_tables: 1146, fatal: 0 wsrep = (exec_mode: 2 conflict_state: 0 seqno: 805439671)
      led 25 02:00:21 db2 mysqld[11890]: 2020-01-25  2:00:21 1 [ERROR] Slave SQL: Error executing row event: 'Table 's_demo.shopio_vouchers_products' doesn't exist', Internal MariaDB error code: 1146
      led 25 02:00:21 db2 mysqld[11890]: 2020-01-25  2:00:21 1 [Warning] WSREP: Event 23 Update_rows_v1 apply failed: 1146, seqno 805439671
      led 25 02:00:21 db2 mysqld[11890]: 2020-01-25  2:00:21 1 [ERROR] WSREP: Failed to apply write set: gtid: 54078096-9d91-11e9-8472-a7c937a2aaa8:805439671 server_id: 34aa6dfd-3d7d-11ea-9f02-1e9305220e61 client_id: 1
      led 25 02:00:21 db2 mysqld[11890]: 2020-01-25  2:00:21 1 [Note] WSREP: Closing send monitor...
      led 25 02:00:21 db2 mysqld[11890]: 2020-01-25  2:00:21 1 [Note] WSREP: Closed send monitor.
      led 25 02:00:21 db2 mysqld[11890]: 2020-01-25  2:00:21 1 [Note] WSREP: gcomm: terminating thread
      led 25 02:00:21 db2 mysqld[11890]: 2020-01-25  2:00:21 1 [Note] WSREP: gcomm: joining thread
      led 25 02:00:21 db2 mysqld[11890]: 2020-01-25  2:00:21 1 [Note] WSREP: gcomm: closing backend
      

      led 25 07:00:19 db2 mysqld[17264]: 2020-01-25  7:00:19 16 [Note] WSREP: MDL BF-BF conflict
      led 25 07:00:19 db2 mysqld[17264]: schema:  s_demo
      led 25 07:00:19 db2 mysqld[17264]: request: (16         seqno 806396466         wsrep (toi, exec, committed) cmd 0 3         /*!40000 ALTER TABLE `shopio_vouchers_products` DISABLE KEYS */)
      led 25 07:00:19 db2 mysqld[17264]: [157B blob data]
      led 25 07:00:19 db2 mysqld[17264]: 2020-01-25  7:00:19 16 [ERROR] Aborting
      led 25 07:10:24 db2 systemd[1]: Stopping MariaDB 10.4.11 database server...
      led 25 07:10:24 db2 mysqld[17264]: 2020-01-25  7:10:24 0 [Warning] WSREP: server: db2 unallowed state transition: disconnecting -> disconnecting
      led 25 07:10:24 db2 mysqld[17264]: mysqld: /home/buildbot/buildbot/build/mariadb-10.4.11/wsrep-lib/src/server_state.cpp:1340: void wsrep::server_state::state(wsrep::unique_lock<wsrep::mutex>&, wsrep::server_state
      led 25 07:10:24 db2 mysqld[17264]: 200125  7:10:24 [ERROR] mysqld got signal 6 ;
      led 25 07:10:24 db2 mysqld[17264]: This could be because you hit a bug. It is also possible that this binary
      led 25 07:10:24 db2 mysqld[17264]: or one of the libraries it was linked against is corrupt, improperly built,
      led 25 07:10:24 db2 mysqld[17264]: or misconfigured. This error can also be caused by malfunctioning hardware.
      led 25 07:10:24 db2 mysqld[17264]: To report this bug, see https://mariadb.com/kb/en/reporting-bugs
      led 25 07:10:24 db2 mysqld[17264]: We will try our best to scrape up some info that will hopefully help
      led 25 07:10:24 db2 mysqld[17264]: diagnose the problem, but since we have already crashed,
      led 25 07:10:24 db2 mysqld[17264]: something is definitely wrong and this may fail.
      led 25 07:10:24 db2 mysqld[17264]: Server version: 10.4.11-MariaDB-1:10.4.11+maria~stretch-log
      led 25 07:10:24 db2 mysqld[17264]: key_buffer_size=1048576
      led 25 07:10:24 db2 mysqld[17264]: read_buffer_size=262144
      led 25 07:10:24 db2 mysqld[17264]: max_used_connections=39
      led 25 07:10:24 db2 mysqld[17264]: max_threads=2327
      led 25 07:10:24 db2 mysqld[17264]: thread_count=26
      led 25 07:10:24 db2 mysqld[17264]: It is possible that mysqld could use up to
      led 25 07:10:24 db2 mysqld[17264]: key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 5419764 K  bytes of memory
      

      Attachments

        1. node1.txt
          17 kB
        2. node2.txt
          0.9 kB

        Issue Links

          Activity

            People

              jplindst Jan Lindström (Inactive)
              Fišer Dominik Fišer
              Votes:
              3 Vote for this issue
              Watchers:
              10 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Git Integration

                  Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.