Uploaded image for project: 'MariaDB Server'
  1. MariaDB Server
  2. MDEV-25427

InnoDB: Conflicting lock on table crash

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Closed (View Workflow)
    • Priority: Blocker
    • Resolution: Fixed
    • Affects Version/s: 10.5.9
    • Fix Version/s: 10.5.10
    • Component/s: Galera
    • Labels:
      None

      Description

      One of our nodes crashed today (3nodes master-master-master setup) because of lock. It seems based on https://jira.mariadb.org/browse/MDEV-24229 there were fixes in 10.5.9 version, however we hit it on 10.5.9 version now.

      2021-04-15 10:28:53 1 [ERROR] InnoDB: Conflicting lock on table: `storage`.`svm` index: PRIMARY that has lock
      RECORD LOCKS space id 325 page no 44 n bits 128 index PRIMARY of table `storage`.`svm` trx id 10709511 lock mode S locks rec but not gap
      Record lock, heap no 24 PHYSICAL RECORD: n_fields 16; compact format; info bits 0
       0: len 8; hex 80000000000008c9; asc         ;;
       1: len 6; hex 000000a369e7; asc     i ;;
       2: len 7; hex 64000001f52dc6; asc d    - ;;
       3: len 30; hex 32383164316164632d313065342d663339622d613237622d336531656638; asc 281d1adc-10e4-f39b-a27b-3e1ef8; (total 36 bytes);
       4: len 4; hex 5f85287f; asc _ ( ;;
       5: len 4; hex 6078154b; asc `x K;;
       6: SQL NULL;
       7: len 30; hex 65376165623235392d306430392d313165622d383030342d643033396561; asc e7aeb259-0d09-11eb-8004-d039ea; (total 36 bytes);
       8: len 30; hex 73766d5f3266383665306538643836613131656161336532333662666663; asc svm_2f86e0e8d86a11eaa3e236bffc; (total 45 bytes);
       9: len 30; hex 697073706163655f32663836653065386438366131316561613365323336; asc ipspace_2f86e0e8d86a11eaa3e236; (total 45 bytes);
       10: len 8; hex 8000000000000003; asc         ;;
       11: len 8; hex 8000000000000014; asc         ;;
       12: len 30; hex 70726f6a656374732f3232313033393533303635332f676c6f62616c2f6e; asc projects/221039530653/global/n; (total 45 bytes);
       13: SQL NULL;
       14: SQL NULL;
       15: len 0; hex ; asc ;;
       
      2021-04-15 10:28:53 1 [ERROR] InnoDB: WSREP state:
      2021-04-15 10:28:53 1 [ERROR] WSREP: Thread BF trx_id: 10709528 thread: 1 seqno: 17823110 client_state: exec client_mode: high priority transaction_mode: replaying applier: 1 toi: 0 local: 0 query: UPDATE `svm` SET `uuid` = ?, `created_at` = ?, `updated_at` = ?, `deleted_at` = ?, `external_uuid` = ?, `name` = ?, `ip_space` = ?, `host_id` = ?, `account_id` = ?, `vendor_subnet_id` = ?, `azure_key_vault_id` = ?, `external_azure_key_vault_uuid` = ?, `app_type` = ?  WHERE `svm`.`deleted_at` IS NULL AND `svm`.`id` = ?Zx`
      2021-04-15 10:28:53 1 [ERROR] WSREP: Thread BF trx_id: 10709511 thread: 10 seqno: 17823105 client_state: exec client_mode: high priority transaction_mode: executing applier: 1 toi: 0 local: 0 query: UPDATE `volume` SET `uuid` = ?, `created_at` = ?, `updated_at` = ?, `deleted_at` = ?, `external_uuid` = ?, `vendor_id` = ?, `vendor_subnet_id` = ?, `creation_token` = ?, `display_name` = ?, `external_name` = ?, `service_level` = ?, `security_style` = ?, `snap_reserve` = ?, `snapshot_directory` = ?, `quota_in_bytes` = ?, `size_in_bytes` = ?, `snapshot_size_in_bytes` = ?, `inodes_total` = ?, `inodes_used` = ?, `inodes_quota` = ?, `protocol_types` = ?, `smb_share_settings` = ?, `life_cycle_state` = ?, `life_cycle_state_details` = ?, `allocation_id` = ?, `pool_id` = ?, `svm_id` = ?, `account_id` = ?, `split_job_id` = ?, `clone_snapshot_id` = ?, `is_data_protection` = ?, `throughput` = ?, `storage_class` = ?, `unix_permissions` = ?, `kerberos_enabled` = ?, `network_proximity_info` = ?, `network_proximity` = ?, `placement_properties` = ?, `ldap_ena
      2021-04-15 10:28:53 0x7f52687df700  InnoDB: Assertion failure in file /home/buildbot/buildbot/build/mariadb-10.5.9/storage/innobase/lock/lock0lock.cc line 674
      InnoDB: We intentionally generate a memory trap.
      InnoDB: Submit a detailed bug report to https://jira.mariadb.org/
      InnoDB: If you get repeated assertion failures or crashes, even
      InnoDB: immediately after the mysqld startup, there may be
      InnoDB: corruption in the InnoDB tablespace. Please refer to
      InnoDB: https://mariadb.com/kb/en/library/innodb-recovery-modes/
      InnoDB: about forcing recovery.
      210415 10:28:53 [ERROR] mysqld got signal 6 ;
      This could be because you hit a bug. It is also possible that this binary
      or one of the libraries it was linked against is corrupt, improperly built,
      or misconfigured. This error can also be caused by malfunctioning hardware.
       
      To report this bug, see https://mariadb.com/kb/en/reporting-bugs
       
      We will try our best to scrape up some info that will hopefully help
      diagnose the problem, but since we have already crashed,
      something is definitely wrong and this may fail.
       
      Server version: 10.5.9-MariaDB-1:10.5.9+maria~focal
      key_buffer_size=629145600
      read_buffer_size=131072
      max_used_connections=48
      max_threads=352
      thread_count=59
      It is possible that mysqld could use up to
      key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 1389275 K  bytes of memory
      Hope that's ok; if not, decrease some variables in the equation.
       
      Thread pointer: 0x7f524c08c598
      Attempting backtrace. You can use the following information to find out
      where mysqld died. If you see no messages after this, something went
      terribly wrong...
      stack_bottom = 0x7f52687ded58 thread_stack 0x49000
      2021-04-15 10:28:53 9819 [Warning] Aborted connection 9819 to db: 'unconnected' user: 'qstack_usage' host: '10.255.4.138' (Got an error reading communication packets)
      2021-04-15 10:28:53 9823 [Warning] Aborted connection 9823 to db: 'unconnected' user: 'qstack_usage' host: '10.255.4.138' (Got an error reading communication packets)
      2021-04-15 10:28:53 9830 [Warning] Aborted connection 9830 to db: 'unconnected' user: 'qstack_usage' host: '10.255.4.138' (Got an error reading communication packets)
      2021-04-15 10:28:53 9825 [Warning] Aborted connection 9825 to db: 'unconnected' user: 'qstack_usage' host: '10.255.4.138' (Got an error reading communication packets)
      2021-04-15 10:28:53 9847 [Warning] IP address '10.255.6.129' could not be resolved: Name or service not known
      mysqld(my_print_stacktrace+0x32)[0x55720eb70af2]
      Printing to addr2line failed
      mysqld(handle_fatal_signal+0x485)[0x55720e5c66f5]
      /lib/x86_64-linux-gnu/libpthread.so.0(+0x153c0)[0x7f527473d3c0]
      /lib/x86_64-linux-gnu/libc.so.6(gsignal+0xcb)[0x7f527424418b]
      /lib/x86_64-linux-gnu/libc.so.6(abort+0x12b)[0x7f5274223859]
      mysqld(+0x62db08)[0x55720e288b08]
      mysqld(+0x608f07)[0x55720e263f07]
      mysqld(+0xcbcdf5)[0x55720e917df5]
      mysqld(+0xcc3551)[0x55720e91e551]
      mysqld(+0xd4dfc4)[0x55720e9a8fc4]
      mysqld(+0xc75db6)[0x55720e8d0db6]
      mysqld(_ZN7handler10ha_rnd_posEPhS0_+0x1c5)[0x55720e5cd685]
      mysqld(+0x8b4040)[0x55720e50f040]
      mysqld(_ZN14Rows_log_event8find_rowEP14rpl_group_info+0x461)[0x55720e6f0c91]
      mysqld(_ZN21Update_rows_log_event11do_exec_rowEP14rpl_group_info+0xee)[0x55720e6f12de]
      mysqld(_ZN14Rows_log_event14do_apply_eventEP14rpl_group_info+0x287)[0x55720e6e54f7]
      mysqld(_Z18wsrep_apply_eventsP3THDP14Relay_log_infoPKvm+0x1e9)[0x55720e89a909]
      mysqld(_ZN22Wsrep_replayer_service15apply_write_setERKN5wsrep7ws_metaERKNS0_12const_bufferERNS0_14mutable_bufferE+0xac)[0x55720e8815fc]
      mysqld(+0xf994e0)[0x55720ebf44e0]
      mysqld(_ZN5wsrep12server_state8on_applyERNS_21high_priority_serviceERKNS_9ws_handleERKNS_7ws_metaERKNS_12const_bufferE+0xc1)[0x55720ebf5531]
      mysqld(+0xfab88c)[0x55720ec0688c]
      /usr/lib/galera/libgalera_smm.so(+0x6241b)[0x7f526bd3441b]
      /usr/lib/galera/libgalera_smm.so(+0x764bd)[0x7f526bd484bd]
      /usr/lib/galera/libgalera_smm.so(+0x541c6)[0x7f526bd261c6]
      

        galera.cnf: |
          [galera]
          user = mysql
          bind-address = 0.0.0.0
          default_storage_engine = InnoDB
          binlog_format = ROW
          innodb_autoinc_lock_mode = 2
          innodb_flush_log_at_trx_commit = 2
          query_cache_size = 0
          query_cache_type = 0
       
          # MariaDB Galera settings
          wsrep_on=ON
          wsrep_provider=/usr/lib/galera/libgalera_smm.so
          wsrep_sst_method=rsync
          wsrep_slave_threads=8
          wsrep_sync_wait=7
       
          # Cluster settings (automatically updated)
          wsrep_cluster_address=gcomm://
          wsrep_cluster_name=mysql
          wsrep_node_address=127.0.0.1
        mariadb.cnf: "[client]\ndefault-character-set = utf8\n[mysqld]\nunix_socket = OFF\nperformance_schema
          = ON\ncharacter-set-server = utf8\ncollation-server = utf8_general_ci\nignore-db-dirs
          = lost+found \nmax_connections = 350\ninteractive_timeout = 450 \nwait_timeout
          = 450\njoin_buffer_size = 524288\nkey_buffer_size = 600MB\n# InnoDB tuning\n innodb_log_file_size
          = 600MB\ninnodb_buffer_pool_size = 8400MB\ncore-file\ninnodb_log_buffer_size =
          67108864\n"
      

        Attachments

          Issue Links

            Activity

              People

              Assignee:
              jplindst Jan Lindström
              Reporter:
              jaroslav Jaroslav
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:

                  Git Integration