InnoDB: Conflicting lock on table crash

2021-04-15 10:28:53 1 [ERROR] InnoDB: Conflicting lock on table: `storage`.`svm` index: PRIMARY that has lock

RECORD LOCKS space id 325 page no 44 n bits 128 index PRIMARY of table `storage`.`svm` trx id 10709511 lock mode S locks rec but not gap

Record lock, heap no 24 PHYSICAL RECORD: n_fields 16; compact format; info bits 0

 0: len 8; hex 80000000000008c9; asc         ;;

 1: len 6; hex 000000a369e7; asc     i ;;

 2: len 7; hex 64000001f52dc6; asc d    - ;;

 3: len 30; hex 32383164316164632d313065342d663339622d613237622d336531656638; asc 281d1adc-10e4-f39b-a27b-3e1ef8; (total 36 bytes);

 4: len 4; hex 5f85287f; asc _ ( ;;

 5: len 4; hex 6078154b; asc `x K;;

 6: SQL NULL;

 7: len 30; hex 65376165623235392d306430392d313165622d383030342d643033396561; asc e7aeb259-0d09-11eb-8004-d039ea; (total 36 bytes);

 8: len 30; hex 73766d5f3266383665306538643836613131656161336532333662666663; asc svm_2f86e0e8d86a11eaa3e236bffc; (total 45 bytes);

 9: len 30; hex 697073706163655f32663836653065386438366131316561613365323336; asc ipspace_2f86e0e8d86a11eaa3e236; (total 45 bytes);

 10: len 8; hex 8000000000000003; asc         ;;

 11: len 8; hex 8000000000000014; asc         ;;

 12: len 30; hex 70726f6a656374732f3232313033393533303635332f676c6f62616c2f6e; asc projects/221039530653/global/n; (total 45 bytes);

 13: SQL NULL;

 14: SQL NULL;

 15: len 0; hex ; asc ;;

2021-04-15 10:28:53 1 [ERROR] InnoDB: WSREP state:

2021-04-15 10:28:53 1 [ERROR] WSREP: Thread BF trx_id: 10709528 thread: 1 seqno: 17823110 client_state: exec client_mode: high priority transaction_mode: replaying applier: 1 toi: 0 local: 0 query: UPDATE `svm` SET `uuid` = ?, `created_at` = ?, `updated_at` = ?, `deleted_at` = ?, `external_uuid` = ?, `name` = ?, `ip_space` = ?, `host_id` = ?, `account_id` = ?, `vendor_subnet_id` = ?, `azure_key_vault_id` = ?, `external_azure_key_vault_uuid` = ?, `app_type` = ?  WHERE `svm`.`deleted_at` IS NULL AND `svm`.`id` = ?Zx`

2021-04-15 10:28:53 1 [ERROR] WSREP: Thread BF trx_id: 10709511 thread: 10 seqno: 17823105 client_state: exec client_mode: high priority transaction_mode: executing applier: 1 toi: 0 local: 0 query: UPDATE `volume` SET `uuid` = ?, `created_at` = ?, `updated_at` = ?, `deleted_at` = ?, `external_uuid` = ?, `vendor_id` = ?, `vendor_subnet_id` = ?, `creation_token` = ?, `display_name` = ?, `external_name` = ?, `service_level` = ?, `security_style` = ?, `snap_reserve` = ?, `snapshot_directory` = ?, `quota_in_bytes` = ?, `size_in_bytes` = ?, `snapshot_size_in_bytes` = ?, `inodes_total` = ?, `inodes_used` = ?, `inodes_quota` = ?, `protocol_types` = ?, `smb_share_settings` = ?, `life_cycle_state` = ?, `life_cycle_state_details` = ?, `allocation_id` = ?, `pool_id` = ?, `svm_id` = ?, `account_id` = ?, `split_job_id` = ?, `clone_snapshot_id` = ?, `is_data_protection` = ?, `throughput` = ?, `storage_class` = ?, `unix_permissions` = ?, `kerberos_enabled` = ?, `network_proximity_info` = ?, `network_proximity` = ?, `placement_properties` = ?, `ldap_ena

2021-04-15 10:28:53 0x7f52687df700  InnoDB: Assertion failure in file /home/buildbot/buildbot/build/mariadb-10.5.9/storage/innobase/lock/lock0lock.cc line 674

InnoDB: We intentionally generate a memory trap.

InnoDB: Submit a detailed bug report to https://jira.mariadb.org/

InnoDB: If you get repeated assertion failures or crashes, even

InnoDB: immediately after the mysqld startup, there may be

InnoDB: corruption in the InnoDB tablespace. Please refer to

InnoDB: https://mariadb.com/kb/en/library/innodb-recovery-modes/

InnoDB: about forcing recovery.

210415 10:28:53 [ERROR] mysqld got signal 6 ;

This could be because you hit a bug. It is also possible that this binary

or one of the libraries it was linked against is corrupt, improperly built,

or misconfigured. This error can also be caused by malfunctioning hardware.

To report this bug, see https://mariadb.com/kb/en/reporting-bugs

We will try our best to scrape up some info that will hopefully help

diagnose the problem, but since we have already crashed,

something is definitely wrong and this may fail.

Server version: 10.5.9-MariaDB-1:10.5.9+maria~focal

key_buffer_size=629145600

read_buffer_size=131072

max_used_connections=48

max_threads=352

thread_count=59

It is possible that mysqld could use up to

key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 1389275 K  bytes of memory

Hope that's ok; if not, decrease some variables in the equation.

Thread pointer: 0x7f524c08c598

Attempting backtrace. You can use the following information to find out

where mysqld died. If you see no messages after this, something went

terribly wrong...

stack_bottom = 0x7f52687ded58 thread_stack 0x49000

2021-04-15 10:28:53 9819 [Warning] Aborted connection 9819 to db: 'unconnected' user: 'qstack_usage' host: '10.255.4.138' (Got an error reading communication packets)

2021-04-15 10:28:53 9823 [Warning] Aborted connection 9823 to db: 'unconnected' user: 'qstack_usage' host: '10.255.4.138' (Got an error reading communication packets)

2021-04-15 10:28:53 9830 [Warning] Aborted connection 9830 to db: 'unconnected' user: 'qstack_usage' host: '10.255.4.138' (Got an error reading communication packets)

2021-04-15 10:28:53 9825 [Warning] Aborted connection 9825 to db: 'unconnected' user: 'qstack_usage' host: '10.255.4.138' (Got an error reading communication packets)

2021-04-15 10:28:53 9847 [Warning] IP address '10.255.6.129' could not be resolved: Name or service not known

mysqld(my_print_stacktrace+0x32)[0x55720eb70af2]

Printing to addr2line failed

mysqld(handle_fatal_signal+0x485)[0x55720e5c66f5]

/lib/x86_64-linux-gnu/libpthread.so.0(+0x153c0)[0x7f527473d3c0]

/lib/x86_64-linux-gnu/libc.so.6(gsignal+0xcb)[0x7f527424418b]

/lib/x86_64-linux-gnu/libc.so.6(abort+0x12b)[0x7f5274223859]

mysqld(+0x62db08)[0x55720e288b08]

mysqld(+0x608f07)[0x55720e263f07]

mysqld(+0xcbcdf5)[0x55720e917df5]

mysqld(+0xcc3551)[0x55720e91e551]

mysqld(+0xd4dfc4)[0x55720e9a8fc4]

mysqld(+0xc75db6)[0x55720e8d0db6]

mysqld(_ZN7handler10ha_rnd_posEPhS0_+0x1c5)[0x55720e5cd685]

mysqld(+0x8b4040)[0x55720e50f040]

mysqld(_ZN14Rows_log_event8find_rowEP14rpl_group_info+0x461)[0x55720e6f0c91]

mysqld(_ZN21Update_rows_log_event11do_exec_rowEP14rpl_group_info+0xee)[0x55720e6f12de]

mysqld(_ZN14Rows_log_event14do_apply_eventEP14rpl_group_info+0x287)[0x55720e6e54f7]

mysqld(_Z18wsrep_apply_eventsP3THDP14Relay_log_infoPKvm+0x1e9)[0x55720e89a909]

mysqld(_ZN22Wsrep_replayer_service15apply_write_setERKN5wsrep7ws_metaERKNS0_12const_bufferERNS0_14mutable_bufferE+0xac)[0x55720e8815fc]

mysqld(+0xf994e0)[0x55720ebf44e0]

mysqld(_ZN5wsrep12server_state8on_applyERNS_21high_priority_serviceERKNS_9ws_handleERKNS_7ws_metaERKNS_12const_bufferE+0xc1)[0x55720ebf5531]

mysqld(+0xfab88c)[0x55720ec0688c]

/usr/lib/galera/libgalera_smm.so(+0x6241b)[0x7f526bd3441b]

/usr/lib/galera/libgalera_smm.so(+0x764bd)[0x7f526bd484bd]

/usr/lib/galera/libgalera_smm.so(+0x541c6)[0x7f526bd261c6]

  galera.cnf: |

    [galera]

    user = mysql

    bind-address = 0.0.0.0

    default_storage_engine = InnoDB

    binlog_format = ROW

    innodb_autoinc_lock_mode = 2

    innodb_flush_log_at_trx_commit = 2

    query_cache_size = 0

    query_cache_type = 0

    # MariaDB Galera settings

    wsrep_on=ON

    wsrep_provider=/usr/lib/galera/libgalera_smm.so

    wsrep_sst_method=rsync

    wsrep_slave_threads=8

    wsrep_sync_wait=7

    # Cluster settings (automatically updated)

    wsrep_cluster_address=gcomm://

    wsrep_cluster_name=mysql

    wsrep_node_address=127.0.0.1

  mariadb.cnf: "[client]\ndefault-character-set = utf8\n[mysqld]\nunix_socket = OFF\nperformance_schema

    = ON\ncharacter-set-server = utf8\ncollation-server = utf8_general_ci\nignore-db-dirs

    = lost+found \nmax_connections = 350\ninteractive_timeout = 450 \nwait_timeout

    = 450\njoin_buffer_size = 524288\nkey_buffer_size = 600MB\n# InnoDB tuning\n innodb_log_file_size

    = 600MB\ninnodb_buffer_pool_size = 8400MB\ncore-file\ninnodb_log_buffer_size =

    67108864\n"