Uploaded image for project: 'MariaDB Server'
  1. MariaDB Server
  2. MDEV-23033

All slaves crash once in ~24 hours and loop restart with signal 11

Details

    Description

      Hello.
      We have a Master-to-multi-Slave config and are regularly facing slaves crashes (they crash together at the same time, approx. once every 24 hours). Maybe this has something to do with load, but crashed did not happen under heavy load and still happened today with relatively (50%) of highest load we've had so far.

      Master 10.4.12 setup is Ubuntu 18.0.4 cloud VPS (yandex cloud) with 6 cores 100% quota, 6 Gbs RAM, 30 Gbs space.
      Slaves 10.4.12 and 10.4.13 setup is Ubuntu 18.0.4 cloud VPS (yandex cloud) with 4 cores 100% quota, 4 Gbs RAM, 30 Gbs space.

      Replication is setup using this guide - https://mariadb.com/kb/en/setting-up-a-replication-slave-with-mariabackup/
      Everything works well, until slaves crash.

      Currently Web API sends DML statements to Master directly and SELECT statements are sent to HAProxy with wieght distribution 20/40/40 (Master/Slave1/Slave2).
      Before that we had Master 10.4.12 and only 1 Slave 10.4.12 with direct requests from API to Slave without HAProxy - Slave still crashed, I hoped with 2 slaves we would at least have 1 standing... but alas...

      We removed all triggers from the DB.
      We use GTIDs.
      We now use mixed binlog (but id did crash with 'row' mode).
      We don't have any DDL statements running (except a few CREATE TEMPORARY TABLE).
      We use InnoDB only to store data.
      We don't have any FullText Indexes.
      We don't use compression.
      We don't use SSL between Mater and Slaves.
      Timezone on all machines is UTC.
      Data size now approximately 800Mb, InnoDb buffer pool size is 1024Mb, InnoDb log file size is128Mb.

      Here's syslog:

      Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: 200627 12:44:00 [ERROR] mysqld got signal 11 ;
      		 
      Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: This could be because you hit a bug. It is also possible that this binary
      		 
      Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: or one of the libraries it was linked against is corrupt, improperly built,
      		 
      Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: or misconfigured. This error can also be caused by malfunctioning hardware.
      		 
      Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: To report this bug, see https://mariadb.com/kb/en/reporting-bugs
      		 
      Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: We will try our best to scrape up some info that will hopefully help
      		 
      Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: diagnose the problem, but since we have already crashed,
      		 
      Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: something is definitely wrong and this may fail.
      		 
      Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: Server version: 10.4.12-MariaDB-1:10.4.12+maria~bionic-log
      		 
      Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: key_buffer_size=134217728
      		 
      Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: read_buffer_size=2097152
      		 
      Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: max_used_connections=15
      		 
      Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: max_threads=77
      		 
      Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: thread_count=24
      		 
      Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: It is possible that mysqld could use up to
      		 
      Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 606038 K  bytes of memory
      		 
      Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: Hope that's ok; if not, decrease some variables in the equation.
      		 
      Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: Thread pointer: 0x7f40500014f8
      		 
      Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: Attempting backtrace. You can use the following information to find out
      		 
      Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: where mysqld died. If you see no messages after this, something went
      		 
      Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: terribly wrong...
      		 
      Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: stack_bottom = 0x7f40fc901e98 thread_stack 0x49000
      		 
      Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: /usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x55b8a063ac3e]
      		 
      Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: /usr/sbin/mysqld(handle_fatal_signal+0x515)[0x55b8a00aecf5]
      		 
      Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: /lib/x86_64-linux-gnu/libpthread.so.0(+0x12890)[0x7f411d583890]
      		 
      Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: /lib/x86_64-linux-gnu/libc.so.6(+0x18f039)[0x7f411c008039]
      		 
      Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: /usr/sbin/mysqld(+0xae4170)[0x55b8a031e170]
      		 
      Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: /usr/sbin/mysqld(+0xa205dd)[0x55b8a025a5dd]
      		 
      Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: /usr/sbin/mysqld(+0xb10a85)[0x55b8a034aa85]
      		 
      Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: /usr/sbin/mysqld(+0xb116be)[0x55b8a034b6be]
      		 
      Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: /usr/sbin/mysqld(+0xb1224c)[0x55b8a034c24c]
      		 
      Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: /usr/sbin/mysqld(+0xae7959)[0x55b8a0321959]
      		 
      Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: /usr/sbin/mysqld(+0xad0db6)[0x55b8a030adb6]
      		 
      Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: /usr/sbin/mysqld(+0xb0b658)[0x55b8a0345658]
      		 
      Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: /usr/sbin/mysqld(+0xb1173f)[0x55b8a034b73f]
      		 
      Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: /usr/sbin/mysqld(+0xb1224c)[0x55b8a034c24c]
      		 
      Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: /usr/sbin/mysqld(+0xae73a4)[0x55b8a03213a4]
      		 
      Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: /usr/sbin/mysqld(+0xa21f8f)[0x55b8a025bf8f]
      		 
      Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: /usr/sbin/mysqld(_ZN7handler13ha_delete_rowEPKh+0x9f)[0x55b8a00ba7af]
      		 
      Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: /usr/sbin/mysqld(_ZN21Delete_rows_log_event11do_exec_rowEP14rpl_group_info+0x12d)[0x55b8a01b12ad]
      		 
      Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: /usr/sbin/mysqld(_ZN14Rows_log_event14do_apply_eventEP14rpl_group_info+0x25d)[0x55b8a01a440d]
      		 
      Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: /usr/sbin/mysqld(+0x5b6e90)[0x55b89fdf0e90]
      		 
      Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: /usr/sbin/mysqld(handle_slave_sql+0x17b4)[0x55b89fdfa344]
      		 
      Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: /usr/sbin/mysqld(+0xdab79a)[0x55b8a05e579a]
      		 
      Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: /lib/x86_64-linux-gnu/libpthread.so.0(+0x76db)[0x7f411d5786db]
      		 
      Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: /lib/x86_64-linux-gnu/libc.so.6(clone+0x3f)[0x7f411bf9a88f]
      		 
      Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: Trying to get some variables.
      		 
      Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: Some pointers may be invalid and cause the dump to abort.
      		 
      Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: Query (0x0):
      		 
      Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: Connection ID (thread ID): 49
      		 
      Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: Status: NOT_KILLED
      		 
      Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=on,condition_pushdown_from_having=on
      		 
      Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
      		 
      Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: information that should help you find out what is causing the crash.
      		 
      Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: Writing a core file...
      		 
      Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: Working directory at /var/lib/mysql
      		 
      Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: Resource Limits:
      		 
      Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: Limit                     Soft Limit           Hard Limit           Units
      		 
      Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: Max cpu time              unlimited            unlimited            seconds
      		 
      Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: Max file size             unlimited            unlimited            bytes
      		 
      Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: Max data size             unlimited            unlimited            bytes
      		 
      Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: Max stack size            8388608              unlimited            bytes
      		 
      Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: Max core file size        0                    unlimited            bytes
      		 
      Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: Max resident set          unlimited            unlimited            bytes
      		 
      Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: Max processes             15602                15602                processes
      		 
      Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: Max open files            16364                16364                files
      		 
      Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: Max locked memory         16777216             16777216             bytes
      		 
      Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: Max address space         unlimited            unlimited            bytes
      		 
      Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: Max file locks            unlimited            unlimited            locks
      		 
      Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: Max pending signals       15602                15602                signals
      		 
      Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: Max msgqueue size         819200               819200               bytes
      		 
      Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: Max nice priority         0                    0
      		 
      Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: Max realtime priority     0                    0
      		 
      Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: Max realtime timeout      unlimited            unlimited            us
      		 
      Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: Core pattern: core
      		 
      Jun 27 12:44:00 vox-db1-read1 kernel: [264684.731198] mysqld[7861]: segfault at 115 ip 00007f411c008039 sp 00007f40fc8fcfc8 error 6 in libc-2.27.so[7f411be79000+1e7000]
      		 
      Jun 27 12:44:00 vox-db1-read1 systemd[1]: mariadb.service: Main process exited, code=killed, status=11/SEGV
      		 
      Jun 27 12:44:00 vox-db1-read1 systemd[1]: mariadb.service: Failed with result 'signal'.
      		 
      Jun 27 12:44:05 vox-db1-read1 systemd[1]: mariadb.service: Service hold-off time over, scheduling restart.
      		 
      Jun 27 12:44:05 vox-db1-read1 systemd[1]: mariadb.service: Scheduled restart job, restart counter is at 1.
      		 
      Jun 27 12:44:05 vox-db1-read1 systemd[1]: Stopped MariaDB 10.4.12 database server.
      		 
      Jun 27 12:44:05 vox-db1-read1 systemd[1]: Starting MariaDB 10.4.12 database server...
      		 
      Jun 27 12:44:06 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:06 0 [Note] /usr/sbin/mysqld (mysqld 10.4.12-MariaDB-1:10.4.12+maria~bionic-log) starting as process 12132 ...
      		 
      Jun 27 12:44:06 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:06 0 [Note] InnoDB: Using Linux native AIO
      		 
      Jun 27 12:44:06 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:06 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
      		 
      Jun 27 12:44:06 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:06 0 [Note] InnoDB: Uses event mutexes
      		 
      Jun 27 12:44:06 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:06 0 [Note] InnoDB: Compressed tables use zlib 1.2.11
      		 
      Jun 27 12:44:06 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:06 0 [Note] InnoDB: Number of pools: 1
      		 
      Jun 27 12:44:06 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:06 0 [Note] InnoDB: Using SSE2 crc32 instructions
      		 
      Jun 27 12:44:06 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:06 0 [Note] mysqld: O_TMPFILE is not supported on /tmp (disabling future attempts)
      		 
      Jun 27 12:44:06 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:06 0 [Note] InnoDB: Initializing buffer pool, total size = 1G, instances = 8, chunk size = 128M
      		 
      Jun 27 12:44:06 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:06 0 [Note] InnoDB: Completed initialization of buffer pool
      		 
      Jun 27 12:44:06 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:06 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority().
      		 
      Jun 27 12:44:07 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:07 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=142473402785
      		 
      Jun 27 12:44:10 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:10 0 [Note] InnoDB: 1 transaction(s) which must be rolled back or cleaned up in total 2 row operations to undo
      		 
      Jun 27 12:44:10 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:10 0 [Note] InnoDB: Trx id counter is 11134299
      		 
      Jun 27 12:44:10 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:10 0 [Note] InnoDB: Starting final batch to recover 63 pages from redo log.
      		 
      Jun 27 12:44:10 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:10 0 [Note] InnoDB: Last binlog file '/var/log/mysql/mariadb-bin.000634', position 59120658
      		 
      Jun 27 12:44:10 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:10 0 [Note] InnoDB: 128 out of 128 rollback segments are active.
      		 
      Jun 27 12:44:10 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:10 0 [Note] InnoDB: Starting in background the rollback of recovered transactions
      		 
      Jun 27 12:44:10 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:10 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1"
      		 
      Jun 27 12:44:10 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:10 0 [Note] InnoDB: Creating shared tablespace for temporary tables
      		 
      Jun 27 12:44:10 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:10 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ...
      		 
      Jun 27 12:44:10 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:10 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB.
      		 
      Jun 27 12:44:10 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:10 0 [Note] InnoDB: 10.4.12 started; log sequence number 142473413250; transaction id 11134301
      		 
      Jun 27 12:44:10 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:10 0 [Note] InnoDB: Loading buffer pool(s) from /var/lib/mysql/ib_buffer_pool
      		 
      Jun 27 12:44:10 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:10 0 [Note] Plugin 'FEEDBACK' is disabled.
      		 
      Jun 27 12:44:10 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:10 0 [Note] Recovering after a crash using /var/log/mysql/mariadb-bin
      		 
      Jun 27 12:44:10 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:10 0 [Note] Starting crash recovery...
      		 
      Jun 27 12:44:10 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:10 0 [Note] Crash recovery finished.
      		 
      Jun 27 12:44:10 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:10 0 [Note] InnoDB: Rolled back recovered transaction 11134298
      		 
      Jun 27 12:44:10 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:10 0 [Note] InnoDB: Rollback of non-prepared transactions completed
      		 
      Jun 27 12:44:10 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:10 0 [Note] Server socket created on IP: '0.0.0.0'.
      		 
      Jun 27 12:44:10 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:10 0 [Note] Reading of all Master_info entries succeeded
      		 
      Jun 27 12:44:10 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:10 0 [Note] Added new Master_info '' to hash table
      		 
      Jun 27 12:44:10 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:10 0 [Warning] Neither --relay-log nor --relay-log-index were used; so replication may break when this MySQL server acts as a slave and has his hostname changed!! Please use '--log-basename=#' or '--relay-log=mysqld-relay-bin' to avoid this problem.
      		 
      Jun 27 12:44:11 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:11 10 [Note] Slave I/O thread: Start asynchronous replication to master 'replication_user@10.130.0.25:3306' in log 'mariadb-bin.000712' at position 4
      		 
      Jun 27 12:44:11 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:11 11 [Note] Slave SQL thread initialized, starting replication in log 'mariadb-bin.000712' at position 4, relay log './mysqld-relay-bin.000001' position: 4; GTID position '0-1-5332814'
      		 
      Jun 27 12:44:11 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:11 0 [Note] /usr/sbin/mysqld: ready for connections.
      		 
      Jun 27 12:44:11 vox-db1-read1 mysqld[12132]: Version: '10.4.12-MariaDB-1:10.4.12+maria~bionic-log'  socket: '/var/run/mysqld/mysqld.sock'  port: 3306  mariadb.org binary distribution
      		 
      Jun 27 12:44:11 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:11 10 [Note] Slave I/O thread: connected to master 'replication_user@10.130.0.25:3306',replication starts at GTID position '0-1-5332814'
      		 
      Jun 27 12:44:11 vox-db1-read1 systemd[1]: Started MariaDB 10.4.12 database server.
      		 
      Jun 27 12:44:11 vox-db1-read1 /etc/mysql/debian-start[12175]: Upgrading MySQL tables if necessary.
      		 
      Jun 27 12:44:11 vox-db1-read1 /etc/mysql/debian-start[12178]: Looking for 'mysql' as: /usr/bin/mysql
      		 
      Jun 27 12:44:11 vox-db1-read1 /etc/mysql/debian-start[12178]: Looking for 'mysqlcheck' as: /usr/bin/mysqlcheck
      		 
      Jun 27 12:44:11 vox-db1-read1 /etc/mysql/debian-start[12178]: This installation of MariaDB is already upgraded to 10.4.12-MariaDB, use --force if you still need to run mysql_upgrade
      		 
      Jun 27 12:44:11 vox-db1-read1 /etc/mysql/debian-start[12186]: Checking for insecure root accounts.
      		 
      Jun 27 12:44:11 vox-db1-read1 /etc/mysql/debian-start[12190]: Triggering myisam-recover for all MyISAM tables and aria-recover for all Aria tables
      		 
      Jun 27 12:44:11 vox-db1-read1 mysqld[12132]: 200627 12:44:11 [ERROR] mysqld got signal 11 ;
      		 
      Jun 27 12:44:11 vox-db1-read1 mysqld[12132]: This could be because you hit a bug. It is also possible that this binary
      		 
      Jun 27 12:44:11 vox-db1-read1 mysqld[12132]: or one of the libraries it was linked against is corrupt, improperly built,
      		 
      Jun 27 12:44:11 vox-db1-read1 mysqld[12132]: or misconfigured. This error can also be caused by malfunctioning hardware.
      		 
      Jun 27 12:44:11 vox-db1-read1 mysqld[12132]: To report this bug, see https://mariadb.com/kb/en/reporting-bugs
      		 
      Jun 27 12:44:11 vox-db1-read1 mysqld[12132]: We will try our best to scrape up some info that will hopefully help
      		 
      Jun 27 12:44:11 vox-db1-read1 mysqld[12132]: diagnose the problem, but since we have already crashed,
      		 
      Jun 27 12:44:11 vox-db1-read1 mysqld[12132]: something is definitely wrong and this may fail.
      		 
      Jun 27 12:44:11 vox-db1-read1 mysqld[12132]: Server version: 10.4.12-MariaDB-1:10.4.12+maria~bionic-log
      		 
      Jun 27 12:44:11 vox-db1-read1 mysqld[12132]: key_buffer_size=134217728
      		 
      Jun 27 12:44:11 vox-db1-read1 mysqld[12132]: read_buffer_size=2097152
      		 
      Jun 27 12:44:11 vox-db1-read1 mysqld[12132]: max_used_connections=1

      ...and so on, it loops indefinitely.

      My theory is that something comes from Master that breaks Slaves.
      When first crash occured (same day I've setup replication, ~4-5 days ago) I tried to restore Slave from same backup I used to initilize it. And Slave could not be started. I suppose while catching up to Master it still encountered this breaking transaction and crashed. Then I just took a fresh backup from Master, and Slave started without problems.

      Since then every time Slaves crash I just take make backup (with mariabackup) and restore slaves. This is stupid I know, but what choice do we currently have?

      What additional information can I provide you with to find the root of this issue?

      Thanks!

      Attachments

        Issue Links

          is duplicated by

          Bug - A problem which impairs or prevents the functions of the product. MDEV-24326 MariaDB crash loop on DELETE

          • Critical - Crashes, loss of data, severe memory leak.
          • Closed
          relates to

          Bug - A problem which impairs or prevents the functions of the product. MDEV-16856 Crash

          • Major - Major loss of function.
          • Closed

          Bug - A problem which impairs or prevents the functions of the product. MDEV-21153 Galera: Replica nodes crash with signal 6 after indexed virtual columns and FK cascading deletes: WSREP has not yet prepared node for application use

          • Major - Major loss of function.
          • Closed

          Activity

            Ascending order - Click to sort in descending order
            Grigoriy Grigoriy created issue -
            Grigoriy Grigoriy made changes -
            Field Original Value New Value
            Environment Ubuntu 18.0.4
            Master 6 cores/6Gb and 2 Slaves 4 cores/4Gbs             		Ubuntu 18.0.4
            Master 6 cores/6Gb and 2 Slaves 4 cores/4Gbs
            Yandex Cloud VPS
            Grigoriy Grigoriy made changes -
            Description Hello.
            We have a Master-to-multi-Slave config and are regularly facing slaves crashes (they crash together at the same time, approx. once every 24 hours). Maybe this has something to do with load, but crashed did not happen under heavy load and still happened today with relatively (50%) of highest load we've had so far.

            Master 10.4.12 setup is Ubuntu 18.0.4 cloud VPS (yandex cloud) with 6 cores 100% quota, 6 Gbs RAM, 30 Gbs space.
            Slaves 10.4.12 and 10.4.13 setup is Ubuntu 18.0.4 cloud VPS (yandex cloud) with 4 cores 100% quota, 4 Gbs RAM, 30 Gbs space.

            Replication is setup using this guide - https://mariadb.com/kb/en/setting-up-a-replication-slave-with-mariabackup/
            Everything works well, until slaves crash.

            Currently Web API sends DML statements to Master directly and SELECT statements are sent to HAProxy with wieght distribution 20/40/40 (Master/Slave1/Slave2).
            Before that we had Master 10.4.12 and only 1 Slave 10.4.12 with direct requests from API to Slave without HAProxy - Slave still crashed, I hoped with 2 slaves we would at least have 1 standing... but alas...

            We removed all triggers from the DB.
            We use GTIDs.
            We don't have any DDL statements running (except a few CREATE TEMPORARY TABLE).
            We use InnoDB only to store data.
            We don't have any FullText Indexes.
            We don't use compression.
            We don't use SSL between Mater and Slaves.
            Timezone on all machines is UTC.

            Here's syslog:

            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: 200627 12:44:00 [ERROR] mysqld got signal 11 ;
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: This could be because you hit a bug. It is also possible that this binary
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: or one of the libraries it was linked against is corrupt, improperly built,
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: or misconfigured. This error can also be caused by malfunctioning hardware.
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: To report this bug, see https://mariadb.com/kb/en/reporting-bugs
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: We will try our best to scrape up some info that will hopefully help
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: diagnose the problem, but since we have already crashed,
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: something is definitely wrong and this may fail.
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: Server version: 10.4.12-MariaDB-1:10.4.12+maria~bionic-log
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: key_buffer_size=134217728
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: read_buffer_size=2097152
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: max_used_connections=15
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: max_threads=77
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: thread_count=24
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: It is possible that mysqld could use up to
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 606038 K bytes of memory
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: Hope that's ok; if not, decrease some variables in the equation.
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: Thread pointer: 0x7f40500014f8
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: Attempting backtrace. You can use the following information to find out
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: where mysqld died. If you see no messages after this, something went
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: terribly wrong...
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: stack_bottom = 0x7f40fc901e98 thread_stack 0x49000
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: /usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x55b8a063ac3e]
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: /usr/sbin/mysqld(handle_fatal_signal+0x515)[0x55b8a00aecf5]
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: /lib/x86_64-linux-gnu/libpthread.so.0(+0x12890)[0x7f411d583890]
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: /lib/x86_64-linux-gnu/libc.so.6(+0x18f039)[0x7f411c008039]
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: /usr/sbin/mysqld(+0xae4170)[0x55b8a031e170]
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: /usr/sbin/mysqld(+0xa205dd)[0x55b8a025a5dd]
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: /usr/sbin/mysqld(+0xb10a85)[0x55b8a034aa85]
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: /usr/sbin/mysqld(+0xb116be)[0x55b8a034b6be]
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: /usr/sbin/mysqld(+0xb1224c)[0x55b8a034c24c]
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: /usr/sbin/mysqld(+0xae7959)[0x55b8a0321959]
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: /usr/sbin/mysqld(+0xad0db6)[0x55b8a030adb6]
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: /usr/sbin/mysqld(+0xb0b658)[0x55b8a0345658]
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: /usr/sbin/mysqld(+0xb1173f)[0x55b8a034b73f]
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: /usr/sbin/mysqld(+0xb1224c)[0x55b8a034c24c]
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: /usr/sbin/mysqld(+0xae73a4)[0x55b8a03213a4]
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: /usr/sbin/mysqld(+0xa21f8f)[0x55b8a025bf8f]
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: /usr/sbin/mysqld(_ZN7handler13ha_delete_rowEPKh+0x9f)[0x55b8a00ba7af]
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: /usr/sbin/mysqld(_ZN21Delete_rows_log_event11do_exec_rowEP14rpl_group_info+0x12d)[0x55b8a01b12ad]
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: /usr/sbin/mysqld(_ZN14Rows_log_event14do_apply_eventEP14rpl_group_info+0x25d)[0x55b8a01a440d]
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: /usr/sbin/mysqld(+0x5b6e90)[0x55b89fdf0e90]
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: /usr/sbin/mysqld(handle_slave_sql+0x17b4)[0x55b89fdfa344]
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: /usr/sbin/mysqld(+0xdab79a)[0x55b8a05e579a]
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: /lib/x86_64-linux-gnu/libpthread.so.0(+0x76db)[0x7f411d5786db]
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: /lib/x86_64-linux-gnu/libc.so.6(clone+0x3f)[0x7f411bf9a88f]
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: Trying to get some variables.
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: Some pointers may be invalid and cause the dump to abort.
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: Query (0x0):
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: Connection ID (thread ID): 49
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: Status: NOT_KILLED
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=on,condition_pushdown_from_having=on
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: information that should help you find out what is causing the crash.
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: Writing a core file...
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: Working directory at /var/lib/mysql
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: Resource Limits:
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: Limit Soft Limit Hard Limit Units
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: Max cpu time unlimited unlimited seconds
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: Max file size unlimited unlimited bytes
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: Max data size unlimited unlimited bytes
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: Max stack size 8388608 unlimited bytes
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: Max core file size 0 unlimited bytes
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: Max resident set unlimited unlimited bytes
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: Max processes 15602 15602 processes
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: Max open files 16364 16364 files
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: Max locked memory 16777216 16777216 bytes
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: Max address space unlimited unlimited bytes
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: Max file locks unlimited unlimited locks
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: Max pending signals 15602 15602 signals
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: Max msgqueue size 819200 819200 bytes
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: Max nice priority 0 0
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: Max realtime priority 0 0
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: Max realtime timeout unlimited unlimited us
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: Core pattern: core
            Jun 27 12:44:00 vox-db1-read1 kernel: [264684.731198] mysqld[7861]: segfault at 115 ip 00007f411c008039 sp 00007f40fc8fcfc8 error 6 in libc-2.27.so[7f411be79000+1e7000]
            Jun 27 12:44:00 vox-db1-read1 systemd[1]: mariadb.service: Main process exited, code=killed, status=11/SEGV
            Jun 27 12:44:00 vox-db1-read1 systemd[1]: mariadb.service: Failed with result 'signal'.
            Jun 27 12:44:05 vox-db1-read1 systemd[1]: mariadb.service: Service hold-off time over, scheduling restart.
            Jun 27 12:44:05 vox-db1-read1 systemd[1]: mariadb.service: Scheduled restart job, restart counter is at 1.
            Jun 27 12:44:05 vox-db1-read1 systemd[1]: Stopped MariaDB 10.4.12 database server.
            Jun 27 12:44:05 vox-db1-read1 systemd[1]: Starting MariaDB 10.4.12 database server...
            Jun 27 12:44:06 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:06 0 [Note] /usr/sbin/mysqld (mysqld 10.4.12-MariaDB-1:10.4.12+maria~bionic-log) starting as process 12132 ...
            Jun 27 12:44:06 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:06 0 [Note] InnoDB: Using Linux native AIO
            Jun 27 12:44:06 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:06 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
            Jun 27 12:44:06 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:06 0 [Note] InnoDB: Uses event mutexes
            Jun 27 12:44:06 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:06 0 [Note] InnoDB: Compressed tables use zlib 1.2.11
            Jun 27 12:44:06 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:06 0 [Note] InnoDB: Number of pools: 1
            Jun 27 12:44:06 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:06 0 [Note] InnoDB: Using SSE2 crc32 instructions
            Jun 27 12:44:06 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:06 0 [Note] mysqld: O_TMPFILE is not supported on /tmp (disabling future attempts)
            Jun 27 12:44:06 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:06 0 [Note] InnoDB: Initializing buffer pool, total size = 1G, instances = 8, chunk size = 128M
            Jun 27 12:44:06 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:06 0 [Note] InnoDB: Completed initialization of buffer pool
            Jun 27 12:44:06 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:06 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority().
            Jun 27 12:44:07 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:07 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=142473402785
            Jun 27 12:44:10 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:10 0 [Note] InnoDB: 1 transaction(s) which must be rolled back or cleaned up in total 2 row operations to undo
            Jun 27 12:44:10 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:10 0 [Note] InnoDB: Trx id counter is 11134299
            Jun 27 12:44:10 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:10 0 [Note] InnoDB: Starting final batch to recover 63 pages from redo log.
            Jun 27 12:44:10 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:10 0 [Note] InnoDB: Last binlog file '/var/log/mysql/mariadb-bin.000634', position 59120658
            Jun 27 12:44:10 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:10 0 [Note] InnoDB: 128 out of 128 rollback segments are active.
            Jun 27 12:44:10 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:10 0 [Note] InnoDB: Starting in background the rollback of recovered transactions
            Jun 27 12:44:10 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:10 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1"
            Jun 27 12:44:10 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:10 0 [Note] InnoDB: Creating shared tablespace for temporary tables
            Jun 27 12:44:10 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:10 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ...
            Jun 27 12:44:10 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:10 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB.
            Jun 27 12:44:10 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:10 0 [Note] InnoDB: 10.4.12 started; log sequence number 142473413250; transaction id 11134301
            Jun 27 12:44:10 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:10 0 [Note] InnoDB: Loading buffer pool(s) from /var/lib/mysql/ib_buffer_pool
            Jun 27 12:44:10 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:10 0 [Note] Plugin 'FEEDBACK' is disabled.
            Jun 27 12:44:10 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:10 0 [Note] Recovering after a crash using /var/log/mysql/mariadb-bin
            Jun 27 12:44:10 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:10 0 [Note] Starting crash recovery...
            Jun 27 12:44:10 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:10 0 [Note] Crash recovery finished.
            Jun 27 12:44:10 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:10 0 [Note] InnoDB: Rolled back recovered transaction 11134298
            Jun 27 12:44:10 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:10 0 [Note] InnoDB: Rollback of non-prepared transactions completed
            Jun 27 12:44:10 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:10 0 [Note] Server socket created on IP: '0.0.0.0'.
            Jun 27 12:44:10 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:10 0 [Note] Reading of all Master_info entries succeeded
            Jun 27 12:44:10 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:10 0 [Note] Added new Master_info '' to hash table
            Jun 27 12:44:10 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:10 0 [Warning] Neither --relay-log nor --relay-log-index were used; so replication may break when this MySQL server acts as a slave and has his hostname changed!! Please use '--log-basename=#' or '--relay-log=mysqld-relay-bin' to avoid this problem.
            Jun 27 12:44:11 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:11 10 [Note] Slave I/O thread: Start asynchronous replication to master 'replication_user@10.130.0.25:3306' in log 'mariadb-bin.000712' at position 4
            Jun 27 12:44:11 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:11 11 [Note] Slave SQL thread initialized, starting replication in log 'mariadb-bin.000712' at position 4, relay log './mysqld-relay-bin.000001' position: 4; GTID position '0-1-5332814'
            Jun 27 12:44:11 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:11 0 [Note] /usr/sbin/mysqld: ready for connections.
            Jun 27 12:44:11 vox-db1-read1 mysqld[12132]: Version: '10.4.12-MariaDB-1:10.4.12+maria~bionic-log' socket: '/var/run/mysqld/mysqld.sock' port: 3306 mariadb.org binary distribution
            Jun 27 12:44:11 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:11 10 [Note] Slave I/O thread: connected to master 'replication_user@10.130.0.25:3306',replication starts at GTID position '0-1-5332814'
            Jun 27 12:44:11 vox-db1-read1 systemd[1]: Started MariaDB 10.4.12 database server.
            Jun 27 12:44:11 vox-db1-read1 /etc/mysql/debian-start[12175]: Upgrading MySQL tables if necessary.
            Jun 27 12:44:11 vox-db1-read1 /etc/mysql/debian-start[12178]: Looking for 'mysql' as: /usr/bin/mysql
            Jun 27 12:44:11 vox-db1-read1 /etc/mysql/debian-start[12178]: Looking for 'mysqlcheck' as: /usr/bin/mysqlcheck
            Jun 27 12:44:11 vox-db1-read1 /etc/mysql/debian-start[12178]: This installation of MariaDB is already upgraded to 10.4.12-MariaDB, use --force if you still need to run mysql_upgrade
            Jun 27 12:44:11 vox-db1-read1 /etc/mysql/debian-start[12186]: Checking for insecure root accounts.
            Jun 27 12:44:11 vox-db1-read1 /etc/mysql/debian-start[12190]: Triggering myisam-recover for all MyISAM tables and aria-recover for all Aria tables
            Jun 27 12:44:11 vox-db1-read1 mysqld[12132]: 200627 12:44:11 [ERROR] mysqld got signal 11 ;
            Jun 27 12:44:11 vox-db1-read1 mysqld[12132]: This could be because you hit a bug. It is also possible that this binary
            Jun 27 12:44:11 vox-db1-read1 mysqld[12132]: or one of the libraries it was linked against is corrupt, improperly built,
            Jun 27 12:44:11 vox-db1-read1 mysqld[12132]: or misconfigured. This error can also be caused by malfunctioning hardware.
            Jun 27 12:44:11 vox-db1-read1 mysqld[12132]: To report this bug, see https://mariadb.com/kb/en/reporting-bugs
            Jun 27 12:44:11 vox-db1-read1 mysqld[12132]: We will try our best to scrape up some info that will hopefully help
            Jun 27 12:44:11 vox-db1-read1 mysqld[12132]: diagnose the problem, but since we have already crashed,
            Jun 27 12:44:11 vox-db1-read1 mysqld[12132]: something is definitely wrong and this may fail.
            Jun 27 12:44:11 vox-db1-read1 mysqld[12132]: Server version: 10.4.12-MariaDB-1:10.4.12+maria~bionic-log
            Jun 27 12:44:11 vox-db1-read1 mysqld[12132]: key_buffer_size=134217728
            Jun 27 12:44:11 vox-db1-read1 mysqld[12132]: read_buffer_size=2097152
            Jun 27 12:44:11 vox-db1-read1 mysqld[12132]: max_used_connections=1
            ...and so on, it loops indefinitely.

            My theory is that something comes from Master that breaks Slaves.
            When first crash occured (same day I've setup replication, ~4-5 days ago) I tried to restore Slave from same backup I used to initilize it. And Slave could not be started. I suppose while catching up to Master it still encountered this breaking transaction and crashed. Then I just took a fresh backup from Master, and Slave started without problems.

            Since then every time Slaves crash I just take make backup (with mariabackup) and restore slaves. This is stupid I know, but what choice do we currently have? :)

            What additional information can I provide you with to find the root of this issue?

            Thanks!
            {code:java}
            // Some comments here
            public String getFoo()
            {
                return foo;
            }
            {code}
            Hello.
            We have a Master-to-multi-Slave config and are regularly facing slaves crashes (they crash together at the same time, approx. once every 24 hours). Maybe this has something to do with load, but crashed did not happen under heavy load and still happened today with relatively (50%) of highest load we've had so far.

            Master 10.4.12 setup is Ubuntu 18.0.4 cloud VPS (yandex cloud) with 6 cores 100% quota, 6 Gbs RAM, 30 Gbs space.
            Slaves 10.4.12 and 10.4.13 setup is Ubuntu 18.0.4 cloud VPS (yandex cloud) with 4 cores 100% quota, 4 Gbs RAM, 30 Gbs space.

            Replication is setup using this guide - https://mariadb.com/kb/en/setting-up-a-replication-slave-with-mariabackup/
            Everything works well, until slaves crash.

            Currently Web API sends DML statements to Master directly and SELECT statements are sent to HAProxy with wieght distribution 20/40/40 (Master/Slave1/Slave2).
            Before that we had Master 10.4.12 and only 1 Slave 10.4.12 with direct requests from API to Slave without HAProxy - Slave still crashed, I hoped with 2 slaves we would at least have 1 standing... but alas...

            We removed all triggers from the DB.
            We use GTIDs.
            We don't have any DDL statements running (except a few CREATE TEMPORARY TABLE).
            We use InnoDB only to store data.
            We don't have any FullText Indexes.
            We don't use compression.
            We don't use SSL between Mater and Slaves.
            Timezone on all machines is UTC.

            Here's syslog:


            {panel:title=My title}

            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: 200627 12:44:00 [ERROR] mysqld got signal 11 ;
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: This could be because you hit a bug. It is also possible that this binary
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: or one of the libraries it was linked against is corrupt, improperly built,
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: or misconfigured. This error can also be caused by malfunctioning hardware.
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: To report this bug, see https://mariadb.com/kb/en/reporting-bugs
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: We will try our best to scrape up some info that will hopefully help
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: diagnose the problem, but since we have already crashed,
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: something is definitely wrong and this may fail.
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: Server version: 10.4.12-MariaDB-1:10.4.12+maria~bionic-log
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: key_buffer_size=134217728
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: read_buffer_size=2097152
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: max_used_connections=15
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: max_threads=77
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: thread_count=24
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: It is possible that mysqld could use up to
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 606038 K bytes of memory
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: Hope that's ok; if not, decrease some variables in the equation.
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: Thread pointer: 0x7f40500014f8
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: Attempting backtrace. You can use the following information to find out
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: where mysqld died. If you see no messages after this, something went
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: terribly wrong...
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: stack_bottom = 0x7f40fc901e98 thread_stack 0x49000
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: /usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x55b8a063ac3e]
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: /usr/sbin/mysqld(handle_fatal_signal+0x515)[0x55b8a00aecf5]
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: /lib/x86_64-linux-gnu/libpthread.so.0(+0x12890)[0x7f411d583890]
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: /lib/x86_64-linux-gnu/libc.so.6(+0x18f039)[0x7f411c008039]
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: /usr/sbin/mysqld(+0xae4170)[0x55b8a031e170]
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: /usr/sbin/mysqld(+0xa205dd)[0x55b8a025a5dd]
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: /usr/sbin/mysqld(+0xb10a85)[0x55b8a034aa85]
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: /usr/sbin/mysqld(+0xb116be)[0x55b8a034b6be]
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: /usr/sbin/mysqld(+0xb1224c)[0x55b8a034c24c]
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: /usr/sbin/mysqld(+0xae7959)[0x55b8a0321959]
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: /usr/sbin/mysqld(+0xad0db6)[0x55b8a030adb6]
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: /usr/sbin/mysqld(+0xb0b658)[0x55b8a0345658]
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: /usr/sbin/mysqld(+0xb1173f)[0x55b8a034b73f]
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: /usr/sbin/mysqld(+0xb1224c)[0x55b8a034c24c]
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: /usr/sbin/mysqld(+0xae73a4)[0x55b8a03213a4]
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: /usr/sbin/mysqld(+0xa21f8f)[0x55b8a025bf8f]
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: /usr/sbin/mysqld(_ZN7handler13ha_delete_rowEPKh+0x9f)[0x55b8a00ba7af]
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: /usr/sbin/mysqld(_ZN21Delete_rows_log_event11do_exec_rowEP14rpl_group_info+0x12d)[0x55b8a01b12ad]
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: /usr/sbin/mysqld(_ZN14Rows_log_event14do_apply_eventEP14rpl_group_info+0x25d)[0x55b8a01a440d]
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: /usr/sbin/mysqld(+0x5b6e90)[0x55b89fdf0e90]
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: /usr/sbin/mysqld(handle_slave_sql+0x17b4)[0x55b89fdfa344]
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: /usr/sbin/mysqld(+0xdab79a)[0x55b8a05e579a]
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: /lib/x86_64-linux-gnu/libpthread.so.0(+0x76db)[0x7f411d5786db]
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: /lib/x86_64-linux-gnu/libc.so.6(clone+0x3f)[0x7f411bf9a88f]
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: Trying to get some variables.
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: Some pointers may be invalid and cause the dump to abort.
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: Query (0x0):
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: Connection ID (thread ID): 49
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: Status: NOT_KILLED
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=on,condition_pushdown_from_having=on
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: information that should help you find out what is causing the crash.
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: Writing a core file...
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: Working directory at /var/lib/mysql
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: Resource Limits:
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: Limit Soft Limit Hard Limit Units
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: Max cpu time unlimited unlimited seconds
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: Max file size unlimited unlimited bytes
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: Max data size unlimited unlimited bytes
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: Max stack size 8388608 unlimited bytes
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: Max core file size 0 unlimited bytes
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: Max resident set unlimited unlimited bytes
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: Max processes 15602 15602 processes
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: Max open files 16364 16364 files
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: Max locked memory 16777216 16777216 bytes
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: Max address space unlimited unlimited bytes
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: Max file locks unlimited unlimited locks
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: Max pending signals 15602 15602 signals
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: Max msgqueue size 819200 819200 bytes
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: Max nice priority 0 0
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: Max realtime priority 0 0
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: Max realtime timeout unlimited unlimited us
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: Core pattern: core
            Jun 27 12:44:00 vox-db1-read1 kernel: [264684.731198] mysqld[7861]: segfault at 115 ip 00007f411c008039 sp 00007f40fc8fcfc8 error 6 in libc-2.27.so[7f411be79000+1e7000]
            Jun 27 12:44:00 vox-db1-read1 systemd[1]: mariadb.service: Main process exited, code=killed, status=11/SEGV
            Jun 27 12:44:00 vox-db1-read1 systemd[1]: mariadb.service: Failed with result 'signal'.
            Jun 27 12:44:05 vox-db1-read1 systemd[1]: mariadb.service: Service hold-off time over, scheduling restart.
            Jun 27 12:44:05 vox-db1-read1 systemd[1]: mariadb.service: Scheduled restart job, restart counter is at 1.
            Jun 27 12:44:05 vox-db1-read1 systemd[1]: Stopped MariaDB 10.4.12 database server.
            Jun 27 12:44:05 vox-db1-read1 systemd[1]: Starting MariaDB 10.4.12 database server...
            Jun 27 12:44:06 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:06 0 [Note] /usr/sbin/mysqld (mysqld 10.4.12-MariaDB-1:10.4.12+maria~bionic-log) starting as process 12132 ...
            Jun 27 12:44:06 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:06 0 [Note] InnoDB: Using Linux native AIO
            Jun 27 12:44:06 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:06 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
            Jun 27 12:44:06 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:06 0 [Note] InnoDB: Uses event mutexes
            Jun 27 12:44:06 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:06 0 [Note] InnoDB: Compressed tables use zlib 1.2.11
            Jun 27 12:44:06 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:06 0 [Note] InnoDB: Number of pools: 1
            Jun 27 12:44:06 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:06 0 [Note] InnoDB: Using SSE2 crc32 instructions
            Jun 27 12:44:06 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:06 0 [Note] mysqld: O_TMPFILE is not supported on /tmp (disabling future attempts)
            Jun 27 12:44:06 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:06 0 [Note] InnoDB: Initializing buffer pool, total size = 1G, instances = 8, chunk size = 128M
            Jun 27 12:44:06 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:06 0 [Note] InnoDB: Completed initialization of buffer pool
            Jun 27 12:44:06 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:06 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority().
            Jun 27 12:44:07 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:07 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=142473402785
            Jun 27 12:44:10 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:10 0 [Note] InnoDB: 1 transaction(s) which must be rolled back or cleaned up in total 2 row operations to undo
            Jun 27 12:44:10 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:10 0 [Note] InnoDB: Trx id counter is 11134299
            Jun 27 12:44:10 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:10 0 [Note] InnoDB: Starting final batch to recover 63 pages from redo log.
            Jun 27 12:44:10 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:10 0 [Note] InnoDB: Last binlog file '/var/log/mysql/mariadb-bin.000634', position 59120658
            Jun 27 12:44:10 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:10 0 [Note] InnoDB: 128 out of 128 rollback segments are active.
            Jun 27 12:44:10 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:10 0 [Note] InnoDB: Starting in background the rollback of recovered transactions
            Jun 27 12:44:10 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:10 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1"
            Jun 27 12:44:10 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:10 0 [Note] InnoDB: Creating shared tablespace for temporary tables
            Jun 27 12:44:10 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:10 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ...
            Jun 27 12:44:10 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:10 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB.
            Jun 27 12:44:10 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:10 0 [Note] InnoDB: 10.4.12 started; log sequence number 142473413250; transaction id 11134301
            Jun 27 12:44:10 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:10 0 [Note] InnoDB: Loading buffer pool(s) from /var/lib/mysql/ib_buffer_pool
            Jun 27 12:44:10 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:10 0 [Note] Plugin 'FEEDBACK' is disabled.
            Jun 27 12:44:10 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:10 0 [Note] Recovering after a crash using /var/log/mysql/mariadb-bin
            Jun 27 12:44:10 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:10 0 [Note] Starting crash recovery...
            Jun 27 12:44:10 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:10 0 [Note] Crash recovery finished.
            Jun 27 12:44:10 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:10 0 [Note] InnoDB: Rolled back recovered transaction 11134298
            Jun 27 12:44:10 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:10 0 [Note] InnoDB: Rollback of non-prepared transactions completed
            Jun 27 12:44:10 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:10 0 [Note] Server socket created on IP: '0.0.0.0'.
            Jun 27 12:44:10 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:10 0 [Note] Reading of all Master_info entries succeeded
            Jun 27 12:44:10 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:10 0 [Note] Added new Master_info '' to hash table
            Jun 27 12:44:10 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:10 0 [Warning] Neither --relay-log nor --relay-log-index were used; so replication may break when this MySQL server acts as a slave and has his hostname changed!! Please use '--log-basename=#' or '--relay-log=mysqld-relay-bin' to avoid this problem.
            Jun 27 12:44:11 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:11 10 [Note] Slave I/O thread: Start asynchronous replication to master 'replication_user@10.130.0.25:3306' in log 'mariadb-bin.000712' at position 4
            Jun 27 12:44:11 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:11 11 [Note] Slave SQL thread initialized, starting replication in log 'mariadb-bin.000712' at position 4, relay log './mysqld-relay-bin.000001' position: 4; GTID position '0-1-5332814'
            Jun 27 12:44:11 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:11 0 [Note] /usr/sbin/mysqld: ready for connections.
            Jun 27 12:44:11 vox-db1-read1 mysqld[12132]: Version: '10.4.12-MariaDB-1:10.4.12+maria~bionic-log' socket: '/var/run/mysqld/mysqld.sock' port: 3306 mariadb.org binary distribution
            Jun 27 12:44:11 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:11 10 [Note] Slave I/O thread: connected to master 'replication_user@10.130.0.25:3306',replication starts at GTID position '0-1-5332814'
            Jun 27 12:44:11 vox-db1-read1 systemd[1]: Started MariaDB 10.4.12 database server.
            Jun 27 12:44:11 vox-db1-read1 /etc/mysql/debian-start[12175]: Upgrading MySQL tables if necessary.
            Jun 27 12:44:11 vox-db1-read1 /etc/mysql/debian-start[12178]: Looking for 'mysql' as: /usr/bin/mysql
            Jun 27 12:44:11 vox-db1-read1 /etc/mysql/debian-start[12178]: Looking for 'mysqlcheck' as: /usr/bin/mysqlcheck
            Jun 27 12:44:11 vox-db1-read1 /etc/mysql/debian-start[12178]: This installation of MariaDB is already upgraded to 10.4.12-MariaDB, use --force if you still need to run mysql_upgrade
            Jun 27 12:44:11 vox-db1-read1 /etc/mysql/debian-start[12186]: Checking for insecure root accounts.
            Jun 27 12:44:11 vox-db1-read1 /etc/mysql/debian-start[12190]: Triggering myisam-recover for all MyISAM tables and aria-recover for all Aria tables
            Jun 27 12:44:11 vox-db1-read1 mysqld[12132]: 200627 12:44:11 [ERROR] mysqld got signal 11 ;
            Jun 27 12:44:11 vox-db1-read1 mysqld[12132]: This could be because you hit a bug. It is also possible that this binary
            Jun 27 12:44:11 vox-db1-read1 mysqld[12132]: or one of the libraries it was linked against is corrupt, improperly built,
            Jun 27 12:44:11 vox-db1-read1 mysqld[12132]: or misconfigured. This error can also be caused by malfunctioning hardware.
            Jun 27 12:44:11 vox-db1-read1 mysqld[12132]: To report this bug, see https://mariadb.com/kb/en/reporting-bugs
            Jun 27 12:44:11 vox-db1-read1 mysqld[12132]: We will try our best to scrape up some info that will hopefully help
            Jun 27 12:44:11 vox-db1-read1 mysqld[12132]: diagnose the problem, but since we have already crashed,
            Jun 27 12:44:11 vox-db1-read1 mysqld[12132]: something is definitely wrong and this may fail.
            Jun 27 12:44:11 vox-db1-read1 mysqld[12132]: Server version: 10.4.12-MariaDB-1:10.4.12+maria~bionic-log
            Jun 27 12:44:11 vox-db1-read1 mysqld[12132]: key_buffer_size=134217728
            Jun 27 12:44:11 vox-db1-read1 mysqld[12132]: read_buffer_size=2097152
            Jun 27 12:44:11 vox-db1-read1 mysqld[12132]: max_used_connections=1
            ...and so on, it loops indefinitely.
            {panel}


            My theory is that something comes from Master that breaks Slaves.
            When first crash occured (same day I've setup replication, ~4-5 days ago) I tried to restore Slave from same backup I used to initilize it. And Slave could not be started. I suppose while catching up to Master it still encountered this breaking transaction and crashed. Then I just took a fresh backup from Master, and Slave started without problems.

            Since then every time Slaves crash I just take make backup (with mariabackup) and restore slaves. This is stupid I know, but what choice do we currently have? :)

            What additional information can I provide you with to find the root of this issue?

            Thanks!
            Grigoriy Grigoriy made changes -
            Description
            {code:java}
            // Some comments here
            public String getFoo()
            {
                return foo;
            }
            {code}
            Hello.
            We have a Master-to-multi-Slave config and are regularly facing slaves crashes (they crash together at the same time, approx. once every 24 hours). Maybe this has something to do with load, but crashed did not happen under heavy load and still happened today with relatively (50%) of highest load we've had so far.

            Master 10.4.12 setup is Ubuntu 18.0.4 cloud VPS (yandex cloud) with 6 cores 100% quota, 6 Gbs RAM, 30 Gbs space.
            Slaves 10.4.12 and 10.4.13 setup is Ubuntu 18.0.4 cloud VPS (yandex cloud) with 4 cores 100% quota, 4 Gbs RAM, 30 Gbs space.

            Replication is setup using this guide - https://mariadb.com/kb/en/setting-up-a-replication-slave-with-mariabackup/
            Everything works well, until slaves crash.

            Currently Web API sends DML statements to Master directly and SELECT statements are sent to HAProxy with wieght distribution 20/40/40 (Master/Slave1/Slave2).
            Before that we had Master 10.4.12 and only 1 Slave 10.4.12 with direct requests from API to Slave without HAProxy - Slave still crashed, I hoped with 2 slaves we would at least have 1 standing... but alas...

            We removed all triggers from the DB.
            We use GTIDs.
            We don't have any DDL statements running (except a few CREATE TEMPORARY TABLE).
            We use InnoDB only to store data.
            We don't have any FullText Indexes.
            We don't use compression.
            We don't use SSL between Mater and Slaves.
            Timezone on all machines is UTC.

            Here's syslog:


            {panel:title=My title}

            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: 200627 12:44:00 [ERROR] mysqld got signal 11 ;
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: This could be because you hit a bug. It is also possible that this binary
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: or one of the libraries it was linked against is corrupt, improperly built,
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: or misconfigured. This error can also be caused by malfunctioning hardware.
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: To report this bug, see https://mariadb.com/kb/en/reporting-bugs
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: We will try our best to scrape up some info that will hopefully help
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: diagnose the problem, but since we have already crashed,
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: something is definitely wrong and this may fail.
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: Server version: 10.4.12-MariaDB-1:10.4.12+maria~bionic-log
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: key_buffer_size=134217728
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: read_buffer_size=2097152
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: max_used_connections=15
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: max_threads=77
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: thread_count=24
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: It is possible that mysqld could use up to
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 606038 K bytes of memory
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: Hope that's ok; if not, decrease some variables in the equation.
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: Thread pointer: 0x7f40500014f8
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: Attempting backtrace. You can use the following information to find out
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: where mysqld died. If you see no messages after this, something went
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: terribly wrong...
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: stack_bottom = 0x7f40fc901e98 thread_stack 0x49000
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: /usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x55b8a063ac3e]
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: /usr/sbin/mysqld(handle_fatal_signal+0x515)[0x55b8a00aecf5]
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: /lib/x86_64-linux-gnu/libpthread.so.0(+0x12890)[0x7f411d583890]
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: /lib/x86_64-linux-gnu/libc.so.6(+0x18f039)[0x7f411c008039]
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: /usr/sbin/mysqld(+0xae4170)[0x55b8a031e170]
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: /usr/sbin/mysqld(+0xa205dd)[0x55b8a025a5dd]
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: /usr/sbin/mysqld(+0xb10a85)[0x55b8a034aa85]
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: /usr/sbin/mysqld(+0xb116be)[0x55b8a034b6be]
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: /usr/sbin/mysqld(+0xb1224c)[0x55b8a034c24c]
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: /usr/sbin/mysqld(+0xae7959)[0x55b8a0321959]
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: /usr/sbin/mysqld(+0xad0db6)[0x55b8a030adb6]
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: /usr/sbin/mysqld(+0xb0b658)[0x55b8a0345658]
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: /usr/sbin/mysqld(+0xb1173f)[0x55b8a034b73f]
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: /usr/sbin/mysqld(+0xb1224c)[0x55b8a034c24c]
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: /usr/sbin/mysqld(+0xae73a4)[0x55b8a03213a4]
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: /usr/sbin/mysqld(+0xa21f8f)[0x55b8a025bf8f]
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: /usr/sbin/mysqld(_ZN7handler13ha_delete_rowEPKh+0x9f)[0x55b8a00ba7af]
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: /usr/sbin/mysqld(_ZN21Delete_rows_log_event11do_exec_rowEP14rpl_group_info+0x12d)[0x55b8a01b12ad]
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: /usr/sbin/mysqld(_ZN14Rows_log_event14do_apply_eventEP14rpl_group_info+0x25d)[0x55b8a01a440d]
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: /usr/sbin/mysqld(+0x5b6e90)[0x55b89fdf0e90]
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: /usr/sbin/mysqld(handle_slave_sql+0x17b4)[0x55b89fdfa344]
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: /usr/sbin/mysqld(+0xdab79a)[0x55b8a05e579a]
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: /lib/x86_64-linux-gnu/libpthread.so.0(+0x76db)[0x7f411d5786db]
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: /lib/x86_64-linux-gnu/libc.so.6(clone+0x3f)[0x7f411bf9a88f]
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: Trying to get some variables.
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: Some pointers may be invalid and cause the dump to abort.
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: Query (0x0):
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: Connection ID (thread ID): 49
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: Status: NOT_KILLED
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=on,condition_pushdown_from_having=on
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: information that should help you find out what is causing the crash.
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: Writing a core file...
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: Working directory at /var/lib/mysql
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: Resource Limits:
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: Limit Soft Limit Hard Limit Units
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: Max cpu time unlimited unlimited seconds
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: Max file size unlimited unlimited bytes
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: Max data size unlimited unlimited bytes
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: Max stack size 8388608 unlimited bytes
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: Max core file size 0 unlimited bytes
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: Max resident set unlimited unlimited bytes
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: Max processes 15602 15602 processes
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: Max open files 16364 16364 files
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: Max locked memory 16777216 16777216 bytes
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: Max address space unlimited unlimited bytes
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: Max file locks unlimited unlimited locks
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: Max pending signals 15602 15602 signals
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: Max msgqueue size 819200 819200 bytes
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: Max nice priority 0 0
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: Max realtime priority 0 0
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: Max realtime timeout unlimited unlimited us
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: Core pattern: core
            Jun 27 12:44:00 vox-db1-read1 kernel: [264684.731198] mysqld[7861]: segfault at 115 ip 00007f411c008039 sp 00007f40fc8fcfc8 error 6 in libc-2.27.so[7f411be79000+1e7000]
            Jun 27 12:44:00 vox-db1-read1 systemd[1]: mariadb.service: Main process exited, code=killed, status=11/SEGV
            Jun 27 12:44:00 vox-db1-read1 systemd[1]: mariadb.service: Failed with result 'signal'.
            Jun 27 12:44:05 vox-db1-read1 systemd[1]: mariadb.service: Service hold-off time over, scheduling restart.
            Jun 27 12:44:05 vox-db1-read1 systemd[1]: mariadb.service: Scheduled restart job, restart counter is at 1.
            Jun 27 12:44:05 vox-db1-read1 systemd[1]: Stopped MariaDB 10.4.12 database server.
            Jun 27 12:44:05 vox-db1-read1 systemd[1]: Starting MariaDB 10.4.12 database server...
            Jun 27 12:44:06 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:06 0 [Note] /usr/sbin/mysqld (mysqld 10.4.12-MariaDB-1:10.4.12+maria~bionic-log) starting as process 12132 ...
            Jun 27 12:44:06 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:06 0 [Note] InnoDB: Using Linux native AIO
            Jun 27 12:44:06 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:06 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
            Jun 27 12:44:06 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:06 0 [Note] InnoDB: Uses event mutexes
            Jun 27 12:44:06 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:06 0 [Note] InnoDB: Compressed tables use zlib 1.2.11
            Jun 27 12:44:06 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:06 0 [Note] InnoDB: Number of pools: 1
            Jun 27 12:44:06 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:06 0 [Note] InnoDB: Using SSE2 crc32 instructions
            Jun 27 12:44:06 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:06 0 [Note] mysqld: O_TMPFILE is not supported on /tmp (disabling future attempts)
            Jun 27 12:44:06 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:06 0 [Note] InnoDB: Initializing buffer pool, total size = 1G, instances = 8, chunk size = 128M
            Jun 27 12:44:06 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:06 0 [Note] InnoDB: Completed initialization of buffer pool
            Jun 27 12:44:06 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:06 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority().
            Jun 27 12:44:07 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:07 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=142473402785
            Jun 27 12:44:10 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:10 0 [Note] InnoDB: 1 transaction(s) which must be rolled back or cleaned up in total 2 row operations to undo
            Jun 27 12:44:10 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:10 0 [Note] InnoDB: Trx id counter is 11134299
            Jun 27 12:44:10 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:10 0 [Note] InnoDB: Starting final batch to recover 63 pages from redo log.
            Jun 27 12:44:10 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:10 0 [Note] InnoDB: Last binlog file '/var/log/mysql/mariadb-bin.000634', position 59120658
            Jun 27 12:44:10 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:10 0 [Note] InnoDB: 128 out of 128 rollback segments are active.
            Jun 27 12:44:10 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:10 0 [Note] InnoDB: Starting in background the rollback of recovered transactions
            Jun 27 12:44:10 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:10 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1"
            Jun 27 12:44:10 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:10 0 [Note] InnoDB: Creating shared tablespace for temporary tables
            Jun 27 12:44:10 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:10 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ...
            Jun 27 12:44:10 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:10 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB.
            Jun 27 12:44:10 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:10 0 [Note] InnoDB: 10.4.12 started; log sequence number 142473413250; transaction id 11134301
            Jun 27 12:44:10 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:10 0 [Note] InnoDB: Loading buffer pool(s) from /var/lib/mysql/ib_buffer_pool
            Jun 27 12:44:10 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:10 0 [Note] Plugin 'FEEDBACK' is disabled.
            Jun 27 12:44:10 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:10 0 [Note] Recovering after a crash using /var/log/mysql/mariadb-bin
            Jun 27 12:44:10 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:10 0 [Note] Starting crash recovery...
            Jun 27 12:44:10 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:10 0 [Note] Crash recovery finished.
            Jun 27 12:44:10 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:10 0 [Note] InnoDB: Rolled back recovered transaction 11134298
            Jun 27 12:44:10 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:10 0 [Note] InnoDB: Rollback of non-prepared transactions completed
            Jun 27 12:44:10 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:10 0 [Note] Server socket created on IP: '0.0.0.0'.
            Jun 27 12:44:10 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:10 0 [Note] Reading of all Master_info entries succeeded
            Jun 27 12:44:10 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:10 0 [Note] Added new Master_info '' to hash table
            Jun 27 12:44:10 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:10 0 [Warning] Neither --relay-log nor --relay-log-index were used; so replication may break when this MySQL server acts as a slave and has his hostname changed!! Please use '--log-basename=#' or '--relay-log=mysqld-relay-bin' to avoid this problem.
            Jun 27 12:44:11 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:11 10 [Note] Slave I/O thread: Start asynchronous replication to master 'replication_user@10.130.0.25:3306' in log 'mariadb-bin.000712' at position 4
            Jun 27 12:44:11 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:11 11 [Note] Slave SQL thread initialized, starting replication in log 'mariadb-bin.000712' at position 4, relay log './mysqld-relay-bin.000001' position: 4; GTID position '0-1-5332814'
            Jun 27 12:44:11 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:11 0 [Note] /usr/sbin/mysqld: ready for connections.
            Jun 27 12:44:11 vox-db1-read1 mysqld[12132]: Version: '10.4.12-MariaDB-1:10.4.12+maria~bionic-log' socket: '/var/run/mysqld/mysqld.sock' port: 3306 mariadb.org binary distribution
            Jun 27 12:44:11 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:11 10 [Note] Slave I/O thread: connected to master 'replication_user@10.130.0.25:3306',replication starts at GTID position '0-1-5332814'
            Jun 27 12:44:11 vox-db1-read1 systemd[1]: Started MariaDB 10.4.12 database server.
            Jun 27 12:44:11 vox-db1-read1 /etc/mysql/debian-start[12175]: Upgrading MySQL tables if necessary.
            Jun 27 12:44:11 vox-db1-read1 /etc/mysql/debian-start[12178]: Looking for 'mysql' as: /usr/bin/mysql
            Jun 27 12:44:11 vox-db1-read1 /etc/mysql/debian-start[12178]: Looking for 'mysqlcheck' as: /usr/bin/mysqlcheck
            Jun 27 12:44:11 vox-db1-read1 /etc/mysql/debian-start[12178]: This installation of MariaDB is already upgraded to 10.4.12-MariaDB, use --force if you still need to run mysql_upgrade
            Jun 27 12:44:11 vox-db1-read1 /etc/mysql/debian-start[12186]: Checking for insecure root accounts.
            Jun 27 12:44:11 vox-db1-read1 /etc/mysql/debian-start[12190]: Triggering myisam-recover for all MyISAM tables and aria-recover for all Aria tables
            Jun 27 12:44:11 vox-db1-read1 mysqld[12132]: 200627 12:44:11 [ERROR] mysqld got signal 11 ;
            Jun 27 12:44:11 vox-db1-read1 mysqld[12132]: This could be because you hit a bug. It is also possible that this binary
            Jun 27 12:44:11 vox-db1-read1 mysqld[12132]: or one of the libraries it was linked against is corrupt, improperly built,
            Jun 27 12:44:11 vox-db1-read1 mysqld[12132]: or misconfigured. This error can also be caused by malfunctioning hardware.
            Jun 27 12:44:11 vox-db1-read1 mysqld[12132]: To report this bug, see https://mariadb.com/kb/en/reporting-bugs
            Jun 27 12:44:11 vox-db1-read1 mysqld[12132]: We will try our best to scrape up some info that will hopefully help
            Jun 27 12:44:11 vox-db1-read1 mysqld[12132]: diagnose the problem, but since we have already crashed,
            Jun 27 12:44:11 vox-db1-read1 mysqld[12132]: something is definitely wrong and this may fail.
            Jun 27 12:44:11 vox-db1-read1 mysqld[12132]: Server version: 10.4.12-MariaDB-1:10.4.12+maria~bionic-log
            Jun 27 12:44:11 vox-db1-read1 mysqld[12132]: key_buffer_size=134217728
            Jun 27 12:44:11 vox-db1-read1 mysqld[12132]: read_buffer_size=2097152
            Jun 27 12:44:11 vox-db1-read1 mysqld[12132]: max_used_connections=1
            ...and so on, it loops indefinitely.
            {panel}


            My theory is that something comes from Master that breaks Slaves.
            When first crash occured (same day I've setup replication, ~4-5 days ago) I tried to restore Slave from same backup I used to initilize it. And Slave could not be started. I suppose while catching up to Master it still encountered this breaking transaction and crashed. Then I just took a fresh backup from Master, and Slave started without problems.

            Since then every time Slaves crash I just take make backup (with mariabackup) and restore slaves. This is stupid I know, but what choice do we currently have? :)

            What additional information can I provide you with to find the root of this issue?

            Thanks!             		Hello.
            We have a Master-to-multi-Slave config and are regularly facing slaves crashes (they crash together at the same time, approx. once every 24 hours). Maybe this has something to do with load, but crashed did not happen under heavy load and still happened today with relatively (50%) of highest load we've had so far.

            Master 10.4.12 setup is Ubuntu 18.0.4 cloud VPS (yandex cloud) with 6 cores 100% quota, 6 Gbs RAM, 30 Gbs space.
            Slaves 10.4.12 and 10.4.13 setup is Ubuntu 18.0.4 cloud VPS (yandex cloud) with 4 cores 100% quota, 4 Gbs RAM, 30 Gbs space.

            Replication is setup using this guide - https://mariadb.com/kb/en/setting-up-a-replication-slave-with-mariabackup/
            Everything works well, until slaves crash.

            Currently Web API sends DML statements to Master directly and SELECT statements are sent to HAProxy with wieght distribution 20/40/40 (Master/Slave1/Slave2).
            Before that we had Master 10.4.12 and only 1 Slave 10.4.12 with direct requests from API to Slave without HAProxy - Slave still crashed, I hoped with 2 slaves we would at least have 1 standing... but alas...

            We removed all triggers from the DB.
            We use GTIDs.
            We don't have any DDL statements running (except a few CREATE TEMPORARY TABLE).
            We use InnoDB only to store data.
            We don't have any FullText Indexes.
            We don't use compression.
            We don't use SSL between Mater and Slaves.
            Timezone on all machines is UTC.

            Here's syslog:

            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: 200627 12:44:00 [ERROR] mysqld got signal 11 ;
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: This could be because you hit a bug. It is also possible that this binary
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: or one of the libraries it was linked against is corrupt, improperly built,
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: or misconfigured. This error can also be caused by malfunctioning hardware.
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: To report this bug, see https://mariadb.com/kb/en/reporting-bugs
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: We will try our best to scrape up some info that will hopefully help
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: diagnose the problem, but since we have already crashed,
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: something is definitely wrong and this may fail.
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: Server version: 10.4.12-MariaDB-1:10.4.12+maria~bionic-log
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: key_buffer_size=134217728
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: read_buffer_size=2097152
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: max_used_connections=15
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: max_threads=77
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: thread_count=24
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: It is possible that mysqld could use up to
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 606038 K bytes of memory
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: Hope that's ok; if not, decrease some variables in the equation.
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: Thread pointer: 0x7f40500014f8
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: Attempting backtrace. You can use the following information to find out
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: where mysqld died. If you see no messages after this, something went
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: terribly wrong...
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: stack_bottom = 0x7f40fc901e98 thread_stack 0x49000
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: /usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x55b8a063ac3e]
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: /usr/sbin/mysqld(handle_fatal_signal+0x515)[0x55b8a00aecf5]
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: /lib/x86_64-linux-gnu/libpthread.so.0(+0x12890)[0x7f411d583890]
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: /lib/x86_64-linux-gnu/libc.so.6(+0x18f039)[0x7f411c008039]
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: /usr/sbin/mysqld(+0xae4170)[0x55b8a031e170]
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: /usr/sbin/mysqld(+0xa205dd)[0x55b8a025a5dd]
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: /usr/sbin/mysqld(+0xb10a85)[0x55b8a034aa85]
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: /usr/sbin/mysqld(+0xb116be)[0x55b8a034b6be]
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: /usr/sbin/mysqld(+0xb1224c)[0x55b8a034c24c]
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: /usr/sbin/mysqld(+0xae7959)[0x55b8a0321959]
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: /usr/sbin/mysqld(+0xad0db6)[0x55b8a030adb6]
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: /usr/sbin/mysqld(+0xb0b658)[0x55b8a0345658]
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: /usr/sbin/mysqld(+0xb1173f)[0x55b8a034b73f]
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: /usr/sbin/mysqld(+0xb1224c)[0x55b8a034c24c]
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: /usr/sbin/mysqld(+0xae73a4)[0x55b8a03213a4]
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: /usr/sbin/mysqld(+0xa21f8f)[0x55b8a025bf8f]
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: /usr/sbin/mysqld(_ZN7handler13ha_delete_rowEPKh+0x9f)[0x55b8a00ba7af]
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: /usr/sbin/mysqld(_ZN21Delete_rows_log_event11do_exec_rowEP14rpl_group_info+0x12d)[0x55b8a01b12ad]
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: /usr/sbin/mysqld(_ZN14Rows_log_event14do_apply_eventEP14rpl_group_info+0x25d)[0x55b8a01a440d]
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: /usr/sbin/mysqld(+0x5b6e90)[0x55b89fdf0e90]
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: /usr/sbin/mysqld(handle_slave_sql+0x17b4)[0x55b89fdfa344]
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: /usr/sbin/mysqld(+0xdab79a)[0x55b8a05e579a]
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: /lib/x86_64-linux-gnu/libpthread.so.0(+0x76db)[0x7f411d5786db]
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: /lib/x86_64-linux-gnu/libc.so.6(clone+0x3f)[0x7f411bf9a88f]
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: Trying to get some variables.
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: Some pointers may be invalid and cause the dump to abort.
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: Query (0x0):
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: Connection ID (thread ID): 49
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: Status: NOT_KILLED
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=on,condition_pushdown_from_having=on
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: information that should help you find out what is causing the crash.
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: Writing a core file...
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: Working directory at /var/lib/mysql
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: Resource Limits:
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: Limit Soft Limit Hard Limit Units
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: Max cpu time unlimited unlimited seconds
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: Max file size unlimited unlimited bytes
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: Max data size unlimited unlimited bytes
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: Max stack size 8388608 unlimited bytes
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: Max core file size 0 unlimited bytes
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: Max resident set unlimited unlimited bytes
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: Max processes 15602 15602 processes
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: Max open files 16364 16364 files
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: Max locked memory 16777216 16777216 bytes
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: Max address space unlimited unlimited bytes
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: Max file locks unlimited unlimited locks
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: Max pending signals 15602 15602 signals
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: Max msgqueue size 819200 819200 bytes
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: Max nice priority 0 0
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: Max realtime priority 0 0
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: Max realtime timeout unlimited unlimited us
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: Core pattern: core
            Jun 27 12:44:00 vox-db1-read1 kernel: [264684.731198] mysqld[7861]: segfault at 115 ip 00007f411c008039 sp 00007f40fc8fcfc8 error 6 in libc-2.27.so[7f411be79000+1e7000]
            Jun 27 12:44:00 vox-db1-read1 systemd[1]: mariadb.service: Main process exited, code=killed, status=11/SEGV
            Jun 27 12:44:00 vox-db1-read1 systemd[1]: mariadb.service: Failed with result 'signal'.
            Jun 27 12:44:05 vox-db1-read1 systemd[1]: mariadb.service: Service hold-off time over, scheduling restart.
            Jun 27 12:44:05 vox-db1-read1 systemd[1]: mariadb.service: Scheduled restart job, restart counter is at 1.
            Jun 27 12:44:05 vox-db1-read1 systemd[1]: Stopped MariaDB 10.4.12 database server.
            Jun 27 12:44:05 vox-db1-read1 systemd[1]: Starting MariaDB 10.4.12 database server...
            Jun 27 12:44:06 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:06 0 [Note] /usr/sbin/mysqld (mysqld 10.4.12-MariaDB-1:10.4.12+maria~bionic-log) starting as process 12132 ...
            Jun 27 12:44:06 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:06 0 [Note] InnoDB: Using Linux native AIO
            Jun 27 12:44:06 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:06 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
            Jun 27 12:44:06 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:06 0 [Note] InnoDB: Uses event mutexes
            Jun 27 12:44:06 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:06 0 [Note] InnoDB: Compressed tables use zlib 1.2.11
            Jun 27 12:44:06 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:06 0 [Note] InnoDB: Number of pools: 1
            Jun 27 12:44:06 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:06 0 [Note] InnoDB: Using SSE2 crc32 instructions
            Jun 27 12:44:06 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:06 0 [Note] mysqld: O_TMPFILE is not supported on /tmp (disabling future attempts)
            Jun 27 12:44:06 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:06 0 [Note] InnoDB: Initializing buffer pool, total size = 1G, instances = 8, chunk size = 128M
            Jun 27 12:44:06 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:06 0 [Note] InnoDB: Completed initialization of buffer pool
            Jun 27 12:44:06 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:06 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority().
            Jun 27 12:44:07 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:07 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=142473402785
            Jun 27 12:44:10 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:10 0 [Note] InnoDB: 1 transaction(s) which must be rolled back or cleaned up in total 2 row operations to undo
            Jun 27 12:44:10 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:10 0 [Note] InnoDB: Trx id counter is 11134299
            Jun 27 12:44:10 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:10 0 [Note] InnoDB: Starting final batch to recover 63 pages from redo log.
            Jun 27 12:44:10 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:10 0 [Note] InnoDB: Last binlog file '/var/log/mysql/mariadb-bin.000634', position 59120658
            Jun 27 12:44:10 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:10 0 [Note] InnoDB: 128 out of 128 rollback segments are active.
            Jun 27 12:44:10 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:10 0 [Note] InnoDB: Starting in background the rollback of recovered transactions
            Jun 27 12:44:10 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:10 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1"
            Jun 27 12:44:10 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:10 0 [Note] InnoDB: Creating shared tablespace for temporary tables
            Jun 27 12:44:10 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:10 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ...
            Jun 27 12:44:10 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:10 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB.
            Jun 27 12:44:10 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:10 0 [Note] InnoDB: 10.4.12 started; log sequence number 142473413250; transaction id 11134301
            Jun 27 12:44:10 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:10 0 [Note] InnoDB: Loading buffer pool(s) from /var/lib/mysql/ib_buffer_pool
            Jun 27 12:44:10 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:10 0 [Note] Plugin 'FEEDBACK' is disabled.
            Jun 27 12:44:10 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:10 0 [Note] Recovering after a crash using /var/log/mysql/mariadb-bin
            Jun 27 12:44:10 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:10 0 [Note] Starting crash recovery...
            Jun 27 12:44:10 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:10 0 [Note] Crash recovery finished.
            Jun 27 12:44:10 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:10 0 [Note] InnoDB: Rolled back recovered transaction 11134298
            Jun 27 12:44:10 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:10 0 [Note] InnoDB: Rollback of non-prepared transactions completed
            Jun 27 12:44:10 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:10 0 [Note] Server socket created on IP: '0.0.0.0'.
            Jun 27 12:44:10 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:10 0 [Note] Reading of all Master_info entries succeeded
            Jun 27 12:44:10 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:10 0 [Note] Added new Master_info '' to hash table
            Jun 27 12:44:10 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:10 0 [Warning] Neither --relay-log nor --relay-log-index were used; so replication may break when this MySQL server acts as a slave and has his hostname changed!! Please use '--log-basename=#' or '--relay-log=mysqld-relay-bin' to avoid this problem.
            Jun 27 12:44:11 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:11 10 [Note] Slave I/O thread: Start asynchronous replication to master 'replication_user@10.130.0.25:3306' in log 'mariadb-bin.000712' at position 4
            Jun 27 12:44:11 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:11 11 [Note] Slave SQL thread initialized, starting replication in log 'mariadb-bin.000712' at position 4, relay log './mysqld-relay-bin.000001' position: 4; GTID position '0-1-5332814'
            Jun 27 12:44:11 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:11 0 [Note] /usr/sbin/mysqld: ready for connections.
            Jun 27 12:44:11 vox-db1-read1 mysqld[12132]: Version: '10.4.12-MariaDB-1:10.4.12+maria~bionic-log' socket: '/var/run/mysqld/mysqld.sock' port: 3306 mariadb.org binary distribution
            Jun 27 12:44:11 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:11 10 [Note] Slave I/O thread: connected to master 'replication_user@10.130.0.25:3306',replication starts at GTID position '0-1-5332814'
            Jun 27 12:44:11 vox-db1-read1 systemd[1]: Started MariaDB 10.4.12 database server.
            Jun 27 12:44:11 vox-db1-read1 /etc/mysql/debian-start[12175]: Upgrading MySQL tables if necessary.
            Jun 27 12:44:11 vox-db1-read1 /etc/mysql/debian-start[12178]: Looking for 'mysql' as: /usr/bin/mysql
            Jun 27 12:44:11 vox-db1-read1 /etc/mysql/debian-start[12178]: Looking for 'mysqlcheck' as: /usr/bin/mysqlcheck
            Jun 27 12:44:11 vox-db1-read1 /etc/mysql/debian-start[12178]: This installation of MariaDB is already upgraded to 10.4.12-MariaDB, use --force if you still need to run mysql_upgrade
            Jun 27 12:44:11 vox-db1-read1 /etc/mysql/debian-start[12186]: Checking for insecure root accounts.
            Jun 27 12:44:11 vox-db1-read1 /etc/mysql/debian-start[12190]: Triggering myisam-recover for all MyISAM tables and aria-recover for all Aria tables
            Jun 27 12:44:11 vox-db1-read1 mysqld[12132]: 200627 12:44:11 [ERROR] mysqld got signal 11 ;
            Jun 27 12:44:11 vox-db1-read1 mysqld[12132]: This could be because you hit a bug. It is also possible that this binary
            Jun 27 12:44:11 vox-db1-read1 mysqld[12132]: or one of the libraries it was linked against is corrupt, improperly built,
            Jun 27 12:44:11 vox-db1-read1 mysqld[12132]: or misconfigured. This error can also be caused by malfunctioning hardware.
            Jun 27 12:44:11 vox-db1-read1 mysqld[12132]: To report this bug, see https://mariadb.com/kb/en/reporting-bugs
            Jun 27 12:44:11 vox-db1-read1 mysqld[12132]: We will try our best to scrape up some info that will hopefully help
            Jun 27 12:44:11 vox-db1-read1 mysqld[12132]: diagnose the problem, but since we have already crashed,
            Jun 27 12:44:11 vox-db1-read1 mysqld[12132]: something is definitely wrong and this may fail.
            Jun 27 12:44:11 vox-db1-read1 mysqld[12132]: Server version: 10.4.12-MariaDB-1:10.4.12+maria~bionic-log
            Jun 27 12:44:11 vox-db1-read1 mysqld[12132]: key_buffer_size=134217728
            Jun 27 12:44:11 vox-db1-read1 mysqld[12132]: read_buffer_size=2097152
            Jun 27 12:44:11 vox-db1-read1 mysqld[12132]: max_used_connections=1
            ...and so on, it loops indefinitely.


            My theory is that something comes from Master that breaks Slaves.
            When first crash occured (same day I've setup replication, ~4-5 days ago) I tried to restore Slave from same backup I used to initilize it. And Slave could not be started. I suppose while catching up to Master it still encountered this breaking transaction and crashed. Then I just took a fresh backup from Master, and Slave started without problems.

            Since then every time Slaves crash I just take make backup (with mariabackup) and restore slaves. This is stupid I know, but what choice do we currently have? :)

            What additional information can I provide you with to find the root of this issue?

            Thanks!
            Grigoriy Grigoriy made changes -
            Description Hello.
            We have a Master-to-multi-Slave config and are regularly facing slaves crashes (they crash together at the same time, approx. once every 24 hours). Maybe this has something to do with load, but crashed did not happen under heavy load and still happened today with relatively (50%) of highest load we've had so far.

            Master 10.4.12 setup is Ubuntu 18.0.4 cloud VPS (yandex cloud) with 6 cores 100% quota, 6 Gbs RAM, 30 Gbs space.
            Slaves 10.4.12 and 10.4.13 setup is Ubuntu 18.0.4 cloud VPS (yandex cloud) with 4 cores 100% quota, 4 Gbs RAM, 30 Gbs space.

            Replication is setup using this guide - https://mariadb.com/kb/en/setting-up-a-replication-slave-with-mariabackup/
            Everything works well, until slaves crash.

            Currently Web API sends DML statements to Master directly and SELECT statements are sent to HAProxy with wieght distribution 20/40/40 (Master/Slave1/Slave2).
            Before that we had Master 10.4.12 and only 1 Slave 10.4.12 with direct requests from API to Slave without HAProxy - Slave still crashed, I hoped with 2 slaves we would at least have 1 standing... but alas...

            We removed all triggers from the DB.
            We use GTIDs.
            We don't have any DDL statements running (except a few CREATE TEMPORARY TABLE).
            We use InnoDB only to store data.
            We don't have any FullText Indexes.
            We don't use compression.
            We don't use SSL between Mater and Slaves.
            Timezone on all machines is UTC.

            Here's syslog:

            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: 200627 12:44:00 [ERROR] mysqld got signal 11 ;
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: This could be because you hit a bug. It is also possible that this binary
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: or one of the libraries it was linked against is corrupt, improperly built,
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: or misconfigured. This error can also be caused by malfunctioning hardware.
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: To report this bug, see https://mariadb.com/kb/en/reporting-bugs
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: We will try our best to scrape up some info that will hopefully help
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: diagnose the problem, but since we have already crashed,
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: something is definitely wrong and this may fail.
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: Server version: 10.4.12-MariaDB-1:10.4.12+maria~bionic-log
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: key_buffer_size=134217728
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: read_buffer_size=2097152
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: max_used_connections=15
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: max_threads=77
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: thread_count=24
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: It is possible that mysqld could use up to
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 606038 K bytes of memory
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: Hope that's ok; if not, decrease some variables in the equation.
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: Thread pointer: 0x7f40500014f8
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: Attempting backtrace. You can use the following information to find out
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: where mysqld died. If you see no messages after this, something went
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: terribly wrong...
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: stack_bottom = 0x7f40fc901e98 thread_stack 0x49000
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: /usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x55b8a063ac3e]
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: /usr/sbin/mysqld(handle_fatal_signal+0x515)[0x55b8a00aecf5]
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: /lib/x86_64-linux-gnu/libpthread.so.0(+0x12890)[0x7f411d583890]
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: /lib/x86_64-linux-gnu/libc.so.6(+0x18f039)[0x7f411c008039]
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: /usr/sbin/mysqld(+0xae4170)[0x55b8a031e170]
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: /usr/sbin/mysqld(+0xa205dd)[0x55b8a025a5dd]
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: /usr/sbin/mysqld(+0xb10a85)[0x55b8a034aa85]
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: /usr/sbin/mysqld(+0xb116be)[0x55b8a034b6be]
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: /usr/sbin/mysqld(+0xb1224c)[0x55b8a034c24c]
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: /usr/sbin/mysqld(+0xae7959)[0x55b8a0321959]
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: /usr/sbin/mysqld(+0xad0db6)[0x55b8a030adb6]
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: /usr/sbin/mysqld(+0xb0b658)[0x55b8a0345658]
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: /usr/sbin/mysqld(+0xb1173f)[0x55b8a034b73f]
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: /usr/sbin/mysqld(+0xb1224c)[0x55b8a034c24c]
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: /usr/sbin/mysqld(+0xae73a4)[0x55b8a03213a4]
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: /usr/sbin/mysqld(+0xa21f8f)[0x55b8a025bf8f]
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: /usr/sbin/mysqld(_ZN7handler13ha_delete_rowEPKh+0x9f)[0x55b8a00ba7af]
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: /usr/sbin/mysqld(_ZN21Delete_rows_log_event11do_exec_rowEP14rpl_group_info+0x12d)[0x55b8a01b12ad]
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: /usr/sbin/mysqld(_ZN14Rows_log_event14do_apply_eventEP14rpl_group_info+0x25d)[0x55b8a01a440d]
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: /usr/sbin/mysqld(+0x5b6e90)[0x55b89fdf0e90]
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: /usr/sbin/mysqld(handle_slave_sql+0x17b4)[0x55b89fdfa344]
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: /usr/sbin/mysqld(+0xdab79a)[0x55b8a05e579a]
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: /lib/x86_64-linux-gnu/libpthread.so.0(+0x76db)[0x7f411d5786db]
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: /lib/x86_64-linux-gnu/libc.so.6(clone+0x3f)[0x7f411bf9a88f]
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: Trying to get some variables.
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: Some pointers may be invalid and cause the dump to abort.
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: Query (0x0):
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: Connection ID (thread ID): 49
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: Status: NOT_KILLED
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=on,condition_pushdown_from_having=on
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: information that should help you find out what is causing the crash.
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: Writing a core file...
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: Working directory at /var/lib/mysql
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: Resource Limits:
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: Limit Soft Limit Hard Limit Units
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: Max cpu time unlimited unlimited seconds
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: Max file size unlimited unlimited bytes
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: Max data size unlimited unlimited bytes
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: Max stack size 8388608 unlimited bytes
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: Max core file size 0 unlimited bytes
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: Max resident set unlimited unlimited bytes
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: Max processes 15602 15602 processes
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: Max open files 16364 16364 files
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: Max locked memory 16777216 16777216 bytes
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: Max address space unlimited unlimited bytes
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: Max file locks unlimited unlimited locks
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: Max pending signals 15602 15602 signals
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: Max msgqueue size 819200 819200 bytes
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: Max nice priority 0 0
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: Max realtime priority 0 0
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: Max realtime timeout unlimited unlimited us
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: Core pattern: core
            Jun 27 12:44:00 vox-db1-read1 kernel: [264684.731198] mysqld[7861]: segfault at 115 ip 00007f411c008039 sp 00007f40fc8fcfc8 error 6 in libc-2.27.so[7f411be79000+1e7000]
            Jun 27 12:44:00 vox-db1-read1 systemd[1]: mariadb.service: Main process exited, code=killed, status=11/SEGV
            Jun 27 12:44:00 vox-db1-read1 systemd[1]: mariadb.service: Failed with result 'signal'.
            Jun 27 12:44:05 vox-db1-read1 systemd[1]: mariadb.service: Service hold-off time over, scheduling restart.
            Jun 27 12:44:05 vox-db1-read1 systemd[1]: mariadb.service: Scheduled restart job, restart counter is at 1.
            Jun 27 12:44:05 vox-db1-read1 systemd[1]: Stopped MariaDB 10.4.12 database server.
            Jun 27 12:44:05 vox-db1-read1 systemd[1]: Starting MariaDB 10.4.12 database server...
            Jun 27 12:44:06 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:06 0 [Note] /usr/sbin/mysqld (mysqld 10.4.12-MariaDB-1:10.4.12+maria~bionic-log) starting as process 12132 ...
            Jun 27 12:44:06 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:06 0 [Note] InnoDB: Using Linux native AIO
            Jun 27 12:44:06 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:06 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
            Jun 27 12:44:06 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:06 0 [Note] InnoDB: Uses event mutexes
            Jun 27 12:44:06 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:06 0 [Note] InnoDB: Compressed tables use zlib 1.2.11
            Jun 27 12:44:06 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:06 0 [Note] InnoDB: Number of pools: 1
            Jun 27 12:44:06 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:06 0 [Note] InnoDB: Using SSE2 crc32 instructions
            Jun 27 12:44:06 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:06 0 [Note] mysqld: O_TMPFILE is not supported on /tmp (disabling future attempts)
            Jun 27 12:44:06 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:06 0 [Note] InnoDB: Initializing buffer pool, total size = 1G, instances = 8, chunk size = 128M
            Jun 27 12:44:06 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:06 0 [Note] InnoDB: Completed initialization of buffer pool
            Jun 27 12:44:06 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:06 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority().
            Jun 27 12:44:07 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:07 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=142473402785
            Jun 27 12:44:10 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:10 0 [Note] InnoDB: 1 transaction(s) which must be rolled back or cleaned up in total 2 row operations to undo
            Jun 27 12:44:10 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:10 0 [Note] InnoDB: Trx id counter is 11134299
            Jun 27 12:44:10 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:10 0 [Note] InnoDB: Starting final batch to recover 63 pages from redo log.
            Jun 27 12:44:10 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:10 0 [Note] InnoDB: Last binlog file '/var/log/mysql/mariadb-bin.000634', position 59120658
            Jun 27 12:44:10 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:10 0 [Note] InnoDB: 128 out of 128 rollback segments are active.
            Jun 27 12:44:10 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:10 0 [Note] InnoDB: Starting in background the rollback of recovered transactions
            Jun 27 12:44:10 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:10 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1"
            Jun 27 12:44:10 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:10 0 [Note] InnoDB: Creating shared tablespace for temporary tables
            Jun 27 12:44:10 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:10 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ...
            Jun 27 12:44:10 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:10 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB.
            Jun 27 12:44:10 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:10 0 [Note] InnoDB: 10.4.12 started; log sequence number 142473413250; transaction id 11134301
            Jun 27 12:44:10 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:10 0 [Note] InnoDB: Loading buffer pool(s) from /var/lib/mysql/ib_buffer_pool
            Jun 27 12:44:10 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:10 0 [Note] Plugin 'FEEDBACK' is disabled.
            Jun 27 12:44:10 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:10 0 [Note] Recovering after a crash using /var/log/mysql/mariadb-bin
            Jun 27 12:44:10 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:10 0 [Note] Starting crash recovery...
            Jun 27 12:44:10 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:10 0 [Note] Crash recovery finished.
            Jun 27 12:44:10 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:10 0 [Note] InnoDB: Rolled back recovered transaction 11134298
            Jun 27 12:44:10 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:10 0 [Note] InnoDB: Rollback of non-prepared transactions completed
            Jun 27 12:44:10 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:10 0 [Note] Server socket created on IP: '0.0.0.0'.
            Jun 27 12:44:10 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:10 0 [Note] Reading of all Master_info entries succeeded
            Jun 27 12:44:10 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:10 0 [Note] Added new Master_info '' to hash table
            Jun 27 12:44:10 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:10 0 [Warning] Neither --relay-log nor --relay-log-index were used; so replication may break when this MySQL server acts as a slave and has his hostname changed!! Please use '--log-basename=#' or '--relay-log=mysqld-relay-bin' to avoid this problem.
            Jun 27 12:44:11 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:11 10 [Note] Slave I/O thread: Start asynchronous replication to master 'replication_user@10.130.0.25:3306' in log 'mariadb-bin.000712' at position 4
            Jun 27 12:44:11 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:11 11 [Note] Slave SQL thread initialized, starting replication in log 'mariadb-bin.000712' at position 4, relay log './mysqld-relay-bin.000001' position: 4; GTID position '0-1-5332814'
            Jun 27 12:44:11 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:11 0 [Note] /usr/sbin/mysqld: ready for connections.
            Jun 27 12:44:11 vox-db1-read1 mysqld[12132]: Version: '10.4.12-MariaDB-1:10.4.12+maria~bionic-log' socket: '/var/run/mysqld/mysqld.sock' port: 3306 mariadb.org binary distribution
            Jun 27 12:44:11 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:11 10 [Note] Slave I/O thread: connected to master 'replication_user@10.130.0.25:3306',replication starts at GTID position '0-1-5332814'
            Jun 27 12:44:11 vox-db1-read1 systemd[1]: Started MariaDB 10.4.12 database server.
            Jun 27 12:44:11 vox-db1-read1 /etc/mysql/debian-start[12175]: Upgrading MySQL tables if necessary.
            Jun 27 12:44:11 vox-db1-read1 /etc/mysql/debian-start[12178]: Looking for 'mysql' as: /usr/bin/mysql
            Jun 27 12:44:11 vox-db1-read1 /etc/mysql/debian-start[12178]: Looking for 'mysqlcheck' as: /usr/bin/mysqlcheck
            Jun 27 12:44:11 vox-db1-read1 /etc/mysql/debian-start[12178]: This installation of MariaDB is already upgraded to 10.4.12-MariaDB, use --force if you still need to run mysql_upgrade
            Jun 27 12:44:11 vox-db1-read1 /etc/mysql/debian-start[12186]: Checking for insecure root accounts.
            Jun 27 12:44:11 vox-db1-read1 /etc/mysql/debian-start[12190]: Triggering myisam-recover for all MyISAM tables and aria-recover for all Aria tables
            Jun 27 12:44:11 vox-db1-read1 mysqld[12132]: 200627 12:44:11 [ERROR] mysqld got signal 11 ;
            Jun 27 12:44:11 vox-db1-read1 mysqld[12132]: This could be because you hit a bug. It is also possible that this binary
            Jun 27 12:44:11 vox-db1-read1 mysqld[12132]: or one of the libraries it was linked against is corrupt, improperly built,
            Jun 27 12:44:11 vox-db1-read1 mysqld[12132]: or misconfigured. This error can also be caused by malfunctioning hardware.
            Jun 27 12:44:11 vox-db1-read1 mysqld[12132]: To report this bug, see https://mariadb.com/kb/en/reporting-bugs
            Jun 27 12:44:11 vox-db1-read1 mysqld[12132]: We will try our best to scrape up some info that will hopefully help
            Jun 27 12:44:11 vox-db1-read1 mysqld[12132]: diagnose the problem, but since we have already crashed,
            Jun 27 12:44:11 vox-db1-read1 mysqld[12132]: something is definitely wrong and this may fail.
            Jun 27 12:44:11 vox-db1-read1 mysqld[12132]: Server version: 10.4.12-MariaDB-1:10.4.12+maria~bionic-log
            Jun 27 12:44:11 vox-db1-read1 mysqld[12132]: key_buffer_size=134217728
            Jun 27 12:44:11 vox-db1-read1 mysqld[12132]: read_buffer_size=2097152
            Jun 27 12:44:11 vox-db1-read1 mysqld[12132]: max_used_connections=1
            ...and so on, it loops indefinitely.


            My theory is that something comes from Master that breaks Slaves.
            When first crash occured (same day I've setup replication, ~4-5 days ago) I tried to restore Slave from same backup I used to initilize it. And Slave could not be started. I suppose while catching up to Master it still encountered this breaking transaction and crashed. Then I just took a fresh backup from Master, and Slave started without problems.

            Since then every time Slaves crash I just take make backup (with mariabackup) and restore slaves. This is stupid I know, but what choice do we currently have? :)

            What additional information can I provide you with to find the root of this issue?

            Thanks!             		Hello.
            We have a Master-to-multi-Slave config and are regularly facing slaves crashes (they crash together at the same time, approx. once every 24 hours). Maybe this has something to do with load, but crashed did not happen under heavy load and still happened today with relatively (50%) of highest load we've had so far.

            Master 10.4.12 setup is Ubuntu 18.0.4 cloud VPS (yandex cloud) with 6 cores 100% quota, 6 Gbs RAM, 30 Gbs space.
            Slaves 10.4.12 and 10.4.13 setup is Ubuntu 18.0.4 cloud VPS (yandex cloud) with 4 cores 100% quota, 4 Gbs RAM, 30 Gbs space.

            binlo

            Replication is setup using this guide - https://mariadb.com/kb/en/setting-up-a-replication-slave-with-mariabackup/
            Everything works well, until slaves crash.

            Currently Web API sends DML statements to Master directly and SELECT statements are sent to HAProxy with wieght distribution 20/40/40 (Master/Slave1/Slave2).
            Before that we had Master 10.4.12 and only 1 Slave 10.4.12 with direct requests from API to Slave without HAProxy - Slave still crashed, I hoped with 2 slaves we would at least have 1 standing... but alas...

            We removed all triggers from the DB.
            We use GTIDs.
            We don't have any DDL statements running (except a few CREATE TEMPORARY TABLE).
            We use InnoDB only to store data.
            We don't have any FullText Indexes.
            We don't use compression.
            We don't use SSL between Mater and Slaves.
            Timezone on all machines is UTC.

            Here's syslog:

            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: 200627 12:44:00 [ERROR] mysqld got signal 11 ;
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: This could be because you hit a bug. It is also possible that this binary
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: or one of the libraries it was linked against is corrupt, improperly built,
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: or misconfigured. This error can also be caused by malfunctioning hardware.
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: To report this bug, see https://mariadb.com/kb/en/reporting-bugs
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: We will try our best to scrape up some info that will hopefully help
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: diagnose the problem, but since we have already crashed,
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: something is definitely wrong and this may fail.
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: Server version: 10.4.12-MariaDB-1:10.4.12+maria~bionic-log
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: key_buffer_size=134217728
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: read_buffer_size=2097152
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: max_used_connections=15
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: max_threads=77
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: thread_count=24
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: It is possible that mysqld could use up to
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 606038 K bytes of memory
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: Hope that's ok; if not, decrease some variables in the equation.
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: Thread pointer: 0x7f40500014f8
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: Attempting backtrace. You can use the following information to find out
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: where mysqld died. If you see no messages after this, something went
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: terribly wrong...
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: stack_bottom = 0x7f40fc901e98 thread_stack 0x49000
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: /usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x55b8a063ac3e]
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: /usr/sbin/mysqld(handle_fatal_signal+0x515)[0x55b8a00aecf5]
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: /lib/x86_64-linux-gnu/libpthread.so.0(+0x12890)[0x7f411d583890]
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: /lib/x86_64-linux-gnu/libc.so.6(+0x18f039)[0x7f411c008039]
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: /usr/sbin/mysqld(+0xae4170)[0x55b8a031e170]
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: /usr/sbin/mysqld(+0xa205dd)[0x55b8a025a5dd]
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: /usr/sbin/mysqld(+0xb10a85)[0x55b8a034aa85]
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: /usr/sbin/mysqld(+0xb116be)[0x55b8a034b6be]
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: /usr/sbin/mysqld(+0xb1224c)[0x55b8a034c24c]
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: /usr/sbin/mysqld(+0xae7959)[0x55b8a0321959]
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: /usr/sbin/mysqld(+0xad0db6)[0x55b8a030adb6]
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: /usr/sbin/mysqld(+0xb0b658)[0x55b8a0345658]
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: /usr/sbin/mysqld(+0xb1173f)[0x55b8a034b73f]
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: /usr/sbin/mysqld(+0xb1224c)[0x55b8a034c24c]
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: /usr/sbin/mysqld(+0xae73a4)[0x55b8a03213a4]
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: /usr/sbin/mysqld(+0xa21f8f)[0x55b8a025bf8f]
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: /usr/sbin/mysqld(_ZN7handler13ha_delete_rowEPKh+0x9f)[0x55b8a00ba7af]
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: /usr/sbin/mysqld(_ZN21Delete_rows_log_event11do_exec_rowEP14rpl_group_info+0x12d)[0x55b8a01b12ad]
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: /usr/sbin/mysqld(_ZN14Rows_log_event14do_apply_eventEP14rpl_group_info+0x25d)[0x55b8a01a440d]
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: /usr/sbin/mysqld(+0x5b6e90)[0x55b89fdf0e90]
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: /usr/sbin/mysqld(handle_slave_sql+0x17b4)[0x55b89fdfa344]
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: /usr/sbin/mysqld(+0xdab79a)[0x55b8a05e579a]
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: /lib/x86_64-linux-gnu/libpthread.so.0(+0x76db)[0x7f411d5786db]
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: /lib/x86_64-linux-gnu/libc.so.6(clone+0x3f)[0x7f411bf9a88f]
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: Trying to get some variables.
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: Some pointers may be invalid and cause the dump to abort.
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: Query (0x0):
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: Connection ID (thread ID): 49
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: Status: NOT_KILLED
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=on,condition_pushdown_from_having=on
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: information that should help you find out what is causing the crash.
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: Writing a core file...
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: Working directory at /var/lib/mysql
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: Resource Limits:
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: Limit Soft Limit Hard Limit Units
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: Max cpu time unlimited unlimited seconds
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: Max file size unlimited unlimited bytes
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: Max data size unlimited unlimited bytes
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: Max stack size 8388608 unlimited bytes
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: Max core file size 0 unlimited bytes
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: Max resident set unlimited unlimited bytes
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: Max processes 15602 15602 processes
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: Max open files 16364 16364 files
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: Max locked memory 16777216 16777216 bytes
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: Max address space unlimited unlimited bytes
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: Max file locks unlimited unlimited locks
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: Max pending signals 15602 15602 signals
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: Max msgqueue size 819200 819200 bytes
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: Max nice priority 0 0
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: Max realtime priority 0 0
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: Max realtime timeout unlimited unlimited us
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: Core pattern: core
            Jun 27 12:44:00 vox-db1-read1 kernel: [264684.731198] mysqld[7861]: segfault at 115 ip 00007f411c008039 sp 00007f40fc8fcfc8 error 6 in libc-2.27.so[7f411be79000+1e7000]
            Jun 27 12:44:00 vox-db1-read1 systemd[1]: mariadb.service: Main process exited, code=killed, status=11/SEGV
            Jun 27 12:44:00 vox-db1-read1 systemd[1]: mariadb.service: Failed with result 'signal'.
            Jun 27 12:44:05 vox-db1-read1 systemd[1]: mariadb.service: Service hold-off time over, scheduling restart.
            Jun 27 12:44:05 vox-db1-read1 systemd[1]: mariadb.service: Scheduled restart job, restart counter is at 1.
            Jun 27 12:44:05 vox-db1-read1 systemd[1]: Stopped MariaDB 10.4.12 database server.
            Jun 27 12:44:05 vox-db1-read1 systemd[1]: Starting MariaDB 10.4.12 database server...
            Jun 27 12:44:06 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:06 0 [Note] /usr/sbin/mysqld (mysqld 10.4.12-MariaDB-1:10.4.12+maria~bionic-log) starting as process 12132 ...
            Jun 27 12:44:06 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:06 0 [Note] InnoDB: Using Linux native AIO
            Jun 27 12:44:06 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:06 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
            Jun 27 12:44:06 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:06 0 [Note] InnoDB: Uses event mutexes
            Jun 27 12:44:06 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:06 0 [Note] InnoDB: Compressed tables use zlib 1.2.11
            Jun 27 12:44:06 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:06 0 [Note] InnoDB: Number of pools: 1
            Jun 27 12:44:06 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:06 0 [Note] InnoDB: Using SSE2 crc32 instructions
            Jun 27 12:44:06 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:06 0 [Note] mysqld: O_TMPFILE is not supported on /tmp (disabling future attempts)
            Jun 27 12:44:06 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:06 0 [Note] InnoDB: Initializing buffer pool, total size = 1G, instances = 8, chunk size = 128M
            Jun 27 12:44:06 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:06 0 [Note] InnoDB: Completed initialization of buffer pool
            Jun 27 12:44:06 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:06 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority().
            Jun 27 12:44:07 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:07 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=142473402785
            Jun 27 12:44:10 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:10 0 [Note] InnoDB: 1 transaction(s) which must be rolled back or cleaned up in total 2 row operations to undo
            Jun 27 12:44:10 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:10 0 [Note] InnoDB: Trx id counter is 11134299
            Jun 27 12:44:10 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:10 0 [Note] InnoDB: Starting final batch to recover 63 pages from redo log.
            Jun 27 12:44:10 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:10 0 [Note] InnoDB: Last binlog file '/var/log/mysql/mariadb-bin.000634', position 59120658
            Jun 27 12:44:10 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:10 0 [Note] InnoDB: 128 out of 128 rollback segments are active.
            Jun 27 12:44:10 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:10 0 [Note] InnoDB: Starting in background the rollback of recovered transactions
            Jun 27 12:44:10 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:10 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1"
            Jun 27 12:44:10 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:10 0 [Note] InnoDB: Creating shared tablespace for temporary tables
            Jun 27 12:44:10 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:10 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ...
            Jun 27 12:44:10 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:10 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB.
            Jun 27 12:44:10 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:10 0 [Note] InnoDB: 10.4.12 started; log sequence number 142473413250; transaction id 11134301
            Jun 27 12:44:10 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:10 0 [Note] InnoDB: Loading buffer pool(s) from /var/lib/mysql/ib_buffer_pool
            Jun 27 12:44:10 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:10 0 [Note] Plugin 'FEEDBACK' is disabled.
            Jun 27 12:44:10 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:10 0 [Note] Recovering after a crash using /var/log/mysql/mariadb-bin
            Jun 27 12:44:10 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:10 0 [Note] Starting crash recovery...
            Jun 27 12:44:10 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:10 0 [Note] Crash recovery finished.
            Jun 27 12:44:10 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:10 0 [Note] InnoDB: Rolled back recovered transaction 11134298
            Jun 27 12:44:10 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:10 0 [Note] InnoDB: Rollback of non-prepared transactions completed
            Jun 27 12:44:10 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:10 0 [Note] Server socket created on IP: '0.0.0.0'.
            Jun 27 12:44:10 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:10 0 [Note] Reading of all Master_info entries succeeded
            Jun 27 12:44:10 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:10 0 [Note] Added new Master_info '' to hash table
            Jun 27 12:44:10 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:10 0 [Warning] Neither --relay-log nor --relay-log-index were used; so replication may break when this MySQL server acts as a slave and has his hostname changed!! Please use '--log-basename=#' or '--relay-log=mysqld-relay-bin' to avoid this problem.
            Jun 27 12:44:11 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:11 10 [Note] Slave I/O thread: Start asynchronous replication to master 'replication_user@10.130.0.25:3306' in log 'mariadb-bin.000712' at position 4
            Jun 27 12:44:11 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:11 11 [Note] Slave SQL thread initialized, starting replication in log 'mariadb-bin.000712' at position 4, relay log './mysqld-relay-bin.000001' position: 4; GTID position '0-1-5332814'
            Jun 27 12:44:11 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:11 0 [Note] /usr/sbin/mysqld: ready for connections.
            Jun 27 12:44:11 vox-db1-read1 mysqld[12132]: Version: '10.4.12-MariaDB-1:10.4.12+maria~bionic-log' socket: '/var/run/mysqld/mysqld.sock' port: 3306 mariadb.org binary distribution
            Jun 27 12:44:11 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:11 10 [Note] Slave I/O thread: connected to master 'replication_user@10.130.0.25:3306',replication starts at GTID position '0-1-5332814'
            Jun 27 12:44:11 vox-db1-read1 systemd[1]: Started MariaDB 10.4.12 database server.
            Jun 27 12:44:11 vox-db1-read1 /etc/mysql/debian-start[12175]: Upgrading MySQL tables if necessary.
            Jun 27 12:44:11 vox-db1-read1 /etc/mysql/debian-start[12178]: Looking for 'mysql' as: /usr/bin/mysql
            Jun 27 12:44:11 vox-db1-read1 /etc/mysql/debian-start[12178]: Looking for 'mysqlcheck' as: /usr/bin/mysqlcheck
            Jun 27 12:44:11 vox-db1-read1 /etc/mysql/debian-start[12178]: This installation of MariaDB is already upgraded to 10.4.12-MariaDB, use --force if you still need to run mysql_upgrade
            Jun 27 12:44:11 vox-db1-read1 /etc/mysql/debian-start[12186]: Checking for insecure root accounts.
            Jun 27 12:44:11 vox-db1-read1 /etc/mysql/debian-start[12190]: Triggering myisam-recover for all MyISAM tables and aria-recover for all Aria tables
            Jun 27 12:44:11 vox-db1-read1 mysqld[12132]: 200627 12:44:11 [ERROR] mysqld got signal 11 ;
            Jun 27 12:44:11 vox-db1-read1 mysqld[12132]: This could be because you hit a bug. It is also possible that this binary
            Jun 27 12:44:11 vox-db1-read1 mysqld[12132]: or one of the libraries it was linked against is corrupt, improperly built,
            Jun 27 12:44:11 vox-db1-read1 mysqld[12132]: or misconfigured. This error can also be caused by malfunctioning hardware.
            Jun 27 12:44:11 vox-db1-read1 mysqld[12132]: To report this bug, see https://mariadb.com/kb/en/reporting-bugs
            Jun 27 12:44:11 vox-db1-read1 mysqld[12132]: We will try our best to scrape up some info that will hopefully help
            Jun 27 12:44:11 vox-db1-read1 mysqld[12132]: diagnose the problem, but since we have already crashed,
            Jun 27 12:44:11 vox-db1-read1 mysqld[12132]: something is definitely wrong and this may fail.
            Jun 27 12:44:11 vox-db1-read1 mysqld[12132]: Server version: 10.4.12-MariaDB-1:10.4.12+maria~bionic-log
            Jun 27 12:44:11 vox-db1-read1 mysqld[12132]: key_buffer_size=134217728
            Jun 27 12:44:11 vox-db1-read1 mysqld[12132]: read_buffer_size=2097152
            Jun 27 12:44:11 vox-db1-read1 mysqld[12132]: max_used_connections=1
            ...and so on, it loops indefinitely.


            My theory is that something comes from Master that breaks Slaves.
            When first crash occured (same day I've setup replication, ~4-5 days ago) I tried to restore Slave from same backup I used to initilize it. And Slave could not be started. I suppose while catching up to Master it still encountered this breaking transaction and crashed. Then I just took a fresh backup from Master, and Slave started without problems.

            Since then every time Slaves crash I just take make backup (with mariabackup) and restore slaves. This is stupid I know, but what choice do we currently have? :)

            What additional information can I provide you with to find the root of this issue?

            Thanks!
            Grigoriy Grigoriy made changes -
            Description Hello.
            We have a Master-to-multi-Slave config and are regularly facing slaves crashes (they crash together at the same time, approx. once every 24 hours). Maybe this has something to do with load, but crashed did not happen under heavy load and still happened today with relatively (50%) of highest load we've had so far.

            Master 10.4.12 setup is Ubuntu 18.0.4 cloud VPS (yandex cloud) with 6 cores 100% quota, 6 Gbs RAM, 30 Gbs space.
            Slaves 10.4.12 and 10.4.13 setup is Ubuntu 18.0.4 cloud VPS (yandex cloud) with 4 cores 100% quota, 4 Gbs RAM, 30 Gbs space.

            binlo

            Replication is setup using this guide - https://mariadb.com/kb/en/setting-up-a-replication-slave-with-mariabackup/
            Everything works well, until slaves crash.

            Currently Web API sends DML statements to Master directly and SELECT statements are sent to HAProxy with wieght distribution 20/40/40 (Master/Slave1/Slave2).
            Before that we had Master 10.4.12 and only 1 Slave 10.4.12 with direct requests from API to Slave without HAProxy - Slave still crashed, I hoped with 2 slaves we would at least have 1 standing... but alas...

            We removed all triggers from the DB.
            We use GTIDs.
            We don't have any DDL statements running (except a few CREATE TEMPORARY TABLE).
            We use InnoDB only to store data.
            We don't have any FullText Indexes.
            We don't use compression.
            We don't use SSL between Mater and Slaves.
            Timezone on all machines is UTC.

            Here's syslog:

            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: 200627 12:44:00 [ERROR] mysqld got signal 11 ;
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: This could be because you hit a bug. It is also possible that this binary
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: or one of the libraries it was linked against is corrupt, improperly built,
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: or misconfigured. This error can also be caused by malfunctioning hardware.
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: To report this bug, see https://mariadb.com/kb/en/reporting-bugs
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: We will try our best to scrape up some info that will hopefully help
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: diagnose the problem, but since we have already crashed,
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: something is definitely wrong and this may fail.
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: Server version: 10.4.12-MariaDB-1:10.4.12+maria~bionic-log
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: key_buffer_size=134217728
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: read_buffer_size=2097152
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: max_used_connections=15
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: max_threads=77
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: thread_count=24
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: It is possible that mysqld could use up to
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 606038 K bytes of memory
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: Hope that's ok; if not, decrease some variables in the equation.
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: Thread pointer: 0x7f40500014f8
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: Attempting backtrace. You can use the following information to find out
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: where mysqld died. If you see no messages after this, something went
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: terribly wrong...
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: stack_bottom = 0x7f40fc901e98 thread_stack 0x49000
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: /usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x55b8a063ac3e]
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: /usr/sbin/mysqld(handle_fatal_signal+0x515)[0x55b8a00aecf5]
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: /lib/x86_64-linux-gnu/libpthread.so.0(+0x12890)[0x7f411d583890]
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: /lib/x86_64-linux-gnu/libc.so.6(+0x18f039)[0x7f411c008039]
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: /usr/sbin/mysqld(+0xae4170)[0x55b8a031e170]
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: /usr/sbin/mysqld(+0xa205dd)[0x55b8a025a5dd]
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: /usr/sbin/mysqld(+0xb10a85)[0x55b8a034aa85]
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: /usr/sbin/mysqld(+0xb116be)[0x55b8a034b6be]
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: /usr/sbin/mysqld(+0xb1224c)[0x55b8a034c24c]
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: /usr/sbin/mysqld(+0xae7959)[0x55b8a0321959]
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: /usr/sbin/mysqld(+0xad0db6)[0x55b8a030adb6]
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: /usr/sbin/mysqld(+0xb0b658)[0x55b8a0345658]
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: /usr/sbin/mysqld(+0xb1173f)[0x55b8a034b73f]
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: /usr/sbin/mysqld(+0xb1224c)[0x55b8a034c24c]
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: /usr/sbin/mysqld(+0xae73a4)[0x55b8a03213a4]
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: /usr/sbin/mysqld(+0xa21f8f)[0x55b8a025bf8f]
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: /usr/sbin/mysqld(_ZN7handler13ha_delete_rowEPKh+0x9f)[0x55b8a00ba7af]
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: /usr/sbin/mysqld(_ZN21Delete_rows_log_event11do_exec_rowEP14rpl_group_info+0x12d)[0x55b8a01b12ad]
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: /usr/sbin/mysqld(_ZN14Rows_log_event14do_apply_eventEP14rpl_group_info+0x25d)[0x55b8a01a440d]
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: /usr/sbin/mysqld(+0x5b6e90)[0x55b89fdf0e90]
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: /usr/sbin/mysqld(handle_slave_sql+0x17b4)[0x55b89fdfa344]
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: /usr/sbin/mysqld(+0xdab79a)[0x55b8a05e579a]
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: /lib/x86_64-linux-gnu/libpthread.so.0(+0x76db)[0x7f411d5786db]
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: /lib/x86_64-linux-gnu/libc.so.6(clone+0x3f)[0x7f411bf9a88f]
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: Trying to get some variables.
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: Some pointers may be invalid and cause the dump to abort.
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: Query (0x0):
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: Connection ID (thread ID): 49
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: Status: NOT_KILLED
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=on,condition_pushdown_from_having=on
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: information that should help you find out what is causing the crash.
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: Writing a core file...
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: Working directory at /var/lib/mysql
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: Resource Limits:
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: Limit Soft Limit Hard Limit Units
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: Max cpu time unlimited unlimited seconds
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: Max file size unlimited unlimited bytes
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: Max data size unlimited unlimited bytes
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: Max stack size 8388608 unlimited bytes
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: Max core file size 0 unlimited bytes
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: Max resident set unlimited unlimited bytes
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: Max processes 15602 15602 processes
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: Max open files 16364 16364 files
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: Max locked memory 16777216 16777216 bytes
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: Max address space unlimited unlimited bytes
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: Max file locks unlimited unlimited locks
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: Max pending signals 15602 15602 signals
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: Max msgqueue size 819200 819200 bytes
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: Max nice priority 0 0
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: Max realtime priority 0 0
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: Max realtime timeout unlimited unlimited us
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: Core pattern: core
            Jun 27 12:44:00 vox-db1-read1 kernel: [264684.731198] mysqld[7861]: segfault at 115 ip 00007f411c008039 sp 00007f40fc8fcfc8 error 6 in libc-2.27.so[7f411be79000+1e7000]
            Jun 27 12:44:00 vox-db1-read1 systemd[1]: mariadb.service: Main process exited, code=killed, status=11/SEGV
            Jun 27 12:44:00 vox-db1-read1 systemd[1]: mariadb.service: Failed with result 'signal'.
            Jun 27 12:44:05 vox-db1-read1 systemd[1]: mariadb.service: Service hold-off time over, scheduling restart.
            Jun 27 12:44:05 vox-db1-read1 systemd[1]: mariadb.service: Scheduled restart job, restart counter is at 1.
            Jun 27 12:44:05 vox-db1-read1 systemd[1]: Stopped MariaDB 10.4.12 database server.
            Jun 27 12:44:05 vox-db1-read1 systemd[1]: Starting MariaDB 10.4.12 database server...
            Jun 27 12:44:06 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:06 0 [Note] /usr/sbin/mysqld (mysqld 10.4.12-MariaDB-1:10.4.12+maria~bionic-log) starting as process 12132 ...
            Jun 27 12:44:06 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:06 0 [Note] InnoDB: Using Linux native AIO
            Jun 27 12:44:06 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:06 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
            Jun 27 12:44:06 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:06 0 [Note] InnoDB: Uses event mutexes
            Jun 27 12:44:06 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:06 0 [Note] InnoDB: Compressed tables use zlib 1.2.11
            Jun 27 12:44:06 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:06 0 [Note] InnoDB: Number of pools: 1
            Jun 27 12:44:06 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:06 0 [Note] InnoDB: Using SSE2 crc32 instructions
            Jun 27 12:44:06 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:06 0 [Note] mysqld: O_TMPFILE is not supported on /tmp (disabling future attempts)
            Jun 27 12:44:06 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:06 0 [Note] InnoDB: Initializing buffer pool, total size = 1G, instances = 8, chunk size = 128M
            Jun 27 12:44:06 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:06 0 [Note] InnoDB: Completed initialization of buffer pool
            Jun 27 12:44:06 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:06 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority().
            Jun 27 12:44:07 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:07 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=142473402785
            Jun 27 12:44:10 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:10 0 [Note] InnoDB: 1 transaction(s) which must be rolled back or cleaned up in total 2 row operations to undo
            Jun 27 12:44:10 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:10 0 [Note] InnoDB: Trx id counter is 11134299
            Jun 27 12:44:10 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:10 0 [Note] InnoDB: Starting final batch to recover 63 pages from redo log.
            Jun 27 12:44:10 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:10 0 [Note] InnoDB: Last binlog file '/var/log/mysql/mariadb-bin.000634', position 59120658
            Jun 27 12:44:10 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:10 0 [Note] InnoDB: 128 out of 128 rollback segments are active.
            Jun 27 12:44:10 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:10 0 [Note] InnoDB: Starting in background the rollback of recovered transactions
            Jun 27 12:44:10 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:10 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1"
            Jun 27 12:44:10 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:10 0 [Note] InnoDB: Creating shared tablespace for temporary tables
            Jun 27 12:44:10 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:10 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ...
            Jun 27 12:44:10 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:10 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB.
            Jun 27 12:44:10 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:10 0 [Note] InnoDB: 10.4.12 started; log sequence number 142473413250; transaction id 11134301
            Jun 27 12:44:10 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:10 0 [Note] InnoDB: Loading buffer pool(s) from /var/lib/mysql/ib_buffer_pool
            Jun 27 12:44:10 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:10 0 [Note] Plugin 'FEEDBACK' is disabled.
            Jun 27 12:44:10 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:10 0 [Note] Recovering after a crash using /var/log/mysql/mariadb-bin
            Jun 27 12:44:10 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:10 0 [Note] Starting crash recovery...
            Jun 27 12:44:10 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:10 0 [Note] Crash recovery finished.
            Jun 27 12:44:10 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:10 0 [Note] InnoDB: Rolled back recovered transaction 11134298
            Jun 27 12:44:10 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:10 0 [Note] InnoDB: Rollback of non-prepared transactions completed
            Jun 27 12:44:10 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:10 0 [Note] Server socket created on IP: '0.0.0.0'.
            Jun 27 12:44:10 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:10 0 [Note] Reading of all Master_info entries succeeded
            Jun 27 12:44:10 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:10 0 [Note] Added new Master_info '' to hash table
            Jun 27 12:44:10 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:10 0 [Warning] Neither --relay-log nor --relay-log-index were used; so replication may break when this MySQL server acts as a slave and has his hostname changed!! Please use '--log-basename=#' or '--relay-log=mysqld-relay-bin' to avoid this problem.
            Jun 27 12:44:11 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:11 10 [Note] Slave I/O thread: Start asynchronous replication to master 'replication_user@10.130.0.25:3306' in log 'mariadb-bin.000712' at position 4
            Jun 27 12:44:11 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:11 11 [Note] Slave SQL thread initialized, starting replication in log 'mariadb-bin.000712' at position 4, relay log './mysqld-relay-bin.000001' position: 4; GTID position '0-1-5332814'
            Jun 27 12:44:11 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:11 0 [Note] /usr/sbin/mysqld: ready for connections.
            Jun 27 12:44:11 vox-db1-read1 mysqld[12132]: Version: '10.4.12-MariaDB-1:10.4.12+maria~bionic-log' socket: '/var/run/mysqld/mysqld.sock' port: 3306 mariadb.org binary distribution
            Jun 27 12:44:11 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:11 10 [Note] Slave I/O thread: connected to master 'replication_user@10.130.0.25:3306',replication starts at GTID position '0-1-5332814'
            Jun 27 12:44:11 vox-db1-read1 systemd[1]: Started MariaDB 10.4.12 database server.
            Jun 27 12:44:11 vox-db1-read1 /etc/mysql/debian-start[12175]: Upgrading MySQL tables if necessary.
            Jun 27 12:44:11 vox-db1-read1 /etc/mysql/debian-start[12178]: Looking for 'mysql' as: /usr/bin/mysql
            Jun 27 12:44:11 vox-db1-read1 /etc/mysql/debian-start[12178]: Looking for 'mysqlcheck' as: /usr/bin/mysqlcheck
            Jun 27 12:44:11 vox-db1-read1 /etc/mysql/debian-start[12178]: This installation of MariaDB is already upgraded to 10.4.12-MariaDB, use --force if you still need to run mysql_upgrade
            Jun 27 12:44:11 vox-db1-read1 /etc/mysql/debian-start[12186]: Checking for insecure root accounts.
            Jun 27 12:44:11 vox-db1-read1 /etc/mysql/debian-start[12190]: Triggering myisam-recover for all MyISAM tables and aria-recover for all Aria tables
            Jun 27 12:44:11 vox-db1-read1 mysqld[12132]: 200627 12:44:11 [ERROR] mysqld got signal 11 ;
            Jun 27 12:44:11 vox-db1-read1 mysqld[12132]: This could be because you hit a bug. It is also possible that this binary
            Jun 27 12:44:11 vox-db1-read1 mysqld[12132]: or one of the libraries it was linked against is corrupt, improperly built,
            Jun 27 12:44:11 vox-db1-read1 mysqld[12132]: or misconfigured. This error can also be caused by malfunctioning hardware.
            Jun 27 12:44:11 vox-db1-read1 mysqld[12132]: To report this bug, see https://mariadb.com/kb/en/reporting-bugs
            Jun 27 12:44:11 vox-db1-read1 mysqld[12132]: We will try our best to scrape up some info that will hopefully help
            Jun 27 12:44:11 vox-db1-read1 mysqld[12132]: diagnose the problem, but since we have already crashed,
            Jun 27 12:44:11 vox-db1-read1 mysqld[12132]: something is definitely wrong and this may fail.
            Jun 27 12:44:11 vox-db1-read1 mysqld[12132]: Server version: 10.4.12-MariaDB-1:10.4.12+maria~bionic-log
            Jun 27 12:44:11 vox-db1-read1 mysqld[12132]: key_buffer_size=134217728
            Jun 27 12:44:11 vox-db1-read1 mysqld[12132]: read_buffer_size=2097152
            Jun 27 12:44:11 vox-db1-read1 mysqld[12132]: max_used_connections=1
            ...and so on, it loops indefinitely.


            My theory is that something comes from Master that breaks Slaves.
            When first crash occured (same day I've setup replication, ~4-5 days ago) I tried to restore Slave from same backup I used to initilize it. And Slave could not be started. I suppose while catching up to Master it still encountered this breaking transaction and crashed. Then I just took a fresh backup from Master, and Slave started without problems.

            Since then every time Slaves crash I just take make backup (with mariabackup) and restore slaves. This is stupid I know, but what choice do we currently have? :)

            What additional information can I provide you with to find the root of this issue?

            Thanks!             		Hello.
            We have a Master-to-multi-Slave config and are regularly facing slaves crashes (they crash together at the same time, approx. once every 24 hours). Maybe this has something to do with load, but crashed did not happen under heavy load and still happened today with relatively (50%) of highest load we've had so far.

            Master 10.4.12 setup is Ubuntu 18.0.4 cloud VPS (yandex cloud) with 6 cores 100% quota, 6 Gbs RAM, 30 Gbs space.
            Slaves 10.4.12 and 10.4.13 setup is Ubuntu 18.0.4 cloud VPS (yandex cloud) with 4 cores 100% quota, 4 Gbs RAM, 30 Gbs space.

            Replication is setup using this guide - https://mariadb.com/kb/en/setting-up-a-replication-slave-with-mariabackup/
            Everything works well, until slaves crash.

            Currently Web API sends DML statements to Master directly and SELECT statements are sent to HAProxy with wieght distribution 20/40/40 (Master/Slave1/Slave2).
            Before that we had Master 10.4.12 and only 1 Slave 10.4.12 with direct requests from API to Slave without HAProxy - Slave still crashed, I hoped with 2 slaves we would at least have 1 standing... but alas...

            We removed all triggers from the DB.
            We use GTIDs.
            We don't have any DDL statements running (except a few CREATE TEMPORARY TABLE).
            We use InnoDB only to store data.
            We don't have any FullText Indexes.
            We don't use compression.
            We don't use SSL between Mater and Slaves.
            Timezone on all machines is UTC.

            Here's syslog:

            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: 200627 12:44:00 [ERROR] mysqld got signal 11 ;
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: This could be because you hit a bug. It is also possible that this binary
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: or one of the libraries it was linked against is corrupt, improperly built,
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: or misconfigured. This error can also be caused by malfunctioning hardware.
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: To report this bug, see https://mariadb.com/kb/en/reporting-bugs
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: We will try our best to scrape up some info that will hopefully help
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: diagnose the problem, but since we have already crashed,
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: something is definitely wrong and this may fail.
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: Server version: 10.4.12-MariaDB-1:10.4.12+maria~bionic-log
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: key_buffer_size=134217728
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: read_buffer_size=2097152
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: max_used_connections=15
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: max_threads=77
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: thread_count=24
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: It is possible that mysqld could use up to
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 606038 K bytes of memory
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: Hope that's ok; if not, decrease some variables in the equation.
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: Thread pointer: 0x7f40500014f8
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: Attempting backtrace. You can use the following information to find out
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: where mysqld died. If you see no messages after this, something went
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: terribly wrong...
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: stack_bottom = 0x7f40fc901e98 thread_stack 0x49000
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: /usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x55b8a063ac3e]
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: /usr/sbin/mysqld(handle_fatal_signal+0x515)[0x55b8a00aecf5]
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: /lib/x86_64-linux-gnu/libpthread.so.0(+0x12890)[0x7f411d583890]
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: /lib/x86_64-linux-gnu/libc.so.6(+0x18f039)[0x7f411c008039]
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: /usr/sbin/mysqld(+0xae4170)[0x55b8a031e170]
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: /usr/sbin/mysqld(+0xa205dd)[0x55b8a025a5dd]
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: /usr/sbin/mysqld(+0xb10a85)[0x55b8a034aa85]
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: /usr/sbin/mysqld(+0xb116be)[0x55b8a034b6be]
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: /usr/sbin/mysqld(+0xb1224c)[0x55b8a034c24c]
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: /usr/sbin/mysqld(+0xae7959)[0x55b8a0321959]
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: /usr/sbin/mysqld(+0xad0db6)[0x55b8a030adb6]
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: /usr/sbin/mysqld(+0xb0b658)[0x55b8a0345658]
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: /usr/sbin/mysqld(+0xb1173f)[0x55b8a034b73f]
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: /usr/sbin/mysqld(+0xb1224c)[0x55b8a034c24c]
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: /usr/sbin/mysqld(+0xae73a4)[0x55b8a03213a4]
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: /usr/sbin/mysqld(+0xa21f8f)[0x55b8a025bf8f]
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: /usr/sbin/mysqld(_ZN7handler13ha_delete_rowEPKh+0x9f)[0x55b8a00ba7af]
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: /usr/sbin/mysqld(_ZN21Delete_rows_log_event11do_exec_rowEP14rpl_group_info+0x12d)[0x55b8a01b12ad]
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: /usr/sbin/mysqld(_ZN14Rows_log_event14do_apply_eventEP14rpl_group_info+0x25d)[0x55b8a01a440d]
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: /usr/sbin/mysqld(+0x5b6e90)[0x55b89fdf0e90]
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: /usr/sbin/mysqld(handle_slave_sql+0x17b4)[0x55b89fdfa344]
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: /usr/sbin/mysqld(+0xdab79a)[0x55b8a05e579a]
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: /lib/x86_64-linux-gnu/libpthread.so.0(+0x76db)[0x7f411d5786db]
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: /lib/x86_64-linux-gnu/libc.so.6(clone+0x3f)[0x7f411bf9a88f]
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: Trying to get some variables.
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: Some pointers may be invalid and cause the dump to abort.
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: Query (0x0):
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: Connection ID (thread ID): 49
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: Status: NOT_KILLED
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=on,condition_pushdown_from_having=on
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: information that should help you find out what is causing the crash.
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: Writing a core file...
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: Working directory at /var/lib/mysql
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: Resource Limits:
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: Limit Soft Limit Hard Limit Units
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: Max cpu time unlimited unlimited seconds
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: Max file size unlimited unlimited bytes
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: Max data size unlimited unlimited bytes
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: Max stack size 8388608 unlimited bytes
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: Max core file size 0 unlimited bytes
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: Max resident set unlimited unlimited bytes
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: Max processes 15602 15602 processes
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: Max open files 16364 16364 files
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: Max locked memory 16777216 16777216 bytes
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: Max address space unlimited unlimited bytes
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: Max file locks unlimited unlimited locks
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: Max pending signals 15602 15602 signals
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: Max msgqueue size 819200 819200 bytes
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: Max nice priority 0 0
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: Max realtime priority 0 0
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: Max realtime timeout unlimited unlimited us
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: Core pattern: core
            Jun 27 12:44:00 vox-db1-read1 kernel: [264684.731198] mysqld[7861]: segfault at 115 ip 00007f411c008039 sp 00007f40fc8fcfc8 error 6 in libc-2.27.so[7f411be79000+1e7000]
            Jun 27 12:44:00 vox-db1-read1 systemd[1]: mariadb.service: Main process exited, code=killed, status=11/SEGV
            Jun 27 12:44:00 vox-db1-read1 systemd[1]: mariadb.service: Failed with result 'signal'.
            Jun 27 12:44:05 vox-db1-read1 systemd[1]: mariadb.service: Service hold-off time over, scheduling restart.
            Jun 27 12:44:05 vox-db1-read1 systemd[1]: mariadb.service: Scheduled restart job, restart counter is at 1.
            Jun 27 12:44:05 vox-db1-read1 systemd[1]: Stopped MariaDB 10.4.12 database server.
            Jun 27 12:44:05 vox-db1-read1 systemd[1]: Starting MariaDB 10.4.12 database server...
            Jun 27 12:44:06 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:06 0 [Note] /usr/sbin/mysqld (mysqld 10.4.12-MariaDB-1:10.4.12+maria~bionic-log) starting as process 12132 ...
            Jun 27 12:44:06 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:06 0 [Note] InnoDB: Using Linux native AIO
            Jun 27 12:44:06 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:06 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
            Jun 27 12:44:06 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:06 0 [Note] InnoDB: Uses event mutexes
            Jun 27 12:44:06 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:06 0 [Note] InnoDB: Compressed tables use zlib 1.2.11
            Jun 27 12:44:06 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:06 0 [Note] InnoDB: Number of pools: 1
            Jun 27 12:44:06 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:06 0 [Note] InnoDB: Using SSE2 crc32 instructions
            Jun 27 12:44:06 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:06 0 [Note] mysqld: O_TMPFILE is not supported on /tmp (disabling future attempts)
            Jun 27 12:44:06 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:06 0 [Note] InnoDB: Initializing buffer pool, total size = 1G, instances = 8, chunk size = 128M
            Jun 27 12:44:06 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:06 0 [Note] InnoDB: Completed initialization of buffer pool
            Jun 27 12:44:06 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:06 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority().
            Jun 27 12:44:07 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:07 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=142473402785
            Jun 27 12:44:10 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:10 0 [Note] InnoDB: 1 transaction(s) which must be rolled back or cleaned up in total 2 row operations to undo
            Jun 27 12:44:10 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:10 0 [Note] InnoDB: Trx id counter is 11134299
            Jun 27 12:44:10 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:10 0 [Note] InnoDB: Starting final batch to recover 63 pages from redo log.
            Jun 27 12:44:10 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:10 0 [Note] InnoDB: Last binlog file '/var/log/mysql/mariadb-bin.000634', position 59120658
            Jun 27 12:44:10 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:10 0 [Note] InnoDB: 128 out of 128 rollback segments are active.
            Jun 27 12:44:10 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:10 0 [Note] InnoDB: Starting in background the rollback of recovered transactions
            Jun 27 12:44:10 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:10 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1"
            Jun 27 12:44:10 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:10 0 [Note] InnoDB: Creating shared tablespace for temporary tables
            Jun 27 12:44:10 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:10 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ...
            Jun 27 12:44:10 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:10 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB.
            Jun 27 12:44:10 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:10 0 [Note] InnoDB: 10.4.12 started; log sequence number 142473413250; transaction id 11134301
            Jun 27 12:44:10 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:10 0 [Note] InnoDB: Loading buffer pool(s) from /var/lib/mysql/ib_buffer_pool
            Jun 27 12:44:10 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:10 0 [Note] Plugin 'FEEDBACK' is disabled.
            Jun 27 12:44:10 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:10 0 [Note] Recovering after a crash using /var/log/mysql/mariadb-bin
            Jun 27 12:44:10 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:10 0 [Note] Starting crash recovery...
            Jun 27 12:44:10 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:10 0 [Note] Crash recovery finished.
            Jun 27 12:44:10 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:10 0 [Note] InnoDB: Rolled back recovered transaction 11134298
            Jun 27 12:44:10 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:10 0 [Note] InnoDB: Rollback of non-prepared transactions completed
            Jun 27 12:44:10 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:10 0 [Note] Server socket created on IP: '0.0.0.0'.
            Jun 27 12:44:10 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:10 0 [Note] Reading of all Master_info entries succeeded
            Jun 27 12:44:10 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:10 0 [Note] Added new Master_info '' to hash table
            Jun 27 12:44:10 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:10 0 [Warning] Neither --relay-log nor --relay-log-index were used; so replication may break when this MySQL server acts as a slave and has his hostname changed!! Please use '--log-basename=#' or '--relay-log=mysqld-relay-bin' to avoid this problem.
            Jun 27 12:44:11 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:11 10 [Note] Slave I/O thread: Start asynchronous replication to master 'replication_user@10.130.0.25:3306' in log 'mariadb-bin.000712' at position 4
            Jun 27 12:44:11 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:11 11 [Note] Slave SQL thread initialized, starting replication in log 'mariadb-bin.000712' at position 4, relay log './mysqld-relay-bin.000001' position: 4; GTID position '0-1-5332814'
            Jun 27 12:44:11 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:11 0 [Note] /usr/sbin/mysqld: ready for connections.
            Jun 27 12:44:11 vox-db1-read1 mysqld[12132]: Version: '10.4.12-MariaDB-1:10.4.12+maria~bionic-log' socket: '/var/run/mysqld/mysqld.sock' port: 3306 mariadb.org binary distribution
            Jun 27 12:44:11 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:11 10 [Note] Slave I/O thread: connected to master 'replication_user@10.130.0.25:3306',replication starts at GTID position '0-1-5332814'
            Jun 27 12:44:11 vox-db1-read1 systemd[1]: Started MariaDB 10.4.12 database server.
            Jun 27 12:44:11 vox-db1-read1 /etc/mysql/debian-start[12175]: Upgrading MySQL tables if necessary.
            Jun 27 12:44:11 vox-db1-read1 /etc/mysql/debian-start[12178]: Looking for 'mysql' as: /usr/bin/mysql
            Jun 27 12:44:11 vox-db1-read1 /etc/mysql/debian-start[12178]: Looking for 'mysqlcheck' as: /usr/bin/mysqlcheck
            Jun 27 12:44:11 vox-db1-read1 /etc/mysql/debian-start[12178]: This installation of MariaDB is already upgraded to 10.4.12-MariaDB, use --force if you still need to run mysql_upgrade
            Jun 27 12:44:11 vox-db1-read1 /etc/mysql/debian-start[12186]: Checking for insecure root accounts.
            Jun 27 12:44:11 vox-db1-read1 /etc/mysql/debian-start[12190]: Triggering myisam-recover for all MyISAM tables and aria-recover for all Aria tables
            Jun 27 12:44:11 vox-db1-read1 mysqld[12132]: 200627 12:44:11 [ERROR] mysqld got signal 11 ;
            Jun 27 12:44:11 vox-db1-read1 mysqld[12132]: This could be because you hit a bug. It is also possible that this binary
            Jun 27 12:44:11 vox-db1-read1 mysqld[12132]: or one of the libraries it was linked against is corrupt, improperly built,
            Jun 27 12:44:11 vox-db1-read1 mysqld[12132]: or misconfigured. This error can also be caused by malfunctioning hardware.
            Jun 27 12:44:11 vox-db1-read1 mysqld[12132]: To report this bug, see https://mariadb.com/kb/en/reporting-bugs
            Jun 27 12:44:11 vox-db1-read1 mysqld[12132]: We will try our best to scrape up some info that will hopefully help
            Jun 27 12:44:11 vox-db1-read1 mysqld[12132]: diagnose the problem, but since we have already crashed,
            Jun 27 12:44:11 vox-db1-read1 mysqld[12132]: something is definitely wrong and this may fail.
            Jun 27 12:44:11 vox-db1-read1 mysqld[12132]: Server version: 10.4.12-MariaDB-1:10.4.12+maria~bionic-log
            Jun 27 12:44:11 vox-db1-read1 mysqld[12132]: key_buffer_size=134217728
            Jun 27 12:44:11 vox-db1-read1 mysqld[12132]: read_buffer_size=2097152
            Jun 27 12:44:11 vox-db1-read1 mysqld[12132]: max_used_connections=1
            ...and so on, it loops indefinitely.


            My theory is that something comes from Master that breaks Slaves.
            When first crash occured (same day I've setup replication, ~4-5 days ago) I tried to restore Slave from same backup I used to initilize it. And Slave could not be started. I suppose while catching up to Master it still encountered this breaking transaction and crashed. Then I just took a fresh backup from Master, and Slave started without problems.

            Since then every time Slaves crash I just take make backup (with mariabackup) and restore slaves. This is stupid I know, but what choice do we currently have? :)

            What additional information can I provide you with to find the root of this issue?

            Thanks!
            Grigoriy Grigoriy made changes -
            Description Hello.
            We have a Master-to-multi-Slave config and are regularly facing slaves crashes (they crash together at the same time, approx. once every 24 hours). Maybe this has something to do with load, but crashed did not happen under heavy load and still happened today with relatively (50%) of highest load we've had so far.

            Master 10.4.12 setup is Ubuntu 18.0.4 cloud VPS (yandex cloud) with 6 cores 100% quota, 6 Gbs RAM, 30 Gbs space.
            Slaves 10.4.12 and 10.4.13 setup is Ubuntu 18.0.4 cloud VPS (yandex cloud) with 4 cores 100% quota, 4 Gbs RAM, 30 Gbs space.

            Replication is setup using this guide - https://mariadb.com/kb/en/setting-up-a-replication-slave-with-mariabackup/
            Everything works well, until slaves crash.

            Currently Web API sends DML statements to Master directly and SELECT statements are sent to HAProxy with wieght distribution 20/40/40 (Master/Slave1/Slave2).
            Before that we had Master 10.4.12 and only 1 Slave 10.4.12 with direct requests from API to Slave without HAProxy - Slave still crashed, I hoped with 2 slaves we would at least have 1 standing... but alas...

            We removed all triggers from the DB.
            We use GTIDs.
            We don't have any DDL statements running (except a few CREATE TEMPORARY TABLE).
            We use InnoDB only to store data.
            We don't have any FullText Indexes.
            We don't use compression.
            We don't use SSL between Mater and Slaves.
            Timezone on all machines is UTC.

            Here's syslog:

            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: 200627 12:44:00 [ERROR] mysqld got signal 11 ;
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: This could be because you hit a bug. It is also possible that this binary
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: or one of the libraries it was linked against is corrupt, improperly built,
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: or misconfigured. This error can also be caused by malfunctioning hardware.
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: To report this bug, see https://mariadb.com/kb/en/reporting-bugs
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: We will try our best to scrape up some info that will hopefully help
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: diagnose the problem, but since we have already crashed,
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: something is definitely wrong and this may fail.
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: Server version: 10.4.12-MariaDB-1:10.4.12+maria~bionic-log
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: key_buffer_size=134217728
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: read_buffer_size=2097152
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: max_used_connections=15
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: max_threads=77
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: thread_count=24
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: It is possible that mysqld could use up to
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 606038 K bytes of memory
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: Hope that's ok; if not, decrease some variables in the equation.
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: Thread pointer: 0x7f40500014f8
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: Attempting backtrace. You can use the following information to find out
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: where mysqld died. If you see no messages after this, something went
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: terribly wrong...
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: stack_bottom = 0x7f40fc901e98 thread_stack 0x49000
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: /usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x55b8a063ac3e]
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: /usr/sbin/mysqld(handle_fatal_signal+0x515)[0x55b8a00aecf5]
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: /lib/x86_64-linux-gnu/libpthread.so.0(+0x12890)[0x7f411d583890]
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: /lib/x86_64-linux-gnu/libc.so.6(+0x18f039)[0x7f411c008039]
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: /usr/sbin/mysqld(+0xae4170)[0x55b8a031e170]
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: /usr/sbin/mysqld(+0xa205dd)[0x55b8a025a5dd]
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: /usr/sbin/mysqld(+0xb10a85)[0x55b8a034aa85]
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: /usr/sbin/mysqld(+0xb116be)[0x55b8a034b6be]
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: /usr/sbin/mysqld(+0xb1224c)[0x55b8a034c24c]
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: /usr/sbin/mysqld(+0xae7959)[0x55b8a0321959]
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: /usr/sbin/mysqld(+0xad0db6)[0x55b8a030adb6]
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: /usr/sbin/mysqld(+0xb0b658)[0x55b8a0345658]
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: /usr/sbin/mysqld(+0xb1173f)[0x55b8a034b73f]
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: /usr/sbin/mysqld(+0xb1224c)[0x55b8a034c24c]
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: /usr/sbin/mysqld(+0xae73a4)[0x55b8a03213a4]
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: /usr/sbin/mysqld(+0xa21f8f)[0x55b8a025bf8f]
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: /usr/sbin/mysqld(_ZN7handler13ha_delete_rowEPKh+0x9f)[0x55b8a00ba7af]
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: /usr/sbin/mysqld(_ZN21Delete_rows_log_event11do_exec_rowEP14rpl_group_info+0x12d)[0x55b8a01b12ad]
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: /usr/sbin/mysqld(_ZN14Rows_log_event14do_apply_eventEP14rpl_group_info+0x25d)[0x55b8a01a440d]
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: /usr/sbin/mysqld(+0x5b6e90)[0x55b89fdf0e90]
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: /usr/sbin/mysqld(handle_slave_sql+0x17b4)[0x55b89fdfa344]
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: /usr/sbin/mysqld(+0xdab79a)[0x55b8a05e579a]
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: /lib/x86_64-linux-gnu/libpthread.so.0(+0x76db)[0x7f411d5786db]
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: /lib/x86_64-linux-gnu/libc.so.6(clone+0x3f)[0x7f411bf9a88f]
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: Trying to get some variables.
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: Some pointers may be invalid and cause the dump to abort.
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: Query (0x0):
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: Connection ID (thread ID): 49
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: Status: NOT_KILLED
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=on,condition_pushdown_from_having=on
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: information that should help you find out what is causing the crash.
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: Writing a core file...
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: Working directory at /var/lib/mysql
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: Resource Limits:
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: Limit Soft Limit Hard Limit Units
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: Max cpu time unlimited unlimited seconds
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: Max file size unlimited unlimited bytes
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: Max data size unlimited unlimited bytes
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: Max stack size 8388608 unlimited bytes
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: Max core file size 0 unlimited bytes
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: Max resident set unlimited unlimited bytes
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: Max processes 15602 15602 processes
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: Max open files 16364 16364 files
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: Max locked memory 16777216 16777216 bytes
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: Max address space unlimited unlimited bytes
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: Max file locks unlimited unlimited locks
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: Max pending signals 15602 15602 signals
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: Max msgqueue size 819200 819200 bytes
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: Max nice priority 0 0
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: Max realtime priority 0 0
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: Max realtime timeout unlimited unlimited us
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: Core pattern: core
            Jun 27 12:44:00 vox-db1-read1 kernel: [264684.731198] mysqld[7861]: segfault at 115 ip 00007f411c008039 sp 00007f40fc8fcfc8 error 6 in libc-2.27.so[7f411be79000+1e7000]
            Jun 27 12:44:00 vox-db1-read1 systemd[1]: mariadb.service: Main process exited, code=killed, status=11/SEGV
            Jun 27 12:44:00 vox-db1-read1 systemd[1]: mariadb.service: Failed with result 'signal'.
            Jun 27 12:44:05 vox-db1-read1 systemd[1]: mariadb.service: Service hold-off time over, scheduling restart.
            Jun 27 12:44:05 vox-db1-read1 systemd[1]: mariadb.service: Scheduled restart job, restart counter is at 1.
            Jun 27 12:44:05 vox-db1-read1 systemd[1]: Stopped MariaDB 10.4.12 database server.
            Jun 27 12:44:05 vox-db1-read1 systemd[1]: Starting MariaDB 10.4.12 database server...
            Jun 27 12:44:06 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:06 0 [Note] /usr/sbin/mysqld (mysqld 10.4.12-MariaDB-1:10.4.12+maria~bionic-log) starting as process 12132 ...
            Jun 27 12:44:06 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:06 0 [Note] InnoDB: Using Linux native AIO
            Jun 27 12:44:06 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:06 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
            Jun 27 12:44:06 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:06 0 [Note] InnoDB: Uses event mutexes
            Jun 27 12:44:06 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:06 0 [Note] InnoDB: Compressed tables use zlib 1.2.11
            Jun 27 12:44:06 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:06 0 [Note] InnoDB: Number of pools: 1
            Jun 27 12:44:06 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:06 0 [Note] InnoDB: Using SSE2 crc32 instructions
            Jun 27 12:44:06 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:06 0 [Note] mysqld: O_TMPFILE is not supported on /tmp (disabling future attempts)
            Jun 27 12:44:06 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:06 0 [Note] InnoDB: Initializing buffer pool, total size = 1G, instances = 8, chunk size = 128M
            Jun 27 12:44:06 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:06 0 [Note] InnoDB: Completed initialization of buffer pool
            Jun 27 12:44:06 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:06 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority().
            Jun 27 12:44:07 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:07 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=142473402785
            Jun 27 12:44:10 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:10 0 [Note] InnoDB: 1 transaction(s) which must be rolled back or cleaned up in total 2 row operations to undo
            Jun 27 12:44:10 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:10 0 [Note] InnoDB: Trx id counter is 11134299
            Jun 27 12:44:10 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:10 0 [Note] InnoDB: Starting final batch to recover 63 pages from redo log.
            Jun 27 12:44:10 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:10 0 [Note] InnoDB: Last binlog file '/var/log/mysql/mariadb-bin.000634', position 59120658
            Jun 27 12:44:10 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:10 0 [Note] InnoDB: 128 out of 128 rollback segments are active.
            Jun 27 12:44:10 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:10 0 [Note] InnoDB: Starting in background the rollback of recovered transactions
            Jun 27 12:44:10 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:10 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1"
            Jun 27 12:44:10 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:10 0 [Note] InnoDB: Creating shared tablespace for temporary tables
            Jun 27 12:44:10 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:10 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ...
            Jun 27 12:44:10 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:10 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB.
            Jun 27 12:44:10 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:10 0 [Note] InnoDB: 10.4.12 started; log sequence number 142473413250; transaction id 11134301
            Jun 27 12:44:10 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:10 0 [Note] InnoDB: Loading buffer pool(s) from /var/lib/mysql/ib_buffer_pool
            Jun 27 12:44:10 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:10 0 [Note] Plugin 'FEEDBACK' is disabled.
            Jun 27 12:44:10 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:10 0 [Note] Recovering after a crash using /var/log/mysql/mariadb-bin
            Jun 27 12:44:10 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:10 0 [Note] Starting crash recovery...
            Jun 27 12:44:10 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:10 0 [Note] Crash recovery finished.
            Jun 27 12:44:10 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:10 0 [Note] InnoDB: Rolled back recovered transaction 11134298
            Jun 27 12:44:10 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:10 0 [Note] InnoDB: Rollback of non-prepared transactions completed
            Jun 27 12:44:10 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:10 0 [Note] Server socket created on IP: '0.0.0.0'.
            Jun 27 12:44:10 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:10 0 [Note] Reading of all Master_info entries succeeded
            Jun 27 12:44:10 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:10 0 [Note] Added new Master_info '' to hash table
            Jun 27 12:44:10 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:10 0 [Warning] Neither --relay-log nor --relay-log-index were used; so replication may break when this MySQL server acts as a slave and has his hostname changed!! Please use '--log-basename=#' or '--relay-log=mysqld-relay-bin' to avoid this problem.
            Jun 27 12:44:11 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:11 10 [Note] Slave I/O thread: Start asynchronous replication to master 'replication_user@10.130.0.25:3306' in log 'mariadb-bin.000712' at position 4
            Jun 27 12:44:11 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:11 11 [Note] Slave SQL thread initialized, starting replication in log 'mariadb-bin.000712' at position 4, relay log './mysqld-relay-bin.000001' position: 4; GTID position '0-1-5332814'
            Jun 27 12:44:11 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:11 0 [Note] /usr/sbin/mysqld: ready for connections.
            Jun 27 12:44:11 vox-db1-read1 mysqld[12132]: Version: '10.4.12-MariaDB-1:10.4.12+maria~bionic-log' socket: '/var/run/mysqld/mysqld.sock' port: 3306 mariadb.org binary distribution
            Jun 27 12:44:11 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:11 10 [Note] Slave I/O thread: connected to master 'replication_user@10.130.0.25:3306',replication starts at GTID position '0-1-5332814'
            Jun 27 12:44:11 vox-db1-read1 systemd[1]: Started MariaDB 10.4.12 database server.
            Jun 27 12:44:11 vox-db1-read1 /etc/mysql/debian-start[12175]: Upgrading MySQL tables if necessary.
            Jun 27 12:44:11 vox-db1-read1 /etc/mysql/debian-start[12178]: Looking for 'mysql' as: /usr/bin/mysql
            Jun 27 12:44:11 vox-db1-read1 /etc/mysql/debian-start[12178]: Looking for 'mysqlcheck' as: /usr/bin/mysqlcheck
            Jun 27 12:44:11 vox-db1-read1 /etc/mysql/debian-start[12178]: This installation of MariaDB is already upgraded to 10.4.12-MariaDB, use --force if you still need to run mysql_upgrade
            Jun 27 12:44:11 vox-db1-read1 /etc/mysql/debian-start[12186]: Checking for insecure root accounts.
            Jun 27 12:44:11 vox-db1-read1 /etc/mysql/debian-start[12190]: Triggering myisam-recover for all MyISAM tables and aria-recover for all Aria tables
            Jun 27 12:44:11 vox-db1-read1 mysqld[12132]: 200627 12:44:11 [ERROR] mysqld got signal 11 ;
            Jun 27 12:44:11 vox-db1-read1 mysqld[12132]: This could be because you hit a bug. It is also possible that this binary
            Jun 27 12:44:11 vox-db1-read1 mysqld[12132]: or one of the libraries it was linked against is corrupt, improperly built,
            Jun 27 12:44:11 vox-db1-read1 mysqld[12132]: or misconfigured. This error can also be caused by malfunctioning hardware.
            Jun 27 12:44:11 vox-db1-read1 mysqld[12132]: To report this bug, see https://mariadb.com/kb/en/reporting-bugs
            Jun 27 12:44:11 vox-db1-read1 mysqld[12132]: We will try our best to scrape up some info that will hopefully help
            Jun 27 12:44:11 vox-db1-read1 mysqld[12132]: diagnose the problem, but since we have already crashed,
            Jun 27 12:44:11 vox-db1-read1 mysqld[12132]: something is definitely wrong and this may fail.
            Jun 27 12:44:11 vox-db1-read1 mysqld[12132]: Server version: 10.4.12-MariaDB-1:10.4.12+maria~bionic-log
            Jun 27 12:44:11 vox-db1-read1 mysqld[12132]: key_buffer_size=134217728
            Jun 27 12:44:11 vox-db1-read1 mysqld[12132]: read_buffer_size=2097152
            Jun 27 12:44:11 vox-db1-read1 mysqld[12132]: max_used_connections=1
            ...and so on, it loops indefinitely.


            My theory is that something comes from Master that breaks Slaves.
            When first crash occured (same day I've setup replication, ~4-5 days ago) I tried to restore Slave from same backup I used to initilize it. And Slave could not be started. I suppose while catching up to Master it still encountered this breaking transaction and crashed. Then I just took a fresh backup from Master, and Slave started without problems.

            Since then every time Slaves crash I just take make backup (with mariabackup) and restore slaves. This is stupid I know, but what choice do we currently have? :)

            What additional information can I provide you with to find the root of this issue?

            Thanks!             		Hello.
            We have a Master-to-multi-Slave config and are regularly facing slaves crashes (they crash together at the same time, approx. once every 24 hours). Maybe this has something to do with load, but crashed did not happen under heavy load and still happened today with relatively (50%) of highest load we've had so far.

            Master 10.4.12 setup is Ubuntu 18.0.4 cloud VPS (yandex cloud) with 6 cores 100% quota, 6 Gbs RAM, 30 Gbs space.
            Slaves 10.4.12 and 10.4.13 setup is Ubuntu 18.0.4 cloud VPS (yandex cloud) with 4 cores 100% quota, 4 Gbs RAM, 30 Gbs space.

            Replication is setup using this guide - https://mariadb.com/kb/en/setting-up-a-replication-slave-with-mariabackup/
            Everything works well, until slaves crash.

            Currently Web API sends DML statements to Master directly and SELECT statements are sent to HAProxy with wieght distribution 20/40/40 (Master/Slave1/Slave2).
            Before that we had Master 10.4.12 and only 1 Slave 10.4.12 with direct requests from API to Slave without HAProxy - Slave still crashed, I hoped with 2 slaves we would at least have 1 standing... but alas...

            We removed all triggers from the DB.
            We use GTIDs.
            We now use mixed binlog (but id did crash with 'row' mode).
            We don't have any DDL statements running (except a few CREATE TEMPORARY TABLE).
            We use InnoDB only to store data.
            We don't have any FullText Indexes.
            We don't use compression.
            We don't use SSL between Mater and Slaves.
            Timezone on all machines is UTC.
            Data size now approximately 800Mb, InnoDb buffer pool size is 1024Mb, InnoDb log file size is128Mb.

            Here's syslog:

            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: 200627 12:44:00 [ERROR] mysqld got signal 11 ;
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: This could be because you hit a bug. It is also possible that this binary
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: or one of the libraries it was linked against is corrupt, improperly built,
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: or misconfigured. This error can also be caused by malfunctioning hardware.
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: To report this bug, see https://mariadb.com/kb/en/reporting-bugs
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: We will try our best to scrape up some info that will hopefully help
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: diagnose the problem, but since we have already crashed,
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: something is definitely wrong and this may fail.
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: Server version: 10.4.12-MariaDB-1:10.4.12+maria~bionic-log
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: key_buffer_size=134217728
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: read_buffer_size=2097152
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: max_used_connections=15
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: max_threads=77
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: thread_count=24
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: It is possible that mysqld could use up to
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 606038 K bytes of memory
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: Hope that's ok; if not, decrease some variables in the equation.
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: Thread pointer: 0x7f40500014f8
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: Attempting backtrace. You can use the following information to find out
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: where mysqld died. If you see no messages after this, something went
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: terribly wrong...
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: stack_bottom = 0x7f40fc901e98 thread_stack 0x49000
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: /usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x55b8a063ac3e]
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: /usr/sbin/mysqld(handle_fatal_signal+0x515)[0x55b8a00aecf5]
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: /lib/x86_64-linux-gnu/libpthread.so.0(+0x12890)[0x7f411d583890]
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: /lib/x86_64-linux-gnu/libc.so.6(+0x18f039)[0x7f411c008039]
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: /usr/sbin/mysqld(+0xae4170)[0x55b8a031e170]
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: /usr/sbin/mysqld(+0xa205dd)[0x55b8a025a5dd]
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: /usr/sbin/mysqld(+0xb10a85)[0x55b8a034aa85]
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: /usr/sbin/mysqld(+0xb116be)[0x55b8a034b6be]
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: /usr/sbin/mysqld(+0xb1224c)[0x55b8a034c24c]
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: /usr/sbin/mysqld(+0xae7959)[0x55b8a0321959]
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: /usr/sbin/mysqld(+0xad0db6)[0x55b8a030adb6]
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: /usr/sbin/mysqld(+0xb0b658)[0x55b8a0345658]
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: /usr/sbin/mysqld(+0xb1173f)[0x55b8a034b73f]
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: /usr/sbin/mysqld(+0xb1224c)[0x55b8a034c24c]
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: /usr/sbin/mysqld(+0xae73a4)[0x55b8a03213a4]
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: /usr/sbin/mysqld(+0xa21f8f)[0x55b8a025bf8f]
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: /usr/sbin/mysqld(_ZN7handler13ha_delete_rowEPKh+0x9f)[0x55b8a00ba7af]
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: /usr/sbin/mysqld(_ZN21Delete_rows_log_event11do_exec_rowEP14rpl_group_info+0x12d)[0x55b8a01b12ad]
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: /usr/sbin/mysqld(_ZN14Rows_log_event14do_apply_eventEP14rpl_group_info+0x25d)[0x55b8a01a440d]
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: /usr/sbin/mysqld(+0x5b6e90)[0x55b89fdf0e90]
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: /usr/sbin/mysqld(handle_slave_sql+0x17b4)[0x55b89fdfa344]
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: /usr/sbin/mysqld(+0xdab79a)[0x55b8a05e579a]
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: /lib/x86_64-linux-gnu/libpthread.so.0(+0x76db)[0x7f411d5786db]
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: /lib/x86_64-linux-gnu/libc.so.6(clone+0x3f)[0x7f411bf9a88f]
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: Trying to get some variables.
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: Some pointers may be invalid and cause the dump to abort.
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: Query (0x0):
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: Connection ID (thread ID): 49
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: Status: NOT_KILLED
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=on,condition_pushdown_from_having=on
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: information that should help you find out what is causing the crash.
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: Writing a core file...
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: Working directory at /var/lib/mysql
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: Resource Limits:
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: Limit Soft Limit Hard Limit Units
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: Max cpu time unlimited unlimited seconds
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: Max file size unlimited unlimited bytes
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: Max data size unlimited unlimited bytes
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: Max stack size 8388608 unlimited bytes
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: Max core file size 0 unlimited bytes
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: Max resident set unlimited unlimited bytes
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: Max processes 15602 15602 processes
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: Max open files 16364 16364 files
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: Max locked memory 16777216 16777216 bytes
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: Max address space unlimited unlimited bytes
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: Max file locks unlimited unlimited locks
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: Max pending signals 15602 15602 signals
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: Max msgqueue size 819200 819200 bytes
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: Max nice priority 0 0
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: Max realtime priority 0 0
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: Max realtime timeout unlimited unlimited us
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: Core pattern: core
            Jun 27 12:44:00 vox-db1-read1 kernel: [264684.731198] mysqld[7861]: segfault at 115 ip 00007f411c008039 sp 00007f40fc8fcfc8 error 6 in libc-2.27.so[7f411be79000+1e7000]
            Jun 27 12:44:00 vox-db1-read1 systemd[1]: mariadb.service: Main process exited, code=killed, status=11/SEGV
            Jun 27 12:44:00 vox-db1-read1 systemd[1]: mariadb.service: Failed with result 'signal'.
            Jun 27 12:44:05 vox-db1-read1 systemd[1]: mariadb.service: Service hold-off time over, scheduling restart.
            Jun 27 12:44:05 vox-db1-read1 systemd[1]: mariadb.service: Scheduled restart job, restart counter is at 1.
            Jun 27 12:44:05 vox-db1-read1 systemd[1]: Stopped MariaDB 10.4.12 database server.
            Jun 27 12:44:05 vox-db1-read1 systemd[1]: Starting MariaDB 10.4.12 database server...
            Jun 27 12:44:06 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:06 0 [Note] /usr/sbin/mysqld (mysqld 10.4.12-MariaDB-1:10.4.12+maria~bionic-log) starting as process 12132 ...
            Jun 27 12:44:06 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:06 0 [Note] InnoDB: Using Linux native AIO
            Jun 27 12:44:06 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:06 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
            Jun 27 12:44:06 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:06 0 [Note] InnoDB: Uses event mutexes
            Jun 27 12:44:06 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:06 0 [Note] InnoDB: Compressed tables use zlib 1.2.11
            Jun 27 12:44:06 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:06 0 [Note] InnoDB: Number of pools: 1
            Jun 27 12:44:06 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:06 0 [Note] InnoDB: Using SSE2 crc32 instructions
            Jun 27 12:44:06 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:06 0 [Note] mysqld: O_TMPFILE is not supported on /tmp (disabling future attempts)
            Jun 27 12:44:06 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:06 0 [Note] InnoDB: Initializing buffer pool, total size = 1G, instances = 8, chunk size = 128M
            Jun 27 12:44:06 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:06 0 [Note] InnoDB: Completed initialization of buffer pool
            Jun 27 12:44:06 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:06 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority().
            Jun 27 12:44:07 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:07 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=142473402785
            Jun 27 12:44:10 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:10 0 [Note] InnoDB: 1 transaction(s) which must be rolled back or cleaned up in total 2 row operations to undo
            Jun 27 12:44:10 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:10 0 [Note] InnoDB: Trx id counter is 11134299
            Jun 27 12:44:10 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:10 0 [Note] InnoDB: Starting final batch to recover 63 pages from redo log.
            Jun 27 12:44:10 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:10 0 [Note] InnoDB: Last binlog file '/var/log/mysql/mariadb-bin.000634', position 59120658
            Jun 27 12:44:10 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:10 0 [Note] InnoDB: 128 out of 128 rollback segments are active.
            Jun 27 12:44:10 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:10 0 [Note] InnoDB: Starting in background the rollback of recovered transactions
            Jun 27 12:44:10 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:10 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1"
            Jun 27 12:44:10 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:10 0 [Note] InnoDB: Creating shared tablespace for temporary tables
            Jun 27 12:44:10 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:10 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ...
            Jun 27 12:44:10 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:10 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB.
            Jun 27 12:44:10 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:10 0 [Note] InnoDB: 10.4.12 started; log sequence number 142473413250; transaction id 11134301
            Jun 27 12:44:10 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:10 0 [Note] InnoDB: Loading buffer pool(s) from /var/lib/mysql/ib_buffer_pool
            Jun 27 12:44:10 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:10 0 [Note] Plugin 'FEEDBACK' is disabled.
            Jun 27 12:44:10 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:10 0 [Note] Recovering after a crash using /var/log/mysql/mariadb-bin
            Jun 27 12:44:10 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:10 0 [Note] Starting crash recovery...
            Jun 27 12:44:10 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:10 0 [Note] Crash recovery finished.
            Jun 27 12:44:10 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:10 0 [Note] InnoDB: Rolled back recovered transaction 11134298
            Jun 27 12:44:10 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:10 0 [Note] InnoDB: Rollback of non-prepared transactions completed
            Jun 27 12:44:10 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:10 0 [Note] Server socket created on IP: '0.0.0.0'.
            Jun 27 12:44:10 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:10 0 [Note] Reading of all Master_info entries succeeded
            Jun 27 12:44:10 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:10 0 [Note] Added new Master_info '' to hash table
            Jun 27 12:44:10 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:10 0 [Warning] Neither --relay-log nor --relay-log-index were used; so replication may break when this MySQL server acts as a slave and has his hostname changed!! Please use '--log-basename=#' or '--relay-log=mysqld-relay-bin' to avoid this problem.
            Jun 27 12:44:11 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:11 10 [Note] Slave I/O thread: Start asynchronous replication to master 'replication_user@10.130.0.25:3306' in log 'mariadb-bin.000712' at position 4
            Jun 27 12:44:11 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:11 11 [Note] Slave SQL thread initialized, starting replication in log 'mariadb-bin.000712' at position 4, relay log './mysqld-relay-bin.000001' position: 4; GTID position '0-1-5332814'
            Jun 27 12:44:11 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:11 0 [Note] /usr/sbin/mysqld: ready for connections.
            Jun 27 12:44:11 vox-db1-read1 mysqld[12132]: Version: '10.4.12-MariaDB-1:10.4.12+maria~bionic-log' socket: '/var/run/mysqld/mysqld.sock' port: 3306 mariadb.org binary distribution
            Jun 27 12:44:11 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:11 10 [Note] Slave I/O thread: connected to master 'replication_user@10.130.0.25:3306',replication starts at GTID position '0-1-5332814'
            Jun 27 12:44:11 vox-db1-read1 systemd[1]: Started MariaDB 10.4.12 database server.
            Jun 27 12:44:11 vox-db1-read1 /etc/mysql/debian-start[12175]: Upgrading MySQL tables if necessary.
            Jun 27 12:44:11 vox-db1-read1 /etc/mysql/debian-start[12178]: Looking for 'mysql' as: /usr/bin/mysql
            Jun 27 12:44:11 vox-db1-read1 /etc/mysql/debian-start[12178]: Looking for 'mysqlcheck' as: /usr/bin/mysqlcheck
            Jun 27 12:44:11 vox-db1-read1 /etc/mysql/debian-start[12178]: This installation of MariaDB is already upgraded to 10.4.12-MariaDB, use --force if you still need to run mysql_upgrade
            Jun 27 12:44:11 vox-db1-read1 /etc/mysql/debian-start[12186]: Checking for insecure root accounts.
            Jun 27 12:44:11 vox-db1-read1 /etc/mysql/debian-start[12190]: Triggering myisam-recover for all MyISAM tables and aria-recover for all Aria tables
            Jun 27 12:44:11 vox-db1-read1 mysqld[12132]: 200627 12:44:11 [ERROR] mysqld got signal 11 ;
            Jun 27 12:44:11 vox-db1-read1 mysqld[12132]: This could be because you hit a bug. It is also possible that this binary
            Jun 27 12:44:11 vox-db1-read1 mysqld[12132]: or one of the libraries it was linked against is corrupt, improperly built,
            Jun 27 12:44:11 vox-db1-read1 mysqld[12132]: or misconfigured. This error can also be caused by malfunctioning hardware.
            Jun 27 12:44:11 vox-db1-read1 mysqld[12132]: To report this bug, see https://mariadb.com/kb/en/reporting-bugs
            Jun 27 12:44:11 vox-db1-read1 mysqld[12132]: We will try our best to scrape up some info that will hopefully help
            Jun 27 12:44:11 vox-db1-read1 mysqld[12132]: diagnose the problem, but since we have already crashed,
            Jun 27 12:44:11 vox-db1-read1 mysqld[12132]: something is definitely wrong and this may fail.
            Jun 27 12:44:11 vox-db1-read1 mysqld[12132]: Server version: 10.4.12-MariaDB-1:10.4.12+maria~bionic-log
            Jun 27 12:44:11 vox-db1-read1 mysqld[12132]: key_buffer_size=134217728
            Jun 27 12:44:11 vox-db1-read1 mysqld[12132]: read_buffer_size=2097152
            Jun 27 12:44:11 vox-db1-read1 mysqld[12132]: max_used_connections=1
            ...and so on, it loops indefinitely.


            My theory is that something comes from Master that breaks Slaves.
            When first crash occured (same day I've setup replication, ~4-5 days ago) I tried to restore Slave from same backup I used to initilize it. And Slave could not be started. I suppose while catching up to Master it still encountered this breaking transaction and crashed. Then I just took a fresh backup from Master, and Slave started without problems.

            Since then every time Slaves crash I just take make backup (with mariabackup) and restore slaves. This is stupid I know, but what choice do we currently have? :)

            What additional information can I provide you with to find the root of this issue?

            Thanks!
            Grigoriy Grigoriy made changes -
            Attachment binlog.000879.zip [ 52437 ]
            Grigoriy Grigoriy made changes -
            Attachment binlog.000879.zip [ 52437 ]
            Grigoriy Grigoriy made changes -
            Attachment binlog.000879.zip [ 52438 ]
            alice Alice Sherepa made changes -
            Assignee Alice Sherepa [ alice ]
            alice Alice Sherepa made changes -
            Description Hello.
            We have a Master-to-multi-Slave config and are regularly facing slaves crashes (they crash together at the same time, approx. once every 24 hours). Maybe this has something to do with load, but crashed did not happen under heavy load and still happened today with relatively (50%) of highest load we've had so far.

            Master 10.4.12 setup is Ubuntu 18.0.4 cloud VPS (yandex cloud) with 6 cores 100% quota, 6 Gbs RAM, 30 Gbs space.
            Slaves 10.4.12 and 10.4.13 setup is Ubuntu 18.0.4 cloud VPS (yandex cloud) with 4 cores 100% quota, 4 Gbs RAM, 30 Gbs space.

            Replication is setup using this guide - https://mariadb.com/kb/en/setting-up-a-replication-slave-with-mariabackup/
            Everything works well, until slaves crash.

            Currently Web API sends DML statements to Master directly and SELECT statements are sent to HAProxy with wieght distribution 20/40/40 (Master/Slave1/Slave2).
            Before that we had Master 10.4.12 and only 1 Slave 10.4.12 with direct requests from API to Slave without HAProxy - Slave still crashed, I hoped with 2 slaves we would at least have 1 standing... but alas...

            We removed all triggers from the DB.
            We use GTIDs.
            We now use mixed binlog (but id did crash with 'row' mode).
            We don't have any DDL statements running (except a few CREATE TEMPORARY TABLE).
            We use InnoDB only to store data.
            We don't have any FullText Indexes.
            We don't use compression.
            We don't use SSL between Mater and Slaves.
            Timezone on all machines is UTC.
            Data size now approximately 800Mb, InnoDb buffer pool size is 1024Mb, InnoDb log file size is128Mb.

            Here's syslog:

            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: 200627 12:44:00 [ERROR] mysqld got signal 11 ;
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: This could be because you hit a bug. It is also possible that this binary
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: or one of the libraries it was linked against is corrupt, improperly built,
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: or misconfigured. This error can also be caused by malfunctioning hardware.
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: To report this bug, see https://mariadb.com/kb/en/reporting-bugs
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: We will try our best to scrape up some info that will hopefully help
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: diagnose the problem, but since we have already crashed,
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: something is definitely wrong and this may fail.
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: Server version: 10.4.12-MariaDB-1:10.4.12+maria~bionic-log
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: key_buffer_size=134217728
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: read_buffer_size=2097152
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: max_used_connections=15
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: max_threads=77
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: thread_count=24
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: It is possible that mysqld could use up to
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 606038 K bytes of memory
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: Hope that's ok; if not, decrease some variables in the equation.
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: Thread pointer: 0x7f40500014f8
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: Attempting backtrace. You can use the following information to find out
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: where mysqld died. If you see no messages after this, something went
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: terribly wrong...
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: stack_bottom = 0x7f40fc901e98 thread_stack 0x49000
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: /usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x55b8a063ac3e]
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: /usr/sbin/mysqld(handle_fatal_signal+0x515)[0x55b8a00aecf5]
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: /lib/x86_64-linux-gnu/libpthread.so.0(+0x12890)[0x7f411d583890]
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: /lib/x86_64-linux-gnu/libc.so.6(+0x18f039)[0x7f411c008039]
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: /usr/sbin/mysqld(+0xae4170)[0x55b8a031e170]
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: /usr/sbin/mysqld(+0xa205dd)[0x55b8a025a5dd]
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: /usr/sbin/mysqld(+0xb10a85)[0x55b8a034aa85]
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: /usr/sbin/mysqld(+0xb116be)[0x55b8a034b6be]
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: /usr/sbin/mysqld(+0xb1224c)[0x55b8a034c24c]
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: /usr/sbin/mysqld(+0xae7959)[0x55b8a0321959]
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: /usr/sbin/mysqld(+0xad0db6)[0x55b8a030adb6]
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: /usr/sbin/mysqld(+0xb0b658)[0x55b8a0345658]
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: /usr/sbin/mysqld(+0xb1173f)[0x55b8a034b73f]
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: /usr/sbin/mysqld(+0xb1224c)[0x55b8a034c24c]
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: /usr/sbin/mysqld(+0xae73a4)[0x55b8a03213a4]
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: /usr/sbin/mysqld(+0xa21f8f)[0x55b8a025bf8f]
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: /usr/sbin/mysqld(_ZN7handler13ha_delete_rowEPKh+0x9f)[0x55b8a00ba7af]
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: /usr/sbin/mysqld(_ZN21Delete_rows_log_event11do_exec_rowEP14rpl_group_info+0x12d)[0x55b8a01b12ad]
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: /usr/sbin/mysqld(_ZN14Rows_log_event14do_apply_eventEP14rpl_group_info+0x25d)[0x55b8a01a440d]
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: /usr/sbin/mysqld(+0x5b6e90)[0x55b89fdf0e90]
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: /usr/sbin/mysqld(handle_slave_sql+0x17b4)[0x55b89fdfa344]
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: /usr/sbin/mysqld(+0xdab79a)[0x55b8a05e579a]
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: /lib/x86_64-linux-gnu/libpthread.so.0(+0x76db)[0x7f411d5786db]
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: /lib/x86_64-linux-gnu/libc.so.6(clone+0x3f)[0x7f411bf9a88f]
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: Trying to get some variables.
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: Some pointers may be invalid and cause the dump to abort.
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: Query (0x0):
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: Connection ID (thread ID): 49
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: Status: NOT_KILLED
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=on,condition_pushdown_from_having=on
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: information that should help you find out what is causing the crash.
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: Writing a core file...
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: Working directory at /var/lib/mysql
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: Resource Limits:
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: Limit Soft Limit Hard Limit Units
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: Max cpu time unlimited unlimited seconds
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: Max file size unlimited unlimited bytes
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: Max data size unlimited unlimited bytes
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: Max stack size 8388608 unlimited bytes
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: Max core file size 0 unlimited bytes
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: Max resident set unlimited unlimited bytes
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: Max processes 15602 15602 processes
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: Max open files 16364 16364 files
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: Max locked memory 16777216 16777216 bytes
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: Max address space unlimited unlimited bytes
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: Max file locks unlimited unlimited locks
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: Max pending signals 15602 15602 signals
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: Max msgqueue size 819200 819200 bytes
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: Max nice priority 0 0
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: Max realtime priority 0 0
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: Max realtime timeout unlimited unlimited us
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: Core pattern: core
            Jun 27 12:44:00 vox-db1-read1 kernel: [264684.731198] mysqld[7861]: segfault at 115 ip 00007f411c008039 sp 00007f40fc8fcfc8 error 6 in libc-2.27.so[7f411be79000+1e7000]
            Jun 27 12:44:00 vox-db1-read1 systemd[1]: mariadb.service: Main process exited, code=killed, status=11/SEGV
            Jun 27 12:44:00 vox-db1-read1 systemd[1]: mariadb.service: Failed with result 'signal'.
            Jun 27 12:44:05 vox-db1-read1 systemd[1]: mariadb.service: Service hold-off time over, scheduling restart.
            Jun 27 12:44:05 vox-db1-read1 systemd[1]: mariadb.service: Scheduled restart job, restart counter is at 1.
            Jun 27 12:44:05 vox-db1-read1 systemd[1]: Stopped MariaDB 10.4.12 database server.
            Jun 27 12:44:05 vox-db1-read1 systemd[1]: Starting MariaDB 10.4.12 database server...
            Jun 27 12:44:06 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:06 0 [Note] /usr/sbin/mysqld (mysqld 10.4.12-MariaDB-1:10.4.12+maria~bionic-log) starting as process 12132 ...
            Jun 27 12:44:06 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:06 0 [Note] InnoDB: Using Linux native AIO
            Jun 27 12:44:06 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:06 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
            Jun 27 12:44:06 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:06 0 [Note] InnoDB: Uses event mutexes
            Jun 27 12:44:06 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:06 0 [Note] InnoDB: Compressed tables use zlib 1.2.11
            Jun 27 12:44:06 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:06 0 [Note] InnoDB: Number of pools: 1
            Jun 27 12:44:06 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:06 0 [Note] InnoDB: Using SSE2 crc32 instructions
            Jun 27 12:44:06 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:06 0 [Note] mysqld: O_TMPFILE is not supported on /tmp (disabling future attempts)
            Jun 27 12:44:06 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:06 0 [Note] InnoDB: Initializing buffer pool, total size = 1G, instances = 8, chunk size = 128M
            Jun 27 12:44:06 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:06 0 [Note] InnoDB: Completed initialization of buffer pool
            Jun 27 12:44:06 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:06 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority().
            Jun 27 12:44:07 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:07 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=142473402785
            Jun 27 12:44:10 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:10 0 [Note] InnoDB: 1 transaction(s) which must be rolled back or cleaned up in total 2 row operations to undo
            Jun 27 12:44:10 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:10 0 [Note] InnoDB: Trx id counter is 11134299
            Jun 27 12:44:10 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:10 0 [Note] InnoDB: Starting final batch to recover 63 pages from redo log.
            Jun 27 12:44:10 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:10 0 [Note] InnoDB: Last binlog file '/var/log/mysql/mariadb-bin.000634', position 59120658
            Jun 27 12:44:10 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:10 0 [Note] InnoDB: 128 out of 128 rollback segments are active.
            Jun 27 12:44:10 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:10 0 [Note] InnoDB: Starting in background the rollback of recovered transactions
            Jun 27 12:44:10 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:10 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1"
            Jun 27 12:44:10 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:10 0 [Note] InnoDB: Creating shared tablespace for temporary tables
            Jun 27 12:44:10 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:10 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ...
            Jun 27 12:44:10 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:10 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB.
            Jun 27 12:44:10 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:10 0 [Note] InnoDB: 10.4.12 started; log sequence number 142473413250; transaction id 11134301
            Jun 27 12:44:10 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:10 0 [Note] InnoDB: Loading buffer pool(s) from /var/lib/mysql/ib_buffer_pool
            Jun 27 12:44:10 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:10 0 [Note] Plugin 'FEEDBACK' is disabled.
            Jun 27 12:44:10 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:10 0 [Note] Recovering after a crash using /var/log/mysql/mariadb-bin
            Jun 27 12:44:10 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:10 0 [Note] Starting crash recovery...
            Jun 27 12:44:10 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:10 0 [Note] Crash recovery finished.
            Jun 27 12:44:10 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:10 0 [Note] InnoDB: Rolled back recovered transaction 11134298
            Jun 27 12:44:10 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:10 0 [Note] InnoDB: Rollback of non-prepared transactions completed
            Jun 27 12:44:10 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:10 0 [Note] Server socket created on IP: '0.0.0.0'.
            Jun 27 12:44:10 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:10 0 [Note] Reading of all Master_info entries succeeded
            Jun 27 12:44:10 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:10 0 [Note] Added new Master_info '' to hash table
            Jun 27 12:44:10 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:10 0 [Warning] Neither --relay-log nor --relay-log-index were used; so replication may break when this MySQL server acts as a slave and has his hostname changed!! Please use '--log-basename=#' or '--relay-log=mysqld-relay-bin' to avoid this problem.
            Jun 27 12:44:11 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:11 10 [Note] Slave I/O thread: Start asynchronous replication to master 'replication_user@10.130.0.25:3306' in log 'mariadb-bin.000712' at position 4
            Jun 27 12:44:11 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:11 11 [Note] Slave SQL thread initialized, starting replication in log 'mariadb-bin.000712' at position 4, relay log './mysqld-relay-bin.000001' position: 4; GTID position '0-1-5332814'
            Jun 27 12:44:11 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:11 0 [Note] /usr/sbin/mysqld: ready for connections.
            Jun 27 12:44:11 vox-db1-read1 mysqld[12132]: Version: '10.4.12-MariaDB-1:10.4.12+maria~bionic-log' socket: '/var/run/mysqld/mysqld.sock' port: 3306 mariadb.org binary distribution
            Jun 27 12:44:11 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:11 10 [Note] Slave I/O thread: connected to master 'replication_user@10.130.0.25:3306',replication starts at GTID position '0-1-5332814'
            Jun 27 12:44:11 vox-db1-read1 systemd[1]: Started MariaDB 10.4.12 database server.
            Jun 27 12:44:11 vox-db1-read1 /etc/mysql/debian-start[12175]: Upgrading MySQL tables if necessary.
            Jun 27 12:44:11 vox-db1-read1 /etc/mysql/debian-start[12178]: Looking for 'mysql' as: /usr/bin/mysql
            Jun 27 12:44:11 vox-db1-read1 /etc/mysql/debian-start[12178]: Looking for 'mysqlcheck' as: /usr/bin/mysqlcheck
            Jun 27 12:44:11 vox-db1-read1 /etc/mysql/debian-start[12178]: This installation of MariaDB is already upgraded to 10.4.12-MariaDB, use --force if you still need to run mysql_upgrade
            Jun 27 12:44:11 vox-db1-read1 /etc/mysql/debian-start[12186]: Checking for insecure root accounts.
            Jun 27 12:44:11 vox-db1-read1 /etc/mysql/debian-start[12190]: Triggering myisam-recover for all MyISAM tables and aria-recover for all Aria tables
            Jun 27 12:44:11 vox-db1-read1 mysqld[12132]: 200627 12:44:11 [ERROR] mysqld got signal 11 ;
            Jun 27 12:44:11 vox-db1-read1 mysqld[12132]: This could be because you hit a bug. It is also possible that this binary
            Jun 27 12:44:11 vox-db1-read1 mysqld[12132]: or one of the libraries it was linked against is corrupt, improperly built,
            Jun 27 12:44:11 vox-db1-read1 mysqld[12132]: or misconfigured. This error can also be caused by malfunctioning hardware.
            Jun 27 12:44:11 vox-db1-read1 mysqld[12132]: To report this bug, see https://mariadb.com/kb/en/reporting-bugs
            Jun 27 12:44:11 vox-db1-read1 mysqld[12132]: We will try our best to scrape up some info that will hopefully help
            Jun 27 12:44:11 vox-db1-read1 mysqld[12132]: diagnose the problem, but since we have already crashed,
            Jun 27 12:44:11 vox-db1-read1 mysqld[12132]: something is definitely wrong and this may fail.
            Jun 27 12:44:11 vox-db1-read1 mysqld[12132]: Server version: 10.4.12-MariaDB-1:10.4.12+maria~bionic-log
            Jun 27 12:44:11 vox-db1-read1 mysqld[12132]: key_buffer_size=134217728
            Jun 27 12:44:11 vox-db1-read1 mysqld[12132]: read_buffer_size=2097152
            Jun 27 12:44:11 vox-db1-read1 mysqld[12132]: max_used_connections=1
            ...and so on, it loops indefinitely.


            My theory is that something comes from Master that breaks Slaves.
            When first crash occured (same day I've setup replication, ~4-5 days ago) I tried to restore Slave from same backup I used to initilize it. And Slave could not be started. I suppose while catching up to Master it still encountered this breaking transaction and crashed. Then I just took a fresh backup from Master, and Slave started without problems.

            Since then every time Slaves crash I just take make backup (with mariabackup) and restore slaves. This is stupid I know, but what choice do we currently have? :)

            What additional information can I provide you with to find the root of this issue?

            Thanks!             		Hello.
            We have a Master-to-multi-Slave config and are regularly facing slaves crashes (they crash together at the same time, approx. once every 24 hours). Maybe this has something to do with load, but crashed did not happen under heavy load and still happened today with relatively (50%) of highest load we've had so far.

            Master 10.4.12 setup is Ubuntu 18.0.4 cloud VPS (yandex cloud) with 6 cores 100% quota, 6 Gbs RAM, 30 Gbs space.
            Slaves 10.4.12 and 10.4.13 setup is Ubuntu 18.0.4 cloud VPS (yandex cloud) with 4 cores 100% quota, 4 Gbs RAM, 30 Gbs space.

            Replication is setup using this guide - https://mariadb.com/kb/en/setting-up-a-replication-slave-with-mariabackup/
            Everything works well, until slaves crash.

            Currently Web API sends DML statements to Master directly and SELECT statements are sent to HAProxy with wieght distribution 20/40/40 (Master/Slave1/Slave2).
            Before that we had Master 10.4.12 and only 1 Slave 10.4.12 with direct requests from API to Slave without HAProxy - Slave still crashed, I hoped with 2 slaves we would at least have 1 standing... but alas...

            We removed all triggers from the DB.
            We use GTIDs.
            We now use mixed binlog (but id did crash with 'row' mode).
            We don't have any DDL statements running (except a few CREATE TEMPORARY TABLE).
            We use InnoDB only to store data.
            We don't have any FullText Indexes.
            We don't use compression.
            We don't use SSL between Mater and Slaves.
            Timezone on all machines is UTC.
            Data size now approximately 800Mb, InnoDb buffer pool size is 1024Mb, InnoDb log file size is128Mb.

            Here's syslog:


            {noformat}
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: 200627 12:44:00 [ERROR] mysqld got signal 11 ;
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: This could be because you hit a bug. It is also possible that this binary
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: or one of the libraries it was linked against is corrupt, improperly built,
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: or misconfigured. This error can also be caused by malfunctioning hardware.
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: To report this bug, see https://mariadb.com/kb/en/reporting-bugs
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: We will try our best to scrape up some info that will hopefully help
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: diagnose the problem, but since we have already crashed,
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: something is definitely wrong and this may fail.
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: Server version: 10.4.12-MariaDB-1:10.4.12+maria~bionic-log
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: key_buffer_size=134217728
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: read_buffer_size=2097152
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: max_used_connections=15
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: max_threads=77
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: thread_count=24
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: It is possible that mysqld could use up to
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 606038 K bytes of memory
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: Hope that's ok; if not, decrease some variables in the equation.
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: Thread pointer: 0x7f40500014f8
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: Attempting backtrace. You can use the following information to find out
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: where mysqld died. If you see no messages after this, something went
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: terribly wrong...
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: stack_bottom = 0x7f40fc901e98 thread_stack 0x49000
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: /usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x55b8a063ac3e]
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: /usr/sbin/mysqld(handle_fatal_signal+0x515)[0x55b8a00aecf5]
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: /lib/x86_64-linux-gnu/libpthread.so.0(+0x12890)[0x7f411d583890]
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: /lib/x86_64-linux-gnu/libc.so.6(+0x18f039)[0x7f411c008039]
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: /usr/sbin/mysqld(+0xae4170)[0x55b8a031e170]
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: /usr/sbin/mysqld(+0xa205dd)[0x55b8a025a5dd]
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: /usr/sbin/mysqld(+0xb10a85)[0x55b8a034aa85]
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: /usr/sbin/mysqld(+0xb116be)[0x55b8a034b6be]
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: /usr/sbin/mysqld(+0xb1224c)[0x55b8a034c24c]
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: /usr/sbin/mysqld(+0xae7959)[0x55b8a0321959]
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: /usr/sbin/mysqld(+0xad0db6)[0x55b8a030adb6]
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: /usr/sbin/mysqld(+0xb0b658)[0x55b8a0345658]
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: /usr/sbin/mysqld(+0xb1173f)[0x55b8a034b73f]
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: /usr/sbin/mysqld(+0xb1224c)[0x55b8a034c24c]
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: /usr/sbin/mysqld(+0xae73a4)[0x55b8a03213a4]
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: /usr/sbin/mysqld(+0xa21f8f)[0x55b8a025bf8f]
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: /usr/sbin/mysqld(_ZN7handler13ha_delete_rowEPKh+0x9f)[0x55b8a00ba7af]
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: /usr/sbin/mysqld(_ZN21Delete_rows_log_event11do_exec_rowEP14rpl_group_info+0x12d)[0x55b8a01b12ad]
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: /usr/sbin/mysqld(_ZN14Rows_log_event14do_apply_eventEP14rpl_group_info+0x25d)[0x55b8a01a440d]
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: /usr/sbin/mysqld(+0x5b6e90)[0x55b89fdf0e90]
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: /usr/sbin/mysqld(handle_slave_sql+0x17b4)[0x55b89fdfa344]
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: /usr/sbin/mysqld(+0xdab79a)[0x55b8a05e579a]
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: /lib/x86_64-linux-gnu/libpthread.so.0(+0x76db)[0x7f411d5786db]
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: /lib/x86_64-linux-gnu/libc.so.6(clone+0x3f)[0x7f411bf9a88f]
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: Trying to get some variables.
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: Some pointers may be invalid and cause the dump to abort.
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: Query (0x0):
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: Connection ID (thread ID): 49
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: Status: NOT_KILLED
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=on,condition_pushdown_from_having=on
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: information that should help you find out what is causing the crash.
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: Writing a core file...
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: Working directory at /var/lib/mysql
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: Resource Limits:
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: Limit Soft Limit Hard Limit Units
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: Max cpu time unlimited unlimited seconds
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: Max file size unlimited unlimited bytes
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: Max data size unlimited unlimited bytes
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: Max stack size 8388608 unlimited bytes
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: Max core file size 0 unlimited bytes
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: Max resident set unlimited unlimited bytes
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: Max processes 15602 15602 processes
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: Max open files 16364 16364 files
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: Max locked memory 16777216 16777216 bytes
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: Max address space unlimited unlimited bytes
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: Max file locks unlimited unlimited locks
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: Max pending signals 15602 15602 signals
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: Max msgqueue size 819200 819200 bytes
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: Max nice priority 0 0
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: Max realtime priority 0 0
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: Max realtime timeout unlimited unlimited us
            Jun 27 12:44:00 vox-db1-read1 mysqld[7728]: Core pattern: core
            Jun 27 12:44:00 vox-db1-read1 kernel: [264684.731198] mysqld[7861]: segfault at 115 ip 00007f411c008039 sp 00007f40fc8fcfc8 error 6 in libc-2.27.so[7f411be79000+1e7000]
            Jun 27 12:44:00 vox-db1-read1 systemd[1]: mariadb.service: Main process exited, code=killed, status=11/SEGV
            Jun 27 12:44:00 vox-db1-read1 systemd[1]: mariadb.service: Failed with result 'signal'.
            Jun 27 12:44:05 vox-db1-read1 systemd[1]: mariadb.service: Service hold-off time over, scheduling restart.
            Jun 27 12:44:05 vox-db1-read1 systemd[1]: mariadb.service: Scheduled restart job, restart counter is at 1.
            Jun 27 12:44:05 vox-db1-read1 systemd[1]: Stopped MariaDB 10.4.12 database server.
            Jun 27 12:44:05 vox-db1-read1 systemd[1]: Starting MariaDB 10.4.12 database server...
            Jun 27 12:44:06 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:06 0 [Note] /usr/sbin/mysqld (mysqld 10.4.12-MariaDB-1:10.4.12+maria~bionic-log) starting as process 12132 ...
            Jun 27 12:44:06 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:06 0 [Note] InnoDB: Using Linux native AIO
            Jun 27 12:44:06 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:06 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
            Jun 27 12:44:06 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:06 0 [Note] InnoDB: Uses event mutexes
            Jun 27 12:44:06 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:06 0 [Note] InnoDB: Compressed tables use zlib 1.2.11
            Jun 27 12:44:06 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:06 0 [Note] InnoDB: Number of pools: 1
            Jun 27 12:44:06 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:06 0 [Note] InnoDB: Using SSE2 crc32 instructions
            Jun 27 12:44:06 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:06 0 [Note] mysqld: O_TMPFILE is not supported on /tmp (disabling future attempts)
            Jun 27 12:44:06 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:06 0 [Note] InnoDB: Initializing buffer pool, total size = 1G, instances = 8, chunk size = 128M
            Jun 27 12:44:06 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:06 0 [Note] InnoDB: Completed initialization of buffer pool
            Jun 27 12:44:06 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:06 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority().
            Jun 27 12:44:07 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:07 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=142473402785
            Jun 27 12:44:10 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:10 0 [Note] InnoDB: 1 transaction(s) which must be rolled back or cleaned up in total 2 row operations to undo
            Jun 27 12:44:10 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:10 0 [Note] InnoDB: Trx id counter is 11134299
            Jun 27 12:44:10 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:10 0 [Note] InnoDB: Starting final batch to recover 63 pages from redo log.
            Jun 27 12:44:10 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:10 0 [Note] InnoDB: Last binlog file '/var/log/mysql/mariadb-bin.000634', position 59120658
            Jun 27 12:44:10 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:10 0 [Note] InnoDB: 128 out of 128 rollback segments are active.
            Jun 27 12:44:10 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:10 0 [Note] InnoDB: Starting in background the rollback of recovered transactions
            Jun 27 12:44:10 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:10 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1"
            Jun 27 12:44:10 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:10 0 [Note] InnoDB: Creating shared tablespace for temporary tables
            Jun 27 12:44:10 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:10 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ...
            Jun 27 12:44:10 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:10 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB.
            Jun 27 12:44:10 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:10 0 [Note] InnoDB: 10.4.12 started; log sequence number 142473413250; transaction id 11134301
            Jun 27 12:44:10 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:10 0 [Note] InnoDB: Loading buffer pool(s) from /var/lib/mysql/ib_buffer_pool
            Jun 27 12:44:10 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:10 0 [Note] Plugin 'FEEDBACK' is disabled.
            Jun 27 12:44:10 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:10 0 [Note] Recovering after a crash using /var/log/mysql/mariadb-bin
            Jun 27 12:44:10 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:10 0 [Note] Starting crash recovery...
            Jun 27 12:44:10 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:10 0 [Note] Crash recovery finished.
            Jun 27 12:44:10 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:10 0 [Note] InnoDB: Rolled back recovered transaction 11134298
            Jun 27 12:44:10 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:10 0 [Note] InnoDB: Rollback of non-prepared transactions completed
            Jun 27 12:44:10 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:10 0 [Note] Server socket created on IP: '0.0.0.0'.
            Jun 27 12:44:10 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:10 0 [Note] Reading of all Master_info entries succeeded
            Jun 27 12:44:10 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:10 0 [Note] Added new Master_info '' to hash table
            Jun 27 12:44:10 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:10 0 [Warning] Neither --relay-log nor --relay-log-index were used; so replication may break when this MySQL server acts as a slave and has his hostname changed!! Please use '--log-basename=#' or '--relay-log=mysqld-relay-bin' to avoid this problem.
            Jun 27 12:44:11 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:11 10 [Note] Slave I/O thread: Start asynchronous replication to master 'replication_user@10.130.0.25:3306' in log 'mariadb-bin.000712' at position 4
            Jun 27 12:44:11 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:11 11 [Note] Slave SQL thread initialized, starting replication in log 'mariadb-bin.000712' at position 4, relay log './mysqld-relay-bin.000001' position: 4; GTID position '0-1-5332814'
            Jun 27 12:44:11 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:11 0 [Note] /usr/sbin/mysqld: ready for connections.
            Jun 27 12:44:11 vox-db1-read1 mysqld[12132]: Version: '10.4.12-MariaDB-1:10.4.12+maria~bionic-log' socket: '/var/run/mysqld/mysqld.sock' port: 3306 mariadb.org binary distribution
            Jun 27 12:44:11 vox-db1-read1 mysqld[12132]: 2020-06-27 12:44:11 10 [Note] Slave I/O thread: connected to master 'replication_user@10.130.0.25:3306',replication starts at GTID position '0-1-5332814'
            Jun 27 12:44:11 vox-db1-read1 systemd[1]: Started MariaDB 10.4.12 database server.
            Jun 27 12:44:11 vox-db1-read1 /etc/mysql/debian-start[12175]: Upgrading MySQL tables if necessary.
            Jun 27 12:44:11 vox-db1-read1 /etc/mysql/debian-start[12178]: Looking for 'mysql' as: /usr/bin/mysql
            Jun 27 12:44:11 vox-db1-read1 /etc/mysql/debian-start[12178]: Looking for 'mysqlcheck' as: /usr/bin/mysqlcheck
            Jun 27 12:44:11 vox-db1-read1 /etc/mysql/debian-start[12178]: This installation of MariaDB is already upgraded to 10.4.12-MariaDB, use --force if you still need to run mysql_upgrade
            Jun 27 12:44:11 vox-db1-read1 /etc/mysql/debian-start[12186]: Checking for insecure root accounts.
            Jun 27 12:44:11 vox-db1-read1 /etc/mysql/debian-start[12190]: Triggering myisam-recover for all MyISAM tables and aria-recover for all Aria tables
            Jun 27 12:44:11 vox-db1-read1 mysqld[12132]: 200627 12:44:11 [ERROR] mysqld got signal 11 ;
            Jun 27 12:44:11 vox-db1-read1 mysqld[12132]: This could be because you hit a bug. It is also possible that this binary
            Jun 27 12:44:11 vox-db1-read1 mysqld[12132]: or one of the libraries it was linked against is corrupt, improperly built,
            Jun 27 12:44:11 vox-db1-read1 mysqld[12132]: or misconfigured. This error can also be caused by malfunctioning hardware.
            Jun 27 12:44:11 vox-db1-read1 mysqld[12132]: To report this bug, see https://mariadb.com/kb/en/reporting-bugs
            Jun 27 12:44:11 vox-db1-read1 mysqld[12132]: We will try our best to scrape up some info that will hopefully help
            Jun 27 12:44:11 vox-db1-read1 mysqld[12132]: diagnose the problem, but since we have already crashed,
            Jun 27 12:44:11 vox-db1-read1 mysqld[12132]: something is definitely wrong and this may fail.
            Jun 27 12:44:11 vox-db1-read1 mysqld[12132]: Server version: 10.4.12-MariaDB-1:10.4.12+maria~bionic-log
            Jun 27 12:44:11 vox-db1-read1 mysqld[12132]: key_buffer_size=134217728
            Jun 27 12:44:11 vox-db1-read1 mysqld[12132]: read_buffer_size=2097152
            Jun 27 12:44:11 vox-db1-read1 mysqld[12132]: max_used_connections=1
            {noformat}

            ...and so on, it loops indefinitely.


            My theory is that something comes from Master that breaks Slaves.
            When first crash occured (same day I've setup replication, ~4-5 days ago) I tried to restore Slave from same backup I used to initilize it. And Slave could not be started. I suppose while catching up to Master it still encountered this breaking transaction and crashed. Then I just took a fresh backup from Master, and Slave started without problems.

            Since then every time Slaves crash I just take make backup (with mariabackup) and restore slaves. This is stupid I know, but what choice do we currently have? :)

            What additional information can I provide you with to find the root of this issue?

            Thanks!
            alice Alice Sherepa made changes -
            Affects Version/s 10.2 [ 14601 ]
            Affects Version/s 10.3 [ 22126 ]
            Affects Version/s 10.4 [ 22408 ]
            Affects Version/s 10.5 [ 23123 ]
            alice Alice Sherepa made changes -
            Fix Version/s 10.2 [ 14601 ]
            Fix Version/s 10.3 [ 22126 ]
            Fix Version/s 10.4 [ 22408 ]
            Fix Version/s 10.5 [ 23123 ]
            alice Alice Sherepa made changes -
            Component/s Replication [ 10100 ]
            Component/s Virtual Columns [ 10803 ]
            alice Alice Sherepa made changes -
            Status Open [ 1 ] Confirmed [ 10101 ]
            alice Alice Sherepa made changes -
            Assignee Alice Sherepa [ alice ] Sachin Setiya [ sachin.setiya.007 ]
            alice Alice Sherepa made changes -
            alice Alice Sherepa made changes -
            alice Alice Sherepa made changes -
            Assignee Sachin Setiya [ sachin.setiya.007 ] Sujatha Sivakumar [ sujatha.sivakumar ]
            maxmether Max Mether made changes -
            Priority Major [ 3 ] Critical [ 2 ]
            alice Alice Sherepa made changes -
            manjot Manjot Singh (Inactive) made changes -
            sujatha.sivakumar Sujatha Sivakumar (Inactive) made changes -
            Status Confirmed [ 10101 ] In Progress [ 3 ]
            sujatha.sivakumar Sujatha Sivakumar (Inactive) made changes -
            Assignee Sujatha Sivakumar [ sujatha.sivakumar ] Andrei Elkin [ elkin ]
            Status In Progress [ 3 ] In Review [ 10002 ]
            Elkin Andrei Elkin made changes -
            Assignee Andrei Elkin [ elkin ] Sujatha Sivakumar [ sujatha.sivakumar ]
            Status In Review [ 10002 ] Stalled [ 10000 ]
            sujatha.sivakumar Sujatha Sivakumar (Inactive) made changes -
            issue.field.resolutiondate 2021-01-04 09:48:27.0 2021-01-04 09:48:27.239
            sujatha.sivakumar Sujatha Sivakumar (Inactive) made changes -
            Fix Version/s 10.2.37 [ 25112 ]
            Fix Version/s 10.3.28 [ 25111 ]
            Fix Version/s 10.4.18 [ 25110 ]
            Fix Version/s 10.5.9 [ 25109 ]
            Fix Version/s 10.2 [ 14601 ]
            Fix Version/s 10.3 [ 22126 ]
            Fix Version/s 10.4 [ 22408 ]
            Fix Version/s 10.5 [ 23123 ]
            Resolution Fixed [ 1 ]
            Status Stalled [ 10000 ] Closed [ 6 ]
            Elkin Andrei Elkin made changes -
            Elkin Andrei Elkin made changes -
            Elkin Andrei Elkin made changes -
            Elkin Andrei Elkin made changes -
            Labels Slave
            serg Sergei Golubchik made changes -
            Workflow MariaDB v3 [ 110582 ] MariaDB v4 [ 158031 ]
            mariadb-jira-automation Jira Automation (IT) made changes -
            Zendesk Related Tickets 202415

            People

              sujatha.sivakumar Sujatha Sivakumar (Inactive)
              Grigoriy Grigoriy
              Votes:
              1 Vote for this issue
              Watchers:
              9 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Git Integration

                  Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.