================================================= 2021-04-21 17:34:00 7 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification. 2021-04-21 17:34:00 7 [Note] WSREP: Lowest cert index boundary for CC from group: 17982550 2021-04-21 17:34:00 7 [Note] WSREP: Min available from gcache for CC from group: 17953174 2021-04-21 17:34:01 0 [Note] WSREP: Member 2.0 (mysql-1) requested state transfer from '*any*'. Selected 1.0 (mysql-2)(SYNCED) as donor. 2021-04-21 17:34:01 0 [Note] WSREP: Shifting SYNCED -> DONOR/DESYNCED (TO: 17982594) 2021-04-21 17:34:01 7 [Note] WSREP: Detected STR version: 1, req_len: 173, req: STRv1 2021-04-21 17:34:01 7 [Note] WSREP: IST request: 6c9afcd7-96b7-11ea-96a5-76e81fcbb085:17982558-17982594|tcp://mysql-1.mysql.sde.svc.cluster.local:4568 2021-04-21 17:34:01 7 [Note] WSREP: Server status change synced -> donor 2021-04-21 17:34:01 7 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification. 2021-04-21 17:34:01 7 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification. 2021-04-21 17:34:01 0 [Note] WSREP: Donor monitor thread started to monitor 2021-04-21 17:34:01 0 [Note] WSREP: Running: 'wsrep_sst_rsync --role 'donor' --address 'mysql-1.mysql.sde.svc.cluster.local:4444/rsync_sst' --socket '/run/mysqld/mysqld.sock' --datadir '/var/lib/mysql/' --gtid '6c9afcd7-96b7-11ea-96a5-76e81fcbb085:17982558' --gtid-domain-id '0' --bypass --mysqld-args --wsrep_start_position=6c9afcd7-96b7-11ea-96a5-76e81fcbb085:17957162' 2021-04-21 17:34:01 7 [Note] WSREP: sst_donor_thread signaled with 0 2021-04-21 17:34:01 0 [Note] WSREP: async IST sender starting to serve tcp://mysql-1.mysql.sde.svc.cluster.local:4568 sending 17982550-17982594, preload starts from 17982550 2021-04-21 17:34:01 0 [Note] WSREP: IST sender 17982550 -> 17982594 WSREP_SST: [INFO] Bypassing state dump. (20210421 17:34:01.568) 2021-04-21 17:34:01 0 [Note] WSREP: SST sent: 6c9afcd7-96b7-11ea-96a5-76e81fcbb085:17982558 2021-04-21 17:34:01 0 [Note] WSREP: Server status change donor -> joined 2021-04-21 17:34:01 0 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification. 2021-04-21 17:34:01 0 [Note] WSREP: 1.0 (mysql-2): State transfer to 2.0 (mysql-1) complete. 2021-04-21 17:34:01 0 [Note] WSREP: Shifting DONOR/DESYNCED -> JOINED (TO: 17982594) 2021-04-21 17:34:01 0 [Note] WSREP: Member 1.0 (mysql-2) synced with group. 2021-04-21 17:34:01 0 [Note] WSREP: Shifting JOINED -> SYNCED (TO: 17982594) 2021-04-21 17:34:01 8 [Note] WSREP: Server mysql-2 synced with group 2021-04-21 17:34:01 8 [Note] WSREP: Server status change joined -> synced 2021-04-21 17:34:01 8 [Note] WSREP: Synchronized with group, ready for connections 2021-04-21 17:34:01 8 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification. 2021-04-21 17:34:01 0 [Note] WSREP: Donor monitor thread ended with total time 0 sec 2021-04-21 17:34:02 43035 [Warning] Aborted connection 43035 to db: 'storage' user: 'root' host: '10.255.5.160' (Got an error reading communication packets) 2021-04-21 17:34:03 0 [Note] WSREP: async IST sender served 2021-04-21 17:34:03 0 [Note] WSREP: 2.0 (mysql-1): State transfer from 1.0 (mysql-2) complete. 2021-04-21 17:34:03 0 [Note] WSREP: Member 2.0 (mysql-1) synced with group. 2021-04-21 17:34:03 0 [Note] WSREP: (8a0be701-a86e, 'tcp://0.0.0.0:4567') turning message relay requesting off 2021-04-21 17:35:02 43062 [Warning] Aborted connection 43062 to db: 'storage' user: 'root' host: '10.255.5.160' (Got an error reading communication packets) 2021-04-21 17:36:02 43119 [Warning] Aborted connection 43119 to db: 'storage' user: 'root' host: '10.255.5.160' (Got an error reading communication packets) 2021-04-21 17:39:01 43260 [Warning] Aborted connection 43260 to db: 'storage' user: 'root' host: '10.255.5.160' (Got an error reading communication packets) 2021-04-21 17:40:01 43554 [Warning] Aborted connection 43554 to db: 'storage' user: 'root' host: '10.255.5.160' (Got an error reading communication packets) 2021-04-21 17:42:01 43655 [Warning] Aborted connection 43655 to db: 'storage' user: 'root' host: '10.255.5.160' (Got an error reading communication packets) 2021-04-21 17:42:02 43656 [Warning] Aborted connection 43656 to db: 'storage' user: 'root' host: '10.255.5.160' (Got an error reading communication packets) 2021-04-21 17:43:02 43689 [Warning] Aborted connection 43689 to db: 'storage' user: 'root' host: '10.255.5.160' (Got an error reading communication packets) 2021-04-21 17:44:02 43728 [Warning] Aborted connection 43728 to db: 'storage' user: 'root' host: '10.255.5.160' (Got an error reading communication packets) 2021-04-21 17:45:02 43764 [Warning] Aborted connection 43764 to db: 'storage' user: 'root' host: '10.255.5.160' (Got an error reading communication packets) 2021-04-21 17:46:02 43788 [Warning] Aborted connection 43788 to db: 'storage' user: 'root' host: '10.255.5.160' (Got an error reading communication packets) 2021-04-21 17:47:01 43825 [Warning] Aborted connection 43825 to db: 'storage' user: 'root' host: '10.255.5.160' (Got an error reading communication packets) 2021-04-21 17:52:02 44320 [Warning] Aborted connection 44320 to db: 'storage' user: 'root' host: '10.255.5.160' (Got an error reading communication packets) 2021-04-21 17:53:02 44367 [Warning] Aborted connection 44367 to db: 'storage' user: 'root' host: '10.255.5.160' (Got an error reading communication packets) 2021-04-21 17:53:02 44368 [Warning] Aborted connection 44368 to db: 'storage' user: 'root' host: '10.255.5.160' (Got an error reading communication packets) 2021-04-21 17:54:02 44411 [Warning] Aborted connection 44411 to db: 'storage' user: 'root' host: '10.255.5.160' (Got an error reading communication packets) 2021-04-21 17:56:01 44505 [Warning] Aborted connection 44505 to db: 'storage' user: 'root' host: '10.255.5.160' (Got an error reading communication packets) 2021-04-21 17:57:01 44529 [Warning] Aborted connection 44529 to db: 'storage' user: 'root' host: '10.255.5.160' (Got an error reading communication packets) 2021-04-21 17:59:02 44608 [Warning] Aborted connection 44608 to db: 'storage' user: 'root' host: '10.255.5.160' (Got an error reading communication packets) 2021-04-21 18:00:02 44659 [Warning] Aborted connection 44659 to db: 'storage' user: 'root' host: '10.255.5.160' (Got an error reading communication packets) 2021-04-21 18:01:02 44723 [Warning] Aborted connection 44723 to db: 'storage' user: 'root' host: '10.255.5.160' (Got an error reading communication packets) 2021-04-21 18:02:02 44761 [Warning] Aborted connection 44761 to db: 'storage' user: 'root' host: '10.255.5.160' (Got an error reading communication packets) 2021-04-21 18:06:01 45184 [Warning] Aborted connection 45184 to db: 'storage' user: 'root' host: '10.255.5.160' (Got an error reading communication packets) 2021-04-21 18:07:01 45214 [Warning] Aborted connection 45214 to db: 'storage' user: 'root' host: '10.255.5.160' (Got an error reading communication packets) 2021-04-21 18:08:02 45238 [Warning] Aborted connection 45238 to db: 'storage' user: 'root' host: '10.255.5.160' (Got an error reading communication packets) 2021-04-21 18:11:02 45640 [Warning] Aborted connection 45640 to db: 'storage' user: 'root' host: '10.255.5.160' (Got an error reading communication packets) 2021-04-21 18:16:01 46199 [Warning] Aborted connection 46199 to db: 'storage' user: 'root' host: '10.255.5.160' (Got an error reading communication packets) 2021-04-21 18:17:01 46234 [Warning] Aborted connection 46234 to db: 'storage' user: 'root' host: '10.255.5.160' (Got an error reading communication packets) 2021-04-21 18:18:02 46255 [Warning] Aborted connection 46255 to db: 'storage' user: 'root' host: '10.255.5.160' (Got an error reading communication packets) 2021-04-21 18:19:02 46292 [Warning] Aborted connection 46292 to db: 'storage' user: 'root' host: '10.255.5.160' (Got an error reading communication packets) 2021-04-21 18:22:01 46423 [Warning] Aborted connection 46423 to db: 'storage' user: 'root' host: '10.255.5.160' (Got an error reading communication packets) 2021-04-21 18:23:01 46478 [Warning] Aborted connection 46478 to db: 'storage' user: 'root' host: '10.255.5.160' (Got an error reading communication packets) 2021-04-21 18:25:01 46541 [Warning] Aborted connection 46541 to db: 'storage' user: 'root' host: '10.255.5.160' (Got an error reading communication packets) 2021-04-21 18:26:02 46579 [Warning] Aborted connection 46579 to db: 'storage' user: 'root' host: '10.255.5.160' (Got an error reading communication packets) 2021-04-21 18:27:02 46648 [Warning] Aborted connection 46648 to db: 'storage' user: 'root' host: '10.255.5.160' (Got an error reading communication packets) 2021-04-21 18:28:02 46691 [Warning] Aborted connection 46691 to db: 'storage' user: 'root' host: '10.255.5.160' (Got an error reading communication packets) 2021-04-21 18:29:02 46736 [Warning] Aborted connection 46736 to db: 'storage' user: 'root' host: '10.255.5.160' (Got an error reading communication packets) 2021-04-21 18:30:02 46775 [Warning] Aborted connection 46775 to db: 'storage' user: 'root' host: '10.255.5.160' (Got an error reading communication packets) 2021-04-21 18:33:01 46963 [Warning] Aborted connection 46963 to db: 'storage' user: 'root' host: '10.255.5.160' (Got an error reading communication packets) 2021-04-21 18:35:02 47040 [Warning] Aborted connection 47040 to db: 'storage' user: 'root' host: '10.255.5.160' (Got an error reading communication packets) 2021-04-21 18:36:02 47082 [Warning] Aborted connection 47082 to db: 'storage' user: 'root' host: '10.255.5.160' (Got an error reading communication packets) 2021-04-21 18:37:02 47139 [Warning] Aborted connection 47139 to db: 'storage' user: 'root' host: '10.255.5.160' (Got an error reading communication packets) 2021-04-21 18:38:02 47175 [Warning] Aborted connection 47175 to db: 'storage' user: 'root' host: '10.255.5.160' (Got an error reading communication packets) 2021-04-21 18:39:01 47218 [Warning] Aborted connection 47218 to db: 'storage' user: 'root' host: '10.255.5.160' (Got an error reading communication packets) 2021-04-21 18:40:01 47276 [Warning] Aborted connection 47276 to db: 'storage' user: 'root' host: '10.255.5.160' (Got an error reading communication packets) 2021-04-21 18:41:01 47306 [Warning] Aborted connection 47306 to db: 'storage' user: 'root' host: '10.255.5.160' (Got an error reading communication packets) 2021-04-21 18:42:02 47329 [Warning] Aborted connection 47329 to db: 'storage' user: 'root' host: '10.255.5.160' (Got an error reading communication packets) 2021-04-21 18:44:02 47410 [Warning] Aborted connection 47410 to db: 'storage' user: 'root' host: '10.255.5.160' (Got an error reading communication packets) 2021-04-21 18:45:02 47686 [Warning] Aborted connection 47686 to db: 'storage' user: 'root' host: '10.255.5.160' (Got an error reading communication packets) 2021-04-21 18:46:02 47777 [Warning] Aborted connection 47777 to db: 'storage' user: 'root' host: '10.255.5.160' (Got an error reading communication packets) 2021-04-21 18:47:01 47828 [Warning] Aborted connection 47828 to db: 'storage' user: 'root' host: '10.255.5.160' (Got an error reading communication packets) 2021-04-21 18:51:02 48237 [Warning] Aborted connection 48237 to db: 'storage' user: 'root' host: '10.255.5.160' (Got an error reading communication packets) 2021-04-21 18:52:02 48261 [Warning] Aborted connection 48261 to db: 'storage' user: 'root' host: '10.255.5.160' (Got an error reading communication packets) 2021-04-21 18:55:02 48613 [Warning] Aborted connection 48613 to db: 'storage' user: 'root' host: '10.255.5.160' (Got an error reading communication packets) 2021-04-21 18:55:02 48614 [Warning] Aborted connection 48614 to db: 'storage' user: 'root' host: '10.255.5.160' (Got an error reading communication packets) 2021-04-21 18:57:01 48720 [Warning] Aborted connection 48720 to db: 'storage' user: 'root' host: '10.255.5.160' (Got an error reading communication packets) 2021-04-21 19:00:02 49107 [Warning] Aborted connection 49107 to db: 'storage' user: 'root' host: '10.255.5.160' (Got an error reading communication packets) 2021-04-21 19:01:02 49182 [Warning] Aborted connection 49182 to db: 'storage' user: 'root' host: '10.255.5.160' (Got an error reading communication packets) 2021-04-21 19:01:02 49183 [Warning] Aborted connection 49183 to db: 'storage' user: 'root' host: '10.255.5.160' (Got an error reading communication packets) 2021-04-21 19:02:02 49212 [Warning] Aborted connection 49212 to db: 'storage' user: 'root' host: '10.255.5.160' (Got an error reading communication packets) 2021-04-21 19:05:01 49344 [Warning] Aborted connection 49344 to db: 'storage' user: 'root' host: '10.255.5.160' (Got an error reading communication packets) 2021-04-21 19:07:02 49403 [Warning] Aborted connection 49403 to db: 'storage' user: 'root' host: '10.255.5.160' (Got an error reading communication packets) 2021-04-21 19:11:01 49807 [Warning] Aborted connection 49807 to db: 'storage' user: 'root' host: '10.255.5.160' (Got an error reading communication packets) 2021-04-21 19:14:01 49974 [Warning] Aborted connection 49974 to db: 'storage' user: 'root' host: '10.255.5.160' (Got an error reading communication packets) 2021-04-21 19:15:02 50009 [Warning] Aborted connection 50009 to db: 'storage' user: 'root' host: '10.255.5.160' (Got an error reading communication packets) 2021-04-21 19:16:02 50034 [Warning] Aborted connection 50034 to db: 'storage' user: 'root' host: '10.255.5.160' (Got an error reading communication packets) mysqld: /home/buildbot/buildbot/build/mariadb-10.5.9/wsrep-lib/include/wsrep/client_state.hpp:320: int wsrep::client_state::start_transaction(const wsrep::transaction_id&): Assertion `state_ == s_exec' failed. 210421 19:16:29 [ERROR] mysqld got signal 6 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. To report this bug, see https://mariadb.com/kb/en/reporting-bugs We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. Server version: 10.5.9-MariaDB-1:10.5.9+maria~focal-log key_buffer_size=629145600 read_buffer_size=131072 max_used_connections=32 max_threads=252 thread_count=41 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 1169132 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. Thread pointer: 0x0 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0x0 thread_stack 0x49000 mysqld(my_print_stacktrace+0x32)[0x55a839818af2] Printing to addr2line failed mysqld(handle_fatal_signal+0x485)[0x55a83926e6f5] /lib/x86_64-linux-gnu/libpthread.so.0(+0x153c0)[0x7ff0435a03c0] /lib/x86_64-linux-gnu/libc.so.6(gsignal+0xcb)[0x7ff0430a718b] /lib/x86_64-linux-gnu/libc.so.6(abort+0x12b)[0x7ff043086859] /lib/x86_64-linux-gnu/libc.so.6(+0x25729)[0x7ff043086729] /lib/x86_64-linux-gnu/libc.so.6(+0x36f36)[0x7ff043097f36] mysqld(_Z14wsrep_bf_abortPK3THDPS_+0x563)[0x55a839543b33] mysqld(wsrep_thd_bf_abort+0x23)[0x55a83954b713] mysqld(+0xc70e3c)[0x55a839573e3c] mysqld(handle_manager+0x15e)[0x55a839050e9e] mysqld(+0xbc1c66)[0x55a8394c4c66] /lib/x86_64-linux-gnu/libpthread.so.0(+0x9609)[0x7ff043594609] /lib/x86_64-linux-gnu/libc.so.6(clone+0x43)[0x7ff043183293] The manual page at https://mariadb.com/kb/en/how-to-produce-a-full-stack-trace-for-mysqld/ contains information that should help you find out what is causing the crash. Writing a core file... Working directory at /var/lib/mysql Resource Limits: Limit Soft Limit Hard Limit Units Max cpu time unlimited unlimited seconds Max file size unlimited unlimited bytes Max data size unlimited unlimited bytes Max stack size 8388608 unlimited bytes Max core file size unlimited unlimited bytes Max resident set unlimited unlimited bytes Max processes unlimited unlimited processes Max open files 1048576 1048576 files Max locked memory 16777216 16777216 bytes Max address space unlimited unlimited bytes Max file locks unlimited unlimited locks Max pending signals 59986 59986 signals Max msgqueue size 819200 819200 bytes Max nice priority 0 0 Max realtime priority 0 0 Max realtime timeout unlimited unlimited us Core pattern: core.%e.%p.%t Fatal signal 11 while backtracing