1 2025-04-14 10:44:02 234416 [Warning] IP address '10.141.83.154' could not be resolved: Name or service not known 2 2025-04-14 10:55:12 5 [Warning] Aborted connection 5 to db: 'unconnected' user: 'maxscale' host: '10.141.83.178' (Got an error reading communication packets) 3 2025-04-14 10:55:12 1599 [Warning] Aborted connection 1599 to db: 'pimcore' user: 'pimcore' host: '10.141.83.178' (Got an error reading communication packets) 4 2025-04-14 10:55:12 232657 [Warning] Aborted connection 232657 to db: 'pimcore' user: 'pimcore' host: '10.141.83.178' (Got an error reading communication packets) 5 2025-04-14 12:52:12 239938 [Warning] Sort aborted, host: 10.141.83.154, user: pimcore, thread: 239938, query: SELECT m.* FROM messenger_messages m WHERE (m.queue_name = 'kafka_send') AND (m.delivered_ at is null OR m.delivered_at < '2025-04-14 08:52:12') AND (m.available_at <= '2025-04-14 09:52:12') ORDER BY available_at ASC LIMIT 1 FOR UPDATE 6 2025-04-14 12:52:15 239930 [Warning] Sort aborted, host: 10.141.83.154, user: pimcore, thread: 239930, query: SELECT m.* FROM messenger_messages m WHERE (m.queue_name = 'pimcore_image_optimize') AND ( m.delivered_at is null OR m.delivered_at < '2025-04-14 08:52:15') AND (m.available_at <= '2025-04-14 09:52:15') ORDER BY available_at ASC LIMIT 1 FOR UPDATE 7 2025-04-14 12:52:15 0x7f58a4066700 InnoDB: Assertion failure in file /home/buildbot/buildbot/padding_for_CPACK_RPM_BUILD_SOURCE_DIRS_PREFIX/mariadb-10.5.18/storage/innobase/trx/trx0trx.cc line 1296 8 InnoDB: Failing assertion: UT_LIST_GET_LEN(lock.trx_locks) == 0 9 InnoDB: We intentionally generate a memory trap. 10 InnoDB: Submit a detailed bug report to https://jira.mariadb.org/ 11 InnoDB: If you get repeated assertion failures or crashes, even 12 InnoDB: immediately after the mysqld startup, there may be 13 InnoDB: corruption in the InnoDB tablespace. Please refer to 14 InnoDB: https://mariadb.com/kb/en/library/innodb-recovery-modes/ 15 InnoDB: about forcing recovery. 16 250414 12:52:15 [ERROR] mysqld got signal 6 ; 17 This could be because you hit a bug. It is also possible that this binary 18 or one of the libraries it was linked against is corrupt, improperly built, 19 or misconfigured. This error can also be caused by malfunctioning hardware. 20 21 To report this bug, see https://mariadb.com/kb/en/reporting-bugs 22 23 We will try our best to scrape up some info that will hopefully help 24 diagnose the problem, but since we have already crashed, 25 something is definitely wrong and this may fail. 26 27 Server version: 10.5.18-MariaDB-log 28 key_buffer_size=134217728 29 read_buffer_size=131072 30 max_used_connections=25 31 max_threads=153 32 thread_count=19 33 It is possible that mysqld could use up to 34 key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 467880 K bytes of memory 35 Hope that's ok; if not, decrease some variables in the equation. 36 37 Thread pointer: 0x7f574c3b01b8 38 Attempting backtrace. You can use the following information to find out 39 where mysqld died. If you see no messages after this, something went 40 terribly wrong... 41 stack_bottom = 0x7f58a4065c90 thread_stack 0x49000 42 Can't start addr2line 43 /usr/sbin/mariadbd(my_print_stacktrace+0x2e)[0x559197c927de] 44 /usr/sbin/mariadbd(handle_fatal_signal+0x307)[0x559197685b17] 45 /lib64/libpthread.so.0(+0xf5f0)[0x7f58be43a5f0] 46 /lib64/libc.so.6(gsignal+0x37)[0x7f58bd886337] 47 /lib64/libc.so.6(abort+0x148)[0x7f58bd887a28] 48 /usr/sbin/mariadbd(+0x66c135)[0x55919734e135] 49 /usr/sbin/mariadbd(+0xdf4650)[0x559197ad6650] 50 /usr/sbin/mariadbd(+0xdf47ab)[0x559197ad67ab] 51 /usr/sbin/mariadbd(+0xdf4a84)[0x559197ad6a84] 52 /usr/sbin/mariadbd(+0xc96de4)[0x559197978de4] 53 /usr/sbin/mariadbd(+0xc9ff99)[0x559197981f99] 54 /usr/sbin/mariadbd(_ZN7handler16ha_external_lockEP3THDi+0x261)[0x559197691f91] 55 /usr/sbin/mariadbd(+0xaa3e0a)[0x559197785e0a] 56 /usr/sbin/mariadbd(_Z19mysql_unlock_tablesP3THDP13st_mysql_lockb+0x14b)[0x55919778606b] 57 /usr/sbin/mariadbd(_ZN4JOIN15optimize_stage2Ev+0x293)[0x5591974d0853] 58 /usr/sbin/mariadbd(_ZN4JOIN14optimize_innerEv+0x1c8e)[0x5591974d45ce] 59 /usr/sbin/mariadbd(_ZN4JOIN8optimizeEv+0xb2)[0x5591974d62e2] 60 /usr/sbin/mariadbd(_Z12mysql_selectP3THDP10TABLE_LISTR4ListI4ItemEPS4_jP8st_orderS9_S7_S9_yP13select_resultP18st_select_lex_unitP13st_select_lex+0xb7)[0x5591974d63b7] 61 /usr/sbin/mariadbd(_Z13handle_selectP3THDP3LEXP13select_resultm+0x1e4)[0x5591974d7024] 62 /usr/sbin/mariadbd(+0x652aeb)[0x559197334aeb] 63 /usr/sbin/mariadbd(_Z21mysql_execute_commandP3THD+0x369a)[0x559197479a7a] 64 /usr/sbin/mariadbd(_Z11mysql_parseP3THDPcjP12Parser_statebb+0x245)[0x55919747be45] 65 /usr/sbin/mariadbd(_Z16dispatch_command19enum_server_commandP3THDPcjbb+0x1605)[0x55919747e435] 66 /usr/sbin/mariadbd(_Z10do_commandP3THD+0x107)[0x55919747fc27] 67 /usr/sbin/mariadbd(_Z24do_handle_one_connectionP7CONNECTb+0x1b2)[0x55919756fd92] 68 /usr/sbin/mariadbd(handle_one_connection+0x34)[0x559197570064] 69 /usr/sbin/mariadbd(+0xbf8857)[0x5591978da857] 70 /lib64/libpthread.so.0(+0x7e65)[0x7f58be432e65] 71 /lib64/libc.so.6(clone+0x6d)[0x7f58bd94e88d] 72 73 Trying to get some variables. 74 Some pointers may be invalid and cause the dump to abort. 75 Query (0x7f574c0457c0): SELECT o_type,o_className,o_classId FROM objects WHERE o_id = '1128779' 76 77 Connection ID (thread ID): 239930 78 Status: NOT_KILLED 79 80 Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on, derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_c ost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,exten ded_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=on,condition_pushdown_from_having=on,not_ null_range_scan=off 81 82 The manual page at https://mariadb.com/kb/en/how-to-produce-a-full-stack-trace-for-mysqld/ contains 83 information that should help you find out what is causing the crash. 84 Writing a core file... 85 Working directory at /u01/mysql 86 Resource Limits: 87 Limit Soft Limit Hard Limit Units 88 Max cpu time unlimited unlimited seconds 89 Max file size unlimited unlimited bytes 90 Max data size unlimited unlimited bytes 91 Max stack size 8388608 unlimited bytes 92 Max core file size 0 unlimited bytes 93 Max resident set unlimited unlimited bytes 94 Max processes 31194 31194 processes 95 Max open files 32768 32768 files 96 Max locked memory 65536 65536 bytes 97 Max address space unlimited unlimited bytes 98 Max file locks unlimited unlimited locks 99 Max pending signals 31194 31194 signals 100 Max msgqueue size 819200 819200 bytes 101 Max nice priority 0 0 102 Max realtime priority 0 0 103 Max realtime timeout unlimited unlimited us 104 Core pattern: core 105 106 Kernel version: Linux version 3.10.0-1062.9.1.el7.x86_64 (mockbuild@kbuilder.bsys.centos.org) (gcc version 4.8.5 20150623 (Red Hat 4.8.5-39) (GCC) ) #1 SMP Fri Dec 6 15:49:49 UTC 2019 107 108 2025-04-14 12:52:23 0 [Warning] The parameter innodb_file_format is deprecated and has no effect. It may be removed in future releases. See https://mariadb.com/kb/en/library/xtradbinnodb-file-format/ 109 2025-04-14 12:52:23 0 [Note] InnoDB: Uses event mutexes 110 2025-04-14 12:52:23 0 [Note] InnoDB: Compressed tables use zlib 1.2.7 111 2025-04-14 12:52:23 0 [Note] InnoDB: Number of pools: 1 112 2025-04-14 12:52:23 0 [Note] InnoDB: Using crc32 + pclmulqdq instructions 113 2025-04-14 12:52:23 0 [Note] InnoDB: Using Linux native AIO 114 2025-04-14 12:52:23 0 [Note] InnoDB: Initializing buffer pool, total size = 4294967296, chunk size = 134217728 115 2025-04-14 12:52:23 0 [Note] InnoDB: Completed initialization of buffer pool 116 2025-04-14 12:52:23 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=208071156636,208072389341 117 2025-04-14 12:52:23 0 [Note] InnoDB: Transaction 461404381 was in the XA prepared state. 118 2025-04-14 12:52:23 0 [Note] InnoDB: 1 transaction(s) which must be rolled back or cleaned up in total 0 row operations to undo 119 2025-04-14 12:52:23 0 [Note] InnoDB: Trx id counter is 461404382 120 2025-04-14 12:52:23 0 [Note] InnoDB: Starting final batch to recover 14201 pages from redo log. 121 2025-04-14 12:52:36 0 [Note] InnoDB: Last binlog file './mariadb-bin.000008', position 1053595311 122 2025-04-14 12:52:36 0 [Note] InnoDB: 128 rollback segments are active. 123 2025-04-14 12:52:36 0 [Note] InnoDB: Starting in background the rollback of recovered transactions 124 2025-04-14 12:52:36 0 [Note] InnoDB: Rollback of non-prepared transactions completed 125 2025-04-14 12:52:36 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1" 126 2025-04-14 12:52:36 0 [Note] InnoDB: Creating shared tablespace for temporary tables 127 2025-04-14 12:52:36 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ... 128 2025-04-14 12:52:36 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB. 129 2025-04-14 12:52:36 0 [Note] InnoDB: 10.5.18 started; log sequence number 208129439881; transaction id 461404383 130 2025-04-14 12:52:36 0 [Note] Plugin 'FEEDBACK' is disabled. 131 2025-04-14 12:52:36 0 [Note] InnoDB: Loading buffer pool(s) from /u01/mysql/ib_buffer_pool 132 2025-04-14 12:52:36 0 [Note] Recovering after a crash using mariadb-bin 133 2025-04-14 12:52:36 0 [Note] InnoDB: Buffer pool(s) load completed at 250414 12:52:36 134 2025-04-14 12:52:41 0 [Note] Starting crash recovery... 135 2025-04-14 12:52:41 0 [Note] InnoDB: Starting recovery for XA transactions... 136 2025-04-14 12:52:41 0 [Note] InnoDB: Transaction 461404381 in prepared state after recovery 137 2025-04-14 12:52:41 0 [Note] InnoDB: Transaction contains changes to 1 rows 138 2025-04-14 12:52:41 0 [Note] InnoDB: 1 transactions in prepared state after recovery 140 2025-04-14 12:52:41 0 [Note] Crash recovery finished. 141 2025-04-14 12:52:41 0 [Note] Server socket created on IP: '::'. 142 2025-04-14 12:52:41 0 [Note] Reading of all Master_info entries succeeded 143 2025-04-14 12:52:41 0 [Note] Added new Master_info '' to hash table 144 2025-04-14 12:52:41 0 [Note] /usr/sbin/mariadbd: ready for connections. 145 Version: '10.5.18-MariaDB-log' socket: '/u01/mysql/mysql.sock' port: 3306 MariaDB Server 146 2025-04-14 12:52:42 4 [Warning] IP address '10.140.104.148' could not be resolved: Name or service not known 147 2025-04-14 12:52:43 5 [Warning] IP address '10.141.83.154' could not be resolved: Name or service not known 148 2025-04-14 12:52:43 5 [Note] Master connection name: '' Master_info_file: 'master.info' Relay_info_file: 'relay-log.info' 149 2025-04-14 12:52:43 5 [Warning] Neither --relay-log nor --relay-log-index were used; so replication may break when this MariaDB server acts as a replica and has its hostname changed. Please use '--log -basename=#' or '--relay-log=pim-1-relay-bin' to avoid this problem. 150 2025-04-14 12:52:43 5 [Note] 'CHANGE MASTER TO executed'. Previous state master_host='', master_port='3306', master_log_file='', master_log_pos='4'. New state master_host='10.141.83.166', master_port= '3306', master_log_file='', master_log_pos='4'. 151 2025-04-14 12:52:43 5 [Note] Previous Using_Gtid=No. New Using_Gtid=Current_Pos 152 2025-04-14 12:52:43 6 [Note] Slave I/O thread: Start asynchronous replication to master 'maxscale@10.141.83.166:3306' in log '' at position 4 153 2025-04-14 12:52:43 7 [Note] Slave SQL thread initialized, starting replication in log 'FIRST' at position 4, relay log './pim-1-relay-bin.000001' position: 4; GTID position '1-1-11133943' 154 2025-04-14 12:52:43 6 [Note] Slave I/O thread: connected to master 'maxscale@10.141.83.166:3306',replication starts at GTID position '1-1-11133943' 155 2025-04-14 15:46:16 6 [ERROR] Error reading packet from server: Lost connection to server during query (server_errno=2013) 156 2025-04-14 15:46:16 6 [Note] Slave I/O thread: Failed reading log event, reconnecting to retry, log 'mariadb-bin.000013' at position 1029920407; GTID position '1-2-11352282' 157 2025-04-14 15:46:16 6 [ERROR] Slave I/O: error reconnecting to master 'maxscale@10.141.83.166:3306' - retry-time: 60 maximum-retries: 86400 message: Can't connect to server on '10.141.83.166' (111 " Connection refused"), Internal MariaDB error code: 2003 158 2025-04-14 15:46:24 7 [Note] Error reading relay log event: slave SQL thread was killed 159 2025-04-14 15:46:24 7 [Note] Slave SQL thread exiting, replication stopped in log 'mariadb-bin.000013' at position 1029920407; GTID position '1-2-11352282' 160 2025-04-14 15:46:24 7 [Note] master was 10.141.83.166:3306 161 2025-04-14 15:46:24 6 [Note] Slave I/O thread killed during or after a reconnect done to recover from failed read 162 2025-04-14 15:46:24 6 [Note] Slave I/O thread exiting, read up to log 'mariadb-bin.000013', position 1029920407; GTID position 1-2-11352282 163 2025-04-14 15:46:24 6 [Note] master was 10.141.83.166:3306 164 2025-04-14 15:46:24 5 [Note] Deleted Master_info file '/u01/mysql/master.info'. 165 2025-04-14 15:46:24 5 [Note] Deleted Master_info file '/u01/mysql/relay-log.info'. 166 2025-04-14 15:46:24 3767 [Warning] IP address '10.140.104.166' could not be resolved: Name or service not known 167 2025-04-14 15:46:25 3768 [Warning] IP address '10.140.104.167' could not be resolved: Name or service not known 168 2025-04-14 15:46:25 3767 [Note] Start binlog_dump to slave_server(3), pos(, 4), using_gtid(1), gtid('1-2-11352282') 169 2025-04-14 15:46:25 3768 [Note] Start binlog_dump to slave_server(4), pos(, 4), using_gtid(1), gtid('1-2-11352282') 170 2025-04-14 15:46:29 3774 [Note] Start binlog_dump to slave_server(2), pos(, 4), using_gtid(1), gtid('1-2-11352282') 171 2025-04-14 15:55:12 3773 [Warning] Sort aborted, host: 10.141.83.154, user: pimcore, thread: 3773, query: SELECT m.* FROM messenger_messages m WHERE (m.queue_name = 'pimcore_data_import') AND (m.deliv ered_at is null OR m.delivered_at < '2025-04-14 11:55:12') AND (m.available_at <= '2025-04-14 12:55:12') ORDER BY available_at ASC LIMIT 1 FOR UPDATE 172 2025-04-14 15:55:13 3773 [Warning] Sort aborted, host: 10.141.83.154, user: pimcore, thread: 3773, query: SELECT m.* FROM messenger_messages m WHERE (m.queue_name = 'pimcore_data_import') AND (m.deliv ered_at is null OR m.delivered_at < '2025-04-14 11:55:13') AND (m.available_at <= '2025-04-14 12:55:13') ORDER BY available_at ASC LIMIT 1 FOR UPDATE 173 2025-04-14 20:31:01 16495 [Warning] Sort aborted, host: 10.141.83.154, user: pimcore, thread: 16495, query: SELECT m.* FROM messenger_messages m WHERE (m.queue_name = 'pimcore_data_import') AND (m.del ivered_at is null OR m.delivered_at < '2025-04-14 16:31:01') AND (m.available_at <= '2025-04-14 17:31:01') ORDER BY available_at ASC LIMIT 1 FOR UPDATE 174 2025-04-15 2:30:02 37849 [Warning] IP address '10.141.83.57' could not be resolved: Name or service not known