Sep 8 03:00:16 breww-db-03 mariadbd[3091530]: 2022-09-08 3:00:16 0 [Note] WSREP: Created page /var/lib/mysql/galera.page.000003 of size 134217696 bytes
|
Sep 8 04:11:38 breww-db-03 mariadbd[3091530]: 2022-09-08 4:11:38 1260551 [Warning] Aborted connection 1260551 to db: 'REDACTED' user: 'REDACTED' host: 'REDACTED' (Got an error reading communication packets)
|
Sep 8 05:12:10 breww-db-03 mariadbd[3091530]: 2022-09-08 5:12:10 0 [Note] WSREP: Deleted page /var/lib/mysql/galera.page.000003
|
Sep 8 07:08:11 breww-db-03 mariadbd[3091530]: mariadbd: /home/buildbot/buildbot/build/mariadb-10.8.4/wsrep-lib/include/wsrep/client_state.hpp:668: int wsrep::client_state::bf_abort(wsrep::seqno): Assertion `mode_ == m_local || transaction_.is_streaming()' failed.
|
Sep 8 07:08:11 breww-db-03 mariadbd[3091530]: 220908 7:08:11 [ERROR] mysqld got signal 6 ;
|
Sep 8 07:08:11 breww-db-03 mariadbd[3091530]: This could be because you hit a bug. It is also possible that this binary
|
Sep 8 07:08:11 breww-db-03 mariadbd[3091530]: or one of the libraries it was linked against is corrupt, improperly built,
|
Sep 8 07:08:11 breww-db-03 mariadbd[3091530]: or misconfigured. This error can also be caused by malfunctioning hardware.
|
Sep 8 07:08:11 breww-db-03 mariadbd[3091530]: To report this bug, see https://mariadb.com/kb/en/reporting-bugs
|
Sep 8 07:08:11 breww-db-03 mariadbd[3091530]: We will try our best to scrape up some info that will hopefully help
|
Sep 8 07:08:11 breww-db-03 mariadbd[3091530]: diagnose the problem, but since we have already crashed,
|
Sep 8 07:08:11 breww-db-03 mariadbd[3091530]: something is definitely wrong and this may fail.
|
Sep 8 07:08:11 breww-db-03 mariadbd[3091530]: Server version: 10.8.4-MariaDB-1:10.8.4+maria~ubu2004
|
Sep 8 07:08:11 breww-db-03 mariadbd[3091530]: key_buffer_size=134217728
|
Sep 8 07:08:11 breww-db-03 mariadbd[3091530]: read_buffer_size=131072
|
Sep 8 07:08:11 breww-db-03 mariadbd[3091530]: max_used_connections=25
|
Sep 8 07:08:11 breww-db-03 mariadbd[3091530]: max_threads=1002
|
Sep 8 07:08:11 breww-db-03 mariadbd[3091530]: thread_count=43
|
Sep 8 07:08:11 breww-db-03 mariadbd[3091530]: It is possible that mysqld could use up to
|
Sep 8 07:08:11 breww-db-03 mariadbd[3091530]: key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 2337747 K bytes of memory
|
Sep 8 07:08:11 breww-db-03 mariadbd[3091530]: Hope that's ok; if not, decrease some variables in the equation.
|
Sep 8 07:08:11 breww-db-03 mariadbd[3091530]: Thread pointer: 0x7f84a82389e8
|
Sep 8 07:08:11 breww-db-03 mariadbd[3091530]: Attempting backtrace. You can use the following information to find out
|
Sep 8 07:08:11 breww-db-03 mariadbd[3091530]: where mysqld died. If you see no messages after this, something went
|
Sep 8 07:08:11 breww-db-03 mariadbd[3091530]: terribly wrong...
|
Sep 8 07:08:11 breww-db-03 mariadbd[3091530]: stack_bottom = 0x7f851c10cd98 thread_stack 0x49000
|
Sep 8 07:08:12 breww-db-03 mariadbd[3091530]: ??:0(my_print_stacktrace)[0x55c972c96ae2]
|
Sep 8 07:08:12 breww-db-03 mariadbd[3091530]: ??:0(handle_fatal_signal)[0x55c97276d9e5]
|
Sep 8 07:08:12 breww-db-03 mariadbd[3091530]: ??:0(__restore_rt)[0x7f87ec247420]
|
Sep 8 07:08:12 breww-db-03 mariadbd[3091530]: ??:0(gsignal)[0x7f87ebd4b00b]
|
Sep 8 07:08:12 breww-db-03 mariadbd[3091530]: ??:0(abort)[0x7f87ebd2a859]
|
Sep 8 07:08:12 breww-db-03 mariadbd[3091530]: /lib/x86_64-linux-gnu/libc.so.6(+0x22729)[0x7f87ebd2a729]
|
Sep 8 07:08:12 breww-db-03 mariadbd[3091530]: ??:0(__assert_fail)[0x7f87ebd3bfd6]
|
Sep 8 07:08:13 breww-db-03 mariadbd[3091530]: ??:0(wsrep_bf_abort(THD*, THD*))[0x55c972a1c9b7]
|
Sep 8 07:08:13 breww-db-03 mariadbd[3091530]: ??:0(wsrep_thd_bf_abort)[0x55c972a2416d]
|
Sep 8 07:08:13 breww-db-03 mariadbd[3091530]: ??:0(wsrep_notify_status(wsrep::server_state::state, wsrep::view const*))[0x55c972a44422]
|
Sep 8 07:08:13 breww-db-03 mariadbd[3091530]: ??:0(Wsrep_server_service::log_dummy_write_set(wsrep::client_state&, wsrep::ws_meta const&))[0x55c972390f36]
|
Sep 8 07:08:13 breww-db-03 mariadbd[3091530]: ??:0(Wsrep_server_service::log_dummy_write_set(wsrep::client_state&, wsrep::ws_meta const&))[0x55c97239195d]
|
Sep 8 07:08:13 breww-db-03 mariadbd[3091530]: ??:0(void std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::_M_construct<char*>(char*, char*, std::forward_iterator_tag))[0x55c972aeab1b]
|
Sep 8 07:08:13 breww-db-03 mariadbd[3091530]: ??:0(void std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::_M_construct<char*>(char*, char*, std::forward_iterator_tag))[0x55c972aebf57]
|
Sep 8 07:08:13 breww-db-03 mariadbd[3091530]: ??:0(void std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::_M_construct<char*>(char*, char*, std::forward_iterator_tag))[0x55c972aec620]
|
Sep 8 07:08:13 breww-db-03 mariadbd[3091530]: ??:0(void std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::_M_construct<char*>(char*, char*, std::forward_iterator_tag))[0x55c972aeca6b]
|
Sep 8 07:08:13 breww-db-03 mariadbd[3091530]: ??:0(void std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::_M_construct<char*>(char*, char*, std::forward_iterator_tag))[0x55c972afeb75]
|
Sep 8 07:08:13 breww-db-03 mariadbd[3091530]: ??:0(wsrep_notify_status(wsrep::server_state::state, wsrep::view const*))[0x55c972a4d6da]
|
Sep 8 07:08:13 breww-db-03 mariadbd[3091530]: ??:0(handler::ha_write_row(unsigned char const*))[0x55c97277cda0]
|
Sep 8 07:08:13 breww-db-03 mariadbd[3091530]: ??:0(Rows_log_event::write_row(rpl_group_info*, bool))[0x55c9728b0174]
|
Sep 8 07:08:13 breww-db-03 mariadbd[3091530]: ??:0(Write_rows_log_event::do_exec_row(rpl_group_info*))[0x55c9728b07f1]
|
Sep 8 07:08:13 breww-db-03 mariadbd[3091530]: ??:0(Rows_log_event::do_apply_event(rpl_group_info*))[0x55c9728a3d8f]
|
Sep 8 07:08:13 breww-db-03 mariadbd[3091530]: ??:0(wsrep_apply_events(THD*, Relay_log_info*, void const*, unsigned long))[0x55c972a1b819]
|
Sep 8 07:08:13 breww-db-03 mariadbd[3091530]: ??:0(Wsrep_replayer_service::apply_write_set(wsrep::ws_meta const&, wsrep::const_buffer const&, wsrep::mutable_buffer&))[0x55c972a010ec]
|
Sep 8 07:08:13 breww-db-03 mariadbd[3091530]: ??:0(wsrep::server_state::start_streaming_applier(wsrep::id const&, wsrep::transaction_id const&, wsrep::high_priority_service*))[0x55c972d2c1d0]
|
Sep 8 07:08:13 breww-db-03 mariadbd[3091530]: ??:0(wsrep::server_state::on_apply(wsrep::high_priority_service&, wsrep::ws_handle const&, wsrep::ws_meta const&, wsrep::const_buffer const&))[0x55c972d2d0a1]
|
Sep 8 07:08:13 breww-db-03 mariadbd[3091530]: ??:0(wsrep::wsrep_provider_v26::options[abi:cxx11]() const)[0x55c972d3e5cc]
|
Sep 8 07:08:13 breww-db-03 mariadbd[3091530]: src/trx_handle.cpp:396(galera::TrxHandleSlave::apply(void*, wsrep_cb_status (*)(void*, wsrep_ws_handle const*, unsigned int, wsrep_buf const*, wsrep_trx_meta const*, bool*), wsrep_trx_meta const&, bool&))[0x7f87eb86d9eb]
|
Sep 8 07:08:13 breww-db-03 mariadbd[3091530]: src/trx_handle.hpp:845(galera::TrxHandleMaster::lock())[0x7f87eb884e8b]
|
Sep 8 07:08:13 breww-db-03 mariadbd[3091530]: src/trx_handle.hpp:1153(galera::TrxHandleLock::~TrxHandleLock())[0x7f87eb859c73]
|
Sep 8 07:08:14 breww-db-03 mariadbd[3091530]: ??:0(wsrep::wsrep_provider_v26::replay(wsrep::ws_handle const&, wsrep::high_priority_service*))[0x55c972d3eecd]
|
Sep 8 07:08:14 breww-db-03 mariadbd[3091530]: ??:0(Wsrep_client_service::replay())[0x55c972a00c92]
|
Sep 8 07:08:14 breww-db-03 mariadbd[3091530]: ??:0(wsrep::transaction::replay(std::unique_lock<wsrep::mutex>&))[0x55c972d37cfa]
|
Sep 8 07:08:14 breww-db-03 mariadbd[3091530]: ??:0(wsrep::transaction::after_statement())[0x55c972d3afe6]
|
Sep 8 07:08:14 breww-db-03 mariadbd[3091530]: ??:0(wsrep::client_state::after_statement())[0x55c972d220a2]
|
Sep 8 07:08:14 breww-db-03 mariadbd[3091530]: ??:0(mysql_init_multi_delete(LEX*))[0x55c972515b38]
|
Sep 8 07:08:14 breww-db-03 mariadbd[3091530]: ??:0(dispatch_command(enum_server_command, THD*, char*, unsigned int, bool))[0x55c972523b1a]
|
Sep 8 07:08:14 breww-db-03 mariadbd[3091530]: ??:0(do_command(THD*, bool))[0x55c972524638]
|
Sep 8 07:08:14 breww-db-03 mariadbd[3091530]: ??:0(do_handle_one_connection(CONNECT*, bool))[0x55c9726461b7]
|
Sep 8 07:08:14 breww-db-03 mariadbd[3091530]: ??:0(handle_one_connection)[0x55c97264650d]
|
Sep 8 07:08:14 breww-db-03 mariadbd[3091530]: ??:0(MyCTX_nopad::finish(unsigned char*, unsigned int*))[0x55c972996226]
|
Sep 8 07:08:14 breww-db-03 mariadbd[3091530]: ??:0(start_thread)[0x7f87ec23b609]
|
Sep 8 07:08:15 breww-db-03 mariadbd[3091530]: ??:0(clone)[0x7f87ebe27133]
|
Sep 8 07:08:15 breww-db-03 mariadbd[3091530]: Trying to get some variables.
|
Sep 8 07:08:15 breww-db-03 mariadbd[3091530]: Some pointers may be invalid and cause the dump to abort.
|
Sep 8 07:08:15 breww-db-03 mariadbd[3091530]: Query (0x7f87e1b6b1b3): INSERT INTO `app_integrationwebhookqueuedtask` (`integration_authorisation_id`, `request_data`, `received_at`, `failed`, `failed_at`, `failed_message`) VALUES (1320, 'REDACTED_DATA_FOR_BUG_REPORT...
|
Sep 8 07:08:15 breww-db-03 mariadbd[3091530]: Connection ID (thread ID): 1
|
Sep 8 07:08:15 breww-db-03 mariadbd[3091530]: Status: NOT_KILLED
|
Sep 8 07:08:15 breww-db-03 mariadbd[3091530]: Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=off,condition_pushdown_for_subquery=on,rowid_filter=on,condition_pushdown_from_having=on,not_null_range_scan=off
|
Sep 8 07:08:15 breww-db-03 mariadbd[3091530]: The manual page at https://mariadb.com/kb/en/how-to-produce-a-full-stack-trace-for-mysqld/ contains
|
Sep 8 07:08:15 breww-db-03 mariadbd[3091530]: information that should help you find out what is causing the crash.
|
Sep 8 07:08:15 breww-db-03 mariadbd[3091530]: Writing a core file...
|
Sep 8 07:08:15 breww-db-03 mariadbd[3091530]: Working directory at /var/lib/mysql
|
Sep 8 07:08:15 breww-db-03 mariadbd[3091530]: Resource Limits:
|
Sep 8 07:08:15 breww-db-03 mariadbd[3091530]: Limit Soft Limit Hard Limit Units
|
Sep 8 07:08:15 breww-db-03 mariadbd[3091530]: Max cpu time unlimited unlimited seconds
|
Sep 8 07:08:15 breww-db-03 mariadbd[3091530]: Max file size unlimited unlimited bytes
|
Sep 8 07:08:15 breww-db-03 mariadbd[3091530]: Max data size unlimited unlimited bytes
|
Sep 8 07:08:15 breww-db-03 mariadbd[3091530]: Max stack size 8388608 unlimited bytes
|
Sep 8 07:08:15 breww-db-03 mariadbd[3091530]: Max core file size 0 unlimited bytes
|
Sep 8 07:08:15 breww-db-03 mariadbd[3091530]: Max resident set unlimited unlimited bytes
|
Sep 8 07:08:15 breww-db-03 mariadbd[3091530]: Max processes 63679 63679 processes
|
Sep 8 07:08:15 breww-db-03 mariadbd[3091530]: Max open files 32768 32768 files
|
Sep 8 07:08:15 breww-db-03 mariadbd[3091530]: Max locked memory 65536 65536 bytes
|
Sep 8 07:08:15 breww-db-03 mariadbd[3091530]: Max address space unlimited unlimited bytes
|
Sep 8 07:08:15 breww-db-03 mariadbd[3091530]: Max file locks unlimited unlimited locks
|
Sep 8 07:08:15 breww-db-03 mariadbd[3091530]: Max pending signals 63679 63679 signals
|
Sep 8 07:08:15 breww-db-03 mariadbd[3091530]: Max msgqueue size 819200 819200 bytes
|
Sep 8 07:08:15 breww-db-03 mariadbd[3091530]: Max nice priority 0 0
|
Sep 8 07:08:15 breww-db-03 mariadbd[3091530]: Max realtime priority 0 0
|
Sep 8 07:08:15 breww-db-03 mariadbd[3091530]: Max realtime timeout unlimited unlimited us
|
Sep 8 07:08:15 breww-db-03 mariadbd[3091530]: Core pattern: |/usr/share/apport/apport -p%p -s%s -c%c -d%d -P%P -u%u -g%g -- %E
|
Sep 8 07:08:15 breww-db-03 mariadbd[3091530]: Kernel version: Linux version 5.4.0-122-generic (buildd@lcy02-amd64-095) (gcc version 9.4.0 (Ubuntu 9.4.0-1ubuntu1~20.04.1)) #138-Ubuntu SMP Wed Jun 22 15:00:31 UTC 2022
|
Sep 8 07:09:05 breww-db-03 systemd[1]: mariadb.service: Main process exited, code=killed, status=6/ABRT
|
Sep 8 07:09:05 breww-db-03 systemd[1]: mariadb.service: Failed with result 'signal'.
|
Sep 8 07:09:10 breww-db-03 systemd[1]: mariadb.service: Scheduled restart job, restart counter is at 1.
|
Sep 8 07:09:10 breww-db-03 systemd[1]: Stopped MariaDB 10.8.4 database server.
|
Sep 8 07:09:10 breww-db-03 systemd[1]: Starting MariaDB 10.8.4 database server...
|
Sep 8 07:09:30 breww-db-03 sh[3691097]: WSREP: Failed to start mysqld for wsrep recovery: '2022-09-08 7:09:11 0 [Note] /usr/sbin/mysqld (server 10.8.4-MariaDB-1:10.8.4+maria~ubu2004) starting as process 3691198 ...
|
Sep 8 07:09:30 breww-db-03 sh[3691097]: 2022-09-08 7:09:11 0 [Note] InnoDB: Compressed tables use zlib 1.2.11
|
Sep 8 07:09:30 breww-db-03 sh[3691097]: 2022-09-08 7:09:11 0 [Note] InnoDB: Number of transaction pools: 1
|
Sep 8 07:09:30 breww-db-03 sh[3691097]: 2022-09-08 7:09:11 0 [Note] InnoDB: Using crc32 + pclmulqdq instructions
|
Sep 8 07:09:30 breww-db-03 sh[3691097]: 2022-09-08 7:09:11 0 [Note] InnoDB: Using Linux native AIO
|
Sep 8 07:09:30 breww-db-03 sh[3691097]: 2022-09-08 7:09:11 0 [Note] InnoDB: Initializing buffer pool, total size = 10.000GiB, chunk size = 160.000MiB
|
Sep 8 07:09:30 breww-db-03 sh[3691097]: 2022-09-08 7:09:11 0 [Note] InnoDB: Completed initialization of buffer pool
|
Sep 8 07:09:30 breww-db-03 sh[3691097]: 2022-09-08 7:09:11 0 [Note] InnoDB: File system buffers for log disabled (block size=512 bytes)
|
Sep 8 07:09:30 breww-db-03 sh[3691097]: 2022-09-08 7:09:11 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=160412372948
|
Sep 8 07:09:30 breww-db-03 sh[3691097]: 2022-09-08 7:09:26 0 [Note] InnoDB: Read redo log up to LSN=161394290156
|
Sep 8 07:09:30 breww-db-03 sh[3691097]: 2022-09-08 7:09:27 0 [Note] InnoDB: 2 transaction(s) which must be rolled back or cleaned up in total 2 row operations to undo
|
Sep 8 07:09:30 breww-db-03 sh[3691097]: 2022-09-08 7:09:27 0 [Note] InnoDB: Trx id counter is 205166915
|
Sep 8 07:09:30 breww-db-03 sh[3691097]: 2022-09-08 7:09:27 0 [Note] InnoDB: Starting final batch to recover 63322 pages from redo log.
|
Sep 8 07:09:30 breww-db-03 sh[3691097]: 2022-09-08 7:09:27 0 [ERROR] InnoDB: Not applying INSERT_HEAP_DYNAMIC due to corruption on [page id: space=2399, page number=8538]
|
Sep 8 07:09:30 breww-db-03 sh[3691097]: 2022-09-08 7:09:27 0 [ERROR] InnoDB: Set innodb_force_recovery=1 to ignore corruption.
|
Sep 8 07:09:30 breww-db-03 sh[3691097]: 2022-09-08 7:09:27 0 [ERROR] InnoDB: Not applying INSERT_HEAP_DYNAMIC due to corruption on [page id: space=2399, page number=8538]
|
Sep 8 07:09:30 breww-db-03 sh[3691097]: 2022-09-08 7:09:27 0 [ERROR] InnoDB: Set innodb_force_recovery=1 to ignore corruption.
|
Sep 8 07:09:30 breww-db-03 sh[3691097]: 2022-09-08 7:09:27 0 [ERROR] InnoDB: Not applying INSERT_HEAP_DYNAMIC due to corruption on [page id: space=2399, page number=8538]
|
Sep 8 07:09:30 breww-db-03 sh[3691097]: 2022-09-08 7:09:27 0 [ERROR] InnoDB: Set innodb_force_recovery=1 to ignore corruption.
|
Sep 8 07:09:30 breww-db-03 sh[3691097]: 2022-09-08 7:09:27 0 [ERROR] InnoDB: Failed to read page 4 from file './breww/app_integrationwebhookqueuedtask.ibd': Page read from tablespace is corrupted.
|
Sep 8 07:09:30 breww-db-03 sh[3691097]: 2022-09-08 7:09:27 0 [ERROR] InnoDB: Failed to read page 8538 from file './breww/app_integrationwebhookqueuedtask.ibd': Page read from tablespace is corrupted.
|
Sep 8 07:09:30 breww-db-03 sh[3691097]: 2022-09-08 7:09:27 0 [ERROR] InnoDB: Failed to read page 2 from file './breww/app_integrationwebhookqueuedtask.ibd': Page read from tablespace is corrupted.
|
Sep 8 07:09:30 breww-db-03 sh[3691097]: 2022-09-08 7:09:27 0 [ERROR] InnoDB: Failed to read page 3 from file './breww/app_location.ibd': Page read from tablespace is corrupted.
|
Sep 8 07:09:30 breww-db-03 sh[3691097]: 2022-09-08 7:09:27 0 [ERROR] InnoDB: Failed to read page 11 from file './breww/app_location.ibd': Page read from tablespace is corrupted.
|
Sep 8 07:09:30 breww-db-03 sh[3691097]: 2022-09-08 7:09:27 0 [ERROR] InnoDB: Failed to read page 12 from file './breww/app_location.ibd': Page read from tablespace is corrupted.
|
Sep 8 07:09:30 breww-db-03 sh[3691097]: 2022-09-08 7:09:27 0 [ERROR] InnoDB: Failed to read page 13 from file './breww/app_productsitetransfer.ibd': Page read from tablespace is corrupted.
|
Sep 8 07:09:30 breww-db-03 sh[3691097]: 2022-09-08 7:09:27 0 [ERROR] InnoDB: Failed to read page 3 from file './breww/app_productsitetransfer.ibd': Page read from tablespace is corrupted.
|
Sep 8 07:09:30 breww-db-03 sh[3691097]: 2022-09-08 7:09:27 0 [ERROR] InnoDB: Failed to read page 27 from file './breww/app_productsitetransfer.ibd': Page read from tablespace is corrupted.
|