Uploaded image for project: 'MariaDB Server'
  1. MariaDB Server
  2. MDEV-29563

MariaDB 10.6.9 crashing every minute

    XMLWordPrintable

Details

    • Bug
    • Status: Closed (View Workflow)
    • Critical
    • Resolution: Duplicate
    • 10.6.9
    • N/A
    • None
    • Centos 7

    Description

      I updated MariaDB to version 10.6.9 from 10.6.8 and after that Mariadb started to crash more or less every minute.

      I downgraded to 10.6.8 and it stopped crashing, which means the issue is with the last version.

      Here is the log of when the crash was occurring:

      Crash Log

      Sep 18 01:20:09 systemd[1]: Starting MariaDB 10.6.9 database server...
      Sep 18 01:20:09 mariadbd[17467]: 2022-09-18 1:20:09 0 [Note] /usr/sbin/mariadbd (server 10.6.9-MariaDB-log) starting as process 17467 ...
      Sep 18 01:20:09 mariadbd[17467]: 2022-09-18 1:20:09 0 [Warning] Could not increase number of max_open_files to more than 32768 (request: 128281)
      Sep 18 01:20:09 mariadbd[17467]: 2022-09-18 1:20:09 0 [Warning] You need to use --log-bin to make --binlog-format work.
      Sep 18 01:20:09 mariadbd[17467]: 2022-09-18 1:20:09 0 [Note] InnoDB: Compressed tables use zlib 1.2.7
      Sep 18 01:20:09 mariadbd[17467]: 2022-09-18 1:20:09 0 [Note] InnoDB: Number of pools: 1
      Sep 18 01:20:09 mariadbd[17467]: 2022-09-18 1:20:09 0 [Note] InnoDB: Using crc32 + pclmulqdq instructions
      Sep 18 01:20:09 mariadbd[17467]: 2022-09-18 1:20:09 0 [Note] InnoDB: Using Linux native AIO
      Sep 18 01:20:09 mariadbd[17467]: 2022-09-18 1:20:09 0 [Note] InnoDB: Initializing buffer pool, total size = 17179869184, chunk size = 134217728
      Sep 18 01:20:09 mariadbd[17467]: 2022-09-18 1:20:09 0 [Note] InnoDB: Completed initialization of buffer pool
      Sep 18 01:20:09 mariadbd[17467]: 2022-09-18 1:20:09 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=4497170045405,4497715985659
      Sep 18 01:20:10 mariadbd[17467]: 2022-09-18 1:20:10 0 [Note] InnoDB: 2 transaction(s) which must be rolled back or cleaned up in total 24118 row operations to undo
      Sep 18 01:20:10 mariadbd[17467]: 2022-09-18 1:20:10 0 [Note] InnoDB: Trx id counter is 954496450
      Sep 18 01:20:10 mariadbd[17467]: 2022-09-18 1:20:10 0 [Note] InnoDB: Starting final batch to recover 52800 pages from redo log.
      Sep 18 01:20:10 mariadbd[17467]: 2022-09-18 1:20:10 0 [Note] InnoDB: 128 rollback segments are active.
      Sep 18 01:20:10 mariadbd[17467]: 2022-09-18 1:20:10 0 [Note] InnoDB: Starting in background the rollback of recovered transactions
      Sep 18 01:20:10 mariadbd[17467]: 2022-09-18 1:20:10 0 [Note] InnoDB: Removed temporary tablespace data file: "./ibtmp1"
      Sep 18 01:20:10 mariadbd[17467]: 2022-09-18 1:20:10 0 [Note] InnoDB: Creating shared tablespace for temporary tables
      Sep 18 01:20:10 mariadbd[17467]: 2022-09-18 1:20:10 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ...
      Sep 18 01:20:10 mariadbd[17467]: 2022-09-18 1:20:10 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB.
      Sep 18 01:20:10 mariadbd[17467]: 2022-09-18 1:20:10 0 [Note] InnoDB: 10.6.9 started; log sequence number 4497715989578; transaction id 954496451
      Sep 18 01:20:10 mariadbd[17467]: 2022-09-18 1:20:10 0 [Note] InnoDB: Loading buffer pool(s) from /var/lib/mysql/ib_buffer_pool
      Sep 18 01:20:10 mariadbd[17467]: 2022-09-18 1:20:10 0 [Note] Plugin 'FEEDBACK' is disabled.
      Sep 18 01:20:10 mariadbd[17467]: 2022-09-18 1:20:10 0 [ERROR] mariadbd: File '/var/log/mysq-slow.log' not found (Errcode: 13 "Permission denied")
      Sep 18 01:20:10 mariadbd[17467]: 2022-09-18 1:20:10 0 [ERROR] Could not use /var/log/mysq-slow.log for logging (error 13). Turning logging off for the whole duration of the MariaDB server process. To tu
      Sep 18 01:20:10 mariadbd[17467]: 2022-09-18 1:20:10 0 [Warning] 'proxies_priv' entry '@% root@app.gods.com' ignored in --skip-name-resolve mode.
      Sep 18 01:20:10 mariadbd[17467]: 2022-09-18 1:20:10 0 [Note] InnoDB: Rolled back recovered transaction 954496381
      Sep 18 01:20:10 mariadbd[17467]: 2022-09-18 1:20:10 0 [Note] /usr/sbin/mariadbd: ready for connections.
      Sep 18 01:20:10 mariadbd[17467]: Version: '10.6.9-MariaDB-log' socket: '/var/lib/mysql/mysql.sock' port: 0 MariaDB Server
      Sep 18 01:20:10 systemd[1]: Started MariaDB 10.6.9 database server.
      Sep 18 01:20:10 mariadbd[17467]: 2022-09-18 1:20:10 0 [Note] InnoDB: Rolled back recovered transaction 954496384
      Sep 18 01:20:10 mariadbd[17467]: 2022-09-18 1:20:10 0 [Note] InnoDB: Rollback of non-prepared transactions completed
      Sep 18 01:20:12 mariadbd[17467]: 2022-09-18 1:20:12 10 [Warning] InnoDB: Total InnoDB FTS size 15451684 for the table `admin_gods`.`ipb__core_search_index` exceeds the innodb_ft_cache_size 8000000
      Sep 18 01:20:12 mariadbd[17467]: 2022-09-18 1:20:12 10 [ERROR] InnoDB: (Duplicate key) writing word node to FTS auxiliary index table `admin_gods`.`ipb__core_search_index`
      Sep 18 01:20:12 mariadbd[17467]: 2022-09-18 01:20:12 0x7f635c21a700 InnoDB: Assertion failure in file /home/buildbot/buildbot/padding_for_CPACK_RPM_BUILD_SOURCE_DIRS_PREFIX/mariadb-10.6.9/storage/innoba
      Sep 18 01:20:12 mariadbd[17467]: InnoDB: Failing assertion: trx->error_state == DB_SUCCESS
      Sep 18 01:20:12 mariadbd[17467]: InnoDB: We intentionally generate a memory trap.
      Sep 18 01:20:12 mariadbd[17467]: InnoDB: Submit a detailed bug report to https://jira.mariadb.org/
      Sep 18 01:20:12 mariadbd[17467]: InnoDB: If you get repeated assertion failures or crashes, even
      Sep 18 01:20:12 mariadbd[17467]: InnoDB: immediately after the mariadbd startup, there may be
      Sep 18 01:20:12 mariadbd[17467]: InnoDB: corruption in the InnoDB tablespace. Please refer to
      Sep 18 01:20:12 mariadbd[17467]: InnoDB: https://mariadb.com/kb/en/library/innodb-recovery-modes/
      Sep 18 01:20:12 mariadbd[17467]: InnoDB: about forcing recovery.
      Sep 18 01:20:12 mariadbd[17467]: 220918 1:20:12 [ERROR] mysqld got signal 6 ;
      Sep 18 01:20:12 mariadbd[17467]: This could be because you hit a bug. It is also possible that this binary
      Sep 18 01:20:12 mariadbd[17467]: or one of the libraries it was linked against is corrupt, improperly built,
      Sep 18 01:20:12 mariadbd[17467]: or misconfigured. This error can also be caused by malfunctioning hardware.
      Sep 18 01:20:12 mariadbd[17467]: To report this bug, see https://mariadb.com/kb/en/reporting-bugs
      Sep 18 01:20:12 mariadbd[17467]: We will try our best to scrape up some info that will hopefully help
      Sep 18 01:20:12 mariadbd[17467]: diagnose the problem, but since we have already crashed,
      Sep 18 01:20:12 mariadbd[17467]: something is definitely wrong and this may fail.
      Sep 18 01:20:12 mariadbd[17467]: Server version: 10.6.9-MariaDB-log
      Sep 18 01:20:12 mariadbd[17467]: key_buffer_size=33554432
      Sep 18 01:20:12 mariadbd[17467]: read_buffer_size=131072
      Sep 18 01:20:12 mariadbd[17467]: max_used_connections=15
      Sep 18 01:20:12 mariadbd[17467]: max_threads=65537
      Sep 18 01:20:12 mariadbd[17467]: thread_count=11
      Sep 18 01:20:12 mariadbd[17467]: It is possible that mysqld could use up to
      Sep 18 01:20:12 mariadbd[17467]: key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 142647817 K bytes of memory
      Sep 18 01:20:12 mariadbd[17467]: Hope that's ok; if not, decrease some variables in the equation.
      Sep 18 01:20:12 mariadbd[17467]: Thread pointer: 0x7f61e4000a98
      Sep 18 01:20:12 mariadbd[17467]: Attempting backtrace. You can use the following information to find out
      Sep 18 01:20:12 mariadbd[17467]: where mysqld died. If you see no messages after this, something went
      Sep 18 01:20:12 mariadbd[17467]: terribly wrong...
      Sep 18 01:20:12 mariadbd[17467]: stack_bottom = 0x7f635c219c20 thread_stack 0x49000
      Sep 18 01:20:12 mariadbd[17467]: ??:0(my_print_stacktrace)[0x558ef910d48e]
      Sep 18 01:20:12 mariadbd[17467]: ??:0(handle_fatal_signal)[0x558ef8b5e217]
      Sep 18 01:20:12 mariadbd[17467]: sigaction.c:0(__restore_rt)[0x7f67969f2630]
      Sep 18 01:20:12 mariadbd[17467]: :0(__GI_raise)[0x7f6795e3d387]
      Sep 18 01:20:12 mariadbd[17467]: :0(__GI_abort)[0x7f6795e3ea78]
      Sep 18 01:20:12 mariadbd[17467]: /usr/sbin/mariadbd(+0x685bcd)[0x558ef8816bcd]
      Sep 18 01:20:12 mariadbd[17467]: ??:0(void std::__introsort_loop<unsigned char*, long>(unsigned char, unsigned char*, long))[0x558ef8ee84c3]
      Sep 18 01:20:12 mariadbd[17467]: ??:0(void std::vector<unsigned long, std::allocator<unsigned long> >::_M_emplace_back_aux<unsigned long>(unsigned long&&))[0x558ef906e665]
      Sep 18 01:20:12 mariadbd[17467]: ??:0(void std::vector<unsigned long, std::allocator<unsigned long> >::_M_emplace_back_aux<unsigned long>(unsigned long&&))[0x558ef905661f]
      Sep 18 01:20:12 mariadbd[17467]: ??:0(void std::vector<unsigned long, std::allocator<unsigned long> >::_M_emplace_back_aux<unsigned long>(unsigned long&&))[0x558ef905c571]
      Sep 18 01:20:12 mariadbd[17467]: ??:0(void std::vector<unsigned long, std::allocator<unsigned long> >::_M_emplace_back_aux<unsigned long>(unsigned long&&))[0x558ef905e52b]
      Sep 18 01:20:12 mariadbd[17467]: ??:0(void std::vector<unsigned long, std::allocator<unsigned long> >::_M_emplace_back_aux<unsigned long>(unsigned long&&))[0x558ef905eb75]
      Sep 18 01:20:12 mariadbd[17467]: ??:0(void std::vector<unsigned long, std::allocator<unsigned long> >::_M_emplace_back_aux<unsigned long>(unsigned long&&))[0x558ef905ee08]
      Sep 18 01:20:12 mariadbd[17467]: ??:0(std::pair<std::_Rb_tree_iterator<unsigned int>, bool> std::_Rb_tree<unsigned int, unsigned int, std::_Identity<unsigned int>, std::less<unsigned int>, std::allocator
      Sep 18 01:20:12 mariadbd[17467]: ??:0(std::pair<std::_Rb_tree_iterator<unsigned int>, bool> std::_Rb_tree<unsigned int, unsigned int, std::_Identity<unsigned int>, std::less<unsigned int>, std::allocator
      Sep 18 01:20:12 mariadbd[17467]: ??:0(std::pair<std::_Rb_tree_iterator<unsigned int>, bool> std::_Rb_tree<unsigned int, unsigned int, std::_Identity<unsigned int>, std::less<unsigned int>, std::allocator
      Sep 18 01:20:12 mariadbd[17467]: ??:0(wsrep_notify_status(wsrep::server_state::state, wsrep::view const*))[0x558ef8e41810]
      Sep 18 01:20:12 mariadbd[17467]: ??:0(ha_check_and_coalesce_trx_read_only(THD*, Ha_trx_info*, bool))[0x558ef8b6178e]
      Sep 18 01:20:12 mariadbd[17467]: ??:0(ha_commit_trans(THD*, bool))[0x558ef8b6e54b]
      Sep 18 01:20:12 mariadbd[17467]: ??:0(trans_commit_stmt(THD*))[0x558ef8a47a85]
      Sep 18 01:20:12 mariadbd[17467]: ??:0(mysql_execute_command(THD*, bool))[0x558ef89398c0]
      Sep 18 01:20:12 mariadbd[17467]: ??:0(Prepared_statement::execute(String*, bool))[0x558ef8954cb2]
      Sep 18 01:20:12 mariadbd[17467]: ??:0(Prepared_statement::execute_loop(String*, bool, unsigned char*, unsigned char*))[0x558ef8954ebe]
      Sep 18 01:20:12 mariadbd[17467]: ??:0(Prepared_statement::execute_bulk_loop(String*, bool, unsigned char*, unsigned char*))[0x558ef8956391]
      Sep 18 01:20:12 mariadbd[17467]: ??:0(mysqld_stmt_execute(THD*, char*, unsigned int))[0x558ef895655d]
      Sep 18 01:20:12 mariadbd[17467]: ??:0(dispatch_command(enum_server_command, THD*, char*, unsigned int, bool))[0x558ef89412ba]
      Sep 18 01:20:12 mariadbd[17467]: ??:0(do_command(THD*, bool))[0x558ef8942273]
      Sep 18 01:20:12 mariadbd[17467]: ??:0(tp_callback(TP_connection*))[0x558ef8adfcd2]
      Sep 18 01:20:12 mariadbd[17467]: ??:0(get_event(worker_thread_t*, thread_group_t*, timespec*))[0x558ef8cf0dd0]
      Sep 18 01:20:12 mariadbd[17467]: ??:0(MyCTX_nopad::finish(unsigned char*, unsigned int*))[0x558ef8da38cc]
      Sep 18 01:20:12 mariadbd[17467]: pthread_create.c:0(start_thread)[0x7f67969eaea5]
      Sep 18 01:20:12 mariadbd[17467]: 2022-09-18 1:20:12 0 [Note] InnoDB: Buffer pool(s) load completed at 220918 1:20:12
      Sep 18 01:20:12 mariadbd[17467]: ??:0(__clone)[0x7f6795f05b0d]
      Sep 18 01:20:12 mariadbd[17467]: Trying to get some variables.
      Sep 18 01:20:12 mariadbd[17467]: Some pointers may be invalid and cause the dump to abort.
      Sep 18 01:20:12 mariadbd[17467]: Query (0x7f61e4010a70): /admin_gods::admin_userig::IPS\Content\Search\Mysql_Index::index:666/ INSERT INTO `ipb__core_search_index` ( `index_class`, `index_object_id
      Sep 18 01:20:13 systemd[1]: mariadb.service: main process exited, code=killed, status=6/ABRT
      Sep 18 01:20:13 systemd[1]: Unit mariadb.service entered failed state.
      Sep 18 01:20:13 systemd[1]: mariadb.service failed.
      Sep 18 01:20:18 systemd[1]: mariadb.service holdoff time over, scheduling restart.
      Sep 18 01:20:18 systemd[1]: Stopped MariaDB 10.6.9 database server.
      Sep 18 01:20:18 systemd[1]: Starting MariaDB 10.6.9 database server...

      Attachments

        Issue Links

          Activity

            People

              Unassigned Unassigned
              Venâncio Ferreira Venâncio Ferreira
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Git Integration

                  Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.