Details
-
Bug
-
Status: Open (View Workflow)
-
Minor
-
Resolution: Unresolved
-
10.3.12
-
None
-
None
Description
Production server restarted earlier this morning, resulting in some corrupt tables:
mysql@outdb-wb-p1:/u01/appl/mariadb/admin/::more error.log
|
logs cleared by cron
|
200909 6:40:51 [ERROR] mysqld got signal 11 ;
|
This could be because you hit a bug. It is also possible that this binary
|
or one of the libraries it was linked against is corrupt, improperly built,
|
or misconfigured. This error can also be caused by malfunctioning hardware.
|
|
To report this bug, see https://mariadb.com/kb/en/reporting-bugs
|
|
We will try our best to scrape up some info that will hopefully help
|
diagnose the problem, but since we have already crashed,
|
something is definitely wrong and this may fail.
|
|
Server version: 10.3.12-MariaDB-log
|
key_buffer_size=134217728
|
read_buffer_size=131072
|
max_used_connections=61
|
max_threads=1852
|
thread_count=27
|
It is possible that mysqld could use up to
|
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 30752567 K bytes of memory
|
Hope that's ok; if not, decrease some variables in the equation.
|
|
Thread pointer: 0x7f22f0065ac8
|
Attempting backtrace. You can use the following information to find out
|
where mysqld died. If you see no messages after this, something went
|
terribly wrong...
|
stack_bottom = 0x7f2344070c4f thread_stack 0x20000
|
/usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x55c35883ed1e]
|
/usr/sbin/mysqld(handle_fatal_signal+0x357)[0x55c3582dfd97]
|
/lib64/libpthread.so.0(+0xf630)[0x7f2602d78630]
|
/usr/lib/oracle/12.2/client64/lib/libclntsh.so.12.1(+0x62376f)[0x7f22b2b7d76f]
|
/usr/lib/oracle/12.2/client64/lib/libclntsh.so.12.1(+0x6221b8)[0x7f22b2b7c1b8]
|
/usr/lib/oracle/12.2/client64/lib/libclntsh.so.12.1(nladini_t+0x1d1)[0x7f22b2b7bfa1]
|
/usr/lib/oracle/12.2/client64/lib/libclntsh.so.12.1(nscall1+0x15c)[0x7f22b2a0cb9c]
|
/usr/lib/oracle/12.2/client64/lib/libclntsh.so.12.1(nscall+0x362)[0x7f22b2a0a992]
|
/usr/lib/oracle/12.2/client64/lib/libclntsh.so.12.1(niotns+0x90f)[0x7f22b2a9f3df]
|
/usr/lib/oracle/12.2/client64/lib/libclntsh.so.12.1(osncon+0x476)[0x7f22b2aaa066]
|
/usr/lib/oracle/12.2/client64/lib/libclntsh.so.12.1(kpuadef+0x3d)[0x7f22b2952cfd]
|
/usr/lib/oracle/12.2/client64/lib/libclntsh.so.12.1(upiini+0x3b7)[0x7f22b2939897]
|
/usr/lib/oracle/12.2/client64/lib/libclntsh.so.12.1(kpuatch+0x47c)[0x7f22b295193c]
|
/usr/lib/oracle/12.2/client64/lib/libclntsh.so.12.1(kwfcrat+0x204)[0x7f22b3712d24]
|
/usr/lib/oracle/12.2/client64/lib/libclntsh.so.12.1(kwfcfoc+0x942)[0x7f22b3713d42]
|
/usr/lib/oracle/12.2/client64/lib/libclntsh.so.12.1(kpcnps+0x261)[0x7f22b52b33b1]
|
/usr/lib/oracle/12.2/client64/lib/libclntsh.so.12.1(upirtrc+0x6c1)[0x7f22b5274331]
|
/usr/lib/oracle/12.2/client64/lib/libclntsh.so.12.1(kpurcsc+0x66)[0x7f22b52802f6]
|
/usr/lib/oracle/12.2/client64/lib/libclntsh.so.12.1(kpufch0+0x780)[0x7f22b527d5a0]
|
/usr/lib/oracle/12.2/client64/lib/libclntsh.so.12.1(kpufch+0x4f4)[0x7f22b527bad4]
|
/usr/lib/oracle/12.2/client64/lib/libsqora.so.12.1(+0x213d8)[0x7f23440b53d8]
|
/usr/lib/oracle/12.2/client64/lib/libsqora.so.12.1(bcoCacheFetch+0xb65)[0x7f23440b3f05]
|
/usr/lib/oracle/12.2/client64/lib/libsqora.so.12.1(bcoSQLScroll+0x5e3)[0x7f23440d8813]
|
/usr/lib/oracle/12.2/client64/lib/libsqora.so.12.1(bcoSQLFetch+0xd3)[0x7f23440d8193]
|
/usr/lib/oracle/12.2/client64/lib/libsqora.so.12.1(SQLFetch+0x2d3)[0x7f2344112b23]
|
/lib64/libodbc.so.2(SQLFetch+0x103)[0x7f25fc732c23]
|
/usr/lib64/mysql/plugin/ha_connect.so(_ZN7ODBConn5FetchEi+0x28)[0x7f25fce18658]
|
/usr/lib64/mysql/plugin/ha_connect.so(_ZN7TDBODBC6ReadDBEP7_global+0x1ce)[0x7f25fce143ce]
|
/usr/lib64/mysql/plugin/ha_connect.so(_Z11CntReadNextP7_globalP3TDB+0x70)[0x7f25fcd86330]
|
/usr/lib64/mysql/plugin/ha_connect.so(_ZN10ha_connect8rnd_nextEPh+0x68)[0x7f25fcd79f38]
|
/usr/sbin/mysqld(_ZN7handler11ha_rnd_nextEPh+0x107)[0x55c3582e4bc7]
|
/usr/sbin/mysqld(_Z13rr_sequentialP11READ_RECORD+0x1c)[0x55c35840885c]
|
/usr/sbin/mysqld(_Z10sub_selectP4JOINP13st_join_tableb+0x20f)[0x55c358141adf]
|
/usr/sbin/mysqld(_ZN4JOIN10exec_innerEv+0xa7b)[0x55c35816324b]
|
/usr/sbin/mysqld(_ZN4JOIN4execEv+0x33)[0x55c358163463]
|
/usr/sbin/mysqld(_Z12mysql_selectP3THDP10TABLE_LISTjR4ListI4ItemEPS4_jP8st_orderS9_S7_S9_yP13select_resultP18st_select_lex_unitP13st_select_lex+0x11a)[0x55c3581635ba]
|
/usr/sbin/mysqld(_Z13handle_selectP3THDP3LEXP13select_resultm+0x1cc)[0x55c3581640cc]
|
/usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x8253)[0x55c358111883]
|
/usr/sbin/mysqld(_ZN13sp_instr_stmt9exec_coreEP3THDPj+0x36)[0x55c358078836]
|
/usr/sbin/mysqld(_ZN13sp_lex_keeper23reset_lex_and_exec_coreEP3THDPjbP8sp_instr+0x99)[0x55c358080259]
|
/usr/sbin/mysqld(_ZN13sp_instr_stmt7executeEP3THDPj+0x63b)[0x55c358080c9b]
|
/usr/sbin/mysqld(_ZN7sp_head7executeEP3THDb+0x860)[0x55c35807c1c0]
|
/usr/sbin/mysqld(_ZN7sp_head17execute_procedureEP3THDP4ListI4ItemE+0x74d)[0x55c35807d3dd]
|
/usr/sbin/mysqld(_ZN14Event_job_data7executeEP3THDb+0x6be)[0x55c3581d045e]
|
/usr/sbin/mysqld(_ZN19Event_worker_thread3runEP3THDP28Event_queue_element_for_exec+0xc9)[0x55c35843b7f9]
|
/usr/sbin/mysqld(event_worker_thread+0x4d)[0x55c35843b8ad]
|
/lib64/libpthread.so.0(+0x7ea5)[0x7f2602d70ea5]
|
/lib64/libc.so.6(clone+0x6d)[0x7f26011118cd]
|
|
Trying to get some variables.
|
Some pointers may be invalid and cause the dump to abort.
|
Query (0x7f226c229a20): INSERT INTO WORDMPF_STAGE SELECT * FROM WORDMPF_LINK
|
Connection ID (thread ID): 57347
|
Status: NOT_KILLED
|
|
Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_p
|
ushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_
|
cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_s
|
ize=off,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on
|
|
The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
|
information that should help you find out what is causing the crash.
|
2020-09-09 6:40:58 0 [Note] CONNECT: Version 1.06.0008 October 06, 2018
|
2020-09-09 6:40:58 0 [Note] InnoDB: Using Linux native AIO
|
2020-09-09 6:40:58 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
|
2020-09-09 6:40:58 0 [Note] InnoDB: Uses event mutexes
|
2020-09-09 6:40:58 0 [Note] InnoDB: Compressed tables use zlib 1.2.7
|
2020-09-09 6:40:58 0 [Note] InnoDB: Number of pools: 1
|
2020-09-09 6:40:58 0 [Note] InnoDB: Using SSE2 crc32 instructions
|
2020-09-09 6:40:58 0 [Note] InnoDB: Initializing buffer pool, total size = 10G, instances = 8, chunk size = 128M
|
2020-09-09 6:40:59 0 [Note] InnoDB: Completed initialization of buffer pool
|
2020-09-09 6:40:59 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority().
|
2020-09-09 6:40:59 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=137536654
|
2020-09-09 6:41:00 0 [Note] InnoDB: Last binlog file '/u02/binarylog/binlog.000763', position 911862
|
2020-09-09 6:41:00 0 [Note] InnoDB: 128 out of 128 rollback segments are active.
|
2020-09-09 6:41:00 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1"
|
2020-09-09 6:41:00 0 [Note] InnoDB: Creating shared tablespace for temporary tables
|
2020-09-09 6:41:00 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ...
|
2020-09-09 6:41:00 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB.
|
2020-09-09 6:41:00 0 [Note] InnoDB: Waiting for purge to start
|
2020-09-09 6:41:00 0 [Note] InnoDB: 10.3.12 started; log sequence number 137536663; transaction id 273391
|
2020-09-09 6:41:00 0 [Note] InnoDB: Loading buffer pool(s) from /u02/mariadata/ib_buffer_pool
|
2020-09-09 6:41:00 0 [Note] Plugin 'FEEDBACK' is disabled.
|
2020-09-09 6:41:00 0 [Note] Recovering after a crash using /u02/binarylog/binlog
|
2020-09-09 6:41:00 0 [Note] InnoDB: Buffer pool(s) load completed at 200909 6:41:00
|
2020-09-09 6:41:00 0 [Note] Starting crash recovery...
|
2020-09-09 6:41:00 0 [Note] Crash recovery finished.
|
2020-09-09 6:41:00 0 [Note] Server socket created on IP: '::'.
|
2020-09-09 6:41:00 0 [ERROR] mysqld: Table './mysql/event' is marked as crashed and should be repaired
|
2020-09-09 6:41:00 0 [Warning] Checking table: './mysql/event'
|
2020-09-09 6:41:00 0 [ERROR] mysql.event: 1 client is using or hasn't closed the table properly
|
2020-09-09 6:41:00 7 [Note] Event Scheduler: scheduler thread started with id 7
|
2020-09-09 6:41:00 0 [Note] Reading of all Master_info entries succeded
|
2020-09-09 6:41:00 0 [Note] Added new Master_info '' to hash table
|
2020-09-09 6:41:00 0 [Note] /usr/sbin/mysqld: ready for connections.
|
Version: '10.3.12-MariaDB-log' socket: '/u01/appl/mariadb/admin/mysqld.sock' port: 3306 MariaDB Server
|
2020-09-09 6:41:23 12 [ERROR] mysqld: Table './outdb_prod/resolve_event' is marked as crashed and should be repaired
|
2020-09-09 6:41:23 12 [Warning] Checking table: './outdb_prod/resolve_event'
|
2020-09-09 6:42:38 12 [ERROR] mysqld: Table './outdb_prod/resolve_registration' is marked as crashed and should be repaired
|
2020-09-09 6:42:38 12 [Warning] Checking table: './outdb_prod/resolve_registration'
|
2020-09-09 6:45:00 23 [ERROR] mysqld: Table './outdb_prod/itsm_10_mobile_outages' is marked as crashed and should be repaired
|
2020-09-09 6:45:00 23 [Warning] Checking table: './outdb_prod/itsm_10_mobile_outages'
|
2020-09-09 6:45:00 23 [ERROR] outdb_prod.ITSM_10_MOBILE_OUTAGES: 1 client is using or hasn't closed the table properly
|
2020-09-09 6:45:53 17 [ERROR] mysqld: Table './outdb_prod/archive_worksheet#P#P_SEP' is marked as crashed and should be repaired
|
2020-09-09 6:45:53 17 [Warning] Checking table: './outdb_prod/archive_worksheet'
|
2020-09-09 6:45:53 17 [Warning] Checking table: './outdb_prod/archive_worksheet'
|
2020-09-09 6:45:53 17 [Warning] Checking table: './outdb_prod/archive_worksheet'
|
2020-09-09 6:45:53 17 [Warning] Checking table: './outdb_prod/archive_worksheet'
|
2020-09-09 6:45:53 17 [Warning] Checking table: './outdb_prod/archive_worksheet'
|
2020-09-09 6:45:53 17 [Warning] Checking table: './outdb_prod/archive_worksheet'
|
2020-09-09 6:45:53 17 [Warning] Checking table: './outdb_prod/archive_worksheet'
|
2020-09-09 6:45:53 17 [Warning] Checking table: './outdb_prod/archive_worksheet'
|
2020-09-09 6:45:53 17 [Warning] Checking table: './outdb_prod/archive_worksheet'
|
2020-09-09 6:45:53 17 [Warning] Checking table: './outdb_prod/archive_worksheet'
|
2020-09-09 6:45:54 17 [Warning] Checking table: './outdb_prod/archive_worksheet'
|
2020-09-09 6:45:54 17 [Warning] Checking table: './outdb_prod/archive_worksheet'
|
2020-09-09 6:45:54 17 [ERROR] mysqld: Table './outdb_prod/archive_action_result_lob#P#P_SEP' is marked as crashed and should be repaired
|
2020-09-09 6:45:54 17 [Warning] Checking table: './outdb_prod/archive_action_result_lob'
|
2020-09-09 6:45:55 17 [Warning] Checking table: './outdb_prod/archive_action_result_lob'
|
2020-09-09 6:45:56 17 [Warning] Checking table: './outdb_prod/archive_action_result_lob'
|
2020-09-09 6:45:57 17 [Warning] Checking table: './outdb_prod/archive_action_result_lob'
|
2020-09-09 6:45:58 17 [Warning] Checking table: './outdb_prod/archive_action_result_lob'
|
2020-09-09 6:45:59 17 [Warning] Checking table: './outdb_prod/archive_action_result_lob'
|
2020-09-09 6:45:59 17 [Warning] Checking table: './outdb_prod/archive_action_result_lob'
|
2020-09-09 6:46:00 17 [Warning] Checking table: './outdb_prod/archive_action_result_lob'
|
2020-09-09 6:46:01 17 [Warning] Checking table: './outdb_prod/archive_action_result_lob'
|
2020-09-09 6:46:01 17 [Warning] Checking table: './outdb_prod/archive_action_result_lob'
|
2020-09-09 6:46:01 17 [Warning] Checking table: './outdb_prod/archive_action_result_lob'
|
2020-09-09 6:46:02 17 [Warning] Checking table: './outdb_prod/archive_action_result_lob'
|
2020-09-09 6:46:02 17 [ERROR] mysqld: Table './outdb_prod/archive_process_request#P#P_SEP' is marked as crashed and should be repaired
|
2020-09-09 6:46:02 17 [Warning] Checking table: './outdb_prod/archive_process_request'
|
2020-09-09 6:46:03 17 [Warning] Checking table: './outdb_prod/archive_process_request'
|
2020-09-09 6:46:03 17 [Warning] Checking table: './outdb_prod/archive_process_request'
|
2020-09-09 6:46:03 17 [Warning] Checking table: './outdb_prod/archive_process_request'
|
2020-09-09 6:46:03 17 [Warning] Checking table: './outdb_prod/archive_process_request'
|
2020-09-09 6:46:04 17 [Warning] Checking table: './outdb_prod/archive_process_request'
|
2020-09-09 6:46:04 17 [Warning] Checking table: './outdb_prod/archive_process_request'
|
2020-09-09 6:46:04 17 [Warning] Checking table: './outdb_prod/archive_process_request'
|
2020-09-09 6:46:04 17 [Warning] Checking table: './outdb_prod/archive_process_request'
|
2020-09-09 6:46:04 17 [Warning] Checking table: './outdb_prod/archive_process_request'
|
2020-09-09 6:46:04 17 [Warning] Checking table: './outdb_prod/archive_process_request'
|
2020-09-09 6:46:04 17 [Warning] Checking table: './outdb_prod/archive_process_request'
|
2020-09-09 6:46:05 17 [ERROR] mysqld: Table './outdb_prod/properties' is marked as crashed and should be repaired
|
2020-09-09 6:46:05 17 [Warning] Checking table: './outdb_prod/properties'
|
2020-09-09 6:46:05 17 [ERROR] mysqld: Table './outdb_prod/wikidoc_statistics' is marked as crashed and should be repaired
|
2020-09-09 6:46:05 17 [Warning] Checking table: './outdb_prod/wikidoc_statistics'
|
2020-09-09 6:46:05 17 [ERROR] mysqld: Table './outdb_prod/archive_action_result#P#P_SEP' is marked as crashed and should be repaired
|
2020-09-09 6:46:05 17 [Warning] Checking table: './outdb_prod/archive_action_result'
|
2020-09-09 6:46:08 17 [Warning] Checking table: './outdb_prod/archive_action_result'
|
2020-09-09 6:46:11 17 [Warning] Checking table: './outdb_prod/archive_action_result'
|
2020-09-09 6:46:13 17 [Warning] Checking table: './outdb_prod/archive_action_result'
|
2020-09-09 6:46:16 17 [Warning] Checking table: './outdb_prod/archive_action_result'
|
2020-09-09 6:46:19 17 [Warning] Checking table: './outdb_prod/archive_action_result'
|
2020-09-09 6:46:21 17 [Warning] Checking table: './outdb_prod/archive_action_result'
|
2020-09-09 6:46:24 17 [Warning] Checking table: './outdb_prod/archive_action_result'
|
2020-09-09 6:46:27 17 [Warning] Checking table: './outdb_prod/archive_action_result'
|
2020-09-09 6:46:27 17 [Warning] Checking table: './outdb_prod/archive_action_result'
|
2020-09-09 6:46:27 17 [Warning] Checking table: './outdb_prod/archive_action_result'
|
2020-09-09 6:46:28 17 [Warning] Checking table: './outdb_prod/archive_action_result'
|
are you to check to see if this has been caused by a bug with the release?
we will repair the affected tables for now.