Elena Stepanova added a comment - 2018-07-16 10:21 Prior to the first occurrence of the problem, did you have any incidents which could cause actual data corruption, disk failures or alike? If it keeps happening all the time, could you please enable the coredump and get the stack trace? Should it be needed, would you be willing to provide access to the crashing instance to one of our developers?

ssauravy added a comment - 2018-07-18 01:06 - edited Good morning. 1. No problems such as data corruption or disk error were found before or after the failure.      / var / log / dmsg, and / var / log / messages. 2. I will share the contents except the logging data for the repeated actions after the restart point. 3. Please understand that the system is closed network and external connection is not possible. 4. I do not know if DB variables, plugins information will help. MariaDB [(none)]> show plugins; +-----------------------------+----------+--------------------+-----------------------+---------+ | Name | Status | Type | Library | License | +-----------------------------+----------+--------------------+-----------------------+---------+ | binlog | ACTIVE | STORAGE ENGINE | NULL | GPL | | mysql_native_password | ACTIVE | AUTHENTICATION | NULL | GPL | | mysql_old_password | ACTIVE | AUTHENTICATION | NULL | GPL | | CSV | ACTIVE | STORAGE ENGINE | NULL | GPL | | MEMORY | ACTIVE | STORAGE ENGINE | NULL | GPL | | MyISAM | ACTIVE | STORAGE ENGINE | NULL | GPL | | MRG_MyISAM | ACTIVE | STORAGE ENGINE | NULL | GPL | | BLACKHOLE | ACTIVE | STORAGE ENGINE | NULL | GPL | | InnoDB | ACTIVE | STORAGE ENGINE | NULL | GPL | | XTRADB_READ_VIEW | ACTIVE | INFORMATION SCHEMA | NULL | GPL | | XTRADB_INTERNAL_HASH_TABLES | ACTIVE | INFORMATION SCHEMA | NULL | GPL | | XTRADB_RSEG | ACTIVE | INFORMATION SCHEMA | NULL | GPL | | INNODB_TRX | ACTIVE | INFORMATION SCHEMA | NULL | GPL | | INNODB_LOCKS | ACTIVE | INFORMATION SCHEMA | NULL | GPL | | INNODB_LOCK_WAITS | ACTIVE | INFORMATION SCHEMA | NULL | GPL | | INNODB_CMP | ACTIVE | INFORMATION SCHEMA | NULL | GPL | | INNODB_CMP_RESET | ACTIVE | INFORMATION SCHEMA | NULL | GPL | | INNODB_CMPMEM | ACTIVE | INFORMATION SCHEMA | NULL | GPL | | INNODB_CMPMEM_RESET | ACTIVE | INFORMATION SCHEMA | NULL | GPL | | INNODB_CMP_PER_INDEX | ACTIVE | INFORMATION SCHEMA | NULL | GPL | | INNODB_CMP_PER_INDEX_RESET | ACTIVE | INFORMATION SCHEMA | NULL | GPL | | INNODB_BUFFER_PAGE | ACTIVE | INFORMATION SCHEMA | NULL | GPL | | INNODB_BUFFER_PAGE_LRU | ACTIVE | INFORMATION SCHEMA | NULL | GPL | | INNODB_BUFFER_POOL_STATS | ACTIVE | INFORMATION SCHEMA | NULL | GPL | | INNODB_METRICS | ACTIVE | INFORMATION SCHEMA | NULL | GPL | | INNODB_FT_DEFAULT_STOPWORD | ACTIVE | INFORMATION SCHEMA | NULL | GPL | | INNODB_FT_DELETED | ACTIVE | INFORMATION SCHEMA | NULL | GPL | | INNODB_FT_BEING_DELETED | ACTIVE | INFORMATION SCHEMA | NULL | GPL | | INNODB_FT_CONFIG | ACTIVE | INFORMATION SCHEMA | NULL | GPL | | INNODB_FT_INDEX_CACHE | ACTIVE | INFORMATION SCHEMA | NULL | GPL | | INNODB_FT_INDEX_TABLE | ACTIVE | INFORMATION SCHEMA | NULL | GPL | | INNODB_SYS_TABLES | ACTIVE | INFORMATION SCHEMA | NULL | GPL | | INNODB_SYS_TABLESTATS | ACTIVE | INFORMATION SCHEMA | NULL | GPL | | INNODB_SYS_INDEXES | ACTIVE | INFORMATION SCHEMA | NULL | GPL | | INNODB_SYS_COLUMNS | ACTIVE | INFORMATION SCHEMA | NULL | GPL | | INNODB_SYS_FIELDS | ACTIVE | INFORMATION SCHEMA | NULL | GPL | | INNODB_SYS_FOREIGN | ACTIVE | INFORMATION SCHEMA | NULL | GPL | | INNODB_SYS_FOREIGN_COLS | ACTIVE | INFORMATION SCHEMA | NULL | GPL | | INNODB_SYS_TABLESPACES | ACTIVE | INFORMATION SCHEMA | NULL | GPL | | INNODB_SYS_DATAFILES | ACTIVE | INFORMATION SCHEMA | NULL | GPL | | INNODB_CHANGED_PAGES | ACTIVE | INFORMATION SCHEMA | NULL | GPL | | FEDERATED | ACTIVE | STORAGE ENGINE | NULL | GPL | | ARCHIVE | ACTIVE | STORAGE ENGINE | NULL | GPL | | PERFORMANCE_SCHEMA | ACTIVE | STORAGE ENGINE | NULL | GPL | | Aria | ACTIVE | STORAGE ENGINE | NULL | GPL | | FEEDBACK | DISABLED | INFORMATION SCHEMA | NULL | GPL | | partition | ACTIVE | STORAGE ENGINE | NULL | GPL | | SERVER_AUDIT | ACTIVE | AUDIT | server_audit.so | GPL | | METADATA_LOCK_INFO | ACTIVE | INFORMATION SCHEMA | metadata_lock_info.so | GPL | | SQL_ERROR_LOG | ACTIVE | AUDIT | sql_errlog.so | GPL | +-----------------------------+----------+--------------------+-----------------------+---------+ 50 rows in set (0.00 sec Variable_name Value aria_block_size 8192 aria_checkpoint_interval 30 aria_checkpoint_log_activity 1048576 aria_force_start_after_recovery_failures 0 aria_group_commit none aria_group_commit_interval 0 aria_log_file_size 1073741824 aria_log_purge_type immediate aria_max_sort_file_size 9223372036853727232 aria_page_checksum ON aria_pagecache_age_threshold 300 aria_pagecache_buffer_size 134217728 aria_pagecache_division_limit 100 aria_pagecache_file_hash_size 512 aria_recover NORMAL aria_repair_threads 1 aria_sort_buffer_size 268434432 aria_stats_method nulls_unequal aria_sync_log_dir NEWFILE aria_used_for_temp_tables ON auto_increment_increment 1 auto_increment_offset 1 autocommit OFF automatic_sp_privileges ON back_log 150 basedir /engn/XXX/mysql big_tables OFF binlog_annotate_row_events OFF binlog_cache_size 32768 binlog_checksum NONE binlog_commit_wait_count 0 binlog_commit_wait_usec 100000 binlog_direct_non_transactional_updates OFF binlog_format STATEMENT binlog_optimize_thread_scheduling ON binlog_stmt_cache_size 32768 bulk_insert_buffer_size 8388608 character_set_client utf8 character_set_connection utf8 character_set_database utf8 character_set_filesystem binary character_set_results utf8 character_set_server utf8 character_set_system utf8 character_sets_dir /engn/XXX/mariadb-10.0.28-linux-x86_64/share/charsets/ collation_connection utf8_bin collation_database utf8_bin collation_server utf8_bin completion_type NO_CHAIN concurrent_insert AUTO connect_timeout 10 datadir /data/XXX/ date_format %Y-%m-%d datetime_format %Y-%m-%d %H:%i:%s deadlock_search_depth_long 15 deadlock_search_depth_short 4 deadlock_timeout_long 50000000 deadlock_timeout_short 10000 debug_no_thread_alarm OFF default_regex_flags default_storage_engine InnoDB default_week_format 0 delay_key_write ON delayed_insert_limit 100 delayed_insert_timeout 300 delayed_queue_size 1000 div_precision_increment 4 event_scheduler OFF expensive_subquery_limit 100 expire_logs_days 0 extra_max_connections 1 extra_port 0 flush OFF flush_time 0 foreign_key_checks ON ft_boolean_syntax + -><()~*:""&| ft_max_word_len 84 ft_min_word_len 4 ft_query_expansion_limit 20 ft_stopword_file (built-in) general_log OFF general_log_file /logs/XXX/general/XXX_general.log group_concat_max_len 67108864 gtid_binlog_pos gtid_binlog_state gtid_current_pos gtid_domain_id 0 gtid_ignore_duplicates OFF gtid_slave_pos gtid_strict_mode OFF have_compress YES have_crypt YES have_dynamic_loading YES have_geometry YES have_openssl NO have_profiling YES have_query_cache YES have_rtree_keys YES have_ssl DISABLED have_symlink DISABLED histogram_size 0 histogram_type SINGLE_PREC_HB host_cache_size 128 hostname LGEDGMRP01 ignore_builtin_innodb OFF ignore_db_dirs init_connect SET NAMES utf8 init_file init_slave innodb_adaptive_flushing ON innodb_adaptive_flushing_lwm 10.000000 innodb_adaptive_hash_index ON innodb_adaptive_hash_index_partitions 64 innodb_adaptive_max_sleep_delay 150000 innodb_additional_mem_pool_size 8388608 innodb_api_bk_commit_interval 5 innodb_api_disable_rowlock OFF innodb_api_enable_binlog OFF innodb_api_enable_mdl OFF innodb_api_trx_level 0 innodb_autoextend_increment 1000 innodb_autoinc_lock_mode 2 innodb_buffer_pool_dump_at_shutdown ON innodb_buffer_pool_dump_now OFF innodb_buffer_pool_dump_pct 100 innodb_buffer_pool_filename ib_buffer_pool innodb_buffer_pool_instances 8 innodb_buffer_pool_load_abort OFF innodb_buffer_pool_load_at_startup ON innodb_buffer_pool_load_now OFF innodb_buffer_pool_populate OFF innodb_buffer_pool_size 171798691840 innodb_change_buffer_max_size 25 innodb_change_buffering all innodb_checksum_algorithm innodb innodb_checksums ON innodb_cleaner_lsn_age_factor high_checkpoint innodb_cmp_per_index_enabled OFF innodb_commit_concurrency 0 innodb_compression_failure_threshold_pct 5 innodb_compression_level 6 innodb_compression_pad_pct_max 50 innodb_concurrency_tickets 5000 innodb_corrupt_table_action assert innodb_data_file_path ibdata1:2048M;ibdata2:2048M;ibdata3:2048M:autoextend innodb_data_home_dir /data/XXX innodb_disable_sort_file_cache OFF innodb_doublewrite ON innodb_empty_free_list_algorithm backoff innodb_fake_changes OFF innodb_fast_shutdown 1 innodb_file_format Barracuda innodb_file_format_check ON innodb_file_format_max Barracuda innodb_file_per_table ON innodb_flush_log_at_timeout 1 innodb_flush_log_at_trx_commit 2 innodb_flush_method O_DIRECT innodb_flush_neighbors 1 innodb_flushing_avg_loops 30 innodb_force_load_corrupted OFF innodb_force_recovery 0 innodb_foreground_preflush exponential_backoff innodb_ft_aux_table innodb_ft_cache_size 8000000 innodb_ft_enable_diag_print OFF innodb_ft_enable_stopword ON innodb_ft_max_token_size 84 innodb_ft_min_token_size 3 innodb_ft_num_word_optimize 2000 innodb_ft_result_cache_limit 2000000000 innodb_ft_server_stopword_table innodb_ft_sort_pll_degree 2 innodb_ft_total_cache_size 640000000 innodb_ft_user_stopword_table innodb_io_capacity 600 innodb_io_capacity_max 2000 innodb_kill_idle_transaction 0 innodb_large_prefix ON innodb_lock_wait_timeout 3600 innodb_locking_fake_changes ON innodb_locks_unsafe_for_binlog OFF innodb_log_arch_dir /logs/XXX/innodblog innodb_log_arch_expire_sec 0 innodb_log_archive OFF innodb_log_block_size 512 innodb_log_buffer_size 134217728 innodb_log_checksum_algorithm innodb innodb_log_compressed_pages ON innodb_log_file_size 10737418240 innodb_log_files_in_group 3 innodb_log_group_home_dir /logs/XXX/innodblog innodb_lru_scan_depth 1024 innodb_max_bitmap_file_size 104857600 innodb_max_changed_pages 1000000 innodb_max_dirty_pages_pct 75.000000 innodb_max_dirty_pages_pct_lwm 0.001000 innodb_max_purge_lag 0 innodb_max_purge_lag_delay 0 innodb_mirrored_log_groups 1 innodb_monitor_disable innodb_monitor_enable innodb_monitor_reset innodb_monitor_reset_all innodb_old_blocks_pct 37 innodb_old_blocks_time 1000 innodb_online_alter_log_max_size 1342177280 innodb_open_files 50000 innodb_optimize_fulltext_only OFF innodb_page_size 16384 innodb_print_all_deadlocks ON innodb_purge_batch_size 300 innodb_purge_threads 1 innodb_random_read_ahead OFF innodb_read_ahead_threshold 56 innodb_read_io_threads 16 innodb_read_only OFF innodb_replication_delay 0 innodb_rollback_on_timeout OFF innodb_rollback_segments 128 innodb_sched_priority_cleaner 19 innodb_show_locks_held 10 innodb_show_verbose_locks 1 innodb_simulate_comp_failures 0 innodb_sort_buffer_size 1048576 innodb_spin_wait_delay 6 innodb_stats_auto_recalc ON innodb_stats_method nulls_equal innodb_stats_modified_counter 0 innodb_stats_on_metadata OFF innodb_stats_persistent ON innodb_stats_persistent_sample_pages 20 innodb_stats_sample_pages 8 innodb_stats_traditional ON innodb_stats_transient_sample_pages 8 innodb_status_output OFF innodb_status_output_locks OFF innodb_strict_mode ON innodb_support_xa ON innodb_sync_array_size 1 innodb_sync_spin_loops 30 innodb_table_locks ON innodb_thread_concurrency 128 innodb_thread_sleep_delay 0 innodb_tmpdir innodb_track_changed_pages OFF innodb_undo_directory . innodb_undo_logs 128 innodb_undo_tablespaces 0 innodb_use_atomic_writes OFF innodb_use_fallocate OFF innodb_use_global_flush_log_at_trx_commit ON innodb_use_native_aio ON innodb_use_stacktrace OFF innodb_use_sys_malloc ON innodb_version 5.6.32-79.0 innodb_write_io_threads 16 interactive_timeout 28800 join_buffer_size 2097152 join_buffer_space_limit 2097152 join_cache_level 2 keep_files_on_create OFF key_buffer_size 25165824 key_cache_age_threshold 300 key_cache_block_size 1024 key_cache_division_limit 100 key_cache_file_hash_size 512 key_cache_segments 0 large_files_support ON large_page_size 0 large_pages OFF lc_messages en_US lc_messages_dir lc_time_names en_US license GPL local_infile ON lock_wait_timeout 60 locked_in_memory OFF log_bin OFF log_bin_trust_function_creators ON log_error /logs/XXX/error/XXX.err log_output FILE log_queries_not_using_indexes OFF log_slave_updates OFF log_slow_filter admin,filesort,filesort_on_disk,full_join,full_scan,query_cache,query_cache_miss,tmp_table,tmp_table_on_disk log_slow_rate_limit 1 log_slow_verbosity query_plan,explain log_warnings 2 long_query_time 3.000000 low_priority_updates OFF lower_case_file_system OFF lower_case_table_names 1 master_verify_checksum OFF max_allowed_packet 1073741824 max_binlog_cache_size 18446744073709547520 max_binlog_size 1073741824 max_binlog_stmt_cache_size 18446744073709547520 max_connect_errors 1000000 max_connections 2000 max_delayed_threads 20 max_digest_length 1024 max_error_count 64 max_heap_table_size 524288000 max_insert_delayed_threads 20 max_join_size 18446744073709551615 max_length_for_sort_data 1024 max_long_data_size 1073741824 max_prepared_stmt_count 16382 max_relay_log_size 1073741824 max_seeks_for_key 4294967295 max_sort_length 1024 max_sp_recursion_depth 0 max_tmp_tables 32 max_user_connections 0 max_write_lock_count 4294967295 metadata_locks_cache_size 1024 metadata_locks_hash_instances 8 min_examined_row_limit 0 mrr_buffer_size 262144 multi_range_count 256 myisam_block_size 1024 myisam_data_pointer_size 6 myisam_max_sort_file_size 9223372036853727232 myisam_mmap_size 18446744073709551615 myisam_recover_options DEFAULT myisam_repair_threads 1 myisam_sort_buffer_size 134216704 myisam_stats_method nulls_unequal myisam_use_mmap OFF net_buffer_length 16384 net_read_timeout 300 net_retry_count 10 net_write_timeout 600 old OFF old_alter_table OFF old_mode old_passwords OFF open_files_limit 1048576 optimizer_prune_level 1 optimizer_search_depth 62 optimizer_selectivity_sampling_limit 100 optimizer_switch index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=off,table_elimination=on,extended_keys=on,exists_to_in=on optimizer_use_condition_selectivity 1 performance_schema ON performance_schema_accounts_size 100 performance_schema_digests_size 10000 performance_schema_events_stages_history_long_size 10000 performance_schema_events_stages_history_size 10 performance_schema_events_statements_history_long_size 10000 performance_schema_events_statements_history_size 10 performance_schema_events_waits_history_long_size 10000 performance_schema_events_waits_history_size 10 performance_schema_hosts_size 100 performance_schema_max_cond_classes 80 performance_schema_max_cond_instances 8900 performance_schema_max_digest_length 1024 performance_schema_max_file_classes 50 performance_schema_max_file_handles 32768 performance_schema_max_file_instances 2154 performance_schema_max_mutex_classes 200 performance_schema_max_mutex_instances 17000 performance_schema_max_rwlock_classes 40 performance_schema_max_rwlock_instances 6800 performance_schema_max_socket_classes 10 performance_schema_max_socket_instances 4020 performance_schema_max_stage_classes 150 performance_schema_max_statement_classes 180 performance_schema_max_table_handles 800 performance_schema_max_table_instances 12500 performance_schema_max_thread_classes 50 performance_schema_max_thread_instances 4100 performance_schema_session_connect_attrs_size 512 performance_schema_setup_actors_size 100 performance_schema_setup_objects_size 100 performance_schema_users_size 100 pid_file /data/XXX/LGEDGMRP01.pid plugin_dir /engn/XXX/mysql/lib/plugin/ plugin_maturity unknown port 3310 preload_buffer_size 32768 profiling OFF profiling_history_size 15 progress_report_time 5 protocol_version 10 query_alloc_block_size 8192 query_cache_limit 1048576 query_cache_min_res_unit 4096 query_cache_size 0 query_cache_strip_comments OFF query_cache_type OFF query_cache_wlock_invalidate OFF query_prealloc_size 8192 range_alloc_block_size 4096 read_buffer_size 2097152 read_only OFF read_rnd_buffer_size 2097152 relay_log relay_log_index relay_log_info_file relay-log.info relay_log_purge ON relay_log_recovery OFF relay_log_space_limit 0 replicate_annotate_row_events OFF replicate_do_db replicate_do_table replicate_events_marked_for_skip replicate replicate_ignore_db replicate_ignore_table replicate_wild_do_table replicate_wild_ignore_table report_host report_password report_port 3310 report_user rowid_merge_buff_size 8388608 rpl_recovery_rank 0 secure_auth OFF secure_file_priv server_audit_events QUERY_DDL server_audit_excl_users server_audit_file_path server_audit.log server_audit_file_rotate_now OFF server_audit_file_rotate_size 1000000 server_audit_file_rotations 9 server_audit_incl_users server_audit_logging ON server_audit_mode 0 server_audit_output_type file server_audit_query_log_limit 1024 server_audit_syslog_facility LOG_USER server_audit_syslog_ident mysql-server_auditing server_audit_syslog_info server_audit_syslog_priority LOG_INFO server_id 0 skip_external_locking ON skip_name_resolve ON skip_networking OFF skip_show_database OFF slave_compressed_protocol OFF slave_ddl_exec_mode IDEMPOTENT slave_domain_parallel_threads 0 slave_exec_mode STRICT slave_load_tmpdir /data/XXX slave_max_allowed_packet 1073741824 slave_net_timeout 3600 slave_parallel_max_queued 131072 slave_parallel_threads 0 slave_skip_errors OFF slave_sql_verify_checksum ON slave_transaction_retries 10 slave_type_conversions slow_launch_time 2 slow_query_log OFF slow_query_log_file /logs/XXX/slow/XXX_slow.log socket /engn/XXX/mysql/mysql.sock sort_buffer_size 2097152 sql_auto_is_null OFF sql_big_selects ON sql_buffer_result OFF sql_error_log_filename sql_errors.log sql_error_log_rate 1 sql_error_log_rotate OFF sql_error_log_rotations 9 sql_error_log_size_limit 1000000 sql_log_bin ON sql_log_off OFF sql_mode PIPES_AS_CONCAT,ANSI_QUOTES,IGNORE_SPACE,ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION sql_notes ON sql_quote_show_create ON sql_safe_updates OFF sql_select_limit 18446744073709551615 sql_slave_skip_counter 0 sql_warnings OFF ssl_ca ssl_capath ssl_cert ssl_cipher ssl_crl ssl_crlpath ssl_key storage_engine InnoDB stored_program_cache 256 sync_binlog 0 sync_frm ON sync_master_info 0 sync_relay_log 0 sync_relay_log_info 0 system_time_zone KST table_definition_cache 50000 table_open_cache 50000 thread_cache_size 100 thread_concurrency 10 thread_handling one-thread-per-connection thread_pool_idle_timeout 60 thread_pool_max_threads 500 thread_pool_oversubscribe 3 thread_pool_size 56 thread_pool_stall_limit 500 thread_stack 294912 time_format %H:%i:%s time_zone SYSTEM timed_mutexes OFF tmp_table_size 524288000 tmpdir /data/XXX transaction_alloc_block_size 8192 transaction_prealloc_size 4096 tx_isolation READ-COMMITTED tx_read_only OFF unique_checks ON updatable_views_with_limit YES use_stat_tables NEVER userstat OFF version 10.0.28-MariaDB version_comment MariaDB Server version_compile_machine x86_64 version_compile_os Linux version_malloc_library bundled jemalloc wait_timeout 28800

Elena Stepanova added a comment - 2018-07-18 12:51 2. I will share the contents except the logging data for the repeated actions after the restart point. Do you mean that you will enable the coredump and provide the information from it? If so, thanks, we'll be waiting for it.

ssauravy added a comment - 2018-07-19 00:22 no. This is happening to us on a very important production server, and core dump can not be activated. Just add a log directly below the above log (from the point of restart). 5564399 180714 19:25:33 mysqld_safe Number of processes running now: 0 5564400 180714 19:25:33 mysqld_safe mysqld restarted 5636219 InnoDB: about forcing recovery. 5636220 InnoDB: Error in pages 12862415 and 12862421 of index "XMVA_ITEM_REQ_DETAIL_N3" of table "gmrpp"."xmva_item_req_detail" 5636221 InnoDB: broken FIL_PAGE_NEXT or FIL_PAGE_PREV links 5636222 2018-07-15 09:10:57 7fac601fc700 InnoDB: Page dump in ascii and hex (16384 bytes): 5582759 2018-07-14 19:48:20 7f7f56ded700 InnoDB: uncompressed page, stored checksum in field1 17170432, calculated checksums for field1: crc32 2753499458, innodb 391883286, none 3735928559, s tored checksum in field2 3590652932, calculated checksums for field2: crc32 2753499458, innodb 653250342, none 3735928559, page LSN 0 4352, low 4 bytes of LSN at page end 839016448, p age number (if stored to page already) 131072, space id (if created with >= MySQL-4.1.1 and stored already) 2650806784 5582760 InnoDB: Page may be a freshly allocated page 5582761 InnoDB: Database page corruption on disk or a failed 5582762 InnoDB: file read of page 12862421. 5582763 InnoDB: You may have to recover from a backup. 5582764 InnoDB: It is also possible that your operating 5582765 InnoDB: system has corrupted its own file cache 5582766 InnoDB: and rebooting your computer removes the 5582767 InnoDB: error. 5582768 InnoDB: If the corrupt page is an index page 5582769 InnoDB: you can also try to fix the corruption 5582770 InnoDB: by dumping, dropping, and reimporting 5582771 InnoDB: the corrupt table. You can use CHECK 5582772 InnoDB: TABLE to scan your table for corruption. 5582773 InnoDB: See also http://dev.mysql.com/doc/refman/5.6/en/forcing-innodb-recovery.html 5582774 InnoDB: about forcing recovery. 5582775 180714 19:48:23 [ERROR] mysqld: Table './gmrpp/jjcr_request_next_id' is marked as crashed and should be repaired 5582776 180714 19:48:23 [ERROR] mysqld: Table 'jjcr_request_next_id' is marked as crashed and should be repaired 5582777 180714 19:48:23 [Warning] Checking table: './gmrpp/jjcr_request_next_id' 5582778 180714 19:48:23 [ERROR] mysqld: Table './gmrpp/jjrs_instance_next_id' is marked as crashed and should be repaired 5582779 180714 19:48:23 [ERROR] mysqld: Table 'jjrs_instance_next_id' is marked as crashed and should be repaired 5582780 180714 19:48:23 [Warning] Checking table: './gmrpp/jjrs_instance_next_id' 5582781 180714 19:48:23 [ERROR] mysqld: Table './gmrpp/jjrs_job_next_id' is marked as crashed and should be repaired 5582782 180714 19:48:23 [ERROR] mysqld: Table 'jjrs_job_next_id' is marked as crashed and should be repaired 5582783 180714 19:48:23 [Warning] Checking table: './gmrpp/jjrs_job_next_id' 5582784 180714 19:48:23 [ERROR] mysqld: Table './gmrpp/jjrs_step_next_id' is marked as crashed and should be repaired 5582785 180714 19:48:23 [ERROR] mysqld: Table 'jjrs_step_next_id' is marked as crashed and should be repaired 5582786 180714 19:48:23 [Warning] Checking table: './gmrpp/jjrs_step_next_id' 5582787 180714 19:48:30 [ERROR] mysqld: Table './gmrpp/batch_job_seq' is marked as crashed and should be repaired 5582788 180714 19:48:30 [ERROR] mysqld: Table 'batch_job_seq' is marked as crashed and should be repaired 5582789 180714 19:48:30 [Warning] Checking table: './gmrpp/batch_job_seq' 5582790 180714 19:48:30 [ERROR] mysqld: Table './gmrpp/batch_job_execution_seq' is marked as crashed and should be repaired 5582791 180714 19:48:30 [ERROR] mysqld: Table 'batch_job_execution_seq' is marked as crashed and should be repaired 5582792 180714 19:48:30 [Warning] Checking table: './gmrpp/batch_job_execution_seq' 5582793 180714 19:48:30 [ERROR] mysqld: Table './gmrpp/batch_step_execution_seq' is marked as crashed and should be repaired 5582794 180714 19:48:30 [ERROR] mysqld: Table 'batch_step_execution_seq' is marked as crashed and should be repaired 5582795 180714 19:48:30 [Warning] Checking table: './gmrpp/batch_step_execution_seq' 5582796 2018-07-14 19:48:51 7f7f4c3dc700 InnoDB: table "gmrpp"."xmva_item_req_detail"is corrupted. Please drop the table and recreate 5582797 2018-07-14 19:48:51 7fa4bd46e700 InnoDB: table 180714 19:48:51 [Warning] InnoDB: Cannot open table gmrpp/xmva_item_req_detail from the internal data dictionary of InnoDB though the . frm file for the table exists. See http://dev.mysql.com/doc/refman/5.6/en/innodb-troubleshooting.html for how you can resolve the problem. 5582798 2018-07-14 19:48:51 7f7f4c194700 InnoDB: table "gmrpp"."xmva_item_req_detail"is corrupted. Please drop the table and recreate 5582799 "gmrpp"."xmva_item_req_detail"is corrupted. Please drop the table and recreate 5582800 180714 19:48:51 [Warning] InnoDB: Cannot open table gmrpp/xmva_item_req_detail from the internal data dictionary of InnoDB though the .frm file for the table exists. See http://dev.my sql.com/doc/refman/5.6/en/innodb-troubleshooting.html for how you can resolve the problem. 5582801 180714 19:48:51 [Warning] InnoDB: Cannot open table gmrpp/xmva_item_req_detail from the internal data dictionary of InnoDB though the .frm file for the table exists. See http://dev.my sql.com/doc/refman/5.6/en/innodb-troubleshooting.html for how you can resolve the problem. 5582802 180714 19:48:52 [Warning] Aborted connection 1091 to db: 'gmrpp' user: 'gmrp_app' host: '10.185.50.12' (Got an error reading communication packets) 5582803 180714 19:48:52 [Warning] Aborted connection 1088 to db: 'gmrpp' user: 'gmrp_app' host: '10.185.50.12' (Got an error reading communication packets) 5582804 180714 19:48:52 [Warning] Aborted connection 1089 to db: 'gmrpp' user: 'gmrp_app' host: '10.185.50.12' (Got an error reading communication packets) 5582805 180714 19:49:02 [Warning] Aborted connection 1375 to db: 'gmrpp' user: 'gmrp_inf' host: '10.185.135.144' (Got an error reading communication packets) 5582806 180714 19:49:02 [Warning] Aborted connection 1368 to db: 'gmrpp' user: 'gmrp_inf' host: '10.185.135.144' (Got an error reading communication packets) 5582807 180714 19:49:02 [Warning] Aborted connection 1376 to db: 'gmrpp' user: 'gmrp_inf' host: '10.185.135.144' (Got an error reading communication packets) 5582808 2018-07-14 19:49:10 7f7ea8cda700 InnoDB: table "gmrpp"."xmva_item_req_detail"is corrupted. Please drop the table and recreate 5582809 180714 19:49:10 [Warning] InnoDB: Cannot open table gmrpp/xmva_item_req_detail from the internal data dictionary of InnoDB though the .frm file for the table exists. See http://dev.my sql.com/doc/refman/5.6/en/innodb-troubleshooting.html for how you can resolve the problem. 5582810 2018-07-14 19:49:10 7fa4bd549700 InnoDB: table "gmrpp"."xmva_item_req_detail"is corrupted. Please drop the table and recreate 5582811 180714 19:49:10 [Warning] InnoDB: Cannot open table gmrpp/xmva_item_req_detail from the internal data dictionary of InnoDB though the .frm file for the table exists. See http://dev.my sql.com/doc/refman/5.6/en/innodb-troubleshooting.html for how you can resolve the problem. 5582812 2018-07-14 19:49:10 7f7f4c26f700 InnoDB: table "gmrpp"."xmva_item_req_detail"is corrupted. Please drop the table and recreate 5582813 180714 19:49:10 [Warning] InnoDB: Cannot open table gmrpp/xmva_item_req_detail from the internal data dictionary of InnoDB though the .frm file for the table exists. See http://dev.my sql.com/doc/refman/5.6/en/innodb-troubleshooting.html for how you can resolve the problem. 5582814 180714 19:49:10 [Warning] Aborted connection 1593 to db: 'gmrpp' user: 'gmrp_app' host: '10.185.50.12' (Got an error reading communication packets) 5582815 180714 19:49:10 [Warning] Aborted connection 1610 to db: 'gmrpp' user: 'gmrp_app' host: '10.185.50.12' (Got an error reading communication packets) 5582816 180714 19:49:10 [Warning] Aborted connection 1607 to db: 'gmrpp' user: 'gmrp_app' host: '10.185.50.12' (Got an error reading communication packets) 5582817 180714 19:50:03 [Warning] Aborted connection 3528 to db: 'gmrpp' user: 'gmrp_inf' host: '10.185.135.144' (Got an error reading communication packets) 5582818 180714 19:50:03 [Warning] Aborted connection 3500 to db: 'gmrpp' user: 'gmrp_inf' host: '10.185.135.144' (Got an error reading communication packets) 5582819 180714 19:50:03 [Warning] Aborted connection 3518 to db: 'gmrpp' user: 'gmrp_inf' host: '10.185.135.144' (Got an error reading communication packets) 5582820 180714 19:50:03 [Warning] Aborted connection 3498 to db: 'gmrpp' user: 'gmrp_inf' host: '10.185.135.144' (Got an error reading communication packets) 5582821 180714 19:50:03 [Warning] Aborted connection 3517 to db: 'gmrpp' user: 'gmrp_inf' host: '10.185.135.144' (Got an error reading communication packets) 5582822 2018-07-14 19:50:25 7f7f56ded700 InnoDB: Error: space id and page n:o stored in the page 5582823 InnoDB: read in are 2650806784:131072, should be 61037:12862421! 5582824 InnoDB: Database page corruption on disk or a failed 5582825 InnoDB: file read of page 12862421. 5582826 InnoDB: You may have to recover from a backup. 5582827 2018-07-14 19:50:25 7f7f56ded700 InnoDB: Page dump in ascii and hex (16384 bytes): no. This is happening to us on a very important production server, and core dump can not be activated. Just add a log directly below the above log (from the point of restart). 5636230 2018-07-15 09:10:57 7fac601fc700 InnoDB: uncompressed page, stored checksum in field1 17170432, calculated checksums for field1: crc32 2753499458, innodb 391883286, none 3735928559, s tored checksum in field2 3590652932, calculated checksums for field2: crc32 2753499458, innodb 653250342, none 3735928559, page LSN 0 4352, low 4 bytes of LSN at page end 839016448, p age number (if stored to page already) 131072, space id (if created with >= MySQL-4.1.1 and stored already) 2650806784 5636231 InnoDB: Page may be a freshly allocated page 5636232 InnoDB: Error in pages 12862415 and 12862421 of index "XMVA_ITEM_REQ_DETAIL_N3" of table "gmrpp"."xmva_item_req_detail" 5636233 InnoDB: 'compact' flag mismatch 5636234 2018-07-15 09:10:57 7fac601fc700 InnoDB: Page dump in ascii and hex (16384 bytes):

ssauravy added a comment - 2018-07-24 03:53 Do you have any additional checks? We are in the operating system, so we have a lot of anxieties about how to follow up in the future. Thank you.

ossk_db added a comment - 2018-07-26 01:21 cpu.txt disk.txt engine.txt gmrpe_table_size.txt gmrpp_table_size.txt memory.txt plugin.txt sts.txt var.txt

Elena Stepanova added a comment - 2018-08-04 20:41 Back to your original question about the cause of signal 6, InnoDB itself rather verbosely explains it in the error log: it's been trying to read a page, fails to do so, apparently due to data corruption, and after a number of attempts gives up. Assigning to marko for further advice/ideas/etc.

ossk_db added a comment - 2018-10-01 02:53 Hi~ Team. 1. Do you know if this text has been patched in the latest version? 2. Is the obvious data corruption a problem with the disk HW?

Marko Mäkelä added a comment - 2018-10-12 09:37 Hi ossk_db , my simplest explanation would be to blame the hardware. I did not see any messages about the page LSN being in the future. Such messages could be displayed when something wrong was done, such as killing the server, deleting ib_logfile* and restarting. That said, there is some room for improvement for InnoDB could in this area: MDEV-13542 should propagate read errors up to the client (so that the user would know which table or SQL statement is in question), and the InnoDB doublewrite buffer could be improved as well ( MDEV-11799 , MDEV-12905 ). The doublewrite buffer only matters if the mysqld process was killed (or the whole system was killed) while InnoDB was in the middle of writing a data page.

Marko Mäkelä added a comment - 2018-10-12 09:41 One thing that could be useful is to pay attention to the diagnostics provided by the storage layer, to get an early warning of upcoming failures. Modern hard disks and SSDs provide the S.M.A.R.T. interface. On GNU/Linux, there is the smartmontools package. If I were running an important server, I would try adding some diagnostics around smartctl -A /dev/sda or similar.