[MDEV-16763] SIGNAL 6 ERROR Created: 2018-07-16  Updated: 2021-03-28  Resolved: 2021-03-28

Status: Closed
Project: MariaDB Server
Component/s: Storage Engine - InnoDB
Affects Version/s: 10.0.28
Fix Version/s: N/A

Type: Bug Priority: Major
Reporter: ssauravy Assignee: Marko Mäkelä
Resolution: Incomplete Votes: 1
Labels: None
Environment:

CentOS 6.9


Attachments: Text File cpu.txt     Text File disk.txt     Text File engine.txt     Text File gmrpe_table_size.txt     Text File gmrpp_table_size.txt     Text File memory.txt     Text File plugin.txt     Text File sts.txt     Text File var.txt    
Issue Links:
Relates
relates to MDEV-11799 InnoDB can abort if the doublewrite b... Closed
relates to MDEV-12905 InnoDB occasionally skips the doublew... Closed
relates to MDEV-13542 Crashing on a corrupted page is unhel... Closed

 Description   

10.0.28

InnoDB: Page may be a freshly allocated page
InnoDB: Database page corruption on disk or a failed
InnoDB: file read of page 12862421.
InnoDB: You may have to recover from a backup.
InnoDB: It is also possible that your operating
InnoDB: system has corrupted its own file cache
InnoDB: and rebooting your computer removes the
InnoDB: error.
InnoDB: If the corrupt page is an index page
InnoDB: you can also try to fix the corruption
InnoDB: by dumping, dropping, and reimporting
InnoDB: the corrupt table. You can use CHECK
InnoDB: TABLE to scan your table for corruption.
InnoDB: See also http://dev.mysql.com/doc/refman/5.6/en/forcing-innodb-recovery.html
InnoDB: about forcing recovery.
InnoDB: Error: Unable to read tablespace 61037 page no 12862421 into the buffer pool after 100 attempts
InnoDB: The most probable cause of this error may be that the table has been corrupted.
InnoDB: You can try to fix this problem by using innodb_force_recovery.
InnoDB: Please see reference manual for more details.
InnoDB: Aborting...
2018-07-14 19:25:31 7edabadfa700  InnoDB: Assertion failure in thread 139477903189760 in file buf0buf.cc line 2895
InnoDB: We intentionally generate a memory trap.
InnoDB: Submit a detailed bug report to http://bugs.mysql.com.
InnoDB: If you get repeated assertion failures or crashes, even
InnoDB: immediately after the mysqld startup, there may be
InnoDB: corruption in the InnoDB tablespace. Please refer to
InnoDB: http://dev.mysql.com/doc/refman/5.6/en/forcing-innodb-recovery.html
InnoDB: about forcing recovery.
180714 19:25:31 [ERROR] mysqld got signal 6 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
 
To report this bug, see https://mariadb.com/kb/en/reporting-bugs
 
We will try our best to scrape up some info that will hopefully help
diagnose the problem, but since we have already crashed,
something is definitely wrong and this may fail.
 
Server version: 10.0.28-MariaDB
key_buffer_size=25165824
read_buffer_size=2097152
max_used_connections=1911
max_threads=2002
thread_count=101
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 8264428 K  bytes of memory
Hope that's ok; if not, decrease some variables in the equation.
 
Thread pointer: 0x0x0
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 0x0 thread_stack 0x48000
(my_addr_resolve failure: fork)
/engn001/masvc01/GMRPP/mysql/bin/mysqld(my_print_stacktrace+0x2e) [0xbda1ae]
/engn001/masvc01/GMRPP/mysql/bin/mysqld(handle_fatal_signal+0x49a) [0x722a9a]
/lib64/libpthread.so.0() [0x3063c0f790]
/lib64/libc.so.6(gsignal+0x35) [0x3063832625]
/lib64/libc.so.6(abort+0x175) [0x3063833e05]
/engn001/masvc01/GMRPP/mysql/bin/mysqld() [0x9d135f]
/engn001/masvc01/GMRPP/mysql/bin/mysqld() [0x9b498e]
/engn001/masvc01/GMRPP/mysql/bin/mysqld() [0x9b69b7]
/engn001/masvc01/GMRPP/mysql/bin/mysqld() [0x9590ea]
/engn001/masvc01/GMRPP/mysql/bin/mysqld() [0x9572f3]
/engn001/masvc01/GMRPP/mysql/bin/mysqld() [0x958c93]
/engn001/masvc01/GMRPP/mysql/bin/mysqld() [0x925ef8]
/engn001/masvc01/GMRPP/mysql/bin/mysqld() [0x982228]
/engn001/masvc01/GMRPP/mysql/bin/mysqld() [0x96d40e]
/engn001/masvc01/GMRPP/mysql/bin/mysqld() [0x9733e0]
/lib64/libpthread.so.0() [0x3063c07a51]
/lib64/libc.so.6(clone+0x6d) [0x30638e896d]
The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
information that should help you find out what is causing the crash.
 
...
 
180714 19:32:48 server_audit: MariaDB Audit Plugin version 1.4.0 STARTED.
180714 19:32:48 server_audit: logging started to the file server_audit.log.
180714 19:32:48 [Note] Server socket created on IP: '::'.
180714 19:32:48 [ERROR] mysqld: Table './mysql/user' is marked as crashed and should be repaired
180714 19:32:48 [ERROR] mysqld: Table 'user' is marked as crashed and should be repaired
180714 19:32:48 [Warning] Checking table:   './mysql/user'
180714 19:32:48 [ERROR] mysql.user: 1 client is using or hasn't closed the table properly
180714 19:32:48 [ERROR] mysqld: Table './mysql/db' is marked as crashed and should be repaired
180714 19:32:48 [ERROR] mysqld: Table 'db' is marked as crashed and should be repaired
180714 19:32:48 [Warning] Checking table:   './mysql/db'
180714 19:32:48 [ERROR] mysql.db: 1 client is using or hasn't closed the table properly
180714 19:32:48 [ERROR] mysqld: Table './mysql/tables_priv' is marked as crashed and should be repaired
180714 19:32:48 [ERROR] mysqld: Table 'tables_priv' is marked as crashed and should be repaired
180714 19:32:48 [Warning] Checking table:   './mysql/tables_priv'
180714 19:32:48 [ERROR] mysql.tables_priv: 1 client is using or hasn't closed the table properly
180714 19:32:48 [ERROR] mysqld: Table './mysql/procs_priv' is marked as crashed and should be repaired

We have confirmed that the db is restarting indefinitely after signal 6 in the error log.
Can you tell us the cause of signal 6?



 Comments   
Comment by Elena Stepanova [ 2018-07-16 ]

Prior to the first occurrence of the problem, did you have any incidents which could cause actual data corruption, disk failures or alike?
If it keeps happening all the time, could you please enable the coredump and get the stack trace?
Should it be needed, would you be willing to provide access to the crashing instance to one of our developers?

Comment by ssauravy [ 2018-07-18 ]

Good morning.
1. No problems such as data corruption or disk error were found before or after the failure.
     / var / log / dmsg, and / var / log / messages.

2. I will share the contents except the logging data for the repeated actions after the restart point.

3. Please understand that the system is closed network and external connection is not possible.

4. I do not know if DB variables, plugins information will help.

MariaDB [(none)]> show plugins;
+-----------------------------+----------+--------------------+-----------------------+---------+
| Name                        | Status   | Type               | Library               | License |
+-----------------------------+----------+--------------------+-----------------------+---------+
| binlog                      | ACTIVE   | STORAGE ENGINE     | NULL                  | GPL     |
| mysql_native_password       | ACTIVE   | AUTHENTICATION     | NULL                  | GPL     |
| mysql_old_password          | ACTIVE   | AUTHENTICATION     | NULL                  | GPL     |
| CSV                         | ACTIVE   | STORAGE ENGINE     | NULL                  | GPL     |
| MEMORY                      | ACTIVE   | STORAGE ENGINE     | NULL                  | GPL     |
| MyISAM                      | ACTIVE   | STORAGE ENGINE     | NULL                  | GPL     |
| MRG_MyISAM                  | ACTIVE   | STORAGE ENGINE     | NULL                  | GPL     |
| BLACKHOLE                   | ACTIVE   | STORAGE ENGINE     | NULL                  | GPL     |
| InnoDB                      | ACTIVE   | STORAGE ENGINE     | NULL                  | GPL     |
| XTRADB_READ_VIEW            | ACTIVE   | INFORMATION SCHEMA | NULL                  | GPL     |
| XTRADB_INTERNAL_HASH_TABLES | ACTIVE   | INFORMATION SCHEMA | NULL                  | GPL     |
| XTRADB_RSEG                 | ACTIVE   | INFORMATION SCHEMA | NULL                  | GPL     |
| INNODB_TRX                  | ACTIVE   | INFORMATION SCHEMA | NULL                  | GPL     |
| INNODB_LOCKS                | ACTIVE   | INFORMATION SCHEMA | NULL                  | GPL     |
| INNODB_LOCK_WAITS           | ACTIVE   | INFORMATION SCHEMA | NULL                  | GPL     |
| INNODB_CMP                  | ACTIVE   | INFORMATION SCHEMA | NULL                  | GPL     |
| INNODB_CMP_RESET            | ACTIVE   | INFORMATION SCHEMA | NULL                  | GPL     |
| INNODB_CMPMEM               | ACTIVE   | INFORMATION SCHEMA | NULL                  | GPL     |
| INNODB_CMPMEM_RESET         | ACTIVE   | INFORMATION SCHEMA | NULL                  | GPL     |
| INNODB_CMP_PER_INDEX        | ACTIVE   | INFORMATION SCHEMA | NULL                  | GPL     |
| INNODB_CMP_PER_INDEX_RESET  | ACTIVE   | INFORMATION SCHEMA | NULL                  | GPL     |
| INNODB_BUFFER_PAGE          | ACTIVE   | INFORMATION SCHEMA | NULL                  | GPL     |
| INNODB_BUFFER_PAGE_LRU      | ACTIVE   | INFORMATION SCHEMA | NULL                  | GPL     |
| INNODB_BUFFER_POOL_STATS    | ACTIVE   | INFORMATION SCHEMA | NULL                  | GPL     |
| INNODB_METRICS              | ACTIVE   | INFORMATION SCHEMA | NULL                  | GPL     |
| INNODB_FT_DEFAULT_STOPWORD  | ACTIVE   | INFORMATION SCHEMA | NULL                  | GPL     |
| INNODB_FT_DELETED           | ACTIVE   | INFORMATION SCHEMA | NULL                  | GPL     |
| INNODB_FT_BEING_DELETED     | ACTIVE   | INFORMATION SCHEMA | NULL                  | GPL     |
| INNODB_FT_CONFIG            | ACTIVE   | INFORMATION SCHEMA | NULL                  | GPL     |
| INNODB_FT_INDEX_CACHE       | ACTIVE   | INFORMATION SCHEMA | NULL                  | GPL     |
| INNODB_FT_INDEX_TABLE       | ACTIVE   | INFORMATION SCHEMA | NULL                  | GPL     |
| INNODB_SYS_TABLES           | ACTIVE   | INFORMATION SCHEMA | NULL                  | GPL     |
| INNODB_SYS_TABLESTATS       | ACTIVE   | INFORMATION SCHEMA | NULL                  | GPL     |
| INNODB_SYS_INDEXES          | ACTIVE   | INFORMATION SCHEMA | NULL                  | GPL     |
| INNODB_SYS_COLUMNS          | ACTIVE   | INFORMATION SCHEMA | NULL                  | GPL     |
| INNODB_SYS_FIELDS           | ACTIVE   | INFORMATION SCHEMA | NULL                  | GPL     |
| INNODB_SYS_FOREIGN          | ACTIVE   | INFORMATION SCHEMA | NULL                  | GPL     |
| INNODB_SYS_FOREIGN_COLS     | ACTIVE   | INFORMATION SCHEMA | NULL                  | GPL     |
| INNODB_SYS_TABLESPACES      | ACTIVE   | INFORMATION SCHEMA | NULL                  | GPL     |
| INNODB_SYS_DATAFILES        | ACTIVE   | INFORMATION SCHEMA | NULL                  | GPL     |
| INNODB_CHANGED_PAGES        | ACTIVE   | INFORMATION SCHEMA | NULL                  | GPL     |
| FEDERATED                   | ACTIVE   | STORAGE ENGINE     | NULL                  | GPL     |
| ARCHIVE                     | ACTIVE   | STORAGE ENGINE     | NULL                  | GPL     |
| PERFORMANCE_SCHEMA          | ACTIVE   | STORAGE ENGINE     | NULL                  | GPL     |
| Aria                        | ACTIVE   | STORAGE ENGINE     | NULL                  | GPL     |
| FEEDBACK                    | DISABLED | INFORMATION SCHEMA | NULL                  | GPL     |
| partition                   | ACTIVE   | STORAGE ENGINE     | NULL                  | GPL     |
| SERVER_AUDIT                | ACTIVE   | AUDIT              | server_audit.so       | GPL     |
| METADATA_LOCK_INFO          | ACTIVE   | INFORMATION SCHEMA | metadata_lock_info.so | GPL     |
| SQL_ERROR_LOG               | ACTIVE   | AUDIT              | sql_errlog.so         | GPL     |
+-----------------------------+----------+--------------------+-----------------------+---------+
50 rows in set (0.00 sec

Variable_name	Value
aria_block_size	8192
aria_checkpoint_interval	30
aria_checkpoint_log_activity	1048576
aria_force_start_after_recovery_failures	0
aria_group_commit	none
aria_group_commit_interval	0
aria_log_file_size	1073741824
aria_log_purge_type	immediate
aria_max_sort_file_size	9223372036853727232
aria_page_checksum	ON
aria_pagecache_age_threshold	300
aria_pagecache_buffer_size	134217728
aria_pagecache_division_limit	100
aria_pagecache_file_hash_size	512
aria_recover	NORMAL
aria_repair_threads	1
aria_sort_buffer_size	268434432
aria_stats_method	nulls_unequal
aria_sync_log_dir	NEWFILE
aria_used_for_temp_tables	ON
auto_increment_increment	1
auto_increment_offset	1
autocommit	OFF
automatic_sp_privileges	ON
back_log	150
basedir	/engn/XXX/mysql
big_tables	OFF
binlog_annotate_row_events	OFF
binlog_cache_size	32768
binlog_checksum	NONE
binlog_commit_wait_count	0
binlog_commit_wait_usec	100000
binlog_direct_non_transactional_updates	OFF
binlog_format	STATEMENT
binlog_optimize_thread_scheduling	ON
binlog_stmt_cache_size	32768
bulk_insert_buffer_size	8388608
character_set_client	utf8
character_set_connection	utf8
character_set_database	utf8
character_set_filesystem	binary
character_set_results	utf8
character_set_server	utf8
character_set_system	utf8
character_sets_dir	/engn/XXX/mariadb-10.0.28-linux-x86_64/share/charsets/
collation_connection	utf8_bin
collation_database	utf8_bin
collation_server	utf8_bin
completion_type	NO_CHAIN
concurrent_insert	AUTO
connect_timeout	10
datadir	/data/XXX/
date_format	%Y-%m-%d
datetime_format	%Y-%m-%d %H:%i:%s
deadlock_search_depth_long	15
deadlock_search_depth_short	4
deadlock_timeout_long	50000000
deadlock_timeout_short	10000
debug_no_thread_alarm	OFF
default_regex_flags	
default_storage_engine	InnoDB
default_week_format	0
delay_key_write	ON
delayed_insert_limit	100
delayed_insert_timeout	300
delayed_queue_size	1000
div_precision_increment	4
event_scheduler	OFF
expensive_subquery_limit	100
expire_logs_days	0
extra_max_connections	1
extra_port	0
flush	OFF
flush_time	0
foreign_key_checks	ON
ft_boolean_syntax	+ -><()~*:""&|
ft_max_word_len	84
ft_min_word_len	4
ft_query_expansion_limit	20
ft_stopword_file	(built-in)
general_log	OFF
general_log_file	/logs/XXX/general/XXX_general.log
group_concat_max_len	67108864
gtid_binlog_pos	
gtid_binlog_state	
gtid_current_pos	
gtid_domain_id	0
gtid_ignore_duplicates	OFF
gtid_slave_pos	
gtid_strict_mode	OFF
have_compress	YES
have_crypt	YES
have_dynamic_loading	YES
have_geometry	YES
have_openssl	NO
have_profiling	YES
have_query_cache	YES
have_rtree_keys	YES
have_ssl	DISABLED
have_symlink	DISABLED
histogram_size	0
histogram_type	SINGLE_PREC_HB
host_cache_size	128
hostname	LGEDGMRP01
ignore_builtin_innodb	OFF
ignore_db_dirs	
init_connect	SET NAMES utf8
init_file	
init_slave	
innodb_adaptive_flushing	ON
innodb_adaptive_flushing_lwm	10.000000
innodb_adaptive_hash_index	ON
innodb_adaptive_hash_index_partitions	64
innodb_adaptive_max_sleep_delay	150000
innodb_additional_mem_pool_size	8388608
innodb_api_bk_commit_interval	5
innodb_api_disable_rowlock	OFF
innodb_api_enable_binlog	OFF
innodb_api_enable_mdl	OFF
innodb_api_trx_level	0
innodb_autoextend_increment	1000
innodb_autoinc_lock_mode	2
innodb_buffer_pool_dump_at_shutdown	ON
innodb_buffer_pool_dump_now	OFF
innodb_buffer_pool_dump_pct	100
innodb_buffer_pool_filename	ib_buffer_pool
innodb_buffer_pool_instances	8
innodb_buffer_pool_load_abort	OFF
innodb_buffer_pool_load_at_startup	ON
innodb_buffer_pool_load_now	OFF
innodb_buffer_pool_populate	OFF
innodb_buffer_pool_size	171798691840
innodb_change_buffer_max_size	25
innodb_change_buffering	all
innodb_checksum_algorithm	innodb
innodb_checksums	ON
innodb_cleaner_lsn_age_factor	high_checkpoint
innodb_cmp_per_index_enabled	OFF
innodb_commit_concurrency	0
innodb_compression_failure_threshold_pct	5
innodb_compression_level	6
innodb_compression_pad_pct_max	50
innodb_concurrency_tickets	5000
innodb_corrupt_table_action	assert
innodb_data_file_path	ibdata1:2048M;ibdata2:2048M;ibdata3:2048M:autoextend
innodb_data_home_dir	/data/XXX
innodb_disable_sort_file_cache	OFF
innodb_doublewrite	ON
innodb_empty_free_list_algorithm	backoff
innodb_fake_changes	OFF
innodb_fast_shutdown	1
innodb_file_format	Barracuda
innodb_file_format_check	ON
innodb_file_format_max	Barracuda
innodb_file_per_table	ON
innodb_flush_log_at_timeout	1
innodb_flush_log_at_trx_commit	2
innodb_flush_method	O_DIRECT
innodb_flush_neighbors	1
innodb_flushing_avg_loops	30
innodb_force_load_corrupted	OFF
innodb_force_recovery	0
innodb_foreground_preflush	exponential_backoff
innodb_ft_aux_table	
innodb_ft_cache_size	8000000
innodb_ft_enable_diag_print	OFF
innodb_ft_enable_stopword	ON
innodb_ft_max_token_size	84
innodb_ft_min_token_size	3
innodb_ft_num_word_optimize	2000
innodb_ft_result_cache_limit	2000000000
innodb_ft_server_stopword_table	
innodb_ft_sort_pll_degree	2
innodb_ft_total_cache_size	640000000
innodb_ft_user_stopword_table	
innodb_io_capacity	600
innodb_io_capacity_max	2000
innodb_kill_idle_transaction	0
innodb_large_prefix	ON
innodb_lock_wait_timeout	3600
innodb_locking_fake_changes	ON
innodb_locks_unsafe_for_binlog	OFF
innodb_log_arch_dir	/logs/XXX/innodblog
innodb_log_arch_expire_sec	0
innodb_log_archive	OFF
innodb_log_block_size	512
innodb_log_buffer_size	134217728
innodb_log_checksum_algorithm	innodb
innodb_log_compressed_pages	ON
innodb_log_file_size	10737418240
innodb_log_files_in_group	3
innodb_log_group_home_dir	/logs/XXX/innodblog
innodb_lru_scan_depth	1024
innodb_max_bitmap_file_size	104857600
innodb_max_changed_pages	1000000
innodb_max_dirty_pages_pct	75.000000
innodb_max_dirty_pages_pct_lwm	0.001000
innodb_max_purge_lag	0
innodb_max_purge_lag_delay	0
innodb_mirrored_log_groups	1
innodb_monitor_disable	
innodb_monitor_enable	
innodb_monitor_reset	
innodb_monitor_reset_all	
innodb_old_blocks_pct	37
innodb_old_blocks_time	1000
innodb_online_alter_log_max_size	1342177280
innodb_open_files	50000
innodb_optimize_fulltext_only	OFF
innodb_page_size	16384
innodb_print_all_deadlocks	ON
innodb_purge_batch_size	300
innodb_purge_threads	1
innodb_random_read_ahead	OFF
innodb_read_ahead_threshold	56
innodb_read_io_threads	16
innodb_read_only	OFF
innodb_replication_delay	0
innodb_rollback_on_timeout	OFF
innodb_rollback_segments	128
innodb_sched_priority_cleaner	19
innodb_show_locks_held	10
innodb_show_verbose_locks	1
innodb_simulate_comp_failures	0
innodb_sort_buffer_size	1048576
innodb_spin_wait_delay	6
innodb_stats_auto_recalc	ON
innodb_stats_method	nulls_equal
innodb_stats_modified_counter	0
innodb_stats_on_metadata	OFF
innodb_stats_persistent	ON
innodb_stats_persistent_sample_pages	20
innodb_stats_sample_pages	8
innodb_stats_traditional	ON
innodb_stats_transient_sample_pages	8
innodb_status_output	OFF
innodb_status_output_locks	OFF
innodb_strict_mode	ON
innodb_support_xa	ON
innodb_sync_array_size	1
innodb_sync_spin_loops	30
innodb_table_locks	ON
innodb_thread_concurrency	128
innodb_thread_sleep_delay	0
innodb_tmpdir	
innodb_track_changed_pages	OFF
innodb_undo_directory	.
innodb_undo_logs	128
innodb_undo_tablespaces	0
innodb_use_atomic_writes	OFF
innodb_use_fallocate	OFF
innodb_use_global_flush_log_at_trx_commit	ON
innodb_use_native_aio	ON
innodb_use_stacktrace	OFF
innodb_use_sys_malloc	ON
innodb_version	5.6.32-79.0
innodb_write_io_threads	16
interactive_timeout	28800
join_buffer_size	2097152
join_buffer_space_limit	2097152
join_cache_level	2
keep_files_on_create	OFF
key_buffer_size	25165824
key_cache_age_threshold	300
key_cache_block_size	1024
key_cache_division_limit	100
key_cache_file_hash_size	512
key_cache_segments	0
large_files_support	ON
large_page_size	0
large_pages	OFF
lc_messages	en_US
lc_messages_dir	
lc_time_names	en_US
license	GPL
local_infile	ON
lock_wait_timeout	60
locked_in_memory	OFF
log_bin	OFF
log_bin_trust_function_creators	ON
log_error	/logs/XXX/error/XXX.err
log_output	FILE
log_queries_not_using_indexes	OFF
log_slave_updates	OFF
log_slow_filter	admin,filesort,filesort_on_disk,full_join,full_scan,query_cache,query_cache_miss,tmp_table,tmp_table_on_disk
log_slow_rate_limit	1
log_slow_verbosity	query_plan,explain
log_warnings	2
long_query_time	3.000000
low_priority_updates	OFF
lower_case_file_system	OFF
lower_case_table_names	1
master_verify_checksum	OFF
max_allowed_packet	1073741824
max_binlog_cache_size	18446744073709547520
max_binlog_size	1073741824
max_binlog_stmt_cache_size	18446744073709547520
max_connect_errors	1000000
max_connections	2000
max_delayed_threads	20
max_digest_length	1024
max_error_count	64
max_heap_table_size	524288000
max_insert_delayed_threads	20
max_join_size	18446744073709551615
max_length_for_sort_data	1024
max_long_data_size	1073741824
max_prepared_stmt_count	16382
max_relay_log_size	1073741824
max_seeks_for_key	4294967295
max_sort_length	1024
max_sp_recursion_depth	0
max_tmp_tables	32
max_user_connections	0
max_write_lock_count	4294967295
metadata_locks_cache_size	1024
metadata_locks_hash_instances	8
min_examined_row_limit	0
mrr_buffer_size	262144
multi_range_count	256
myisam_block_size	1024
myisam_data_pointer_size	6
myisam_max_sort_file_size	9223372036853727232
myisam_mmap_size	18446744073709551615
myisam_recover_options	DEFAULT
myisam_repair_threads	1
myisam_sort_buffer_size	134216704
myisam_stats_method	nulls_unequal
myisam_use_mmap	OFF
net_buffer_length	16384
net_read_timeout	300
net_retry_count	10
net_write_timeout	600
old	OFF
old_alter_table	OFF
old_mode	
old_passwords	OFF
open_files_limit	1048576
optimizer_prune_level	1
optimizer_search_depth	62
optimizer_selectivity_sampling_limit	100
optimizer_switch	index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=off,table_elimination=on,extended_keys=on,exists_to_in=on
optimizer_use_condition_selectivity	1
performance_schema	ON
performance_schema_accounts_size	100
performance_schema_digests_size	10000
performance_schema_events_stages_history_long_size	10000
performance_schema_events_stages_history_size	10
performance_schema_events_statements_history_long_size	10000
performance_schema_events_statements_history_size	10
performance_schema_events_waits_history_long_size	10000
performance_schema_events_waits_history_size	10
performance_schema_hosts_size	100
performance_schema_max_cond_classes	80
performance_schema_max_cond_instances	8900
performance_schema_max_digest_length	1024
performance_schema_max_file_classes	50
performance_schema_max_file_handles	32768
performance_schema_max_file_instances	2154
performance_schema_max_mutex_classes	200
performance_schema_max_mutex_instances	17000
performance_schema_max_rwlock_classes	40
performance_schema_max_rwlock_instances	6800
performance_schema_max_socket_classes	10
performance_schema_max_socket_instances	4020
performance_schema_max_stage_classes	150
performance_schema_max_statement_classes	180
performance_schema_max_table_handles	800
performance_schema_max_table_instances	12500
performance_schema_max_thread_classes	50
performance_schema_max_thread_instances	4100
performance_schema_session_connect_attrs_size	512
performance_schema_setup_actors_size	100
performance_schema_setup_objects_size	100
performance_schema_users_size	100
pid_file	/data/XXX/LGEDGMRP01.pid
plugin_dir	/engn/XXX/mysql/lib/plugin/
plugin_maturity	unknown
port	3310
preload_buffer_size	32768
profiling	OFF
profiling_history_size	15
progress_report_time	5
protocol_version	10
query_alloc_block_size	8192
query_cache_limit	1048576
query_cache_min_res_unit	4096
query_cache_size	0
query_cache_strip_comments	OFF
query_cache_type	OFF
query_cache_wlock_invalidate	OFF
query_prealloc_size	8192
range_alloc_block_size	4096
read_buffer_size	2097152
read_only	OFF
read_rnd_buffer_size	2097152
relay_log	
relay_log_index	
relay_log_info_file	relay-log.info
relay_log_purge	ON
relay_log_recovery	OFF
relay_log_space_limit	0
replicate_annotate_row_events	OFF
replicate_do_db	
replicate_do_table	
replicate_events_marked_for_skip	replicate
replicate_ignore_db	
replicate_ignore_table	
replicate_wild_do_table	
replicate_wild_ignore_table	
report_host	
report_password	
report_port	3310
report_user	
rowid_merge_buff_size	8388608
rpl_recovery_rank	0
secure_auth	OFF
secure_file_priv	
server_audit_events	QUERY_DDL
server_audit_excl_users	
server_audit_file_path	server_audit.log
server_audit_file_rotate_now	OFF
server_audit_file_rotate_size	1000000
server_audit_file_rotations	9
server_audit_incl_users	
server_audit_logging	ON
server_audit_mode	0
server_audit_output_type	file
server_audit_query_log_limit	1024
server_audit_syslog_facility	LOG_USER
server_audit_syslog_ident	mysql-server_auditing
server_audit_syslog_info	
server_audit_syslog_priority	LOG_INFO
server_id	0
skip_external_locking	ON
skip_name_resolve	ON
skip_networking	OFF
skip_show_database	OFF
slave_compressed_protocol	OFF
slave_ddl_exec_mode	IDEMPOTENT
slave_domain_parallel_threads	0
slave_exec_mode	STRICT
slave_load_tmpdir	/data/XXX
slave_max_allowed_packet	1073741824
slave_net_timeout	3600
slave_parallel_max_queued	131072
slave_parallel_threads	0
slave_skip_errors	OFF
slave_sql_verify_checksum	ON
slave_transaction_retries	10
slave_type_conversions	
slow_launch_time	2
slow_query_log	OFF
slow_query_log_file	/logs/XXX/slow/XXX_slow.log
socket	/engn/XXX/mysql/mysql.sock
sort_buffer_size	2097152
sql_auto_is_null	OFF
sql_big_selects	ON
sql_buffer_result	OFF
sql_error_log_filename	sql_errors.log
sql_error_log_rate	1
sql_error_log_rotate	OFF
sql_error_log_rotations	9
sql_error_log_size_limit	1000000
sql_log_bin	ON
sql_log_off	OFF
sql_mode	PIPES_AS_CONCAT,ANSI_QUOTES,IGNORE_SPACE,ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION
sql_notes	ON
sql_quote_show_create	ON
sql_safe_updates	OFF
sql_select_limit	18446744073709551615
sql_slave_skip_counter	0
sql_warnings	OFF
ssl_ca	
ssl_capath	
ssl_cert	
ssl_cipher	
ssl_crl	
ssl_crlpath	
ssl_key	
storage_engine	InnoDB
stored_program_cache	256
sync_binlog	0
sync_frm	ON
sync_master_info	0
sync_relay_log	0
sync_relay_log_info	0
system_time_zone	KST
table_definition_cache	50000
table_open_cache	50000
thread_cache_size	100
thread_concurrency	10
thread_handling	one-thread-per-connection
thread_pool_idle_timeout	60
thread_pool_max_threads	500
thread_pool_oversubscribe	3
thread_pool_size	56
thread_pool_stall_limit	500
thread_stack	294912
time_format	%H:%i:%s
time_zone	SYSTEM
timed_mutexes	OFF
tmp_table_size	524288000
tmpdir	/data/XXX
transaction_alloc_block_size	8192
transaction_prealloc_size	4096
tx_isolation	READ-COMMITTED
tx_read_only	OFF
unique_checks	ON
updatable_views_with_limit	YES
use_stat_tables	NEVER
userstat	OFF
version	10.0.28-MariaDB
version_comment	MariaDB Server
version_compile_machine	x86_64
version_compile_os	Linux
version_malloc_library	bundled jemalloc
wait_timeout	28800

Comment by Elena Stepanova [ 2018-07-18 ]

2. I will share the contents except the logging data for the repeated actions after the restart point.

Do you mean that you will enable the coredump and provide the information from it?
If so, thanks, we'll be waiting for it.

Comment by ssauravy [ 2018-07-19 ]

no. This is happening to us on a very important production server, and core dump can not be activated.
Just add a log directly below the above log (from the point of restart).

5564399 180714 19:25:33 mysqld_safe Number of processes running now: 0
5564400 180714 19:25:33 mysqld_safe mysqld restarted

5636219 InnoDB: about forcing recovery.
5636220 InnoDB: Error in pages 12862415 and 12862421 of index "XMVA_ITEM_REQ_DETAIL_N3" of table "gmrpp"."xmva_item_req_detail"
5636221 InnoDB: broken FIL_PAGE_NEXT or FIL_PAGE_PREV links
5636222 2018-07-15 09:10:57 7fac601fc700 InnoDB: Page dump in ascii and hex (16384 bytes):

5582759 2018-07-14 19:48:20 7f7f56ded700 InnoDB: uncompressed page, stored checksum in field1 17170432, calculated checksums for field1: crc32 2753499458, innodb 391883286, none 3735928559, s tored checksum in field2 3590652932, calculated checksums for field2: crc32 2753499458, innodb 653250342, none 3735928559, page LSN 0 4352, low 4 bytes of LSN at page end 839016448, p age number (if stored to page already) 131072, space id (if created with >= MySQL-4.1.1 and stored already) 2650806784
5582760 InnoDB: Page may be a freshly allocated page
5582761 InnoDB: Database page corruption on disk or a failed
5582762 InnoDB: file read of page 12862421.
5582763 InnoDB: You may have to recover from a backup.
5582764 InnoDB: It is also possible that your operating
5582765 InnoDB: system has corrupted its own file cache
5582766 InnoDB: and rebooting your computer removes the
5582767 InnoDB: error.
5582768 InnoDB: If the corrupt page is an index page
5582769 InnoDB: you can also try to fix the corruption
5582770 InnoDB: by dumping, dropping, and reimporting
5582771 InnoDB: the corrupt table. You can use CHECK
5582772 InnoDB: TABLE to scan your table for corruption.
5582773 InnoDB: See also http://dev.mysql.com/doc/refman/5.6/en/forcing-innodb-recovery.html
5582774 InnoDB: about forcing recovery.
5582775 180714 19:48:23 [ERROR] mysqld: Table './gmrpp/jjcr_request_next_id' is marked as crashed and should be repaired
5582776 180714 19:48:23 [ERROR] mysqld: Table 'jjcr_request_next_id' is marked as crashed and should be repaired
5582777 180714 19:48:23 [Warning] Checking table: './gmrpp/jjcr_request_next_id'
5582778 180714 19:48:23 [ERROR] mysqld: Table './gmrpp/jjrs_instance_next_id' is marked as crashed and should be repaired
5582779 180714 19:48:23 [ERROR] mysqld: Table 'jjrs_instance_next_id' is marked as crashed and should be repaired
5582780 180714 19:48:23 [Warning] Checking table: './gmrpp/jjrs_instance_next_id'
5582781 180714 19:48:23 [ERROR] mysqld: Table './gmrpp/jjrs_job_next_id' is marked as crashed and should be repaired
5582782 180714 19:48:23 [ERROR] mysqld: Table 'jjrs_job_next_id' is marked as crashed and should be repaired
5582783 180714 19:48:23 [Warning] Checking table: './gmrpp/jjrs_job_next_id'
5582784 180714 19:48:23 [ERROR] mysqld: Table './gmrpp/jjrs_step_next_id' is marked as crashed and should be repaired
5582785 180714 19:48:23 [ERROR] mysqld: Table 'jjrs_step_next_id' is marked as crashed and should be repaired
5582786 180714 19:48:23 [Warning] Checking table: './gmrpp/jjrs_step_next_id'
5582787 180714 19:48:30 [ERROR] mysqld: Table './gmrpp/batch_job_seq' is marked as crashed and should be repaired
5582788 180714 19:48:30 [ERROR] mysqld: Table 'batch_job_seq' is marked as crashed and should be repaired
5582789 180714 19:48:30 [Warning] Checking table: './gmrpp/batch_job_seq'
5582790 180714 19:48:30 [ERROR] mysqld: Table './gmrpp/batch_job_execution_seq' is marked as crashed and should be repaired
5582791 180714 19:48:30 [ERROR] mysqld: Table 'batch_job_execution_seq' is marked as crashed and should be repaired
5582792 180714 19:48:30 [Warning] Checking table: './gmrpp/batch_job_execution_seq'
5582793 180714 19:48:30 [ERROR] mysqld: Table './gmrpp/batch_step_execution_seq' is marked as crashed and should be repaired
5582794 180714 19:48:30 [ERROR] mysqld: Table 'batch_step_execution_seq' is marked as crashed and should be repaired
5582795 180714 19:48:30 [Warning] Checking table: './gmrpp/batch_step_execution_seq'
5582796 2018-07-14 19:48:51 7f7f4c3dc700 InnoDB: table "gmrpp"."xmva_item_req_detail"is corrupted. Please drop the table and recreate
5582797 2018-07-14 19:48:51 7fa4bd46e700 InnoDB: table 180714 19:48:51 [Warning] InnoDB: Cannot open table gmrpp/xmva_item_req_detail from the internal data dictionary of InnoDB though the . frm file for the table exists. See http://dev.mysql.com/doc/refman/5.6/en/innodb-troubleshooting.html for how you can resolve the problem.
5582798 2018-07-14 19:48:51 7f7f4c194700 InnoDB: table "gmrpp"."xmva_item_req_detail"is corrupted. Please drop the table and recreate
5582799 "gmrpp"."xmva_item_req_detail"is corrupted. Please drop the table and recreate
5582800 180714 19:48:51 [Warning] InnoDB: Cannot open table gmrpp/xmva_item_req_detail from the internal data dictionary of InnoDB though the .frm file for the table exists. See http://dev.my sql.com/doc/refman/5.6/en/innodb-troubleshooting.html for how you can resolve the problem.
5582801 180714 19:48:51 [Warning] InnoDB: Cannot open table gmrpp/xmva_item_req_detail from the internal data dictionary of InnoDB though the .frm file for the table exists. See http://dev.my sql.com/doc/refman/5.6/en/innodb-troubleshooting.html for how you can resolve the problem.
5582802 180714 19:48:52 [Warning] Aborted connection 1091 to db: 'gmrpp' user: 'gmrp_app' host: '10.185.50.12' (Got an error reading communication packets)
5582803 180714 19:48:52 [Warning] Aborted connection 1088 to db: 'gmrpp' user: 'gmrp_app' host: '10.185.50.12' (Got an error reading communication packets)
5582804 180714 19:48:52 [Warning] Aborted connection 1089 to db: 'gmrpp' user: 'gmrp_app' host: '10.185.50.12' (Got an error reading communication packets)
5582805 180714 19:49:02 [Warning] Aborted connection 1375 to db: 'gmrpp' user: 'gmrp_inf' host: '10.185.135.144' (Got an error reading communication packets)
5582806 180714 19:49:02 [Warning] Aborted connection 1368 to db: 'gmrpp' user: 'gmrp_inf' host: '10.185.135.144' (Got an error reading communication packets)
5582807 180714 19:49:02 [Warning] Aborted connection 1376 to db: 'gmrpp' user: 'gmrp_inf' host: '10.185.135.144' (Got an error reading communication packets)
5582808 2018-07-14 19:49:10 7f7ea8cda700 InnoDB: table "gmrpp"."xmva_item_req_detail"is corrupted. Please drop the table and recreate
5582809 180714 19:49:10 [Warning] InnoDB: Cannot open table gmrpp/xmva_item_req_detail from the internal data dictionary of InnoDB though the .frm file for the table exists. See http://dev.my sql.com/doc/refman/5.6/en/innodb-troubleshooting.html for how you can resolve the problem.
5582810 2018-07-14 19:49:10 7fa4bd549700 InnoDB: table "gmrpp"."xmva_item_req_detail"is corrupted. Please drop the table and recreate
5582811 180714 19:49:10 [Warning] InnoDB: Cannot open table gmrpp/xmva_item_req_detail from the internal data dictionary of InnoDB though the .frm file for the table exists. See http://dev.my sql.com/doc/refman/5.6/en/innodb-troubleshooting.html for how you can resolve the problem.
5582812 2018-07-14 19:49:10 7f7f4c26f700 InnoDB: table "gmrpp"."xmva_item_req_detail"is corrupted. Please drop the table and recreate
5582813 180714 19:49:10 [Warning] InnoDB: Cannot open table gmrpp/xmva_item_req_detail from the internal data dictionary of InnoDB though the .frm file for the table exists. See http://dev.my sql.com/doc/refman/5.6/en/innodb-troubleshooting.html for how you can resolve the problem.
5582814 180714 19:49:10 [Warning] Aborted connection 1593 to db: 'gmrpp' user: 'gmrp_app' host: '10.185.50.12' (Got an error reading communication packets)
5582815 180714 19:49:10 [Warning] Aborted connection 1610 to db: 'gmrpp' user: 'gmrp_app' host: '10.185.50.12' (Got an error reading communication packets)
5582816 180714 19:49:10 [Warning] Aborted connection 1607 to db: 'gmrpp' user: 'gmrp_app' host: '10.185.50.12' (Got an error reading communication packets)
5582817 180714 19:50:03 [Warning] Aborted connection 3528 to db: 'gmrpp' user: 'gmrp_inf' host: '10.185.135.144' (Got an error reading communication packets)
5582818 180714 19:50:03 [Warning] Aborted connection 3500 to db: 'gmrpp' user: 'gmrp_inf' host: '10.185.135.144' (Got an error reading communication packets)
5582819 180714 19:50:03 [Warning] Aborted connection 3518 to db: 'gmrpp' user: 'gmrp_inf' host: '10.185.135.144' (Got an error reading communication packets)
5582820 180714 19:50:03 [Warning] Aborted connection 3498 to db: 'gmrpp' user: 'gmrp_inf' host: '10.185.135.144' (Got an error reading communication packets)
5582821 180714 19:50:03 [Warning] Aborted connection 3517 to db: 'gmrpp' user: 'gmrp_inf' host: '10.185.135.144' (Got an error reading communication packets)
5582822 2018-07-14 19:50:25 7f7f56ded700 InnoDB: Error: space id and page n:o stored in the page
5582823 InnoDB: read in are 2650806784:131072, should be 61037:12862421!
5582824 InnoDB: Database page corruption on disk or a failed
5582825 InnoDB: file read of page 12862421.
5582826 InnoDB: You may have to recover from a backup.
5582827 2018-07-14 19:50:25 7f7f56ded700 InnoDB: Page dump in ascii and hex (16384 bytes):
no. This is happening to us on a very important production server, and core dump can not be activated.
Just add a log directly below the above log (from the point of restart).

5636230 2018-07-15 09:10:57 7fac601fc700 InnoDB: uncompressed page, stored checksum in field1 17170432, calculated checksums for field1: crc32 2753499458, innodb 391883286, none 3735928559, s tored checksum in field2 3590652932, calculated checksums for field2: crc32 2753499458, innodb 653250342, none 3735928559, page LSN 0 4352, low 4 bytes of LSN at page end 839016448, p age number (if stored to page already) 131072, space id (if created with >= MySQL-4.1.1 and stored already) 2650806784
5636231 InnoDB: Page may be a freshly allocated page
5636232 InnoDB: Error in pages 12862415 and 12862421 of index "XMVA_ITEM_REQ_DETAIL_N3" of table "gmrpp"."xmva_item_req_detail"
5636233 InnoDB: 'compact' flag mismatch
5636234 2018-07-15 09:10:57 7fac601fc700 InnoDB: Page dump in ascii and hex (16384 bytes):

Comment by ssauravy [ 2018-07-24 ]

Do you have any additional checks?
We are in the operating system, so we have a lot of anxieties about how to follow up in the future. Thank you.

Comment by ossk_db [ 2018-07-26 ]

cpu.txt disk.txt engine.txt gmrpe_table_size.txt gmrpp_table_size.txt memory.txt plugin.txt sts.txt var.txt

Comment by Elena Stepanova [ 2018-08-04 ]

Back to your original question about the cause of signal 6, InnoDB itself rather verbosely explains it in the error log: it's been trying to read a page, fails to do so, apparently due to data corruption, and after a number of attempts gives up.
Assigning to marko for further advice/ideas/etc.

Comment by ossk_db [ 2018-10-01 ]

Hi~ Team.

1. Do you know if this text has been patched in the latest version?

2. Is the obvious data corruption a problem with the disk HW?

Comment by Marko Mäkelä [ 2018-10-12 ]

Hi ossk_db, my simplest explanation would be to blame the hardware.
I did not see any messages about the page LSN being in the future. Such messages could be displayed when something wrong was done, such as killing the server, deleting ib_logfile* and restarting.

That said, there is some room for improvement for InnoDB could in this area: MDEV-13542 should propagate read errors up to the client (so that the user would know which table or SQL statement is in question), and the InnoDB doublewrite buffer could be improved as well (MDEV-11799, MDEV-12905).

The doublewrite buffer only matters if the mysqld process was killed (or the whole system was killed) while InnoDB was in the middle of writing a data page.

Comment by Marko Mäkelä [ 2018-10-12 ]

One thing that could be useful is to pay attention to the diagnostics provided by the storage layer, to get an early warning of upcoming failures. Modern hard disks and SSDs provide the S.M.A.R.T. interface. On GNU/Linux, there is the smartmontools package. If I were running an important server, I would try adding some diagnostics around smartctl -A /dev/sda or similar.

Generated at Thu Feb 08 08:31:22 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.