[MDEV-19613] Got error 128 when reading "Table Name" - when i tried to truncate a partition in a table Created: 2019-05-28  Updated: 2021-12-06

Status: Open
Project: MariaDB Server
Component/s: Data Definition - Alter Table
Affects Version/s: 10.3.12, 10.3.21
Fix Version/s: None

Type: Bug Priority: Major
Reporter: Stuart Slater Assignee: Unassigned
Resolution: Unresolved Votes: 2
Labels: None
Environment:

Mariadb 10.3.12 32bit Windows 2008 R1


Issue Links:
Relates
relates to MDEV-18815 Server crashes on TRUNCATE TABLE Closed

 Description   

When I was trying to truncate a partition i received the following error
2019-05-28 4:47:50 24189 [ERROR] Got error 128 when reading table '.\datafactorydb\rjis_jqi_perf'

I am on Windows 2008 R1, 32-bit

the statement
alter table rjis_jqi_perf truncate partition p20 just hangs; and then I get the above error 128 in the mariadb log file.

I cancel the statement; and retry 5minutes later and still the same problem.

I tried a third time and the statement worked.

The statements I was running on a large table was (all worked first time except it hung on P20):-
alter table rjis_jqi_perf truncate partition p1;
alter table rjis_jqi_perf truncate partition p2;
alter table rjis_jqi_perf truncate partition p3;
alter table rjis_jqi_perf truncate partition p4;
alter table rjis_jqi_perf truncate partition p5;
alter table rjis_jqi_perf truncate partition p6;
alter table rjis_jqi_perf truncate partition p7;
alter table rjis_jqi_perf truncate partition p8;
alter table rjis_jqi_perf truncate partition p9;
alter table rjis_jqi_perf truncate partition p10;
alter table rjis_jqi_perf truncate partition p11;
alter table rjis_jqi_perf truncate partition p12;
alter table rjis_jqi_perf truncate partition p13;
alter table rjis_jqi_perf truncate partition p14;
alter table rjis_jqi_perf truncate partition p15;
alter table rjis_jqi_perf truncate partition p16;
alter table rjis_jqi_perf truncate partition p17;
alter table rjis_jqi_perf truncate partition p18;
alter table rjis_jqi_perf truncate partition p19;
alter table rjis_jqi_perf truncate partition p20;
alter table rjis_jqi_perf truncate partition p21;
alter table rjis_jqi_perf truncate partition p22;
alter table rjis_jqi_perf truncate partition p23;
alter table rjis_jqi_perf truncate partition p24;
alter table rjis_jqi_perf truncate partition p25;
alter table rjis_jqi_perf truncate partition p29;
alter table rjis_jqi_perf truncate partition p30;
alter table rjis_jqi_perf truncate partition p31;

Not sure if this relates to the bug in MDEV-17877



 Comments   
Comment by Elena Stepanova [ 2019-05-28 ]

Error 128 appears to be the "Out of memory in engine" error. Do you have any kind of memory monitoring on your machine?

Comment by Stuart Slater [ 2019-05-29 ]

Hi,
Many Thanks you for looking into this.
No memory monitoring; and no s/w configuration changes on this machine for over 6 months.
It has plenty of physical memory left - physical 4GB

my.ini configuration file

# Remove leading # and set to the amount of RAM for the most important data
# cache in MySQL. Start at 70% of total RAM for dedicated server, else 10%.
# innodb_buffer_pool_size = 128M
 
# Remove leading # to turn on a very important data integrity option: logging
# changes to the binary log between backups.
# log_bin
 
# These are commonly set, remove the # and set as required.
# basedir = .....
# datadir = .....
# port = .....
# server_id = .....
 
# Remove leading # to set options mainly useful for reporting servers.
# The server defaults are faster for transactions and fast SELECTs.
# Adjust sizes as needed, experiment to find the optimal values.
# join_buffer_size = 128M
# sort_buffer_size = 2M
# read_rnd_buffer_size = 2M 
 
[client]
 
# pipe
# socket=mysql
port=3306
 
[mysql]
default-character-set=latin1
 
[mysqld]
# The TCP/IP Port the MySQL Server will listen on
port=3306
 
# Path to installation directory. All paths are usually resolved relative to this.
 
# Path to the database root
datadir=D:/MariaDB/data/
character-set-server=latin1
 
# Set the SQL mode to strict
sql-mode="STRICT_TRANS_TABLES,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION"
 
# Enable Windows Authentication
# plugin-load=authentication_windows.dll
 
# General and Slow logging.
log-output=FILE
general-log=0
general_log_file="jpdpt3.log"
slow-query-log=1
 
slow_query_log_file="D:/MariaDB/Logs/jpdpt3-slow.log"
long_query_time=10
 
# Binary Logging.
# log-bin
 
# Error Logging.
log-error="D:/MariaDB/Logs/jpdpt3.log"
 
# The maximum amount of concurrent sessions the MySQL server will
# allow. One of these connections will be reserved for a user with
# SUPER privileges to allow the administrator to login even if the
# connection limit has been reached.
max_connections=1000
 
# Query cache is used to cache SELECT results and later return them
# without actual executing the same query once again. Having the query
# cache enabled may result in significant speed improvements, if your
# have a lot of identical queries and rarely changing tables. See the
# "Qcache_lowmem_prunes" status variable to check if the current value
# is high enough for your load.
# Note: In case your tables change very often or if your queries are
# textually different every time, the query cache may result in a
# slowdown instead of a performance improvement.
query_cache_size=0
 
# The number of open tables for all threads. Increasing this value
# increases the number of file descriptors that mysqld requires.
# Therefore you have to make sure to set the amount of open files
# allowed to at least 4096 in the variable "open-files-limit" in
# section [mysqld_safe]
table_open_cache=4000
 
# Maximum size for internal (in-memory) temporary tables. If a table
# grows larger than this value, it is automatically converted to disk
# based table This limitation is for a single table. There can be many
# of them.
tmp_table_size=30M
 
# How many threads we should keep in a cache for reuse. When a client
# disconnects, the client's threads are put in the cache if there aren't
# more than thread_cache_size threads from before.  This greatly reduces
# the amount of thread creations needed if you have a lot of new
# connections. (Normally this doesn't give a notable performance
# improvement if you have a good thread implementation.)
thread_cache_size=9
 
#*** MyISAM Specific options
# The maximum size of the temporary file MySQL is allowed to use while
# recreating the index (during REPAIR, ALTER TABLE or LOAD DATA INFILE.
# If the file-size would be bigger than this, the index will be created
# through the key cache (which is slower).
myisam_max_sort_file_size=100G
 
# If the temporary file used for fast index creation would be bigger
# than using the key cache by the amount specified here, then prefer the
# key cache method.  This is mainly used to force long character keys in
# large tables to use the slower key cache method to create the index.
myisam_sort_buffer_size=17M
 
# Size of the Key Buffer, used to cache index blocks for MyISAM tables.
# Do not set it larger than 30% of your available memory, as some memory
# is also required by the OS to cache rows. Even if you're not using
# MyISAM tables, you should still set it to 8-64M as it will also be
# used for internal temporary disk tables.
key_buffer_size=8M
 
# Size of the buffer used for doing full table scans of MyISAM tables.
# Allocated per thread, if a full scan is needed.
read_buffer_size=36K
read_rnd_buffer_size=512K
 
# This buffer is allocated when MySQL needs to rebuild the index in
# REPAIR, OPTIMZE, ALTER table statements as well as in LOAD DATA INFILE
# into an empty table. It is allocated per thread so be careful with
# large settings.
sort_buffer_size=1024K
 
#*** INNODB Specific options ***
# innodb_data_home_dir=0.0
 
# Use this option if you have a MySQL server with InnoDB support enabled
# but you do not plan to use it. This will save memory and disk space
# and speed up some things.
# skip-innodb
 
# Additional memory pool that is used by InnoDB to store metadata
# information.  If InnoDB requires more memory for this purpose it will
# start to allocate it from the OS.  As this is fast enough on most
# recent operating systems, you normally do not need to change this
# value. SHOW INNODB STATUS will display the current amount used.
 
# If set to 1, InnoDB will flush (fsync) the transaction logs to the
# disk at each commit, which offers full ACID behavior. If you are
# willing to compromise this safety, and you are running small
# transactions, you may set this to 0 or 2 to reduce disk I/O to the
# logs. Value 0 means that the log is only written to the log file and
# the log file flushed to disk approximately once per second. Value 2
# means the log is written to the log file at each commit, but the log
# file is only flushed to disk approximately once per second.
innodb_flush_log_at_trx_commit=1
 
# The size of the buffer InnoDB uses for buffering log data. As soon as
# it is full, InnoDB will have to flush it to disk. As it is flushed
# once per second anyway, it does not make sense to have it very large
# (even with long transactions).
innodb_log_buffer_size=1M
 
# InnoDB, unlike MyISAM, uses a buffer pool to cache both indexes and
# row data. The bigger you set this the less disk I/O is needed to
# access data in tables. On a dedicated database server you may set this
# parameter up to 80% of the machine physical memory size. Do not set it
# too large, though, because competition of the physical memory may
# cause paging in the operating system.  Note that on 32bit systems you
# might be limited to 2-3.5G of user level memory per process, so do not
# set it too high.
innodb_buffer_pool_size=300M
 
# Size of each log file in a log group. You should set the combined size
# of log files to about 25%-100% of your buffer pool size to avoid
# unneeded buffer pool flush activity on log file overwrite. However,
# note that a larger logfile size will increase the time needed for the
# recovery process.
innodb_log_file_size=100M
 
# Number of threads allowed inside the InnoDB kernel. The optimal value
# depends highly on the application, hardware as well as the OS
# scheduler properties. A too high value may lead to thread thrashing.
innodb_thread_concurrency=9
 
# The increment size (in MB) for extending the size of an auto-extend InnoDB system tablespace file when it becomes full.
innodb_autoextend_increment=64
 
# The number of regions that the InnoDB buffer pool is divided into.
# For systems with buffer pools in the multi-gigabyte range, dividing the buffer pool into separate instances can improve concurrency,
# by reducing contention as different threads read and write to cached pages.
innodb_buffer_pool_instances=8
 
# Determines the number of threads that can enter InnoDB concurrently.
innodb_concurrency_tickets=8000
 
# Specifies how long in milliseconds (ms) a block inserted into the old sublist must stay there after its first access before
# it can be moved to the new sublist.
innodb_old_blocks_time=1000
 
# It specifies the maximum number of .ibd files that MySQL can keep open at one time. The minimum value is 10.
innodb_open_files=500
 
# When this variable is enabled, InnoDB updates statistics during metadata statements.
innodb_stats_on_metadata=0
 
# When innodb_file_per_table is enabled (the default in 5.6.6 and higher), InnoDB stores the data and indexes for each newly created table
# in a separate .ibd file, rather than in the system tablespace.
innodb_file_per_table=1
 
# Use the following list of values: 0 for crc32, 1 for strict_crc32, 2 for innodb, 3 for strict_innodb, 4 for none, 5 for strict_none.
innodb_checksum_algorithm=0
 
# The number of outstanding connection requests MySQL can have.
# This option is useful when the main MySQL thread gets many connection requests in a very short time.
# It then takes some time (although very little) for the main thread to check the connection and start a new thread.
# The back_log value indicates how many requests can be stacked during this short time before MySQL momentarily
# stops answering new requests.
# You need to increase this only if you expect a large number of connections in a short period of time.
back_log=70
 
# If this is set to a nonzero value, all tables are closed every flush_time seconds to free up resources and
# synchronize unflushed data to disk.
# This option is best used only on systems with minimal resources.
flush_time=0
 
# The minimum size of the buffer that is used for plain index scans, range index scans, and joins that do not use
# indexes and thus perform full table scans.
join_buffer_size=256K
 
# The maximum size of one packet or any generated or intermediate string, or any parameter sent by the
# mysql_stmt_send_long_data() C API function.
max_allowed_packet=4M
 
# If more than this many successive connection requests from a host are interrupted without a successful connection,
# the server blocks that host from performing further connections.
max_connect_errors=200
 
# Changes the number of file descriptors available to mysqld.
# You should try increasing the value of this option if mysqld gives you the error "Too many open files".
open_files_limit=6110
 
# Set the query cache type. 0 for OFF, 1 for ON and 2 for DEMAND.
query_cache_type=0
 
# If you see many sort_merge_passes per second in SHOW GLOBAL STATUS output, you can consider increasing the
# sort_buffer_size value to speed up ORDER BY or GROUP BY operations that cannot be improved with query optimization
# or improved indexing.
sort_buffer_size=256K
 
# The number of table definitions (from .frm files) that can be stored in the definition cache.
# If you use a large number of tables, you can create a large table definition cache to speed up opening of tables.
# The table definition cache takes less space and does not use file descriptors, unlike the normal table cache.
# The minimum and default values are both 400.
table_definition_cache=1400
 
# Specify the maximum size of a row-based binary log event, in bytes.
# Rows are grouped into events smaller than this size if possible. The value should be a multiple of 256.
binlog_row_event_max_size=8K
 
# If the value of this variable is greater than 0, a replication slave synchronizes its master.info file to disk.
# (using fdatasync()) after every sync_master_info events.
sync_master_info=10000
 
# If the value of this variable is greater than 0, the MySQL server synchronizes its relay log to disk.
# (using fdatasync()) after every sync_relay_log writes to the relay log.
sync_relay_log=10000
 
# If the value of this variable is greater than 0, a replication slave synchronizes its relay-log.info file to disk.
# (using fdatasync()) after every sync_relay_log_info transactions.
sync_relay_log_info=10000
 
# Added 16/04/14 S. Slater to allow a remote connection when the reverse DNS lookup does not resolve
skip-name-resolve
 
# Added BY S.Slater for DF_2.0.6 to enable Table Compression on RCS_FLOW
# CR6031
# innodb_file_format=Barracuda
 
#
# Setup for Replication 
#
server-id=1
binlog_do_db=datafactorydb
log-bin=d:/MariaDB/master/logbin/Master-bin.log
innodb_flush_log_at_trx_commit=1
sync_binlog=1
expire-logs-days=7
max_binlog_size=214748365
binlog-format=MIXED

Comment by Marko Mäkelä [ 2019-05-29 ]

If InnoDB is being used, and innodb_adaptive_hash_index=ON (the default), then this might be related to MDEV-18815, where InnoDB crashes due to corruption in the adaptive hash index. That same problem ought to be possible also with DROP TABLE or DROP PARTITION.

Comment by Stuart Slater [ 2019-05-29 ]

Marko; many thanks.
I can confirm adaptive hash index=ON

Variable_nameValue
innodb_adaptive_flushingON
innodb_adaptive_flushing_lwm10.000000
innodb_adaptive_hash_indexON
innodb_adaptive_hash_index_parts8
innodb_adaptive_max_sleep_delay150000
innodb_autoextend_increment64
innodb_autoinc_lock_mode1
innodb_background_scrub_data_check_interval3600
innodb_background_scrub_data_compressedOFF
innodb_background_scrub_data_interval604800
innodb_background_scrub_data_uncompressedOFF
innodb_buf_dump_status_frequency0
innodb_buffer_pool_chunk_size134217728
innodb_buffer_pool_dump_at_shutdownON
innodb_buffer_pool_dump_nowOFF
innodb_buffer_pool_dump_pct25
innodb_buffer_pool_filenameib_buffer_pool
innodb_buffer_pool_instances1
innodb_buffer_pool_load_abortOFF
innodb_buffer_pool_load_at_startupON
innodb_buffer_pool_load_nowOFF
innodb_buffer_pool_size402653184
innodb_change_buffer_max_size25
innodb_change_bufferingall
innodb_checksum_algorithmcrc32
innodb_checksumsON
innodb_cmp_per_index_enabledOFF
innodb_commit_concurrency0
innodb_compression_algorithmzlib
innodb_compression_defaultOFF
innodb_compression_failure_threshold_pct5
innodb_compression_level6
innodb_compression_pad_pct_max50
innodb_concurrency_tickets8000
innodb_data_file_pathibdata1:12M:autoextend
innodb_data_home_dir
innodb_deadlock_detectON
innodb_default_encryption_key_id1
innodb_default_row_formatdynamic
innodb_defragmentOFF
innodb_defragment_fill_factor0.900000
innodb_defragment_fill_factor_n_recs20
innodb_defragment_frequency40
innodb_defragment_n_pages7
innodb_defragment_stats_accuracy0
innodb_disable_sort_file_cacheOFF
innodb_doublewriteON
innodb_encrypt_logOFF
innodb_encrypt_tablesOFF
innodb_encryption_rotate_key_age1
innodb_encryption_rotation_iops100
innodb_encryption_threads0
innodb_fast_shutdown1
innodb_fatal_semaphore_wait_threshold600
innodb_file_per_tableON
innodb_fill_factor100
innodb_flush_log_at_timeout1
innodb_flush_log_at_trx_commit1
innodb_flush_methodunbuffered
innodb_flush_neighbors1
innodb_flush_syncON
innodb_flushing_avg_loops30
innodb_force_load_corruptedOFF
innodb_force_primary_keyOFF
innodb_force_recovery0
innodb_ft_aux_table
innodb_ft_cache_size8000000
innodb_ft_enable_diag_printOFF
innodb_ft_enable_stopwordON
innodb_ft_max_token_size84
innodb_ft_min_token_size3
innodb_ft_num_word_optimize2000
innodb_ft_result_cache_limit2000000000
innodb_ft_server_stopword_table
innodb_ft_sort_pll_degree2
innodb_ft_total_cache_size640000000
innodb_ft_user_stopword_table
innodb_idle_flush_pct100
innodb_immediate_scrub_data_uncompressedOFF
innodb_io_capacity200
innodb_io_capacity_max2000
innodb_lock_schedule_algorithmfcfs
innodb_lock_wait_timeout50
innodb_locks_unsafe_for_binlogOFF
innodb_log_buffer_size1048576
innodb_log_checksumsON
innodb_log_compressed_pagesON
innodb_log_file_size104857600
innodb_log_files_in_group2
innodb_log_group_home_dir.\
innodb_log_optimize_ddlON
innodb_log_write_ahead_size8192
innodb_lru_scan_depth1024
innodb_max_dirty_pages_pct75.000000
innodb_max_dirty_pages_pct_lwm0.000000
innodb_max_purge_lag0
innodb_max_purge_lag_delay0
innodb_max_undo_log_size10485760
innodb_monitor_disable
innodb_monitor_enable
innodb_monitor_reset
innodb_monitor_reset_all
innodb_old_blocks_pct37
innodb_old_blocks_time1000
innodb_online_alter_log_max_size134217728
innodb_open_files500
innodb_optimize_fulltext_onlyOFF
innodb_page_cleaners1
innodb_page_size16384
innodb_prefix_index_cluster_optimizationOFF
innodb_print_all_deadlocksOFF
innodb_purge_batch_size300
innodb_purge_rseg_truncate_frequency128
innodb_purge_threads4
innodb_random_read_aheadOFF
innodb_read_ahead_threshold56
innodb_read_io_threads4
innodb_read_onlyOFF
innodb_replication_delay0
innodb_rollback_on_timeoutOFF
innodb_rollback_segments128
innodb_scrub_logOFF
innodb_scrub_log_speed256
innodb_sort_buffer_size1048576
innodb_spin_wait_delay4
innodb_stats_auto_recalcON
innodb_stats_include_delete_markedOFF
innodb_stats_methodnulls_equal
innodb_stats_modified_counter0
innodb_stats_on_metadataOFF
innodb_stats_persistentON
innodb_stats_persistent_sample_pages20
innodb_stats_sample_pages8
innodb_stats_traditionalON
innodb_stats_transient_sample_pages8
innodb_status_outputOFF
innodb_status_output_locksOFF
innodb_strict_modeON
innodb_sync_array_size1
innodb_sync_spin_loops30
innodb_table_locksON
innodb_temp_data_file_pathibtmp1:12M:autoextend
innodb_thread_concurrency9
innodb_thread_sleep_delay0
innodb_tmpdir
innodb_undo_directory.\
innodb_undo_log_truncateOFF
innodb_undo_logs128
innodb_undo_tablespaces0
innodb_use_atomic_writesON
innodb_use_native_aioON
innodb_version10.3.12
innodb_write_io_threads4

Comment by Stuart Slater [ 2019-05-30 ]

Just wondering if this is fixed in a later version of 10.3.* please?

Comment by Stuart Slater [ 2019-06-06 ]

Hi, Any update on this please? - many thanks.

Comment by Stuart Slater [ 2019-06-18 ]

Hi, Any update on this issue \ bug please ?

Comment by Vladislav Vaintroub [ 2019-06-18 ]

slaterss, this works like this : marko commented previously that there might be a problem with adaptive_hash_index=ON, in version that you are using (10.3.12), and referenced bug MDEV-18815, that is closed with target version 10.3.13

If I were you, I would first run with adaptive_hash_index=OFF, and see if problem if reproducible, or alternatively try out latest 10.3.

Comment by Stuart Slater [ 2019-06-18 ]

Thank you Vladislav Vaintroub for your guidance and patience.
I am not sure if the adaptive_hash_index=OFF will have a major performance impact for us, so I will try the latest version 10.3.15 which has just been released, and report back.

Comment by Stuart Slater [ 2019-06-18 ]

Just reading MDEV-18815 - 10.3.13 suffers the same problem; and does not seem to be fixed.
i have not seen a version that fixes MDEV-18815 , so I will have to try adapative_hash_index=OFF

thanks for your help.

Comment by Vladislav Vaintroub [ 2019-06-18 ]

Please let us know if switching this feature off fixes the problem.

Comment by Stuart Slater [ 2019-07-22 ]

Hi,
I have found by setting adaptive_hash_index=OFF does have an overall performance impact on our main application by at least 10-15%, so is not ideal.
I have not been able to replicate the issue recently with adaptive_hash_index=OFF or adaptive_hash_index=ON, however when I reported it back on the start of June, this was the first time I have come across this problem when we do our monthly truncate of partitions.

As it stands because of the perceived performance impact of switch this adaptive hash index=OFF, we have out it back to ON; is there any timescales for a fix for this problem yet please?

thank you

Comment by Marko Mäkelä [ 2019-09-04 ]

slaterss, ALTER TABLE…TRUNCATE PARTITION should internally use the same logic as TRUNCATE TABLE, which the InnoDB implementation of MariaDB Server since MDEV-13564 (10.2.19, 10.3.11) is internally executing by the equivalent of RENAME TABLE, CREATE TABLE, and DROP TABLE, so that the operation will be compatible with the concurrent execution of mariabackup.

The DROP TABLE part must wait for any adaptive hash index entries for the old partition (or table) to be removed before the data dictionary cache entries for the table can be removed.

I fixed some bugs that were related to the MDEV-13564 rewrite of TRUNCATE. The latest fix that I was able to find was MDEV-17540 in 10.2.25, 10.3.15.

I have a feeling that the InnoDB adaptive hash index was broken in MySQL 5.6 when the field buf_page_t::buf_fix_count ceased to be protected by a mutex. Since that change, MySQL is not only using a mixture of atomic and non-atomic operations on that field, but it could also be that the consistency conditions listed in a comment "Hash search fields" in the source file buf0buf.h could be violated due to race conditions. The value buf_fix_count=0 is special for the adaptive hash index. MariaDB Server may use proper atomic operations on that field, but the problem when the field is changed from or to 0 would still exist in our implementation.

It is very hard to debug the adaptive hash index due to its nondeterministic nature. In MDEV-18815, there is a report of a crash on a DDL operation (TRUNCATE executing DROP) due to corrupted adaptive hash index. For debug builds, there is also MDEV-20203, but it is not failing frequently or consistently enough so that we could fix it.

We conducted some benchmarks in MDEV-17492, which suggest that the adaptive hash index should rarely be useful. In MDEV-20487, I would like to remove the adaptive hash index altogether. Can you describe your workload where the adaptive hash index is bringing that much improvement? Can you post some profiler traces that indicate the bottlenecks when the adaptive hash index is disabled?

Comment by Stuart Slater [ 2019-09-04 ]

Hi,

Thanks for the very comprehensive response; that is very helpful.

in terms of the performance degradation i quoted when disabling the adapative hash indexing; was more of a feel for our general data factory - data processing of large batches of inserts and updates, followed then by throughout the day querying of this data. more so it was also seen on a transactional table that has many inserts throughout a day, with lots of queries from users with concurrency hitting >20 users querying the data in these tables.

There is not much of a definitive profiler \ real statistics I can give; except our end to end application response times seen against the DB , but of course there is never an exact set of circumstances , i.e. each day the pattern is different and each day the concurrency is different. but I did see for a number of days a general slow down. but I only had the hash indexing disabled for a few days. I can disable it again for a longer period to see if I can get a better feel on any performance impact.

we are currently running 10.3.12; i see you have made some bug fixes in the rewrite of Truncate; do you think we would benefit going to the latest stable release of 10.3 , i.e.. 10.3.17 (as this includes your bug fixes in 10.3.15). I have been running with the latest 10.3.* release in our Development environment for a few weeks now and all has been ok; but this is very low usage compared to where we saw this error in our live environment.

Comment by Marko Mäkelä [ 2019-09-05 ]

slaterss, it might be even better to wait a few days for the upcoming 10.3.18 release, which contains a few important InnoDB bug fixes.

I would prefer to proceed with the MDEV-20487 plan and remove the InnoDB adaptive hash index in 10.5, but I guess that it would not hurt to see some more performance data from the field. If you can run the server with both settings for a couple of weeks and try to get some aggregated performance metrics, that could be helpful.

Comment by Stuart Slater [ 2019-09-06 ]

thank you; will wait for the 10.3.18

and also i wil gather some performance data in regards to the adaptive hash index ON \ OFF comparison.

Comment by Marko Mäkelä [ 2019-09-30 ]

slaterss, can you give any update?

Comment by Stuart Slater [ 2019-10-01 ]

Having run with the hash index on \ off for a few each; i do seem to see around a 5% slowdown overall with hash index off.
I am in the process of applying 10.3.18 everywhere as you say this may of fixed the problem due to lots of code changes in around trucate partitition and other partition statements.

thanks

Comment by Marko Mäkelä [ 2019-10-01 ]

slaterss, thank you. Do you have any estimate of the read/write ratio? In MDEV-17492 the conclusion was that the adaptive hash index would mostly be helpful when there are at most 1% writes.

Comment by Stuart Slater [ 2019-10-03 ]

Hi, After looking at some stats for the just a day; our main database tends to have around 5% writes; i.e. 1:20 ratio from Writes:Reads
Thanks

Comment by Marko Mäkelä [ 2019-10-23 ]

slaterss, thank you. axel ran benchmarks where all writes were interleaved with reads, while your use case probably involves a burst of writes and then a long period of mostly reads. It looks like we must preserve the innodb_adaptive_hash_index. But I would disable it by default in 10.5 because of the known bugs.

By the way, did the server upgrade solve the TRUNCATE PARTITION problem for you? That is, can we close this bug (with the original description)?

Comment by Stuart Slater [ 2020-08-11 ]

we are running version 10.3.21 32bit
(i.e, 2020-07-27 1:28:18 0 [Note] InnoDB: 10.3.21 started; log sequence number 281849526018; transaction id 520983628)

we have had a further occurrence of this again.
it has occurred during truncates of partitions of a table that has many rows inserting \ deleting.
2020-07-26 13:41:50 130997 [ERROR] Got error 128 when reading table '.\datafactorydb\rjis_jqi_perf'
no other messages in the log file.

exactly the same as before.

Any ideas when this will be fixed please?

Comment by Stuart Slater [ 2020-08-24 ]

For now on all databases that have partitioned tables \ indexes - i have put in the workaround as stated in:-
https://jira.mariadb.org/browse/MDEV-18815

i.e.

SET GLOBAL innodb_adaptive_hash_index=OFF;

I will report back if the issue reoccurs or not.

Comment by Marko Mäkelä [ 2020-08-24 ]

It is possible that this bug has already been fixed, but I cannot be sure. This year, we have caught and fixed some race conditions in the adaptive hash index, mostly involving buffer pool resizing.

Did you try upgrading to a newer 10.3 version? Also 10.3.21 is rather old. And out of curiosity, is there a specific reason why you are using a 32-bit executable instead of 64-bit?

Also, MDEV-22456 improved the way how the adaptive hash index is dropped. But, note that a fix one regression (MDEV-23452) was not included in any release yet.

Comment by Stuart Slater [ 2020-08-25 ]

Many thanks for your reply marko.makela@mariadb.com
Sorry no; I did not upgrade to 10.3.24 from our current 10.3.21 as I looked at the bugs fixed and nothing stood out that said it fixed this bug or MDEV-18815; I will take a further look at MDEV-22456 thank you.

We have some legacy systems whose OS is still 32-bit , we in the process of moving them all to 64-bit but this is a slow process.

Comment by Mark Reibert [ 2021-12-06 ]

I am wondering if there is anything new on this issue. I continue to get this error in my logs (currently running MariaDB 10.4.22) even with adaptive hash index disabled. The errors often correlate (in time) to when my partitioning script kicks off.

When this error occurs I have seen as many as 28 thousand messages of this type spit out to the logs. It can actually cause a brief service interruption, so this is not just a benign issue.

Do I have latent corruption in my table from the time before I disabled adaptive hash index? I should note none of the partitions that existed when I disabled AHI currently exist.

Do I need to dump the table, drop the table, recreate the table (and its partitions), and restore the data in order to fix this? Is there a less-drastic fix?

Generated at Thu Feb 08 08:53:01 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.