Rafael Gallastegui added a comment - 2017-03-29 15:30 - edited My config file: # # This group is read both both by the client and the server # use it for options that affect everything # [client-server] # # This group is read both both by the client and the server # use it for options that affect everything # [client-server] # # These groups are read by MariaDB server. # Use it for options that only the server (but not clients) should see # # See the examples of server my.cnf files in /usr/share/mysql/ # # this is read by the standalone daemon and embedded servers [server] tmpdir = /var/lib/mysql/tmpdir   # Disable binary logs, since we don't have replication running here   #log-bin = binlog #binlog_format = mixed server_id = 1 #log-basename = server1 expire-logs-days = 7 # this is only for the mysqld standalone daemon [mysqld] transaction-isolation = READ-COMMITTED ignore_db_dirs=lost+found performance-schema=on # Set host name for reports report_host = '172.16.14.241' default_time_zone='America/Chicago' # Set default character set and collation character_set_server = 'latin1' collation-server = 'latin1_bin' max_allowed_packet = 512M # Corresponding value for replication slave_max_allowed_packet = 512M # log file location log_error=/var/log/mysqld.log # directory for the data datadir=/var/lib/mysql/ # Max number of opened connections max_connections=500 # Perf tuning thread_cache_size=512 # Increased from the default value to handle large number of tables table_open_cache=30000 # Max number of open files. It has to correlate to an OS change open_files_limit = 102400 # INNODB OPTIONS # Buffer Pool Size - (IMPORTANT - DEPENDS on MEMORY!!! - 70 to 80% of memory is recommended) #innodb_buffer_pool_size = 4G # Use one file per table in the schema innodb_file_per_table=1 # lock timeout innodb_lock_wait_timeout = 300 # Perf improvement to decrease flushing innodb_log_file_size=512M # Decreases I/O to disk for larger transaction innodb_log_buffer_size=64M innodb_buffer_pool_instances=16 innodb_log_files_in_group=2 # Recommended twice CPUs plus number of disks innodb_thread_concurrency=10 # Next 2 Recommended to ensure data consistency innodb_flush_method = O_DIRECT innodb_doublewrite=1 ## Avoid statistics update when doing some ops, e.g. show tables innodb_stats_on_metadata=0 default_storage_engine=innodb

Rafael Gallastegui added a comment - 2017-03-29 19:42 Don't know if it's relevant, but the IBD files came from a 10.1.19 server

John W Smith added a comment - 2017-04-03 19:36 - edited I can confirm that the same happens on a Galera cluster in our environment after upgrading to 10.1.22 (ius.io repos on CentOS 7). This unfortunately also causes a cascade in the cluster and all nodes go down. Downgrading to 10.1.21 seems to have resolved the issue for us.

Marko Mäkelä added a comment - 2017-05-09 12:16 This could be a regression caused by MDEV-11759 , which was released in MariaDB 10.1.22. In MariaDB 10.1.23 there are multiple fixes to encryption; look at the related issues listed in MDEV-12602 .

Marko Mäkelä added a comment - 2017-05-09 12:17 This report could be a duplicate of MDEV-12419 which was fixed as part of MDEV-12253 . Does the large import work with MariaDB 10.1.23?

Marko Mäkelä added a comment - 2017-05-17 05:40 On a somewhat related note, Valgrind reported an error for a memcpy() operation that is executed when importing encrypted files: http://buildbot.askmonty.org/buildbot/builders/work-amd64-valgrind/builds/10056/steps/test/logs/stdio encryption.innodb-discard-import-change 'cbc,innodb' w2 [ fail ] Found warnings/errors in server log file! Test ended at 2017-05-17 07:42:11 line ==31112== Thread 27: ==31112== Source and destination overlap in memcpy(0x11bb8000, 0x11bb8000, 16384) ==31112== at 0x4C2C51C: memcpy@@GLIBC_2.14 (vg_replace_strmem.c:1018) ==31112== by 0x10048A5: fil_iterate(fil_iterator_t const&, buf_block_t*, PageCallback&) (fil0fil.cc:6047) ==31112== by 0x1005392: fil_tablespace_iterate(dict_table_t*, unsigned long, PageCallback&) (fil0fil.cc:6283) ==31112== by 0xE03456: row_import_for_mysql(dict_table_t*, row_prebuilt_t*) (row0import.cc:3496) I do not think that the buffers should legitimately overlap in any case here: if (encrypted) { memcpy(writeptr + (i * size), row_compressed ? block->page.zip.data : block->frame, size); } I was unable to repeat this failure locally. The arithmetics done to n_bytes in fil_iterate() look a bit suspicious to me. Even more alarming is this code: byte* io_buffer = iter.io_buffer; block->frame = io_buffer;   if (row_compressed) { … block->page.zip.data = block->frame + UNIV_PAGE_SIZE; … io_buffer = block->page.zip.data; } Could block->page.zip.data point past the end of the iter.io_buffer array, overlapping with writeptr (which points to iter.crypt_io_buffer)?

Marko Mäkelä added a comment - 2018-03-19 15:20 - edited I pushed some clean-up to 10.1 so that page checksums will be validated in one function. I did not find anything that could explain this crash.

Marko Mäkelä added a comment - 2018-03-21 21:11 I merged the clean-up to 10.2, so it will be in both 10.1.32 and 10.2.14 (and 10.3.6). The main functional difference should be that IMPORT no longer fills the error log with an unbounded number of messages about retrying partial page reads when you try to import an .ibd file that is shorter than innodb_page_size . For 10.2, there is also the difference that IMPORT will now continue to read and write multiple pages at a time, like it used to work. page_compressed tables will grow during import; it looks like hole-punching has not been implemented yet. The originally reported error ought to occur somewhere in the final page flushing after doing some final adjustments through the buffer pool. rgallastpsi , I wonder if you have enabled innodb_use_mtflush , which is known to cause occasional hangs ( MDEV-10843 ); maybe it could cause crashes as well? In MariaDB 10.2, an alternative to innodb_use_mtflush was provided by innodb_page_cleaners , and accordingly, innodb_use_mtflush was deprecated in 10.2 ( MDEV-13674 ) and removed in 10.3 ( MDEV-13690 ).

Marko Mäkelä added a comment - 2020-01-24 08:15 rgallastpsi , I just noticed that this ticket is still open. We made significant improvements to InnoDB table encryption in MariaDB, especially in 10.1.23,10.2.6 but also in later. Do you still experience this kind of problems with more recent versions?