[MCOL-4207] Cross storage engine replication innodb - ColumnStore Created: 2020-07-23  Updated: 2021-01-14  Resolved: 2021-01-14

Status: Closed
Project: MariaDB ColumnStore
Component/s: MariaDB Server
Affects Version/s: 1.5.2
Fix Version/s: N/A

Type: Bug Priority: Minor
Reporter: Matias Sanchez Assignee: Unassigned
Resolution: Not a Bug Votes: 1
Labels: None
Environment:

Ubuntu 18.04 with single cluster columnstore
https://mariadb.com/docs/deploy/community-single-columnstore-cs105-ubuntu18/#install-community-single-columnstore-col15-cs105-ubuntu18



 Description   

Hi !! We're on process of implementing an cross storage engine native replication between Innodb and Columnstore having source database as Mariadb Server 10.5.4 and target database as Mariadb Server 10.5.4 with Columnstore storage engine plugin 1.5.2, these are latests release on both products. In the process we adapt tables syntax and import data on target columnstore with cpimport, but once the replication is configured in this cross-storage engine manner, it works correctly for a little while but eventually gets crashed with following errors:

.............

_
terminate called after throwing an instance of 'std::logic_error'
what(): basic_string::_M_construct null not valid
200723 1:48:37 [ERROR] mysqld got signal 6 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.

To report this bug, see https://mariadb.com/kb/en/reporting-bugs

We will try our best to scrape up some info that will hopefully help
diagnose the problem, but since we have already crashed,
something is definitely wrong and this may fail.

Server version: 10.5.4-MariaDB-1:10.5.4+maria~bionic-log
key_buffer_size=134217728
read_buffer_size=131072
max_used_connections=1
max_threads=153
thread_count=5
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 467802 K bytes of memory
Hope that's ok; if not, decrease some variables in the equation.

Thread pointer: 0x7f00700008e8
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 0x7f00e8080658 thread_stack 0x49000
??:0(my_print_stacktrace)[0x556cd71aa6ee]
??:0(handle_fatal_signal)[0x556cd6b9fb05]
??:0(__restore_rt)[0x7f00ebeb0890]
??:0(gsignal)[0x7f00eb1c0e97]
??:0(abort)[0x7f00eb1c2801]
/usr/lib/x86_64-linux-gnu/libstdc++.so.6(+0x8c957)[0x7f00eb99d957]
??:0(std::rethrow_exception(std::__exception_ptr::exception_ptr))[0x7f00eb9a3ae6]
??:0(std::terminate())[0x7f00eb9a3b21]
??:0(__cxa_throw)[0x7f00eb9a3d54]
/usr/lib/x86_64-linux-gnu/libstdc++.so.6(+0x8e79f)[0x7f00eb99f79f]
??:0(handler::change_table_ptr(TABLE*, TABLE_SHARE*))[0x7f00e898f93b]
??:0(ha_mcs_impl_rnd_init(TABLE*, std::vector<Item*, std::allocator<Item*> > const&))[0x7f00e89a4d1c]
??:0(ha_mcs::rnd_init(bool))[0x7f00e898cc77]
??:0(handler::ha_rnd_init_with_error(bool))[0x556cd6ba6369]
??:0(Rows_log_event::find_row(rpl_group_info*))[0x556cd6caae76]
??:0(Update_rows_log_event::do_exec_row(rpl_group_info*))[0x556cd6cab3eb]
??:0(Rows_log_event::do_apply_event(rpl_group_info*))[0x556cd6c9f4ec]
??:0(Item_string::get_copy(THD*))[0x556cd68f4ce2]
??:0(handle_slave_sql)[0x556cd68fda0e]
??:0(MyCTX_nopad::finish(unsigned char*, unsigned int*))[0x556cd6ddea0a]
??:0(start_thread)[0x7f00ebea56db]
??:0(clone)[0x7f00eb2a388f]
_
.............

After this happens, MariaDB target replication tries crash recovery intermittently always with the same error and can't recover anymore.
Tried resetting environments multiple times with same error, tried setting columnstore_replication_slave on both sides of replication, have no clue what gets wrong.
As this type of cross storage engine is not on documentation at the moment, is it possible?? have any indication of best practices on configuration on target and replica if this is feasible??

Thanks you so much in advance.



 Comments   
Comment by Todd Stoffel (Inactive) [ 2020-07-27 ]

matias.sanchez@avature.net Can you please post the output of:

SHOW VARIABLES LIKE 'binlog_format';

And compare that to the documentation here:

https://mariadb.com/docs/deploy/enterprise-htap/#configuring-mariadb

Comment by Matias Sanchez [ 2020-07-27 ]

Hi Todd, thanks you for your fast reply, I was able to set replication strictly with binlog_format=STATEMENT as until now ROW format it's not supported as it's detailed on forwarded documentation, so this issue may be closed.
Anyway, right away we faced another big issue on performance regression when trying to sync any data from Innodb to Columnstore,

I have opened another issue for this as I could imagine it's a different problem https://jira.mariadb.org/browse/MCOL-4215 any recommendation for tuning replication for this Hybrid Transactional/Analytical Processing (HTAP) solution?

Generated at Thu Feb 08 02:48:36 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.