[MCOL-4936] Disable bingloging for MCS tables running DML/bulk insertion(LDIF) Created: 2021-12-03  Updated: 2022-03-29  Resolved: 2022-02-18

Status: Closed
Project: MariaDB ColumnStore
Component/s: MDB Plugin
Affects Version/s: 6.2.1
Fix Version/s: 6.3.1

Type: Task Priority: Blocker
Reporter: Roman Assignee: Daniel Lee (Inactive)
Resolution: Fixed Votes: 0
Labels: None

Attachments: File binlogTestSQL.sql     File binlogtesttable.ddl    
Issue Links:
Relates
relates to MCOL-4960 Allow a user to enable DML binlog rep... Open
Sprint: 2021-16, 2021-17

 Description   

LOAD DATA IN FILE and DML are the only ways to ingest data into MCS tables in the cloud. Both LDIF and DML produces binlog entries for MCS tables. The binlogs themselves doesn't make any sense in MCS cluster b/c of its distributed storage and also binlogs can take a lot of disk space in case of LDIF. The immediate solution is to disable binlogs for DML/bulk insertion for MCS tables.

 Comments   
Comment by Gagan Goel (Inactive) [ 2022-01-04 ]

For QA: Set up an MCS cluster as usual and test the queries mentioned in the issue description:

  1. Test to make sure entries are not written to the binlog for the LDI statement.
  2. Test to make sure entries are not written to the binlog for other DML statements (INSERT/UPDATE/DELETE/INSERT..SELECT).
  3. Test that DDLs are replicated as usual to the replica nodes (binlog entries are written to the primary node for DDL statements).

Also test that the HTAP topology (InnoDB primary replicating to a replica MCS cluster) is not affected, i.e. the DMLs performed on the InnoDB table are replicated successfully on the replica MCS cluster. For the DMLs to be replicated in the MCS cluster, the system variable columnstore_replication_slave should be set to ON in the master node in the MCS cluster.

Comment by Gagan Goel (Inactive) [ 2022-01-10 ]

NOTE With the fix for this issue, not writing to primary node's binlog in a ColumnStore cluster means DML replication from a ColumnStore cluster to another ColumnStore cluster or to another foreign engine will not work.

Comment by Daniel Lee (Inactive) [ 2022-01-10 ]

Build tested: 6.2.3-1 (#3651)

Reproduced the behavior the binlog growing behavior for LDI in 6.2.1
Verified that LDI is no longer log to binlog after this fix.
Verified replication continues to work between clusters (innodb-to-innodb and innodb-to-columnstore)

DML comands INSERT, UPDATE, DELETE are not being logged to the binlog, but 'INSERT..SELECT' and 'INSERT INTO..SELECT' are being logged. Hence, reopening the ticket.

Comment by Roman [ 2022-01-11 ]

Good point about RD allen.herrera. I don't think binlog replication via MDB would be a scalable solution but it can be a first cut of the feature.
We will discuss how to enable/disable binlog replication using MDB session variable, e.g. columnstore_binglog.

Comment by Gagan Goel (Inactive) [ 2022-01-12 ]

allen.herrera and drrtuy I have created MCOL-4960 to allow a user to enable back binlog for DMLs statements for cross-cluster replication.

Comment by Daniel Lee (Inactive) [ 2022-01-12 ]

It would be more natural and intuitive for this MCOL-4936 to be configurable instead, instead of disabling by default and have another variable to enable it back.

Comment by Gagan Goel (Inactive) [ 2022-02-15 ]

dleeyh I have confirmed that none of the DML statements, including INSERT .. SELECT and INSERT INTO SELECT (in addition to LDI/INSERT/UPDATE/DELETE) are getting logged to the primary node's binlog. I am assigning the ticket back to you to retest.

Comment by Daniel Lee (Inactive) [ 2022-02-18 ]

Build verified: 6.3.1 (b3885)

Generated at Thu Feb 08 02:54:07 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.