[MDEV-26307] multi-source-replication support mysql syntax(for channel) Created: 2021-08-05  Updated: 2023-03-21  Resolved: 2021-09-14

Status: Closed
Project: MariaDB Server
Component/s: Parser, Replication
Fix Version/s: 10.7.1

Type: Task Priority: Critical
Reporter: woqutech.com Assignee: Michael Widenius
Resolution: Fixed Votes: 1
Labels: mysql, replication

Attachments: File for_chanel_210904.diff     File for_channel.patch     File for_channel_test_210915.diff    
Issue Links:
Relates
relates to MDEV-23912 Orphan files left after failed relay ... Open

 Description   

MariaDB and MySQL both implement the multi-source replication, but their usage and internal implementation are different. In order to facilitate the migration of MySQL users to MariaDB, we want to support MySQL syntax

describe

  • For CHANGE MASTER/FLUSH RELAY LOG/RESET SLAVE/SHOW RELAY LOG EVENT/SHOW SLAVE STATUS/START SLAVE/STOP SLAVE commands support FOR CHANNEL syntax
  • "FOR CHANNEL" is just an alias for connection_name, does not change the original implementation, all behaviors are consistent with MariaDB, so some operations are not compatible with MySQL, For example:
    (MySQL) When a replica has multiple channels and a FOR CHANNEL channel option is not specified, a valid statement generally acts on all available channels, with some specific exceptions.
    (MariaDB) You specify which primary connection you want to work with by either specifying the connection name in the command or setting default_master_connection to the connection you want to work with
  • The channel name supports up to 64 bytes, and is not case sensitive
  • The connection name and for channel syntax cannot be used at the same time, otherwise an error will be reported
  • MariaDB syntax is highly compatible with MySQL, so there is no special SQL_MODE set like Oracle/SQL Server and other databases, FOR CHANNEL is not affected by SQL_MODE

new syntax for MariaDB

CHANGE MASTER

CHANGE MASTER ['connection_name'] TO master_def [, master_def] ... [FOR CHANNEL 'connection_name'] 

FLUSH RELAY LOG

FLUSH RELAY LOGS ['connection_name' | FOR CHANNEL 'connection_name']

RESET SLAVE

RESET SLAVE ['connection_name'] [ALL] [FOR CHANNEL 'connection_name'] 

SHOW RELAY LOG EVENT

SHOW RELAYLOG ['connection_name'] EVENTS [IN 'log_name'] [FROM pos] [LIMIT [offset,] row_count] [ FOR CHANNEL 'connection_name']

SHOW SLAVE STATUS

SHOW SLAVE ['connection_name'] STATUS [FOR CHANNEL 'connection_name'] 

START SLAVE

START SLAVE ['connection_name'] [thread_type [, thread_type] ... ] [FOR CHANNEL 'connection_name']

STOP SLAVE

STOP SLAVE ['connection_name'] [thread_type [, thread_type] ... ] [FOR CHANNEL 'connection_name']



 Comments   
Comment by woqutech.com [ 2021-08-16 ]
  • Without changing the original implementation of MariaDB, it is compatible with MySQL syntax, so we can only modify yacc and lex
  • Define CHANNEL related tokens in yacc and lex
    at sql_yacc.yy

    %token  <kwd>  CHANNEL_SYM
    

    at lex.h

    { "CHANNEL",		SYM(CHANNEL_SYM)},
    

  • Modify the detection of the rule part, and the corresponding results, so that MariaDB performs the same operation after detecting [FOR CHANNEL'channel_name'] and after detecting ['connection_name'].
    define [FOR CHANNEL'channel_name']

optional_for_channel:
        /* empty */
          {
            /*do nothing */
          }
        | for_channel
 
        ;
 
for_channel:
        FOR_SYM CHANNEL_SYM TEXT_STRING_sys
        {
          if (Lex->mi.connection_name.str != NULL)
          {
            my_yyabort_error((ER_WRONG_ARGUMENTS, MYF(0), "CONNECTION_NAME AND FOR CHANNEL CAN NOT BE SPECIFIED AT THE SAME TIME)"));
          }
          else
          {
            Lex->mi.connection_name= $3;
#ifdef HAVE_REPLICATION
           if (unlikely(check_master_connection_name(&$3)))
              my_yyabort_error((ER_WRONG_ARGUMENTS, MYF(0), "MASTER_CONNECTION_NAME"));
#endif
          }
 
         }
         ;

CHANGE MASTER

CHANGE MASTER ['connection_name'] TO master_def [, master_def] ... [FOR CHANNEL 'channel_name'] 

change:
          CHANGE MASTER_SYM optional_connection_name TO_SYM  
          {
            Lex->sql_command = SQLCOM_CHANGE_MASTER;
          } 
          master_defs 
          { }
          optional_for_channel
          { }
        ;

FLUSH RELAY LOGS

FLUSH RELAY LOGS ['connection_name' | FOR CHANNEL 'channel_name']

| RELAY LOGS_SYM optional_connection_name optional_for_channel
{
    LEX *lex= Lex;
    if (unlikely(lex->type & REFRESH_RELAY_LOG))
      my_yyabort_error((ER_WRONG_USAGE, MYF(0), "FLUSH", "RELAY LOGS"));
    lex->type|= REFRESH_RELAY_LOG;
    lex->relay_log_connection_name= lex->mi.connection_name;
}

RESET SLAVE

RESET SLAVE ['connection_name'] [ALL] [FOR CHANNEL 'channel_name'] 

reset_option:
          SLAVE               { Lex->type|= REFRESH_SLAVE; }
          optional_connection_name
          slave_reset_options optional_for_channel
          { }

SHOW RELAYLOG

SHOW RELAYLOG ['connection_name'] EVENTS [IN 'log_name'] [FROM pos] [LIMIT [offset,] row_count] [ FOR CHANNEL 'channel_name']

| RELAYLOG_SYM optional_connection_name EVENTS_SYM binlog_in binlog_from 
{
    LEX *lex= Lex;
    lex->sql_command= SQLCOM_SHOW_RELAYLOG_EVENTS;
}
opt_global_limit_clause optional_for_channel
{ }

SHOW SLAVE STATUS

SHOW SLAVE ['connection_name'] STATUS [FOR CHANNEL 'channel_name'] 

 | SLAVE optional_connection_name STATUS_SYM optional_for_channel 
 {
    if (!(Lex->m_sql_cmd= new (thd->mem_root)
      Sql_cmd_show_slave_status()))
    MYSQL_YYABORT;
    Lex->sql_command = SQLCOM_SHOW_SLAVE_STAT;
 }

START SLAVE

START SLAVE ['connection_name'] [thread_type [, thread_type] ... ] [FOR CHANNEL 'channel_name']

START_SYM SLAVE optional_connection_name slave_thread_opts optional_for_channel
{
    LEX *lex=Lex;
    lex->sql_command = SQLCOM_SLAVE_START;
    lex->type = 0;
    /* If you change this code don't forget to update SLAVE START too */
}
slave_until 
{}

STOP SLAVE

STOP SLAVE ['connection_name'] [thread_type [, thread_type] ... ] [FOR CHANNEL 'channel_name']

| STOP_SYM SLAVE optional_connection_name slave_thread_opts optional_for_channel
{
    LEX *lex=Lex;
    lex->sql_command = SQLCOM_SLAVE_STOP;
    lex->type = 0;
    /* If you change this code don't forget to update SLAVE STOP too */
}

Comment by woqutech.com [ 2021-08-18 ]

for_channel.patch

result of all testcase

The servers were restarted 1785 times
Spent 11415.718 of 1068 seconds executing testcases
 
Completed: All 6262 tests were successful.
 
362 tests were skipped, 254 by the test itself.
 

Comment by woqutech.com [ 2021-08-20 ]

We need to modify the MariaDB Server Documentation , the content is as follows:

Multi-Source Replication

Document modification website: Multi-Source Replication

...
 
 The new syntax introduced to handle many connections:
 
 * <<fixed>>[[change-master-to|CHANGE MASTER ['connection_name'] TO ...]]<</fixed>>.  This creates or modifies a connection to a primary.
-* <<fixed>>FLUSH RELAY LOGS ['connection_name']<</fixed>>
+* <<fixed>>FLUSH RELAY LOGS ['connection_name' | ...] <</fixed>>
 * <<fixed>>[[master_pos_wait|MASTER_POS_WAIT(....,['connection_name'])]]<</fixed>>
-* <<fixed>>[[reset-slave|RESET SLAVE ['connection_name'] [ALL]]]<</fixed>>.  This is used to reset a replica's replication position or to remove a replica permanently.
-* <<fixed>>[[show-relaylog-events|SHOW RELAYLOG ['connection_name'] EVENTS]]<</fixed>>
-* <<fixed>>[[show-slave-status|SHOW SLAVE ['connection_name'] STATUS]]<</fixed>>
+* <<fixed>>[[reset-slave|RESET SLAVE ['connection_name'] [ALL] ...]]<</fixed>>.  This is used to reset a replica's replication position or to remove a replica permanently.
+* <<fixed>>[[show-relaylog-events|SHOW RELAYLOG ['connection_name'] EVENTS ... ]]<</fixed>>
+* <<fixed>>[[show-slave-status|SHOW SLAVE ['connection_name'] STATUS ...]]<</fixed>>
 * <<fixed>>[[show-slave-status|SHOW ALL SLAVES STATUS]]<</fixed>>
-* <<fixed>>[[start-slave|START SLAVE ['connection_name']]...]]<</fixed>>
+* <<fixed>>[[start-slave|START SLAVE ['connection_name']] ...]]<</fixed>>
 * <<fixed>>[[start-slave|START ALL SLAVES ...]]<</fixed>>
 * <<fixed>>[[stop-slave|STOP SLAVE ['connection_name'] ...]]<</fixed>>
 * <<fixed>>[[stop-slave|STOP ALL SLAVES ...]]<</fixed>>
 
...
 
 * If the server variable <<fixed>>log_warnings<</fixed>> > 1 then you will get some information in the log about how the multi-master-info file is updated (mainly for debugging).
 * [[show-slave-status|SHOW [FULL] SLAVE STATUS]] has one line per connection and more columns than before. **Note that the first column is the <<fixed>>connection_name<</fixed>>!**
 * <<fixed>>[[reset-slave|RESET SLAVE]]<</fixed>> now deletes all relay-log files.
+* ['connection_name'] and [FOR CHANNEL 'connection_name'] are equivalent, but cannot be specified at the same time.
 
 == replicate-... Variables
 * One can set the values for the <<fixed>>replicate-...<</fixed>> variables from the command line or in <<fixed>>my.cnf<</fixed>> for a given connection by prefixing the variable with the connection name.

CHANGE MASTER

Document modification website: CHANGE MASTER

...
 
 == Syntax
 
 <<code>>
-CHANGE MASTER ['connection_name'] TO master_def [, master_def] ...
+CHANGE MASTER ['connection_name'] TO master_def [, master_def] ... [FOR CHANNEL 'connection_name'] 
 
 master_def:
     MASTER_BIND = 'interface_name'
 
...
 
 == Multi-Source Replication
 
-If you are using [[multi-source-replication|multi-source replication]], then you need to specify a connection name when you execute ##CHANGE MASTER##. There are two ways to do this:
+If you are using [[multi-source-replication|multi-source replication]], then you need to specify a connection name when you execute ##CHANGE MASTER##. There are three ways to do this:
 
 * Setting the [[replication-and-binary-log-server-system-variables#default_master_connection|default_master_connection]] system variable prior to executing ##CHANGE MASTER##.
 * Setting the ##connection_name## parameter when executing ##CHANGE MASTER##.
+* Setting the ##connection_name## parameter by [FOR CHANNEL 'connection_name'] when executing ##CHANGE MASTER##.
 
+['connection_name'] and [FOR CHANNEL 'connection_name'] are equivalent, but cannot be specified at the same time.
 === ##default_master_connection##
 
 <<code lang=mysql>>
 
...
 
 CHANGE MASTER 'gandalf' TO 
    MASTER_PASSWORD='new3cret';
 START SLAVE 'gandalf';
+
+# You can also use for channel statement
+STOP SLAVE 'gandalf';
+CHANGE MASTER TO 
+   MASTER_PASSWORD='new3cret' FOR CHANNEL 'gandalf';
+START SLAVE 'gandalf';
 <</code>>
 
 == Options
 
...

RESET SLAVE

Document modification website: RESET SLAVE

 <<include slug="replica_slave">>
 == Syntax
 <<code>>
-RESET SLAVE ["connection_name"] [ALL]                
+RESET SLAVE ["connection_name"] [ALL] [FOR CHANNEL 'connection_name']               
 <</code>>
 
 <<toc>>
 
...
 
 The ##connection_name## option was added as part of [[multi-source-replication|multi-source replication]] added in MariaDB 10.0
 <</product>>
 If there is only one nameless master, or the default master (as specified by the [[replication-and-binary-log-server-system-variables#default_master_connection|default_master_connection]] system variable) is intended, ##connection_name## can be omitted. If provided, the ##RESET SLAVE## statement will apply to the specified master. ##connection_name## is case-insensitive.
+['connection_name'] and [FOR CHANNEL 'connection_name'] are equivalent, but cannot be specified at the same time.
 
 ==== RESET REPLICA
 <<product mariadb from=10.5.1>>
 
...

SHOW RELAYLOG EVENTS

Document modification website: SHOW RELAYLOG EVENTS

<<include slug="replica_slave">>
 == Syntax
 
 <<code>>
 SHOW RELAYLOG ['connection_name'] EVENTS
-    [IN 'log_name'] [FROM pos] [LIMIT [offset,] row_count]
+    [IN 'log_name'] [FROM pos] [LIMIT [offset,] row_count] [FOR CHANNEL 'connection_name']
 <</code>>
 
 == Description
 
...
 
 ==== connection_name
 If there is only one nameless primary, or the default primary (as specified by the [[replication-and-binary-log-server-system-variables#default_master_connection|default_master_connection]] system variable) is intended, ##connection_name## can be omitted. If provided, the ##SHOW RELAYLOG## statement will apply to the specified primary. ##connection_name## is case-insensitive.
+['connection_name'] and [FOR CHANNEL 'connection_name'] are equivalent, but cannot be specified at the same time.

SHOW SLAVE STATUS

Document modification website: SHOW SLAVE STATUS

 == Syntax
 
 <<code>>
-SHOW SLAVE ["connection_name"] STATUS
+SHOW SLAVE ["connection_name"] STATUS [FOR CHANNEL 'connection_name']
 SHOW REPLICA ["connection_name"] STATUS -- From MariaDB 10.5.1
 <</code>>
 or
 
...
 
 connection. If ##connection_name## is not used, then the name set by ##default_master_connection## is used. If the connection name doesn't exist you will get an error:
 ##There is no master connection for 'xxx'##.
 
+
+['connection_name'] and [FOR CHANNEL 'connection_name'] are equivalent, but cannot be specified at the same time.
+
 === Column Descriptions
 
 <<style class="darkheader-nospace-borders">>
 
...

START SLAVE

Document modification website: START SLAVE

 <<include slug="replica_slave">>
 == Syntax
 <<code>>
-START SLAVE ["connection_name"] [thread_type [, thread_type] ... ]
+START SLAVE ["connection_name"] [thread_type [, thread_type] ... ] [FOR CHANNEL 'connection_name']
 START SLAVE ["connection_name"] [SQL_THREAD] UNTIL                
     MASTER_LOG_FILE = 'log_name', MASTER_LOG_POS = log_pos
 START SLAVE ["connection_name"] [SQL_THREAD] UNTIL
 
...
 
 ==== connection_name
 If there is only one nameless primary, or the default primary (as specified by the [[replication-and-binary-log-server-system-variables#default_master_connection|default_master_connection]] system variable) is intended, ##connection_name## can be omitted. If provided, the ##START SLAVE## statement will apply to the specified primary. ##connection_name## is case-insensitive.
 
+['connection_name'] and [FOR CHANNEL 'connection_name'] are equivalent, but cannot be specified at the same time.
+
 ==== START ALL SLAVES
 ##START ALL SLAVES## starts all configured replicas (replicas with master_host not empty) that were not started before.  It will give a ##note## for all started connections. You can check the notes with [[show-warnings|SHOW WARNINGS]].
 
...

STOP SLAVE

Document modification website: [STOP SLAVE| https://mariadb.com/kb/en/stop-replica/+edit]

 <<include slug="replica_slave">>
 == Syntax
 <<code>>
-START SLAVE ["connection_name"] [thread_type [, thread_type] ... ]
+START SLAVE ["connection_name"] [thread_type [, thread_type] ... ] [FOR CHANNEL 'connection_name']
 START SLAVE ["connection_name"] [SQL_THREAD] UNTIL                
     MASTER_LOG_FILE = 'log_name', MASTER_LOG_POS = log_pos
 START SLAVE ["connection_name"] [SQL_THREAD] UNTIL
 
...
 
 ==== connection_name
 If there is only one nameless primary, or the default primary (as specified by the [[replication-and-binary-log-server-system-variables#default_master_connection|default_master_connection]] system variable) is intended, ##connection_name## can be omitted. If provided, the ##START SLAVE## statement will apply to the specified primary. ##connection_name## is case-insensitive.
 
+['connection_name'] and [FOR CHANNEL 'connection_name'] are equivalent, but cannot be specified at the same time.
+
 ==== START ALL SLAVES
 ##START ALL SLAVES## starts all configured replicas (replicas with master_host not empty) that were not started before.  It will give a ##note## for all started connections. You can check the notes with [[show-warnings|SHOW WARNINGS]].
 
...

Comment by Michael Widenius [ 2021-08-23 ]

The Jira specification looks ok
Great that you also provided a patch to the documentation!

Looking at patch:
Good test case (well covered)
Patch looks ok

I will merge it to 10.7 ASAP (hopefully later today)

Comment by Michael Widenius [ 2021-08-23 ]

Pushed into bb-10.7-monty for testing and to be checked by Elena.

Comment by Alice Sherepa [ 2021-09-02 ]

I haven't found any problems, that are specific to CHANNEL.

Currently the test multisource_for_channel fails,
show slave status, that is right after FLUSH LOGS still shows the previous file.

 show slave status for channel 'master1'
 Master_Port = 'MYPORT_1'
-Relay_Log_File = 'mysqld-relay-bin-master1.000006'
+Relay_Log_File = 'mysqld-relay-bin-master1.000005'
 Slave_IO_Running = 'Yes'
 Slave_SQL_Running = 'Yes'
 Last_Errno = '0'
@@ -308,7 +308,7 @@
 
 show slave status for channel 'master1'
 Master_Port = 'MYPORT_1'
-Relay_Log_File = 'mysqld-relay-bin-master1.000006'
+Relay_Log_File = 'mysqld-relay-bin-master1.000005'
 Slave_IO_Running = 'No'
 Slave_SQL_Running = 'No'
 Last_Errno = '0'
 
mysqltest: Result content mismatch

Comment by woqutech.com [ 2021-09-04 ]

I am very sorry, because we did not find this problem in the 10.6.4 and 10.7.0 version code, as well as in some different environments, including ubuntu, centos, etc., and even the test in docker did not repeat it.
I think we need information from your test environment to repeat this problem.
Also, in this test we found an omission in the test case, here is our new patch – for_chanel_210904.diff. for_chanel_210904.diff

Comment by Michael Widenius [ 2021-09-07 ]

I have pushed a fix to bb-10.7-monty. Waiting for Alice to verify it fixed the problem

Comment by Michael Widenius [ 2021-09-14 ]

Just about to push the final channel patch to 10.7, but there is one open question:
Why did the patch introduce an test file that only includes an .inc file (doing nothing else) and the inc file is only used in one test?
I think it would be better to remove the include file and put it directly in the test file.
Any comments?

Comment by Elena Stepanova [ 2021-09-14 ]

It looks like a copy-paste issue. The include file was clearly derived from multisource.inc.
multisource.inc is a separate file because it's used in two suites – multisource and binlog_encryption. The comment in the new include file still refers to the same reason, only apparently there isn't actual multisource_channel.test in the binlog_encryption suite, neither do I think it's worth adding one. So, it should of course be fine to move the logic directly into the test, of course the comments related to the usage of the include file should be dropped then.

Comment by woqutech.com [ 2021-09-14 ]

I’m sorry, because of my negligence, this problem was left over.
Your idea is right, we can move the logic directly to the test, and delete the comments related to the use of the include file.
To make sure it’s correct, I move the logic to the test and test it again.

Comment by woqutech.com [ 2021-09-15 ]

for_channel_test_210915.diff

Generated at Thu Feb 08 09:44:19 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.