Uploaded image for project: 'MariaDB MaxScale'
  1. MariaDB MaxScale
  2. MXS-2264

Schema aware avrorouter ignores table renames

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Closed (View Workflow)
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 2.2.17
    • Fix Version/s: 2.4.3
    • Component/s: avrorouter, cdc
    • Labels:
      None
    • Sprint:
      MXS-SPRINT-91

      Description

      We rely on https://github.com/github/gh-ost to make online schema changes in our MariaDB OLTP environment without downtime.

      gh-ost like many other tools of this kind relies on atomic table renames in its operation.

      It is critical for Maxscale CDC to process table renames similarly to ALTER TABLE statements, generate new/updated AVRO spec and continue converting events.

      In current Maxscale version, renaming the table effectively breaks subsequent avro conversion.

      To reproduce:

      CREATE TABLE `rename_1` (
        `some_id` mediumint(8) unsigned DEFAULT NULL
      ) ENGINE=MyISAM DEFAULT CHARSET=utf8mb4;
       
      INSERT INTO `rename_1` (`some_id`)
      VALUES
      	(1),
      	(2);
       
       
      CREATE TABLE `rename_2` (
        `some_id` mediumint(8) unsigned DEFAULT NULL,
        `additional_field` mediumint(8) unsigned DEFAULT NULL
      ) ENGINE=MyISAM DEFAULT CHARSET=utf8mb4;
       
      INSERT INTO `rename_2` (`some_id`)
      VALUES
      	(1),
      	(2);
       
      RENAME TABLE rename_1 TO rename_old, rename_2 TO rename_1;
       
      INSERT INTO `rename_1` (`some_id`, `additional_field`)
      VALUES
      	(3, 1);
       
      INSERT INTO `rename_1` (`some_id`, `additional_field`)
      VALUES
      	(4, 2);
      

      Table contents are:

      some_id	additional_field
      1	NULL
      2	NULL
      3	1
      4	2
      

      JSON output as follows and last two inserts (some_id =3 and some_id=4) never make it to avro file:

      {"namespace": "MaxScaleChangeDataSchema.avro", "type": "record", "name": "ChangeRecord", "fields": [{"name": "domain", "type": "int"}, {"name": "server_id", "type": "int"}, {"name": "sequence", "type": "int"}, {"name": "event_number", "type": "int"}, {"name": "timestamp", "type": "int"}, {"name": "event_type", "type": {"type": "enum", "name": "EVENT_TYPES", "symbols": ["insert", "update_before", "update_after", "delete"]}}, {"name": "some_id", "type": ["null", "int"], "real_type": "mediumint", "length": 8}]}
      {"domain": 0, "server_id": 11009, "sequence": 9784476, "event_number": 1, "timestamp": 1547530357, "event_type": "insert", "some_id": 1}
      {"domain": 0, "server_id": 11009, "sequence": 9784500, "event_number": 1, "timestamp": 1547530360, "event_type": "insert", "some_id": 2}
      

      Also, performing following insert into brand new table created by same rename does not result in avro spec generation for table `rename_old` so this new table gets completely ignored by Maxscale:

      INSERT INTO `rename_old` (`some_id`)
      VALUES
      	(7);
      

        Attachments

          Activity

            People

            Assignee:
            markus makela markus makela
            Reporter:
            sergiy.kyrychenko Sergiy Kyrychenko
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Dates

              Created:
              Updated:
              Resolved: