Uploaded image for project: 'MariaDB MaxScale'
  1. MariaDB MaxScale
  2. MXS-1081

Avro data file corruption

    XMLWordPrintable

Details

    • Bug
    • Status: Closed (View Workflow)
    • Major
    • Resolution: Fixed
    • 2.0.3
    • 2.0.5, 2.1.1
    • avrorouter
    • None
    • Ubuntu 14.04.5 LTS. mysql Ver 15.1 Distrib 10.1.20-MariaDB
    • 2017-26, 2017-27, 2017-28, 2017-29

    Description

      We set up maxscale on it's own server to connect to our galera cluster. Pretty straightforward configuration taken directly from your guides:

      https://gist.github.com/Geesu/949a9660a3068e9f48b2877ec395da06

      We then used cdc_schema.go to create schemas from our existing mariadb database (we don't have all binlogs to replay).

      So far so good, but once we started up maxscale the data that was created in the avro files didn't make any sense whatsoever. The data was being corrupted somehow, and I suspect it is in the avrorouter.

      An example avro file converted to JSON w/json-tools:
      java -jar ~/avro-tools-1.8.1.jar tojson ebth_production.followed_items_users.000001.avro

      {{"domain":0,"server_id":1,"sequence":28480571,"event_number":1,"timestamp":1484234657,"event_type":"insert","user_id":2158615,"item_id":4966469,"id":149400690,"deleted":0,"created_at":"34568-489768447-32668 489689435:32668:446036064","updated_at":"34568-489768447-32668 489689435:32668:446036064"}
      {"domain":0,"server_id":1,"sequence":28480571,"event_number":2,"timestamp":1484234657,"event_type":"insert","user_id":2158615,"item_id":1480089267,"id":18,"deleted":0,"created_at":"34568-489768447-32668 489689435:32668:446036064","updated_at":"34568-489768447-32668 489689435:32668:446036064"}}

      This data is all garbage. For other avro files the strings are completely mangled as well. Although if you cat the file you can sometimes see text that should be on the record (but it won't be the entire string).

      I've attached the schema and the avro file.

      I suspected it was using snappy but when using the getmeta command (avro tools) it says null:

      avro.codec null
      avro.schema {"type":"record","name":"ChangeRecord","namespace":"MaxScaleChangeDataSchema.avro","fields":[{"name":"domain","type":{"type":"int"}},{"name":"server_id","type":{"type":"int"}},{"name":"sequence","type":{"type":"int"}},{"name":"event_number","type":{"type":"int"}},{"name":"timestamp","type":{"type":"int"}},{"name":"event_type","type":{"type":"enum","name":"EVENT_TYPES","symbols":["insert","update_before","update_after","delete"]}},{"name":"user_id","type":{"type":"int"}},{"name":"item_id","type":{"type":"int"}},{"name":"id","type":{"type":"int"}},{"name":"deleted","type":{"type":"int"}},{"name":"created_at","type":{"type":"string"}},{"name":"updated_at","type":{"type":"string"}}]}

      Attachments

        1. ebth_production.followed_items_users.000001.avro
          1.0 kB
          Josh Becker
        2. ebth_production.followed_items_users.000001.avsc
          0.3 kB
          Josh Becker
        3. ebth_production.items.000001.avro
          47 kB
          Josh Becker
        4. ebth_production.items.000001.avro
          5 kB
          Josh Becker
        5. ebth_production.items.000001.avsc
          2 kB
          Josh Becker
        6. items.sql
          3 kB
          Josh Becker
        7. my.cnf
          5 kB
          Josh Becker

        Activity

          People

            markus makela markus makela
            geesu Josh Becker
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Git Integration

                Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.