Uploaded image for project: 'MariaDB Server'
  1. MariaDB Server
  2. MDEV-9383

Server fails to read master.info after upgrade 10.0 -> 10.1

Details

    • Bug
    • Status: Closed (View Workflow)
    • Major
    • Resolution: Fixed
    • 10.1.10, 10.0(EOL), 10.1(EOL)
    • 10.1.14
    • Replication
    • None
    • Linux

    Description

      I tried to upgrade the server 10.0.22 -> 10.1.10.
      Old version was shutdown, but new failed to start:

      2016-01-08 22:25:00 7f8017479700 InnoDB: Error: Column last_update in table "mysql"."innodb_table_stats" is INT UNSIGNED NOT NULL but should be BINARY(4) NOT NULL (type mismatch).
      2016-01-08 22:25:00 7f8017479700 InnoDB: Error: Fetch of persistent statistics requested for table "mysql"."gtid_slave_pos" but the required system tables mysql.innodb_table_stats and mysql.innodb_index_stats are not present or have unexpected structure. Using transient stats instead.
      2016-01-08 22:25:01 140188123334592 [Note] Server socket created on IP: '::'.
      2016-01-08 22:25:01 140188123334592 [ERROR] Failed to initialize master info using_gtid
      2016-01-08 22:25:01 140188123334592 [ERROR] Error reading master configuration
      2016-01-08 22:25:01 140188123334592 [ERROR] Initialized Master_info from 'master-bob@002dhk.info' failed
      2016-01-08 22:25:01 140188123334592 [ERROR] Failed to initialize master info using_gtid
      2016-01-08 22:25:01 140188123334592 [ERROR] Error reading master configuration
      2016-01-08 22:25:01 140188123334592 [ERROR] Initialized Master_info from 'master-bob@002dth.info' failed
      2016-01-08 22:25:01 140188123334592 [ERROR] Failed to initialize master info using_gtid
      2016-01-08 22:25:01 140188123334592 [ERROR] Error reading master configuration
      2016-01-08 22:25:01 140188123334592 [ERROR] Initialized Master_info from 'master-bob@002dsg.info' failed
      2016-01-08 22:25:01 140188123334592 [ERROR] Failed to initialize master info using_gtid
      2016-01-08 22:25:01 140188123334592 [ERROR] Error reading master configuration
      2016-01-08 22:25:01 140188123334592 [ERROR] Initialized Master_info from 'master-delivery.info' failed
      2016-01-08 22:25:01 140188123334592 [Warning] Reading of some Master_info entries failed
      2016-01-08 22:25:01 140188123334592 [ERROR] Failed to initialize multi master structures
      2016-01-08 22:25:01 140188123334592 [ERROR] Aborting
      

      I guess I can't use mysql_upgrade when the server is not running.

      Attachments

        1. log.txt
          23 kB
        2. master.info.example
          0.1 kB
        3. master-info.txt
          4 kB
        4. MDEV-9383-100grama.sample
          0.5 kB
        5. MDEV-9383-ck.sample
          0.7 kB
        6. MDEV-9383-miss60.sample
          0.9 kB
        7. mysqld.cnf
          1 kB
        8. relay.ls.txt.gz
          34 kB

        Activity

          BTW, the issue with leaving extra stuff at the end of master.info is fixed in 10.1 with END_MARKER.

          knielsen Kristian Nielsen added a comment - BTW, the issue with leaving extra stuff at the end of master.info is fixed in 10.1 with END_MARKER.
          lukav Anton Avramov added a comment -

          MDEV-9383-100grama.sample MDEV-9383-ck.sample MDEV-9383-miss60.sample

          I've compiled the ending of the .info files in some of our clients multi-masters.
          I've used the command: tail n 6 /var/lib/mysql/master*.info > /tmp/MDEV-9383-[host].sample

          I think this would give you a good samples of the "damages" the bug made

          lukav Anton Avramov added a comment - MDEV-9383-100grama.sample MDEV-9383-ck.sample MDEV-9383-miss60.sample I've compiled the ending of the .info files in some of our clients multi-masters. I've used the command: tail n 6 /var/lib/mysql/master *.info > /tmp/ MDEV-9383 - [host] .sample I think this would give you a good samples of the "damages" the bug made
          knielsen Kristian Nielsen added a comment - Suggested fix: http://lists.askmonty.org/pipermail/commits/2016-April/009251.html

          BTW, what happens here is that when a new master.info file is written, it
          does not truncate the file (for performance reasons).

          So if the new file happens to be shorter, extra junk is left at the
          end. This has always been so, but it has become more of a problem now that
          MySQL and MariaDB are each adding new lines to the file without
          coordinating.

          In 10.1, a line "END_MARKER" is added at the bottom to easily avoid reading
          any left-over junk, but this change was deemed too dangerous for stable 10.0
          (which is fine, since 10.0 is not going to add anothing more to
          master.info).

          But then apparently some bugs were introduced with the previously mentioned
          patch, which could cause 10.1 to incorrectly interpret some cases of extra
          junk left by 10.0, causing the problems seen here. In particular, an empty
          line was treated as an empty "using_gtid=" line, causing the failure.

          An easy work-around, until a fixed 10.1 release becomes available, is to
          just remove anything after the "using_gtid=X" line (but do not remove the
          newline character at the end of that line).

          knielsen Kristian Nielsen added a comment - BTW, what happens here is that when a new master.info file is written, it does not truncate the file (for performance reasons). So if the new file happens to be shorter, extra junk is left at the end. This has always been so, but it has become more of a problem now that MySQL and MariaDB are each adding new lines to the file without coordinating. In 10.1, a line "END_MARKER" is added at the bottom to easily avoid reading any left-over junk, but this change was deemed too dangerous for stable 10.0 (which is fine, since 10.0 is not going to add anothing more to master.info). But then apparently some bugs were introduced with the previously mentioned patch, which could cause 10.1 to incorrectly interpret some cases of extra junk left by 10.0, causing the problems seen here. In particular, an empty line was treated as an empty "using_gtid=" line, causing the failure. An easy work-around, until a fixed 10.1 release becomes available, is to just remove anything after the "using_gtid=X" line (but do not remove the newline character at the end of that line).

          Pushed to 10.1

          knielsen Kristian Nielsen added a comment - Pushed to 10.1

          People

            knielsen Kristian Nielsen
            ip1981 Igor Pashev
            Votes:
            1 Vote for this issue
            Watchers:
            5 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Git Integration

                Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.