Uploaded image for project: 'MariaDB Server'
  1. MariaDB Server
  2. MDEV-10754

wsrep_sst_rsync does not support innodb_data_home_dir

Details

    Description

      If you have an entry in your configuration file like this:

      innodb_data_home_dir=/innodbdata
      

      Then rsync SSTs will fail.

      The first time a joiner attempts to SST, the node will see log entries like this, showing that a new InnoDB data file is being created after the SST:

      2016-09-06 15:47:41 140177158698752 [Note] WSREP: Running: 'wsrep_sst_rsync --role 'joiner' --address '172.31.22.174' --datadir '/var/lib/mysql/'   --parent '1957' --binlog '/var/mysql/binlogs/mariadb-bin' '
      2016-09-06 15:47:41 140177453222656 [Note] WSREP: Prepared SST request: rsync|172.31.22.174:4444/rsync_sst
      2016-09-06 15:47:41 140177453222656 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
      2016-09-06 15:47:41 140177453222656 [Note] WSREP: REPL Protocols: 7 (3, 2)
      2016-09-06 15:47:41 140177231623936 [Note] WSREP: Service thread queue flushed.
      2016-09-06 15:47:41 140177453222656 [Note] WSREP: Assign initial position for certification: 0, protocol version: 3
      2016-09-06 15:47:41 140177231623936 [Note] WSREP: Service thread queue flushed.
      2016-09-06 15:47:41 140177453222656 [Warning] WSREP: Failed to prepare for incremental state transfer: Local state UUID (71de705d-6faa-11e6-9a78-8295746971ef) does not match group state UUID (bae8d70c-746a-11e6-869f-632ad20bfb5c): 1 (Operation not permitted)
               at galera/src/replicator_str.cpp:prepare_for_IST():482. IST will be unavailable.
      2016-09-06 15:47:41 140177175471872 [Note] WSREP: Member 1.0 () requested state transfer from '*any*'. Selected 0.0 (ip-172-31-19-192.us-west-2.compute.internal)(SYNCED) as donor.
      2016-09-06 15:47:41 140177175471872 [Note] WSREP: Shifting PRIMARY -> JOINER (TO: 0)
      2016-09-06 15:47:41 140177453222656 [Note] WSREP: Requesting state transfer: success, donor: 0
      2016-09-06 15:47:43 140177183864576 [Note] WSREP: (c52d1933, 'tcp://0.0.0.0:4567') turning message relay requesting off
      2016-09-06 15:47:44 140177175471872 [Note] WSREP: 0.0 (ip-172-31-19-192.us-west-2.compute.internal): State transfer to 1.0 () complete.
      2016-09-06 15:47:44 140177175471872 [Note] WSREP: Member 0.0 (ip-172-31-19-192.us-west-2.compute.internal) synced with group.
      WSREP_SST: [INFO] Extracting binlog files: (20160906 15:47:44.804)
      mariadb-bin.000006
      WSREP_SST: [INFO] Joiner cleanup. rsync PID: 1997 (20160906 15:47:44.815)
      WSREP_SST: [INFO] Joiner cleanup done. (20160906 15:47:45.320)
      2016-09-06 15:47:45 140177453537408 [Note] WSREP: SST complete, seqno: 0
      2016-09-06 15:47:45 140177453537408 [Note] InnoDB: Using mutexes to ref count buffer pool pages
      2016-09-06 15:47:45 140177453537408 [Note] InnoDB: The InnoDB memory heap is disabled
      2016-09-06 15:47:45 140177453537408 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
      2016-09-06 15:47:45 140177453537408 [Note] InnoDB: Memory barrier is not used
      2016-09-06 15:47:45 140177453537408 [Note] InnoDB: Compressed tables use zlib 1.2.7
      2016-09-06 15:47:45 140177453537408 [Note] InnoDB: Using Linux native AIO
      2016-09-06 15:47:45 140177453537408 [Note] InnoDB: Using SSE crc32 instructions
      2016-09-06 15:47:45 140177453537408 [Note] InnoDB: Initializing buffer pool, size = 128.0M
      2016-09-06 15:47:45 140177453537408 [Note] InnoDB: Completed initialization of buffer pool
      2016-09-06 15:47:45 140177453537408 [Note] InnoDB: The first specified data file /innodbdata/ibdata1 did not exist: a new database to be created!
      2016-09-06 15:47:45 140177453537408 [Note] InnoDB: Setting file /innodbdata/ibdata1 size to 12 MB
      2016-09-06 15:47:45 140177453537408 [Note] InnoDB: Database physically writes the file full: wait...
      2016-09-06 15:47:45 140177453537408 [Note] InnoDB: Setting log file ./ib_logfile101 size to 48 MB
      2016-09-06 15:47:46 140177453537408 [ERROR] InnoDB: Cannot create ./ib_logfile1
      2016-09-06 15:47:46 140177453537408 [ERROR] Plugin 'InnoDB' init function returned error.
      2016-09-06 15:47:46 140177453537408 [ERROR] Plugin 'InnoDB' registration as a STORAGE ENGINE failed.
      2016-09-06 15:47:46 140177453537408 [Note] Plugin 'FEEDBACK' is disabled.
      2016-09-06 15:47:46 140177453537408 [ERROR] Unknown/unsupported storage engine: InnoDB
      2016-09-06 15:47:46 140177453537408 [ERROR] Aborting
      

      And then every time the joiner performs an SST after the first, the error log will show issues with the InnoDB tablespace:

      2016-09-06 15:51:15 139831426414336 [Note] WSREP: Running: 'wsrep_sst_rsync --role 'joiner' --address '172.31.22.174' --datadir '/var/lib/mysql/'   --parent '2092' --binlog '/var/mysql/binlogs/mariadb-bin' '
      2016-09-06 15:51:15 139831721270016 [Note] WSREP: Prepared SST request: rsync|172.31.22.174:4444/rsync_sst
      2016-09-06 15:51:15 139831721270016 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
      2016-09-06 15:51:15 139831721270016 [Note] WSREP: REPL Protocols: 7 (3, 2)
      2016-09-06 15:51:15 139831499339520 [Note] WSREP: Service thread queue flushed.
      2016-09-06 15:51:15 139831721270016 [Note] WSREP: Assign initial position for certification: 0, protocol version: 3
      2016-09-06 15:51:15 139831499339520 [Note] WSREP: Service thread queue flushed.
      2016-09-06 15:51:15 139831721270016 [Warning] WSREP: Failed to prepare for incremental state transfer: Local state UUID (00000000-0000-0000-0000-000000000000) does not match group state UUID (bae8d70c-746a-11e6-869f-632ad20bfb5c): 1 (Operation not permitted)
               at galera/src/replicator_str.cpp:prepare_for_IST():482. IST will be unavailable.
      2016-09-06 15:51:15 139831443187456 [Note] WSREP: Member 0.0 () requested state transfer from '*any*'. Selected 1.0 (ip-172-31-19-192.us-west-2.compute.internal)(SYNCED) as donor.
      2016-09-06 15:51:15 139831443187456 [Note] WSREP: Shifting PRIMARY -> JOINER (TO: 0)
      2016-09-06 15:51:15 139831721270016 [Note] WSREP: Requesting state transfer: success, donor: 1
      2016-09-06 15:51:17 139831443187456 [Note] WSREP: 1.0 (ip-172-31-19-192.us-west-2.compute.internal): State transfer to 0.0 () complete.
      2016-09-06 15:51:17 139831443187456 [Note] WSREP: Member 1.0 (ip-172-31-19-192.us-west-2.compute.internal) synced with group.
      2016-09-06 15:51:17 139831451580160 [Note] WSREP: (44e8acf0, 'tcp://0.0.0.0:4567') turning message relay requesting off
      WSREP_SST: [INFO] Extracting binlog files: (20160906 15:51:18.363)
      mariadb-bin.000007
      WSREP_SST: [INFO] Joiner cleanup. rsync PID: 2132 (20160906 15:51:18.369)
      WSREP_SST: [INFO] Joiner cleanup done. (20160906 15:51:18.874)
      2016-09-06 15:51:18 139831721584768 [Note] WSREP: SST complete, seqno: 0
      2016-09-06 15:51:18 139831721584768 [Note] InnoDB: Using mutexes to ref count buffer pool pages
      2016-09-06 15:51:18 139831721584768 [Note] InnoDB: The InnoDB memory heap is disabled
      2016-09-06 15:51:18 139831721584768 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
      2016-09-06 15:51:18 139831721584768 [Note] InnoDB: Memory barrier is not used
      2016-09-06 15:51:18 139831721584768 [Note] InnoDB: Compressed tables use zlib 1.2.7
      2016-09-06 15:51:18 139831721584768 [Note] InnoDB: Using Linux native AIO
      2016-09-06 15:51:18 139831721584768 [Note] InnoDB: Using SSE crc32 instructions
      2016-09-06 15:51:18 139831721584768 [Note] InnoDB: Initializing buffer pool, size = 128.0M
      2016-09-06 15:51:19 139831721584768 [Note] InnoDB: Completed initialization of buffer pool
      2016-09-06 15:51:19 139831721584768 [Note] InnoDB: Restoring page 0 of tablespace 0
      2016-09-06 15:51:19 139831721584768 [Warning] InnoDB: Doublewrite does not have page_no=0 of space: 0
      2016-09-06 15:51:19 139831721584768 [ERROR] InnoDB: space header page consists of zero bytes in data file /innodbdata/ibdata1
      2016-09-06 15:51:19 139831721584768 [ERROR] InnoDB: Could not open or create the system tablespace. If you tried to add new data files to the system tablespace, and it failed here, you should now edit innodb_data_file_path in my.cnf back to what it was, and remove the new ibdata files InnoDB created in this failed attempt. InnoDB only wrote those files full of zeros, but did not yet use them in any way. But be careful: do not remove old data files which contain your precious data!
      2016-09-06 15:51:19 139831721584768 [ERROR] Plugin 'InnoDB' init function returned error.
      

      Attachments

        Issue Links

          Activity

            GeoffMontee Geoff Montee (Inactive) created issue -
            elenst Elena Stepanova made changes -
            Field Original Value New Value
            Assignee Nirbhay Choubey [ nirbhay_c ]
            serg Sergei Golubchik made changes -
            Assignee Nirbhay Choubey [ nirbhay_c ]
            jplindst Jan Lindström (Inactive) made changes -
            Assignee Sachin Setiya [ sachin.setiya.007 ]
            serg Sergei Golubchik made changes -
            Assignee Sachin Setiya [ sachin.setiya.007 ] Jan Lindström [ jplindst ]
            ratzpo Rasmus Johansson (Inactive) made changes -
            Assignee Jan Lindström [ jplindst ] Seppo Jaakola [ seppo ]
            jplindst Jan Lindström (Inactive) made changes -
            jplindst Jan Lindström (Inactive) made changes -
            Fix Version/s 10.0-galera [ 21901 ]
            jplindst Jan Lindström (Inactive) made changes -
            Assignee Seppo Jaakola [ seppo ] Julius Goryavsky [ sysprg ]
            sysprg Julius Goryavsky made changes -
            Status Open [ 1 ] In Progress [ 3 ]

            https://github.com/MariaDB/server/pull/824

            If innodb_data_home_dir path specified in the configuration file
            then rsync IST/SST will fail because the wsrep_sst_rsync.sh script
            does not read this parameter from the configuration file and then
            tries to find the data files in the default directory.

            To fix this error, we need to add reading of the innodb_data_home_dir
            configuration parameter to the rsync-related SST script.

            sysprg Julius Goryavsky added a comment - https://github.com/MariaDB/server/pull/824 If innodb_data_home_dir path specified in the configuration file then rsync IST/SST will fail because the wsrep_sst_rsync.sh script does not read this parameter from the configuration file and then tries to find the data files in the default directory. To fix this error, we need to add reading of the innodb_data_home_dir configuration parameter to the rsync-related SST script.
            sysprg Julius Goryavsky made changes -
            Status In Progress [ 3 ] In Review [ 10002 ]
            sysprg Julius Goryavsky made changes -
            Assignee Julius Goryavsky [ sysprg ] Jan Lindström [ jplindst ]
            jplindst Jan Lindström (Inactive) made changes -
            Assignee Jan Lindström [ jplindst ] Julius Goryavsky [ sysprg ]
            Status In Review [ 10002 ] Stalled [ 10000 ]
            sysprg Julius Goryavsky added a comment - Patch for 10.1 branch: https://github.com/MariaDB/server/pull/828
            jplindst Jan Lindström (Inactive) made changes -
            Comment [ Ok to push to 10.1 branch. ]

            Patch needs adjustment and please run regression testing! See current regressions on git.

            jplindst Jan Lindström (Inactive) added a comment - Patch needs adjustment and please run regression testing! See current regressions on git.
            sysprg Julius Goryavsky made changes -
            Status Stalled [ 10000 ] In Progress [ 3 ]

            Added new fixes after review and correction of the problems caused by the fact that during the SST innodb plugin is not yet initialized, as well as problems with running tests from the root user (not directly related to the MDEV-10754).

            sysprg Julius Goryavsky added a comment - Added new fixes after review and correction of the problems caused by the fact that during the SST innodb plugin is not yet initialized, as well as problems with running tests from the root user (not directly related to the MDEV-10754 ).
            sysprg Julius Goryavsky made changes -
            Assignee Julius Goryavsky [ sysprg ] Jan Lindström [ jplindst ]
            Status In Progress [ 3 ] In Review [ 10002 ]
            jplindst Jan Lindström (Inactive) made changes -
            Status In Review [ 10002 ] Stalled [ 10000 ]
            jplindst Jan Lindström (Inactive) made changes -
            Status Stalled [ 10000 ] In Progress [ 3 ]
            jplindst Jan Lindström (Inactive) made changes -
            issue.field.resolutiondate 2018-08-21 12:59:01.0 2018-08-21 12:59:01.043
            jplindst Jan Lindström (Inactive) made changes -
            Fix Version/s 10.1.36 [ 23117 ]
            Fix Version/s 10.2.18 [ 23112 ]
            Fix Version/s 10.3.11 [ 23141 ]
            Fix Version/s 10.1 [ 16100 ]
            Resolution Fixed [ 1 ]
            Status In Progress [ 3 ] Closed [ 6 ]
            ralf.gebhardt Ralf Gebhardt made changes -
            NRE Projects AC-2328 AC-3299
            GeoffMontee Geoff Montee (Inactive) made changes -
            GeoffMontee Geoff Montee (Inactive) made changes -
            GeoffMontee Geoff Montee (Inactive) made changes -
            serg Sergei Golubchik made changes -
            Workflow MariaDB v3 [ 76933 ] MariaDB v4 [ 150878 ]
            mariadb-jira-automation Jira Automation (IT) made changes -
            Zendesk Related Tickets 138807 143621

            People

              jplindst Jan Lindström (Inactive)
              GeoffMontee Geoff Montee (Inactive)
              Votes:
              1 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Git Integration

                  Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.