Uploaded image for project: 'MariaDB Server'
  1. MariaDB Server
  2. MDEV-15209

mdb cluster node fails to start with wsrep_sst_mariabackup due to mariabackup not in path

    XMLWordPrintable

Details

    • Bug
    • Status: Closed (View Workflow)
    • Minor
    • Resolution: Fixed
    • 10.2
    • 10.2.42
    • Galera SST
    • None

    Description

      mdb cluster node fails to start with wsrep_sst_mariabackup due to mariabackup not in path

      after installation of MariaDB-server ,setting up wsrep_sst_method=mariabackup
      and enabling wsrep Node fails to start due to dependancy from MariaDB-backup package,

      it would be expected that MariaDB-server is including mariabackup while the mariabackup
      is a part of the supported wsrep_sst methods

      2018-02-05 14:17:15 140016945899264 [Warning] WSREP: Gap in state sequence. Need state transfer.
      2018-02-05 14:17:15 140016688793344 [Note] WSREP: Running: 'wsrep_sst_mariabackup --role 'joiner' --address '192.168.104.193' --datadir '/var/lib/mysql/'   --parent '27597'  '' '
      which: no mariabackup in (/usr/sbin:/sbin:/usr//bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin)
      

      [root@t4w6 ~]# yum install MariaDB-server MariaDB-client MariaDB-common
      Loaded plugins: fastestmirror, langpacks
      Loading mirror speeds from cached hostfile
       * base: mirrors.netix.net
       * epel: mirrors.netix.net
       * extras: mirrors.netix.net
       * updates: mirrors.netix.net
      Resolving Dependencies
      --> Running transaction check
      ---> Package MariaDB-client.x86_64 0:10.2.12-1.el7.centos will be installed
      ---> Package MariaDB-common.x86_64 0:10.2.12-1.el7.centos will be installed
      --> Processing Dependency: MariaDB-compat for package: MariaDB-common-10.2.12-1.el7.centos.x86_64
      ---> Package MariaDB-server.x86_64 0:10.2.12-1.el7.centos will be installed
      --> Running transaction check
      ---> Package MariaDB-compat.x86_64 0:10.2.12-1.el7.centos will be installed
      --> Finished Dependency Resolution
       
      Dependencies Resolved
       
      =============================================================================================================================================================
       Package                                Arch                           Version                                         Repository                       Size
      =============================================================================================================================================================
      Installing:
       MariaDB-client                         x86_64                         10.2.12-1.el7.centos                            mariadb                          48 M
       MariaDB-common                         x86_64                         10.2.12-1.el7.centos                            mariadb                         154 k
       MariaDB-server                         x86_64                         10.2.12-1.el7.centos                            mariadb                         110 M
      Installing for dependencies:
       MariaDB-compat                         x86_64                         10.2.12-1.el7.centos                            mariadb                         2.8 M
       
      Transaction Summary
      =============================================================================================================================================================
      Install  3 Packages (+1 Dependent package)
       
      Total download size: 160 M
      Installed size: 683 M
      Is this ok [y/d/N]: y
      Downloading packages:
      (1/4): MariaDB-10.2.12-centos7-x86_64-common.rpm                                                                                      | 154 kB  00:00:00
      (2/4): MariaDB-10.2.12-centos7-x86_64-compat.rpm                                                                                      | 2.8 MB  00:00:01
      (3/4): MariaDB-10.2.12-centos7-x86_64-client.rpm                                                                                      |  48 MB  00:00:05
      (4/4): MariaDB-10.2.12-centos7-x86_64-server.rpm                                                                                      | 110 MB  00:00:07
      -------------------------------------------------------------------------------------------------------------------------------------------------------------
      Total                                                                                                                         16 MB/s | 160 MB  00:00:09
      Running transaction check
      Running transaction test
      Transaction test succeeded
      Running transaction
        Installing : MariaDB-common-10.2.12-1.el7.centos.x86_64                                                                                                1/4
        Installing : MariaDB-compat-10.2.12-1.el7.centos.x86_64                                                                                                2/4
        Installing : MariaDB-client-10.2.12-1.el7.centos.x86_64                                                                                                3/4
        Installing : MariaDB-server-10.2.12-1.el7.centos.x86_64                                                                                                4/4
      chown: cannot access ‘/var/lib/mysql’: No such file or directory
       
      PLEASE REMEMBER TO SET A PASSWORD FOR THE MariaDB root USER !
      To do so, start the server, then issue the following commands:
       
      '/usr/bin/mysqladmin' -u root password 'new-password'
      '/usr/bin/mysqladmin' -u root -h t4w6.xentio.lan password 'new-password'
       
      Alternatively you can run:
      '/usr/bin/mysql_secure_installation'
       
      which will also give you the option of removing the test
      databases and anonymous user created by default.  This is
      strongly recommended for production servers.
       
      See the MariaDB Knowledgebase at http://mariadb.com/kb or the
      MySQL manual for more instructions.
       
      Please report any problems at http://mariadb.org/jira
       
      The latest information about MariaDB is available at http://mariadb.org/.
      You can find additional information about the MySQL part at:
      http://dev.mysql.com
      Consider joining MariaDB's strong and vibrant community:
      https://mariadb.org/get-involved/
       
        Verifying  : MariaDB-client-10.2.12-1.el7.centos.x86_64                                                                                                1/4
        Verifying  : MariaDB-compat-10.2.12-1.el7.centos.x86_64                                                                                                2/4
        Verifying  : MariaDB-common-10.2.12-1.el7.centos.x86_64                                                                                                3/4
        Verifying  : MariaDB-server-10.2.12-1.el7.centos.x86_64                                                                                                4/4
       
      Installed:
        MariaDB-client.x86_64 0:10.2.12-1.el7.centos        MariaDB-common.x86_64 0:10.2.12-1.el7.centos        MariaDB-server.x86_64 0:10.2.12-1.el7.centos
       
      Dependency Installed:
        MariaDB-compat.x86_64 0:10.2.12-1.el7.centos
       
      Complete!
      [root@t4w6 ~]#
      
      

      [root@t4w3 log]# service mysql start --wsrep_cluster_address=gcomm://192.168.104.196
      Starting mysql (via systemctl): Job for mariadb.service failed because the control process exited with error code. See "systemctl status mariadb.service" and "journalctl -xe" for details.
      [FAILED]

      2018-02-05 14:17:14 140017037990016 [Note] WSREP: Read nil XID from storage engines, skipping position init
      2018-02-05 14:17:14 140017037990016 [Note] WSREP: wsrep_load(): loading provider library '/usr/lib64/galera/libgalera_smm.so'
      2018-02-05 14:17:14 140017037990016 [Note] WSREP: wsrep_load(): Galera 25.3.22(r3764) by Codership Oy <info@codership.com> loaded successfully.
      2018-02-05 14:17:14 140017037990016 [Note] WSREP: CRC-32C: using "slicing-by-8" algorithm.
      2018-02-05 14:17:14 140017037990016 [Note] WSREP: Found saved state: 00000000-0000-0000-0000-000000000000:-1, safe_to_bootstrap: 1
      2018-02-05 14:17:14 140017037990016 [Note] WSREP: Passing config to GCS: base_dir = /var/lib/mysql/; base_host = 192.168.104.193; base_port = 4567; cert.log_conflicts = no; debug = no; evs.auto_evict = 0; evs.delay_margin = PT1S; evs.delayed_keep_period = PT30S; evs.inactive_check_period = PT0.5S; evs.inactive_timeout = PT15S; evs.join_retrans_period = PT1S; evs.max_install_timeouts = 3; evs.send_window = 4; evs.stats_report_period = PT1M; evs.suspect_timeout = PT5S; evs.user_send_window = 2; evs.view_forget_timeout = PT24H; gcache.dir = /var/lib/mysql/; gcache.keep_pages_size = 0; gcache.mem_size = 0; gcache.name = /var/lib/mysql//galera.cache; gcache.page_size = 128M; gcache.recover = no; gcache.size = 128M; gcomm.thread_prio = ; gcs.fc_debug = 0; gcs.fc_factor = 1.0; gcs.fc_limit = 16; gcs.fc_master_slave = no; gcs.max_packet_size = 64500; gcs.max_throttle = 0.25; gcs.recv_q_hard_limit = 9223372036854775807; gcs.recv_q_soft_limit = 0.25; gcs.sync_donor = no; gmcast.segment = 0; gmcast.version = 0; pc.announce_timeout = PT3S; pc.checksum = false; p
      2018-02-05 14:17:14 140017037990016 [Note] WSREP: GCache history reset: 00000000-0000-0000-0000-000000000000:0 -> 00000000-0000-0000-0000-000000000000:-1
      2018-02-05 14:17:14 140017037990016 [Note] WSREP: Assign initial position for certification: -1, protocol version: -1
      2018-02-05 14:17:14 140017037990016 [Note] WSREP: wsrep_sst_grab()
      2018-02-05 14:17:14 140017037990016 [Note] WSREP: Start replication
      2018-02-05 14:17:14 140017037990016 [Note] WSREP: Setting initial position to 00000000-0000-0000-0000-000000000000:-1
      2018-02-05 14:17:14 140017037990016 [Note] WSREP: protonet asio version 0
      2018-02-05 14:17:14 140017037990016 [Note] WSREP: Using CRC-32C for message checksums.
      2018-02-05 14:17:14 140017037990016 [Note] WSREP: backend: asio
      2018-02-05 14:17:14 140017037990016 [Note] WSREP: gcomm thread scheduling priority set to other:0
      2018-02-05 14:17:14 140017037990016 [Warning] WSREP: access file(/var/lib/mysql//gvwstate.dat) failed(No such file or directory)
      2018-02-05 14:17:14 140017037990016 [Note] WSREP: restore pc from disk failed
      2018-02-05 14:17:14 140017037990016 [Note] WSREP: GMCast version 0
      2018-02-05 14:17:14 140017037990016 [Note] WSREP: (80294d65, 'tcp://0.0.0.0:4567') listening at tcp://0.0.0.0:4567
      2018-02-05 14:17:14 140017037990016 [Note] WSREP: (80294d65, 'tcp://0.0.0.0:4567') multicast: , ttl: 1
      2018-02-05 14:17:14 140017037990016 [Note] WSREP: EVS version 0
      2018-02-05 14:17:14 140017037990016 [Note] WSREP: gcomm: connecting to group 'cluster1', peer '192.168.104.193:,192.168.104.196:'
      2018-02-05 14:17:14 140017037990016 [Note] WSREP: (80294d65, 'tcp://0.0.0.0:4567') connection established to 80294d65 tcp://192.168.104.193:4567
      2018-02-05 14:17:14 140017037990016 [Warning] WSREP: (80294d65, 'tcp://0.0.0.0:4567') address 'tcp://192.168.104.193:4567' points to own listening address, blacklisting
      2018-02-05 14:17:14 140017037990016 [Note] WSREP: (80294d65, 'tcp://0.0.0.0:4567') connection established to 76c5a0e7 tcp://192.168.104.196:4567
      2018-02-05 14:17:14 140017037990016 [Note] WSREP: (80294d65, 'tcp://0.0.0.0:4567') turning message relay requesting on, nonlive peers:
      2018-02-05 14:17:15 140017037990016 [Note] WSREP: declaring 76c5a0e7 at tcp://192.168.104.196:4567 stable
      2018-02-05 14:17:15 140017037990016 [Note] WSREP: Node 76c5a0e7 state prim
      2018-02-05 14:17:15 140017037990016 [Note] WSREP: view(view_id(PRIM,76c5a0e7,2) memb {
              76c5a0e7,0
              80294d65,0
      } joined {
      } left {
      } partitioned {
      })
      2018-02-05 14:17:15 140017037990016 [Note] WSREP: save pc into disk
      2018-02-05 14:17:15 140017037990016 [Note] WSREP: gcomm: connected
      2018-02-05 14:17:15 140017037990016 [Note] WSREP: Changing maximum packet size to 64500, resulting msg size: 32636
      2018-02-05 14:17:15 140017037990016 [Note] WSREP: Shifting CLOSED -> OPEN (TO: 0)
      2018-02-05 14:17:15 140017037990016 [Note] WSREP: Opened channel 'cluster1'
      2018-02-05 14:17:15 140016697186048 [Note] WSREP: New COMPONENT: primary = yes, bootstrap = no, my_idx = 1, memb_num = 2
      2018-02-05 14:17:15 140016697186048 [Note] WSREP: STATE EXCHANGE: Waiting for state UUID.
      2018-02-05 14:17:15 140017037990016 [Note] WSREP: Waiting for SST to complete.
      2018-02-05 14:17:15 140016697186048 [Note] WSREP: STATE EXCHANGE: sent state msg: 88badf22-0a6e-11e8-91b4-bad85022d9a0
      2018-02-05 14:17:15 140016697186048 [Note] WSREP: STATE EXCHANGE: got state msg: 88badf22-0a6e-11e8-91b4-bad85022d9a0 from 0 (t4w6)
      2018-02-05 14:17:15 140016697186048 [Note] WSREP: STATE EXCHANGE: got state msg: 88badf22-0a6e-11e8-91b4-bad85022d9a0 from 1 (t4w3)
      2018-02-05 14:17:15 140016697186048 [Note] WSREP: Quorum results:
              version    = 4,
              component  = PRIMARY,
              conf_id    = 1,
              members    = 1/2 (joined/total),
              act_id     = 0,
              last_appl. = -1,
              protocols  = 0/7/3 (gcs/repl/appl),
              group UUID = 376a9b49-0a68-11e8-abee-56e9cbc26fb1
      2018-02-05 14:17:15 140016697186048 [Note] WSREP: Flow-control interval: [23, 23]
      2018-02-05 14:17:15 140016697186048 [Note] WSREP: Trying to continue unpaused monitor
      2018-02-05 14:17:15 140016697186048 [Note] WSREP: Shifting OPEN -> PRIMARY (TO: 0)
      2018-02-05 14:17:15 140016945899264 [Note] WSREP: State transfer required:
              Group state: 376a9b49-0a68-11e8-abee-56e9cbc26fb1:0
              Local state: 00000000-0000-0000-0000-000000000000:-1
      2018-02-05 14:17:15 140016945899264 [Note] WSREP: New cluster view: global state: 376a9b49-0a68-11e8-abee-56e9cbc26fb1:0, view# 2: Primary, number of nodes: 2, my index: 1, protocol version 3
      2018-02-05 14:17:15 140016945899264 [Warning] WSREP: Gap in state sequence. Need state transfer.
      2018-02-05 14:17:15 140016688793344 [Note] WSREP: Running: 'wsrep_sst_mariabackup --role 'joiner' --address '192.168.104.193' --datadir '/var/lib/mysql/'   --parent '27597'  '' '
      which: no mariabackup in (/usr/sbin:/sbin:/usr//bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin)
      WSREP_SST: [ERROR] mariabackup not in path: /usr/sbin:/sbin:/usr//bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin (20180205 14:17:15.807)
      2018-02-05 14:17:15 140016688793344 [ERROR] WSREP: Failed to read 'ready <addr>' from: wsrep_sst_mariabackup --role 'joiner' --address '192.168.104.193' --datadir '/var/lib/mysql/'   --parent '27597'  ''
              Read: '(null)'
      2018-02-05 14:17:15 140016688793344 [ERROR] WSREP: Process completed with error: wsrep_sst_mariabackup --role 'joiner' --address '192.168.104.193' --datadir '/var/lib/mysql/'   --parent '27597'  '' : 2 (No such file or directory)
      2018-02-05 14:17:15 140016945899264 [ERROR] WSREP: Failed to prepare for 'mariabackup' SST. Unrecoverable.
      2018-02-05 14:17:15 140016945899264 [ERROR] Aborting
       
      Error in my_thread_global_end(): 1 threads didn't exit
       
      
      

      Attachments

        Activity

          People

            jplindst Jan Lindström (Inactive)
            winstone Zdravelina Sokolovska (Inactive)
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Git Integration

                Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.