Uploaded image for project: 'MariaDB Server'
  1. MariaDB Server
  2. MDEV-14063

mariadb102-server 10.2.9 cannot startup with galera

Details

    • Bug
    • Status: Closed (View Workflow)
    • Major
    • Resolution: Duplicate
    • 10.2.9
    • N/A
    • Galera
    • None
    • FreeBSD 11.1

    Description

      I am running galera multi-master. After upgrading one node from 10.2.7_1 to 10.2.9. I couldn't start it. When I run bash -x /usr/local/etc/rc.d/mysql-server start, I found that mysqld_safe is looking for "WSREP: Running position" in wsrep_recovery.XXXXXX. Not sure if this is related to MDEV-13950.

      I compared mysqld_safe with another machine which is running 10.2.7_1.

      --- /usr/local/bin/mysqld_safe	2017-09-12 23:14:57.000000000 +1000
      +++ mysqld_safe.10.2.9	2017-10-13 15:17:02.066860000 +1100
      @@ -245,7 +245,7 @@
         local euid=$(id -u)
         local ret=0
       
      -  local wr_logfile=$(mktemp $DATADIR/wsrep_recovery.XXXXXX)
      +  local wr_logfile=$(mktemp wsrep_recovery.XXXXXX)
       
         # safety checks
         if [ -z $wr_logfile ]; then
      @@ -263,11 +263,11 @@
       
         local wr_pidfile="$DATADIR/"`hostname`"-recover.pid"
       
      -  local wr_options="--log_error='$wr_logfile' --pid-file='$wr_pidfile'"
      +  local wr_options="--disable-log-error  --pid-file='$wr_pidfile'"
       
         log_notice "WSREP: Running position recovery with $wr_options"
       
      -  eval_log_error "$mysqld_cmd --wsrep_recover $wr_options"
      +  eval_log_error "$mysqld_cmd --wsrep_recover $wr_options 2> $wr_logfile"
       
         local rp="$(grep 'WSREP: Recovered position:' $wr_logfile)"
         if [ -z "$rp" ]; then
      

      So I replaced mysqld_safe with version 10.2.7_1 and now I can start it up.
      I don't see mysqld_safe from 10.2.7_1 has any problems. Why change it in 10.2.9?

      Attachments

        Issue Links

          Activity

            laocius TAO ZHOU added a comment -

            I can only start it with /usr/local/bin/mysqld_safe from command line. Still couldn't start with 'service mysql-server start'.

            laocius TAO ZHOU added a comment - I can only start it with /usr/local/bin/mysqld_safe from command line. Still couldn't start with 'service mysql-server start'.

            In addition to mentioned mysqld_safe patch, 10.2.9 may have sst scripts problems . Could you check in joiner node if sst was used and if it was rsync or some other ? In any case we need to see error log from both joiner and donor to assist

            anikitin Andrii Nikitin (Inactive) added a comment - In addition to mentioned mysqld_safe patch, 10.2.9 may have sst scripts problems . Could you check in joiner node if sst was used and if it was rsync or some other ? In any case we need to see error log from both joiner and donor to assist
            laocius TAO ZHOU added a comment -

            There are no logs on the donor side because the joiner exited before it started to join.
            On the joiner side:

            2017-10-13 12:52:34 34426937344 [Note] /usr/local/libexec/mysqld (mysqld 10.2.9-MariaDB-log) starting as process 7117 ...
            2017-10-13 12:52:34 34426937344 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
            2017-10-13 12:52:34 34426937344 [Note] InnoDB: Uses event mutexes
            2017-10-13 12:52:34 34426937344 [Note] InnoDB: Compressed tables use zlib 1.2.11
            2017-10-13 12:52:34 34426937344 [Note] InnoDB: Number of pools: 1
            2017-10-13 12:52:34 34426937344 [Note] InnoDB: Using SSE2 crc32 instructions
            2017-10-13 12:52:34 34426937344 [Note] InnoDB: Initializing buffer pool, total size = 12G, instances = 8, chunk size = 128M
            2017-10-13 12:52:35 34426937344 [Note] InnoDB: Completed initialization of buffer pool
            2017-10-13 12:52:35 34426937344 [Note] InnoDB: Highest supported file format is Barracuda.
            2017-10-13 12:52:35 34426937344 [Note] InnoDB: 128 out of 128 rollback segments are active.
            2017-10-13 12:52:35 34426937344 [Note] InnoDB: Creating shared tablespace for temporary tables
            2017-10-13 12:52:35 34426937344 [Note] InnoDB: Setting file '/var/mysql-data/ibtmp1' size to 12 MB. Physically writing the file full; Please wait ...
            2017-10-13 12:52:36 34426937344 [Note] InnoDB: File '/var/mysql-data/ibtmp1' size is now 12 MB.
            2017-10-13 12:52:36 34426937344 [Note] InnoDB: Waiting for purge to start
            2017-10-13 12:52:36 34426937344 [Note] InnoDB: 5.7.19 started; log sequence number 714948452513
            2017-10-13 12:52:36 34426937344 [Warning] InnoDB: Skipping buffer pool dump/restore during wsrep recovery.
            2017-10-13 12:52:36 34426937344 [Note] Plugin 'FEEDBACK' is disabled.
            2017-10-13 12:52:36 34426937344 [Note] Server socket created on IP: '192.168.11.34'.
            2017-10-13 12:52:36 34426937344 [Note] WSREP: Recovered position: 7a79d7e6-7ca9-11e7-b980-ef84f0f16abf:52913163
            171013 12:56:32 mysqld_safe Starting mysqld daemon with databases from /var/mysql-data
            171013 12:56:32 mysqld_safe WSREP: Running position recovery with --disable-log-error --pid-file='/var/mysql-data/db2.ish.com.au-recover.pid'

            I am using mariabackup for sst.

            laocius TAO ZHOU added a comment - There are no logs on the donor side because the joiner exited before it started to join. On the joiner side: 2017-10-13 12:52:34 34426937344 [Note] /usr/local/libexec/mysqld (mysqld 10.2.9-MariaDB-log) starting as process 7117 ... 2017-10-13 12:52:34 34426937344 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins 2017-10-13 12:52:34 34426937344 [Note] InnoDB: Uses event mutexes 2017-10-13 12:52:34 34426937344 [Note] InnoDB: Compressed tables use zlib 1.2.11 2017-10-13 12:52:34 34426937344 [Note] InnoDB: Number of pools: 1 2017-10-13 12:52:34 34426937344 [Note] InnoDB: Using SSE2 crc32 instructions 2017-10-13 12:52:34 34426937344 [Note] InnoDB: Initializing buffer pool, total size = 12G, instances = 8, chunk size = 128M 2017-10-13 12:52:35 34426937344 [Note] InnoDB: Completed initialization of buffer pool 2017-10-13 12:52:35 34426937344 [Note] InnoDB: Highest supported file format is Barracuda. 2017-10-13 12:52:35 34426937344 [Note] InnoDB: 128 out of 128 rollback segments are active. 2017-10-13 12:52:35 34426937344 [Note] InnoDB: Creating shared tablespace for temporary tables 2017-10-13 12:52:35 34426937344 [Note] InnoDB: Setting file '/var/mysql-data/ibtmp1' size to 12 MB. Physically writing the file full; Please wait ... 2017-10-13 12:52:36 34426937344 [Note] InnoDB: File '/var/mysql-data/ibtmp1' size is now 12 MB. 2017-10-13 12:52:36 34426937344 [Note] InnoDB: Waiting for purge to start 2017-10-13 12:52:36 34426937344 [Note] InnoDB: 5.7.19 started; log sequence number 714948452513 2017-10-13 12:52:36 34426937344 [Warning] InnoDB: Skipping buffer pool dump/restore during wsrep recovery. 2017-10-13 12:52:36 34426937344 [Note] Plugin 'FEEDBACK' is disabled. 2017-10-13 12:52:36 34426937344 [Note] Server socket created on IP: '192.168.11.34'. 2017-10-13 12:52:36 34426937344 [Note] WSREP: Recovered position: 7a79d7e6-7ca9-11e7-b980-ef84f0f16abf:52913163 171013 12:56:32 mysqld_safe Starting mysqld daemon with databases from /var/mysql-data 171013 12:56:32 mysqld_safe WSREP: Running position recovery with --disable-log-error --pid-file='/var/mysql-data/db2.ish.com.au-recover.pid' I am using mariabackup for sst.

            My understanding is that if you see message "WSREP: Recovered position" in MariaDB Server error log - then problem is related to MDEV-13950 and mysqld_safe wasn't patched properly.
            Thus this call will be closed with 'Duplicate' of MDEV-13950 resolution.

            anikitin Andrii Nikitin (Inactive) added a comment - My understanding is that if you see message "WSREP: Recovered position" in MariaDB Server error log - then problem is related to MDEV-13950 and mysqld_safe wasn't patched properly. Thus this call will be closed with 'Duplicate' of MDEV-13950 resolution.

            People

              anikitin Andrii Nikitin (Inactive)
              laocius TAO ZHOU
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Git Integration

                  Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.