Uploaded image for project: 'MariaDB Server'
  1. MariaDB Server
  2. MDEV-27682

Bundled wsrep_notify.sh causes mariadbd to freeze during start

    XMLWordPrintable

Details

    Description

      starting with galera_new_cluster stuck at this point

      mysql      29610  0.6  2.5 1159880 99828 ?       Ssl  02:48   0:00 /usr/sbin/mariadbd --wsrep-new-cluster --wsrep_start_position=09e652bc-8169-11ec-848c-dfaaaea09e0a:999
      mysql      29638  0.0  0.0   2420   528 ?        S    02:48   0:00  \_ sh -c /usr/local/bin/wsrep_notify.sh --status initialized
      mysql      29639  0.0  0.0   2420   524 ?        S    02:48   0:00      \_ /bin/sh -eu /usr/local/bin/wsrep_notify.sh --status initialized
      mysql      29641  0.0  0.1  20104  7828 ?        S    02:48   0:00          \_ mysql -B
      

      root@frytka:~# strace -p 29610
      strace: Process 29610 attached
      wait4(29638,
      

      There are connection errors in error.log on every attempt to execute wsrep_notify.sh

      2022-01-30  2:48:48 1 [Note] WSREP: Server status change disconnected -> connected
      ERROR 2002 (HY000): Can't connect to local MySQL server through socket '/run/mysqld/mysqld.sock' (111)
      2022-01-30  2:48:48 1 [ERROR] WSREP: Process completed with error: /usr/local/bin/wsrep_notify.sh --status connected: 1 (Operation not permitted)
      2022-01-30  2:48:48 1 [ERROR] WSREP: Notification command failed: 1 (Operation not permitted): "/usr/local/bin/wsrep_notify.sh --status connected"
       
      2022-01-30  2:48:48 1 [Note] WSREP: Server status change connected -> joiner
      ERROR 2002 (HY000): Can't connect to local MySQL server through socket '/run/mysqld/mysqld.sock' (111)
      2022-01-30  2:48:48 1 [ERROR] WSREP: Process completed with error: /usr/local/bin/wsrep_notify.sh --status joiner: 1 (Operation not permitted)
      2022-01-30  2:48:48 1 [ERROR] WSREP: Notification command failed: 1 (Operation not permitted): "/usr/local/bin/wsrep_notify.sh --status joiner"
       
      2022-01-30  2:48:48 1 [Note] WSREP: Server status change joiner -> initializing
      ERROR 2002 (HY000): Can't connect to local MySQL server through socket '/run/mysqld/mysqld.sock' (111)
      2022-01-30  2:48:48 1 [ERROR] WSREP: Process completed with error: /usr/local/bin/wsrep_notify.sh --status initializing: 1 (Operation not permitted)
      2022-01-30  2:48:48 1 [ERROR] WSREP: Notification command failed: 1 (Operation not permitted): "/usr/local/bin/wsrep_notify.sh --status initializing"
      
      

      But:
      1: when it's in frozen state, I can't connect either
      2: after killing (SIGKILL) mariadbd and restarting without wsrep_notify_cmd everything is working normally
      3: I've modified the script to connect as service account (mysql) via socket, but had the same issue when trying to use TCP to localhost

      Config I'm running:

      [sst]
      encrypt=3
      sst-log-archive-dir=/var/log/mysql/
      sst-log-archive=0
      tkey = /etc/mysql/ssl/key.pem
      tcert = /etc/mysql/ssl/cert.pem
      tca = /etc/mysql/ssl/ca.pem
      ssl-mode=VERIFY_CA
       
      [galera]
      wsrep_on                 = ON
      wsrep_cluster_name       = "clustername"
      wsrep_provider           = /usr/lib/galera/libgalera_smm.so
      wsrep_cluster_address    = gcomm://host02.domain.com,host01.domain.com,host03.domain.com
      binlog_format            = row
      default_storage_engine   = InnoDB
      innodb_autoinc_lock_mode = 2
      innodb_doublewrite = 1
       
      wsrep_notify_cmd=/usr/local/bin/wsrep_notify.sh
       
      wsrep_provider_options="socket.ssl_cert=/etc/mysql/ssl/cert.pem;socket.ssl_key=/etc/mysql/ssl/key.pem;socket.ssl_ca=/etc/mysql/ssl/ca.pem"
      #wsrep_sst_method = rsync_wan
      wsrep_sst_method = mariabackup
      wsrep_sst_auth = mysql:
       
       
      wsrep_node_address = host01.domain.com
      
      

      Attachments

        Activity

          People

            sysprg Julius Goryavsky
            mkozlowski Michal Kozlowski
            Votes:
            1 Vote for this issue
            Watchers:
            9 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Git Integration

                Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.