Uploaded image for project: 'MariaDB Server'
  1. MariaDB Server
  2. MDEV-20821

mariadb parallel slave server shutdown hang - v10.4.8

Details

    Description

      During on-site engagement, I installed master-slave configuration with MaxScale v2.3 on top of master node.

      OS: CentOS 7.4.1708
      MariaDB: v10.4.8
      MaxScale: v2.3.11

      Customer is using GTID to utilize auto failover of MaxScale and HA test has been done successfully.

      But when I tried to shutdown mariadb server it failed with timeout.
      I tried
      $ systemctl stop mysql
      and
      $ mysqladmin shutdown and the result is the same.

      The last error log is follows..
      ...
      [Note] Slave I/O thread: connected to master 'xxx@xxxx:3306', replication starts at GTID position '0-1000-382'
      ----------------------------------------------

      So If I stop the slave with "mysql> stop slave;" and then tried to stop the mariadb server, it stops immediately.
      It looks like shutdown never stops "Slave I/O thread"... and got timeout.

      Is it a known issue of v10.4.8 or any OS setting dependancies ?

      FYI. innodb-thread-concurrency is set to 0.

      Thanks

      Attachments

        Activity

          kb Kwangbock Lee added a comment -

          FYI. Customer is utilizing their infrastructures with OpenStack on top of their on-premises.

          kb Kwangbock Lee added a comment - FYI. Customer is utilizing their infrastructures with OpenStack on top of their on-premises.
          Elkin Andrei Elkin added a comment - - edited

          kb, thank you for the report! I am sure there were more data in the error log around and after the attempted shutdown. It would be very helpful to share with us. Could you please?

          Secondly, if the issue is reproducible, could you try out SHOW PROCESSLIST when the server is hanging.

          And finally, we also need to know SHOW GLOBAL for both VARIABLES and STATUS of your slave server.

          Elkin Andrei Elkin added a comment - - edited kb , thank you for the report! I am sure there were more data in the error log around and after the attempted shutdown. It would be very helpful to share with us. Could you please? Secondly, if the issue is reproducible, could you try out SHOW PROCESSLIST when the server is hanging. And finally, we also need to know SHOW GLOBAL for both VARIABLES and STATUS of your slave server.
          Elkin Andrei Elkin added a comment -

          alice: could you please verify the case? Support is also interested though there's not ticket associated. The report may related to MDEV-18450.

          Elkin Andrei Elkin added a comment - alice : could you please verify the case? Support is also interested though there's not ticket associated. The report may related to MDEV-18450 .
          vklasen Volker Klasen added a comment -

          Hi,

          we've got the same issue as the original reporter. I attached the output of SHOW GLOBAL VARIABLES and STATUS. However, I can't issue SHOW PROCESSLIST as my connection gets killed when issuing the shutdown command.

          Here is the tail of mysql-error.log:

          2020-01-02 10:53:45 0 [Note] /opt/mysql/bin/mysqld (initiated by: root[root] @ localhost []): Normal shutdown
          2020-01-02 10:53:45 0 [Note] Event Scheduler: Purging the queue. 0 events
          2020-01-02 10:53:45 8 [Note] Error reading relay log event: slave SQL thread was killed
          2020-01-02 10:53:45 8 [Note] Slave SQL thread exiting, replication stopped in log 'mysql-bin.000045' at position 3382; GTID position '0-2-174394'
          2020-01-02 10:53:45 0 [Note] InnoDB: FTS optimize thread exiting.
          2020-01-02 10:53:45 7 [Note] Slave I/O thread exiting, read up to log 'mysql-bin.000045', position 3382; GTID position 0-2-174394
          

          Cheers,
          Volker

          P.S. Setting innodb-thread-concurrency to 0 makes no difference (cf. MDEV-20288)

          vklasen Volker Klasen added a comment - Hi, we've got the same issue as the original reporter. I attached the output of SHOW GLOBAL VARIABLES and STATUS. However, I can't issue SHOW PROCESSLIST as my connection gets killed when issuing the shutdown command. Here is the tail of mysql-error.log: 2020-01-02 10:53:45 0 [Note] /opt/mysql/bin/mysqld (initiated by: root[root] @ localhost []): Normal shutdown 2020-01-02 10:53:45 0 [Note] Event Scheduler: Purging the queue. 0 events 2020-01-02 10:53:45 8 [Note] Error reading relay log event: slave SQL thread was killed 2020-01-02 10:53:45 8 [Note] Slave SQL thread exiting, replication stopped in log 'mysql-bin.000045' at position 3382; GTID position '0-2-174394' 2020-01-02 10:53:45 0 [Note] InnoDB: FTS optimize thread exiting. 2020-01-02 10:53:45 7 [Note] Slave I/O thread exiting, read up to log 'mysql-bin.000045', position 3382; GTID position 0-2-174394 Cheers, Volker P.S. Setting innodb-thread-concurrency to 0 makes no difference (cf. MDEV-20288 )
          devhen Devin Henderson added a comment - - edited

          Happening to me on a fresh MariaDB 10.4.11 install from the mariadb.org CentOS 8 repo on CentOS 8.1.1911 (fully updated). `innodb-thread-concurrency=0` makes no difference for me either. I'm using `slave_parallel_threads=2` but commenting that out doesn't help. I'm also using `thread_handling=pool-of-threads` but I can't recall if I tried commenting that out, I will report back on that.

          What I have noticed is that a maraidb.service stop or restart does work if I first do `STOP SLAVE` otherwise it hangs. See attached my journalctl for mariadb.service as well as GLOBAL VARIABLES and STATUS.

          mariadb.service.log
          mariadb.VARIABLES.log
          mariadb.STATUS.log

          devhen Devin Henderson added a comment - - edited Happening to me on a fresh MariaDB 10.4.11 install from the mariadb.org CentOS 8 repo on CentOS 8.1.1911 (fully updated). `innodb-thread-concurrency=0` makes no difference for me either. I'm using `slave_parallel_threads=2` but commenting that out doesn't help. I'm also using `thread_handling=pool-of-threads` but I can't recall if I tried commenting that out, I will report back on that. What I have noticed is that a maraidb.service stop or restart does work if I first do `STOP SLAVE` otherwise it hangs. See attached my journalctl for mariadb.service as well as GLOBAL VARIABLES and STATUS. mariadb.service.log mariadb.VARIABLES.log mariadb.STATUS.log

          Can verify, this is no longer happening to me on 10.4.12. Thanks!!!

          devhen Devin Henderson added a comment - Can verify, this is no longer happening to me on 10.4.12. Thanks!!!

          People

            Elkin Andrei Elkin
            kb Kwangbock Lee
            Votes:
            2 Vote for this issue
            Watchers:
            8 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Git Integration

                Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.