Uploaded image for project: 'MariaDB ColumnStore'
  1. MariaDB ColumnStore
  2. MCOL-3831

um1 got "MAN_DISABLED" on every week automatically

    XMLWordPrintable

Details

    • Bug
    • Status: Closed (View Workflow)
    • Critical
    • Resolution: Fixed
    • 1.2.5
    • Icebox
    • DMLProc
    • None
    • MariaDB Column Store 1.2.5-1

    Description

      Customer reported their UM1 got MAN_DISABLED state every weekend. This is 2UM(replication enabled) + 3PM CS nodes.
      Chaning ModuleHeartbeatCount from 3 to 10 didn't help. They do not run any import job or any type of workload during the weekend,

      • Report log attached

      [ERROR]
      Feb 23 10:07:32 x01tepmsdb8a writeengine[6661]: 32.598163 |0|0|0| E 19 CAL0001: ReadThreadFactory::CreateReadThread: read() error InetStreamSocket::readToMagic: Remote is closed
      Feb 23 10:08:32 x01tepmsdb8a writeengine[6661]: 32.600318 |0|0|0| E 19 CAL0001: ReadThreadFactory::CreateReadThread: read() error InetStreamSocket::readToMagic: Remote is closed
      Feb 23 10:09:44 x01tepmsdb8a writeengine[6661]: 44.459956 |0|0|0| E 19 CAL0001: ReadThreadFactory::CreateReadThread: read() error InetStreamSocket::readToMagic(): I/O error1: rc-1; poll signal interrupt ( POLLHUP POLLERR )
      Feb 23 10:09:44 x01tepmsdb8a writeengine[6661]: 44.460285 |0|0|0| E 19 CAL0001: ReadThreadFactory::CreateReadThread: read() error InetStreamSocket::readToMagic: Remote is closed
      Feb 23 10:09:44 x01tepmsdb8a writeengine[6661]: 44.460455 |0|0|0| E 19 CAL0001: ReadThreadFactory::CreateReadThread: read() error InetStreamSocket::readToMagic: Remote is closed
      Feb 23 10:10:44 x01tepmsdb8a writeengine[6661]: 44.480446 |0|0|0| E 19 CAL0001: ReadThreadFactory::CreateReadThread: read() error InetStreamSocket::readToMagic: Remote is closed
      Feb 23 10:11:09 x01tepmsdb8a writeengine[6661]: 09.508097 |0|0|0| E 19 CAL0001: ReadThreadFactory::CreateReadThread: read() error InetStreamSocket::readToMagic: Remote is closed
      Feb 23 10:11:49 x01tepmsdb8a writeengine[6661]: 49.534477 |0|0|0| E 19 CAL0001: ReadThreadFactory::CreateReadThread: read() error InetStreamSocket::readToMagic: Remote is closed
      Feb 23 10:12:49 x01tepmsdb8a writeengine[6661]: 49.583981 |0|0|0| E 19 CAL0001: ReadThreadFactory::CreateReadThread: read() error InetStreamSocket::readToMagic: Remote is closed
      Feb 23 10:13:14 x01tepmsdb8a writeengine[6661]: 14.613427 |0|0|0| E 19 CAL0001: ReadThreadFactory::CreateReadThread: read() error InetStreamSocket::readToMagic: Remote is closed
      Feb 23 10:13:24 x01tepmsdb8a writeengine[6661]: 24.619921 |0|0|0| E 19 CAL0001: ReadThreadFactory::CreateReadThread: read() error InetStreamSocket::readToMagic: Remote is closed
      Feb 23 10:13:34 x01tepmsdb8a writeengine[6661]: 34.632247 |0|0|0| E 19 CAL0001: ReadThreadFactory::CreateReadThread: read() error InetStreamSocket::readToMagic: Remote is closed
      Feb 23 10:13:34 x01tepmsdb8a writeengine[6661]: 34.632623 |0|0|0| E 19 CAL0001: ReadThreadFactory::CreateReadThread: read() error InetStreamSocket::readToMagic: Remote is closed
      Feb 23 10:13:44 x01tepmsdb8a writeengine[6661]: 44.644498 |0|0|0| E 19 CAL0001: ReadThreadFactory::CreateReadThread: read() error InetStreamSocket::readToMagic: Remote is closed
      Feb 23 10:13:54 x01tepmsdb8a writeengine[6661]: 54.659748 |0|0|0| E 19 CAL0001: ReadThreadFactory::CreateReadThread: read() error InetStreamSocket::readToMagic: Remote is closed
      Feb 23 10:14:32 x01tepmsdb8a writeengine[6661]: 32.599303 |0|0|0| E 19 CAL0001: ReadThreadFactory::CreateReadThread: read() error InetStreamSocket::readToMagic: Remote is closed
      Feb 23 10:15:32 x01tepmsdb8a writeengine[6661]: 32.605321 |0|0|0| E 19 CAL0001: ReadThreadFactory::CreateReadThread: read() error InetStreamSocket::readToMagic: Remote is closed
      Feb 23 10:16:32 x01tepmsdb8a writeengine[6661]: 32.606814 |0|0|0| E 19 CAL0001: ReadThreadFactory::CreateReadThread: read() error InetStreamSocket::readToMagic: Remote is closed
      Feb 23 10:17:32 x01tepmsdb8a writeengine[6661]: 32.600418 |0|0|0| E 19 CAL0001: ReadThreadFactory::CreateReadThread: read() error InetStreamSocket::readToMagic: Remote is closed
      Feb 23 10:18:32 x01tepmsdb8a writeengine[6661]: 32.599711 |0|0|0| E 19 CAL0001: ReadThreadFactory::CreateReadThread: read() error InetStreamSocket::readToMagic: Remote is closed
      Feb 23 10:19:32 x01tepmsdb8a writeengine[6661]: 32.599982 |0|0|0| E 19 CAL0001: ReadThreadFactory::CreateReadThread: read() error InetStreamSocket::readToMagic: Remote is closed
      Feb 23 10:20:32 x01tepmsdb8a writeengine[6661]: 32.606291 |0|0|0| E 19 CAL0001: ReadThreadFactory::CreateReadThread: read() error InetStreamSocket::readToMagic: Remote is closed
      Feb 23 10:21:32 x01tepmsdb8a writeengine[6661]: 32.606890 |0|0|0| E 19 CAL0001: ReadThreadFactory::CreateReadThread: read() error InetStreamSocket::readToMagic: Remote is closed
      Feb 23 10:22:32 x01tepmsdb8a writeengine[6661]: 32.603574 |0|0|0| E 19 CAL0001: ReadThreadFactory::CreateReadThread: read() error InetStreamSocket::readToMagic: Remote is closed
      Feb 23 13:24:37 x01tepmsdb8a workernode[6428]: 37.761151 |0|0|0| C 30 CAL0000: An error occurred: CompressedInetStreamSocket::accept sync: Connection reset by peer
      Feb 23 13:24:48 x01tepmsdb8a controllernode[5023]: 48.669462 |0|0|0| E 29 CAL0000: DBRM: error: SessionManager::getSystemState() failed (network)
      Feb 23 13:25:02 x01tepmsdb8a controllernode[5023]: 02.953492 |0|0|0| E 29 CAL0000: DBRM: error: SessionManager::clearSystemState() failed (network)
      Feb 23 13:25:02 x01tepmsdb8a controllernode[5023]: 02.953923 |0|0|0| E 29 CAL0000: DBRM: error: SessionManager::clearSystemState() failed (network)
      Feb 23 13:25:02 x01tepmsdb8a controllernode[5023]: 02.962947 |0|0|0| E 29 CAL0000: DBRM: error: SessionManager::clearSystemState() failed (network)
      Feb 23 13:25:02 x01tepmsdb8a controllernode[5023]: 02.964940 |0|0|0| E 29 CAL0000: DBRM: error: SessionManager::clearSystemState() failed (network)
      Feb 23 13:25:03 x01tepmsdb8a controllernode[5023]: 03.824580 |0|0|0| E 29 CAL0000: DBRM: error: SessionManager::clearSystemState() failed (network)
      Feb 23 13:25:03 x01tepmsdb8a controllernode[5023]: 03.825384 |0|0|0| E 29 CAL0000: DBRM: error: SessionManager::clearSystemState() failed (network)
      Feb 23 13:25:03 x01tepmsdb8a controllernode[5023]: 03.825586 |0|0|0| E 29 CAL0000: DBRM: error: SessionManager::clearSystemState() failed (network)
      Feb 23 13:25:03 x01tepmsdb8a controllernode[5023]: 03.825837 |0|0|0| E 29 CAL0000: DBRM: error: SessionManager::clearSystemState() failed (network)
      Feb 23 13:25:07 x01tepmsdb8a controllernode[5023]: 07.911440 |0|0|0| E 29 CAL0000: DBRM: error: SessionManager::setSystemState() failed (network)
      Feb 23 13:25:07 x01tepmsdb8a controllernode[5023]: 07.912174 |0|0|0| E 29 CAL0000: DBRM: error: SessionManager::setSystemState() failed (network)
      Feb 23 13:25:07 x01tepmsdb8a controllernode[5023]: 07.917168 |0|0|0| E 29 CAL0000: DBRM: error: SessionManager::setSystemState() failed (network)
      Feb 23 13:25:07 x01tepmsdb8a controllernode[5023]: 07.918586 |0|0|0| E 29 CAL0000: DBRM: error: SessionManager::setSystemState() failed (network)
      Feb 23 13:26:29 x01tepmsdb8a controllernode[5023]: 29.528970 |0|0|0| E 29 CAL0000: DBRM: error: SessionManager::getSystemState() failed (network)

      [UM1 log]
      2020-02-23 13:24:14 0 [Note] /home/mcsadm/mariadb/columnstore/mysql//bin/mysqld (initiated by: unknown): Normal shutdown
      2020-02-23 13:24:14 0 [Note] Event Scheduler: Purging the queue. 0 events
      2020-02-23 13:24:14 0 [Note] InnoDB: FTS optimize thread exiting.
      2020-02-23 13:24:14 0 [Note] InnoDB: Starting shutdown...
      2020-02-23 13:24:14 0 [Note] InnoDB: Dumping buffer pool(s) to /home/mcsadm/mariadb/columnstore/mysql/db/ib_buffer_pool
      2020-02-23 13:24:14 0 [Note] InnoDB: Buffer pool(s) dump completed at 200223 13:24:14
      2020-02-23 13:24:15 0 [Note] InnoDB: Shutdown completed; log sequence number 2533040783; transaction id 3785986
      2020-02-23 13:24:15 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1"
      2020-02-23 13:24:15 0 [Note] /home/mcsadm/mariadb/columnstore/mysql//bin/mysqld: Shutdown complete

      [Critical Log]
      Feb 23 13:24:37 x01tepmsdb8a workernode[6428]: 37.761151 |0|0|0| C 30 CAL0000: An error occurred: CompressedInetStreamSocket::accept sync: Connection reset by peer

      Attachments

        Activity

          People

            Unassigned Unassigned
            allen.lee@mariadb.com Allen Lee (Inactive)
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Git Integration

                Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.