Details
-
Bug
-
Status: Closed (View Workflow)
-
Critical
-
Resolution: Fixed
-
1.2.5
-
None
-
MariaDB Column Store 1.2.5-1
Description
Customer reported their UM1 got MAN_DISABLED state every weekend. This is 2UM(replication enabled) + 3PM CS nodes.
Chaning ModuleHeartbeatCount from 3 to 10 didn't help. They do not run any import job or any type of workload during the weekend,
- Report log attached
[ERROR]
Feb 23 10:07:32 x01tepmsdb8a writeengine[6661]: 32.598163 |0|0|0| E 19 CAL0001: ReadThreadFactory::CreateReadThread: read() error InetStreamSocket::readToMagic: Remote is closed
Feb 23 10:08:32 x01tepmsdb8a writeengine[6661]: 32.600318 |0|0|0| E 19 CAL0001: ReadThreadFactory::CreateReadThread: read() error InetStreamSocket::readToMagic: Remote is closed
Feb 23 10:09:44 x01tepmsdb8a writeengine[6661]: 44.459956 |0|0|0| E 19 CAL0001: ReadThreadFactory::CreateReadThread: read() error InetStreamSocket::readToMagic(): I/O error1: rc-1; poll signal interrupt ( POLLHUP POLLERR )
Feb 23 10:09:44 x01tepmsdb8a writeengine[6661]: 44.460285 |0|0|0| E 19 CAL0001: ReadThreadFactory::CreateReadThread: read() error InetStreamSocket::readToMagic: Remote is closed
Feb 23 10:09:44 x01tepmsdb8a writeengine[6661]: 44.460455 |0|0|0| E 19 CAL0001: ReadThreadFactory::CreateReadThread: read() error InetStreamSocket::readToMagic: Remote is closed
Feb 23 10:10:44 x01tepmsdb8a writeengine[6661]: 44.480446 |0|0|0| E 19 CAL0001: ReadThreadFactory::CreateReadThread: read() error InetStreamSocket::readToMagic: Remote is closed
Feb 23 10:11:09 x01tepmsdb8a writeengine[6661]: 09.508097 |0|0|0| E 19 CAL0001: ReadThreadFactory::CreateReadThread: read() error InetStreamSocket::readToMagic: Remote is closed
Feb 23 10:11:49 x01tepmsdb8a writeengine[6661]: 49.534477 |0|0|0| E 19 CAL0001: ReadThreadFactory::CreateReadThread: read() error InetStreamSocket::readToMagic: Remote is closed
Feb 23 10:12:49 x01tepmsdb8a writeengine[6661]: 49.583981 |0|0|0| E 19 CAL0001: ReadThreadFactory::CreateReadThread: read() error InetStreamSocket::readToMagic: Remote is closed
Feb 23 10:13:14 x01tepmsdb8a writeengine[6661]: 14.613427 |0|0|0| E 19 CAL0001: ReadThreadFactory::CreateReadThread: read() error InetStreamSocket::readToMagic: Remote is closed
Feb 23 10:13:24 x01tepmsdb8a writeengine[6661]: 24.619921 |0|0|0| E 19 CAL0001: ReadThreadFactory::CreateReadThread: read() error InetStreamSocket::readToMagic: Remote is closed
Feb 23 10:13:34 x01tepmsdb8a writeengine[6661]: 34.632247 |0|0|0| E 19 CAL0001: ReadThreadFactory::CreateReadThread: read() error InetStreamSocket::readToMagic: Remote is closed
Feb 23 10:13:34 x01tepmsdb8a writeengine[6661]: 34.632623 |0|0|0| E 19 CAL0001: ReadThreadFactory::CreateReadThread: read() error InetStreamSocket::readToMagic: Remote is closed
Feb 23 10:13:44 x01tepmsdb8a writeengine[6661]: 44.644498 |0|0|0| E 19 CAL0001: ReadThreadFactory::CreateReadThread: read() error InetStreamSocket::readToMagic: Remote is closed
Feb 23 10:13:54 x01tepmsdb8a writeengine[6661]: 54.659748 |0|0|0| E 19 CAL0001: ReadThreadFactory::CreateReadThread: read() error InetStreamSocket::readToMagic: Remote is closed
Feb 23 10:14:32 x01tepmsdb8a writeengine[6661]: 32.599303 |0|0|0| E 19 CAL0001: ReadThreadFactory::CreateReadThread: read() error InetStreamSocket::readToMagic: Remote is closed
Feb 23 10:15:32 x01tepmsdb8a writeengine[6661]: 32.605321 |0|0|0| E 19 CAL0001: ReadThreadFactory::CreateReadThread: read() error InetStreamSocket::readToMagic: Remote is closed
Feb 23 10:16:32 x01tepmsdb8a writeengine[6661]: 32.606814 |0|0|0| E 19 CAL0001: ReadThreadFactory::CreateReadThread: read() error InetStreamSocket::readToMagic: Remote is closed
Feb 23 10:17:32 x01tepmsdb8a writeengine[6661]: 32.600418 |0|0|0| E 19 CAL0001: ReadThreadFactory::CreateReadThread: read() error InetStreamSocket::readToMagic: Remote is closed
Feb 23 10:18:32 x01tepmsdb8a writeengine[6661]: 32.599711 |0|0|0| E 19 CAL0001: ReadThreadFactory::CreateReadThread: read() error InetStreamSocket::readToMagic: Remote is closed
Feb 23 10:19:32 x01tepmsdb8a writeengine[6661]: 32.599982 |0|0|0| E 19 CAL0001: ReadThreadFactory::CreateReadThread: read() error InetStreamSocket::readToMagic: Remote is closed
Feb 23 10:20:32 x01tepmsdb8a writeengine[6661]: 32.606291 |0|0|0| E 19 CAL0001: ReadThreadFactory::CreateReadThread: read() error InetStreamSocket::readToMagic: Remote is closed
Feb 23 10:21:32 x01tepmsdb8a writeengine[6661]: 32.606890 |0|0|0| E 19 CAL0001: ReadThreadFactory::CreateReadThread: read() error InetStreamSocket::readToMagic: Remote is closed
Feb 23 10:22:32 x01tepmsdb8a writeengine[6661]: 32.603574 |0|0|0| E 19 CAL0001: ReadThreadFactory::CreateReadThread: read() error InetStreamSocket::readToMagic: Remote is closed
Feb 23 13:24:37 x01tepmsdb8a workernode[6428]: 37.761151 |0|0|0| C 30 CAL0000: An error occurred: CompressedInetStreamSocket::accept sync: Connection reset by peer
Feb 23 13:24:48 x01tepmsdb8a controllernode[5023]: 48.669462 |0|0|0| E 29 CAL0000: DBRM: error: SessionManager::getSystemState() failed (network)
Feb 23 13:25:02 x01tepmsdb8a controllernode[5023]: 02.953492 |0|0|0| E 29 CAL0000: DBRM: error: SessionManager::clearSystemState() failed (network)
Feb 23 13:25:02 x01tepmsdb8a controllernode[5023]: 02.953923 |0|0|0| E 29 CAL0000: DBRM: error: SessionManager::clearSystemState() failed (network)
Feb 23 13:25:02 x01tepmsdb8a controllernode[5023]: 02.962947 |0|0|0| E 29 CAL0000: DBRM: error: SessionManager::clearSystemState() failed (network)
Feb 23 13:25:02 x01tepmsdb8a controllernode[5023]: 02.964940 |0|0|0| E 29 CAL0000: DBRM: error: SessionManager::clearSystemState() failed (network)
Feb 23 13:25:03 x01tepmsdb8a controllernode[5023]: 03.824580 |0|0|0| E 29 CAL0000: DBRM: error: SessionManager::clearSystemState() failed (network)
Feb 23 13:25:03 x01tepmsdb8a controllernode[5023]: 03.825384 |0|0|0| E 29 CAL0000: DBRM: error: SessionManager::clearSystemState() failed (network)
Feb 23 13:25:03 x01tepmsdb8a controllernode[5023]: 03.825586 |0|0|0| E 29 CAL0000: DBRM: error: SessionManager::clearSystemState() failed (network)
Feb 23 13:25:03 x01tepmsdb8a controllernode[5023]: 03.825837 |0|0|0| E 29 CAL0000: DBRM: error: SessionManager::clearSystemState() failed (network)
Feb 23 13:25:07 x01tepmsdb8a controllernode[5023]: 07.911440 |0|0|0| E 29 CAL0000: DBRM: error: SessionManager::setSystemState() failed (network)
Feb 23 13:25:07 x01tepmsdb8a controllernode[5023]: 07.912174 |0|0|0| E 29 CAL0000: DBRM: error: SessionManager::setSystemState() failed (network)
Feb 23 13:25:07 x01tepmsdb8a controllernode[5023]: 07.917168 |0|0|0| E 29 CAL0000: DBRM: error: SessionManager::setSystemState() failed (network)
Feb 23 13:25:07 x01tepmsdb8a controllernode[5023]: 07.918586 |0|0|0| E 29 CAL0000: DBRM: error: SessionManager::setSystemState() failed (network)
Feb 23 13:26:29 x01tepmsdb8a controllernode[5023]: 29.528970 |0|0|0| E 29 CAL0000: DBRM: error: SessionManager::getSystemState() failed (network)
[UM1 log]
2020-02-23 13:24:14 0 [Note] /home/mcsadm/mariadb/columnstore/mysql//bin/mysqld (initiated by: unknown): Normal shutdown
2020-02-23 13:24:14 0 [Note] Event Scheduler: Purging the queue. 0 events
2020-02-23 13:24:14 0 [Note] InnoDB: FTS optimize thread exiting.
2020-02-23 13:24:14 0 [Note] InnoDB: Starting shutdown...
2020-02-23 13:24:14 0 [Note] InnoDB: Dumping buffer pool(s) to /home/mcsadm/mariadb/columnstore/mysql/db/ib_buffer_pool
2020-02-23 13:24:14 0 [Note] InnoDB: Buffer pool(s) dump completed at 200223 13:24:14
2020-02-23 13:24:15 0 [Note] InnoDB: Shutdown completed; log sequence number 2533040783; transaction id 3785986
2020-02-23 13:24:15 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1"
2020-02-23 13:24:15 0 [Note] /home/mcsadm/mariadb/columnstore/mysql//bin/mysqld: Shutdown complete
[Critical Log]
Feb 23 13:24:37 x01tepmsdb8a workernode[6428]: 37.761151 |0|0|0| C 30 CAL0000: An error occurred: CompressedInetStreamSocket::accept sync: Connection reset by peer