|
Hi
I have a MariaDB Galera cluster with three nodes. Let's say the node A, B and C.
The node A was detected that disk is full on 2023-01-20 at 20:07:27. I giving you the log results from three nodes. The log content is really a sample aboout what was happening.
The message '[Warning] Aborted connection 0 to db: 'unconnected' user: 'unauthenticated' host: 'connecting host' (Too many connections' appeared on node C 500 times more, on node B 1000 times more and node A only 10 times.
Node A, mysql_error.log:
2023-01-20 17:47:46 10913016 [Warning] Aborted connection 10913016 to db: 'app1' user: 'app1' host: '192.168.1.16' (Got an error reading communi cation packets)
|
2023-01-20 18:09:08 10914951 [Warning] Aborted connection 10914951 to db: 'app1' user: 'app1' host: '192.168.1.16' (Got an error reading communi cation packets)
|
2023-01-20 19:33:14 10922448 [Warning] Aborted connection 10922448 to db: 'app2' user: 'app2' host: '192.168.1.16' (Got an error reading communi cation packets)
|
2023-01-20 20:07:27 3449683 [ERROR] mariadbd: Disk full (/tmp/#sql-temptable-cfe-34a353-3c714c.MAI); waiting for someone to free some space... ( errno: 28 "No space left on device")
|
2023-01-20 20:11:04 2 [Warning] mariadbd: Disk is full writing './mariadb-bin.000201' (Errcode: 28 "No space left on device"). Waiting for someo ne to free space... (Expect up to 60 secs delay for server to continue after freeing disk space)
|
2023-01-20 20:11:04 2 [Warning] mariadbd: Retry in 60 secs. Message reprinted in 600 secs
|
2023-01-20 20:12:03 10926329 [ERROR] mariadbd: Disk full (/tmp/#sql-temptable-cfe-a6b831-58d1c.MAI); waiting for someone to free some space... ( errno: 28 "No space left on device")
|
2023-01-20 20:12:03 10926329 [ERROR] mariadbd: Disk full (/tmp/#sql-temptable-cfe-a6b831-58d1f.MAI); waiting for someone to free some space... ( errno: 28 "No space left on device")
|
2023-01-20 20:12:03 10926329 [ERROR] mariadbd: Disk full (/tmp/#sql-temptable-cfe-a6b831-58d22.MAI); waiting for someone to free some space... ( errno: 28 "No2023-01-21 11:34:30 0 [Warning] Aborted connection 0 to db: 'unconnected' user: 'unauthenticated' host: 'connecting host' (Too many connections)
|
2023-01-21 11:35:30 0 [Warning] Aborted connection 0 to db: 'unconnected' user: 'unauthenticated' host: 'connecting host' (Too many connections)
|
2023-01-21 11:35:30 0 [Warning] Aborted connection 0 to db: 'unconnected' user: 'unauthenticated' host: 'connecting host' (Too many connections)
|
2023-01-21 11:35:31 0 [Warning] Aborted connection 0 to db: 'unconnected' user: 'unauthenticated' host: 'connecting host' (Too many connections)
|
Node B, mysql_error.log:
2023-01-20 18:38:38 7131340 [Warning] Aborted connection 7131340 to db: 'app3' user: 'app3' host: '192.168.8.93' (Got timeout reading communication packets)
|
2023-01-20 19:41:57 7136919 [Warning] Aborted connection 7136919 to db: 'app3' user: 'app3' host: '192.168.8.93' (Got timeout reading communication packets)
|
2023-01-20 20:37:44 13 [Warning] Aborted connection 13 to db: 'information_schema' user: 'cmon' host: '192.168.8.12' (Got timeout reading communication packets)
|
2023-01-20 20:43:18 7142525 [Warning] Aborted connection 7142525 to db: 'information_schema' user: 'cmon' host: '192.168.8.12' (Got timeout reading communication packets)
|
2023-01-21 0:49:41 7164883 [Warning] Aborted connection 7164883 to db: 'app1' user: 'app1' host: '192.168.1.16' (Got an error reading communication packets)
|
2023-01-21 1:35:23 7168774 [Warning] Aborted connection 7168774 to db: 'app2' user: 'app2' host: '192.168.1.16' (Got an error reading communication packets)
|
2023-01-21 6:03:30 7192617 [Warning] Aborted connection 7192617 to db: 'app4' user: 'app4' host: '192.168.1.16' (Got an error reading communication packets)
|
2023-01-21 8:29:38 7205876 [Warning] Aborted connection 7205876 to db: 'app1' user: 'app1' host: '192.168.1.16' (Got an error reading communication packets)
|
2023-01-21 8:41:59 7206581 [Warning] Aborted connection 7206581 to db: 'information_schema' user: 'cmon' host: '192.168.8.12' (Got timeout reading communication packets)
|
2023-01-21 8:57:14 0 [Warning] Aborted connection 0 to db: 'unconnected' user: 'unauthenticated' host: 'connecting host' (Too many connections)
|
2023-01-21 8:57:44 0 [Warning] Aborted connection 0 to db: 'unconnected' user: 'unauthenticated' host: 'connecting host' (Too many connections)
|
2023-01-21 8:57:59 0 [Warning] Aborted connection 0 to db: 'unconnected' user: 'unauthenticated' host: 'connecting host' (Too many connections)
|
2023-01-21 8:57:59 0 [Warning] Aborted connection 0 to db: 'unconnected' user: 'unauthenticated' host: 'connecting host' (Too many connections)
|
Node C, mysql_error.log:
2023-01-20 18:38:38 7131340 [Warning] Aborted connection 7131340 to db: 'app3' user: 'app3' host: '192.168.8.93' (Got timeout reading communication packets)
|
2023-01-20 19:40:57 7136919 [Warning] Aborted connection 7136919 to db: 'app3' user: 'app3' host: '192.168.8.93' (Got timeout reading communication packets)
|
2023-01-20 20:37:44 13 [Warning] Aborted connection 13 to db: 'information_schema' user: 'cmon' host: '192.168.8.12' (Got timeout reading communication packets)
|
2023-01-20 20:43:18 7142525 [Warning] Aborted connection 7142525 to db: 'information_schema' user: 'cmon' host: '192.168.8.12' (Got timeout reading communication packets)
|
2023-01-21 0:49:41 7164883 [Warning] Aborted connection 7164883 to db: 'app1' user: 'app1' host: '192.168.1.16' (Got an error reading communication packets)
|
2023-01-21 1:33:23 7168774 [Warning] Aborted connection 7168774 to db: 'app2' user: 'app2' host: '192.168.1.16' (Got an error reading communication packets)
|
2023-01-21 6:00:50 7192617 [Warning] Aborted connection 7192617 to db: 'app4' user: 'app4' host: '192.168.1.16' (Got an error reading communication packets)
|
2023-01-21 8:29:38 7205876 [Warning] Aborted connection 7205876 to db: 'app1' user: 'app1' host: '192.168.1.16' (Got an error reading communication packets)
|
2023-01-21 8:41:59 7206581 [Warning] Aborted connection 7206581 to db: 'information_schema' user: 'cmon' host: '192.168.8.12' (Got timeout reading communication packets)
|
2023-01-21 8:55:04 0 [Warning] Aborted connection 0 to db: 'unconnected' user: 'unauthenticated' host: 'connecting host' (Too many connections)
|
2023-01-21 8:55:34 0 [Warning] Aborted connection 0 to db: 'unconnected' user: 'unauthenticated' host: 'connecting host' (Too many connections)
|
2023-01-21 8:55:39 0 [Warning] Aborted connection 0 to db: 'unconnected' user: 'unauthenticated' host: 'connecting host' (Too many connections)
|
2023-01-21 8:55:59 0 [Warning] Aborted connection 0 to db: 'unconnected' user: 'unauthenticated' host: 'connecting host' (Too many connections)
|
I executed the instruction on one node:
show status where `variable_name` = 'Threads_connected';
|
It returned Thread connected 101. It should return between 3 and 10, normally. MariaDB services on two nodes were active (systemctl status mariadb: active) but they weren't available for client applications to do connections and query executions.
I understand the two nodes couldn't attend the client applications because the connection pool was full. Even I can suspect if someone tried to go down the cluster by sending tcp requests.
What I did in order to make the cluster available was deleting files on node A to free space. Then I restarted the server from node A and it started to work again.
Questions:
If one node is disk full.
Could be this the real reason so that the cluster are not available?
Could be this the real reason for the message '[Warning] Aborted connection 0 to db: 'unconnected' user: 'unauthenticated' host: 'connecting host' (Too many connections)'?
|