[MDEV-21438] Can't create a new thread (errno 35 "Resource temporarily unavailable") Created: 2020-01-08  Updated: 2020-01-08

Status: Open
Project: MariaDB Server
Component/s: None
Affects Version/s: 10.3.21
Fix Version/s: None

Type: Bug Priority: Major
Reporter: TAO ZHOU Assignee: Unassigned
Resolution: Unresolved Votes: 0
Labels: None
Environment:

FreeBSD 12.1


Attachments: File my.cnf    

 Description   

We are running mariadb galera cluster on three nodes(two data nodes, one garb).
We just had a huge outage.
The error we are getting is

ERROR 1135 (HY000): Can't create a new thread (errno 35 "Resource temporarily unavailable"); if you are not out of available memory, you can consult the manual for a possible OS-dependent bug

The error appeared intermittently on both data nodes.

The servers got plenty of memory, so not a memory issue.

I have attached our `my.cnf`



 Comments   
Comment by TAO ZHOU [ 2020-01-08 ]

Here are some `sysctl` that might worth looking

kern.maxvnodes: 1097062
kern.maxproc: 4116
kern.maxfiles: 2092259
kern.argmax: 262144
kern.maxfilesperproc: 1883025
kern.maxprocperuid: 3704
kern.ipc.maxsockbuf: 2097152
kern.ipc.max_linkhdr: 16
kern.ipc.max_protohdr: 60
kern.ipc.max_hdr: 76
kern.ipc.max_datalen: 92
kern.ipc.maxsockets: 2092260
kern.ipc.shmmax: 536870912
kern.ipc.msgmax: 16384
kern.ipc.aio.max_procs: 32
kern.ipc.maxpipekva: 1071235072
kern.ipc.umtx_max_robust: 1000
kern.ipc.maxmbufmem: 33397762048
kern.iov_max: 1024
kern.maxphys: 131072
kern.smp.maxcpus: 256
kern.smp.maxid: 15
kern.maxssiz: 536870912
kern.maxdsiz: 34359738368
kern.maxtsiz: 34359738368
kern.bio_transient_maxcnt: 1024
kern.maxbcache: 0
kern.maxswzone: 0
kern.threads.max_threads_hits: 856
kern.threads.max_threads_per_proc: 1500
kern.sigqueue.max_pending_per_proc: 128
kern.racct.rctl.throttle_max: 4294967295
kern.racct.rctl.maxbufsize: 16777216
kern.pid_max: 99999
kern.maxusers: 256

Generated at Thu Feb 08 09:07:08 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.