I recently encountered a limitation during an experiment I was conducting; after some trial and error, I recognized that the limitation was due to cgroups.
But let’s start from the beginning. I open sourced docker-salt, a small pet project I had in mind in order to have a full blown setup for SaltStack: a master with an army of minions. Now for the fun part: what if I really start a hundred of minions on a server that has 16GB of RAM ready to be stressed with SaltStack?
yankee:~ # docker run -d --hostname saltmaster --name saltmaster -v `pwd`/srv/salt:/srv/salt -p 8000:8000 -ti mbologna/saltstack-master
yankee:~ # for i in {1..100}; do docker run -d --hostname saltminion$i --name saltminion$i --link saltmaster:salt mbologna/saltstack-minion ; done
When reaching around the ~50th container created, Docker cannot start containers anymore:
[...]
a9e72a3b9452d1ff23628ab431e1b3127a0cbf218bfa179d602230f676e3740
docker: Error response from daemon: containerd: container not started.
a827de31439a2937ceebd8769e742038c395c9543e548071f36058789b9b144c
docker: Error response from daemon: invalid header field value "oci runtime error: container_linux.go:247: starting container process caused \"process_linux.go:237: starting init process command caused \\\"fork/exec /proc/self/exe: resource temporarily unavailable\\\"\"\n".
[...]
By looking at the logs, we can see a more verbose message:
yankee containerd[2072]: time="2017-04-20T22:59:10.608383236+02:00" level=error msg="containerd: start container" error="oci runtime error: container_linux.go:247: starting container process caused \"process_linux.go:243: running exec setns process for init caused \\\"exit status 6\\\"\"\n" id=aa642284b64dc97a519f6d33004d4a1468c13b9ef52bb05338fc09396631567f
The problem here is that we reached the limit of the cgroup imposed for containerd, so we cannot fork any new process to spawn a new container.
The solution is pretty easy: open /usr/lib/systemd/system/containerd.service and add the directive TasksMax=infinity to overcome the problem:
[Service]
[...]
TasksMax=infinity
[...]
Issue a systemctl daemon-reload followed by systemctl restart containerd and you are good to go. Now the army of 100 minions can be started (sky is the limit!)