![]() Trying to reproduce the hypothesis by other means Various possibilities were considered: that the too many clients error was a red herring, or that prlimit was not really working because some odd kernel setting was missing, or even that the ubuntu kernel did not support it.Īt this point I still wasn't able to confirm my hypothesis, and the errors were (of course) only appearing in production. I was puzzled because the linux kernel reported the limits being changed. ![]() I compiled the package and uploaded to our packagecloud repository.Īfter deploying the package I tried to change the limits of the running process, but the next day the errors were still there. Fortunately the engineers at Shopify have already done that in a Github repository with util-linux backports. I tried writing directly to /proc//limits, as sugested by this tweet, but that didn't work either. ![]() There is a tool called prlimit that provides a way to change limits on a running process, but for reasons I haven't been able to determine it was removed from the util-linux pacakge in Ubuntu 14.04, the Linux distribution we run. The first issue I ran into was that I did not want to restart stunnel in order to interrupt our workers from accessing redis. My first decision was to verify that hypothesis. Increasing limits in a running process to confirm the first hypothesis 2016.10.27 12:59:20 LOG4: Connection rejected: too many clients (>=500)Īfter a quick search on Google, I found out that stunnel was somehow dependant on the limits of resources imposed on the shell by the Linux kernel, specifically the total number of open file descriptors in this case.
0 Comments
Leave a Reply. |
AuthorWrite something about yourself. No need to be fancy, just an overview. ArchivesCategories |