out of memory - "java.lang.OutOfMemoryError : unable to create new native Thread"

Question

We are getting "java.lang.OutOfMemoryError : unable to create new native Thread" on 8GB RAM VM after 32k threads (ps -eLF| grep -c java)

However, "top" and "free -m" shows 50% free memory available. JDk is 64 bit and tried with both HotSpot and JRockit.Server has Linux 2.6.18

We also tried OS stack size (ulimit -s) tweaking and max process(ulimit -u) limits, limit.conf increase but all in vain.

Also we tried almost all possible of heap size combinations, keeping it low, high etc.

The script we use to run application is

/opt/jrockit-jdk1.6/bin/java -Xms512m -Xmx512m -Xss128k -jar JavaNatSimulator.jar /opt/tools/jnatclients/natSimulator.properties

Thanks for the reply.

We have tried editing /etc/security/limits.conf and ulimit but still that same

[root@jboss02 ~]# ulimit -a
core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 72192
max locked memory       (kbytes, -l) 32
max memory size         (kbytes, -m) unlimited
open files                      (-n) 65535
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) 10240
cpu time               (seconds, -t) unlimited
max user processes              (-u) 72192
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited

Answer 1

This is not a memory problem even though the exception name highly suggests so, but an operating system resource problem. You are running out of native threads, i.e. how many threads the operating system will allow your JVM to use.

This is an uncommon problem, because you rarely need that many. Do you have a lot of unconditional thread spawning where the threads should but doesn't finish?

You might consider rewriting into using Callable/Runnables under the control of an Executor if at all possible. There are plenty of standard executors with various behavior which your code can easily control.

(There are many reasons why the number of threads is limited, but they vary from operating system to operating system)