out of memory - Java OutOfMemory exception: mmap error on loading zip file

Question

I run my app on production env (rhel 5.2 x64, oracle jre 1.7_05, tomcat 7.0.28) with JVM arguments:

-Xms8192m -Xmx8192m -XX:MaxPermSize=1024m 
-Doracle.net.tns_admin=/var/ora_net -XX:ReservedCodeCacheSize=512m -XX:+AggressiveOpts -XX:+UseFastAccessorMethods 
-XX:+UseStringCache -XX:+OptimizeStringConcat -XX:+UseCompressedOops -XX:+UseG1GC -Dcom.sun.management.jmxremote 
-Dcom.sun.management.jmxremote.port=9026 -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.authenticate=false

After several time i've got stack trace like that:

Java HotSpot(TM) 64-Bit Server VM warning: Attempt to deallocate stack guard pages failed.
Java HotSpot(TM) 64-Bit Server VM warning: Attempt to allocate stack guard pages failed.
mmap failed for CEN and END part of zip file
[...]
Caused by: java.lang.OutOfMemoryError: null
    at java.util.zip.ZipFile.$$YJP$$open(Native Method) ~[na:1.7.0_05]
    at java.util.zip.ZipFile.open(Unknown Source) ~[na:1.7.0_05]
    at java.util.zip.ZipFile.<init>(Unknown Source) ~[na:1.7.0_05]
    at java.util.zip.ZipFile.<init>(Unknown Source) ~[na:1.7.0_05]
    at java.util.jar.JarFile.<init>(Unknown Source) ~[na:1.7.0_05]
    at java.util.jar.JarFile.<init>(Unknown Source) ~[na:1.7.0_05]
    at sun.net.www.protocol.jar.URLJarFile.<init>(Unknown Source) ~[na:1.7.0_05]
    at sun.net.www.protocol.jar.URLJarFile.getJarFile(Unknown Source) ~[na:1.7.0_05]
    at sun.net.www.protocol.jar.JarFileFactory.get(Unknown Source) ~[na:1.7.0_05]
    at sun.net.www.protocol.jar.JarURLConnection.connect(Unknown Source) ~[na:1.7.0_05]
    at sun.net.www.protocol.jar.JarURLConnection.getInputStream(Unknown Source) ~[na:1.7.0_05]
    at java.net.URL.openStream(Unknown Source) ~[na:1.7.0_05]
    at org.apache.catalina.loader.WebappClassLoader.findLoadedResource(WebappClassLoader.java:3279) ~[na:na]
    at org.apache.catalina.loader.WebappClassLoader.getResourceAsStream(WebappClassLoader.java:1478) ~[na:na]
    at org.apache.http.util.VersionInfo.loadVersionInfo(VersionInfo.java:242) ~[httpcore-4.2.jar:4.2]
    at org.apache.http.impl.client.DefaultHttpClient.setDefaultHttpParams(DefaultHttpClient.java:180) ~[httpclient-4.2.jar:4.2]
    at org.apache.http.impl.client.DefaultHttpClient.createHttpParams(DefaultHttpClient.java:158) ~[httpclient-4.2.jar:4.2]
    at org.apache.http.impl.client.AbstractHttpClient.getParams(AbstractHttpClient.java:448) ~[httpclient-4.2.jar:4.2]

Looking to my profiler - everthing is ok (heap and non-heap memory used for 10%) and i have no idea where is the problem.

This problem's happening every day at same time and it's not connected to application uptime. What is cause it problem?

Edited:

New output in log file:

Java HotSpot(TM) 64-Bit Server VM warning: CodeCache is full. Compiler has been disabled.
Java HotSpot(TM) 64-Bit Server VM warning: Try increasing the code cache size using -XX:ReservedCodeCacheSize=
Code Cache  [0x00002aaaab790000, 0x00002aaaad240000, 0x00002aaacb790000)
 total_blobs=4223 nmethods=3457 adapters=707 free_code_cache=497085Kb largest_free_block=508887936

But i have enough memory: http://i.stack.imgur.com/K8VMx.jpg

Answer: Problem in java version. It descripted here: https://forums.oracle.com/forums/thread.jspa?messageID=10369413

Answer 1

I have seen these error before when running out of resources such as running out of swap space or running out of allowed memory mapping. Have a look at sudo cat /proc/$PID/maps | wc -l compared with cat /proc/sys/vm/max_map_count

See comments below.

---

I also suggested ....

You appear to have run into a bug with YourKit. What version are you using?

I would cut down most of your options as they either are the default and don't do anything or could be complicating matters.

-mx8g -XX:MaxPermSize=1g -Doracle.net.tns_admin=/var/ora_net 
-XX:ReservedCodeCacheSize=512m -XX:+UseG1GC -Dcom.sun.management.jmxremote.port=9026

I would try dropping -XX:+UseG1GC as well as this is a relatively new collector and shouldn't change your results.