Short answer
Doubling java heap size, doubles the waiting time for Garbage Collection pauses, that with current JVM technologies become of multiple seconds when the heap is in the Gb order. It seems that the newly released Java 7 is going to change that.
Long answer
The MaxPermSize is the maximum size for the permanent generation heap, a heap that holds the byte code of classes and is kept separated from the object heap containing the actual instances. One thing to keep in mind in case of web applications, is that in each hot re-deployment this memory usage is going to increase with multiple copies of the same classes. Unless -XX:+CMSClassUnloadingEnabled is specified.
Max Heap Size (-Xmx) and MaxPermSize have to be set taking into consideration how much memory is needed by the application classes and instances, the total memory of the server and the memory needed by other applications.
Another important point is that expanding the memory from the Min Heap Size (-Xms) is a costly operation. Especially in case of financial applications this could mean delays. With similar real-time requirements, it could be a good idea to set -Xms and -Xmx to the same value.
The following talk may be of interest http://www.infoq.com/presentations/Java-without-the-GC-Pauses
From the talk you can evince that the side effect of increasing the heap more than a certain limit 2Gb, 10Gb, 100Gb, means longer pause time for Garbage Collection that could stop everything for seconds or a minute in case of very large heaps.
For that reason with current JVM technology, you want to set the heap large enough to run your application, but not too large. A way to find the right size, could be set it to the largest possible value, then cut in half and keeping on cutting in half until you find problems, when you do, double the last found value and keep it as the chosen heap size for production.
This advice of course applies for large heaps, in the Gb order, if your application runs fine with 256Mb of memory, I would just keep that value without further investigation.
And finally for reference here are some example settings:
-Xms512m -Xmx512m -XX:MaxPermSize=256m -XX:+CMSClassUnloadingEnabled
与恶龙缠斗过久,自身亦成为恶龙;凝视深渊过久,深渊将回以凝视…