OutOfMemoryError

Revision as of 18:04, 20 June 2008

Most memory problems are due to memory leaks in the application program. For example, a cache or a vector that fills up with stale data, or a singleton or static variable which doesn't properly detect a web-app reload. Some more exotic memory issues relate to running out of heap memory or virtual memory when using a large number of classes or threads.

Troubleshooting steps

The steps to track down a memory problem are:

• Enable -verbosegc. The -verbosegc flag causing logging of the garbage collection of the heap, which will let you know if you're running out of heap memory (the most common case).
• If you are using Java 6, specify <java-arg>-XX:+HeapDumpOnOutOfMemoryError</java-arg> in resin.conf
• Get a heap profiler or use the heap dump in the JVM. JProfiler is an inexpensive commercial heap profiler. Although the JVM's heap dump is not very user friendly, it's always available. Java developers should be using a heap profiler as part of the development process and before any production launch.
• With the heap profiler, find the 2-3 top memory users and fix those memory leaks.

Common application errors

Common application errors include:

• Singleton or static hash maps and caches, esp check for clearing web-app restarts.
• Spawned threads which are not stopped on web-app restart.
• web-app variables (like the "application (variable)" variable), stored in a static member of a class
• ThreadLocal variables that are not properly cleared at the end of each request.

-verbosegc

Main article: Garbage collection

-verbosegc causes the JVM to output debugging information about garbage colleciton. It is very convenient for checking basic memory usage and garbage collection time. It's a good idea to use -verbosegc for any production system. When starting Resin, you'll use -J-verbosegc.

The specific output depends on the JVM, and looks something like the following:

[GC 9176K->8647K(9768K), 0.0014790 secs]
[GC 9287K->8668K(9768K), 0.0011120 secs]
[GC 9308K->8668K(9768K), 0.0007810 secs]

The "(9768K)" is the most important number, showing that the maximum allocated heap is about 10 meg. The other numbers show the actual heap used before and after garbage collection.

Heap dump

Main article: Heap dump

If an application gobbles up memory until it throws an OutOfMemory exception, or seems to be spending an excessive amount of time doing garbage collection, a heap dump can help track down the source of the problem.

Types of memory

Normal heap 

The JVM heap size. Use -verbosegc to debug this.

Permanent heap 

An index into the heap for permanent objects such as classes. Configured with -XX:MaxPermSize. The default is generally too small.

malloc() memory 

Used by JNI and in small parts by the JVM, does not show up as part of the JVM heap but should be shown by operating system tools.

mmap memory 

On Unix .jar files are mmapped taking up virtual memory. mmap memory does not show up as part of the JVM heap but should be shown by operating system tools.

Thread stacks 

Each thread has preallocated stack memory. Resin httpd sets the stack size tpo a reasonable 1 meg unless an explicit value is provided on the command line.

Further troubleshooting

If the heap is clean (-verbosegc shows a steady heap), look at non-heap memory:

• JNI memory. If you're using JNI libraries or drivers that use JNI, it's possible that the JNI can allocate more memory than is available.
• fork/exec and OS limits. If the OS does not have enough available swap space, for example, the OS might refuse a "jikes" compilation.
• NIO, memory mapping, and .jar files. The JDK memory-maps .jar files. In some cases with very large numbers of jars, this can result in running out of virtual memory. The same issue can appear for NIO memory mapping.

If all of these are eliminated, it might be a Resin bug. However, you will need to have located the memory leak as Resin-related before any memory-related bug report, i.e. it's necessary to go through the above steps before reporting a bug. Bug reports that complain about OutOfMemory errors, but have not even gotten a JDK memory dump are most likely application errors. You must provide a heap dump when reporting any potential Resin memory problems.

OutOfMemoryError: PermGen

Main article: java.lang.OutOfMemoryError: PermGen space

This error occurs when the JVM runs out of space in the permanent generation heap.

If the application(s) in the server use a large number of classes, the solution is to increase the value specified with -XX:MaxPermSize.

If the error occurs only after the redeployment and restart of new applications, then the likely cause is that the JVM cannot garbage collect old classes that are replaced because there are references to the old classes, as discussed in Classloader references.

The error can also be caused by an interaction between the JDK and debugging agents (triggered by introspecting fields with primitive arrays.) You may need to disable agents with some JDKs to avoid these memory problems.

OutOfMemoryError: excessive garbage collection time

If the JVM is taking an excessive amount of time in garbage collection, it may through an OutOfMemoryError. The documentation on Garbage collection contains information about how to monitor and adjust the garbage collector.

See also

Garbage collection 

Java.lang.OutOfMemoryError: PermGen space