java.lang.OutOfMemoryError: Metaspace

Solution Verified - Updated -

Environment

  • OpenJDK 1.8
  • Oracle JDK 1.8
  • JBoss Enterprise Application Platform (EAP)
    • 6.4.x
    • 7.x

Issue

  • java.lang.OutOfMemoryError: Metaspace

Resolution

  • Increase the size of the Metaspace using the -XX:MaxMetaspaceSize argument. For example:

    -XX:MaxMetaspaceSize=2g

  • When UseCompressedOops and UseCompressedClassesPointers are enabled (default), increase the size of the class metadata space. the following options will result in a 2g space for class metadata and 1g space for compressed class pointers:

    -XX:MetaspaceSize=2g -XX:MaxMetaspaceSize=2g -XX:CompressedClassSpaceSize=1g

  • See "Metadata GC Threshold" garbage collections are not able to free space.

  • See JDK 8 Metaspace tuning for JBoss EAP .

Root Cause

Diagnostic Steps

Server Log Analysis

Verify the OOME is due to Metaspace.

GC Log Analysis

Verify a full collection does not reclaim Metaspace.

Classloader Stats Analysis

JDK8 u13+

  • Enable -XX:+UnlockDiagnosticVMOptions then capture GC.class_stats with jcmd (JDK 1.8 u13+):

    $JAVA_HOME/bin/jcmd <JBOSS_JAVA_PID> GC.class_stats -all -csv > class_stats.csv

    Notes:

    • Forces a full gc.
    • Open class_stats.csv in a spreadsheet (e.g. LibreOffice Calc).
    • Check InstBytes column for largest retainers.
    • Review the largest retainers in the heap dump to look for clues. See below.

JDK 1.8 u13-

  • Get clstats output from a time of high Metaspace piped to a file:

    $JAVA_HOME/bin/jmap -clstats <JAVA_PID> > clstats.out

    Example output:

    Attaching to process ID 12345, please wait...
Debugger attached successfully.
Server compiler detected.
JVM version is 25.101-b13
class_loader    classes bytes   parent_loader   alive?  type

<bootstrap>     2631    4705272   null          live    <internal>
0x00000000c91be518      1       1455    0x00000000fdcf0978      dead    sun/reflect/DelegatingClassLoader@0x00000001000099d8
0x00000000c377a770      0       0       0x00000000fd618000      live    org/eclipse/osgi/internal/loader/EquinoxClassLoader@0x00000001000bf438
0x00000000c64f4a48      454     772114  0x00000000fd618000      live    org/eclipse/osgi/internal/loader/EquinoxClassLoader@0x00000001000bf438
...
0x00000000c7651c90      1       864     0x00000000c35c1d98      dead    sun/reflect/DelegatingClassLoader@0x00000001000099d8
0x00000000c2ff5108      54      136855  0x00000000fd618000      live    org/eclipse/osgi/internal/loader/EquinoxClassLoader@0x00000001000bf438
0x00000000c76f9898      1       1455    0x00000000c3eb0f60      dead    sun/reflect/DelegatingClassLoader@0x00000001000099d8

total = 426     31560   64301535            N/A         alive=277, dead=149         N/A

    Notes:

    • Do not use this for JDK 1.8 u13+. It does not seem to provide useful information.
    • Requires debuginfo.
    • Requires running jmap as the user that started the Java process.
    • Will put the JVM in a safepoint (all threads stopped) while running.
    • Make take several minutes to produce output.
    • Forces a full gc.

Heap Dump Analysis

Get a heap dump when OOME happens using the -XX:+HeapDumpOnOutOfMemoryError option and check the following:

  • Use the following Object Query Language (OQL) query to find duplicate deployments (e.g. duplicate application war/ear/jars):

    #EAP6
SELECT module.identifier.name.value.toString() FROM org.jboss.modules.ModuleClassLoader
#EAP7
SELECT module.name.value.toString() FROM org.jboss.modules.ModuleClassLoader

  • Check for multiple instances of org.apache.cxf.common.jaxb.JAXBContextCache$2.

  • Review the classes that classloader stats analysis showed are the largest retainers for clues. For example, what are the immediate dominators? What are the GC root(s)?
  • Check the stack traces of running threads and review for patterns.

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.

Was this helpful?

Yes No

Get notified when this content is updated

You'll get an email whenever this content is updated or others comment. You can manage all of your notifications in your profile

1 Comments

XX
HT Community Member 35 points
24 October 2018 7:47 AM

In our environment the metaspace problem could be solved by changing the undertow configuration. We changed jsp-config development back to "false" (default). With "true" metaspace is growing permanently. <websockets/> Perhaps this helps as another possible solution.