Troubleshooting Native Memory Leaks in Java Applications

50 %
50 %
Information about Troubleshooting Native Memory Leaks in Java Applications

Published on October 25, 2018

Author: PoonamBajaj5

Source: slideshare.net

1. Copyright © 2018, Oracle and/or its affiliates. All rights reserved. | Troubleshooting Native Memory Leaks in Java Applications Poonam Parhar Consulting Member of Technical Staff Java Platform Group October, 2018

2. Copyright © 2018, Oracle and/or its affiliates. All rights reserved. | Safe Harbor Statement The following is intended to outline our general product direction. It is intended for information purposes only, and may not be incorporated into any contract. It is not a commitment to deliver any material, code, or functionality, and should not be relied upon in making purchasing decisions. The development, release, timing, and pricing of any features or functionality described for Oracle’s products may change and remains at the sole discretion of Oracle Corporation. Confidential – Oracle Internal/Restricted/Highly Restricted

3. Copyright © 2018, Oracle and/or its affiliates. All rights reserved. | About me • Poonam Parhar • JVM Sustaining Engineer at Oracle • Co-author of the book ‘Java Performance Companion’ • JavaOne RockStar • Speaker at Java Conferences • Published articles • Blog: https://blogs.oracle.com/poonam/ 3

4. Copyright © 2018, Oracle and/or its affiliates. All rights reserved. | Agenda • Layout of Memory From the Perspective of the JVM • Detect Native Memory Leaks • OutOfMemoryError: Native Memory • Common Issues and their Solutions • Approaches/Tools to Troubleshoot a Native Memory Leak

5. Copyright © 2018, Oracle and/or its affiliates. All rights reserved. | Virtual and Physical Memory • Address Space or Virtual Memory determined by address size e.g. 32-bit or 64-bit • Physical memory (RAM) Image source: https://en.wikipedia.org

6. Copyright © 2018, Oracle and/or its affiliates. All rights reserved. | Memory allocated by a Java application Java Heap - Memory allocated for JVM’s internal functioning such as Java Heap memory management - Memory used by loaded jars and native libraries - Thread stacks and thread-local storage - … Memory Storage of classes and metadata, including dynamically generated classes through reflection (Metaspace) JIT bytecode compilation and storage of compiled code (codecache) - JNI/JVMTI code allocating memory - NIO allocations - Direct Bytebuffers

7. Copyright © 2018, Oracle and/or its affiliates. All rights reserved. | Native leaks are hard to diagnose • Native allocations can come from several different places • Could be due to JNI allocations, JIT compiler allocations or creation of threads, …

8. Copyright © 2018, Oracle and/or its affiliates. All rights reserved. | Detect Native Memory Leaks

9. Copyright © 2018, Oracle and/or its affiliates. All rights reserved. | Demo A simple Java program with a native memory leak

10. Copyright © 2018, Oracle and/or its affiliates. All rights reserved. | Monitor Native Memory • ps, top, pmap on Unix systems – watch -n 1 ps v <pid> – top –p <pid> – Examine pmap output • PerfMon or VMMap on Windows

11. Copyright © 2018, Oracle and/or its affiliates. All rights reserved. | bash-3.2$ ps v 27890 PID STAT TIME SL RE PAGEIN VSZ RSS LIM TSIZ %CPU %MEM COMMAND 27890 S+ 0:01.68 0 0 0 10068904 63652 - 0 3.4 0.4 java -Djava.library.pat 27890 S+ 0:02.09 0 0 0 10076072 64604 - 0 3.6 0.4 java -Djava.library.pat 27890 S+ 0:02.57 0 0 0 10084268 65996 - 0 4.6 0.4 java -Djava.library.pat 27890 S+ 0:03.29 0 0 0 10099628 71592 - 0 3.6 0.4 java -Djava.library.pat 27890 S+ 0:03.60 0 0 0 10104748 72100 - 0 3.6 0.4 java -Djava.library.pat 27890 S+ 0:03.96 0 0 0 10111916 72644 - 0 3.5 0.4 java -Djava.library.pat 27890 S+ 0:04.37 0 0 0 10121136 73328 - 0 3.4 0.4 java -Djava.library.pat 27890 S+ 0:04.93 0 0 0 10129928 83688 - 0 3.1 0.5 java -Djava.library.pat 27890 S+ 0:05.44 0 0 0 10137060 84004 - 0 3.6 0.5 java -Djava.library.pat 27890 S+ 0:05.59 0 0 0 10139108 84272 - 0 3.2 0.5 java -Djava.library.pat ...

12. Copyright © 2018, Oracle and/or its affiliates. All rights reserved. |

13. Copyright © 2018, Oracle and/or its affiliates. All rights reserved. | poonam@poonam-VirtualBox:/media/sf_shared/native_leak_tests$ diff pmap.1 pmap.9 12a13,14 > 00007f4060000000 32276K rw--- [ anon ] > 00007f4061f85000 33260K ----- [ anon ] 56,57c58 < 00007f40a4000000 18952K rw--- [ anon ] < 00007f40a5282000 46584K ----- [ anon ] --- > 00007f40a4000000 65536K rw--- [ anon ] 146c147 < total 3222140K --- > total 3287676K poonam@poonam-VirtualBox:/media/sf_shared/native_leak_tests$ diff pmap.2 pmap.9 12a13,14 > 00007f4060000000 32276K rw--- [ anon ] > 00007f4061f85000 33260K ----- [ anon ] 56,57c58 < 00007f40a4000000 25600K rw--- [ anon ] < 00007f40a5900000 39936K ----- [ anon ] --- > 00007f40a4000000 65536K rw--- [ anon ] 146c147 < total 3222140K --- > total 3287676K

14. Copyright © 2018, Oracle and/or its affiliates. All rights reserved. | OutOfMemoryError: Native Memory 14

15. Copyright © 2018, Oracle and/or its affiliates. All rights reserved. | # A fatal error has been detected by the Java Runtime Environment: # # java.lang.OutOfMemoryError : unable to create new native Thread # A fatal error has been detected by the Java Runtime Environment: # # java.lang.OutOfMemoryError: requested 32756 bytes for ChunkPool::allocate. Out of swap space? # # Internal Error (allocation.cpp:166), pid=2290, tid=27 # Error: ChunkPool::allocate 15 Native OutOfMemoryError

16. Copyright © 2018, Oracle and/or its affiliates. All rights reserved. | Native OutOfMemoryError • JVM is not able to allocate from native memory – Not managed by the JVM • This process or other processes on the system are eating up native memory • Can make more room for native allocations by: – Reducing the Java Heap, PermGen/Metaspace, number of threads and their stack sizes etc. – Reducing the number of processes running on the system • If the above don’t help, we might be facing a native memory leak – Example: JNI code allocating native buffers 16

17. Copyright © 2018, Oracle and/or its affiliates. All rights reserved. | Common Issues: Native OutOfMemoryError 17

18. Copyright © 2018, Oracle and/or its affiliates. All rights reserved. | Native Heap OutOfMemoryError with 64-bit JVM • Running with 32-bit JVM puts a maximum limit of 4GB on the process size – So more likely you’ll run out of native memory with 32-bit Java processes • Running with a 64-bit JVM gets us access to unlimited memory, so we would expect never to run out of native memory • However, we might see OutOfMemoryErrors occurring in a 64-bit JVM too • CompressedOops feature implementation determines where the Java heap should be placed in the address space Ø Below 4GB address boundary – no scaling, no offset addition ØBelow 32GB address boundary – no offset addition • The position of the Java heap can put a cap on the maximum capacity of the native heap. 18

19. Copyright © 2018, Oracle and/or its affiliates. All rights reserved. | Memory Map 0000000100000000 8K r-x-- /sw/.es-base/sparc/pkg/jdk-1.7.0_60/bin/sparcv9/java 0000000100100000 8K rwx-- /sw/.es-base/sparc/pkg/jdk-1.7.0_60/bin/sparcv9/java 0000000100102000 56K rwx-- [ heap ] 0000000100110000 2624K rwx-- [ heap ] <--- native heap 00000001FB000000 24576K rw--- [ anon ] <--- Java Heap starts here 0000000200000000 1396736K rw--- [ anon ] 0000000600000000 700416K rw--- [ anon ] 19

20. Copyright © 2018, Oracle and/or its affiliates. All rights reserved. | Solution for OutOfMemoryError with 64-bit JVM • Can be resolved by using option -XX:HeapBaseMinAddress=n to specify the address the Java heap should be based at • Setting it to a higher address would leave more room for native heap 20

21. Copyright © 2018, Oracle and/or its affiliates. All rights reserved. | JAXB Issues • JAXB internally uses Inflater/Deflater to compress/uncompress files – Inflater/Deflater use native memory to hold their data – Depend on Finalizer to deallocate the java objects and the associated native memory data – Delay in running Finalizer can exhaust native memory • JAXBContext.newInstance() called for every new request – Classes from the context get reloaded again – Increases the native memory usage • Environment upgrade fails to upgrade all the JAXB jar files – Classes linking errors leading to re-loading of classes 21

22. Copyright © 2018, Oracle and/or its affiliates. All rights reserved. | OutOfMemoryError: Direct buffer memory • ByteBuffer.allocateDirect(SIZE_OF_BUFFER) • DirectByteBuffers are garbage collected by using a phantom reference and a reference queue • Maximum direct memory is unbounded, can be limited by using the JVM option -XX:MaxDirectMemorySize=n • We can explicitly call: ((DirectBuffer)buffer).cleaner().clean(); • Make GCs more frequent allowing DirectByteBuffers to be GC’ed 22

23. Copyright © 2018, Oracle and/or its affiliates. All rights reserved. | NIO ByteBuffers • Java NIO APIs use ByteBuffers as the source and destination of I/O calls • Java Heap ByteBuffers and Native Heap ByteBuffers • Java Heap ByteBuffer for I/O use a temporary direct ByteBuffer per thread • If large native ByteBuffers from multiple threads are used for I/O calls ØNative Memory Exhaustion • -Djdk.nio.maxCachedBufferSize=m (1.8.0_102, 9 and above) 23

24. Copyright © 2018, Oracle and/or its affiliates. All rights reserved. | Native Memory: Diagnostic Data 24

25. Copyright © 2018, Oracle and/or its affiliates. All rights reserved. | Native Memory: Diagnostic Data • Native memory leaks in the JVM • Native Memory Tracker (NMT) output • Native Memory Leaks outside the JVM – Tools like jemalloc, valgrind, libumem, dbx, purify, UMDH • Process map output with tools like pmap • Core file 25

26. Copyright © 2018, Oracle and/or its affiliates. All rights reserved. | Native Memory Tracker • Native Memory Tracker (NMT) that can be used to track native memory that is used internally by the JVM • It cannot track memory allocated outside the JVM or by native libraries • https://docs.oracle.com/javase/8/docs/technotes/guides/troubleshoot/to oldescr007.html 26

27. Copyright © 2018, Oracle and/or its affiliates. All rights reserved. | NMT • Start the process with NMT enabled using NativeMemoryTracking • The output level can be set to a ‘summary’ or ‘detail’ level: • -XX:NativeMemoryTracking=summary • -XX:NativeMemoryTracking=detail • Use jcmd to get the native memory usage details: • jcmd <pid> VM.native_memory • jcmd <pid> VM.native_memory baseline • jcmd <pid> VM.native_memory detail.diff/summary.diff 27

28. Copyright © 2018, Oracle and/or its affiliates. All rights reserved. | Demo Classes space leak

29. Copyright © 2018, Oracle and/or its affiliates. All rights reserved. |

30. Copyright © 2018, Oracle and/or its affiliates. All rights reserved. | 30 • bash-3.2$ jcmd 39057 VM.native_memory summary.diff • 39057: • Native Memory Tracking: • Total: reserved=5761678KB +52943KB, committed=472350KB +104143KB • - Java Heap (reserved=4194304KB, committed=163328KB +7680KB) • (mmap: reserved=4194304KB, committed=163328KB +7680KB) • • - Class (reserved=1118333KB +47579KB, committed=117949KB +89963KB) • (classes #68532 +58527) • (malloc=8317KB +2523KB #5461 +3371) • (mmap: reserved=1110016KB +45056KB, committed=109632KB +87440KB) • • - Thread (reserved=21594KB -2057KB, committed=21594KB -2057KB) • (thread #22 -2) • (stack: reserved=21504KB -2048KB, committed=21504KB -2048KB) • (malloc=65KB -6KB #111 -10) • (arena=25KB -2 #42 -4) • • - Code (reserved=250400KB +244KB, committed=5612KB +1348KB) • (malloc=800KB +244KB #1498 +234) • (mmap: reserved=249600KB, committed=4812KB +1104KB) • • - GC (reserved=159039KB +18KB, committed=145859KB +50KB) • (malloc=5795KB +18KB #856 +590) • (mmap: reserved=153244KB, committed=140064KB +32KB) • • - Compiler (reserved=153KB, committed=153KB) • (malloc=22KB #72 -2) • (arena=131KB #3) • • - Internal (reserved=13537KB +6949KB, committed=13537KB +6949KB) • (malloc=13505KB +6949KB #70630 +59119) • (mmap: reserved=32KB, committed=32KB) • • - Symbol (reserved=2715KB +9KB, committed=2715KB +9KB) • (malloc=1461KB +9KB #702 +29) • (arena=1255KB #1) • • - Native Memory Tracking (reserved=1416KB +1031KB, committed=1416KB +1031KB) • (malloc=140KB +34KB #2197 +518) • (tracking overhead=1275KB +997KB) • • - Arena Chunk (reserved=186KB -832KB, committed=186KB -832KB) • (malloc=186KB -832KB)

31. Copyright © 2018, Oracle and/or its affiliates. All rights reserved. | Native Memory Leaks Outside the JVM • For the native memory leaks stemming from outside the JVM, we need to rely on the native memory leak tools for their detection and troubleshooting • Native Memory Leak Detection Tools – jemalloc – valgrind – libumem – dbx – Purify – User-Mode Dump Heap (UMDH) 31

32. Copyright © 2018, Oracle and/or its affiliates. All rights reserved. | jemalloc • Memory allocator and profiler • FreeBSD – Scalability in terms of multithreading by avoiding lock contentions – Heap profiling capabilities • Very little performance impact (heavy malloc use, ~2%) – Can be used in production systems • Download it from https://github.com/jemalloc/jemalloc • ./configure --prefix=/usr/local --enable-prof • make

33. Copyright © 2018, Oracle and/or its affiliates. All rights reserved. | Demo Memory Leak with Deflater

34. Copyright © 2018, Oracle and/or its affiliates. All rights reserved. | Run application with jemalloc export JEMALLOC_PATH=/media/sf_shared/jemalloc-5.1.0 export MALLOC_CONF=prof:true,lg_prof_sample:0,prof_final:true Or export MALLOC_CONF=prof:true,lg_prof_interval:30,lg_prof_sample:21 LD_PRELOAD=${JEMALLOC_PATH}/lib/libjemalloc.so.2 java -Xmx1024m -Xms1024m - XX:MaxNewSize=20m TestDeflater

35. Copyright © 2018, Oracle and/or its affiliates. All rights reserved. | jeprof --show_bytes `which java` jeprof.23299.0.f.heap (jeprof) top Total: 341324080 B 325433344 95.3% 95.3% 325433344 95.3% deflateInit2_ 15292392 4.5% 99.8% 15292392 4.5% os::malloc@926750 368704 0.1% 99.9% 368704 0.1% readCEN 116448 0.0% 100.0% 325549792 95.4% Java_java_util_zip_Deflater_init 81920 0.0% 100.0% 81920 0.0% std::__once_callable 12016 0.0% 100.0% 12016 0.0% _dl_new_object 3840 0.0% 100.0% 3840 0.0% allocate_dtv 3616 0.0% 100.0% 3616 0.0% _nl_intern_locale_data 2048 0.0% 100.0% 2208 0.0% __tzfile_read 1984 0.0% 100.0% 1984 0.0% _dl_check_map_versions

36. Copyright © 2018, Oracle and/or its affiliates. All rights reserved. | • jeprof --show_bytes --gif `which java` jeprof.23299.0.f.heap > leak.gif

37. Copyright © 2018, Oracle and/or its affiliates. All rights reserved. | Valgrind • valgrind --leak-check=yes --log-file=valgrind_full.log -v java <options and class file> • --leak-check=summary | yes | full • May not be effective for Java applications – Reports allocations done by the JVM for its internal functioning • Impacts application performance significantly – 20 to 30 times slower • Can be useful in tracking down JVM memory leaks but we have NMT for that

38. Copyright © 2018, Oracle and/or its affiliates. All rights reserved. | Windows: User Mode Dump Heap (UMDH) tool

39. Copyright © 2018, Oracle and/or its affiliates. All rights reserved. | UMDH • Install Debugging Tools for Windows • Use ’gflags’ to set the ‘Create user mode stack trace database flag’ for the process – gflags /i java.exe +ust • https://docs.microsoft.com/en-us/windows- hardware/drivers/debugger/umdh • umdh -p:<pid> -f:first.log , umdh -p:<pid> -f:second.log, … • umdh -d -v -l first.log second.log > report.log

40. Copyright © 2018, Oracle and/or its affiliates. All rights reserved. | pmap and core file

41. Copyright © 2018, Oracle and/or its affiliates. All rights reserved. | $ diff pmap.15767.1 pmap.15767.3 69,70c69,70 < 00007f6d68000000 17036K rw--- [ anon ] < 00007f6d690a3000 48500K ----- [ anon ] --- > 00007f6d68000000 63816K rw--- [ anon ] > 00007f6d6be52000 1720K ----- [ anon ] ------------------------------------------------------------- $ gdb `which java` core.15767 GNU gdb (Ubuntu 7.11.1-0ubuntu1~16.5) 7.11.1 Copyright (C) 2016 Free Software Foundation, Inc. ... (gdb) x/100s 0x00007f6d690a3000 0x7f6d690a3000: "mory Leak " 0x7f6d690a300c: "Alert: JNI Memory Leak " 0x7f6d690a3025: "Alert: JNI Memory Leak " 0x7f6d690a303e: "Alert: JNI Memory Leak " 0x7f6d690a3057: "Alert: JNI Memory Leak " 0x7f6d690a3070: "Alert: JNI Memory Leak "

42. Copyright © 2018, Oracle and/or its affiliates. All rights reserved. | #include "JNINativeLeak.h" #include <stdlib.h> #include <memory.h> JNIEXPORT void JNICALL Java_JNINativeLeak_allocateMemory (JNIEnv *env, jobject obj, jint size) { char* bytes = (char*) malloc(size); printf("Allocated %d bytes at %p n", size, (void*)bytes); for (int i=0; i<40; i++) { strcpy(bytes+i*25, "Alert: JNI Memory Leak "); } }

43. Copyright © 2018, Oracle and/or its affiliates. All rights reserved. | Summary • Identify native memory leak • Monitor using j* tools and native tools such as ps, top, pmap • Rule out common issues • Leaks from the JVM – NMT • Leaks outside the JVM – Native tools: jemalloc, valgrind, libumem, purify, UMDH, pmap and core file • https://github.com/poonamparhar/native_leaks

44. Copyright © 2018, Oracle and/or its affiliates. All rights reserved. | Troubleshooting Native Memory Leaks in Java Applications Poonam Parhar Consulting Member of Technical Staff Java Platform Group October, 2018

Add a comment