On Android, a direct ByteBuffer does not ever seem to release its memory, not even when calling System.gc().
Example: doing
Log.v("?", Long.toString(Debug.getNativeHeapAllocatedSize()));
ByteBuffer buffer = allocateDirect(LARGE_NUMBER);
buffer=null;
System.gc();
Log.v("?", Long.toString(Debug.getNativeHeapAllocatedSize()));
gives two numbers in the log, the second one being at least LARGE_NUMBER larger than the first.
How do I get rid of this leak?
Added:
Following the suggestion by Gregory to handle alloc/free on the C++ side, I then defined
JNIEXPORT jobject JNICALL Java_com_foo_bar_allocNative(JNIEnv* env, jlong size)
{
void* buffer = malloc(size);
jobject directBuffer = env->NewDirectByteBuffer(buffer, size);
jobject globalRef = env->NewGlobalRef(directBuffer);
return globalRef;
}
JNIEXPORT void JNICALL Java_com_foo_bar_freeNative(JNIEnv* env, jobject globalRef)
{
void *buffer = env->GetDirectBufferAddress(globalRef);
free(buffer);
env->DeleteGlobalRef(globalRef);
}
I then get my ByteBuffer on the JAVA side with
ByteBuffer myBuf = allocNative(LARGE_NUMBER);
and free it with
freeNative(myBuf);
Unfortunately, while it does allocate fine, it a) still keeps the memory allocated according to Debug.getNativeHeapAllocatedSize() and b) leads to an error
W/dalvikvm(26733): JNI: DeleteGlobalRef(0x462b05a0) failed to find entry (valid=1)
I am now thoroughly confused, I thought I at least understood the C++ side of things... Why is free() not returning the memory? And what am I doing wrong with the DeleteGlobalRef()?
There is no leak.
ByteBuffer.allocateDirect() allocates memory from the native heap / free store (think malloc()) which is in turn wrapped in to a ByteBuffer instance.
When the ByteBuffer instance gets garbage collected, the native memory is reclaimed (otherwise you would leak native memory).
You're calling System.gc() in hope the native memory is reclaimed immediately. However, calling System.gc() is only a request which explains why your second log statement doesn't tell you memory has been released: it's because it hasn't yet!
In your situation, there is apparently enough free memory in the Java heap and the garbage collector decides to do nothing: as a consequence, unreachable ByteBuffer instances are not collected yet, their finalizer is not run and native memory is not released.
Also, keep in mind this bug in the JVM (not sure how it applies to Dalvik though) where heavy allocation of direct buffers leads to unrecoverable OutOfMemoryError.
You commented about doing controlling things from JNI. This is actually possible, you could implement the following:
publish a native ByteBuffer allocateNative(long size) entry point that:
calls void* buffer = malloc(size) to allocate native memory
wraps the newly allocated array into a ByteBuffer instance with a call to (*env)->NewDirectByteBuffer(env, buffer, size);
converts the ByteBuffer local reference to a global one with (*env)->NewGlobalRef(env, directBuffer);
publish a native void disposeNative(ByteBuffer buffer) entry point that:
calls free() on the direct buffer address returned by *(env)->GetDirectBufferAddress(env, directBuffer);
deletes the global ref with (*env)->DeleteGlobalRef(env, directBuffer);
Once you call disposeNative on the buffer, you're not supposed to use the reference anymore, so it could be very error prone. Reconsider whether you really need such explicit control over the allocation pattern.
Forget what I said about global references. Actually global references are a way to store a reference in native code (like in a global variable) so that a further call to JNI methods can use that reference. So you would have for instance:
from Java, call native method foo() which creates a global reference out of a local reference (obtained by creating an object from native side) and stores it in a native global variable (as a jobject)
once back, from Java again, call native method bar() which gets the jobject stored by foo() and further processes it
finally, still from Java, a last call to native baz() deletes the global reference
Sorry for the confusion.
I was using TurqMage's solution until I tested it on a Android 4.0.3 emulator (Ice Cream Sandwich). For some reason, the call to DeleteGlobalRef fails with a jni warning: JNI WARNING: DeleteGlobalRef on non-global 0x41301ea8 (type=1), followed by a segmentation fault.
I took out the calls to create a NewGlobalRef and DeleteGlobalRef (see below) and it seems to work fine on the Android 4.0.3 emulator.. As it turns out, I'm only using the created byte buffer on the java side, which should hold a java reference to it anyways, so I think the call to NewGlobalRef() was not needed in the first place..
JNIEXPORT jobject JNICALL Java_com_foo_allocNativeBuffer(JNIEnv* env, jobject thiz, jlong size)
{
void* buffer = malloc(size);
jobject directBuffer = env->NewDirectByteBuffer(buffer, size);
return directBuffer;
}
JNIEXPORT void JNICALL Java_comfoo_freeNativeBuffer(JNIEnv* env, jobject thiz, jobject bufferRef)
{
void *buffer = env->GetDirectBufferAddress(bufferRef);
free(buffer);
}
Not sure if your last comments are old or what Kasper. I did the following...
JNIEXPORT jobject JNICALL Java_com_foo_allocNativeBuffer(JNIEnv* env, jobject thiz, jlong size)
{
void* buffer = malloc(size);
jobject directBuffer = env->NewDirectByteBuffer(buffer, size);
jobject globalRef = env->NewGlobalRef(directBuffer);
return globalRef;
}
JNIEXPORT void JNICALL Java_comfoo_freeNativeBuffer(JNIEnv* env, jobject thiz, jobject globalRef)
{
void *buffer = env->GetDirectBufferAddress(globalRef);
env->DeleteGlobalRef(globalRef);
free(buffer);
}
Then in Java...
mImageData = (ByteBuffer)allocNativeBuffer( mResX * mResY * mBPP );
and
freeNativeBuffer(mImageData);
mImageData = null;
and everything seems to be working fine for me. Thanks a lot Gregory for this idea. The link to the referenced Bug in the JVM has gone bad.
Use the reflection to call java.nio.DirectByteBuffer.free(). I remind you that Android DVM is inspired by Apache Harmony, which supports the method above.
The direct NIO buffers are allocated on the native heap, not on the Java heap managed by the garbage collection. It's up to the developer to release their native memory. It's a bit different with OpenJDK and Oracle Java because they try to call the garbage collector when the creation of a direct NIO buffer fails but there is no guarantee that it helps.
N.B: You'll have to tinker a bit more if you use asFloatBuffer(), asIntBuffer(), ... because only the direct byte buffer can be "freed".
Related
Library written in c++ produces continuous stream of data and same has to be ported on different platforms. Now integrating the lib to android application, I am trying to create shared memory between NDK and SDK.
Below is working snippet,
Native code:
#include <jni.h>
#include <fcntl.h>
#include <sys/mman.h>
#include <linux/ashmem.h>
#include <android/log.h>
#include <string>
char *buffer;
constexpr size_t BufferSize=100;
extern "C" JNIEXPORT jobject JNICALL
Java_test_com_myapplication_MainActivity_getSharedBufferJNI(
JNIEnv* env,
jobject /* this */) {
int fd = open("/dev/ashmem", O_RDWR);
ioctl(fd, ASHMEM_SET_NAME, "shared_memory");
ioctl(fd, ASHMEM_SET_SIZE, BufferSize);
buffer = (char*) mmap(NULL, BufferSize, PROT_READ | PROT_WRITE, MAP_SHARED, fd, 0);
return (env->NewDirectByteBuffer(buffer, BufferSize));
}
extern "C" JNIEXPORT void JNICALL
Java_test_com_myapplication_MainActivity_TestBufferCopy(
JNIEnv* env,
jobject /* this */) {
for(size_t i=0;i<BufferSize;i = i+2) {
__android_log_print(ANDROID_LOG_INFO, "native_log", "Count %d value:%d", i,buffer[i]);
}
//pass `buffer` to dynamically loaded library to update share memory
//
}
SDK code:
//MainActivity.java
public class MainActivity extends AppCompatActivity {
// Used to load the 'native-lib' library on application startup.
static {
System.loadLibrary("native-lib");
}
final int BufferSize = 100;
#RequiresApi(api = Build.VERSION_CODES.Q)
#Override
protected void onCreate(Bundle savedInstanceState) {
super.onCreate(savedInstanceState);
setContentView(R.layout.activity_main);
ByteBuffer byteBuffer = getSharedBufferJNI();
//update the command to shared memory here
//byteBuffer updated with commands
//Call JNI to inform update and get the response
TestBufferCopy();
}
/**
* A native method that is implemented by the 'native-lib' native library,
* which is packaged with this application.
*/
public native ByteBuffer getSharedBufferJNI();
public native int TestBufferCopy();
}
Question:
Accessing primitive arrays from Java to native is reference only if garbage collector supports pinning. Is it true for other way around ?
Is it guaranteed by android platform that ALWAYS reference is shared from NDK to SDK without redundant copy?
Is it the right way to share memory?
You only need /dev/ashmem to share memory between processes. NDK and SDK (Java/Kotlin) work in same Linux process and have full access to same memory space.
The usual way to define memory that can be used both from C++ and Java is by creating a Direct ByteBuffer. You don't need JNI for that, Java API has ByteBuffer.allocateDirect(int capacity). If it's more natural for your logical flow to allocate the buffer on the C++ side, JNI has the NewDirectByteBuffer(JNIEnv* env, void* address, jlong capacity) function that you used in your question.
Working with Direct ByteBuffer is very easy on the C++ side, but not so efficient on the JVM side. The reason is that this buffer is not backed by array, and the only API you have involves ByteBuffer.get() with typed variations (getting byte array, char, int, …). You have control of current position in the buffer, but working this way requires certain discipline: every get() operation updates the current position. Also, random access to this buffer is rather slow, because it involves calling both positioning and get APIs. Therefore, in some cases of non-trivial data structures, it may be easier to write your custom access code in C++ and have 'intelligent' getters called through JNI.
It's important not to forget to set ByteBuffer.order(ByteOrder.nativeOrder()). The order of a newly-created byte buffer is counterintuitively BIG_ENDIAN. This applies both to buffer created from Java and from C++.
If you can isolate the instances when C++ needs access to such shared memory, and don't really need it to be pinned all the time, it's worth to consider working with byte array. In Java, you have more efficient random access. On the NDK side, you will call GetByteArrayElements() or GetPrimitiveArrayCritical(). The latter is more efficient, but its use imposes restrictions on what Java functions you can call until the array is released. On Android, both methods don't involve memory allocation and copy (with no official guarantee, though). Even though C++ side uses the same memory as Java, your JNI code must call the appropriate Release…() function, and better do that as early as possible. It's a good practice to handle this Get/Release via RAII.
Let me summarize my findings,
Accessing primitive arrays from Java to native is reference only if garbage collector supports pinning. Is it true for other way around ?
The contents of a direct buffer can, potentially, reside in native memory outside of the ordinary garbage-collected heap. And hence garbage collector can't claim the memory.
Is it guaranteed by android platform that ALWAYS reference is shared from NDK to SDK without redundant copy?
Yes, As per documentation of NewDirectByteBuffer.
jobject NewDirectByteBuffer(JNIEnv* env, void* address, jlong capacity);
Allocates and returns a direct java.nio.ByteBuffer referring to the block of memory starting at the memory address address and extending capacity bytes.
I'm trying to cause a leak using this native method from my app. I can see "Method returned." in my Logs but I don't seem to lose any RAM. I'm using(MemoryInfo.availMem / 1048576L) for tracking usage.
JNIEXPORT jstring JNICALL Java_com_app_native_Wrapper_causeLeak(JNIEnv *je, jclass jc, jint bytes) {
char *p_array = calloc(bytes,sizeof(char));
return (*je)->NewStringUTF(je, "Method returned.");
}
And trying to cause 10MB leak via this method:
Wrapper.causeLeak(10 * 1024 * 1024)
EDIT:
I'm doing this because I want to test my app in a low memory situation.
I couldn't get this working but I found someone on GitHub who built a better approach. If someone ever need to test memory leaks use the repository here: https://github.com/T-Spoon/Android-Developer-Toolbelt
So I noticed that my app crashes after repeated calls of the following method.
JNIEXPORT void JNICALL Java_com_kitware_VolumeRender_VolumeRenderLib_DummyFunction(JNIEnv * env,jobject obj, jlong udp, jdoubleArray rotation, jdoubleArray translation){
jboolean isCopy1, isCopy2 ;
jdouble* rot = env->GetDoubleArrayElements(rotation,&isCopy1);
jdouble* trans = env->GetDoubleArrayElements(translation,&isCopy2);
if(isCopy1 == JNI_TRUE){
env->ReleaseDoubleArrayElements(rotation,rot, JNI_ABORT);
}
if(isCopy2 == JNI_TRUE){
env->ReleaseDoubleArrayElements(translation,trans, JNI_ABORT);
}
}
I thought this would be due to some missing memory space but I do free the memory here don't I? Still after 512 calls to that method I get my app crashing.
I could provide you with the Logcat if needed but it's a pretty long one. And after investigating a little I'm pretty sure the error is in the memory allocation/free process (i.e commenting out the two GetDoubleArrayElements() get me a running app no matter how many times I call the function).
In android docs: http://developer.android.com/training/articles/perf-jni.html
it is clearly stated:
You must Release every array you Get. Also, if the Get call fails, you must ensure that your code doesn't try to Release a NULL pointer later.
the number 512 is as far as I remember a limit on the number of local references which your code exceeds. So you should remove those checks: if(isCopy2 == JNI_TRUE){.
Still, above docs has a paragraph on JNI_ABORT, which explains it might be used together with isCopy - but its a bit confusing. You might search android sources on how to use JNI_ABORT, ie here is some code:
http://androidxref.com/6.0.0_r1/xref/frameworks/ml/bordeaux/learning/multiclass_pa/jni/jni_multiclass_pa.cpp#77
In my code I often use PushLocalFrame/PopLocalFrame to prevent local references leaks.
I my cpp code contains a jni function that i wish to convert to const char*. This is the code i am using
extern "C" {
void Java_com_sek_test_JNITest_printSomething(JNIEnv * env, jclass cl, jstring str) {
const char* mystring = env->GetStringUTFChars(env, str, 0);
PingoScreen::notify();
}
I get an error that
no matching function for call to '_JNIEnv::GetStringUTFChars(JNIEnv*&, _jstring*&, int)
What am i doing wrong ?
There several things that aren't quite right with your code and approach:
As you've discovered, (env*)->JNIFunc(env,...) should be env->JNIFunc(...) in C++. Your vendor's (Google Android's) jni.h simplifies the C++ syntax over the C syntax.
You're not calling the "Release" function (ReleaseStringUTFChars) corresponding to the "pinning" function (GetStringUTFChars). This is very important because pinned objects reduce the memory efficiency of the JVM's garbage collector.
You've misinterpreted the final argument to GetStringUTFChars. It's a pointer for an output parameter. The result isn't very interesting so pass nullptr.
You're using JNI functions that deal with the modified UTF-8 encoding (GetStringUTFChars et al). There should be no need to ever use that encoding. Java classes are very capable at converting encodings. They also give you control over what happens when a character cannot be encoded in the target encoding. (The default is to convert it to a ?.)
The idea of converting a JVM object reference (jstring) to a pointer to one byte storage (char*) needs a lot of refinement. You probably want to copy the characters in a JVM java.lang.String to a "native" string using a specific or OS-default encoding. The Java string has Unicode characters with a UTF-16 encoding. Android generally uses the Unicode character set with the UTF-8 encoding. If do you need something else, you can specify it with a Charset object.
Also, in C++, it is more convenient to use STL std::string to hold counted byte-sequences for a string. You can get a pointer to a null-terminated buffer from a std::string if you need it.
Be sure to read Android's JNI Tips.
Here is an implementation of your function that lets the vendor's JVM implementation pick the target encoding (which is UTF-8 for Android):
extern "C" JNIEXPORT void Java_com_sek_test_JNITest_printSomething
(JNIEnv * env, jclass cl, jstring str) {
// TODO check for JVM exceptions where appropriate
// javap -s -public java.lang.String | egrep -A 2 "getBytes"
const auto stringClass = env->FindClass("java/lang/String");
const auto getBytes = env->GetMethodID(stringClass, "getBytes", "()[B");
const auto stringJbytes = (jbyteArray) env->CallObjectMethod(str, getBytes);
const auto length = env->GetArrayLength(stringJbytes);
const auto pBytes = env->GetByteArrayElements(stringJbytes, nullptr);
std::string s((char *)pBytes, length);
env->ReleaseByteArrayElements(stringJbytes, pBytes, JNI_ABORT);
const auto pChars = s.c_str(); // if you really do need a pointer
}
However, I'd probably do the call to String.getBytes on the Java side, defining the native method to take a byte array instead of a string.
(Of course, implementations that use GetStringUTFChars do work for some subset of Unicode strings but why impose an esoteric and needless limit?)
According to the documentation,
GetStringUTFChars
const jbyte* GetStringUTFChars(JNIEnv *env, jstring string,
jboolean *isCopy);
Returns a pointer to an array of bytes representing the string in modified UTF-8 encoding. This array is valid until it is released by ReleaseStringUTFChars().
If isCopy is not NULL, then *isCopy is set to JNI_TRUE if a copy is made; or it is set to JNI_FALSE if no copy is made.
So the last parameter should be a jboolean;
Try... Change this line
const char* mystring = env->GetStringUTFChars(env, str, 0);
to
const char *mystring = (*env)->GetStringUTFChars(env, str, 0);
Hope it works:)
I made hello world application from book Android apps for Absolute Beginners and Temperature Convertor app from here
Both is running fine on Emulator but when I try to run it on Samsung Note 2 following error is coming on LogCat
02-08 07:22:18.665: E/dalvikvm(30944): JNI ERROR (app bug): accessed stale local reference 0xbc00021 (index 8 in a table of size 8)
02-08 07:22:18.665: E/dalvikvm(30944): VM aborting
02-08 07:22:18.665: A/libc(30944): Fatal signal 11 (SIGSEGV) at 0xdeadd00d (code=1), thread 30944 (oid.temperature)
Both applications do open shows layout with title but do not shows any other views in layout
Samples runs fine
device: note 2 Samsung-gt_n7100
IDE:Eclipse version 3.8
OS: 64bit Windows 7
Since android 4.0 garbage collector was changed. Now it moves object around during garbage collection, which can cause a lot of problems.
Imagine that you have a static variable pointing to an object, and then this object gets moved by gc. Since android uses direct pointers for java objects, this would mean that your static variable is now pointing to a random address in the memory, unoccupied by any object or occupied by an object of different sort. This will almost guarantee that you'll get EXC_BAD_ACCESS next time you use this variable.
So android gives you JNI ERROR (app bug) error to prevent you from getting undebugable EXC_BAD_ACCESS. Now there are two ways to avoid this error.
You can set targetSdkVersion in your manifest to version 11 or less. This will enable JNI bug compatibility mode and prevent any problems altogether. This is the reason why your old examples are working.
You can avoid using static variables pointing to java objects or make jobject references global before storing them by calling env->NewGlobalRef(ref).
Perhaps on of the biggest examples here is keeping jclass objects. Normally, you'll initialize static jclass variable during JNI_OnLoad, since class objects remain in the memory as long as the application is running.
This code will lead to a crash:
static jclass myClass;
JNIEXPORT jint JNICALL JNI_OnLoad (JavaVM * vm, void * reserved) {
myClass = env->FindClass("com/example/company/MyClass");
return JNI_VERSION_1_6;
}
While this code will run fine:
static jclass myClass;
JNIEXPORT jint JNICALL JNI_OnLoad (JavaVM * vm, void * reserved) {
jclass tmp = env->FindClass("com/example/company/MyClass");
myClass = (jclass)env->NewGlobalRef(tmp);
return JNI_VERSION_1_6;
}
For more examples see link provided by Marek Sebera: http://android-developers.blogspot.cz/2011/11/jni-local-reference-changes-in-ics.html