Develop Reference

Converting bitmap to Opencv::Mat in JNI

I am building a image processing project in Android. I capture bitmap pictures through camera, and feed it to the opencv C++ function through JNI.
Firstly, I test my opencv c++ function using saved bitmap pictures (PNG format), and it it successful.
// in Android, save bitmap
Bitmap bmp = YUV_420_888_toRGB(img,img.getWidth(),img.getHeight());
try {
FileOutputStream fos = new FileOutputStream(pictureFile);
bmp.compress(Bitmap.CompressFormat.PNG, 100, fos);
fos.flush();
fos.close();
Log.e(TAG,"saved successfully.)");
} catch (FileNotFoundException e) {
Log.d(TAG, "File not found: " + e.getMessage());
} catch (IOException e) {
Log.d(TAG, "Error accessing file: " + e.getMessage());
}
// in opencv c++ function
Mat im = imread("/Users/Jun/Downloads/20170227/P9/1488167433596_frame.PNG");
// processing im
Then i feed each captured bitmap picture to the same opencv c++ function. However, the detected result is totally different. I think there must be some errors when converting bitmap in Java to opencv mat in C++ through JNI. Please find the converting codes below:
//Java side:
public static int[] detector(Bitmap bitmap) {
int w = bitmap.getWidth();
int h = bitmap.getHeight();
int []pixels = new int[w*h];
bitmap.getPixels(pixels,0,w,0,0,w,h);
return detect(pixels,w,h);
}
private static native int[] detect(int pixels[],int w,int h);
// c++ side:
JNIEXPORT jintArray JNICALL Java_com_example_jun_helloworld_JNIUtils_detect(JNIEnv *env, jclass cls, jintArray buf, jint w, jint h) {
jint* cbuf = env->GetIntArrayElements(buf, false);
if (cbuf == NULL) {
return NULL;
}
Mat im(h, w, CV_8UC4, (unsigned char *) cbuf);
// processing im
The two "im"s should be different. Can someone tell me what's wrong in the converting? thanks.

In your code, you cast int pointer to char pointer. So, you will change the way your code treats your data.
Take a look here:
#include <stdio.h>
int main() {
// what you have in the code is array of ints
int iarray[5] = {1, 2, 3, 4, 5};
int *iarray_ptr = iarray;
// and you cast int pointer to char pointer
char *carray_ptr = (char *) iarray_ptr;
// so, you simply skip some values because of
// pointer aritmetics; your data are shifted
for(int i=0;i<5;i++) {
printf("int: %p, char %p\n", iarray_ptr + i, carray_ptr + i);
}
// you can always do something like this
char carray2[5];
for(int p=0;p<5;p++) {
// you can loose precision here!
carray2[p] = (char) iarray[p];
}
// and then, you can simply pass &carray2 to
// to your code
}
If you run the code, you can clearly see what will be the difference in pointer arithmetics:
./pointer
int: 0x7fff51d859f0, char 0x7fff51d859f0
int: 0x7fff51d859f4, char 0x7fff51d859f1
int: 0x7fff51d859f8, char 0x7fff51d859f2
int: 0x7fff51d859fc, char 0x7fff51d859f3
int: 0x7fff51d85a00, char 0x7fff51d859f4
After casting to char *, you will simply "scatter" your data.

JNI Android jbytearray to unsigned char* and viceversa

I have 2 separate pthread, and one static struct array. One of the pthread writes the decoding object which include bytes, size, width and height. the other pthread is actually reading the stack and doing some image manipulation and posting to a java function the result.
Here is the problem, on pthread1 I convert the jbytearray to unsigned char*, and store to the position 0 on the static array.
But when it comes pthread2 to convert it back to jbytearray something happens and i always get fatal signal.
This is the top of my cpp class
struct DecodeObject {
unsigned char* data;
int data_size;
int width;
int height;
int orientation;
};
static int decodeLimit = 200 ;
static DecodeObject decodeList[200] ;
static int decodeSize = -1 ;
Here is part of my pthread1
//Values
jbyteArray imageData = (jbyteArray) env->CallObjectMethod(decodeObject,getData);
jint width = (jint) env->CallIntMethod(decodeObject,getWidth);
jint height = (jint) env->CallIntMethod(decodeObject,getHeight);
jint orientation = (jint) env->CallIntMethod(decodeObject,getOrientation);
if(decodeSize<decodeLimit-1){
DecodeObject object;
object.data_size = env->GetArrayLength (imageData);
object.data = as_unsigned_char_array(env,imageData);
object.width = width;
object.height = height;
object.orientation = orientation;
decodeSize++;
decodeList[decodeSize] = object;
}
else {
LOGD("ERROR => BUFFER IS FULL");
}
Here is part of my pthread2
//PREPARE RUNS OK
LOGD("PREPARE"); // RUNS OK
tempObject.data = Prepare(tempObject.data,tempObject.width,tempObject.height);
LOGD("CONVERT BACK TO JBYTEARRAY"); //HERE FAILS
jbyteArray converted = as_byte_array(env,tempObject.data,tempObject.data_size);
LOGD("DONE CONVERTING");
And finally here is the function i am using to convert
unsigned char* as_unsigned_char_array(JNIEnv* &env,jbyteArray array) {
int len = env->GetArrayLength (array);
unsigned char* buf = new unsigned char[len];
env->GetByteArrayRegion (array, 0, len, reinterpret_cast<jbyte*>(buf));
return buf;
}
jbyteArray as_byte_array(JNIEnv* &env,unsigned char* buf, jsize len) {
jbyteArray array = env->NewByteArray(len);
//HERE I GET THE ERROR, I HAVE BEEN TRYING WITH len/2 and WORKS , PROBABLY SOME BYTS ARE GETTING LOST.
env->SetByteArrayRegion (array, 0, len, (jbyte*)(buf));
return array;
}

How to use JNI bitmap operations for helping to avoid OOM when using large images? [closed]

Closed. This question needs details or clarity. It is not currently accepting answers.
Want to improve this question? Add details and clarify the problem by editing this post.
Closed 9 years ago.
Improve this question
Background
most of the times, getting OOM on android is due to using too many bitmaps and/or creating large bitmaps.
recently i've decided to try out JNI in order to allow avoiding OOM by storing the data itself on the JNI side.
after messing around with JNI for a while, i've created some posts on SO asking for help and sharing my knowledge, and i've now decided to share some more code with you. here are the posts in case anyone is interested in reading the findings or contributing :
How to cache bitmaps into native memory
image decoding and manipulation using JNI on android
JNI - how to use multiple Jni wrapper instances with different fields?
Rotating a bitmap using JNI & NDK
this time, i've added the ability to store,restore, crop and rotate bitmaps. it should be easy to add more options and I would be happy if other people here would add their own code to more useful functions .
so the code i'm about to show is actually merging of all the things i've created.
Sample code of usage:
Bitmap bitmap=BitmapFactory.decodeResource(getResources(),R.drawable.ic_launcher);
final int width=bitmap.getWidth(),height=bitmap.getHeight();
// store the bitmap in the JNI "world"
final JniBitmapHolder bitmapHolder=new JniBitmapHolder(bitmap);
// no need for the bitmap on the java "world", since the operations are done on the JNI "world"
bitmap.recycle();
// crop a center square from the bitmap, from (0.25,0.25) to (0.75,0.75) of the bitmap.
bitmapHolder.cropBitmap(width/4,height/4,width*3/4,height*3/4);
//rotate the bitmap:
bitmapHolder.rotateBitmapCcw90();
//get the output java bitmap , and free the one on the JNI "world"
bitmap=bitmapHolder.getBitmapAndFree();
The project is available on github
project page is available on github here .
feel free to give advises and contribute.
Important notes
same notes as shown here, plus:
current features that are written here (more updated on the project page) :
store
restore
rotate 90 degrees CCW
crop.
the approach i've taken for this code is both memory efficiency (use only memory that i need, and free it when not needed), and CPU efficiency (i tried to use pointers and CPU memory cache optimizations whenever possible).
for best performance, i've done really few validations, especially on the JNI part. it might be best to manage the validations on the java "world".
there are still many missing features that i think should be added, and i hope that i will have the time to add them . if anyone wishes to contribute, i will be glad to add they code too. here are the functions that i think could be useful:
get current bitmap info
scale bitmaps, including choice of which algorithm to use (nearest neighbour and bilinear interpolation should be enough).
use different bitmap formats
do the decoding within JNI, to avoid creation of the java bitmap (and not use the heap on the java world) from the beginning, only at the end, when you finished with all of the operations.
face detection
rotation in any angle, or at least the obvious ones . currently i only added rotation of 90 degrees counter clock wise .

NOTE: this is a bit old code. for the most updated one, check out the project page on github.
jni/Android.mk
LOCAL_PATH := $(call my-dir)
#bitmap operations module
include $(CLEAR_VARS)
LOCAL_MODULE := JniBitmapOperations
LOCAL_SRC_FILES := JniBitmapOperations.cpp
LOCAL_LDLIBS := -llog
LOCAL_LDFLAGS += -ljnigraphics
include $(BUILD_SHARED_LIBRARY)
APP_OPTIM := debug
LOCAL_CFLAGS := -g
#if you need to add more module, do the same as the one we started with (the one with the CLEAR_VARS)
jni/JniBitmapOperations.cpp
#include <jni.h>
#include <jni.h>
#include <android/log.h>
#include <stdio.h>
#include <android/bitmap.h>
#include <cstring>
#include <unistd.h>
#define LOG_TAG "DEBUG"
#define LOGD(...) __android_log_print(ANDROID_LOG_DEBUG,LOG_TAG,__VA_ARGS__)
#define LOGE(...) __android_log_print(ANDROID_LOG_ERROR,LOG_TAG,__VA_ARGS__)
extern "C"
{
JNIEXPORT jobject JNICALL Java_com_jni_bitmap_1operations_JniBitmapHolder_jniStoreBitmapData(JNIEnv * env, jobject obj, jobject bitmap);
JNIEXPORT jobject JNICALL Java_com_jni_bitmap_1operations_JniBitmapHolder_jniGetBitmapFromStoredBitmapData(JNIEnv * env, jobject obj, jobject handle);
JNIEXPORT void JNICALL Java_com_jni_bitmap_1operations_JniBitmapHolder_jniFreeBitmapData(JNIEnv * env, jobject obj, jobject handle);
JNIEXPORT void JNICALL Java_com_jni_bitmap_1operations_JniBitmapHolder_jniRotateBitmapCcw90(JNIEnv * env, jobject obj, jobject handle);
JNIEXPORT void JNICALL Java_com_jni_bitmap_1operations_JniBitmapHolder_jniCropBitmap(JNIEnv * env, jobject obj, jobject handle, uint32_t left, uint32_t top, uint32_t right, uint32_t bottom);
}
class JniBitmap
{
public:
uint32_t* _storedBitmapPixels;
AndroidBitmapInfo _bitmapInfo;
JniBitmap()
{
_storedBitmapPixels = NULL;
}
};
/**crops the bitmap within to be smaller. note that no validations are done*/ //
JNIEXPORT void JNICALL Java_com_jni_bitmap_1operations_JniBitmapHolder_jniCropBitmap(JNIEnv * env, jobject obj, jobject handle, uint32_t left, uint32_t top, uint32_t right, uint32_t bottom)
{
JniBitmap* jniBitmap = (JniBitmap*) env->GetDirectBufferAddress(handle);
if (jniBitmap->_storedBitmapPixels == NULL)
return;
uint32_t* previousData = jniBitmap->_storedBitmapPixels;
uint32_t oldWidth = jniBitmap->_bitmapInfo.width;
uint32_t newWidth = right - left, newHeight = bottom - top;
uint32_t* newBitmapPixels = new uint32_t[newWidth * newHeight];
uint32_t* whereToGet = previousData + left + top * oldWidth;
uint32_t* whereToPut = newBitmapPixels;
for (int y = top; y < bottom; ++y)
{
memcpy(whereToPut, whereToGet, sizeof(uint32_t) * newWidth);
whereToGet += oldWidth;
whereToPut += newWidth;
}
//done copying , so replace old data with new one
delete[] previousData;
jniBitmap->_storedBitmapPixels = newBitmapPixels;
jniBitmap->_bitmapInfo.width = newWidth;
jniBitmap->_bitmapInfo.height = newHeight;
}
/**rotates the inner bitmap data by 90 degress counter clock wise*/ //
JNIEXPORT void JNICALL Java_com_jni_bitmap_1operations_JniBitmapHolder_jniRotateBitmapCcw90(JNIEnv * env, jobject obj, jobject handle)
{
JniBitmap* jniBitmap = (JniBitmap*) env->GetDirectBufferAddress(handle);
if (jniBitmap->_storedBitmapPixels == NULL)
return;
uint32_t* previousData = jniBitmap->_storedBitmapPixels;
AndroidBitmapInfo bitmapInfo = jniBitmap->_bitmapInfo;
uint32_t* newBitmapPixels = new uint32_t[bitmapInfo.height * bitmapInfo.width];
int whereToPut = 0;
// A.D D.C
// ...>...
// B.C A.B
for (int x = bitmapInfo.width - 1; x >= 0; --x)
for (int y = 0; y < bitmapInfo.height; ++y)
{
uint32_t pixel = previousData[bitmapInfo.width * y + x];
newBitmapPixels[whereToPut++] = pixel;
}
delete[] previousData;
jniBitmap->_storedBitmapPixels = newBitmapPixels;
uint32_t temp = bitmapInfo.width;
bitmapInfo.width = bitmapInfo.height;
bitmapInfo.height = temp;
}
/**free bitmap*/ //
JNIEXPORT void JNICALL Java_com_jni_bitmap_1operations_JniBitmapHolder_jniFreeBitmapData(JNIEnv * env, jobject obj, jobject handle)
{
JniBitmap* jniBitmap = (JniBitmap*) env->GetDirectBufferAddress(handle);
if (jniBitmap->_storedBitmapPixels == NULL)
return;
delete[] jniBitmap->_storedBitmapPixels;
jniBitmap->_storedBitmapPixels = NULL;
delete jniBitmap;
}
/**restore java bitmap (from JNI data)*/ //
JNIEXPORT jobject JNICALL Java_com_jni_bitmap_1operations_JniBitmapHolder_jniGetBitmapFromStoredBitmapData(JNIEnv * env, jobject obj, jobject handle)
{
JniBitmap* jniBitmap = (JniBitmap*) env->GetDirectBufferAddress(handle);
if (jniBitmap->_storedBitmapPixels == NULL)
{
LOGD("no bitmap data was stored. returning null...");
return NULL;
}
//
//creating a new bitmap to put the pixels into it - using Bitmap Bitmap.createBitmap (int width, int height, Bitmap.Config config) :
//
//LOGD("creating new bitmap...");
jclass bitmapCls = env->FindClass("android/graphics/Bitmap");
jmethodID createBitmapFunction = env->GetStaticMethodID(bitmapCls, "createBitmap", "(IILandroid/graphics/Bitmap$Config;)Landroid/graphics/Bitmap;");
jstring configName = env->NewStringUTF("ARGB_8888");
jclass bitmapConfigClass = env->FindClass("android/graphics/Bitmap$Config");
jmethodID valueOfBitmapConfigFunction = env->GetStaticMethodID(bitmapConfigClass, "valueOf", "(Ljava/lang/String;)Landroid/graphics/Bitmap$Config;");
jobject bitmapConfig = env->CallStaticObjectMethod(bitmapConfigClass, valueOfBitmapConfigFunction, configName);
jobject newBitmap = env->CallStaticObjectMethod(bitmapCls, createBitmapFunction, jniBitmap->_bitmapInfo.width, jniBitmap->_bitmapInfo.height, bitmapConfig);
//
// putting the pixels into the new bitmap:
//
int ret;
void* bitmapPixels;
if ((ret = AndroidBitmap_lockPixels(env, newBitmap, &bitmapPixels)) < 0)
{
LOGE("AndroidBitmap_lockPixels() failed ! error=%d", ret);
return NULL;
}
uint32_t* newBitmapPixels = (uint32_t*) bitmapPixels;
int pixelsCount = jniBitmap->_bitmapInfo.height * jniBitmap->_bitmapInfo.width;
memcpy(newBitmapPixels, jniBitmap->_storedBitmapPixels, sizeof(uint32_t) * pixelsCount);
AndroidBitmap_unlockPixels(env, newBitmap);
//LOGD("returning the new bitmap");
return newBitmap;
}
/**store java bitmap as JNI data*/ //
JNIEXPORT jobject JNICALL Java_com_jni_bitmap_1operations_JniBitmapHolder_jniStoreBitmapData(JNIEnv * env, jobject obj, jobject bitmap)
{
AndroidBitmapInfo bitmapInfo;
uint32_t* storedBitmapPixels = NULL;
//LOGD("reading bitmap info...");
int ret;
if ((ret = AndroidBitmap_getInfo(env, bitmap, &bitmapInfo)) < 0)
{
LOGE("AndroidBitmap_getInfo() failed ! error=%d", ret);
return NULL;
}
LOGD("width:%d height:%d stride:%d", bitmapInfo.width, bitmapInfo.height, bitmapInfo.stride);
if (bitmapInfo.format != ANDROID_BITMAP_FORMAT_RGBA_8888)
{
LOGE("Bitmap format is not RGBA_8888!");
return NULL;
}
//
//read pixels of bitmap into native memory :
//
//LOGD("reading bitmap pixels...");
void* bitmapPixels;
if ((ret = AndroidBitmap_lockPixels(env, bitmap, &bitmapPixels)) < 0)
{
LOGE("AndroidBitmap_lockPixels() failed ! error=%d", ret);
return NULL;
}
uint32_t* src = (uint32_t*) bitmapPixels;
storedBitmapPixels = new uint32_t[bitmapInfo.height * bitmapInfo.width];
int pixelsCount = bitmapInfo.height * bitmapInfo.width;
memcpy(storedBitmapPixels, src, sizeof(uint32_t) * pixelsCount);
AndroidBitmap_unlockPixels(env, bitmap);
JniBitmap *jniBitmap = new JniBitmap();
jniBitmap->_bitmapInfo = bitmapInfo;
jniBitmap->_storedBitmapPixels = storedBitmapPixels;
return env->NewDirectByteBuffer(jniBitmap, 0);
}
src/com/jni/bitmap_operations/JniBitmapHolder.java
package com.jni.bitmap_operations;
import java.nio.ByteBuffer;
import android.graphics.Bitmap;
import android.util.Log;
public class JniBitmapHolder
{
ByteBuffer _handler =null;
static
{
System.loadLibrary("JniBitmapOperations");
}
private native ByteBuffer jniStoreBitmapData(Bitmap bitmap);
private native Bitmap jniGetBitmapFromStoredBitmapData(ByteBuffer handler);
private native void jniFreeBitmapData(ByteBuffer handler);
private native void jniRotateBitmapCcw90(ByteBuffer handler);
private native void jniCropBitmap(ByteBuffer handler,final int left,final int top,final int right,final int bottom);
public JniBitmapHolder()
{}
public JniBitmapHolder(final Bitmap bitmap)
{
storeBitmap(bitmap);
}
public void storeBitmap(final Bitmap bitmap)
{
if(_handler!=null)
freeBitmap();
_handler=jniStoreBitmapData(bitmap);
}
public void rotateBitmapCcw90()
{
if(_handler==null)
return;
jniRotateBitmapCcw90(_handler);
}
public void cropBitmap(final int left,final int top,final int right,final int bottom)
{
if(_handler==null)
return;
jniCropBitmap(_handler,left,top,right,bottom);
}
public Bitmap getBitmap()
{
if(_handler==null)
return null;
return jniGetBitmapFromStoredBitmapData(_handler);
}
public Bitmap getBitmapAndFree()
{
final Bitmap bitmap=getBitmap();
freeBitmap();
return bitmap;
}
public void freeBitmap()
{
if(_handler==null)
return;
jniFreeBitmapData(_handler);
_handler=null;
}
#Override
protected void finalize() throws Throwable
{
super.finalize();
if(_handler==null)
return;
Log.w("DEBUG","JNI bitmap wasn't freed nicely.please rememeber to free the bitmap as soon as you can");
freeBitmap();
}
}

convert createBitmap from Java to JNI

I want to use this code in JNI, without going back to Java.
I already converted bitmap manipulation to JNI (thanks to other stackoverflow posters), but this seems more complicated because I do not understand how to call constructors.
Bitmap bmp;
bmp = ((BitmapDrawable)imgview.getDrawable()).getBitmap();
if (bmp == null || !bmp.isMutable())
Bitmap bmp = Bitmap.createBitmap(w, h, Config.ARGB_8888);
// bitmap manipulations goes here
jclass java_bitmap_class = (env)->GetObjectClass(java_bitmap);
class SkBitmap;
SkBitmap *sk_bitmap = (SkBitmap*)(env)->CallIntMethod(
java_bitmap, (env)->GetMethodID(java_bitmap_class, "ni", "()I"));
// there is more c++ code to manipulate bmp, but it is not relevant to a question
imgview.setImageBitmap(bmp);

Ok, it is actually very simple once you master java->jni translation. Basically you can do anything in JNI side what you can do in Java. Yes, it looks messy. I decided not to create the bitmap in JNI but to access the existing one though.
JNIEnv* Env = 0; jobject Obj;
jclass cls = 0, ClassImageView = 0, class_drawable = 0, java_bitmap_class = 0;
jmethodID jcontrol_ui = 0, jfindViewById = 0, jgetBitmap = 0, jgetDrawable = 0;
int *getViewBitmapBuffer(int ID) {
jobject image_view = (jobject) (Env)->CallObjectMethod(Obj, jfindViewById, ID);
// some values can be cached, hence the checks for "(something == 0)"
if (ClassImageView == 0) ClassImageView = (Env)->GetObjectClass(image_view);
if (jgetDrawable == 0) jgetDrawable = (Env)->GetMethodID(ClassImageView, "getDrawable", sig_drawable);
jobject drawable = (jobject) (Env)->CallObjectMethod(image_view, jgetDrawable);
if (class_drawable == 0) class_drawable = (Env)->GetObjectClass(drawable);
if (jgetBitmap == 0) jgetBitmap = (Env)->GetMethodID(class_drawable, "getBitmap", sig_bitmap);
jobject java_bitmap = (jobject) (Env)->CallObjectMethod(drawable, jgetBitmap);
if (java_bitmap_class == 0) java_bitmap_class = (Env)->GetObjectClass(java_bitmap);
class SkBitmap;
SkBitmap *sk_bitmap = (SkBitmap*)(Env)->CallIntMethod(java_bitmap, (Env)->GetMethodID(java_bitmap_class, "ni", "()I"));
SkPixelRef *sk_pix_ref;
sk_pix_ref = (SkPixelRef*)((int*)sk_bitmap)[1];
int *B = (int*) sk_pix_ref->GetPixels();
return B;
}

Android ndk out of memory issue

i am trying to do real time image processing in android using jni. I have a native method to decode image data and i call this method for every frame. After a few seconds later i get out of memory and my app terminats.
LOG OUTPUT:
12-03 20:54:19.780: E/dalvikvm-heap(8119): Out of memory on a 3686416-byte allocation.
MY NATIVE METHOD:
JNIEXPORT jintArray JNICALL Java_net_oyunyazar_arcc_data_FrameManager_processImage(JNIEnv* env, jobject javaThis, jint width, jint height, jbyteArray arr) {
jint *convertedData;
convertedData = (jint*)malloc((width*height) * sizeof(jint));
jintArray result = (*env)->NewIntArray(env, width*height);
jint y,x;
jbyte grey;
jsize len = (*env)->GetArrayLength(env, arr);
jbyte *YUVData = (*env)->GetByteArrayElements(env, arr, 0);
for (y = 0; y < height; y++){
for (x = 0; x < width; x++){
grey = YUVData[y * width + x];
convertedData[y*width+x] =(jint) grey & 0xff;
}
}
LOGD("Random [%d]",len);
(*env)->SetIntArrayRegion(env, result, 0, (width*height),convertedData );
free(convertedData);
(*env)->ReleaseByteArrayElements(env, YUVData, (jbyte*)arr, 0);
return result;
}
Thanks for any help.

I have the same problem as yours.
In your specific case, while you are using pixel (and probably bitmap) you can send a bitmap instead of your bytearray and modify it
void *pixel_bm;
int retValue;
AndroidBitmapInfo info;
if ((retValue = AndroidBitmap_getInfo(env, bitmap, &info)) < 0) return 0;
if ((retValue = AndroidBitmap_lockPixels(env, bitmap, &pixel_bm)) < 0) return 0;
// you can now read an write into pixel_bm
AndroidBitmap_unlockPixels(env, bitmap);
If you find a solution to correctly free a GetByteArrayElement result, I'm instrested by the solution !!!

I have solved this problem by releasing the parameters.
(*env)->ReleaseByteArrayElements(env, arr, YUVData, 0);
It works great now.