I'm trying to inference my tflite model on c++ code at an embedded device.
So, I type simple tflite inference code which is using GPU.
And I cross-compile in my PC and run at the embedded device which is running android.
However, (1) If I use delegate gpu option, c++ codes give random results.
(2) It has given same input, but the results changed every time.
(3) When I turn off the gpu option, it gives me a correct results.
When I test my tflite model in python, it gives the correct output.
So I think the model file has no problem.
Should I re-build TensorFlow-lite because I use a prebuilt .so file? or Should I change the GPU option?
I don't know what should I check more.
Please Help!
Here is my C++ code
// Load model
std::unique_ptr<tflite::FlatBufferModel> model =
tflite::FlatBufferModel::BuildFromFile(model_file.c_str());
// Build the interpreter
tflite::ops::builtin::BuiltinOpResolver resolver;
std::unique_ptr<tflite::Interpreter> interpreter;
tflite::InterpreterBuilder(*model, resolver)(&interpreter);
// set delegate option
bool use_gpu = true;
if(use_gpu)
{
TfLiteDelegate* delegate;
auto options = TfLiteGpuDelegateOptionsV2Default();
options.inference_preference = TFLITE_GPU_INFERENCE_PREFERENCE_FAST_SINGLE_ANSWER;
options.inference_priority1 = TFLITE_GPU_INFERENCE_PRIORITY_AUTO;
delegate = TfLiteGpuDelegateV2Create(&options);
interpreter->ModifyGraphWithDelegate(delegate);
}
interpreter->AllocateTensors();
// set input
float* input = interpreter->typed_input_tensor<float>(0);
for(int i=0; i<width*height*channel; i++)
*(input+i) = 1;
TfLiteTensor* output_tensor = nullptr;
// Inference
interpreter->Invoke();
// Check output
output_tensor = interpreter->tensor(interpreter->outputs()[0]);
printf("Result : %f\n",output_tensor->data.f[0]);
//float* output = interpreter->typed_output_tensor<float>(0);
//printf("output : %f\n",*(output));
Luckily, I found the answer.
This was caused by the mismatch of the NDK version. (The prebuilt so file I used is built another version)
After unifying the NDK version to 21, it was tested again, and it worked normally.
In my case same application using gpu on S10 lite was not working but on S22,S21 Ultra 5G was working.
So you can consider running on different chipset configuration on different devices with same application.
Related
I am attempting to use a new NLP model within the PyTorch android demo app Demo App Git however I am struggling to serialize the model so that it works with Android.
The demonstration given by PyTorch is as follows for a Resnet model:
model = torchvision.models.resnet18(pretrained=True)
model.eval()
example = torch.rand(1, 3, 224, 224)
traced_script_module = torch.jit.trace(model, example)
traced_script_module.save("app/src/main/assets/model.pt")
However I am not sure what to use for the 'example' input with my NLP model.
The model that I am using from a fastai tutorial and the python is linked here: model
Here is the Python used to create my model (using the Fastai library). It is the same as in the model link above, but in a simplified form.
from fastai.text import *
path = untar_data('http://files.fast.ai/data/examples/imdb_sample')
path.ls()
#: [PosixPath('/storage/imdb_sample/texts.csv')]
data_lm = TextDataBunch.from_csv(path, 'texts.csv')
data = (TextList.from_csv(path, 'texts.csv', cols='text')
.split_from_df(col=2)
.label_from_df(cols=0)
.databunch())
bs=48
path = untar_data('https://s3.amazonaws.com/fast-ai-nlp/imdb')
data_lm = (TextList.from_folder(path)
.filter_by_folder(include=['train', 'test', 'unsup'])
.split_by_rand_pct(0.1)
.label_for_lm()
.databunch(bs=bs))
learn = language_model_learner(data_lm, AWD_LSTM, drop_mult=0.3)
learn.fit_one_cycle(1, 1e-2, moms=(0.8,0.7))
learn.unfreeze()
learn.fit_one_cycle(10, 1e-3, moms=(0.8,0.7))
learn.save_encoder('fine_tuned_enc')
path = untar_data('https://s3.amazonaws.com/fast-ai-nlp/imdb')
data_clas = (TextList.from_folder(path, vocab=data_lm.vocab)
.split_by_folder(valid='test')
.label_from_folder(classes=['neg', 'pos'])
.databunch(bs=bs))
learn = text_classifier_learner(data_clas, AWD_LSTM, drop_mult=0.5)
learn.load_encoder('fine_tuned_enc')
learn.fit_one_cycle(1, 2e-2, moms=(0.8,0.7))
learn.freeze_to(-2)
learn.fit_one_cycle(1, slice(1e-2/(2.6**4),1e-2), moms=(0.8,0.7))
learn.freeze_to(-3)
learn.fit_one_cycle(1, slice(5e-3/(2.6**4),5e-3), moms=(0.8,0.7))
learn.unfreeze()
learn.fit_one_cycle(2, slice(1e-3/(2.6**4),1e-3), moms=(0.8,0.7))
I worked out how to do this after a while. The issue was that the Fastai model wasn't tracing correctly no matter what shape of input I was using.
In the end, I used another text classification model and got it to work. I wrote a tutorial about how I did it, in case it can help anyone else.
NLP PyTorch Tracing Tutorial
Begin by opening a new Jupyter Python Notebook using your preferred cloud machine provider (I use Paperspace).
Next, copy and run the code in the PyTorch Text Classification tutorial. But replace the line…
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
With…
device = torch.device("cpu")
NOTE: It caused issues tracing when the device was set to CUDA so I forced it on to the CPU. (this will slow training, but inference on the mobile will run at the same speed as it is cpu anyway)
Lastly, run the code below to correctly trace the model to allow it to be run on Android:
data = DataLoader(test_dataset, batch_size=1, collate_fn=generate_batch)
for text, offsets, cls in data:
text, offsets, cls = text.to(device), offsets.to(device), cls.to(device)
example = text, offsets
traced_script_module = torch.jit.trace(model, example)
traced_script_module.save("model.pt")
In addition, if you would like a CSV copy of the vocab list for use on Android when you are making predictions, run the following code afterwards:
import pandas as pd
vocab = train_dataset.get_vocab()
df = pd.DataFrame.from_dict(vocab.stoi, orient='index', columns=['token'])
df[:30]
df.to_csv('out.csv')
This model should work fine on Android using the PyTorch API.
I have been trying a ( i hope) simple bit of Android hyperloop code directly within a titanium project (using SDK 7.0.1.GA and hyperloop 3).
var sysProp = require('android.os.SystemProperties');
var serialNumber = sysProp.get("sys.serialnumber", "none");
But when the app is run it reports
Requested module not found:android.os.SystemProperties
I think this maybe due to the fact that when compiling the app (using the cli) it reports
hyperloop:generateSources: Skipping Hyperloop wrapper generation, no usage found ...
I have similar code in a jar and if I use this then it does work, so I am wondering why the hyperloop generation is not being triggered, as I assume that is the issue.
Sorry should have explained better.
This is the jar source that I use, the extraction of the serial number was just an example (I need access to other info manufacturer specific data as well), I wanted to see if I could replicate the JAR functionality using just hyperloop rather that including the JAR file. Guess if it's not broke don't fix it, but was curious to see if it could be done.
So with the feedback from #miga and a bit of trial and error, I have come up with a solution that works really well and will do the method reflection that is required. My new Hyperloop function is
function getData(data){
var result = false;
var Class = require("java.lang.Class");
var String = require("java.lang.String");
var c = Class.forName("android.os.SystemProperties");
var get = c.getMethod("get", String.class, String.class);
result = get.invoke(c, data, "Error");
return result;
}
Where data is a string of the system property I want.
I am using it to extract and match a serial number from a Samsung device that is a System Property call "ril.serialnumber" or "sys.serialnumber". Now I can use the above function to do what I was using the JAR file for. Just thought I'd share in case anyone else needed something similar.
It is because android.os.SystemProperties is not class you can import. Check the android documentation at https://developer.android.com/reference/android/os/package-summary.html
You could use
var build = require('android.os.Build');
console.log(build.SERIAL);
to access the serial number.
I'm studying the NFC code of Google Pixel AOSP build (android-8.0.0_r3). I can understand the call flow from kernel all the way to Android hardware module system\nfc\halimpl\pn54x\nfc_nci.c as following:
struct nfc_nci_module_t HAL_MODULE_INFO_SYM = {
.common =
{
.tag = HARDWARE_MODULE_TAG,
.module_api_version = 0x0100, /* [15:8] major, [7:0] minor (1.0) */
.hal_api_version = 0x00, /* 0 is only valid value */
.id = NFC_NCI_HARDWARE_MODULE_ID,
.name = "NXP PN54X NFC NCI HW HAL",
.author = "NXP Semiconductors",
.methods = &nfc_module_methods,
},
};
This hardware module is loaded via hw_get_module(NFC_NCI_HARDWARE_MODULE_ID) found it in hardware/interface/nfc/1.0/default/nfc.cpp : HIDL_FETCH_INfc(...). Now I'm stuck here unable to find where in the AOSP invoked HIDL_FETCH_INfc(...)and how does it link to the next layer?
For android services that interact with hardware it connects in the following way.
In the nfc.cpp there should be a functions that are similar to:
JNINativeMethod method_table[]= {
{"external_name","(I)Z)",(void*)internal_name},
};
int register_android_server_FlexcanService(JNIEnv* env)
{
return jniRegisterNativeMethods(env,"com/android/server/MyService",method_table,NELEM(method_table));
}
This jniRegisterNativeMethods is called in onload.cpp and nfc.cpp is added to the Android.mk in the same folder
Then in the java package com.android.server.MyService (core/java/com/android/server) there will be a call to the native function:
public static native boolean external_name(int someVar);
From there there will be an .aidl binder file to expose the service methods to the SDK and eventually into top level Android apps.
In Android 7.x, NFC JNI locates at packages/apps/Nfc/nci/jni/. I haven't checked Android 8.x source, but I think it's still there.
I'm having a confusing problem. I'm trying to make a Web cleint that uses WSDL.
I'm using C++ RAD Studio 10 Seattle, but the same problem occured in RAD Studio XE8(older version).
1.I create a Multi-Device Application, add one Edit component and one Button.
2.I create a WSDL Importer by changing the location of the WSDL file to : "http://www.w3schools.com/webservices/tempconvert.asmx?WSDL" and leave all other setting to default.
3.On ButtonClick event of the button I write two lines of code :
_di_TempConvertSoap Converter = GetTempConvertSoap(true,
"http://www.w3schools.com/webservices/tempconvert.asmx?WSDL");
Edit1->Text = Converter->CelsiusToFahrenheit("32");
So after these three steps I have one unit, which is the main Unit with the Form and with the button event. And one file "tempconvert.cpp" that the WSDL Importer has generated. It quite actually just translates the WSDL code to a C++ one and defines the method to communicate with the server. In my case I have two methods : FahrenheitToCelsius() and CelsiusToFahrenheit(), in the example I use CelsiusToFahrenheit().
I compile it to 32-bit Windows platform, run it and when I click the button, the result "89.6" appears in the text of the Edit component. So this is working as expected.
But when I change the target platform to "Android" and use my mobile phone "Samsung GT-I8262" with Android 4.1.2 and run the project, it just stops and exits. I debugged the problem and it stops at the first command in "tempconvert.cpp" in RegTypes() method.
// ************************************************************************
//
// This routine registers the interfaces and types exposed by the WebService.
// ************************************************************************ //
static void RegTypes()
{
/* TempConvertSoap */
InvRegistry()->RegisterInterface(__delphirtti(TempConvertSoap), L"http://www.w3schools.com/webservices/", L"utf-8");
InvRegistry()->RegisterDefaultSOAPAction(__delphirtti(TempConvertSoap), L"http://www.w3schools.com/webservices/%operationName%");
InvRegistry()->RegisterInvokeOptions(__delphirtti(TempConvertSoap), ioDocument);
/* TempConvertSoap.FahrenheitToCelsius */
InvRegistry()->RegisterMethodInfo(__delphirtti(TempConvertSoap), "FahrenheitToCelsius", "",
"[ReturnName='FahrenheitToCelsiusResult']", IS_OPTN);
/* TempConvertSoap.CelsiusToFahrenheit */
InvRegistry()->RegisterMethodInfo(__delphirtti(TempConvertSoap), "CelsiusToFahrenheit", "",
"[ReturnName='CelsiusToFahrenheitResult']", IS_OPTN);
/* TempConvertHttpPost */
InvRegistry()->RegisterInterface(__delphirtti(TempConvertHttpPost), L"http://www.w3schools.com/webservices/", L"utf-8");
InvRegistry()->RegisterDefaultSOAPAction(__delphirtti(TempConvertHttpPost), L"");
}
#pragma startup RegTypes 32
Does someone have any idea why this might be happening? I tried on two other Samsung phones and it didn't work. The error that shuts the program down is "Segmentation fault(11)", and more precisely it stops at the following line of code in "System.pas" file :
u_strFromUTF8(PUChar(Dest), MaxDestChars, DestLen, MarshaledAString(Source), SourceBytes, ErrorConv);
Here is some info that I've found about the function:
u_strFromUTF8 - function that converts a UTF-8 string to UTF-16.
UCHAR is a Byte(in Delphi), so PUCHAR is a pointer to Byte.
I cannot se what could possibly go wrong with this function which apparently only converts a string.
So my question is why does the project work on Windows 32 bit version, but on Android it throws Segmentation fault(11)?
I hope I could find a solution for this problem. I will keep looking.
Thank you,
Zdravko Donev :)
UPDATE:
I disassembled the line:
InvRegistry()->RegisterInterface(__delphirtti(TempConvertSoap), L"http://www.w3schools.com/webservices/", L"utf-16");
to get :
TInvokableClassRegistry *Class = InvRegistry();
TTypeInfo *Info = __delphirtti(TempConvertSoap);
UnicodeString Namespace = "http://www.w3schools.com/webservices/";
UnicodeString WSDLEncoding = "utf-8";
Class->RegisterInterface(Info, Namespace, WSDLEncoding);
And I saw that the problem occurs when calling InvRegistry() function, but I still haven't found the problem as I cannot reach the source code of the function.
I found a solution.
I deleted the line
#pragma startup RegTypes 32
and called the method RegTypes() on my own when I create the form and it worked.
I am referring "Android crypto implementation" document. w.r.t Android 4.3 source.
In point 2 it says The framework starts up, and sees that vold.decrypt is set to "1"
But I am not able to locate the code that does things mentioned in point 2. Is anybody able to locate it? Is it somewhere related to class BackupRestoreConfirmation?
Just a simple git grep will get the all the reference to this property:
cmds/bootanimation/BootAnimation.cpp:276: property_get("vold.decrypt", decrypt, "");
services/java/com/android/server/DevicePolicyManagerService.java:1869: String state = SystemProperties.get("vold.decrypt");
services/java/com/android/server/SystemServer.java:239: String cryptState = SystemProperties.get("vold.decrypt");
services/java/com/android/server/usb/UsbDeviceManager.java:173: boolean dataEncrypted = "1".equals(SystemProperties.get("vold.decrypt"));
And I think the one you need is in DevicePolicyManagerService.java or SystemServer.java