Develop Reference

How to add a delay inside the Executor bound to CameraX's analyzer?

Inside my overridden analyze() I need to add some kind of throttle before performing an IO operation. Without the throttle, this operation gets executed immediately at each call of analyze() and it actually completes quickly, but apparently the calls are too fast and after a while the camera preview freezes for eternity (the app is still running because Logcat keeps displaying new messages).
I'm currently investigating if it has something to do with my code, like forgetting to call imageProxy.close(). So far everything seems fine and I'm afraid the device that performs the IO operation is raising too many interrupts for the CPU to handle, or something along the lines.
I've tried the good old Thread.sleep() but obviously it blocks the main thread and freezes the UI; I've seen some examples with Handler#postDelayed() but I don't think it does what I want; I've tried wrapping the IO call in a coroutine with a delay() at the beginning but again I don't think it does what I want. Basically I'd like to call some form of sleep() on the Executor thread itself, from within the code executed by it.

after a while the camera preview freezes for eternity
I've seen this issue occur many times, and it's usually due to an image that the Analyzer doesn't close. Are you seeing the issue even when the image analysis use case isn't used?
I've tried the good old Thread.sleep() but obviously it blocks the main thread and freezes the UI
Why's that? This shouldn't be the case if you're adding the call to Thread.sleep() inside Analyzer.analyze(), since it'll block the thread of the Executor you provided when calling ImageAnalysis.setAnalyzer(), which shouldn't be tied to the main thread.
One option to perform analysis fewer times is to drop images inside the Analyzer, something like the following:
private static final int ANALYSIS_DELAY_MS = 1_000;
private static final int INVALID_TIME = -1;
private long lastAnalysisTime = INVALID_TIME;
public void analyze (ImageProxy image) {
final long now = SystemClock.uptimeMillis();
// Drop frame if an image has been analyzed less than ANALYSIS_DELAY_MS ms ago
if (lastAnalysisTime != INVALID_TIME && (now - lastAnalysisTime < ANALYSIS_DELAY_MS)) {
image.close();
}
lastAnalysisTime = now;
// Analyze image
image.close();
}

There is one more way, This is how I am doing it in a local project. So in my Image Analyzer, I am rebinding the camera using Handler#PostDelayed and this gives a small black screen in preview, but it won't process the next image that quickly.
My use case was that I had to continuously scan barcodes, but the scanning was too fast and it kept on scanning one code many times. So, I just needed 100ms wait. So this works out for me.
try {
cameraProvider.unbindAll()
Handler(Looper.getMainLooper()).postDelayed({
cameraProvider.bindToLifecycle(
this, CameraSelector.DEFAULT_BACK_CAMERA, useCaseGroup.build())
}, 100)
}
catch(exc: Exception) {
Log.e(TAG, "Use case binding failed", exc)
}

Android NDK Sensor strange report interval to event queue

I try to access the accelerometer from the NDK. So far it works. But the way events are written to the eventqueue seems a little bit strange.
See the following code:
ASensorManager* AcquireASensorManagerInstance(void) {
typedef ASensorManager *(*PF_GETINSTANCEFORPACKAGE)(const char *name);
void* androidHandle = dlopen("libandroid.so", RTLD_NOW);
PF_GETINSTANCEFORPACKAGE getInstanceForPackageFunc = (PF_GETINSTANCEFORPACKAGE) dlsym(androidHandle, "ASensorManager_getInstanceForPackage");
if (getInstanceForPackageFunc) {
return getInstanceForPackageFunc(kPackageName);
}
typedef ASensorManager *(*PF_GETINSTANCE)();
PF_GETINSTANCE getInstanceFunc = (PF_GETINSTANCE) dlsym(androidHandle, "ASensorManager_getInstance");
return getInstanceFunc();
}
void init() {
sensorManager = AcquireASensorManagerInstance();
accelerometer = ASensorManager_getDefaultSensor(sensorManager, ASENSOR_TYPE_ACCELEROMETER);
looper = ALooper_prepare(ALOOPER_PREPARE_ALLOW_NON_CALLBACKS);
accelerometerEventQueue = ASensorManager_createEventQueue(sensorManager, looper, LOOPER_ID_USER, NULL, NULL);
auto status = ASensorEventQueue_enableSensor(accelerometerEventQueue,
accelerometer);
status = ASensorEventQueue_setEventRate(accelerometerEventQueue,
accelerometer,
SENSOR_REFRESH_PERIOD_US);
}
That's how I initialize everything. My SENSOR_REFRESH_PERIOD_US is 100.000 - so 10 refreshs per second. Now I have the following method to receive the events of the event queue.
vector<sensorEvent> update() {
ALooper_pollAll(0, NULL, NULL, NULL);
vector<sensorEvent> listEvents;
ASensorEvent event;
while (ASensorEventQueue_getEvents(accelerometerEventQueue, &event, 1) > 0) {
listEvents.push_back(sensorEvent{event.acceleration.x, event.acceleration.y, event.acceleration.z, (long long) event.timestamp});
}
return listEvents;
}
sensorEvent at this point is a custom struct which I use. This update method gets called via JNI from Android every 10 seconds from an IntentService (to make sure it runs even when the app itself is killed). Now I would expect to receive 100 values (10 per second * 10 seconds). In different tests I received around 130 which is also completly fine for me even it's a bit off. Then I read in the documentation of ASensorEventQueue_setEventRate that it's not forced to follow the given refresh period. So if I would get more than I wanted it would be totally fine.
But now the problem: Sometimes I receive like 13 values in 10 seconds and when I continue to call update 10 secods later I get the 130 values + the missing 117 of the run before. This happens completly random and sometimes it's not the next run but the fourth following or something like that.
I am completly fine with being off from the refresh period by having more values. But can anyone explain why it happens that there are so many values missing and they appear 10 seconds later in the next run? Or is there maybe a way to make sure I receive them in their desired run?

Your code is correct and as i see only one reason can be cause such behaviour. It is android system, for avoid drain battery, decreases frequency of accelerometer stream of events in some time after app go to background or device fall asleep.
You need to revise all axelerometer related logic and optimize according
Doze and App Standby
Also you can try to work with axelerometer in foreground service.

cannot trace frames rendered from another thread

Our company develops several games for mobile platforms, including Android. We use OpenGL for all visual items, including UI (more technical details are below).
We have received some weird warnings from Google Play Console in Pre-launch report, like “Your app took 20764 ms to launch”. On the video provided with this report, the game took about a second to start.
After some investigation we found that Android Systrace cannot detect OpenGL draws made from another thread. So Pre-launch tests think (wrongly) that our game is super-slow.
Is there some method to notify the system that a frame is drawn? It’s seems that eglSwapBuffers() is not enough.
There’s a link to the same problem with Cocos2d: https://discuss.cocos2d-x.org/t/frozen-frames-warnings-by-google-play-pre-launch-report-for-3-17-cocos-demo-app/42894
Some details
When a new build is published to Google Play Console, some automated tests are performed on different devices. Results of these tests are available in Pre-launch report section of the Google Play Console.
Starting from beginning of April we receive strange performance warnings on some of devices (always the same ones). Two examples:
Startup time: Your app took 20764 ms to launch…
Frozen frames: 33.33% of the frames took longer than 700ms to render
Both problems sound dreadful--would have they be true. But when we examined videos of testing, we could not see any problems. All games started pretty fast and ran without visual stuttering.
Systrace report
This is the picture of systrace showing 5 seconds of our game being started (rectangles were drawn by me).
systrace
As you can see, the systrace have found only 4 frames rendered (the pink rect), which were drawn from the RenderThread. But by some reason Android cannot detect our GL draw calls which are performed in another thread (blue rects).
Pre-launch reports also displays only 3 to 4 frames, each 300-400 ms long.
Initialization code
Our game engine runs all game logic and render code in a separate thread. This is simplified initialization code.
The worker thread is created from our Activity’s onStart() overriden method.
public class MyActivity extends Activity
{
protected Thread worker = null;
private native void Run();
#Override
protected void onStart()
{
super.onStart();
if(worker == null)
{
worker = new Thread()
{
public void run()
{
Run();
}
};
worker.start();
}
}
}
The only thing the thread does is the Run() native function. This function may be resolved into something like this:
void MyActivity::Run()
{
initApp();
while(!destroyRequested())
{
// Process the game logic.
if (activated && window != NULL)
{
time->process();
input->process();
sound->process();
logic->process();
graphics->draw();
}
}
clearApp();
}
As you can see, the worker thread constantly spins the update-and-draw loop. Vsync protects the loop from overperforming. Heavy operations like resource loading are done asynchronously to avoid freezes.
From the user side this approach works just fine. Games are loading fast and go smoothly.

Android with Nexus 6 -- how to avoid decreased OpenSL audio thread priority relating to app focus?

I'm encountering a strange problem when trying to implement low-latency streaming audio playback on a Nexus 6 running Android 6.0.1 using OpenSL ES.
My initial attempt seemed to be suffering from starvation issues, so I added some basic timing benchmarks in the buffer completion callback function. What I've found is that audio plays back fine if I continually tap the screen while my app is open, but if I leave it alone for a few seconds, the callback starts to take much longer. I'm able to reproduce this behavior consistently. A couple of things to note:
"a few seconds" ~= 3-5 seconds, not long enough to trigger a screen change
My application's activity sets FLAG_KEEP_SCREEN_ON, so no screen changes should occur anyway
I have taken no action to try to increase the audio callback thread's priority, since I was under the impression that Android reserves high priority for these threads already
The behavior occurs on my Nexus 6 (Android 6.0.1), but not on a Galaxy S6 I also have available (Android 5.1.1).
The symptoms I'm seeing really seem like the OS kicks down the audio thread priority after a few seconds of non-interaction with the phone. Is this right? Is there any way I can avoid this behavior?

While watching the latest Google I/O 2016 audio presentation, I finally found the cause and the (ugly) solution for this problem.
Just watch the around one minute of this you tube clip (starting at 8m56s):
https://youtu.be/F2ZDp-eNrh4?t=8m56s
It explains why this is happening and how you can get rid of it.
In fact, Android slows the CPU down after a few seconds of touch inactivity to reduce the battery usage. The guy in the video promises a proper solution for this soon, but for now the only way to get rid of it is to send fake touches (that's the official recommendation).
Instrumentation instr = new Instrumentation();
instr.sendKeyDownUpSync(KeyEvent.KEYCODE_BACKSLASH); // or whatever event you prefer
Repeat this with a timer every 1.5 seconds and the problem will vanish.
I know, this is an ugly hack, and it might have ugly side effects which must be handled. But for now, it is simply the only solution.
Update:
Regarding your latest comment ... here's my solution.
I'm using a regular MotionEvent.ACTION_DOWN at a location outside of the screen bounds. Everything else interfered in an unwanted way with the UI. To avoid the SecurityException, initialize the timer in the onStart() handler of the main activity and terminate it in the onStop() handler. There are still situations when the app goes to the background (depending on the CPU load) in which you might run into a SecurityException, therefore you must surround the fake touch call with a try catch block.
Please note, that I'm using my own timer framework, so you have to transform the code to use whatever timer you want to use.
Also, I cannot ensure yet that the code is 100% bulletproof. My apps have that hack applied, but are currently in beta state, therefore I cannot give you any guarantee if this is working correctly on all devices and Android versions.
Timer fakeTouchTimer = null;
Instrumentation instr;
void initFakeTouchTimer()
{
if (this.fakeTouchTimer != null)
{
if (this.instr == null)
{
this.instr = new Instrumentation();
}
this.fakeTouchTimer.restart();
}
else
{
if (this.instr == null)
{
this.instr = new Instrumentation();
}
this.fakeTouchTimer = new Timer(1500, Thread.MIN_PRIORITY, new TimerTask()
{
#Override
public void execute()
{
if (instr != null && fakeTouchTimer != null && hasWindowFocus())
{
try
{
long downTime = SystemClock.uptimeMillis();
MotionEvent event = MotionEvent.obtain(downTime, downTime, MotionEvent.ACTION_DOWN, -100, -100, 0);
instr.sendPointerSync(event);
event.recycle();
}
catch (Exception e)
{
}
}
}
}, true/*isInfinite*/);
}
}
void killFakeTouchTimer()
{
if (this.fakeTouchTimer != null)
{
this.fakeTouchTimer.interupt();
this.fakeTouchTimer = null;
this.instr = null;
}
}
#Override
protected void onStop()
{
killFakeTouchTimer();
super.onStop();
.....
}
#Override
protected void onStart()
{
initFakeTouchTimer();
super.onStart();
.....
}

It is well known that the audio pipeline in Android 6 has been completely rewritten. While this improved latency-related issues in most cases, it is not impossible that it generated a number of undesirable side-effects, as is usually the case with such large-scale changes.
While your issue does not seem to be a common one, there are a few things you might be able to try:
Increase the audio thread priority. The default priority for audio threads in Android is -16, with the maximum being -20, usually only available to system services. While you can't assign this value to you audio thread, you can assign the next best thing: -19 by using the ANDROID_PRIORITY_URGENT_AUDIO flag when setting the thread's priority.
Increase the number of buffers to prevent any kind of jitter or latency (you can even go up to 16). However on some devices the callback to fill a new buffer isn’t always called when it should.
This SO post has several suggestions to improve audio latency on Anrdoid. Of particular interest are points 3, 4 and 5 in the accepted answer.
Check whether the current Android system is low-latency-enabled by querying whether hasSystemFeature(FEATURE_AUDIO_LOW_LATENCY) or hasSystemFeature(FEATURE_AUDIO_PRO).
Additionally, this academic paper discusses strategies for improving audio latency-related issues in Android/OpenSL, including buffer- and callback interval-related approaches.

Force resampling to native device sample rate on Android 6.
Use the device's native sample rate of 48000. For example:
SLDataFormat_PCM dataFormat;
dataFormat.samplesPerSec = 48000;

OpenSL ES Android: "Too many objects" SL_RESULT_MEMORY_FAILURE

I'm having a problem with OpenSL ES on Android. I'm using OpenSL to play sound effects. Currently I'm creating a new player each time I play a sound. (I know this isn't terribly efficient, but it's "good enough" for the time being.)
After a while of playback, I start to get these errors:
E/libOpenSLES(25131): Too many objects
W/libOpenSLES(25131): Leaving Engine::CreateAudioPlayer (SL_RESULT_MEMORY_FAILURE)
I'm tracking my create/destroy pattern and I never go above 4 outstanding objects at any given time, well below the system limit of 32. Of course, this is assuming that the Destroy is properly working.
My only guess right now is that I'm doing something incorrectly when I clean up the player objects. One possible issue is that the Destroy is often called in the context of the player callback (basically destroying the player after it's finished playing), although I can't find any reference suggesting this is a problem. Are there any other cleanup steps I should be taking besides "Destroy"-ing the player object? Do the Interfaces need to be cleaned up somehow as well?
-- Added --
After more testing, it happens consistently after the 30th player is created (there is an engine and a mix too, so that brings the total to 32 objects). So I must not be destroying the object properly. Here's the code--I'd love to know what's going wrong:
SLuint32 playerState = 0;
SLresult result = (*pPlayerObject)->GetState(pPlayerObject, &playerState);
return_if_fail(result);
if (playerState == SL_OBJECT_STATE_REALIZED)
{
(*pPlayerObject)->AbortAsyncOperation(pPlayerObject);
(*pPlayerObject)->Destroy(pPlayerObject);
}
else
{
__android_log_print(1, LOG_TAG, "Player object in unexpected state (%d)", playerState);
return 1002;
}

if (playerState == SL_OBJECT_STATE_REALIZED)
is not needed. Try to do it always.
AbortAsyncOperation is called in Destroy => not needed.
So try just (*pPlayerObject)->Destroy(pPlayerObject); it should be enough.
Edit:
I tested, and found solution.
You cannot call Destroy() from player callback. Should make "destroy" list and destroy it somewhere else, for example, in main thread.