In my VR app for Android Smartphones (I think it doesn't matter whether it is a VR app or not) I download 6 big images from Street View and built a skybox out of it. During this process the whole app incl. environment and UI freezes. After ~10sec the process is down and Street View is there.
How can I seperate the main processing from UI? So the phone is working, but should not freeze. This is a common web problem, but how can I solve this in Unity for Android?
Thank you!
code:
private byte[] GetStreetviewTexture(string url) {
WWW www = new WWW(url);
while (!www.isDone) ;
if (!string.IsNullOrEmpty(www.error))
{
Debug.LogWarning("Unable to DL texture: " + www.error);
}
else
{
bytes = www.texture.EncodeToPNG();
}
return bytes;
}
You probably want to use an IEnumerator to thread it with a Unity Coroutine. If you're already using this, which I'm assuming you actually are - could it be a performance issue that freezes the device? If so, check the Unity profiler and/or Unity Remote. Coroutines are, like comments suggest, not a Thread - but emulates a threaded behaviour.
Basic Coroutine code:
void SomeMethod() {
StartCoroutine(Threaded());
}
IEnumerator Threaded() {
// Do something
yield return new WaitForSeconds(3f);
}
Check out the WWW class and how to make an IEnumerator wait for download to complete
Update responding to OP's follow-up question:
How to tell when a Coroutine is done and get a value
Update responding to OPs follow-up follow-up question:
This is a simplified example of your current logic/flow. NOTE: THIS DOES NOT WORK. Do not use this code as an example for how it's supposed to be done, I just want to illustrate the differences in how you should approach your problem.
void ButtonClicked() {
SetTexture()
}
void SetTexture() {
Texture texture = GetTexture()
Object.texture = texture;
}
Texture GetTexture() {
Texture texture;
StartCoroutine(DownloadTexture((textureCallback) => {
texture = textureCallback;
}));
return texture;
}
IEnumerator DownloadTexture(Action<Texture> callbackTexture)
{
WWW www = new WWW(URL);
yield return www;
callback(www.texture);
}
Not only does this not work, because a Coroutine is run asyncronously with the rest of the code, but it's not a good approach for this task.
Instead of doing
Button click -> set texture -> start download -> error setting texture -> download finished
you want:
Button click -> download texture -> wait for it to be done -> set texture
Like this:
void ButtonClick() {
StartCoroutine(DownloadTexture((callbackTexture) => {
SetTexture(callbackTexture); // Will run SetTexture when Coroutine DownloadTexture is completed.
}));
}
IEnumerator DownloadTexture(Action<Texture> callbackTexture)
{
WWW www = new WWW(URL);
yield return www;
callback(www.texture);
}
void SetTexture(Texture texture) {
object.texture = texture;
}
Note that this is just flow-code. Your code will look different, using byte[] instead of Texture and I don't know if you're using a Button to start the texture-setting. This can be any sort of trigger/starting point.
Related
My first attempt at AndroidPlot. The data I want to plot (and update every 5 seconds when a new data point arrives) comes from an ArrayBlockingQueue of up to 720 timestamped points. I have a class that implements the XYSeries and PlotListener interfaces. It has a method updatePlotData that just extracts the data from the queue into an array:
class TempPlotSeries implements XYSeries, PlotListener {
private final static String TAG = TempPlotSeries.class.getSimpleName();
private Pair<Date, Float>[] plotArray;
void updatePlotData( ArrayBlockingQueue<Pair<Date, Float>> dataQueue ) throws InterruptedException {
synchronized ( this ) {
wait(); // don't update data until we're notified that current plot is done (& we can get lock)
plotArray = dataQueue.toArray( new Pair[0] );
if( DEBUG ) Log.d( TAG, "updatePlotData run with " + plotArray.length + " data points" );
notifyAll(); // release lock & let other threads know they can continue
}
}
// XYSeries implementation
#Override
public int size( ) {
return plotArray.length;
}
#Override
public Number getX( int index ) {
return (index - HISTORY_BUFFER_SIZE) / (60/TEMP_UPDATE_SECONDS); // e.g., -60 minutes at left edge of graph, -1/12 min at right
}
#Override
public Number getY( int index ) {
return plotArray[index].second; // the temp value
}
#Override
public String getTitle( ) {
return "Temp History";
}
// PlotListener Implementation
#Override
public void onBeforeDraw( Plot source, Canvas canvas ) {
synchronized ( this ) {
try {
wait(); // wait for data updating to finish if it's in progress on another thread
} catch ( InterruptedException e ) {
// unlikely to be interrupted?
}
}
}
// between these 2 calls the plot is redrawn
#Override
public void onAfterDraw( Plot source, Canvas canvas ) {
synchronized ( this ) {
notifyAll( ); // plot done, OK to update data
}
}
}
I don't have much experience with synchronization--does this look reasonable?
My plot setup is:
tempHistoryPlot = (XYPlot) findViewById(R.id.temp_history);
tempPlotSeries = new TempPlotSeries();
tempHistoryPlot.setRenderMode( Plot.RenderMode.USE_BACKGROUND_THREAD );
tempGraphFormatter = new LineAndPointFormatter(this, R.xml.line_point_formatter_with_labels);
tempHistoryPlot.addSeries(tempPlotSeries, tempGraphFormatter);
tempGraphWidget = tempHistoryPlot.getGraph();
(couldn't find any documentation on the purpose of getGraph() so don't know if I need it.)
I have an Observable (RxJava) that emits the entire data queue when a new sample is available (every 5 seconds). If the queue is full I discard the oldest value. Then I have:
tempPlotSeries.updatePlotData( newTempHistory );
tempHistoryPlot.redraw();
But the plot isn't drawn. When the app first launches, the "dummy" plot appears in its View, but as soon as I try to draw the plot the entire ConstraintLayout containing the XYPlot element (and other UI elements) is completely blanked. What's going on here?
Other questions: it's my understanding that any code affecting the Android UI must run on the main thread. But we're using a background thread to render the plot. How does this work? Do I perhaps need to insert a .observeOn( AndroidSchedulers.mainThread() operator in my Observable chain?
I don't have much experience with synchronization--does this look reasonable?
I don't think you need the wait() inside the synchronized block inside updatePlotData. You can also use SimpleXYSeries as a reference for how to setup synchronization of this sort.
When the app first launches, the "dummy" plot appears in its View, but as soon as I try to draw the plot the entire ConstraintLayout containing the XYPlot element (and other UI elements) is completely blanked.
I'm having trouble visualizing this. Could you add a screenshot of the "dummy" plot and the subsequent blank plot?
it's my understanding that any code affecting the Android UI must run on the main thread. But we're using a background thread to render the plot. How does this work?
The general rules of using the main thread to update the UI still exist, Androidplot is just using a technique to minimize the main thread usage during intensive rendering: A background thread is used to fill a bitmap buffer with the data to be shown, then notifies the main thread when the buffer is ready to be displayed.
Somewhat Unrelated Suggestion: Looking at your TempPlotSeries implementation, I notice that you are modeling your data as a Pair<Date, Float>[] but your getX() implementation does not make use of the Date part. It appears you're trying to model your data using what I assume is your desired display format for your domain, ie. -60 to -1/12 minutes. For simplicity I'd suggest making getX() return the Date's long epoch value instead. You can apply a display format to these values later.
I have some Android app that should load some data on start and process it as soon as possible. For better performance data processing done in native code.
But in some time data amount was increased and it becomes to block UI while processing. The weird thing is that I launch JNI call from background thread and I believe native code should also work in background.
So few questions:
how to detect is native code works in main thread?
how to detect who blocks UI?
Some code sample:
// java
new Thread(new Runnable() {
#Override
public void run() {
ResourceLoader.loadListsAndParse(); // native call
}
}).start();
// c++
void RulesManager::loadRules() {
if (!rules_loaded_) {
using namespace std;
base::FilePath my_dir;
GetCacheDirectory(&my_dir);
this->loadRulesInternal(my_dir, true);
}
}
void RulesManager::loadRulesInternal(base::FilePath& path, bool processExceptions) {
base::FileEnumerator files(path, false, base::FileEnumerator::FILES);
for (base::FilePath name = files.Next(); !name.empty(); name = files.Next()) {
this->loadRulesFromFile(name.AsUTF8Unsafe(), processExceptions);
}
}
bool RulesManager::loadRulesFromFile(const std::string& filePath, bool processException) {
bool result = false;
std::ifstream file(filePath);
std::string str;
while (std::getline(file, str)) {
result = this->addRule(str, processException);
}
return result;
}
Android has system calls getpid() and gettid() defined in <unistd.h>. If the pid_t return values are equal, then your native code runs on the UI thread.
I use BlockCanary for that purposes. If you know about LeakCanary, BlockCanary provides reports about UI blocking in the same way.
How to detect is native code works in main thread?
You can apply Log method inside your code.
For example your code is:
if(myInt > 0){
doSth();
}
Insert Log method;
if(myInt > 0){
doSth();
Log.i("Info", "if statement is true.");
}
now your device will type to the console if your if statement runs. So in your thread, you can insert Log method to critical lines to see if it is running or not.
how to detect who blocks UI?
As in my first answer, again you can see it from the console if you implement Log methods to your code.
But if you want to do it properly, I suggest you to check Asynctask. This will take time to learn how to implement it properly but the most practical way to pass the long jobs to background in Android is Asynctask.
i have an android mobile app and im trying to check if a specific LatLng is at water, so im using google static map api to get an image of the location, and then to check if the image is blue.
im using this code -
private boolean result;
public boolean IsWater(LatLng position)
{
imageView = (ImageView) this.findViewById(R.id.imageView);
checkText= (TextView) this.findViewById(R.id.checkText);
String lati = Double.toString(position.latitude);
String longi = Double.toString(position.longitude);
String url = "http://maps.googleapis.com/maps/api/staticmap?center="+lati+"," + longi + "&zoom=20&size=1x1&style=element:labels%7Cvisibility:off&style=element:geometry.stroke%7Cvisibility:off";
Picasso.with(MainActivity.this).load(url)
.into(imageView, new com.squareup.picasso.Callback() {
#Override
public void onSuccess() {
Bitmap bitmap = ((BitmapDrawable) imageView.getDrawable()).getBitmap();
int pixel = bitmap.getPixel(0, 0);
int blueValue = Color.blue(pixel);
if(blueValue>250)
result =true;
}
#Override
public void onError() {
result =false;
}
});
return result;
}
the problem, i think, is that it is not synchronized, and IsWater get to the last line and return a null for result before the onSuccess kicks in...
any thoughts?
Picasso loads images on a background thread by default. The operation you are running is asynchronous. Therefore, it does not block your method from returning result before the onSuccess callback has been called.
The problem is Picasso is running Async. within the calling method "isWater", so what ends up happening is the method will return 'false' because instead of waiting on Picasso to finish because it isn't in serial execution. This is due to the function call's stack frame being popped off the stack once it reaches the return statement.
What you need to do is the following by using a Target.
// make sure to set Target as strong reference
private Target loadtarget;
public void loadBitmap(String url) {
if (loadtarget == null){
loadtarget = new Target() {
#Override
public void onBitmapLoaded(Bitmap bitmap, Picasso.LoadedFrom from) {
// do something with the Bitmap
handleLoadedBitmap(bitmap);
}
#Override
public void onBitmapFailed() {
}
};
}
Picasso.with(this).load(url).into(loadtarget);
}
public void handleLoadedBitmap(Bitmap b) {
// do something here
}
This code was taken from here, and should offer you some insight on how to get it work for your goal.
A Target is essentially an object that holds the bitmap you need so it is still in memory. Generally used for custom view objects though as a field. Here is documentation Target docs
Asynchronous execution is one of the hardest things to wrap ones head (and subsequently ones code) around. In all of the JavaScript frameworks I've used, the network communication is done in a background thread. The intended effect is that the User Interface thread is left free to keep the user from thinking that things locked up. Mouse-overs and tool-tips will all still work, while a background thread is dragging data out of a slow server.
The code patterns, on the other hand, aren't as nicely shaped.
My problem is/was still basically thinking linearly, or functionally, instead of embracing the event-driven nature of modern JavaScript: Passing a function to an asynchronous method to completely handle that response. Not just return a value, but perform the full task that the value was needed for. The callback can call the other functions to assist with that task, and may be able to fill in a cache (of whatever sort) so that other functions that may need this data do not necessarily have to wait for another response. This often (to me) feels backwards from the logic pattern I was following to solve the original purpose of the code.
I've stumbled on this pattern-flip many times, coming from C/C++ as my first programming language. It can sometimes help to avoid the anonymous function pattern of callback definition and define one's callback functions with names, then pass the name to the asynchronous call, but that is extra steps and extra memory use in the long run. The big hurdle is thinking in terms of Event and EventHandler, versus function and data.
I hope this helps a little.
I use OpenCV's JavaCameraViewto capture images. So it has onCameraFrame method.
#Override
public Mat onCameraFrame(CvCameraViewFrame inputFrame) {
rgb = inputFrame.rgba();
if(viewMode == VIEW_MODE_EDGE){
Mat mIntermediateMat = new Mat(rgb.size(), rgb.type());
Imgproc.Canny(rgb, mIntermediateMat, 80, 90);
Imgproc.cvtColor(mIntermediateMat, rgb, Imgproc.COLOR_GRAY2BGRA, 4);
mIntermediateMat.release();
}
return rgb;
}
That method is always runing and I convert the incoming image to Canny edge and display.
I implemented a button to copy that Canny image and pass to another activity. For that I made a global public static Mat Canny_image; When the button is pressed the rgb image from onCameraFrame is copied to Canny_image. Then the next activity is called.
My code is
ImageCapture.setOnClickListener(new OnClickListener() {
public void onClick(View v) {
if (mOpenCvCameraView != null)
{
Canny_image= rgb.clone();
}
final Intent intent = new Intent(ctxt, TextExtraction.class);
intent.setFlags(Intent.FLAG_ACTIVITY_REORDER_TO_FRONT);
startActivity(intent);
}
});
Problem: when i press the button and if canny conversion is finished in onCameraFramemethod, I receive the canny image (the image with only edges). But if it is not converted yet or half way converted, I receive original rgb image (original rgb image) or half way converted image. How can I make sure I receive only canny image and make sure the conversion process is completed and do cloning as Canny_image= rgb.clone();in ImageCapture.setOnClickListener(new OnClickListener(){};
Thanks
To synchronise two methods in Android Java you can use the synchronized(this) syntax:
Public void OnFrameAvailable() {
synchronized(this) {
// Wair for condition false...
// Flag condition true.
}
}
And
Public void ProcessFrame() {
synchronized(this) {
// Wair for condition true...
// Do something with incoming data...
// Flag condition false
}
}
This would ensure that both methods execute in mutual exclusion, although in general is not a good idea to make a callback wait... If you fully want to decouple both, you should have a thread for processing, with a Looper, and post() a Runnable to it from the OnFrameAvailable().
I don't have any knowledge of android but just a suggestion..isn't possible for you to set a flag (e.g set Flag=1) after getting the Canny image onCameraFrame method and in the another method you made a check that whether the flag ==1 or not?
first()
{
flag = 0;
convert to canny;
flag=1;
}
second()
{
if(flag ==1)
try to get the Canny image
}
And excuse me if its not possible :)
Assuming rgb is a static then one immediate thing to point out is that this will be 4 channel Mat on the call (use inputFrame.gray() if you want a single channel). Canny is expecting a single channel as is the call to CvColor viz the COLOR_GRAY2BGRA conversion type. I don't think that is causing your particular problem but worth checking out.
As onCameraFrame is a callback you could well be processing rgb when another call comes in so rgb could be getting overwritten before you return it in the method. Adding synchronized to the onCameraFrame method should help in that regard but this might impact performance. Alternatively, making rgb local to onCameraFrame should also avoid that particular concurrency issue.
One final thought is rather than do the copy when you click the button, you could do the copy to Canny_image when you finish processing in onCameraFrame so it is always available and onClick will take the current version.
All examples of the use of a SurfaceView seems to use a run method that performs a busy loop. Is that a valid way to do this? All the code I can see follows this paradigm from the lunar lander sample. However, creating a busy while loop seems to be a strange way to code multi threaded apps. Shouldnt the drawing code wait on a queue of drawing commands, or something similar. I would have implemented it that way, but the amount of code that I see that does is like below makes me ask the question... What is the best semantics for a thread drawing on a SurfaceView.
public void run() {
while (mRun) {
Canvas c = null;
try {
c = mSurfaceHolder.lockCanvas(null);
synchronized (mSurfaceHolder) {
// DO DRAWING HERE
}
} finally {
if (c != null) {
mSurfaceHolder.unlockCanvasAndPost(c);
}
}
}
}
I don't know what is best practice in this case, but I have successfully used a slightly modified version of that example in my apps. Since I respond to touch input (rather than continuously updating the canvas) I added a flag to test if drawing even needs to be done. I also added a sleep after each refresh to limit system load. This is my code inside of the try block:
if(mPanel.needsRefresh()) {
c = mSurfaceHolder.lockCanvas(null);
synchronized (mSurfaceHolder) {
mPanel.onDraw(c);
}
} else {
SystemClock.sleep(10);
}