For my master thesis I need to analyse iris recognition data. Therefore I have about 400 templates created. Each of this templates has to be compared to each other template, resulting in ~160.000 matching results.
Those results need to be uploaded to my Azure Easy Table. I really don't know where to start as the ThreadPoolExecutor cannot handle more than 128 Threads in parallel.
What is the correct approach to do something like this? Time is not really an issue.
This is my current approach:
for (int i = 0; i < mIrisEntries.size(); i++){
match(i);
}
public void match(final int position) {
IrisEntry inputEntry = mIrisEntries.get(position);
// takes about 10ms
List<IrisResult> results = mUSITHelper.matchEntries(inputEntry, mIrisEntries, this);
for (IrisResult s : results) {
try {
Thread.sleep(1000);
mAzureTableManager.addIrisResult(s); // here the AsyncTask is started
} catch (InterruptedException e) {
e.printStackTrace();
}
}
}
This approach works for some time,. but then the system starts to kill all threads and uploading is cancelled.
I do not completely follow all of what you describe in the question and comments.
However, at that amount of processing time (~30 minutes if I understood you), you need to use a foreground service, as otherwise your process may not survive that long. In that service, use your own ThreadPoolExecutor, with the number of threads in the pool tuned based on the number of CPU cores. Ideally, I would not post ~160,000 jobs to that executor, but rather 400, where each of those 400 jobs performs the work for one of your templates. The last job you post to the executor would do any final cleanup, plus stop the service.
Also, you may be able to halve your work. Suppose A and B are two of your templates. If comparing A to B is the same as comparing B to A, you do not need to do both comparisons. Your 400 templates expanding to ~160,000 jobs seems to imply that you are comparing A to B and B to A for all pairs.
As #CommonsWare said, I think it's a good idea for reducing the number of upload request & the data size.
However, per my experience, I suggest that you need to try to enable offline sync for your android app so that don't need to consider to the issue about uploading data asynchronously.
Hope it helps.
Related
Since I am new to android programming, I am not sure how to write code efficiently hence the reason for this question. I am creating an app. A basic app in which the app generates 10 random math questions and evaluates it from left to right (ignoring orders of operations). E.g. 3+5/2 should equal 4 instead of 5.5.
I am getting the error Launch timeout has expired. I have researched this and found out that its because the main thread is doing too much work. How do I overcome this? My app first does alot randomizing integers, could that be the case?
This is the code. It is pretty long.
P.S. in the display method, i hrdcoded it to display the first elements just to see if it will display.
public void initAnswers(String[] questionToBeLooped){
for(int i =0; i < questionToBeLooped.length; i++){
if(mathOperations.length == 2){
runningTotal = evaluateAnswerTwoOperations(mathOperations[0], mathNumbersInIntFormat.get(0), mathNumbersInIntFormat.get(1));
}else{
int operationsCounter =0;
int numbersCounter =1;
runningTotal = mathNumbersInIntFormat.get(0);
while(mathOperations[operationsCounter] != "="){
runningTotal = evaluateAnswerTwoOperations(mathOperations[0],runningTotal,mathNumbersInIntFormat.get(numbersCounter));
}
}
answers[i] = runningTotal;
}
}
Could someone tell me how to write this efficiently and also can you provide some tips to generate fluent and efficient apps.
Not sure what in your code is the cause, but you should offload the heavy work into an AsyncTask. AsyncTask will allow you to execute code in a background thread via doInBackground and callback to the UI thread via onPostExecute. The heavy work you are doing is probably something around the looping constructs you have.
Keep in mind that you cannot modify the UI from the background, so if you need to update UI, wait until the heavy work is finished and do it in onPostExecute.
See the docs here on AsyncTask.
Also worth noting that AsyncTask has a lot of flaws, but since you are new - it is where I would recommend to start.
I'd like to ask for some help about the following problem I have.
I'd like to create an application that solves the Rubik's cube with an optimal solution. I downloaded the following library, whitch supposedly does just that using the Kociemba's Algorithm.
http://kociemba.org/twophase.jar
Apparently it can solve the cube in under 0.5 sec, but in my app it never returned the solution due to memory problems. I know it works, I tested it with wrong inputs and it returns the documented error codes.
I call it in my onCreate method like this:
resultTxt = Search.solution(data, 21, 10, true);
resultTxt is a String variable and it should contain the solution.
It quickly eats up the memory.
I tried it with IntentService without success. By this I mean it didn't really changed anything.
As i didn't find any evidence of anyone using this library in any android application, I thought I would ask someone who is more experienced than me.
Is there any way I could make this work on Android, or is this as impossible as I thought?
It may be a bit late, but I was also facing this issue quite recently when I was working on a Rubik's-Cube-solving-robot using an Android-Smartphone for scanning the cube and computing the solution, so I'll put here what I have found out.
What is the problem?
Let's start off by discussing where the problem causing that performance issue actually is located.
The reason for that being so slow is the class CoordCube, which looks (very simplified) like this:
class CoordCube {
short[] pruneTables;
static {
/* compute and save ~50MB data in `pruneTables` */
}
}
Basically what it does, is to load a pretty big amount of data into lookup-tables which are required for a fast solving procedure. This loading is automatically executed by the JVM once this class is first instantiated. That happens on line 159 in Search.solution():
/* simplified code of solution() */
if (cube.isValid()) {
CoordCube c = new CoordCube(); // pruning tables will be loaded on this line
That is also the reason why this method executes in negligible time as long as the passed cube is invalid, since it never gets to load the tables.
Possible Solutions:
Now that we have identified where the problem is located, let's focus on how to solve it.
I have come up with 3 different approaches, where the first one is probably the easiest (but also the slowest execution wise ...) and is also used in my App. The other two are just ideas on how to improve the performance even more.
Approach 1:
The first and most simple approach is to just manually preload the lookup tables in a kind of LoadingActivity with a ProgressBar showing our current progress. For that we first want to be able to manually control exactly when which tables are loaded (when the class is first instantiated is not precise enough), like this:
loadTable1() {
/* load pruning table number 1 */
}
For that I have written some simple utility here (code is too long to paste). Make sure to check out my instructions there on how to properly import that code in your App.
Also we will probably want to do the loading in the background, namely in an AsyncTask. This is how I have done it in my application (PruneTableLoader is included in the previous link):
private class LoadPruningTablesTask extends AsyncTask<Void, Void, Void> {
private PruneTableLoader tableLoader = new PruneTableLoader();
#Override
protected Void doInBackground(Void... params) {
/* load all tables if they are not already in RAM */
while (!tableLoader.loadingFinished()) { // while tables are left to load
tableLoader.loadNext(); // load next pruning table
publishProgress(); // increment `ProgressBar` by one
}
return null;
}
#Override
protected void onProgressUpdate(Void... values) {
super.onProgressUpdate(values);
/* increment `ProgressBar` by 1 */
}
}
When using my PruneTableLoader, the loading of all tables needs about 40s on my Samsung Galaxy S3 with 250 MB RAM free. (in contrast it needs well over 2min when loading them automatically and in addition often results in a crash ...)
That may still sound quite slow considering it needs < 1s on PC, but at least you must only do that once, since Android caches the static-variables and you so don't have to load them on every startup of your App.
Approach 2: (untested)
I assume it would be faster to save the pruning tables in a file or a database and load them from there instead of always recomputing them. I have not yet tested that though and it will probably require quite some work getting the saving and loading to work properly. (also maybe it's not even faster because of access times)
Approach 3: (untested)
Well, the hardest and also by decades most work expensive solution would be, to simply rewrite the whole algorithm in C or C++ and invoke it in the App via JNI. (Herbert Kociemba has not published his C-sourcecode yet as far as I know ...)
This is going to be the performance wise fastest solution for sure. (also for the solving procedure itself)
All in all approach 1 is probably the effort/benefit-wise best approach for the beginning (and also was for me), so I would recommend you to go with that, in case the loading time is not such a huge issue for your Application.
I'm not completely satisfied with the performance of that myself though, so I may try out approach 2 and maybe even approach 3 some time in the future. In case I do that, I will update this post with my results then.
I have a test case for my app which fills in the TextViews in an Activity and then simulates clicking the Save button which commits the data to a database. I repeat this several times with different data, call Instrumentation.waitForIdleSync(), and then check that the data inserted is in fact in the database. I recently ran this test three times in a row without changing or recompiling my code. The result each time was different: one test run passed and the other two test runs reported different data items missing from the database. What could cause this kind of behavior? Is it possibly due to some race condition between competing threads? How do I debug this when the outcome differs each time I run it?
Looks like a race condition.
remember in the world of threading there is no way to ensure runtime order.
I'm not an android dev so I'm only speculating but UI is only on one event thread generally so when you call the method from another thread (your test) you're probably breaking that as you're outside of the event thread.
You could try using a semaphore or more likely a lock on the resource.
http://docs.oracle.com/javase/1.5.0/docs/api/java/util/concurrent/locks/Lock.html
http://docs.oracle.com/javase/1.5.0/docs/api/java/util/concurrent/Semaphore.html
I (finally!) found a solution to this problem. I now call finish() on the tested Activity to make sure that all of its connections to the database are closed. This seems to ensure consistency in the data when I run the assertions.
I would suggest making a probe for the database data rather than a straight assert on it. By this I mean make a piece of code that will keep checking the database for up to a certain amount of time for a condition rather than waiting for x seconds (or idle time) then check, I am not on a proper computer so the following is only pseudo code
public static void assertDatabaseHasData(String message, String dataExpected, long maxTimeToWaitFor){
long timeToWaitUntil = System.getCurrentTimeMillis() + maxTimeToWaitFor;
boolean expectationMatched = false;
do {
if(databaseCheck() == dataExpected){
expecttionMatched == true;
}
}while(!expectationMatched && System.getCurrentTimeMillis() < timeToWaituntil);
assertTrue(message, expectationMatched);
}
When i get to a computer i will try to relook into the above and make it better (I would actually of used hamcrest rather than asserts but that is personal preference)
I'm getting data from a server for my app. The "getData" functions are included in the app's main Activity, in a splash thread. The problem I'm having is this:
If I quickly enter news or description after loading up the app, I notice that not all info has loaded (last 2 or 3 strings that needed to be saved are null). If, however, I allow the app a few more seconds after displaying the main menu (after completing the splash thread), the problem doesn't occur, all info is stored correctly on the phone. I tried delaying the splash screen by a few seconds but that's not really an elegant solution nor does it always work.
My question is how can I make sure that the functions have been completed before it jumps to "finally"
I'm not storing the data in any database, just in public static string arrays in another class.
You have my code below:
if(networkAvailable()){
Thread splashTread = new Thread() {
#Override
public void run() {
try {
getData.execute(description_Hyperlinks);
getNews.execute(new String[]{newsJSON_Hyperlink});
getOffers.execute(new String[]{offersJSON_Hyperlink});
for(int i = 0; i<3; i++)
Thread.sleep(1000);
} catch (InterruptedException e) {
// TODO Auto-generated catch block
e.printStackTrace();
} finally {
finish();
startActivity(new Intent(FlexFormActivity.this, MainMenu.class));
stop();
}
}
};
splashTread.start();
For the purpose of making the discussion in the comments on the initial question more readable and physically visible
It would be a better idea to use the Android build-in support for asynchronous task-handling in form of the AsyncTask-class. This allows you to "hook in" the task, giving you the opportunity to react on the different stages of the progress.
The idea would be to not make getData, getNews and getOffers each extend AsyncTask, but to rather have a single task (called e.g. "LoadContents"), which loads the data, the news and the offers one after another.
It would then be possible to determine, when the whole initial work has been done, which makes it easy to react on this "completion of task", in whatever form you can imagine.
As a little code-review, It should normally never be necessary to use the Thread-class itself, as Java and Android provide many wrappers around it (in particular the Java Executor Framework), which should be favored in order to produce more clean and reliable code.
Also, as a general advice on "disabling the back-button" (which is used by #Eugen to ensure that the splash-screen stays present): Don't do it. It's not the kind of behavior a user expects when he uses an application.
Imagine someone has accedently opened an app, which takes ~10 seconds for the initial loading of contents, and this process can't be canceled. The user will have to wait the entire time, only to then leave the app without using it.
Therefore, you should not "deactivate" the back-button, but rather make your task (and therefore the initial loading of your application) "cancel-able". When using an AsyncTask, this is already implemented for you.
I am not fully understanding what the synchronization block is doing nor why it is necessary.
Can someone explain in a "synchronizing for dummies" kind of way?
In a book I am reading, the author tells me "The synchronization is necessary, since the members we manipulate within the
synchronized block could be manipulated in the onPause() method on the UI thread."
He creates an Object named stateChanged and instantiates it as a new object.
Then, in the synchronization block he uses the stateChanged object as the argument.
This whole thing is throwing me off and I do not like to move on until I have a pretty good understanding of what is going on.
The classic example is: Imagine you have two threads of operation, and both of them reference the same method:
public void addToGlobalVar(int y) {
int x = globalVar; //what if a thread stops right after this line?
x += y;
globalVar = y;
}
where globalVar is some other predefined number that this method can interact with and set. Lets say globalVar is 50.
Threads get computing time on a somewhat arbitrary basis, so you never fully know the precise nanosecond one stops and the other gets CPU time.
In this example, if you launched an AsyncTask in addition to the UI thread, and both at some point use addToGlobalVar(10), what can happen is that one thread might be interrupted at line 2 of that code block. If the other thread goes through while that one is sleeping, it will successfully set globalVar to 60. But when the other one wakes up, it still thinks x = 50, and its going to then set it to 60. So in essence you just made 50+10+10 = 60. Hopefully you can see how this becomes a problem.
You can fix this simple example by making the calculation atomic (skip declaring x, 1 line, all calcs done) or if the logic wasn't able to be condensed to 1 line, you make a block of code atomic by using synchronized.
The book to read is Java Concurrency in Practice.
You should really just segregate this idea from Android, although your code is going to be running on Dalvik this is a Java concept. Not an Android one.
The synchronized block takes an object as a parameter, any object, and when flow enters the body of the synchronized block, any other thread that runs in to a synchronized block with the same instance (object) as the parameter has to wait for the previous one to complete. That's a very basic description.
This is an entire sub-field of computer science and without serious study you will probably not understand it.
You have to fully understand it before you use it. It is standard android synchronization using object-oriented monitors. You have to understand it to write multi-threaded programs, however it is somehow dated (better use java.util.concurrent for anything thread/synchronisation related instead).
Anyhow - you need to know what it is about - read the related java tutorial part:
http://download.oracle.com/javase/tutorial/essential/concurrency/sync.html