My question is very simple, what is the best approach to work with Parse using the local store at the time I want to query the saved objects.
Is it better to trigger several queries to the local store directly on the main thread and avoid nesting a lot of anonymous classes or using a background thread?
It's important thing to notice is that this method is going to be called very frequently and the pattern will be repeated in several places with different queries. I'm evaluating both efficiency and code quality in readability. These methods will be called synchronously so we can assume the data will be consistent at any time.
As the objects are being saved locally I would expect the queries to be very fast in response. Here's a rough sample of how the code would look like in both cases.
Option one:
public void processBatches() {
ParseQuery<Batch> batchQuery = Batch.getQuery();
int batchCount = batchQuery.fromLocalDatastore().count();
List<Batch> batches = batchQuery.fromLocalDatastore().find();
for(Batch b : batches) {
// do whatever I need to do
}
}
Option two:
public void processBatches() {
ParseQuery<Batch> batchQuery = Batch.getQuery();
int batchCount = batchQuery.fromLocalDatastore().countInBackground(new CountCallback() {
#Override
public void done(int i, ParseException e) {
if (i > 0) {
batchQuery.findInBackground(new FindCallback<Batch>() {
#Override
public void done(List<Batch> list, ParseException e) {
for (Batch batch : list) {
// do whatever I need to do
}
}
});
}
}
});
}
Well since in option one you are blocking the UI thread, there could be a delay in the user's ability to interact with your application. This is not a very good option since even if it is for just a moment, users don't want to be waiting unless they know operations are happening. But, if you know that at any time there will be little to no delay, go ahead and do it.
Nevertheless, I argue that option two is going to be the best option. This is because, in general, all network operations should be performed in the background. Although in your case you are performing local datastore queries, suppose a user has gone to their application task manager and cleared the data (very rare this will happen) what happens now when you perform the find from local data store and processing of Batch objects? Well, the app crashes. Again, this is not a very good option for the usability for your application.
Choose the second option, and allow an AsyncThread to run the find() and count() query operations to the network if there is nothing found from local data store queries. Also, from the Parse documentation for find:
public Task<List<T>> findInBackground()
Retrieves a list of ParseObjects that satisfy this query from the source in a background thread.
This is preferable to using ParseQuery.find(), unless your code is already running in a background thread.
Returns:
A Task that will be resolved when the find has completed.
Parse's creators prefers that the users of their API use a background thread to perform operations.
It really depends.
Is the user triggering the update? If so then do it on the main thread because you don't want them waiting
If not, then is the data access a result of fetching data from the web (and hence you should already be on a background thread) so could probably just remain on the background thread
Also what happens in "// do whatever I need to do"? Is it an update to the UI or more background processing?
Related
I'm still fairly new to RxJava and I'm using it in an Android application. I've read a metric ton on the subject but still feel like I'm missing something.
I have the following scenario:
I have data stored in the system which is accessed via various service connections (AIDL) and I need to retrieve data from this system (1-n number of async calls can happen). Rx has helped me a ton in simplifying this code. However, this entire process tends to take a few seconds (upwards of 5 seconds+) therefore I need to cache this data to speed up the native app.
The requirements at this point are:
Initial subscription, the cache will be empty, therefore we have to wait the required time to load. No big deal. After that the data should be cached.
Subsequent loads should pull the data from cache, but then the data should be reloaded and the disk cache should be behind the scenes.
The Problem: I have two Observables - A and B. A contains the nested Observables that pull data from the local services (tons going on here). B is much simpler. B simply contains the code to pull the data from disk cache.
Need to solve:
a) Return a cached item (if cached) and continue to re-load the disk cache.
b) Cache is empty, load the data from system, cache it and return it. Subsequent calls go back to "a".
I've had a few folks recommend a few operations such as flatmap, merge and even subjects but for some reason I'm having trouble connecting the dots.
How can I do this?
Here are a couple options on how to do this. I'll try to explain them as best I can as I go along. This is napkin-code, and I'm using Java8-style lambda syntax because I'm lazy and it's prettier. :)
A subject, like AsyncSubject, would be perfect if you could keep these as instance states in memory, although it sounds like you need to store these to disk. However, I think this approach is worth mentioning just in case you are able to. Also, it's just a nifty technique to know.
AsyncSubject is an Observable that only emits the LAST value published to it (A Subject is both an Observer and an Observable), and will only start emitting after onCompleted has been called. Thus, anything that subscribes after that complete will receive the next value.
In this case, you could have (in an application class or other singleton instance at the app level):
public class MyApplication extends Application {
private final AsyncSubject<Foo> foo = AsyncSubject.create();
/** Asynchronously gets foo and stores it in the subject. */
public void fetchFooAsync() {
// Gets the observable that does all the heavy lifting.
// It should emit one item and then complete.
FooHelper.getTheFooObservable().subscribe(foo);
}
/** Provides the foo for any consumers who need a foo. */
public Observable<Foo> getFoo() {
return foo;
}
}
Deferring the Observable. Observable.defer lets you wait to create an Observable until it is subscribed to. You can use this to allow the disk cache fetch to run in the background, and then return the cached version or, if not in cache, make the real deal.
This version assumes that your getter code, both cache fetch and non- catch creation, are blocking calls, not observables, and the defer does work in the background. For example:
public Observable<Foo> getFoo() {
Observable.defer(() -> {
if (FooHelper.isFooCached()) {
return Observable.just(FooHelper.getFooFromCacheBlocking());
}
return Observable.just(FooHelper.createNewFooBlocking());
}).subscribeOn(Schedulers.io());
}
Use concatWith and take. Here we assume our method to get the Foo from the disk cache either emits a single item and completes or else just completes without emitting, if empty.
public Observable<Foo> getFoo() {
return FooHelper.getCachedFooObservable()
.concatWith(FooHelper.getRealFooObservable())
.take(1);
}
That method should only attempt to fetch the real deal if the cached observable finished empty.
Use amb or ambWith. This is probably one the craziest solutions, but fun to point out. amb basically takes a couple (or more with the overloads) observables and waits until one of them emits an item, then it completely discards the other observable and just takes the one that won the race. The only way this would be useful is if it's possible for the computation step of creating a new Foo to be faster than fetching it from disk. In that case, you could do something like this:
public Observable<Foo> getFoo() {
return Observable.amb(
FooHelper.getCachedFooObservable(),
FooHelper.getRealFooObservable());
}
I kinda prefer Option 3. As far as actually caching it, you could have something like this at one of the entry points (preferably before we're gonna need the Foo, since as you said this is a long-running operation) Later consumers should get the cached version as long as it has finished writing. Using an AsyncSubject here may help as well, to make sure we don't trigger the work multiple times while waiting for it to be written. The consumers would only get the completed result, but again, that only works if it can be reasonably kept around in memory.
if (!FooHelper.isFooCached()) {
getFoo()
.subscribeOn(Schedulers.io())
.subscribe((foo) -> FooHelper.cacheTheFoo(foo));
}
Note that, you should either keep around a single thread scheduler meant for disk writing (and reading) and use .observeOn(foo) after .subscribeOn(...), or otherwise synchronize access to the disk cache to prevent concurrency issues.
I’ve recently published a library on Github for Android and Java, called RxCache, which meets your needs about caching data using observables.
RxCache implements two caching layers -memory and disk, and it counts with several annotations in order to configure the behaviour of every provider.
It is highly recommended to use with Retrofit for data retrieved from http calls. Using lambda expression, you can formulate expression as follows:
rxCache.getUser(retrofit.getUser(id), () -> true).flatmap(user -> user);
I hope you will find it interesting :)
Take a look at the project below. This is my personal take on things and I have used this pattern in a number of apps.
https://github.com/zsiegel/rxandroid-architecture-sample
Take a look at the PersistenceService. Rather than hitting the database (or MockService in the example project) you could simply have a local list of users that are updated with the save() method and just return that in the get().
Let me know if you have any questions.
I am working on an Android application that uses greenDAO as a data persistence layer. The application downloads data from various different sources across multiple threads (determined by a thread pool), each piece of data is inserted into the database in a transaction using insertOrReplaceInTx. This is working fine.
My question is whether it is technically possible, using greenDAO, to encapsulate these different transactions (which occur on different threads) into an overall transaction, using nested transactions. I know in theory it is possible to do this if all the transactions were taking place on a single thread, however I am unsure if this possible with the insertOrReplaceInTx calls occurring on different threads.
The reason I wish to encapsulate these into a single overall transaction is because they represent a synchronisation process within an app. In the event of any single part of the import failing, I wish to abort and rollback all of the modifications within the overall transaction.
If I begin a transaction with db.beginTransaction on the main thread where I initiate the import process, this creates a deadlock when another thread tries to insertOrReplaceInTxt.
Is the correct way to counter this to ensure that all greenDAO transactions are taking place on the same thread?
Afaik, you cannot because each thread manages its own connection.
If you have such dependency between these operations, you probably want to sync them anyways.
e.g. what if Job A finishes way before Job B and Job B's db connection fails. Your data will go out of sync again. You still need some logic for the other job.
Also, writers are mutually exclusive.
I would suggest creating a utility class that can run a list of runnables in a transaction. Each job, when finished, enqueues a Runnable to this utility. These runnables include the actual database commands.
When the last one arrives (this depends on your dependency logic), the utility will run all runnables in a transaction.
A sample implementation may look like this: (I used a simple counter but you may need a more complex logic)
class DbBundle {
AtomicInteger mLatch;
List<Runnable> mRunnables = new ArrayList();
DbBundle(int numberOfTx) {
mLatch = new AtomicInteger(numberOfTx);
}
void cancel() {
mLatch.set(-1); // so decrement can never reach 0 in submit
}
boolean isCanceled() {
mLatch.count() < 0;
}
void submit(Runnable runnable) {
mRunnables.add(runnable);
if (mLatch.decrementAndGet() == 0) {
db.beginTransaction();
try {
for (Runnable r : mRunnables) r.run();
db.setTransactionSuccessful()
} finally {
db.endTransaction();
}
}
}
}
When you create each job, you pass this shared DbBundle and the last one will execute them all.
So a job would look like:
....
try {
if (!dbBundle.isCanceled()) { // avoid extra request if it is already canceled
final List<User> users = webservice.getUsers();
dbBundle.submit(new Runnable() {
void onRun() {
saveUsers(users);//which calls db. no transaction code.
});
});
} catch(Throwable t) {
dbBundle.cancel();
}
I have severals URLs I need to get data from, this should happen in order, one by one. The amount of data returned by requesting those URLs is relatively big. I need to be able to reschedule particular downloads which failed.
What is the best way to go? Shall I use IntentService, Loaders or something else?
Additional note: I would need not only to download, but also post process the data (create tables in db, fill it with data, etc). So DownloadManger can't be of help here.
I would use an IntentService.
It has a number of advantages that are suitable for your needs, including being able to download the data without your application running and supporting automatic restart of the service using setIntentRedelivery().
You can set a number of identifiers for the particular job, you need to perform using Intent extras, and you can keep track of the progress using SharedPreferences - that way you can also resume the work if it's been cancelled previously.
The easiest way is probably to use the system DownloadManager http://developer.android.com/reference/android/app/DownloadManager.html
(answering from my phone, so please excuse the lack of formatting)
I would suggest a service for this. Having service resolves many problems
It would allow reporting of progress asynchronously to the application so you can enable or disable a specific gui in application based on the download status of data
It will allow you to continue the download even if the user switches to other application or closes the application.
Will allow you to establish independent communication with server to prioritize downloads without user interaction.
Try a WakefulIntentService for creating a long-running job that uses wakelocks to keep your task alive and running https://github.com/commonsguy/cwac-wakeful .
Also, if your whole app process is getting killed, you may want to look into persisting the task queue to disk, using something like Tape, from Square
I think the way to go is loading urls in an array, then starting an AsyncTask, returning a boolean to onPostExecute indicating if the operation has success or not. then, keeping a global int index, you can run the AsyncTask with the next index if success, or the same index otherwise. Here is a pseudocode
private int index=0;
//this array must be loaded with urls
private ArrayList<String> urlsArray;
new MyDownloaderAsyncTask().execute(urlsArray.get(index));
class MyDownloaderAsyncTask extends AsyncTask<String,String,Boolean>{
#Override
doInBackground(String... input){
//downlaod my data is the function which download data and return a boolean
return downloadMyData();
}
#Override
onPostExecute(Boolean result){
if(result)
new MyDownloaderAsyncTask().execute(urlsArray.get(++index));
else
new MyDownloaderAsyncTask().execute(urlsArray.get(index));
}
}
hope this help
I have just completed an open source library that can do exactly what you need. Using droidQuery, you can do something like this:
$.ajax(new AjaxOptions().url("http://www.example.com")
.type("GET")
.dataType("JSON")
.context(this)
.success(new Function() {
#Override
public void invoke($ droidQuery, Object... params) {
//since dataType is JSON, params[0] is a JSONObject
JSONObject obj = (JSONObject) params[0];
//TODO handle data
//TODO start the next ajax task
}
})
.error(new Function() {
#Override
public void invoke($ droidQuery, Object... params) {
AjaxError error = params[0];
//TODO adjust error.options before retry:
$.ajax(error.request, error.options);
}
}));
You can specify other data types, which will return different object types, such as JSONObject, String, Document, etc.
Similar to #Murtuza Kabul I'd say use a service, but it's a little complicated than that. We have a similar situation related to constant internet access and updates, although ours places greater focus on keeping the service running. I'll try to highlight the main features without drowning you in too much detail (and code is owned by the company ;) )
android.permission.RECEIVE_BOOT_COMPLETED permission and a BroadcastReceiver listening for android.intent.action.BOOT_COMPLETED to poke the service awake.
Don't link the service to the Activity, you want it running all the time. eg we call context.startService(new Intent(context.getApplicationContext(), OurService.class))
The service class is just a simple class which registers and calls an OurServiceHandler (as in our case we fire off repeated checks and the Handler manages the 'ticks')
We have an OurServiceRunnable which is a singleton which is checked and called by the Handler for each test. It protects against overlapping updates. It delegates to an OurServiceWorker to do the actual lifting.
Sounds heavy handed, but you want to ensure that the service is always running, always ticking (via the Handler) but only running a single check at a time. You're also going to run into database issue if you use the standard SqlLite DbHelper paradigm, as you can't open the DB on multiple threads and you definitely want the internet access off the main thread. Our hack was a java.util.concurrent.locks.ReentrantLock protecting access to the DB, but you could probably keep DB access on the UI thread and pass DB operations via the Handler.
Beyond this it's just a matter of keeping the downloads atomic in terms of "get task, download task, complete task" or enabling it to pick up from a failed state eg downloaded OK, attempt to complete.
You should take a look at the volley library :
http://www.javacodegeeks.com/2013/06/android-volley-library-example.html
There is also an interesting video of the author that took place at google io 2013 :
http://www.youtube.com/watch?v=yhv8l9F44qo
Mainly because it eases the process of managing a lot of these fastidious tasks that are connection checking, connection interruption, queue management, retry, resume, etc.
Quoting from the javacodegeeks "Advantages of using Volley :
Volley automatically schedule all network requests. It means that Volley will be taking care of all the network requests your app executes for fetching response or image from web.
Volley provides transparent disk and memory caching.
Volley provides powerful cancellation request API. It means that you can cancel a single request or you can set blocks or scopes of requests to cancel.
Volley provides powerful customization abilities.
Volley provides Debugging and tracing tools"
Update from dennisdrew :
For large file, better use a variant of volley which authorize using another http client implementation. This link gives more details :
The volley article about this modification :
http://ogrelab.ikratko.com/android-volley-examples-samples-and-demos/
The github file detail :
https://github.com/ogrebgr/android_volley_examples/blob/master/src/com/github/volley_examples/toolbox/ExtHttpClientStack.java
public class FetchDataFromDBThread implements Runnable {
/*
* Defines the code to run for this task.
*/
#Override
public void run() {
// Moves the current Thread into the background
android.os.Process
.setThreadPriority(android.os.Process.THREAD_PRIORITY_BACKGROUND);
FetchDataFromDB();
}
}
I'm looking for a design pattern or approach for the following scenario. I wish to kick off two separate background threads for data retrieval from different sources. I then want one method (on the UI thread) to be called once both background threads have completed their work. As the data from the two sources must be combined to be useful, I must wait until both have finished retrieving before manipulating the data. How can I achieve this on the Android platform?
Edit: My first version has been bothering me, and I didn't like the necessary added boolean with it, so here's another version. Call it with this from onPostExecute of each added task.
ArrayList<AsyncTask> tasks;
public void doStuffWhenDone(AsyncTask finishedTask)
{
tasks.remove(finishedTask);
if(tasks.size() > 0)
return;
... do stuff
}
I'll keep the older one up also, since they both work, but I think the above is much cleaner. Now to go tidy up one of my earlier projects.
ArrayList<AsyncTask> tasks;
boolean hasBeenDone = false;
public void doStuffWhenDone()
{
for(int i=0;i<tasks.size();i++)
if(hasBeenDone || (tasks.get(i).getStatus() != AsyncTask.Status.FINISHED))
return;
hasBeenDone = true;
... do stuff
}
It's easily extendable to however many tasks you have, and there's no need for a thread to handle the threads. Just call the method at the end of each task. If it's not the last one done, nothing happens.
Edit: Good point, but I don't think it needs to be atomic. Since both AsyncTasks' onPostExecute methods run on the UI thread, they'll be called one after the other.
Use a CountDownLatch, like this:
CountDownLatch barrier = new CountDownLatch(2); // init with count=2
startWorkerThread1(barrier);
startWorkerThread2(barrier);
barrier.await(); // it will wait here until the count is zero
doStuffWithTheResult();
when a worker thread finishes, call barrier.countDown() from it.
You can use AsyncTask and an int to know if both jobs are finished...
I've been investigating alternative methods for saving my game's data between turns, and wonder if anyone can point me in the right direction.
I have approximately 32k of data which must be saved during onPause. I ruled out preferences due to the sheer quantity of data. I spent a few days playing around with SQLite but couldn't get the data to save in less than two seconds (although the time certainly hasn't been wasted).
I've decided that I'll use the database for loading constant data at the beginning of the game. This will certainly make it easier to tweak various parameters and default values in the game. But this still leaves me looking for the ideal method for writing data.
The data that needs to be saved is basically nine occurrences of class A and nine occurrences of class B. I'm an intensive month into the learning curve of Android (and the nuances of Java, coming from a C++ background) and have been googling like crazy. This brought two possibilities to mind -
1) Serialization (ObjectOutputStream)
I thought this would be the perfect solution but, having read several other posts regarding the subject, gather that it isn't highly recommended on the Android platform due to speed and memory allocations provoking the garbage collector into a potential rage.
2) DataOutputStream class
My current thought is to add Load and Save functions to both classes and to use DataOutputStream and DataInputStream calls in them to write and read the data respectively.
The data in the classes are primitives (strings and ints mostly) and arrays of primitives, so there's nothing too complicated in there to break down. Would this second solution seem a good, viable one? Or are there other solutions that I am unaware of as yet?
You should use an Async task to save the data, I used this method to fetch highscores at the start a game:
new HighscoreTask().execute(this);
the Async task looks like this:
public class HighscoreTask extends AsyncTask<MainView, Void, Void> {
protected void onPreExecute() {
}
protected void onPostExecute(final Void unused) {
}
#Override
protected Void doInBackground(MainView... params) {
HighScoreFactory.syncScores();
return null;
}
}
All the database interaction happens in HighScoreFactory.syncScores() this can take as long as it needs because it happens in the background. In my case it sends an HTTP request to an external server and loads these into a database. It's never caused any problems and works seamlessly.
Why do you have a 2 second limit on your database write? If it is purely for the sake of UI responsiveness, then there is another approach you can take.
You don't actually have to perform the save within your onPause method itself, you could just kick off a new Thread that actually does the save for you.
private void backgroundSave(){
Thread backgroundThread = new Thread() {
#Override
public void run() {
//do save here
}
};
backgroundThread.start();
}
#Override
protected void onPause() {
super.onPause();
backgroundSave();
}
You could alternatively use an AsyncTask for this.
You might have to consider the case when a user attempts to restart your app before the save is complete, but that shouldn't be too hard to take into account.
Have you tried insert data to the database in transaction?
try{
db.beginTransaction();
//here insert data to database
db.setTransactionSuccessful();
} finally {
db.endTranscation();
}
That can speed up operation.
Create a new Thread that does the data writing using Context.openFileOutput(String name, int mode) with this as the context. You can then write it in the background with the new thread and retrieve it with: Context.openFileInput(String name) again with this as the context. Hopefully this helps.