I have a method to ping all computers on a list called computerItemList and check if it's online or not. I want to do it by sending parallel requests to API. Code below simulates response from server and it is working but I need information when all requests finish unblocking ping button. How to do that?
Generally, I want to execute async command on all items on list and then wait until all finish.
public void pingComputers() {
for (ComputerItem computerItem : _computerItemList) {
Observable.just(computerItem)
.subscribeOn(Schedulers.io())
.subscribe(item -> {
int sleepTime = randomizer.nextInt(1000) + 200;
int status = randomizer.nextInt(ComputerStatus.values().length - 1) + 1;
Log.d(TAG, Thread.currentThread().getName() + " >>pingSelected: " + item + " sleep: " + sleepTime);
Thread.sleep(sleepTime);
item.setStatus(ComputerStatus.values()[status]);
updateComputerList();
});
}
}
UPDATE
I wrote something like this. Is it okay?
public void ping2() {
Observable.fromIterable(_computerItemList)
.flatMap(computerItem -> {
return Observable.just(computerItem)
.subscribeOn(Schedulers.io())
.map(item -> {
int sleepTime = randomizer.nextInt(1000) + 200;
Log.d(TAG, "ping2: Sleeping for " + sleepTime);
Thread.sleep(sleepTime);
int status = randomizer.nextInt(ComputerStatus.values().length - 1) + 1;
item.setStatus(ComputerStatus.values()[status]);
updateComputerList();
return item;
});
}).doOnComplete(() -> Log.d(TAG, "ping2: COMPLETE")).subscribe();
}
UPDATE - IT IS WORKING! But is it okay???
public void executeCommand(CommandType command) {
isWorking.setValue(true);
Observable.fromIterable(_computerItemList)
.subscribeOn(Schedulers.io())
.flatMap(computerItem -> createPingObservable(computerItem))
.observeOn(AndroidSchedulers.mainThread())
.doOnComplete(() -> isWorking.setValue(false))
.subscribe();
}
private Observable<ComputerItem> createPingObservable(ComputerItem computerItem) {
return Observable.just(computerItem)
.subscribeOn(Schedulers.io())
.map(item -> {
int sleepTime = randomizer.nextInt(1000) + 200;
int status = randomizer.nextInt(ComputerStatus.values().length - 1) + 1;
Log.d(TAG, Thread.currentThread().getName() + ">>> executeCommand: PING");
Thread.sleep(sleepTime);
item.setStatus(ComputerStatus.values()[status]);
updateComputerList();
return item;
});
}
UPDATE - IT IS WORKING! But is it okay???
No, it's not ok to call updateComputerList(); that way, you would have to synchronized it because you will be calling it from multiple threads, each of them trying to mutate the state.
I would do it this way:
Observable.fromIterable(list)
.flatMap(computerItem -> createPingObservable(computerItem))
.observeOn(AndroidSchedulers.mainThread())
.subscribe(computerItem -> updateComputerList(computerItem));
}
Also, don't use Thread.sleep(sleepTime);, just use
.delay(randomizer.nextInt(1000) + 200, TimeUnit.MILLISECONDS)
Related
My app is making repeat API requests to download chunks of data. Unfortunately the server limits the API requests to 3 per second. In the code below, how can I rate limit the requests to X per second ?
private void getHistoricalPrices(String currency, String start_date, String end_date, int granularity){
// show download status
currentDlPageIndex++;
view.showDownloadingStatus(currentDlPageIndex, totalDlPages);
// make the API call
addDisposable(
model.isNetworkAvailable()
.doOnSuccess(isNetworkAvailable -> {
if (!isNetworkAvailable) {
showErrorMessage();
Timber.v("no internet");
}
})
.filter(isNetworkAvailable -> true)
.flatMapSingle(isNetworkAvailable -> model.getHistoricalPrices(currency, start_date, end_date, String.valueOf(granularity)))
.subscribeOn(rxSchedulers.io())
.observeOn(rxSchedulers.mainThread())
.subscribe((Response<List<List<String>>> responseData) -> {
if (responseData != null && responseData.code() == HTTP_OK) {
List<List<String>> response = responseData.body();
if (response != null) {
// create list from downloaded data
ArrayList<HistoricPrice> tmpHistoricPriceList = new ArrayList<>(response.size());
for (List<String> rawHistoricPrice : response) {
HistoricPrice historicPrice = new HistoricPrice(rawHistoricPrice.get(0), rawHistoricPrice.get(1), rawHistoricPrice.get(2), rawHistoricPrice.get(3), rawHistoricPrice.get(4), rawHistoricPrice.get(5));
tmpHistoricPriceList.add(0, historicPrice);
}
// add the downloaded list to the main list being recreated
model.historicPriceList.addAll(tmpHistoricPriceList);
Timber.d("added %d records to memory", response.size());
// if there's more to download, download another chunk
if (intermediateDateSecs != null && intermediateDateSecs < endDateSecs){
startDateSecs = tmpHistoricPriceList.get(tmpHistoricPriceList.size()-1).time + granularity;// add "granularity" to startDateSecs to avoid getting two exact data for the same time
Date startDate = new Date(startDateSecs * 1000L);//requires milliseconds, not epoch
String startStrDate = DateUtils.fromDateToString(startDate);
intermediateDateSecs = startDateSecs + ((ApiService.MAX_HISTORIC_RETURN_VALUES - 1) * granularity);
if (intermediateDateSecs > endDateSecs) intermediateDateSecs = endDateSecs;
Date intermediateDate = new Date(intermediateDateSecs * 1000L);
String intermediateStrDate = DateUtils.fromDateToString(intermediateDate);
getHistoricalPrices(currency, startStrDate, intermediateStrDate, granularity);
} else {
// no more to download, save data
Timber.d("downloaded total of %d records", model.historicPriceList.size());
view.hideDownloadingStatus();
showSaveDataMessage();
}
}
}
}, error -> {
showErrorMessage();
})
);
}
you can see that the method getHistoricalPrices() calls itself to continue downloading. This implementation works well, except for the server complaining when there's too many API requests per second.
You can perform your request every X milliseconds / seconds like so:
long interval = 0L; // Your desired interval here in milliseconds
Observable.interval(0, interval, TimeUnit.MILLISECONDS)
.takeUntil(x -> isTaskComplete == true)
// Other configuration calls go here
.subscribe(); // make request here or in doOnNext()
To say that I have some code like this
Observable.concat(scores)
.observeOn(AndroidSchedulers.mainThread())
.subscribeOn(Schedulers.io())
.subscribe({...});
Due to the instability on the server side, an onError() notification (eg error code 500) may be poped out when emitting one of the Observables and the concat() operator will be stopped from emitting the rest of Observables as noted in the doc.
So, the failed Observable needs to be emitting again as well as the rest of Observables.
From my point of view, I try to use toBlocking() operator
to turn the sequence of Observables into a blocking Observable and forEach() it
List<Observable> list = createListOfScores();
Observable.from(list)
.toBlocking()
.forEach(new Action1<Observable>() {
#Override
public void call(Observable observable) {
observable.subscribe(...).onErrorResumeNext(...)
}
});
There will be better solution than this. I hope someone can enlighten me.
Another way could to use retry method if onError is called for some Observable. However, in this case, Observables that run without any errors would have to be removed from the list so that they're not run again. Here's an example (here I retry to run a task 10 times before giving up):
#RunWith(RobolectricTestRunner.class)
#Config(manifest = Config.NONE)
public class RxTest {
#Test
public void testRetryObservableConcat() throws Exception {
final List<Observable<String>> observables = new CopyOnWriteArrayList<>(getObservables());
Observable.concat(observables)
//remove the observable if it's successful
.doOnNext(result -> observables.remove(0))
.doOnError(error -> System.out.println(error.getMessage()))
//if an error is thrown then retry the task only for this Observable
.retry(10, error -> true)
.subscribeOn(Schedulers.io())
.observeOn(AndroidSchedulers.mainThread())
.subscribe(result -> System.out.println(result),
error -> System.out.println("onError called!"));
Thread.sleep(1000);
ShadowLooper.runUiThreadTasks();
}
private List<Observable<String>> getObservables() {
List<Observable<String>> observables = new ArrayList<>();
final Random random = new Random();
final AtomicInteger atomicInteger = new AtomicInteger(1);
for (int i = 0; i < 3; i++) {
final int id = i;
observables.add(Observable.fromCallable(() -> {
//for second Observable there's a 2/3 probability that it won't succeed
if (id == 1 && random.nextInt() % 3 != 0) {
throw new RuntimeException("Observable " + id + " failed!");
}
return "Observable #" + id + " returns " + atomicInteger.getAndIncrement();
}));
}
return observables;
}
}
Output
Observable 1 failed!
Observable 1 failed!
Observable 1 failed!
Observable #0 returns 1
Observable #1 returns 2
Observable #2 returns 3
As it can be seen from the example, after the second Observable finally succeeds each result is delivered in order.
I do a huge polling when user login into my app. At the moment, we don't have much payload. But, the payloads are growing by the 100s daily at the moment. Which makes us have around 300 records per-user. And this, is the first week. So, we started optimizing for the next thousands. API has been well modified, but the Android app is shaky.
I have a request where a user have max of Two Territories for now, and I need to fetch all the Items in each Territory. So, I did this:
/**
* This is to get all user territories and for each and every territory, fetch all items there;
*/
private Observable<ItemListResponse> getAlltemIByTerritory() {
List<String> territories = PrefUtils.getUserTerritories(context);
return Observable.from(territories).flatMap(territory -> fetchAllTerritoryItemPaged(territory, 1));
}
/**
* This is to fetch all items in a passed territory. There's a per_page of 21.
*/
private Observable<ItemListResponse> fetchAllTerritoryItemPaged(String territory, int page) {
Log.e(TAG, "Each territory page: " + page);
return itemAPI.getIems("Bearer " + PrefUtils.getToken(context), page, 21, territory)
.flatMap(itemListResponse -> {
Meta meta = itemListResponse.getBakeResponse().getMeta();
Observable<ItemListResponse> thisPage = Observable.just(itemListResponse);
if (meta.getPage() != meta.getPageCount() && meta.getPageCount() > 0) {
Observable<ItemListResponse> nextPage = fetchAllTerritoryItemPaged(territory, page + 1);
return thisPage.concatWith(nextPage);
} else {
return thisPage;
}
});
}
Utilizing the Observable
public void getAlltemIByTerritory(APIRequestListener apiRequestListener) {
realm.executeTransaction(realm1 -> realm1.where(RealmItem.class).findAll().deleteAllFromRealm());
getAlltemIByTerritory()
.subscribeOn(Schedulers.io())
.observeOn(AndroidSchedulers.mainThread())
.subscribe(new Subscriber<ItemListResponse>() {
#Override
public void onCompleted() {
Log.e(TAG, "Completed Item");
apiRequestListener.didComplete(WhichSync.ITEM);
unsubscribe();
}
#Override
public void onError(Throwable e) {
e.printStackTrace();
handleError(e, apiRequestListener, WhichSync.ITEM);
unsubscribe();
}
#Override
public void onNext(ItemListResponse itemListResponse) {
Log.e(TAG, "In the handler next " + itemListResponse.toString());
realm.executeTransaction(realm1 -> {
for (Item itemData : itemListResponse.getItemResponse().getData()) {
RealmItem realmItem = realm1.where(RealmItem.class).equalTo("id", itemData.getId()).findFirst();
if (realmItem != null)
realmItem.deleteFromRealm();
realm1.copyToRealm(Item.copyBakeryToRealm(itemData));
}
});
apiRequestListener.onProgress(itemListResponse.getItemResponse().getMeta(), itemListResponse.getItemResponse().getData().size(), WhichSync.ITEM);
}
});
}
Log.e(TAG, "Each territory page: " + page); This line and this line Log.e(TAG, "In the handler next " + itemListResponse.toString()); reads to 23 for a 1780 record. With 21 records per_page. And then, the second line stop showing. Then, the view has hang. And then, when the first one is done, I then see the second spitting out logs afterwards.
This works fine, just that when the record is over 700 (The first try was with a 1780 record), there's a huge lag on the UI. And on mostly pre-lollipops it crashes with NoClassDefined Exception and sometimes it works. But, the lag is still always there.
Any help with optimizing the code, thanks.
Tail Recursion here:
/**
* This is to fetch all items in a passed territory. There's a per_page of 21.
*/
private Observable<ItemListResponse> fetchAllTerritoryItemPaged(String territory, int page) {
Log.e(TAG, "Each territory page: " + page);
return itemAPI.getIems("Bearer " + PrefUtils.getToken(context), page, 21, territory)
.flatMap(itemListResponse -> {
Meta meta = itemListResponse.getItemResponse().getMeta();
Observable<ItemListResponse> thisPage = Observable.just(itemListResponse);
if (meta.getPage() != meta.getPageCount() && meta.getPageCount() > 0) {
Observable<ItemListResponse> nextPage = fetchAllTerritoryItemPaged(territory, page + 1);
return thisPage.concatWith(nextPage);
} else {
return thisPage;
}
});
}
is the one causing the problem. Luckily, Rx provides us with BehaviorSubject.
So, I did something like this instead
private Observable<ItemListResponse> fetchAllTerritoryItemPaged(String territory, int page) {
BehaviorSubject<Integer> pageControl = BehaviorSubject.create(page);
Observable<ItemListResponse> ret = pageControl.asObservable().concatMap(integer -> {
if (integer > 0) {
return itemAPI.getIems("Bearer " + PrefUtils.getToken(context), integer, 21, territory)
.doOnNext(itemListResponse -> {
Meta meta = itemListResponse.getItemResponse().getMeta();
if (meta.getPage() != meta.getPageCount() && meta.getPageCount() > 0) {
pageControl.onNext(integer + 1);
} else {
pageControl.onNext(-1);
}
});
} else {
return Observable.<ItemListResponse>empty().doOnCompleted(pageControl::onCompleted);
}
});
return Observable.defer(() -> ret);
}
Which not only remove the lag, but also request comes in pretty fast
I have two observables(A, B), and I want the first to finish running before the second runs. But, that's not even the problem I'm having. The problem is that, when A is added before B, B doesn't run at all unless I place B before A then, the two runs. But, the scenario I'm in is like thus:
A - Pickup
B - Delivery
There are three types of orders. Pickup Only, Delivery Only and Pickup And Delivery. Pickups need to run before Deliveries in every situation. A Delivery only already have Pickup marked as true. A Pickup only, needs to be picked up and delivered on it being closed. Which is why I need Pickup to send all locally saved pickups first before sending deliveries. So, I did this:
Pickup
private Observable<UpdateMainResponse> getDeliveredOrders() {
String token = PrefUtil.getToken(context);
BehaviorSubject<Integer> pageControl = BehaviorSubject.create(1);
Observable<UpdateMainResponse> ret = pageControl.asObservable().concatMap(integer -> {
if (integer - 1 != deliveryUpdate.size()) {
Log.e(TAG, "DeliveredOrders: " + deliveryUpdate.size());
RealmOrderUpdate theDel = deliveryUpdate.get(integer-1);
Log.e(TAG, "DeliveryUpdate: " + theDel.toString());
DeliverOrder pickupOrder = new DeliverOrder();
pickupOrder.setUuid(theDel.getUuid());
pickupOrder.setCode(theDel.getDest_code());
pickupOrder.setDelivered_lat(theDel.getLoc_lat());
pickupOrder.setDelivered_long(theDel.getLoc_long());
return apiService.deliverOrder(theDel.getOrderId(), token, pickupOrder)
.subscribeOn(Schedulers.immediate())
.doOnNext(updateMainResponse -> {
try {
Log.e(TAG, updateMainResponse.toString());
realm.executeTransaction(realm1 -> theDel.deleteFromRealm());
} catch (Exception e) {
e.printStackTrace();
} finally {
pageControl.onNext(integer + 1);
}
});
} else {
return Observable.<UpdateMainResponse>empty().doOnCompleted(pageControl::onCompleted);
}
});
return Observable.defer(() -> ret);
}
Delivery
private Observable<UpdateMainResponse> getPickedOrders() {
Log.e(TAG, "PickedOrders: " + pickUpdate.size());
String token = PrefUtil.getToken(context);
BehaviorSubject<Integer> pageControl = BehaviorSubject.create(1);
Observable<UpdateMainResponse> ret = pageControl.asObservable().concatMap(integer -> {
Log.e(TAG, "MainPickedInteger: " + integer);
if (integer - 1 != pickUpdate.size()) {
RealmOrderUpdate thePick = pickUpdate.get(integer - 1);
Log.e(TAG, "PickedUpdate: " + thePick.toString());
PickupOrder pickupOrder = new PickupOrder();
pickupOrder.setUuid(thePick.getUuid());
pickupOrder.setCode(thePick.getSource_code());
pickupOrder.setPicked_lat(thePick.getLoc_lat());
pickupOrder.setPicked_long(thePick.getLoc_long());
return apiService.pickupOrder(thePick.getOrderId(), token, pickupOrder)
.subscribeOn(Schedulers.immediate())
.doOnNext(updateMainResponse -> {
try {
Log.e(TAG, updateMainResponse.toString());
realm.executeTransaction(realm1 -> thePick.deleteFromRealm());
} catch (Exception e) {
e.printStackTrace();
} finally {
pageControl.onNext(integer + 1);
}
});
} else {
return Observable.<UpdateMainResponse>empty().doOnCompleted(pageControl::onCompleted);
}
});
return Observable.defer(() -> ret);
}
Zipper
private Observable<ZipperResponse> batchedZip() {
return Observable.zip(getPickedOrders(), getDeliveredOrders(), (updateMainResponse, updateMainResponse2) -> {
List<UpdateMainResponse> orders = new ArrayList<>();
bakeries.add(updateMainResponse);
bakeries.add(updateMainResponse2);
return new ZipperResponse(orders);
});
}
Utilizing Zipper
public void generalUpload(APIRequestListener listener) {
batchedZip.subscribe(new Subscriber<ZipperResponse>() {
#Override
public void onCompleted() {
listener.didComplete();
unsubscribe();
}
#Override
public void onError(Throwable e) {
listener.handleDefaultError(e);
unsubscribe();
}
#Override
public void onNext(ZipperResponse zipperResponse) {
Log.e(TAG, zipperResponse.size());
}
});
}
Problem
I don't know why getDeliveredOrders() doesn't get called unless I move it to the first before getPickedOrders()
Reading through Rx Documentation for Zip I can see that it's not going to work as I expected where all of getPickedOrders() runs first before getDeliveredOrders() runs. It'll have to do it one by one. E.g: One of Pickup and then One of Delivery
Any help to understand what's going on would be appreciated. Thanks
Ok, so if I got that right:
Pickup only: need to run through the Pickup process, then they complete.
Delivery only: need to run through the Delivery process, then they complete.
Pickup and Delivery: need to run through Pickup first, then through Delivery.
On a very high level, almost preudo-code, why does this process not work?
Observable<Item> performPickup(Item item);
Observable<Item> performDelivery(Item item);
Observable<Items> items = ...;
items
.flatMap(item -> item.needsPickup() ? performPickup(item) : Observable.just(item))
.flatMap(item -> item.needsDelivery() ? performDelivery(item) : Observable.just(item))
.doOnNext(completedItem -> ...)
If you have different sources for the three types:
Observable<Item> items = Observable.merge(
pickupSource(),
deliverySource(),
pickupAndDeliverySource());
I have to download a Json with a list of files, and then parallel download the files in the list. I would like to update periodically the ProgressDialog, so I implemented in this way
I create and show the dialog
I start an AsyncTask
onProgressUpdate receives 2 Integers, current progress and max progress, and updates the progress bar
doInBackground
downloads the json file and obtains the list of files to download
creates a ThreadPoolExecutor (tpe), with a LinkedBlockingQueue<Runnable>
submit a runnable for each file, that download the file to disk using Apache commons-io FileUtils.copyURLToFile
exec shutdown
in a while cycle. tpe.awaitTermination(1, TimeUnit.SECONDS) invokes periodically publishProgress( (int) tpe.getCompletedTaskCount(), tot), to update the progress bar
onPostExecute hides and dismisses the progres bar, and manages the files downloades
is there any problem in using ThreadPoolExecutor inside an AsynTask?
I am discussing with a colleague who claims that there could be problems in the threads management, that could deadlock, and that might give us problems on future versions
that's the code
public static void syncFiles(...)
{
PowerManager pm = (PowerManager) context.getSystemService(Context.POWER_SERVICE);
sWakelock = pm.newWakeLock(PowerManager.PARTIAL_WAKE_LOCK, TAG);
sWakelock.acquire();
sProgress = new ProgressDialog(context);
sProgress.setCancelable(false);
sProgress.setTitle("MyTitle");
sProgress.setMessage("Sincronizzazione in corso");
sProgress.setProgressStyle(ProgressDialog.STYLE_HORIZONTAL);
sProgress.setIndeterminate(false);
sProgress.show();
sCurrentTask = new AsyncTask<Void, Integer, Manifest>()
{
#Override
protected void onCancelled()
{
if ((sProgress != null) && sProgress.isShowing())
sProgress.dismiss();
if ((sWakelock != null) && sWakelock.isHeld())
sWakelock.release();
};
#Override
protected Manifest doInBackground(Void... params)
{
ArrayList files = getFiles(....)// download the jsonfile, and return the list of files
final String baseurl = ... // get the remote base url
final String baselocal = ... //get the local base path ;
int tot = m.size();
publishProgress(0, tot);
final int MAX_THREADS = Runtime.getRuntime().availableProcessors(); * 4;
ThreadPoolExecutor tpe = new ThreadPoolExecutor(
MAX_THREADS,
MAX_THREADS,
1,
TimeUnit.MINUTES,
new LinkedBlockingQueue<Runnable>()
);
for (final String s: files)
{
tpe.submit(new Runnable()
{
#Override
public void run()
{
try
{
URL remoteUrl = new URL(baseurl + s);
File localUrl = new File(baselocal, s);
FileUtils.copyURLToFile(remoteUrl, localUrl, 60000, 60000);
Log.w(TAG, "Downloaded " + localUrl.getAbsolutePath() + " in " + remoteUrl);
} catch (Exception e)
{
e.printStackTrace();
Log.e(TAG, "download error " + e);
// error management logic
}
}
});
}
tpe.shutdown();
int num = 0;
publishProgress(num, tot);
try
{
while (!tpe.awaitTermination(1, TimeUnit.SECONDS))
{
int n = (int) tpe.getCompletedTaskCount();
Log.w(TAG, "COUTN: " + n + "/" + tot);
if (n != num)
{
num = n;
publishProgress(num, tot);
}
}
} catch (InterruptedException e)
{
// TODO Auto-generated catch block
e.printStackTrace();
}
return m;
}
protected void onProgressUpdate(Integer... prog)
{
if (sProgress.getMax() != prog[1]) {
sProgress.setMax(prog[1]);
}
sProgress.setProgress(prog[0]);
}
#Override
protected void onPostExecute(Manifest result)
{
sWakelock.release();
sProgress.hide();
sProgress.dismiss();
// manage results
}
}.execute();
}
If you'll checkout the implementation of AsyncTask then youi can find that AsyncTask itself has ThreadPool so it will start the task on separate thread. Acutually when we can the .execute() to start the background task this method is typically used with THREAD_POOL_EXECUTOR to allow multiple tasks to run in parallel on a pool of threads managed by AsyncTask. So why you need to implement another.
Update
Read about executeOnExecutor in this may be this can help you... It clearly says that if you are allowing multiple tasks to run in parallel from a thread pool is generally not what one wants, because the order of their operation is not defined....but here you want to download the files so I don't think the order is important so in my view you can use it and it'll not create any issue.