My app is making repeat API requests to download chunks of data. Unfortunately the server limits the API requests to 3 per second. In the code below, how can I rate limit the requests to X per second ?
private void getHistoricalPrices(String currency, String start_date, String end_date, int granularity){
// show download status
currentDlPageIndex++;
view.showDownloadingStatus(currentDlPageIndex, totalDlPages);
// make the API call
addDisposable(
model.isNetworkAvailable()
.doOnSuccess(isNetworkAvailable -> {
if (!isNetworkAvailable) {
showErrorMessage();
Timber.v("no internet");
}
})
.filter(isNetworkAvailable -> true)
.flatMapSingle(isNetworkAvailable -> model.getHistoricalPrices(currency, start_date, end_date, String.valueOf(granularity)))
.subscribeOn(rxSchedulers.io())
.observeOn(rxSchedulers.mainThread())
.subscribe((Response<List<List<String>>> responseData) -> {
if (responseData != null && responseData.code() == HTTP_OK) {
List<List<String>> response = responseData.body();
if (response != null) {
// create list from downloaded data
ArrayList<HistoricPrice> tmpHistoricPriceList = new ArrayList<>(response.size());
for (List<String> rawHistoricPrice : response) {
HistoricPrice historicPrice = new HistoricPrice(rawHistoricPrice.get(0), rawHistoricPrice.get(1), rawHistoricPrice.get(2), rawHistoricPrice.get(3), rawHistoricPrice.get(4), rawHistoricPrice.get(5));
tmpHistoricPriceList.add(0, historicPrice);
}
// add the downloaded list to the main list being recreated
model.historicPriceList.addAll(tmpHistoricPriceList);
Timber.d("added %d records to memory", response.size());
// if there's more to download, download another chunk
if (intermediateDateSecs != null && intermediateDateSecs < endDateSecs){
startDateSecs = tmpHistoricPriceList.get(tmpHistoricPriceList.size()-1).time + granularity;// add "granularity" to startDateSecs to avoid getting two exact data for the same time
Date startDate = new Date(startDateSecs * 1000L);//requires milliseconds, not epoch
String startStrDate = DateUtils.fromDateToString(startDate);
intermediateDateSecs = startDateSecs + ((ApiService.MAX_HISTORIC_RETURN_VALUES - 1) * granularity);
if (intermediateDateSecs > endDateSecs) intermediateDateSecs = endDateSecs;
Date intermediateDate = new Date(intermediateDateSecs * 1000L);
String intermediateStrDate = DateUtils.fromDateToString(intermediateDate);
getHistoricalPrices(currency, startStrDate, intermediateStrDate, granularity);
} else {
// no more to download, save data
Timber.d("downloaded total of %d records", model.historicPriceList.size());
view.hideDownloadingStatus();
showSaveDataMessage();
}
}
}
}, error -> {
showErrorMessage();
})
);
}
you can see that the method getHistoricalPrices() calls itself to continue downloading. This implementation works well, except for the server complaining when there's too many API requests per second.
You can perform your request every X milliseconds / seconds like so:
long interval = 0L; // Your desired interval here in milliseconds
Observable.interval(0, interval, TimeUnit.MILLISECONDS)
.takeUntil(x -> isTaskComplete == true)
// Other configuration calls go here
.subscribe(); // make request here or in doOnNext()
Related
I've problem with max repeat count for request that returns error. In my case I need to send a new request to backend, if previous was falling, but 3 time max. I'm using option to get result, that returns data or error:
fun sendRequest(): Single<Option<BeResponse>
Inside of sendRequest() function I transform it from Either<ApiError, Data> to BeResponse
data class BeReponse(
val error: ApiError? = null,
val data: Data? = null
)
When I'm trying to get data I need to send request if previous one failed. So I'm using:
sendRequest().flatMap{ option ->
option.map {
if (option.error != null) {
Single.just(null.toOption())
} else {
Single.just(it)
}
}.getOrElse{ sendRequest() }
}
But in case when I have error I will call sendRequest endless. How I can set max attempts count for it?
create temp var, then increase it when the program do looping..
See this
var temp = 0
sendRequest().flatMap{ option ->
option.map {
if (option.error != null) {
Single.just(null.toOption())
} else {
Single.just(it)
}
}.getOrElse{
if (temp < 3){
sendRequest()
temp = temp+1
} else{
//do your code to take action
}
}
I am trying to learn work manager chaining with passing output of one to another
Here are my objectives,
I have two work request WR1 (gets me the url) and WR2 (sends request to the url)
I can't and should not start the WR2 until the WR1 completes.
WR1 is supposed to return a url to which i have to send to WR2 as inputData
I can mostly do this without chaining. But i would to like explore it in the chaining.
Here is my snippet in progress. Please help.
WorkManager mWorkManager = WorkManager.getInstance(this);
//WR1
OneTimeWorkRequest urlRequest = new
OneTimeWorkRequest.Builder(UriResolveWorker.class).build();
//WR2
OneTimeWorkRequest pullRequest = new
OneTimeWorkRequest.Builder(PullReplicator.class).build();
btnStart.setOnClickListener(new View.OnClickListener() {
#Override
public void onClick(View v) {
mWorkManager.beginWith(urlRequest)
.then(pullRequest) // I should be able to pass the result of urlRequest.
.enqueue();
}
});
mWorkManager.getWorkInfoByIdLiveData(urlRequest.getId()).observe(this, new
Observer<WorkInfo>() {
#Override
public void onChanged(WorkInfo workInfo) {
if (workInfo != null) {
WorkInfo.State state = workInfo.getState();
// I will get the URL here and i want to pass this to WR2
message = workInfo.getOutputData().getString("work_result");
tvStatus.append("\n"+"state : "+state.toString() + "message : " +message + "\n");
}
}
});
mWorkManager.getWorkInfoByIdLiveData(pullRequest.getId()).observe(this, new Observer<WorkInfo>() {
#Override
public void onChanged(WorkInfo workInfo) {
if (workInfo != null) {
WorkInfo.State state = workInfo.getState();
String count = workInfo.getOutputData().getString("work_result");
tvStatus.append("\n"+"state : "+state.toString() + " No of Documents : " +count + "\n");o
}
}
});
Before returning the Result object in doWork() method of UriResolveWorker class, You can pass the url to Result.success().
First create object of Data.Builder() and then put the url into:
Data.Builder outputDataBuilder = Data.Builder();
outputDataBuilder.putString(KEY_URL_STRING, url.toString());
after that, create Data object with outputDataBuilder:
Data outputData = outputDataBuilder.build();
now you can return the Result with outputData :
return Result.success(outputData);
Workmanager sends the data to pullRequest when the first one has been done.
Please check the state of the WorkRequest before getting the data.
For example:
private final Observer<WorkInfo> urlWorkInfo = workInfo -> {
if (workInfo == null) {
return;
}
WorkInfo.State state = workInfo.getState();
if (state.isFinished()) {
String url = workInfo.getOutputData()
.getString(KEY_URL)
if (url != null) {
Log.d(TAG_MAIN_ACTIVITY, url);
} else {
Log.d(TAG_MAIN_ACTIVITY, "Url not found!");
}
} else if (state == WorkInfo.State.RUNNING) {
Log.d(TAG_MAIN_ACTIVITY, "Associated WorkRequest is being executed");
}
};
I know it's been a while since you asked this question, but maybe it might help someone in the feature:
In WR1 (Worker 1) you will have output with specific key - data that you want in your second worker (WR2):
Result.success(workDataOf(MY_TEST_KEY to someData.toString()))
Now if you want to get that data in WR2 you cannot send it trough setInputData() method when creating your OneTimeWorkRequest and chaining it.
As everything is forwarded from WR1 to WR2, in WR2 you have to fetch that data like:
inputData.getString(WR1.MY_TEST_KEY)
And than use it as you want in your WR2.
Hopefully this will help someone in the future, official documentation can be found here: https://developer.android.com/topic/libraries/architecture/workmanager/how-to/chain-work
I am trying to fetch data from server and sync it on the background of app. I did it using AsyncTask in an Intent Service class, but now I want to make network call using Retrofit. So, I fetched data from server using retrofit but while I am saving them on the local database the main thread freezes, only after completion of the process I can do something on the Main thread. Why is this happening?
I tried both synchronus and Asynchronus request of retrofit but the problem remains. This is what I have tried so far..
//calling company api in synchronus way
try {
val responseCompany = apiService.company(page, headers, bodyModel).execute()
Log.e("onResponse", "Company Response Code: ${responseCompany.code()}")
Log.e("onResponse", "Company Response Body: ${Gson().toJson(responseCompany.body())}")
if (responseCompany.isSuccessful) {
val response = responseCompany.body()
//delete company data
if (response?.delete?.data?.size!! > 0) {
for (i in response.delete.data.indices) {
val delete = response.delete.data[i]
Log.e(tag, "Delete Company data $i: ${delete.company_id}")
dbAdapter.open()
dbAdapter.Delete_COMPANY_NAME(delete.company_id)
dbAdapter.close()
}
}
//insert company data
if (response.insert.data.isNotEmpty()) {
for (i in response.insert.data.indices) {
val insert = response.insert.data[i]
Log.e(tag, "Insert company data $i: ${insert.company_id}")
dbAdapter.open()
dbAdapter.Insert_COMPANY_NAME(insert.company_id.toString(), insert.company_name)
dbAdapter.close()
}
}
//update company data
if (response.update.data.isNotEmpty()) {
for (i in response.update.data.indices) {
val update = response.update.data[i]
Log.e(tag, "Update Company data $i: ${update.company_id}")
dbAdapter.open()
dbAdapter.Update_COMPANY_NAME(update.company_id.toString(), update.company_name)
dbAdapter.close()
}
}
val totalPage = largest(response.delete.meta.pagination.total_pages, response.insert.meta.pagination.total_pages, response.update.meta.pagination.total_pages)
if (page < totalPage) {
prefManager.pageNumber = page + 1
bodyModel.date = lastAdUpdate2
bodyModel.limit = 500
companyData(bodyModel)
} else {
prefManager.T_COMPANY_NAME = currentTime
prefManager.PAGE_NUMBER = 1
prefManager.TOTAL_PAGE = 1
prefManager.pageNumber = 1
prefManager.FIRST = "1"
pagenumber = prefManager.PAGE_NUMBER
Handler().postDelayed({
bodyModel.limit = 100
bodyModel.date = lastAdUpdate3
generics(bodyModel)
}, 1000)
}
} else {
prefManager.dbUpdateStatus = false
Log.i("dataNotInsert", "data Not Insert")
}
} catch (e: Exception) {
Log.e("Exception", "Company: ${e.localizedMessage}")
e.printStackTrace()
}
N.B: I made network call (Retrofit request) in an Intent Service class..
Any Kind of help is highly appreciated. Thank you
This problem was actually solved by replacing Intent Service with Work Manager. And Using Kotlin Coroutine
And if you want to persist with Intent Service rather than using WorkManager. Just wrapping your network call with AsyncTask will solve the problem
I'm using Google Fit API to develop a fitness app.
I'm trying to use SessionApi with SessionReadRequest to read all the sessions in the past week created with my app. Here is the code I wrote:
/**
* Creates and executes a {#link SessionReadRequest} using {#link
* com.google.android.gms.fitness.SessionsClient} to get last week sessions data
*/
private Task<SessionReadResponse> readLastWeekSessions() {
loader.smoothToShow();
// Begin by creating the query.
SessionReadRequest readRequest = readLastWeekFitnessSessions();
// [START read_session]
// Invoke the Sessions API to fetch the session with the query and wait for the result
// of the read request. Note: Fitness.SessionsApi.readSession() requires the
// ACCESS_FINE_LOCATION permission.
return Fitness.getSessionsClient(getActivity(), GoogleSignIn.getLastSignedInAccount(getContext()))
.readSession(readRequest)
.addOnCompleteListener(new OnCompleteListener<SessionReadResponse>() {
#Override
public void onComplete(#NonNull Task<SessionReadResponse> task) {
Log.i(TAG, "Session read completed");
}
})
.addOnSuccessListener(new OnSuccessListener<SessionReadResponse>() {
#Override
public void onSuccess(SessionReadResponse sessionReadResponse) {
// Get a list of the sessions that match the criteria to check the result.
List<Session> sessions = sessionReadResponse.getSessions();
Log.i(TAG, "Session read was successful. Number of returned sessions is: "
+ sessions.size());
for (Session session : sessions) {
// Process the session
dumpSession(session);
// Process the data sets for this session
List<DataSet> dataSets = sessionReadResponse.getDataSet(session);
for (DataSet dataSet : dataSets) {
dumpDataSet(dataSet);
}
}
loader.smoothToHide();
}
})
.addOnFailureListener(new OnFailureListener() {
#Override
public void onFailure(#NonNull Exception e) {
Log.i(TAG, "Failed to read sessions");
}
});
// [END read_session]
}
/**
* Returns a {#link SessionReadRequest} for all the data of the last week sessions
*/
private SessionReadRequest readLastWeekFitnessSessions() {
Log.i(TAG, "Reading History API results for all the last week session");
// [START build_read_session_request]
Calendar cal = Calendar.getInstance();
Date now = new Date();
cal.setTime(now);
long endTime = cal.getTimeInMillis();
cal.add(Calendar.WEEK_OF_YEAR, -1);
long startTime = cal.getTimeInMillis();
// Build a session read request
SessionReadRequest readRequest = new SessionReadRequest.Builder()
.setTimeInterval(startTime, endTime, TimeUnit.MILLISECONDS)
.read(DataType.TYPE_CALORIES_EXPENDED)
.read(DataType.TYPE_DISTANCE_DELTA)
.read(DataType.TYPE_SPEED)
.enableServerQueries()
.build();
// [END build_read_session_request]
return readRequest;
}
It works when I have a small amount of data (only a few sessions). But if I want to read calories burned, total distance and average speed for the last month sessions (which is around 45 sessions I created for testing), it is loading infinitely (I waited more than 10 minutes wich is unacceptable for such a request).
Is it a normal behavior? Am I supposed to destroy regularly the old sessions? Am I supposed the use aggregate data? (which I don't know how to do...)
EDIT: I guess that the problem comes from the fact that I'm getting a huge amount of datapoint for the speed which is not what I want. I want to get the average speed but don't know how to do...
I do a huge polling when user login into my app. At the moment, we don't have much payload. But, the payloads are growing by the 100s daily at the moment. Which makes us have around 300 records per-user. And this, is the first week. So, we started optimizing for the next thousands. API has been well modified, but the Android app is shaky.
I have a request where a user have max of Two Territories for now, and I need to fetch all the Items in each Territory. So, I did this:
/**
* This is to get all user territories and for each and every territory, fetch all items there;
*/
private Observable<ItemListResponse> getAlltemIByTerritory() {
List<String> territories = PrefUtils.getUserTerritories(context);
return Observable.from(territories).flatMap(territory -> fetchAllTerritoryItemPaged(territory, 1));
}
/**
* This is to fetch all items in a passed territory. There's a per_page of 21.
*/
private Observable<ItemListResponse> fetchAllTerritoryItemPaged(String territory, int page) {
Log.e(TAG, "Each territory page: " + page);
return itemAPI.getIems("Bearer " + PrefUtils.getToken(context), page, 21, territory)
.flatMap(itemListResponse -> {
Meta meta = itemListResponse.getBakeResponse().getMeta();
Observable<ItemListResponse> thisPage = Observable.just(itemListResponse);
if (meta.getPage() != meta.getPageCount() && meta.getPageCount() > 0) {
Observable<ItemListResponse> nextPage = fetchAllTerritoryItemPaged(territory, page + 1);
return thisPage.concatWith(nextPage);
} else {
return thisPage;
}
});
}
Utilizing the Observable
public void getAlltemIByTerritory(APIRequestListener apiRequestListener) {
realm.executeTransaction(realm1 -> realm1.where(RealmItem.class).findAll().deleteAllFromRealm());
getAlltemIByTerritory()
.subscribeOn(Schedulers.io())
.observeOn(AndroidSchedulers.mainThread())
.subscribe(new Subscriber<ItemListResponse>() {
#Override
public void onCompleted() {
Log.e(TAG, "Completed Item");
apiRequestListener.didComplete(WhichSync.ITEM);
unsubscribe();
}
#Override
public void onError(Throwable e) {
e.printStackTrace();
handleError(e, apiRequestListener, WhichSync.ITEM);
unsubscribe();
}
#Override
public void onNext(ItemListResponse itemListResponse) {
Log.e(TAG, "In the handler next " + itemListResponse.toString());
realm.executeTransaction(realm1 -> {
for (Item itemData : itemListResponse.getItemResponse().getData()) {
RealmItem realmItem = realm1.where(RealmItem.class).equalTo("id", itemData.getId()).findFirst();
if (realmItem != null)
realmItem.deleteFromRealm();
realm1.copyToRealm(Item.copyBakeryToRealm(itemData));
}
});
apiRequestListener.onProgress(itemListResponse.getItemResponse().getMeta(), itemListResponse.getItemResponse().getData().size(), WhichSync.ITEM);
}
});
}
Log.e(TAG, "Each territory page: " + page); This line and this line Log.e(TAG, "In the handler next " + itemListResponse.toString()); reads to 23 for a 1780 record. With 21 records per_page. And then, the second line stop showing. Then, the view has hang. And then, when the first one is done, I then see the second spitting out logs afterwards.
This works fine, just that when the record is over 700 (The first try was with a 1780 record), there's a huge lag on the UI. And on mostly pre-lollipops it crashes with NoClassDefined Exception and sometimes it works. But, the lag is still always there.
Any help with optimizing the code, thanks.
Tail Recursion here:
/**
* This is to fetch all items in a passed territory. There's a per_page of 21.
*/
private Observable<ItemListResponse> fetchAllTerritoryItemPaged(String territory, int page) {
Log.e(TAG, "Each territory page: " + page);
return itemAPI.getIems("Bearer " + PrefUtils.getToken(context), page, 21, territory)
.flatMap(itemListResponse -> {
Meta meta = itemListResponse.getItemResponse().getMeta();
Observable<ItemListResponse> thisPage = Observable.just(itemListResponse);
if (meta.getPage() != meta.getPageCount() && meta.getPageCount() > 0) {
Observable<ItemListResponse> nextPage = fetchAllTerritoryItemPaged(territory, page + 1);
return thisPage.concatWith(nextPage);
} else {
return thisPage;
}
});
}
is the one causing the problem. Luckily, Rx provides us with BehaviorSubject.
So, I did something like this instead
private Observable<ItemListResponse> fetchAllTerritoryItemPaged(String territory, int page) {
BehaviorSubject<Integer> pageControl = BehaviorSubject.create(page);
Observable<ItemListResponse> ret = pageControl.asObservable().concatMap(integer -> {
if (integer > 0) {
return itemAPI.getIems("Bearer " + PrefUtils.getToken(context), integer, 21, territory)
.doOnNext(itemListResponse -> {
Meta meta = itemListResponse.getItemResponse().getMeta();
if (meta.getPage() != meta.getPageCount() && meta.getPageCount() > 0) {
pageControl.onNext(integer + 1);
} else {
pageControl.onNext(-1);
}
});
} else {
return Observable.<ItemListResponse>empty().doOnCompleted(pageControl::onCompleted);
}
});
return Observable.defer(() -> ret);
}
Which not only remove the lag, but also request comes in pretty fast