Kotlin/Android novice here :). I'm playing around with chunked uploads using a CoroutineWorker and don't see a built-in way to maintain state for my worker in case a retry happens, but I'm having sort of a hard time believing smth like that would be missing...
My use case is the following:
Create the worker request with the path to the file to upload as input data
Worker loops over the file and performs uploads in chunks. The latest uploaded chunkIndex is being tracked.
In case of an error and subsequent Retry(), the worker somehow retrieves the current chunk index and resumes rather than starting from at the beginning again.
So basically, I really just need to preserve that chunkIndex flag. I looked into setting progress, but this seems to be hit or miss on retries (worked once, wasn't available on another attempt).
override suspend fun doWork(): Result {
try {
// TODO check if we are resuming with a given chunk index
chunkIndex = ...
// do the work
performUpload(...)
return Result.success()
} catch (e: Exception) {
// TODO cache the chunk index
return Result.retry()
}
}
Did I overlook something, or would I really have to store that index outside the worker?
You have a pretty good use-case but unfortunately you cannot cache data within Worker class or pass on the data to the next Worker object on retry! As you suspected, you will have to store the index outside of the WorkManager provided constructs!
Long answer,
The Worker object can receive and return data. It can access the data from getInputData() method. If you chain tasks, the output of one worker can be input for the next-in-line worker. This can be done by returning Result.success(output) (see below code)
public Result doWork() {
int chunkIndex = upload();
//...set the output, and we're done!
Data output = new Data.Builder()
.putInt(KEY_RESULT, result)
.build();
return Result.success(output);
}
So the problem is we cannot return data for the retry case, only for failure and success case! (Result.retry(Data data) method is missing!)
Reference: official documentation and API.
As stated in GB's answer, there seems to be no way to cache data with in the worker, or do a Result.retry(data). I ended up just doing a quick hack with SharedPreferences instead.
Solution below. Take it with a grain of salt, I have a total of ~10 hours of Kotlin under my belt ;)
var latestChunkIndex = -1
override suspend fun doWork(): Result = withContext(Dispatchers.IO) {
try {
// get cached entry (simplified - no checking for fishy status or anything)
val transferId = id.toString()
var uploadInfo: UploadInfo = TransferCache.tryGetUpload(applicationContext, transferId) ?: TransferCache.registerUpload(applicationContext, transferId, TransferStatus.InProgress)
if(uploadInfo.status != TransferStatus.InProgress) {
TransferCache.setUploadStatus(applicationContext, transferId, TransferStatus.InProgress)
}
// resolve the current chunk - this will allow us to resume in case we're retrying
latestChunkIndex = uploadInfo.latestChunkIndex
// do the actual work
upload()
// update status and complete
TransferCache.setUploadStatus(applicationContext, id.toString(), TransferStatus.Success)
Result.success()
} catch (e: Exception) {
if (runAttemptCount > 20) {
// give up
TransferCache.setUploadStatus(applicationContext, id.toString(), TransferStatus.Error)
Result.failure()
}
// update status and schedule retry
TransferCache.setUploadStatus(applicationContext, id.toString(), TransferStatus.Paused)
Result.retry()
}
}
Within my upload function, I'm simply keeping track of my cache (I could also just do it in the exception handler of the doWork method, but I'll use the cache entry for status checks as well, and it's cheap):
private suspend fun upload() {
while ((latestChunkIndex + 1) * defaultChunkSize < fileSize) {
// doing the actual upload
...
// increment chunk number and store as progress
latestChunkIndex += 1
TransferCache.cacheUploadProgress(applicationContext, id.toString(), latestChunkIndex)
}
}
and the TransferCache looking like this (note that there is no housekeeping there, so without cleanup, this would just continue to grow!)
class UploadInfo() {
var transferId: String = ""
var status: TransferStatus = TransferStatus.Undefined
var latestChunkIndex: Int = -1
constructor(transferId: String) : this() {
this.transferId = transferId
}
}
object TransferCache {
private const val PREFERENCES_NAME = "${BuildConfig.APPLICATION_ID}.transfercache"
private val gson = Gson()
fun tryGetUpload(context: Context, transferId: String): UploadInfo? {
return getPreferences(context).tryGetUpload(transferId);
}
fun cacheUploadProgress(context: Context, transferId: String, transferredChunkIndex: Int): UploadInfo {
getPreferences(context).run {
// get or create entry, update and save
val uploadInfo = tryGetUpload(transferId)!!
uploadInfo.latestChunkIndex = transferredChunkIndex
return saveUpload(uploadInfo)
}
}
fun setUploadStatus(context: Context, transferId: String, status: TransferStatus): UploadInfo {
getPreferences(context).run {
val upload = tryGetUpload(transferId) ?: registerUpload(context, transferId, status)
if (upload.status != status) {
upload.status = status
saveUpload(upload)
}
return upload
}
}
/**
* Registers a new upload transfer. This would simply (and silently) override any
* existing registration.
*/
fun registerUpload(context: Context, transferId: String, status: TransferStatus): UploadInfo {
getPreferences(context).run {
val upload = UploadInfo(transferId).apply {
this.status = status
}
return saveUpload(upload)
}
}
private fun getPreferences(context: Context): SharedPreferences {
return context.getSharedPreferences(
PREFERENCES_NAME,
Context.MODE_PRIVATE
)
}
private fun SharedPreferences.tryGetUpload(transferId: String): UploadInfo? {
val data: String? = getString(transferId, null)
return if (data == null)
null
else
gson.fromJson(data, UploadInfo::class.java)
}
private fun SharedPreferences.saveUpload(uploadInfo: UploadInfo): UploadInfo {
val editor = edit()
editor.putString(uploadInfo.transferId, gson.toJson(uploadInfo))
editor.apply()
return uploadInfo;
}
}
Related
I am a newbie to android coroutines my requirements
Need to upload 20 images
Keep track of upload(at least when it gets finished I need to hide progressBar of each image)
After uploading all the images need to enable a "next" button also
Here is my try:
private fun startUploading(){
// Get AWS data
val accessKey = sharedPreferences.getString(getString(R.string.aws_access_key), "").toString()
val secretKey = sharedPreferences.getString(getString(R.string.aws_secret_key), "").toString()
val bucketName = sharedPreferences.getString(getString(R.string.aws_bucket_name), "").toString()
val region = sharedPreferences.getString(getString(R.string.aws_region), "").toString()
val distributionUrl = sharedPreferences.getString(getString(R.string.aws_distribution_url), "").toString()
var totalImagesNeedToUpload = 0
var totalImagesUploaded = 0
CoroutineScope(Dispatchers.IO).launch {
for (i in allCapturedImages.indices) {
val allImageFiles = allCapturedImages[i].viewItem.ImageFiles
totalImagesNeedToUpload += allImageFiles.size
for (j in allImageFiles.indices) {
CoroutineScope(Dispatchers.IO).launch {
while (true) {
val internetActive = utilsClassInstance.hasInternetConnected()
if (internetActive){
try {
val file = allImageFiles[j]
if (!file.uploaded) {
// Upload the file
val cfUrl = utilsClassInstance.uploadFile(file.imageFile, accessKey, secretKey, bucketName, region, distributionUrl)
// Set the uploaded status to true
file.uploaded = true
file.uploadedUrl = cfUrl
// Increment the count of total uploaded images
totalImagesUploaded += 1
// Upload is done for that particular set image
CoroutineScope(Dispatchers.Main).launch {
mainRecyclerAdapter?.uploadCompleteForViewItemImage(i, j, cfUrl)
// Set the next button enabled
if (totalImagesUploaded == totalImagesNeedToUpload){
binding.btnNext.isEnabled = true
}
}
break
}else{
totalImagesUploaded += 1
break
}
} catch (e: Exception) {
println(e.printStackTrace())
}
}
}
CoroutineScope(Dispatchers.Main).launch {
if (totalImagesUploaded == totalImagesNeedToUpload){
updateProgressForAllImages()
binding.btnNext.isEnabled = true
}
}
}
}
}
}
}
fun uploadFile(file: File, accessKey:String, secretKey:String, bucketName: String, region:String, distributionUrl: String): String{
// Create a S3 client
val s3Client = AmazonS3Client(BasicAWSCredentials(accessKey, secretKey))
s3Client.setRegion(Region.getRegion(region))
// Create a put object
val por = PutObjectRequest(bucketName, file.name, file)
s3Client.putObject(por)
// Override the response headers
val override = ResponseHeaderOverrides()
override.contentType = "image/jpeg"
// Generate the url request
val urlRequest = GeneratePresignedUrlRequest(bucketName, file.name)
urlRequest.responseHeaders = override
// Get the generated url
val url = s3Client.generatePresignedUrl(urlRequest)
return url.toString().replace("https://${bucketName}.s3.amazonaws.com/", distributionUrl)
}
There are total "n" images that I need to upload
every image is getting uploaded in different Coroutine because I need to do the parallel upload
The whole question is how to know that all the images are uploaded and enable a next button?
Your code seems very unstructured. You have an infinite loop checking for network availability. You have a nested loop here to upload images (Why?). You are creating a lot of coroutine scopes and have no control over them
Based on the 3 requirements that you mentioned in the question, you can do something like this:
val imagesToUpload: List<File> = /* ... */
var filesUploaded = 0
lifecycleScope.launchWhenStarted {
coroutineScope { // This will return only when all child coroutines have finished
imagesToUpload.forEach { imageFile ->
launch { // Run every upload in parallel
val url = utilsClassInstance.uploadFile(file.imageFile, ...) // Assuming this is a non-blocking suspend function.
filesUploaded++
// Pass the `url` to your adapter to display the image
binding.progressBar.progress = (filesUploaded * 100) / imagesToUpload.size // Update progress bar
}
}
}
// All images have been uploaded at this point.
binding.btnNext.enabled = true
}
Ideally you should have used a viewModelScope and the upload code should be in a repository, but since you don't seem to have a proper architecture in place, I have used lifecycleScope which you can get inside an Activity or Fragment
I want to call an api multiple times using WorkManager.
where idsArrayList is a list of ids.
I send each id in the api as Path to get response and similarly for other ids.
I want the workManager to return success after it has called api for all ids.
But the problem is WorkManager only returns SUCCESS for one id from the list. This is the first time I'm using WorkManager and I tried starting work manager for every id too by iterating over idsList one by one and making instance of workManger for every id in the for loop. But I thought sending the idsList as data in the workmanager and then itering over ids from inside doWork() would be better, but it's not working like I want and I don't understand why. Here's my code:
class MyWorkManager(appContext: Context, workerParams: WorkerParameters):
Worker(appContext, workerParams) {
private lateinit var callGrabShifts: Call<ConfirmStatus>
override fun doWork(): Result {
val idsList = inputData.getStringArray("IDS_LIST")
val idsArrayList = idsList?.toCollection(ArrayList())
var response = ""
if (idsArrayList != null) {
try {
response = callConfirmShiftApi(idsArrayList)
if (response.contains("CONFIRM")) {
return Result.success()
}
} catch (e: Exception) {
e.printStackTrace()
return Result.failure()
}
}
return Result.retry()
}
private fun callConfirmShiftApi(idsArrayList: ArrayList<String>): String {
var response = ""
for ((index, id) in idsArrayList.withIndex()) {
response = callApiForId(id)
if(index == idsArrayList.lastIndex) {
response = "CONFIRM"
}
}
return response
}
private fun callApiForId(id: String): String {
var shiftGrabStatus = ""
callGrabShifts = BaseApp.apiInterface.confirmGrabAllShifts(BaseApp.userId, id)
callGrabShifts.enqueue(object : Callback<ConfirmStatus> {
override fun onResponse(call: Call<ConfirmStatus>, response: Response<ConfirmStatus>) {
if (response.body() != null) {
shiftGrabStatus = response.body()!!.status
if (shiftGrabStatus != null) {
if (shiftGrabStatus.contains("CONFIRM")) {
val shiftNumber = ++BaseApp.noOfShiftsGrabbed
sendNotification(applicationContext)
shiftGrabStatus = "CONFIRM"
return
} else {
shiftGrabStatus = "NOT CONFIRM"
return
}
} else {
shiftGrabStatus = "NULL"
return
}
} else {
shiftGrabStatus = "NULL"
return
}
}
override fun onFailure(call: Call<ConfirmStatus>, t: Throwable) {
shiftGrabStatus = "FAILURE"
return
}
})
return shiftGrabStatus
}
}
And this is the code where I'm starting the WorkManager:
private fun confirmShiftApi(availableShiftsIdList: ArrayList<String>) {
val data = Data.Builder()
data.putStringArray("IDS_LIST", availableShiftsIdList.toArray(arrayOfNulls<String>(availableShiftsIdList.size)))
val oneTimeWorkRequest = OneTimeWorkRequestBuilder<MyWorkManager>().setInputData(data.build())
.build()
WorkManager.getInstance(applicationContext).enqueue(oneTimeWorkRequest)
WorkManager.getInstance(this).getWorkInfoByIdLiveData(oneTimeWorkRequest.id)
.observe(this, Observer { workInfo: WorkInfo? ->
if (workInfo != null && workInfo.state.isFinished) {
val progress = workInfo.progress
}
Log.d("TESTING", "(MainActivity) : observing work manager - workInfo?.state - ${workInfo?.state}")
})
}
Any suggestions what I might be doing wrong or any other alternative to perform the same? I chose workmanager basicaly to perform this task even when app is closed and for learning purposes as I haven't used WorkManager before. But would switch to other options if this doesn't work.
I tried the following things:
removed the 'var response line in every method that I'm using to set the response, though I added it temporarily just for debugging earlier but it was causing an issue.
I removed the check for "CONFIRM" in doWork() method and just made the api calls, removed the extra return lines.
I tried adding manual delay in between api calls for each id.
I removed the code where I'm sending the ids data from my activity before calling workmanager and made the api call to fetch those ids inside workmanager and added more delay in between those calls to that keep running in background to check for data one round completes(to call api for all ids that were fetched earlier, it had to call api again to check for more ids on repeat)
I removed the extra api calls from onRestart() and from other conditons that were required to call api again.
I tested only one round of api calls for all ids with delay and removed the repeated call part just to test first. Didn't work.
None of the above worked, it just removed extra lines of code.
This is my final code that is tested and It cleared my doubt. Though it didn't fix this issue as the problem was because of backend server and Apis were returning failure in onResponse callback for most ids(when calls are made repeatedly using a for loop for each id) except first id and randomly last id from the list sometimes(with delay) for the rest of the ids it didn't return CONFIRM status message from api using Workmanager. Adding delay didn't make much difference.
Here's my Workmanager code:
class MyWorkManager(appContext: Context, workerParams: WorkerParameters):
Worker(appContext, workerParams) {
private lateinit var callGrabShifts: Call<ConfirmStatus>
override fun doWork(): Result {
val idsList = inputData.getStringArray("IDS_LIST")
val idsArrayList = idsList?.toCollection(ArrayList())
if (idsArrayList != null) {
try {
response = callConfirmShiftApi(idsArrayList)
if (response.contains("CONFIRM")) {
return Result.success()
}
} catch (e: Exception) {
e.printStackTrace()
return Result.failure()
}
}
return Result.success()
}
private fun callConfirmShiftApi(idsArrayList: ArrayList<String>): String {
for ((index, id) in idsArrayList.withIndex()) {
response = callApiForId(id)
Thread.sleep(800)
if(index == idsArrayList.lastIndex) {
response = "CONFIRM"
}
}
return response
}
private fun callApiForId(id: String): String {
callGrabShifts = BaseApp.apiInterface.confirmGrabAllShifts(BaseApp.userId, id)
callGrabShifts.enqueue(object : Callback<ConfirmStatus> {
override fun onResponse(call: Call<ConfirmStatus>, response: Response<ConfirmStatus>) {
if (response.body() != null) {
shiftGrabStatus = response.body()!!.status
if (shiftGrabStatus != null) {
if (shiftGrabStatus.contains("CONFIRM")) {
return
} else {
return
}
} else {
return
}
} else {
return
}
}
override fun onFailure(call: Call<ConfirmStatus>, t: Throwable) {
return
}
})
return shiftGrabStatus
}
Eventually this problem(when an individual call is made for an id, it always returns success but when i call the api for every id using a loop, it only returns success for first call and failure for others) was solved using Service, it didn't have a complete success rate from apis either, but for 6/11 ids the api returned success(400ms delay between each api call), so it served the purpose for now.
var datasFromApi: ArrayList = arrayListOf()
btnNextCharacter.setOnClickListener(View.OnClickListener {
if(checkForInternetConnection()) {
try {
val innerClassObject = DownloadingCharacterTask()
innerClassObject.execute()
}catch (e: Exception){
e.printStackTrace()
}
}
}
inner class DownloadingCharacterTask : AsyncTask>() {
override fun doInBackground(vararg params: String?): List<Characters>? {
var parseCharacter = ParseCharacterUtility()
return parseCharacter.parseCharacterObjectFromJSONData()
}
override fun onPostExecute(result: List<Characters>?) {
super.onPostExecute(result)
var numberOfCharacters = result?.size ?: 0
var index: Int = 0
while (index < result!!.size) {
datasFromApi[index].name = result[index].name
datasFromApi[index].height = result[index].height
datasFromApi[index].mass = result[index].mass
datasFromApi[index].gender = result[index].gender
datasFromApi[index].birthYear = result[index].birthYear
}
}
}
Try this,
datasFromApi = innerClassObject.execute().get();
It can get a return value, but blocks the UIThread for that time.
That is, when get() is used, it is synchronous rather than asynchronous while getting the return value.
It's okay to use get() if there's no UI-related processing.
AsyncTasks uses 3 type parameters namely
private class DownloadFilesTask extends AsyncTask<URL, Integer, Long> {
}
From the docs:
Params, the type of the parameters sent to the task upon execution.
Progress, the type of the progress units published during the background computation.
Result, the type of the result of the background computation.
You can use the Result type to return some value and store it in some global variable.
Having said that, you should reconsider the use of AsyncTask (its a horrid thing from the past) and Android is about to ditch this monster soon.
Hope that helps.
I'm having a hard time making a call to my api. I'm using Reactivex with kotlin and Flowables. My API returns a list of items if the date I passed by the "If-Modified_since" header is less than the last update.
If there is no update I get as an app return android app a 304 error.
I need to do the following procedure.
1-> I make a call to the api
2-> If the call is successful, save the list in Realm and return to the viewmodel
3-> If the error is 304, I perform a cache search (Realm) of the items
4-> If it is another error, I return the error normally for the ViewModel
Here is the code below, but I'm not sure if it's that way.
override fun getTickets(eventId: String): Flowable<List<Ticket>> {
return factory
.retrieveRemoteDataStore()
.getTickets(eventId)
.map {
saveTickets(it)
it
}.onErrorResumeNext { t: Throwable ->
if (t is HttpException && t.response().code() == 304) {
factory.retrieveCacheDataStore().getTickets(eventId)
} else
//Should return error
}
The question is, what is the best way to do this?
Thank you.
I'm going to assume, that you're using Retrofit. If that's the case, then you could wrap your getTickets call in Single<Response<SomeModel>>. This way, on first map you can check the errorcode, something among the lines of:
...getTickets(id)
.map{ response ->
when {
response.isSuccessful && response.body!=null -> {
saveTickets(it)
it
}
!response.isSuccessful && response.errorCode() == 304 -> {
factory.retrieveCacheDataStore().getTickets(eventId)
}
else -> throw IOException()
}
}
This could of course be made pretty using standard/extension functions but wanted to keep it simple for readability purposes.
Hope this helps!
Most of my comments are my explanations.
data class Ticket(val id:Int) {
companion object {
fun toListFrom(jsonObject: JSONObject): TICKETS {
/**do your parsing of data transformation here */
return emptyList()
}
}
}
typealias TICKETS = List<Ticket>
class ExampleViewModel(): ViewModel() {
private var error: BehaviorSubject<Throwable> = BehaviorSubject.create()
private var tickets: BehaviorSubject<TICKETS> = BehaviorSubject.create()
/**public interfaces that your activity or fragment talk to*/
fun error(): Observable<Throwable> = this.error
fun tickets(): Observable<TICKETS> = this.tickets
fun start() {
fetch("http://api.something.com/v1/tickets/")
.subscribeOn(Schedulers.io())
.onErrorResumeNext { t: Throwable ->
if (t.message == "304") {
get(3)
} else {
this.error.onNext(t)
/** this makes the chain completed gracefuly without executing flatMap or any other operations*/
Observable.empty()
}
}
.flatMap(this::insertToRealm)
.subscribe(this.tickets)
}
private fun insertToRealm(tickets: TICKETS) : Observable<TICKETS> {
/**any logic here is mainly to help you save into Realm**/
/** I think realm has the option to ignore items that are already in the db*/
return Observable.empty()
}
private fun get(id: Int): Observable<TICKETS> {
/**any logic here is mainly to help you fetch from your cache**/
return Observable.empty()
}
private fun fetch(apiRoute: String): Observable<TICKETS> {
/**
* boilerplate code
wether you're using Retrofit or Okhttp, that's the logic you
should try to have
* */
val status: Int = 0
val rawResponse = ""
val error: Throwable? = null
val jsonResponse = JSONObject(rawResponse)
return Observable.defer {
if (status == 200) {
Observable.just(Ticket.toListFrom(jsonResponse))
}
else if (status == 304) {
Observable.error<TICKETS>(Throwable("304"))
}
else {
Observable.error<TICKETS>(error)
}
}
}
override fun onCleared() {
super.onCleared()
this.error = BehaviorSubject.create()
this.tickets = BehaviorSubject.create()
}
}
I have an array of URLs, each providing a zip file. I want to download them and store them in my app folders, inside the internal memory.
Question:
Since I do not know the number of URLs I will need to access, what is the best way to go about this? I am just beginning to work with Kotlin coroutines.
This is my 'download from url' method
fun downloadResourceArchiveFromUrl(urlString: String, context: Context): Boolean {
Timber.d("-> Started downloading resource archive.. $urlString")
lateinit var file: File
try {
val url = URL(urlString)
val urlConn = url.openConnection()
urlConn.readTimeout = 5000
urlConn.connectTimeout = 10000
val inputStream = urlConn.getInputStream()
val buffInStream = BufferedInputStream(inputStream, 1024 * 5)
val fileNameFromUrl = urlString.substringAfterLast("/")
file = File(context.getDir("resources", Context.MODE_PRIVATE) , fileNameFromUrl)
val outStream = FileOutputStream(file)
val buff = ByteArray(5 * 1024)
while (buffInStream.read(buff) != -1){
outStream.write(buff, 0, buffInStream.read(buff))
}
outStream.flush()
outStream.close()
buffInStream.close()
} catch (e: Exception) {
e.printStackTrace()
Timber.d("Download finished with exception: ${e.message} -<")
return false
}
Timber.d("Download finished -<")
return true
}
Could you simply create a loop and call download method each time?
for (i in resources.indices) {
asyncAwait {
downloadResourcesFromUrl(resources[i].url, context)
return#asyncAwait
}
Also, is it a good idea to do this synchronously? Wait for every file to download then proceed to the next one?
Turn your blocking download function into a suspending one:
suspend fun downloadResourceArchiveFromUrl(
urlString: String, context: Context
): Boolean = withContext(Dispatchers.IO) {
... your function body
}
Now run your loop inside a coroutine you launch:
myActivity.launch {
resources.forEach {
val success = downloadResourceArchiveFromUrl(it.url, context)
... react to success/failure ...
}
}
Also be sure to properly implement structured concurrency on your activity.