Develop Reference

Write to function, is called multi time in request body

I have a progreesBar for uploading with retrofit and I implementation that with some of examples.
my problem is 'WriteTo' function in my custom requestBody class.
This function send progress value for use in my progressBar but this function is called twice. I used debugger and I think some interceptors call WriteTo function.
Let me explain the problem more clearly,When I click Upload button, The number of progress bar reaches one hundred and then it starts again from zero.
Some of the things I did:
I removed HttpLoggingInterceptor.
I used a boolean variable for check that 'writeTo' don't post anything the first time
I don't have any extra interceptors
Also I read this topics:
Retrofit 2 RequestBody writeTo() method called twice
using Retrofit2/okhttp3 upload file,the upload action always performs twice,one fast ,and other slow
Interceptor Problem
My codes:
ProgressRequestBody class
class ProgressRequestBody : RequestBody() {
var mutableLiveData = MutableLiveData<Int>()
lateinit var mFile: File
lateinit var contentType: String
companion object {
private const val DEFAULT_BUFFER_SIZE = 2048
}
override fun contentType(): MediaType? {
return "$contentType/*".toMediaTypeOrNull()
}
#Throws(IOException::class)
override fun contentLength(): Long {
return mFile.length()
}
#Throws(IOException::class)
override fun writeTo(sink: BufferedSink) {
val fileLength = mFile.length()
val buffer = ByteArray(DEFAULT_BUFFER_SIZE)
val `in` = FileInputStream(mFile)
var uploaded: Long = 0
`in`.use { `in` ->
var read: Int
while (`in`.read(buffer).also { read = it } != -1) {
val percentage = (100 * uploaded / fileLength).toInt()
mutableLiveData.postValue(percentage)
uploaded += read.toLong()
sink.write(buffer, 0, read)
}
}
}
}
private fun upload(file: File, fileType: FileType) {
val fileBody = ProgressRequestBody()
fileBody.mFile = file
fileBody.contentType = file.name
uploadImageJob = viewModelScope.launch(Dispatchers.IO) {
val body = MultipartBody.Part.createFormData("File", file.name, fileBody)
fileUploadRepo.upload(body).catch {
// ...
}.collect {
when (it) {
// ...
}
}
}
}
In my fragment I use liveData for collect progressBar progress value.

Android download image with Workmanager and show it in gallery

I wrote code that downloads some source of data from the internet (in this example picture), shows downloadPercentages while the process of downloading is ongoing and writes this file on android storage. works well and looks very nice everything except saving on android storage.
Code is written in 3 classes, but I will show only one that I think is relevant (DownloadWorker). If anyone thinks other classes might help, let me now in comment and I will add them.
DownloadWorker:
class DownloadWorker(val context: Context, params: WorkerParameters) : Worker(context, params) {
companion object {
const val FILE_NAME = "image.jpg"
}
override fun doWork(): Result {
try {
if (DownloadHelper.url == null) {
DownloadHelper.downloadState.postValue(DownloadState.Failure)
return Result.failure()
}
DownloadHelper.url?.apply {
if(!startsWith("https")) {
DownloadHelper.url = replace("http", "https")
}
}
val url = URL(DownloadHelper.url)
val connection = url.openConnection()
connection.connect()
val fileSize = connection.contentLength
val inputStream = connection.getInputStream()
val buffer = ByteArray(1024)
val file = context.getExternalFilesDir(Environment.DIRECTORY_DOWNLOADS)
val outputFile = File(file, FILE_NAME)
var len = 0
var total = 0L
val fos = FileOutputStream(outputFile)
len = inputStream.read(buffer)
while (len != -1) {
fos.write(buffer, 0, len)
total += len
val percent = ((total * 100) / fileSize).toInt()
DownloadHelper.downloadState.postValue(DownloadState.InProgress(percent))
len = inputStream.read(buffer)
}
fos.close()
inputStream.close()
DownloadHelper.downloadState.postValue(DownloadState.Success(outputFile))
} catch (e: Exception) {
DownloadHelper.downloadState.postValue(DownloadState.Failure)
return Result.failure()
}
return Result.success()
}
}
After download success, my image is not shown in gallery, or in downloaded files folder. To see this image you need to enter android storage, find in android data app package by name and navigate all the way to the image. Pretty complicated.
Can anyone help, thanks.

Using nested CoroutineScopes to upload images and keeping track of them

I am a newbie to android coroutines my requirements
Need to upload 20 images
Keep track of upload(at least when it gets finished I need to hide progressBar of each image)
After uploading all the images need to enable a "next" button also
Here is my try:
private fun startUploading(){
// Get AWS data
val accessKey = sharedPreferences.getString(getString(R.string.aws_access_key), "").toString()
val secretKey = sharedPreferences.getString(getString(R.string.aws_secret_key), "").toString()
val bucketName = sharedPreferences.getString(getString(R.string.aws_bucket_name), "").toString()
val region = sharedPreferences.getString(getString(R.string.aws_region), "").toString()
val distributionUrl = sharedPreferences.getString(getString(R.string.aws_distribution_url), "").toString()
var totalImagesNeedToUpload = 0
var totalImagesUploaded = 0
CoroutineScope(Dispatchers.IO).launch {
for (i in allCapturedImages.indices) {
val allImageFiles = allCapturedImages[i].viewItem.ImageFiles
totalImagesNeedToUpload += allImageFiles.size
for (j in allImageFiles.indices) {
CoroutineScope(Dispatchers.IO).launch {
while (true) {
val internetActive = utilsClassInstance.hasInternetConnected()
if (internetActive){
try {
val file = allImageFiles[j]
if (!file.uploaded) {
// Upload the file
val cfUrl = utilsClassInstance.uploadFile(file.imageFile, accessKey, secretKey, bucketName, region, distributionUrl)
// Set the uploaded status to true
file.uploaded = true
file.uploadedUrl = cfUrl
// Increment the count of total uploaded images
totalImagesUploaded += 1
// Upload is done for that particular set image
CoroutineScope(Dispatchers.Main).launch {
mainRecyclerAdapter?.uploadCompleteForViewItemImage(i, j, cfUrl)
// Set the next button enabled
if (totalImagesUploaded == totalImagesNeedToUpload){
binding.btnNext.isEnabled = true
}
}
break
}else{
totalImagesUploaded += 1
break
}
} catch (e: Exception) {
println(e.printStackTrace())
}
}
}
CoroutineScope(Dispatchers.Main).launch {
if (totalImagesUploaded == totalImagesNeedToUpload){
updateProgressForAllImages()
binding.btnNext.isEnabled = true
}
}
}
}
}
}
}
fun uploadFile(file: File, accessKey:String, secretKey:String, bucketName: String, region:String, distributionUrl: String): String{
// Create a S3 client
val s3Client = AmazonS3Client(BasicAWSCredentials(accessKey, secretKey))
s3Client.setRegion(Region.getRegion(region))
// Create a put object
val por = PutObjectRequest(bucketName, file.name, file)
s3Client.putObject(por)
// Override the response headers
val override = ResponseHeaderOverrides()
override.contentType = "image/jpeg"
// Generate the url request
val urlRequest = GeneratePresignedUrlRequest(bucketName, file.name)
urlRequest.responseHeaders = override
// Get the generated url
val url = s3Client.generatePresignedUrl(urlRequest)
return url.toString().replace("https://${bucketName}.s3.amazonaws.com/", distributionUrl)
}
There are total "n" images that I need to upload
every image is getting uploaded in different Coroutine because I need to do the parallel upload
The whole question is how to know that all the images are uploaded and enable a next button?

Your code seems very unstructured. You have an infinite loop checking for network availability. You have a nested loop here to upload images (Why?). You are creating a lot of coroutine scopes and have no control over them
Based on the 3 requirements that you mentioned in the question, you can do something like this:
val imagesToUpload: List<File> = /* ... */
var filesUploaded = 0
lifecycleScope.launchWhenStarted {
coroutineScope { // This will return only when all child coroutines have finished
imagesToUpload.forEach { imageFile ->
launch { // Run every upload in parallel
val url = utilsClassInstance.uploadFile(file.imageFile, ...) // Assuming this is a non-blocking suspend function.
filesUploaded++
// Pass the `url` to your adapter to display the image
binding.progressBar.progress = (filesUploaded * 100) / imagesToUpload.size // Update progress bar
}
}
}
// All images have been uploaded at this point.
binding.btnNext.enabled = true
}
Ideally you should have used a viewModelScope and the upload code should be in a repository, but since you don't seem to have a proper architecture in place, I have used lifecycleScope which you can get inside an Activity or Fragment

Download images from a URL, save them to App Internal Storage without blocking calls (multiple files in parallel). Using Kotlin Coroutines on Android

Basically, I am trying to download three different images(bitmaps) from a URL and save them to Apps Internal storage, and then use the URI's from the saved file to save a new Entity to my database. I am having a lot of issues with running this in parallel and getting it to work properly. As ideally all three images would be downloaded, saved and URI's returned simultaneously. Most of my issues come from blocking calls that I cannot seem to avoid.
Here's all of the relevant code
private val okHttpClient: OkHttpClient = OkHttpClient()
suspend fun saveImageToDB(networkImageModel: CBImageNetworkModel): Result<Long> {
return withContext(Dispatchers.IO) {
try {
//Upload all three images to local storage
val edgesUri = this.async {
val req = Request.Builder().url(networkImageModel.edgesImageUrl).build()
val response = okHttpClient.newCall(req).execute() // BLOCKING
val btEdges = BitmapFactory.decodeStream(response.body?.byteStream())
return#async saveBitmapToAppStorage(btEdges, ImageType.EDGES)
}
val finalUri = this.async {
val urlFinal = URL(networkImageModel.finalImageUrl) // BLOCKING
val btFinal = BitmapFactory.decodeStream(urlFinal.openStream())
return#async saveBitmapToAppStorage(btFinal, ImageType.FINAL)
}
val labelUri = this.async {
val urlLabels = URL(networkImageModel.labelsImageUrl)
val btLabel = BitmapFactory.decodeStream(urlLabels.openStream())
return#async saveBitmapToAppStorage(btLabel, ImageType.LABELS)
}
awaitAll(edgesUri, finalUri, labelUri)
if(edgesUri.getCompleted() == null || finalUri.getCompleted() == null || labelUri.getCompleted() == null) {
return#withContext Result.failure(Exception("An image couldn't be saved"))
}
} catch (e: Exception) {
Result.failure<Long>(e)
}
try {
// Result.success( db.imageDao().insertImage(image))
Result.success(123) // A placeholder untill I actually get the URI's to create my Db Entity
} catch (e: Exception) {
Timber.e(e)
Result.failure(e)
}
}
}
//Save the bitmap and return Uri or null if failed
private fun saveBitmapToAppStorage(bitmap: Bitmap, imageType: ImageType): Uri? {
val type = when (imageType) {
ImageType.EDGES -> "edges"
ImageType.LABELS -> "labels"
ImageType.FINAL -> "final"
}
val filename = "img_" + System.currentTimeMillis().toString() + "_" + type
val file = File(context.filesDir, filename)
try {
val fos = file.outputStream()
bitmap.compress(Bitmap.CompressFormat.PNG, 100, fos);
fos.close();
} catch (e: Exception) {
Timber.e(e)
return null
}
return file.toUri()
}
Here I am calling this function
viewModelScope.launch {
val imageID = appRepository.saveImageToDB(imageNetworkModel)
withContext(Dispatchers.Main) {
val uri = Uri.parse("$PAINT_DEEPLINK/$imageID")
navManager.navigate(uri)
}
}
Another issue I am facing is returning the URI in the first place and handling errors. As if one of these parts fails, I'd like to cancel the whole thing and return Result.failure(), but I am unsure on how to achieve that. As returning null just seems meh, I'd much prefer to have an error message or something along those lines.

Android worker - update and preserve state across retries

Kotlin/Android novice here :). I'm playing around with chunked uploads using a CoroutineWorker and don't see a built-in way to maintain state for my worker in case a retry happens, but I'm having sort of a hard time believing smth like that would be missing...
My use case is the following:
Create the worker request with the path to the file to upload as input data
Worker loops over the file and performs uploads in chunks. The latest uploaded chunkIndex is being tracked.
In case of an error and subsequent Retry(), the worker somehow retrieves the current chunk index and resumes rather than starting from at the beginning again.
So basically, I really just need to preserve that chunkIndex flag. I looked into setting progress, but this seems to be hit or miss on retries (worked once, wasn't available on another attempt).
override suspend fun doWork(): Result {
try {
// TODO check if we are resuming with a given chunk index
chunkIndex = ...
// do the work
performUpload(...)
return Result.success()
} catch (e: Exception) {
// TODO cache the chunk index
return Result.retry()
}
}
Did I overlook something, or would I really have to store that index outside the worker?

You have a pretty good use-case but unfortunately you cannot cache data within Worker class or pass on the data to the next Worker object on retry! As you suspected, you will have to store the index outside of the WorkManager provided constructs!
Long answer,
The Worker object can receive and return data. It can access the data from getInputData() method. If you chain tasks, the output of one worker can be input for the next-in-line worker. This can be done by returning Result.success(output) (see below code)
public Result doWork() {
int chunkIndex = upload();
//...set the output, and we're done!
Data output = new Data.Builder()
.putInt(KEY_RESULT, result)
.build();
return Result.success(output);
}
So the problem is we cannot return data for the retry case, only for failure and success case! (Result.retry(Data data) method is missing!)
Reference: official documentation and API.

As stated in GB's answer, there seems to be no way to cache data with in the worker, or do a Result.retry(data). I ended up just doing a quick hack with SharedPreferences instead.
Solution below. Take it with a grain of salt, I have a total of ~10 hours of Kotlin under my belt ;)
var latestChunkIndex = -1
override suspend fun doWork(): Result = withContext(Dispatchers.IO) {
try {
// get cached entry (simplified - no checking for fishy status or anything)
val transferId = id.toString()
var uploadInfo: UploadInfo = TransferCache.tryGetUpload(applicationContext, transferId) ?: TransferCache.registerUpload(applicationContext, transferId, TransferStatus.InProgress)
if(uploadInfo.status != TransferStatus.InProgress) {
TransferCache.setUploadStatus(applicationContext, transferId, TransferStatus.InProgress)
}
// resolve the current chunk - this will allow us to resume in case we're retrying
latestChunkIndex = uploadInfo.latestChunkIndex
// do the actual work
upload()
// update status and complete
TransferCache.setUploadStatus(applicationContext, id.toString(), TransferStatus.Success)
Result.success()
} catch (e: Exception) {
if (runAttemptCount > 20) {
// give up
TransferCache.setUploadStatus(applicationContext, id.toString(), TransferStatus.Error)
Result.failure()
}
// update status and schedule retry
TransferCache.setUploadStatus(applicationContext, id.toString(), TransferStatus.Paused)
Result.retry()
}
}
Within my upload function, I'm simply keeping track of my cache (I could also just do it in the exception handler of the doWork method, but I'll use the cache entry for status checks as well, and it's cheap):
private suspend fun upload() {
while ((latestChunkIndex + 1) * defaultChunkSize < fileSize) {
// doing the actual upload
...
// increment chunk number and store as progress
latestChunkIndex += 1
TransferCache.cacheUploadProgress(applicationContext, id.toString(), latestChunkIndex)
}
}
and the TransferCache looking like this (note that there is no housekeeping there, so without cleanup, this would just continue to grow!)
class UploadInfo() {
var transferId: String = ""
var status: TransferStatus = TransferStatus.Undefined
var latestChunkIndex: Int = -1
constructor(transferId: String) : this() {
this.transferId = transferId
}
}
object TransferCache {
private const val PREFERENCES_NAME = "${BuildConfig.APPLICATION_ID}.transfercache"
private val gson = Gson()
fun tryGetUpload(context: Context, transferId: String): UploadInfo? {
return getPreferences(context).tryGetUpload(transferId);
}
fun cacheUploadProgress(context: Context, transferId: String, transferredChunkIndex: Int): UploadInfo {
getPreferences(context).run {
// get or create entry, update and save
val uploadInfo = tryGetUpload(transferId)!!
uploadInfo.latestChunkIndex = transferredChunkIndex
return saveUpload(uploadInfo)
}
}
fun setUploadStatus(context: Context, transferId: String, status: TransferStatus): UploadInfo {
getPreferences(context).run {
val upload = tryGetUpload(transferId) ?: registerUpload(context, transferId, status)
if (upload.status != status) {
upload.status = status
saveUpload(upload)
}
return upload
}
}
/**
* Registers a new upload transfer. This would simply (and silently) override any
* existing registration.
*/
fun registerUpload(context: Context, transferId: String, status: TransferStatus): UploadInfo {
getPreferences(context).run {
val upload = UploadInfo(transferId).apply {
this.status = status
}
return saveUpload(upload)
}
}
private fun getPreferences(context: Context): SharedPreferences {
return context.getSharedPreferences(
PREFERENCES_NAME,
Context.MODE_PRIVATE
)
}
private fun SharedPreferences.tryGetUpload(transferId: String): UploadInfo? {
val data: String? = getString(transferId, null)
return if (data == null)
null
else
gson.fromJson(data, UploadInfo::class.java)
}
private fun SharedPreferences.saveUpload(uploadInfo: UploadInfo): UploadInfo {
val editor = edit()
editor.putString(uploadInfo.transferId, gson.toJson(uploadInfo))
editor.apply()
return uploadInfo;
}
}