Develop Reference

Write to function, is called multi time in request body

I have a progreesBar for uploading with retrofit and I implementation that with some of examples.
my problem is 'WriteTo' function in my custom requestBody class.
This function send progress value for use in my progressBar but this function is called twice. I used debugger and I think some interceptors call WriteTo function.
Let me explain the problem more clearly,When I click Upload button, The number of progress bar reaches one hundred and then it starts again from zero.
Some of the things I did:
I removed HttpLoggingInterceptor.
I used a boolean variable for check that 'writeTo' don't post anything the first time
I don't have any extra interceptors
Also I read this topics:
Retrofit 2 RequestBody writeTo() method called twice
using Retrofit2/okhttp3 upload file,the upload action always performs twice,one fast ,and other slow
Interceptor Problem
My codes:
ProgressRequestBody class
class ProgressRequestBody : RequestBody() {
var mutableLiveData = MutableLiveData<Int>()
lateinit var mFile: File
lateinit var contentType: String
companion object {
private const val DEFAULT_BUFFER_SIZE = 2048
}
override fun contentType(): MediaType? {
return "$contentType/*".toMediaTypeOrNull()
}
#Throws(IOException::class)
override fun contentLength(): Long {
return mFile.length()
}
#Throws(IOException::class)
override fun writeTo(sink: BufferedSink) {
val fileLength = mFile.length()
val buffer = ByteArray(DEFAULT_BUFFER_SIZE)
val `in` = FileInputStream(mFile)
var uploaded: Long = 0
`in`.use { `in` ->
var read: Int
while (`in`.read(buffer).also { read = it } != -1) {
val percentage = (100 * uploaded / fileLength).toInt()
mutableLiveData.postValue(percentage)
uploaded += read.toLong()
sink.write(buffer, 0, read)
}
}
}
}
private fun upload(file: File, fileType: FileType) {
val fileBody = ProgressRequestBody()
fileBody.mFile = file
fileBody.contentType = file.name
uploadImageJob = viewModelScope.launch(Dispatchers.IO) {
val body = MultipartBody.Part.createFormData("File", file.name, fileBody)
fileUploadRepo.upload(body).catch {
// ...
}.collect {
when (it) {
// ...
}
}
}
}
In my fragment I use liveData for collect progressBar progress value.

OutOfMemory Exception raised in okHttp onFailure callback

I am using Kotlin and writing a function to upload a file. While testing, I have observed that if I do the following operations, OutOfMemory exception is raised and onFailure callback is called.
Select a file of size 100 MB to upload it to server. Upload it by calling uploadFile.
While it uploads, disconnect the internet. onFailure is called with an exception (timeout).
Enable the internet again and try to upload it again by calling uploadFile function.
Repeat the step 2 and 3 for 1-2 more times and app crashes. onFailure is called this time with OutOfMemory exception.
Here is my code.
class UploadManager(
private val fileTransferDataSource: IListFileTransferDataSource
) {
private val uploadClient by lazy {
OkHttpClient.Builder()
.retryOnConnectionFailure(false)
.writeTimeout(40, TimeUnit.SECONDS)
.readTimeout(40, TimeUnit.SECONDS)
.build()
}
suspend fun uploadFile(url: String,
fileUri: String,
downloadUrl: String?,
stream: InputStream,
callback: ((success: Boolean, filePath: String, url: String?, responseCode: Int?) -> Unit)? = null) {
val baseUrl = FileTransferUtility.getBaseUrl(url)
val authToken = fileTransferDataSource.getAuthenticationToken(baseUrl)
if (baseUrl.isEmpty() || authToken.isEmpty()) {
stream.close()
callback?.let { it(false, fileUri, null, null) }
return
}
kotlin.runCatching {
val buf = ByteArray(stream.available())
val bytesRead = stream.read(buf)
stream.close()
if (bytesRead == -1) {
callback?.let { it(false, fileUri, null, null) }
return#runCatching
}
val requestBody = create(FileTransferUtility.contentTypeStream.toMediaType(), buf)
val request = requestBody.let {
Request.Builder()
.url(url)
.post(it)
.addHeader(HttpConstants.Headers.AUTHORIZATION, HttpConstants.Values.AUTHORIZATION_TOKEN_BEARER_FORMAT.format(authToken))
.addHeader(HttpConstants.Headers.CONTENT_TYPE, HttpConstants.Values.APPLICATION_JSON)
.addHeader(HttpConstants.Headers.ACCEPT, HttpConstants.Values.APPLICATION_JSON)
.build()
}
if (request == null) {
callback?.let { it(false, fileUri, null, null) }
return#runCatching
}
uploadClient.newCall(request).enqueue(object : Callback {
override fun onFailure(call: Call, e: IOException) {
call.cancel()
callback?.let { it(false, fileUri, null, null) }
}
override fun onResponse(call: Call, response: Response) {
response.body?.close()
callback?.let { it(response.code == 200, fileUri, downloadUrl, response.code) }
}
})
}
}
//Restricting object creation for this class by making it singleton
companion object : SingletonHolder<UploadManager, IListFileTransferDataSource>(::UploadManager)
}
However, it doesn't crash even if I upload multiple files of size 100 MB and they gets upload without any issue. Problem happens only when onFailure is triggered multiple times. I am suspecting the some internal buffer aren't getting deallocating on failure.
I have tired the following.
Cancelling the dispatcher
Add interceptor for Logging with log level None.
cancelling call explicitly in onFailure callback
Nothing seems to resolve this issue. Please help me to identify the memory leak.
Stack Trace:
java.io.IOException: canceled due to java.lang.OutOfMemoryError: Failed to allocate a 8208 byte allocation with 200 free bytes and 200B until OOM, target footprint 268435456, growth limit 268435456
0 = {StackTraceElement#16551} "okhttp3.internal.connection.RealCall$AsyncCall.run(RealCall.kt:515)"
1 = {StackTraceElement#16552} "java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1167)"
2 = {StackTraceElement#16553} "java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:641)"
3 = {StackTraceElement#16554} "java.lang.Thread.run(Thread.java:919)"

I have realized the problem was with val buf = ByteArray(inputStream.available()). It creates a ByteArray of huge size in a single go. If we have to upload multiple files of larger size (in my case, 100MB), garbage collector take a while to free the memory from last uploads. It was causing the memory exception. I have changed my code and it stopped being a problem:
suspend fun uploadFile(
url: String,
fileUri: String,
downloadUrl: String?,
stream: InputStream,
coroutineScope: CoroutineScope
): Flow<Result> = flow {
kotlin.runCatching {
val baseUrl = FileTransferUtility.getBaseUrl(url)
val authToken = fileTransferDataSource.getAuthenticationToken(baseUrl)
if (baseUrl.isEmpty() || authToken.isEmpty()) {
stream.close()
emit(Result.Failure(fileUri))
return#flow
}
var connection: HttpURLConnection? = null
val bufferedInputStream = stream.buffered()
try {
val fileSize = stream.available()
if (fileSize == 0)
throw IOException("Unable to read file $fileUri")
connection = (URL(url).openConnection() as HttpURLConnection).also { conn ->
addHeadersForUpload(conn, authToken, fileSize)
conn.readTimeout = UPLOAD_READ_TIMEOUT
}
var progress = 0
emit(Result.InProgress(fileUri, progress))
BufferedOutputStream(connection.outputStream).use { uploadStream ->
var bytesWritten = 0
val buffer = ByteArray(DEFAULT_UPLOAD_BUFFER_SIZE)
while (true) {
val size = bufferedInputStream.read(buffer)
if (size <= 0)
break
bytesWritten += size
progress = ((100f * bytesWritten) / fileSize).toInt()
emit(Result.InProgress(fileUri, progress))
uploadStream.write(buffer, 0, size)
}
uploadStream.flush()
}
val responseCode = connection.responseCode
val isSuccess = responseCode == 200
if (isSuccess) {
if (progress < 100)
emit(Result.InProgress(fileUri, 100))
emit(Result.Success(fileUri, downloadUrl, readResponseBody(connection)))
} else {
emit(Result.Failure(fileUri))
}
} catch (e: Exception) {
emit(Result.Failure(fileUri))
} finally {
bufferedInputStream.close()
connection?.disconnect()
}
}
}.flowOn(Dispatchers.IO)
private fun readResponseBody(connection: HttpURLConnection): String? {
val builder = StringBuilder()
val lineBreak = System.getProperty("line.separator")
BufferedReader(connection.inputStream.reader()).use {
val line = it.readLine() ?: return#use
builder.append(line + lineBreak)
}
return builder.toString().trim(*lineBreak.toCharArray())
}

How to upload large file to the server using retrofit multipart

I have the request which works well in postman:
and I'm trying to make it with Retrofit. In general file sizes will be >500MB that. I did such uploading method:
fun uploadFile(file:File) {
val client = OkHttpClient().newBuilder()
.build()
val mediaType: MediaType? = "text/plain".toMediaTypeOrNull()
val body: RequestBody = MultipartBody.Builder().setType(MultipartBody.FORM)
.addFormDataPart(
"data", file.name,
file.asRequestBody()
)
.build()
val request: Request = Request.Builder()
.url("https://..../upload.php")
.method("POST", body)
.build()
val response: okhttp3.Response = client.newCall(request).execute()
println(response.message)
}
but I need to have file for uploading. I can create temporary file with such way:
val path = requireContext().cacheDir
val file = File.createTempFile(
name ?: "",
fileUri.lastPathSegment,
path
)
val os = FileOutputStream(file)
os.write(string)
os.close()
but I usually receive outOfMemoryException. I also added to the AndroidManifest.xml heap param:
android:largeHeap="true"
but it didn't help me at all during temp file creating. I don't know how postman uploads files, but in general I managed to upload with his help file with size about 600Mb. I can also cut selected file with chunks:
val data = result.data
data?.let {
val fileUri = data.data
var name: String? = null
var size: Long? = null
fileUri.let { returnUri ->
contentResolver?.query(returnUri!!, null, null, null, null)
}?.use { cursor ->
val nameIndex = cursor.getColumnIndex(OpenableColumns.DISPLAY_NAME)
val sizeIndex = cursor.getColumnIndex(OpenableColumns.SIZE)
cursor.moveToFirst()
name = cursor.getString(nameIndex)
size = cursor.getLong(sizeIndex)
}
val inputStream: InputStream? = fileUri?.let { it1 ->
contentResolver.openInputStream(
it1
)
}
val fileData = inputStream?.readBytes()
val mimeType = fileUri.let { returnUri ->
returnUri.let { retUri ->
if (retUri != null) {
contentResolver.getType(retUri)
}
}
}
fileData?.let {
val MAX_SUB_SIZE = 4194304 // 4*1024*1024 == 4MB
var start = 0 // From 0
var end = MAX_SUB_SIZE // To MAX_SUB_SIZE bytes
var subData: ByteArray // 4MB Sized Array
val max = fileData.size
if (max > 0) {
while (end < max) {
subData = fileData.copyOfRange(start, end)
start = end
end += MAX_SUB_SIZE
if (end >= max) {
end = max
}
println("file handling" + subData.size)
}
end-- // To avoid a padded zero
subData = fileData.copyOfRange(start, end)
println("file handling" + subData.size)
}
}
}
all actions will be made in:
private val filesReceiver =
registerForActivityResult(ActivityResultContracts.StartActivityForResult()) { result ->
if (result.resultCode == Activity.RESULT_OK) {
}
}
so I won't have any file path in normal way. Anyway I think I did something wrong.
UPDATE
right now I have such file uploading from inputStream:
private fun doSomeNetworkStuff(file:InputStream, name:String) {
GlobalScope.launch(Dispatchers.IO) {
val client = OkHttpClient()
.newBuilder()
.protocols(listOf(Protocol.HTTP_1_1))
.connectTimeout(10, TimeUnit.MINUTES)
.readTimeout(10, TimeUnit.MINUTES)
.build()
val mediaType: MediaType? = "text/plain".toMediaTypeOrNull()
val body: RequestBody = MultipartBody.Builder().setType(MultipartBody.FORM)
.addFormDataPart(
"data", name,
file.readBytes().toRequestBody(mediaType)
)
.build()
val request: Request = Request.Builder()
.url("https://.../upload.php")
.method("POST", body)
.build()
val response: Response = client.newCall(request).execute()
println(response.body)
}
}
and receive such error:
java.lang.OutOfMemoryError: Failed to allocate a 173410912 byte allocation with 25165824 free bytes and 89MB until OOM, max allowed footprint 199761800, growth limit 268435456
but I can upload with this code file with size about 90mb

The retrofit multipart stuff has a member that takes an Uri for a request body.
You try to use the one for a File instance.

Have you set log in loggingInterceptor or restadapter ?
if yes then try to set it NONE.

Trying to upload image from Android to AWS Lambda

I am using this code in Kotlin/Android to upload image to server:
fun uploadImageWithLambda(){
val file = File(filePath)
val retrofit = NetworkClient.getRetrofitObj()
val uploadApis = retrofit.create(UploadApis::class.java)
val requestBody = RequestBody.create(MediaType.parse("image/*"), file)
val part = MultipartBody.Part.createFormData("image", file.name, requestBody)
val call = uploadApis.uploadImage(imageRequest = part)
call.enqueue(object : Callback<ImageResponse> {
override fun onFailure(call: Call<ImageResponse>, t: Throwable) {
Log.e(TAG, t.toString())
}
override fun onResponse(call: Call<ImageResponse>, response: Response<ImageResponse>) {
Log.i(TAG, "success")
}
})
}
This is function in UploadApis
#Multipart
#POST("prod/res")
fun uploadImage(#Part imageRequest: MultipartBody.Part) : Call<ImageResponse>
The file that I received in Lambda cannot be open. When I removed some bytes from that file I got proper image file.
First lines of code in my lambda function:
const multipart = require('aws-multipart-parser')
exports.handler = async (event) => {
try {
CONSTANTS.logger.info("POST image in S3...")
let spotText = {}
const result = multipart.parse(event, spotText)
fs.writeFileSync('original.jpg', result.image.content)
const content = result.image.content;
console.log(result)
var newBuffer = new Buffer(content.length - 22);
content.copy(newBuffer, 0, 22, content.length);
// const sliced = content.slice(25);
console.log(newBuffer);
fs.writeFileSync('new.jpg', content);
....
I also made a small express application with one route to upload the image to S3 and I sent the image from Android in the same way and it worked, I could open the image.
Is there anyone having the same/similar issue?

How to use okhttp to create a stream of data and write into device in android

What I am doing: I am using a connection to download file from server and write into a storage
What is happening: Code is working
What I am trying to do: How to achieve the same using okhttp
try {
val url = URL(DICTIONARY_FILE_URL)
val conection = url.openConnection()
conection.connect()
// getting file length
// input stream to read file - with 8k buffer
val input = BufferedInputStream(url.openStream(), 8192)
// Output stream to write file
val directoryPathName = Environment.getExternalStorageDirectory().absolutePath.plus("/CNX/dictionary/")
val dictionaryFileName = "DictionaryEN.quickdic"
var f = File(directoryPathName)
if(!f.isDirectory) {
f.mkdirs()
}
val output = FileOutputStream(directoryPathName.plus(dictionaryFileName))
val data = ByteArray(1024)
var count: Int? = 0
while ({ count = input.read(data);count }() != -1) {
output.write(data, 0, count!!)
}
// flushing output
output.flush()
// closing streams
output.close()
input.close()
isJobSuccess = true
//sharedPreferences[IS_DICTIONARY_DOWNLOADED] = true
} catch (e: Exception) {
Log.e("Exception",e.message)
isJobSuccess = false
//sharedPreferences[IS_DICTIONARY_DOWNLOADED] = false
}

Sample for Downloading a file with OkHttp asynchronously
fun downloadFileAsync(downloadUrl: String) {
val client = OkHttpClient();
val request = Request.Builder().url(downloadUrl).build();
client.newCall(request).enqueue(object : Callback {
override fun onFailure(call: Call, e: IOException) {
Log.e("fail", e.printStackTrace().toString())
}
override fun onResponse(call: Call, response: Response) {
if (!response.isSuccessful) {
Log.e("fail", "Download error")
}else{
// Output stream to write file
val directoryPathName = Environment.getExternalStorageDirectory().absolutePath.plus("/CNX/dictionary/")
val dictionaryFileName = "DictionaryEN.quickdic"
val f = File(directoryPathName)
if(!f.isDirectory) {
f.mkdirs()
}
val output = FileOutputStream(directoryPathName.plus(dictionaryFileName))
response.body?.let {
output.write(it.bytes())
}
// flushing output
output.flush()
// closing streams
output.close()
}
}
})
}
Sample for Downloading a file with OkHttp synchronously
fun downloadFileSync(downloadUrl: String) {
val client = OkHttpClient();
val request = Request.Builder().url(downloadUrl).build();
val response = client.newCall (request).execute();
if (!response.isSuccessful) {
Log.e("fail", "Failed to download file: " + response)
}else {
val directoryPathName = Environment.getExternalStorageDirectory().absolutePath.plus("/CNX/dictionary/")
val dictionaryFileName = "DictionaryEN.quickdic"
val f = File(directoryPathName)
if (!f.isDirectory) {
f.mkdirs()
}
val output = FileOutputStream(directoryPathName.plus(dictionaryFileName))
response.body?.let {
output.write(it.bytes())
}
// flushing output
output.flush()
// closing streams
output.close()
}
}
build.gradle
implementation("com.squareup.okhttp3:okhttp:4.5.0")