How to upload large file to the server using retrofit multipart - android

I have the request which works well in postman:
and I'm trying to make it with Retrofit. In general file sizes will be >500MB that. I did such uploading method:
fun uploadFile(file:File) {
val client = OkHttpClient().newBuilder()
.build()
val mediaType: MediaType? = "text/plain".toMediaTypeOrNull()
val body: RequestBody = MultipartBody.Builder().setType(MultipartBody.FORM)
.addFormDataPart(
"data", file.name,
file.asRequestBody()
)
.build()
val request: Request = Request.Builder()
.url("https://..../upload.php")
.method("POST", body)
.build()
val response: okhttp3.Response = client.newCall(request).execute()
println(response.message)
}
but I need to have file for uploading. I can create temporary file with such way:
val path = requireContext().cacheDir
val file = File.createTempFile(
name ?: "",
fileUri.lastPathSegment,
path
)
val os = FileOutputStream(file)
os.write(string)
os.close()
but I usually receive outOfMemoryException. I also added to the AndroidManifest.xml heap param:
android:largeHeap="true"
but it didn't help me at all during temp file creating. I don't know how postman uploads files, but in general I managed to upload with his help file with size about 600Mb. I can also cut selected file with chunks:
val data = result.data
data?.let {
val fileUri = data.data
var name: String? = null
var size: Long? = null
fileUri.let { returnUri ->
contentResolver?.query(returnUri!!, null, null, null, null)
}?.use { cursor ->
val nameIndex = cursor.getColumnIndex(OpenableColumns.DISPLAY_NAME)
val sizeIndex = cursor.getColumnIndex(OpenableColumns.SIZE)
cursor.moveToFirst()
name = cursor.getString(nameIndex)
size = cursor.getLong(sizeIndex)
}
val inputStream: InputStream? = fileUri?.let { it1 ->
contentResolver.openInputStream(
it1
)
}
val fileData = inputStream?.readBytes()
val mimeType = fileUri.let { returnUri ->
returnUri.let { retUri ->
if (retUri != null) {
contentResolver.getType(retUri)
}
}
}
fileData?.let {
val MAX_SUB_SIZE = 4194304 // 4*1024*1024 == 4MB
var start = 0 // From 0
var end = MAX_SUB_SIZE // To MAX_SUB_SIZE bytes
var subData: ByteArray // 4MB Sized Array
val max = fileData.size
if (max > 0) {
while (end < max) {
subData = fileData.copyOfRange(start, end)
start = end
end += MAX_SUB_SIZE
if (end >= max) {
end = max
}
println("file handling" + subData.size)
}
end-- // To avoid a padded zero
subData = fileData.copyOfRange(start, end)
println("file handling" + subData.size)
}
}
}
all actions will be made in:
private val filesReceiver =
registerForActivityResult(ActivityResultContracts.StartActivityForResult()) { result ->
if (result.resultCode == Activity.RESULT_OK) {
}
}
so I won't have any file path in normal way. Anyway I think I did something wrong.
UPDATE
right now I have such file uploading from inputStream:
private fun doSomeNetworkStuff(file:InputStream, name:String) {
GlobalScope.launch(Dispatchers.IO) {
val client = OkHttpClient()
.newBuilder()
.protocols(listOf(Protocol.HTTP_1_1))
.connectTimeout(10, TimeUnit.MINUTES)
.readTimeout(10, TimeUnit.MINUTES)
.build()
val mediaType: MediaType? = "text/plain".toMediaTypeOrNull()
val body: RequestBody = MultipartBody.Builder().setType(MultipartBody.FORM)
.addFormDataPart(
"data", name,
file.readBytes().toRequestBody(mediaType)
)
.build()
val request: Request = Request.Builder()
.url("https://.../upload.php")
.method("POST", body)
.build()
val response: Response = client.newCall(request).execute()
println(response.body)
}
}
and receive such error:
java.lang.OutOfMemoryError: Failed to allocate a 173410912 byte allocation with 25165824 free bytes and 89MB until OOM, max allowed footprint 199761800, growth limit 268435456
but I can upload with this code file with size about 90mb

The retrofit multipart stuff has a member that takes an Uri for a request body.
You try to use the one for a File instance.

Have you set log in loggingInterceptor or restadapter ?
if yes then try to set it NONE.

Related

Upload image to Graphql server using apollo kotlin

I need to upload image to my Graphql server from android application. The details in the documentation is not working. I need an example.
Came up with the solution. 1st I needed to create an upload scalar type. in Fragment class:
requireContext().contentResolver.openFileDescriptor(
selectedImageUri!!,
"r",
null
) ?: return
val file = File(
requireContext().cacheDir, requireContext().contentResolver.getFileName(
selectedImageUri
)
)
val body = UploadRequestBody(file, "image")
val upload = DefaultUpload.Builder()
.content(file)
.fileName(file.name)
.contentType(body.contentType().toString())
.build()
In case what the UploadRequestBody class does:
class UploadRequestBody(
private val file: File,
private val contentType: String
) : RequestBody() {
override fun contentType() = "$contentType/*".toMediaTypeOrNull()
override fun contentLength() = file.length()
override fun writeTo(sink: BufferedSink) {
val length = file.length()
val buffer = ByteArray(DEFAULT_BUFFER_SIZE)
val fileInputStream = FileInputStream(file)
var uploaded = 0L
fileInputStream.use { inputStream ->
var read: Int
while (inputStream.read(buffer).also { read = it } != -1) {
uploaded += read
sink.write(buffer, 0, read)
}
}
}
companion object {
private const val DEFAULT_BUFFER_SIZE = 2048
}
}

OutOfMemory Exception raised in okHttp onFailure callback

I am using Kotlin and writing a function to upload a file. While testing, I have observed that if I do the following operations, OutOfMemory exception is raised and onFailure callback is called.
Select a file of size 100 MB to upload it to server. Upload it by calling uploadFile.
While it uploads, disconnect the internet. onFailure is called with an exception (timeout).
Enable the internet again and try to upload it again by calling uploadFile function.
Repeat the step 2 and 3 for 1-2 more times and app crashes. onFailure is called this time with OutOfMemory exception.
Here is my code.
class UploadManager(
private val fileTransferDataSource: IListFileTransferDataSource
) {
private val uploadClient by lazy {
OkHttpClient.Builder()
.retryOnConnectionFailure(false)
.writeTimeout(40, TimeUnit.SECONDS)
.readTimeout(40, TimeUnit.SECONDS)
.build()
}
suspend fun uploadFile(url: String,
fileUri: String,
downloadUrl: String?,
stream: InputStream,
callback: ((success: Boolean, filePath: String, url: String?, responseCode: Int?) -> Unit)? = null) {
val baseUrl = FileTransferUtility.getBaseUrl(url)
val authToken = fileTransferDataSource.getAuthenticationToken(baseUrl)
if (baseUrl.isEmpty() || authToken.isEmpty()) {
stream.close()
callback?.let { it(false, fileUri, null, null) }
return
}
kotlin.runCatching {
val buf = ByteArray(stream.available())
val bytesRead = stream.read(buf)
stream.close()
if (bytesRead == -1) {
callback?.let { it(false, fileUri, null, null) }
return#runCatching
}
val requestBody = create(FileTransferUtility.contentTypeStream.toMediaType(), buf)
val request = requestBody.let {
Request.Builder()
.url(url)
.post(it)
.addHeader(HttpConstants.Headers.AUTHORIZATION, HttpConstants.Values.AUTHORIZATION_TOKEN_BEARER_FORMAT.format(authToken))
.addHeader(HttpConstants.Headers.CONTENT_TYPE, HttpConstants.Values.APPLICATION_JSON)
.addHeader(HttpConstants.Headers.ACCEPT, HttpConstants.Values.APPLICATION_JSON)
.build()
}
if (request == null) {
callback?.let { it(false, fileUri, null, null) }
return#runCatching
}
uploadClient.newCall(request).enqueue(object : Callback {
override fun onFailure(call: Call, e: IOException) {
call.cancel()
callback?.let { it(false, fileUri, null, null) }
}
override fun onResponse(call: Call, response: Response) {
response.body?.close()
callback?.let { it(response.code == 200, fileUri, downloadUrl, response.code) }
}
})
}
}
//Restricting object creation for this class by making it singleton
companion object : SingletonHolder<UploadManager, IListFileTransferDataSource>(::UploadManager)
}
However, it doesn't crash even if I upload multiple files of size 100 MB and they gets upload without any issue. Problem happens only when onFailure is triggered multiple times. I am suspecting the some internal buffer aren't getting deallocating on failure.
I have tired the following.
Cancelling the dispatcher
Add interceptor for Logging with log level None.
cancelling call explicitly in onFailure callback
Nothing seems to resolve this issue. Please help me to identify the memory leak.
Stack Trace:
java.io.IOException: canceled due to java.lang.OutOfMemoryError: Failed to allocate a 8208 byte allocation with 200 free bytes and 200B until OOM, target footprint 268435456, growth limit 268435456
0 = {StackTraceElement#16551} "okhttp3.internal.connection.RealCall$AsyncCall.run(RealCall.kt:515)"
1 = {StackTraceElement#16552} "java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1167)"
2 = {StackTraceElement#16553} "java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:641)"
3 = {StackTraceElement#16554} "java.lang.Thread.run(Thread.java:919)"
I have realized the problem was with val buf = ByteArray(inputStream.available()). It creates a ByteArray of huge size in a single go. If we have to upload multiple files of larger size (in my case, 100MB), garbage collector take a while to free the memory from last uploads. It was causing the memory exception. I have changed my code and it stopped being a problem:
suspend fun uploadFile(
url: String,
fileUri: String,
downloadUrl: String?,
stream: InputStream,
coroutineScope: CoroutineScope
): Flow<Result> = flow {
kotlin.runCatching {
val baseUrl = FileTransferUtility.getBaseUrl(url)
val authToken = fileTransferDataSource.getAuthenticationToken(baseUrl)
if (baseUrl.isEmpty() || authToken.isEmpty()) {
stream.close()
emit(Result.Failure(fileUri))
return#flow
}
var connection: HttpURLConnection? = null
val bufferedInputStream = stream.buffered()
try {
val fileSize = stream.available()
if (fileSize == 0)
throw IOException("Unable to read file $fileUri")
connection = (URL(url).openConnection() as HttpURLConnection).also { conn ->
addHeadersForUpload(conn, authToken, fileSize)
conn.readTimeout = UPLOAD_READ_TIMEOUT
}
var progress = 0
emit(Result.InProgress(fileUri, progress))
BufferedOutputStream(connection.outputStream).use { uploadStream ->
var bytesWritten = 0
val buffer = ByteArray(DEFAULT_UPLOAD_BUFFER_SIZE)
while (true) {
val size = bufferedInputStream.read(buffer)
if (size <= 0)
break
bytesWritten += size
progress = ((100f * bytesWritten) / fileSize).toInt()
emit(Result.InProgress(fileUri, progress))
uploadStream.write(buffer, 0, size)
}
uploadStream.flush()
}
val responseCode = connection.responseCode
val isSuccess = responseCode == 200
if (isSuccess) {
if (progress < 100)
emit(Result.InProgress(fileUri, 100))
emit(Result.Success(fileUri, downloadUrl, readResponseBody(connection)))
} else {
emit(Result.Failure(fileUri))
}
} catch (e: Exception) {
emit(Result.Failure(fileUri))
} finally {
bufferedInputStream.close()
connection?.disconnect()
}
}
}.flowOn(Dispatchers.IO)
private fun readResponseBody(connection: HttpURLConnection): String? {
val builder = StringBuilder()
val lineBreak = System.getProperty("line.separator")
BufferedReader(connection.inputStream.reader()).use {
val line = it.readLine() ?: return#use
builder.append(line + lineBreak)
}
return builder.toString().trim(*lineBreak.toCharArray())
}

How to Encrypt Request Data in Retrofit

There are many examples of encryption.
It's not available at the moment.
The methods have changed a lot.
All I want is to send data using Retrofit.
I'm encrypting it with AES256.
What can you do?
class Encryptor: Interceptor {
override fun intercept(chain: Interceptor.Chain): Response {
var request = chain.request()
val body = request.body()
//------------------------------ HOW...
var byteArrayOutputStream = ByteArrayOutputStream()
var sink = Okio.sink(byteArrayOutputStream)
var bufferedSink = Okio.buffer(sink)
body!!.writeTo(bufferedSink)
bufferedSink.close()
var buffer: String = "a"
bufferedSink.writeUtf8(buffer)
//------------------------------
// val buffer = Buffer()
//
// body!!.writeTo(buffer)
// val strOldBody: String = body. buffer.readUtf8()
// val mediaType: MediaType = MediaType.parse("text/plain; charset=utf-8")
// val strNewBody: String = encrypt(strOldBody)
// val body = RequestBody.create(mediaType, strNewBody)
// request = request.newBuilder().header("Content-Type", body.contentType().toString())
// .header("Content-Length", body.contentLength().toString())
// .method(request.method(), body).build()
return chain.proceed(request)
}
private fun encrypt(text: String): String? {
//your code
return text
}
}
I know how to do AES256.
I just want to know how to encrypt the body.

Downloading mp3 file with okhttp produces corrupt file

I am trying to download an mp3 file from an http link and save the file to local storage. All the code I have tried saves a corrupt file that is slightly twice as large as it should be.
File should be 1,204,787
File saved is 2,478,272
The file I am trying to download is: rise-stage.bioinf.unc.edu/cue_audio/sampleaudio.mp3 –
It plays fine when downloaded manually.
fun downloadFilea(url:String , localFileName:String)
{
val request: Request = Request.Builder()
.url(url)
.build()
client.newCall(request).enqueue(object : Callback {
override fun onFailure(call: Call, e: IOException) {
println("error"+e.toString())
}
#Throws(IOException::class)
override fun onResponse(call: Call, response: Response) {
if (!response.isSuccessful) throw IOException("Unexpected code $response")
var uri = dataMgr.getLocalURI(localFileName)
var file = File(uri)
val body = response.body
val contentLength = body!!.contentLength()
val source = body.source()
val DOWNLOAD_CHUNK_SIZE:Long = 2048
val sink: BufferedSink = file.sink().buffer()
var totalRead: Long = 0
var read: Long = 0
while (source.read(sink.buffer(), DOWNLOAD_CHUNK_SIZE).also {
read = it
} != -1L) {
totalRead += read
val progress = (totalRead * 100 / contentLength).toInt()
}
sink.writeAll(source)
sink.flush()
sink.close()
Log.d(logTag, "downloaded file")
}
})
}

Upload image from Android to Server that accepts only JPEG and PNG

MediaStore.Images.Media.Data is deprecated so I tried using byte array, but my server throws an error that it accepts only file types of jpeg and png. I have read #commonsware's answers a couple of times from other questions but I can't seem to get it right. How do I get the file type and also the image so it can be attached to my network call? Thanks in advance
override fun onActivityResult(requestCode: Int, resultCode: Int, data: Intent?) {
if (resultCode == RESULT_OK && requestCode == PROFILE_REQUEST_CODE) {
val uri = data?.data
createImageData(uri)
profile_image.setImageURI(data?.data)
}
}
private fun createImageData(uri: Uri?) {
val inputStream = activity?.contentResolver?.openInputStream(uri!!)
processImage(inputStream)
}
private fun processImage(inputStream: InputStream?) {
imageData = inputStream?.readBytes()
bitmap = BitmapFactory.decodeStream(inputStream)
val imageBody: RequestBody = RequestBody.create(MediaType.parse("multipart/form-data"), imageData!!)
val image: MultipartBody.Part = MultipartBody.Part.createFormData("image", "user", imageBody)
uploadImage(image)
}
private fun uploadImage(image: MultipartBody.Part) {
val user = appPreferences?.getUser()
val userToken = user?.jwt_token
val token = "Bearer $userToken"
val headers = HashMap<String, String>()
//headers["Authorization"] = token
val uploadResponseCall: Call<ProfileImageResponse> = client.getApi().upload(token, image)
uploadResponseCall.enqueue(object : retrofit2.Callback<ProfileImageResponse> {
override fun onResponse(call: Call<ProfileImageResponse>, response: Response<ProfileImageResponse>) {
val imageResponse = response.body()
val resCode = imageResponse?.statuscode
val msg = imageResponse?.message
if (resCode == 200) {
appUtils.showSnackBar(requireActivity().applicationContext, profile_frame, msg!!)
} else {
appUtils.showSnackBar(requireActivity().applicationContext, profile_frame, "wrong file type")
}
}
override fun onFailure(call: Call<ProfileImageResponse>, t: Throwable) {
if (t is IOException) {
call.cancel()
Log.d("profilefragment", "issue")
appUtils.showSnackBar(requireActivity().applicationContext, profile_frame, "server error")
}
}
})
}
Here is example
#Multipart
#POST(ApiConstants.SIGNUP)
fun signUp(
#Query("first_name") firstName: String,
#Query("last_name") lastName: String,
#Query("email") email: String,
#Query("password") password: String,
#Query("gender") gender: String,
#Query("phone_code") phoneCode: String,
#Query("phone_no") phoneNo: String,
#Part("id_card_front\"; filename=\"pp.png") id_card_front: RequestBody?
) : Observable<SignUpModel>
Here is usage
var image = RequestBody.create(
MediaType.parse("image/*"),
getFileFromBitmap((imageview.drawable as BitmapDrawable).bitmap, 13)
)
disposable = retroClient.signUp(
fname,
lname,
email,
password,
gender,
phoneCode,
phone,
image,
)
.subscribeOn(Schedulers.io())
.observeOn(AndroidSchedulers.mainThread())
.subscribe({ result ->
Log.d("response", result.toString())
}, { error ->
Log.e("error", error.toString())
})
the method is used to get file from bitmap
fun getFileFromBitmap(bitmap: Bitmap, index: Int): File {
val f = File(cacheDir, "something+$index")
f.createNewFile()
val bitmap = bitmap
bitmap
val bos = ByteArrayOutputStream()
bitmap.compress(Bitmap.CompressFormat.PNG, 25 /*ignored for PNG*/, bos)
val bitmapdata = bos.toByteArray()
val fos = FileOutputStream(f)
fos.write(bitmapdata)
fos.flush()
fos.close()
return f
}
Try to add an appropriate Content-Type header. image/png or image/jpg should fit. Also, it possibly, should not be the multipart request. So you'll need such kind of request.
private fun processImage(inputStream: InputStream?) {
imageData = inputStream?.readBytes()
bitmap = BitmapFactory.decodeStream(inputStream)
val imageBody: RequestBody = imageData!.toRequestBody()
uploadImage(imageBody)
}
or if it's really should be multipart/form-data, you should correctly set mime type for the image part like this:
private fun processImage(inputStream: InputStream?) {
imageData = inputStream?.readBytes()
bitmap = BitmapFactory.decodeStream(inputStream)
val imageBody: RequestBody = RequestBody.create(MediaType.parse("image/png"), imageData!!)
val image: MultipartBody.Part = MultipartBody.Part.createFormData("image", "user", imageBody)
uploadImage(image)
}

Categories

Resources