I am using Kotlin and writing a function to upload a file. While testing, I have observed that if I do the following operations, OutOfMemory exception is raised and onFailure callback is called.
Select a file of size 100 MB to upload it to server. Upload it by calling uploadFile.
While it uploads, disconnect the internet. onFailure is called with an exception (timeout).
Enable the internet again and try to upload it again by calling uploadFile function.
Repeat the step 2 and 3 for 1-2 more times and app crashes. onFailure is called this time with OutOfMemory exception.
Here is my code.
class UploadManager(
private val fileTransferDataSource: IListFileTransferDataSource
) {
private val uploadClient by lazy {
OkHttpClient.Builder()
.retryOnConnectionFailure(false)
.writeTimeout(40, TimeUnit.SECONDS)
.readTimeout(40, TimeUnit.SECONDS)
.build()
}
suspend fun uploadFile(url: String,
fileUri: String,
downloadUrl: String?,
stream: InputStream,
callback: ((success: Boolean, filePath: String, url: String?, responseCode: Int?) -> Unit)? = null) {
val baseUrl = FileTransferUtility.getBaseUrl(url)
val authToken = fileTransferDataSource.getAuthenticationToken(baseUrl)
if (baseUrl.isEmpty() || authToken.isEmpty()) {
stream.close()
callback?.let { it(false, fileUri, null, null) }
return
}
kotlin.runCatching {
val buf = ByteArray(stream.available())
val bytesRead = stream.read(buf)
stream.close()
if (bytesRead == -1) {
callback?.let { it(false, fileUri, null, null) }
return#runCatching
}
val requestBody = create(FileTransferUtility.contentTypeStream.toMediaType(), buf)
val request = requestBody.let {
Request.Builder()
.url(url)
.post(it)
.addHeader(HttpConstants.Headers.AUTHORIZATION, HttpConstants.Values.AUTHORIZATION_TOKEN_BEARER_FORMAT.format(authToken))
.addHeader(HttpConstants.Headers.CONTENT_TYPE, HttpConstants.Values.APPLICATION_JSON)
.addHeader(HttpConstants.Headers.ACCEPT, HttpConstants.Values.APPLICATION_JSON)
.build()
}
if (request == null) {
callback?.let { it(false, fileUri, null, null) }
return#runCatching
}
uploadClient.newCall(request).enqueue(object : Callback {
override fun onFailure(call: Call, e: IOException) {
call.cancel()
callback?.let { it(false, fileUri, null, null) }
}
override fun onResponse(call: Call, response: Response) {
response.body?.close()
callback?.let { it(response.code == 200, fileUri, downloadUrl, response.code) }
}
})
}
}
//Restricting object creation for this class by making it singleton
companion object : SingletonHolder<UploadManager, IListFileTransferDataSource>(::UploadManager)
}
However, it doesn't crash even if I upload multiple files of size 100 MB and they gets upload without any issue. Problem happens only when onFailure is triggered multiple times. I am suspecting the some internal buffer aren't getting deallocating on failure.
I have tired the following.
Cancelling the dispatcher
Add interceptor for Logging with log level None.
cancelling call explicitly in onFailure callback
Nothing seems to resolve this issue. Please help me to identify the memory leak.
Stack Trace:
java.io.IOException: canceled due to java.lang.OutOfMemoryError: Failed to allocate a 8208 byte allocation with 200 free bytes and 200B until OOM, target footprint 268435456, growth limit 268435456
0 = {StackTraceElement#16551} "okhttp3.internal.connection.RealCall$AsyncCall.run(RealCall.kt:515)"
1 = {StackTraceElement#16552} "java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1167)"
2 = {StackTraceElement#16553} "java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:641)"
3 = {StackTraceElement#16554} "java.lang.Thread.run(Thread.java:919)"
I have realized the problem was with val buf = ByteArray(inputStream.available()). It creates a ByteArray of huge size in a single go. If we have to upload multiple files of larger size (in my case, 100MB), garbage collector take a while to free the memory from last uploads. It was causing the memory exception. I have changed my code and it stopped being a problem:
suspend fun uploadFile(
url: String,
fileUri: String,
downloadUrl: String?,
stream: InputStream,
coroutineScope: CoroutineScope
): Flow<Result> = flow {
kotlin.runCatching {
val baseUrl = FileTransferUtility.getBaseUrl(url)
val authToken = fileTransferDataSource.getAuthenticationToken(baseUrl)
if (baseUrl.isEmpty() || authToken.isEmpty()) {
stream.close()
emit(Result.Failure(fileUri))
return#flow
}
var connection: HttpURLConnection? = null
val bufferedInputStream = stream.buffered()
try {
val fileSize = stream.available()
if (fileSize == 0)
throw IOException("Unable to read file $fileUri")
connection = (URL(url).openConnection() as HttpURLConnection).also { conn ->
addHeadersForUpload(conn, authToken, fileSize)
conn.readTimeout = UPLOAD_READ_TIMEOUT
}
var progress = 0
emit(Result.InProgress(fileUri, progress))
BufferedOutputStream(connection.outputStream).use { uploadStream ->
var bytesWritten = 0
val buffer = ByteArray(DEFAULT_UPLOAD_BUFFER_SIZE)
while (true) {
val size = bufferedInputStream.read(buffer)
if (size <= 0)
break
bytesWritten += size
progress = ((100f * bytesWritten) / fileSize).toInt()
emit(Result.InProgress(fileUri, progress))
uploadStream.write(buffer, 0, size)
}
uploadStream.flush()
}
val responseCode = connection.responseCode
val isSuccess = responseCode == 200
if (isSuccess) {
if (progress < 100)
emit(Result.InProgress(fileUri, 100))
emit(Result.Success(fileUri, downloadUrl, readResponseBody(connection)))
} else {
emit(Result.Failure(fileUri))
}
} catch (e: Exception) {
emit(Result.Failure(fileUri))
} finally {
bufferedInputStream.close()
connection?.disconnect()
}
}
}.flowOn(Dispatchers.IO)
private fun readResponseBody(connection: HttpURLConnection): String? {
val builder = StringBuilder()
val lineBreak = System.getProperty("line.separator")
BufferedReader(connection.inputStream.reader()).use {
val line = it.readLine() ?: return#use
builder.append(line + lineBreak)
}
return builder.toString().trim(*lineBreak.toCharArray())
}
Related
I have the request which works well in postman:
and I'm trying to make it with Retrofit. In general file sizes will be >500MB that. I did such uploading method:
fun uploadFile(file:File) {
val client = OkHttpClient().newBuilder()
.build()
val mediaType: MediaType? = "text/plain".toMediaTypeOrNull()
val body: RequestBody = MultipartBody.Builder().setType(MultipartBody.FORM)
.addFormDataPart(
"data", file.name,
file.asRequestBody()
)
.build()
val request: Request = Request.Builder()
.url("https://..../upload.php")
.method("POST", body)
.build()
val response: okhttp3.Response = client.newCall(request).execute()
println(response.message)
}
but I need to have file for uploading. I can create temporary file with such way:
val path = requireContext().cacheDir
val file = File.createTempFile(
name ?: "",
fileUri.lastPathSegment,
path
)
val os = FileOutputStream(file)
os.write(string)
os.close()
but I usually receive outOfMemoryException. I also added to the AndroidManifest.xml heap param:
android:largeHeap="true"
but it didn't help me at all during temp file creating. I don't know how postman uploads files, but in general I managed to upload with his help file with size about 600Mb. I can also cut selected file with chunks:
val data = result.data
data?.let {
val fileUri = data.data
var name: String? = null
var size: Long? = null
fileUri.let { returnUri ->
contentResolver?.query(returnUri!!, null, null, null, null)
}?.use { cursor ->
val nameIndex = cursor.getColumnIndex(OpenableColumns.DISPLAY_NAME)
val sizeIndex = cursor.getColumnIndex(OpenableColumns.SIZE)
cursor.moveToFirst()
name = cursor.getString(nameIndex)
size = cursor.getLong(sizeIndex)
}
val inputStream: InputStream? = fileUri?.let { it1 ->
contentResolver.openInputStream(
it1
)
}
val fileData = inputStream?.readBytes()
val mimeType = fileUri.let { returnUri ->
returnUri.let { retUri ->
if (retUri != null) {
contentResolver.getType(retUri)
}
}
}
fileData?.let {
val MAX_SUB_SIZE = 4194304 // 4*1024*1024 == 4MB
var start = 0 // From 0
var end = MAX_SUB_SIZE // To MAX_SUB_SIZE bytes
var subData: ByteArray // 4MB Sized Array
val max = fileData.size
if (max > 0) {
while (end < max) {
subData = fileData.copyOfRange(start, end)
start = end
end += MAX_SUB_SIZE
if (end >= max) {
end = max
}
println("file handling" + subData.size)
}
end-- // To avoid a padded zero
subData = fileData.copyOfRange(start, end)
println("file handling" + subData.size)
}
}
}
all actions will be made in:
private val filesReceiver =
registerForActivityResult(ActivityResultContracts.StartActivityForResult()) { result ->
if (result.resultCode == Activity.RESULT_OK) {
}
}
so I won't have any file path in normal way. Anyway I think I did something wrong.
UPDATE
right now I have such file uploading from inputStream:
private fun doSomeNetworkStuff(file:InputStream, name:String) {
GlobalScope.launch(Dispatchers.IO) {
val client = OkHttpClient()
.newBuilder()
.protocols(listOf(Protocol.HTTP_1_1))
.connectTimeout(10, TimeUnit.MINUTES)
.readTimeout(10, TimeUnit.MINUTES)
.build()
val mediaType: MediaType? = "text/plain".toMediaTypeOrNull()
val body: RequestBody = MultipartBody.Builder().setType(MultipartBody.FORM)
.addFormDataPart(
"data", name,
file.readBytes().toRequestBody(mediaType)
)
.build()
val request: Request = Request.Builder()
.url("https://.../upload.php")
.method("POST", body)
.build()
val response: Response = client.newCall(request).execute()
println(response.body)
}
}
and receive such error:
java.lang.OutOfMemoryError: Failed to allocate a 173410912 byte allocation with 25165824 free bytes and 89MB until OOM, max allowed footprint 199761800, growth limit 268435456
but I can upload with this code file with size about 90mb
The retrofit multipart stuff has a member that takes an Uri for a request body.
You try to use the one for a File instance.
Have you set log in loggingInterceptor or restadapter ?
if yes then try to set it NONE.
Basically, I am trying to download three different images(bitmaps) from a URL and save them to Apps Internal storage, and then use the URI's from the saved file to save a new Entity to my database. I am having a lot of issues with running this in parallel and getting it to work properly. As ideally all three images would be downloaded, saved and URI's returned simultaneously. Most of my issues come from blocking calls that I cannot seem to avoid.
Here's all of the relevant code
private val okHttpClient: OkHttpClient = OkHttpClient()
suspend fun saveImageToDB(networkImageModel: CBImageNetworkModel): Result<Long> {
return withContext(Dispatchers.IO) {
try {
//Upload all three images to local storage
val edgesUri = this.async {
val req = Request.Builder().url(networkImageModel.edgesImageUrl).build()
val response = okHttpClient.newCall(req).execute() // BLOCKING
val btEdges = BitmapFactory.decodeStream(response.body?.byteStream())
return#async saveBitmapToAppStorage(btEdges, ImageType.EDGES)
}
val finalUri = this.async {
val urlFinal = URL(networkImageModel.finalImageUrl) // BLOCKING
val btFinal = BitmapFactory.decodeStream(urlFinal.openStream())
return#async saveBitmapToAppStorage(btFinal, ImageType.FINAL)
}
val labelUri = this.async {
val urlLabels = URL(networkImageModel.labelsImageUrl)
val btLabel = BitmapFactory.decodeStream(urlLabels.openStream())
return#async saveBitmapToAppStorage(btLabel, ImageType.LABELS)
}
awaitAll(edgesUri, finalUri, labelUri)
if(edgesUri.getCompleted() == null || finalUri.getCompleted() == null || labelUri.getCompleted() == null) {
return#withContext Result.failure(Exception("An image couldn't be saved"))
}
} catch (e: Exception) {
Result.failure<Long>(e)
}
try {
// Result.success( db.imageDao().insertImage(image))
Result.success(123) // A placeholder untill I actually get the URI's to create my Db Entity
} catch (e: Exception) {
Timber.e(e)
Result.failure(e)
}
}
}
//Save the bitmap and return Uri or null if failed
private fun saveBitmapToAppStorage(bitmap: Bitmap, imageType: ImageType): Uri? {
val type = when (imageType) {
ImageType.EDGES -> "edges"
ImageType.LABELS -> "labels"
ImageType.FINAL -> "final"
}
val filename = "img_" + System.currentTimeMillis().toString() + "_" + type
val file = File(context.filesDir, filename)
try {
val fos = file.outputStream()
bitmap.compress(Bitmap.CompressFormat.PNG, 100, fos);
fos.close();
} catch (e: Exception) {
Timber.e(e)
return null
}
return file.toUri()
}
Here I am calling this function
viewModelScope.launch {
val imageID = appRepository.saveImageToDB(imageNetworkModel)
withContext(Dispatchers.Main) {
val uri = Uri.parse("$PAINT_DEEPLINK/$imageID")
navManager.navigate(uri)
}
}
Another issue I am facing is returning the URI in the first place and handling errors. As if one of these parts fails, I'd like to cancel the whole thing and return Result.failure(), but I am unsure on how to achieve that. As returning null just seems meh, I'd much prefer to have an error message or something along those lines.
I am trying to download an mp3 file from an http link and save the file to local storage. All the code I have tried saves a corrupt file that is slightly twice as large as it should be.
File should be 1,204,787
File saved is 2,478,272
The file I am trying to download is: rise-stage.bioinf.unc.edu/cue_audio/sampleaudio.mp3 –
It plays fine when downloaded manually.
fun downloadFilea(url:String , localFileName:String)
{
val request: Request = Request.Builder()
.url(url)
.build()
client.newCall(request).enqueue(object : Callback {
override fun onFailure(call: Call, e: IOException) {
println("error"+e.toString())
}
#Throws(IOException::class)
override fun onResponse(call: Call, response: Response) {
if (!response.isSuccessful) throw IOException("Unexpected code $response")
var uri = dataMgr.getLocalURI(localFileName)
var file = File(uri)
val body = response.body
val contentLength = body!!.contentLength()
val source = body.source()
val DOWNLOAD_CHUNK_SIZE:Long = 2048
val sink: BufferedSink = file.sink().buffer()
var totalRead: Long = 0
var read: Long = 0
while (source.read(sink.buffer(), DOWNLOAD_CHUNK_SIZE).also {
read = it
} != -1L) {
totalRead += read
val progress = (totalRead * 100 / contentLength).toInt()
}
sink.writeAll(source)
sink.flush()
sink.close()
Log.d(logTag, "downloaded file")
}
})
}
I'm trying to upload a file to Google Drive using Google Drive REST API v3. After the upload process is completed, it returns a status code of 200 (successful). But I can't find the files in my Google Drive. Please tell me what am I doing wrong? I will really appreciate if you provide a proper illustration or better still code snippet while helping me with this problem of mine. I am really anticipating your answers.
I have tried following the documentation but I am still getting the same error. I have searched everywhere online and stackoverflow, but none seems to provide the solution to my problem.
here is the code
private val AUTHORIZATION_PARAM = "Authorization"
private val BEARER_VAL = "Bearer "
private val CONTENT_TYPE_PARAM = "Content-Type: "
private val LINE_FEED = "\r\n"
private val APP_FOLDER_ID = "appDataFolder"
fun connectAndStartOperation() {
if (mAuthCode == null) {
signInOptions = GoogleSignInOptions.Builder(GoogleSignInOptions.DEFAULT_SIGN_IN)
.requestEmail()
.requestProfile()
.requestScopes(Scope(Scopes.DRIVE_APPFOLDER))
.requestIdToken(resources.getString(R.string.gdrive_clientId))
.requestServerAuthCode(resources.getString(R.string.gdrive_clientId))
.build()
mGoogleSignInClient = GoogleSignIn.getClient(this, signInOptions!!)
startActivityForResult(mGoogleSignInClient?.signInIntent, CLOUD_STORAGE)
Log.i("mAuthCode", "false")
} else {
Log.i("mAuthCode", "true")
writeDbToDrive()
mNextGoogleApiOperation = INVALID;
}
}
fun disconnect() {
mGoogleSignInClient?.signOut()
mActivity = null
mNextGoogleApiOperation = INVALID
mAuthCode = null
mAccessToken = null
mTokenExpired = 0
}
override fun onDestroy() {
disconnect()
super.onDestroy()
}
public override fun onActivityResult(requestCode: Int, resultCode: Int, data: Intent?) {
super.onActivityResult(requestCode, resultCode, data)
if (requestCode == CLOUD_STORAGE) {
val task = GoogleSignIn.getSignedInAccountFromIntent(data)
.addOnSuccessListener(this)
.addOnFailureListener(this)
}
}
override fun onSuccess(googleSignInAccount: GoogleSignInAccount?) {
Log.i("mAuthCode", "Success")
val scope = "oauth2:https://www.googleapis.com/auth/plus.me https://www.googleapis.com/auth/userinfo.profile"
idTokenString = googleSignInAccount?.idToken
mAuthCode = googleSignInAccount?.serverAuthCode
mGoogleSignInAccount = googleSignInAccount
doAsync {
try {
mAccessToken = GoogleAuthUtil.getToken(this#SettingsActivity, mGoogleSignInAccount?.account, scope)
} catch (e: Exception) {
Log.i("Error AccessToken", "${e.message}")
e.printStackTrace()
}
uiThread {
Log.i("AccessTokenMy", "$mAccessToken")
}
}
}
override fun onFailure(p0: java.lang.Exception) {
Log.i("mAuthCode", "Failed")
p0.printStackTrace()
}
private fun writeDbToDrive() {
var conn: HttpURLConnection? = null
var os: OutputStream? = null
val accessToken = requestAccessToken(mGoogleSignInAccount!!)
if (accessToken == null)
return
try {
val boundary = "pb" + System.currentTimeMillis()
val url = URL("https://www.googleapis.com/upload/drive/v3/files?uploadType=multipart")
conn = url.openConnection() as HttpURLConnection
conn.requestMethod = "POST"
conn.useCaches = false
conn.doOutput = true
conn.doInput = true
conn.connectTimeout = 5000
conn.setRequestProperty(AUTHORIZATION_PARAM, BEARER_VAL + accessToken!!)
conn.setRequestProperty("Content-Type", "multipart/related; boundary=$boundary")
Log.i("Action", "Parameter set for server")
/////// Prepare data
//val timestamp = SimpleDateFormat("yyyy-MM-dd_HH:mm:ss", Locale.getDefault()).format(Date())
// Prepare file metadata (Change your backup file name here)
val b = StringBuilder()
b.append('{')
.append("\"name\":\"").append(exportedFileName).append('\"')
.append(',')
.append("\"mimeType\":").append("\"text\\/csv\"")
.append(',') //"\"application\\/vnd.google-apps.unknown\""
.append("\"parents\":").append("[\"").append(APP_FOLDER_ID).append("\"]")
.append('}')
val metadata = b.toString()
val data = readFile(File(filePath))
/////// Calculate body length
var bodyLength = 0
// MetaData part
b.setLength(0)
b.append("--").append(boundary).append(LINE_FEED)
b.append(CONTENT_TYPE_PARAM).append("application/json; charset=UTF-8")
.append(LINE_FEED)
b.append(LINE_FEED)
b.append(metadata).append(LINE_FEED)
b.append(LINE_FEED)
b.append("--").append(boundary).append(LINE_FEED)
b.append(CONTENT_TYPE_PARAM).append("text/csv").append(LINE_FEED)
b.append(LINE_FEED)
val beforeFilePart = b.toString().toByteArray(charset("UTF_8"))
bodyLength += beforeFilePart.size
bodyLength += data.size // File
b.setLength(0)
b.append(LINE_FEED)
b.append("--").append(boundary).append("--")
val afterFilePart = b.toString().toByteArray(charset("UTF_8"))
bodyLength += afterFilePart.size
conn.setRequestProperty("Content-Length", bodyLength.toString())
//if (BuildConfig.DEBUG_MODE) DebugHelper.log("LENGTH", bodyLength)
/////// Write to socket
os = conn.outputStream
try {
os!!.write(beforeFilePart)
os!!.write(data)
os!!.write(afterFilePart)
os!!.flush()
} catch (e: Exception) {
e.printStackTrace()
}
val msg = conn.responseMessage
val code = conn.responseCode
if (code == 200) {
Log.i("writeDbToDrive", "Exported Successfully: $code $msg")
} else {
Log.i("writeDbToDrive", "Error: $code $msg")
}
} catch (e: Exception) {
e.printStackTrace()
Log.i("writeDbToDrive", e.message!!)
} finally {
if (os != null) {
try {
os!!.close()
} catch (e: IOException) {
}
}
conn?.disconnect()
}
}
#Throws(IOException::class)
private fun readFile(file: File): ByteArray {
val f = RandomAccessFile(file, "r")
try {
val longlength = f.length()
val length = longlength.toInt()
if (length.toLong() != longlength)
throw IOException("File size >= 10 Mb")
val data = ByteArray(length)
f.readFully(data)
return data
} finally {
f.close()
}
}
private fun requestAccessToken(mGoogleSignInAccount: GoogleSignInAccount): String? {
val scope = "oauth2:https://www.googleapis.com/auth/plus.me https://www.googleapis.com/auth/userinfo.profile"
doAsync {
try {
mAccessToken = GoogleAuthUtil.getToken(this#SettingsActivity, mGoogleSignInAccount?.account, scope)
} catch (e: Exception) {
Log.i("Error AccessToken", "${e.message}")
e.printStackTrace()
}
uiThread {
Log.i("AccessTokenMy", "$mAccessToken")
}
}
return mAccessToken
}
After reading through this Files: create Documentation, I have finally fixed the problem. Unknown to me is that the files where being saved in the AppData folder created by my app. The AppData folder is hidden which can only be accessible by and through my app. For me to be able to save the file to My Drive folder, I removed the part of the metadata
` .append(',')
.append("\"parents\":").append("[\"").append(APP_FOLDER_ID).append("\"]")`
So the metadata part is now like this
val b = StringBuilder()
b.append('{')
.append("\"name\":\"").append(exportedFileName).append('\"')
.append(',')
.append("\"mimeType\":").append("\"text\\/csv\"")
.append('}')
val metadata = b.toString()
Every other thing remains the same
I am downloading the file using OkHttp3, I want to see the downloading speed. but I am confused about how to measure the speed.
I tried getting the current millis before reading the buffer and calculating after it is written, but it always returns a static value.
Following is my download function.
fun download(fileName: String) {
val request = Request.Builder().url(url)
.get().build()
val call = OkHttpClient().newCall(request)
val response = call.execute()
if (response.isSuccessful) {
var inputStream: InputStream? = null
try {
inputStream = response.body()?.byteStream()
val buffer = ByteArray(8192)
val mediaFile = File(downloadDir, fileName)
val output = RandomAccessFile(mediaFile, "rw")
output.seek(0)
while (true) {
val readed = inputStream?.read(buffer)
if (readed == -1 || readed == null) {
break
}
output.write(buffer, 0, readed)
downloaded.append(readed.toLong())
}
output.close()
} catch (e: IOException) {
// TODO: handle IOException
console.log("${e.message}")
} finally {
inputStream?.close()
}
}
}
It's a very simple problem, I got confused by overthinking. Anyway here is the solution.
all I need to do is store the downloaded bytes in a variable after 1s subtract downloaded bytes from newly downloaded bytes, that will give me the downloaded bytes in 1s, then I can use those bytes to convert into speed like kbps or Mbps.
fun getSpeed(callback: (String) -> Unit) {
doAsync {
var prevDownloaded = 0L
while (true) {
if (contentLength != null) {
if (downloaded.get() >= contentLength!!) {
break
}
}
if (prevDownloaded != 0L) {
callback(formatBytes(downloaded.get() - prevDownloaded))
}
prevDownloaded = downloaded.get()
Thread.sleep(1000)
}
}
}