Develop Reference

Android - Best Practice for Downloading Medium - Large Files Quickly

I need to download a few large zip files into my application (each approx 25mb) however it seems quite slow (5 minutes +) and when we test the same files being downloaded on an iPad it's downloading several times faster. I've considered using Volley, however it seems asynctask is the best for large files (from what I've read).
Does anyone have any suggestions or ideas on how I might be able to download/write these files faster?
My current implementation is show below:
My AsyncTaskExample:
#Override
protected String doInBackground(String... sUrl) {
InputStream input = null;
OutputStream output = null;
HttpURLConnection connection = null;
try {
URL url = new URL(sUrl[0]);
connection = (HttpURLConnection) url.openConnection();
connection.connect();
// expect HTTP 200 OK, so we don't mistakenly save error report
// instead of the file
if (connection.getResponseCode() != HttpURLConnection.HTTP_OK) {
return "Server returned HTTP " + connection.getResponseCode()
+ " " + connection.getResponseMessage();
}
// this will be useful to display download percentage
// might be -1: server did not report the length
int fileLength = connection.getContentLength();
// download the file
input = connection.getInputStream();
output = new FileOutputStream("/sdcard/file_name.extension");
byte data[] = new byte[1024];
long total = 0;
int count;
while ((count = input.read(data)) != -1) {
// allow canceling with back button
if (isCancelled()) {
input.close();
return null;
}
total += count;
// publishing the progress....
if (fileLength > 0) // only if total length is known
publishProgress((int) (total * 100 / fileLength));
output.write(data, 0, count);
}
} catch (Exception e) {
return e.toString();
} finally {
try {
if (output != null)
output.close();
if (input != null)
input.close();
} catch (IOException ignored) {
}
if (connection != null)
connection.disconnect();
}
return null;
}

AsyncTask should be used only for relatively short background processes (i.e. processes that last a few seconds). From the docs:
AsyncTask is designed to be a helper class around Thread and Handler
and does not constitute a generic threading framework. AsyncTasks
should ideally be used for short operations (a few seconds at the
most.) If you need to keep threads running for long periods of time,
it is highly recommended you use the various APIs provided by the
java.util.concurrent package such as Executor, ThreadPoolExecutor and
FutureTask.
For long operations you should use a Service:
A Service is an application component representing either an
application's desire to perform a longer-running operation while not
interacting with the user or to supply functionality for other
applications to use.

Check integrity of Database SQLite Android

I have a SQLite database file in my server, and from time to time my Android App checks if there is a new SQLite database file. If true the App downloads the File and replaces the old database.
The problem is, that some times the new database file gets corrupted and the App start to crashing and never recovers if I dont manualy clean the app in the Android Settings.
My question is, there is a way to check the integrity of SQLite Database after the Downloaded?
This is my code for download the new Database from the server this code is placed in an AssyncTask :
protected Boolean doInBackground(String... Url) {
try {
URL url = null;
if(Url[0].equals("")){
mSyncDate = mConnectionManager.getSyncDate();
url = new URL(Constants.HF_SERVER_DATABASE+"db_fxbus_"+convertDateToFormatYYYYMMDD(mSyncDate.getServerDate())+".sqlite");
}else{
url = new URL(Url[0]);
}
URLConnection connection = url.openConnection();
connection.connect();
// this will be useful so that you can show a typical 0-100% progress bar
int fileLength = connection.getContentLength();
mDB.getReadableDatabase();
// download the file
InputStream input = new BufferedInputStream(url.openStream());
Log.i(TAG, "Path:"+mContext.getDatabasePath("HorariosDoFunchal").getAbsolutePath());
OutputStream output = new FileOutputStream(mContext.getDatabasePath("HorariosDoFunchal").getAbsolutePath());
startWriting = true;
byte data[] = new byte[1024];
long total = 0;
int count;
while ((count = input.read(data)) != -1) {
total += count;
// publishing the progress....
publishProgress((int) (total * 100 / fileLength));
output.write(data, 0, count);
//Log.i(TAG, "Executing ...");
}
//Log.i(TAG, "Finish ...");
output.flush();
output.close();
input.close();
} catch (Exception e) {
Log.e(TAG, e.toString());
return false;
}
return true;
}

Look into:
pragma integrity_check;
it will scan the Database and check it for errors and other things too.
More info(and more commands) can be found at this link:
http://www.sqlite.org/pragma.html
also check out the documentation of isDatabaseIntegrityOk().

You could try to use PRAGMA integrity_check (or Android's equivalent isDatabaseIntegrityOk()), but this checks only the database structure for errors, and can detect only errors where it can prove that the structure is wrong.
To be able to detect all errors (especially in your own data), you need to compute a checksum for the entire database file.

0-byte files not detected when downloading files on Android

I have an app for Android which downloads hundreds of files from the Internet. Some files turn out to be 0-byte after download. The app attempts to detect such cases and delete such files after download but sometimes it fails. The problem is more frequent on Android 4.x devices.
Here is the method which does the downloading. I gets the number of actually read bytes from inputStream.read(buffer).
public class Utils
{
public static class DownloadFileData
{
int nTotalSize;
int nDownloadedSize;
}
public interface ProgressCallback
{
void onProgress(long nCurrent, long nMax);
}
public static boolean downloadFile(String sFileURL, File whereToSave, DownloadFileData fileData, ProgressCallback progressCallback)
{
InputStream inputStream = null;
FileOutputStream fileOutput = null;
try
{
URL url = new URL(sFileURL);
URLConnection connection = url.openConnection();
//set up some things on the connection
connection.setDoOutput(true);
connection.connect();
fileOutput = new FileOutputStream(whereToSave);
inputStream = connection.getInputStream();
fileData.nTotalSize = connection.getContentLength();
fileData.nDownloadedSize = 0;
byte[] buffer = new byte[1024];
int bufferLength = 0; //used to store a temporary size of the buffer
// now, read through the input buffer and write the contents to the file
while ((bufferLength = inputStream.read(buffer)) > 0)
{
// if interrupted, don't download the file further and return
// also restore the interrupted flag so that the caller stopped also
if (Thread.interrupted())
{
Thread.currentThread().interrupt();
return false;
}
// add the data in the buffer to the file in the file output stream
fileOutput.write(buffer, 0, bufferLength);
// add up the size so we know how much is downloaded
fileData.nDownloadedSize += bufferLength;
if (null != progressCallback && fileData.nTotalSize > 0)
{
progressCallback.onProgress(fileData.nDownloadedSize, fileData.nTotalSize);
}
}
return true;
}
catch (FileNotFoundException e)
{
return false; // swallow a 404
}
catch (IOException e)
{
return false; // swallow a 404
}
catch (Throwable e)
{
return false;
}
finally
{
// in any case close input and output streams
if (null != inputStream)
{
try
{
inputStream.close();
inputStream = null;
}
catch (Exception e)
{
}
}
if (null != fileOutput)
{
try
{
fileOutput.close();
fileOutput = null;
}
catch (Exception e)
{
}
}
}
}
Here is the piece of code which processes the downloads. Since sometimes the number of read bytes is incorrect (it is > 0 and the real file has the size 0 bytes) I check the size of the downloaded file with outputFile.length(). But this again gives a value > 0 even though the file is really 0 byte. I tried to also just create a new file and read its size with recheckSizeFile.length(). Still the size is determined as > 0 while it's really 0 byte.
Utils.DownloadFileData fileData = new Utils.DownloadFileData();
boolean bDownloadedSuccessully = Utils.downloadFile(app.sCurrenltyDownloadedFile, outputFile, fileData, new Utils.ProgressCallback()
{
... // progress bar is updated here
});
if (bDownloadedSuccessully)
{
boolean bIsGarbage = false;
File recheckSizeFile = new File(sFullPath);
long nDownloadedFileSize = Math.min(recheckSizeFile.length(), Math.min(outputFile.length(), fileData.nDownloadedSize));
// if the file is 0bytes, it's garbage
if (0 == nDownloadedFileSize)
{
bIsGarbage = true;
}
// if this is a video and if of suspiciously small size, it's
// garbage, too
else if (Utils.isStringEndingWith(app.sCurrenltyDownloadedFile, App.VIDEO_FILE_EXTENSIONS) && nDownloadedFileSize < Constants.MIN_NON_GARBAGE_VIDEO_FILE_SIZE)
{
bIsGarbage = true;
}
if (bIsGarbage)
{
++app.nFilesGarbage;
app.updateLastMessageInDownloadLog("File is fake, deleting: " + app.sCurrenltyDownloadedFile);
// delete the garbage file
if (null != outputFile)
{
if (!outputFile.delete())
{
Log.e("MyService", "Failed to delete garbage file " + app.sCurrenltyDownloadedFile);
}
}
}
else
{
... // process the normally downloaded file
}
I am not sure but I think there is a bug in Android with reading file size. Has anyone seen a similar problem? Or am I maybe doing something wrong here?
Thanks!
EDIT: how i determine that the files are 0-byte:
all the files which get downloaded go thru the described routines. When I then later view the download folder with a file browser (Ghost Commander), some of the files (like maybe 10%) are 0-byte. They can't be played by a video player (shown as "broken file" icon).

It looks to me like your problem is that you only check for "garbage" files if the Utils.downloadFile call returns true. If the download fails in the getInputStream call or the first read, you will have created a file with zero length which will never be deleted.

You should call flush() on your FileOutputStream to ensure that all data is written to the file. This should make your issue with 0-byte files occur less often.
To check for 0 byte files using File.length() should work properly. Can you open a shell (adb shell) on the device and run ls -l to see the byte count displayed by it is 0 (maybe your file manager has some weird issues). Also please debug (or put some log statements) that sFullPath contains the correct file paths. I can't see where sFullPath gets set in your code above and why you don't just use outputFile but recreate another File object.

AsyncTask - slow download

I am using AsyncTask to download ~50 MB files from internet. Sometimes, when I download this file, progress bar gain is very slow (even when I am on Wi-Fi). And after minute, phone shows me, download complete, but the file itself has only ~100kB, no more. But when I restart device, and try to download file, download is executed briefly and quick. Has anyone faced same problem? Do I need to erase same download memory before downloading new file? I am downloading file to Environment.externalStoryDirectory().
Thx
Calling download from activity:
mProgressDialog = new ProgressDialog(ItemDetails.this);
mProgressDialog.setTitle("Downloading");
mProgressDialog.setMessage("Downloading sth...");
mProgressDialog.setIndeterminate(false);
mProgressDialog.setMax(100);
mProgressDialog.setProgressStyle(ProgressDialog.STYLE_HORIZONTAL);
DownloadMapTask downloadFile = new DownloadMapTask(ItemDetails.this);
downloadFile.execute(web_location_url);
mProgressDialog.show();
Download Async Task (two methods):
#Override
protected String doInBackground(String... urls) {
int count;
PATH=maps_loc+"/Android/data/test/maps/";
try {
URL url = new URL(urls[0]);
HttpURLConnection connection2 = (HttpURLConnection) url.openConnection();
connection2.setRequestMethod("GET");
connection2.setDoOutput(true);
connection2.connect();
int lenghtOfFile = connection2.getContentLength();
File apkdir = new File(PATH);
apkdir.mkdirs();
File newInstall = new File(PATH, name+".tmp");
InputStream input = new BufferedInputStream(url.openStream());
OutputStream output = new FileOutputStream(newInstall);
byte data[] = new byte[1024];
long total = 0;
while ((count = input.read(data)) != -1 && running==true) {
total += count;
publishProgress((int) (total * 100 / lenghtOfFile));
output.write(data, 0, count);
}
output.flush();
output.close();
input.close();
} catch (Exception e) {
e.printStackTrace();
}
return null;
}
public void onProgressUpdate(Integer... args) {
ItemDetails.mProgressDialog.setProgress(args[0]);
}

Some servers will close the connection if the client has slow speed and the download takes long time, which can be the case if your program is connected to the Internet through mobile data not Wi-Fi.
You should consider supporting download resume in your program to not start from scratch every time.
I do not think there is sort of download memory that you need to clear. I have an app that can easily downloads over 50MB with no problems.
Also, you might consider obtaining a lock for both Wi-Fi and processor to keep your program running until the download finishes.
Edit
In your code, try to print the value lenghtOfFile after the line int lenghtOfFile = connection2.getContentLength(); to make sure that it is the same as the actual file size you are downloading.
Below is alternative example code which supports resume that I am using in my projects. (it is just to illustrate the idea, you will need to modify the code to your needs)
HttpClient httpClient = new DefaultHttpClient();
HttpGet request = new HttpGet(new URI(fileURL)));
HttpResponse response;
InputStream is = null;
FileOutputStream fos = null;
try {
boolean continueDownloading = false;
String tmpFileName = fileName + "_tmp";
outputFile = new File(downloadFolder, tmpFileName);
if (outputFile.exists()) {
localFileLength = outputFile.length();
if (localFileLength > 0) {
continueDownloading = true;
}
if (continueDownloading) {
request.addHeader("Range", "bytes=" + localFileLength + "-");
}
response = httpClient.execute(request);
long remoteFileLength = 0;
Header contentLengthHeader = response.getFirstHeader("Content-Length");
if (contentLengthHeader != null) {
remoteFileLength = Integer.parseInt(contentLengthHeader.getValue());
}
long downloaded = 0;
if (continueDownloading) {
downloaded = localFileLength;
}
long fullFileLength = downloaded + remoteFileLength;
fos = new FileOutputStream(outputFile, true);
is = response.getEntity().getContent();
byte[] buffer = new byte[DOWNLOAD_BUFFER_SIZE];
int len = 0;
while ((len = is.read(buffer)) != -1 && isDownloading) {
fos.write(buffer, 0, len);
downloaded += len;
}
fos.flush();
boolean success = downloaded == fullFileLength;
if (success) {
outputFile.renameTo(new File(downloadFolder, fileName));
}
} catch (Throwable ex) {
ex.printStackTrace();
} finally {
// clean up resources
}

Try using downloadManager instead of downloading manually , there are many advantages to using it.
Here is an example for it : DownloadManager Example
and take a look at the documentations : DownloadManager

Simultaneous downloads result in corrupted files

I'm downloading sets of images in separated threads and saving them to the SD card. The problem is that when I run two or more downloading threads the saved images are corrupted. If just one thread is running the images are ok. I'm downloading it from the same domain but different url, e.g. www.test.com/set1/img1.jpg, www.test.com/set2/img1.jpg etc.
I'm saving them to different folders by the name of the set. I noticed that mostly larger images are corrupted (over 500 KB), smaller are usually ok, but not always.
Do you have any clue why the files get corrupted when multiple threads are running?
Here's a part of the code I'm using:
protected class DownloadTask extends DownloadRunnable {
#Override
public void run() {
InputStream is = null;
OutputStream os = null;
File bitmapFile = null;
/** some more declarations and preparations are here */
for (int pg=getDownloadedPages(); pg < numPages; ++pg) {
for (char ch='a'; ch <= 'e'; ++ch) {
/* check for pause */
synchronized (pauseLock) {
while (paused && !aborted) {
try {
pauseLock.wait();
} catch (InterruptedException e) {
}
}
}
fileName = "page-" + df.format(pg) + "-" + ch;
url = MainApp.getRestrictedUrl(MainApp.tstcode, urlFile + fileName+ ".jpg");
is = new BufferedInputStream(new URL(url).openStream());
if(android.os.Environment.getExternalStorageState().equals(android.os.Environment.MEDIA_MOUNTED)) {
bitmapFile = new File(pathToSave, fileName + MagazinePage.FILE_EXT);
MainApp.encryptToFile(bitmapFile, is);
dwnlSize += bitmapFile.length();
}
is.close();
}
}
}
public static void encryptToFile(File file, InputStream is) throws IOException {
BufferedOutputStream os = null;
try {
if (file.exists()) {
file.delete();
} else {
file.getParentFile().mkdirs();
}
file.createNewFile();
os = new BufferedOutputStream(new FileOutputStream(file));
IkioskContentProvider.getInstance().encrypt(is, os);
} finally {
os.close();
}
}
}
DownloadRunnable is custom abstract class implementing Runnable. And I'm using it in thread the regular way:
protected void downloadIssuePages() {
dwnlTask = new DownloadTask();
new Thread(dwnlTask).start();
}
I'm calling downloadIssuePages() on two different objects to download two sets for example.
Using SDK version 11 (Android 3.0), device Acer Iconia Tab A500 with Android 3.1

I've tried to disable writing for the second thread, only the first thread was saving files to find out if there's a problem in reading from the stream or writing. Apparently writing was the problem as data was correct in this case.
So I decided to use lock around writing to the file and looks like it's working fine:
synchronized (MainApp.fileWritingLockObj) {
while (MainApp.fileWritingLocked) {
try {
MainApp.fileWritingLockObj.wait();
} catch (InterruptedException e) {
}
}
MainApp.fileWritingLocked = true;
if(android.os.Environment.getExternalStorageState().equals(android.os.Environment.MEDIA_MOUNTED)) {
bitmapFile = new File(pathToSave, fileName + MagazinePage.FILE_EXT);
MainApp.encryptToFile(bitmapFile, is);
dwnlSize += bitmapFile.length();
}
is.close();
MainApp.fileWritingLocked = false;
MainApp.fileWritingLockObj.notifyAll();
}