I have an app for Android which downloads hundreds of files from the Internet. Some files turn out to be 0-byte after download. The app attempts to detect such cases and delete such files after download but sometimes it fails. The problem is more frequent on Android 4.x devices.
Here is the method which does the downloading. I gets the number of actually read bytes from inputStream.read(buffer).
public class Utils
{
public static class DownloadFileData
{
int nTotalSize;
int nDownloadedSize;
}
public interface ProgressCallback
{
void onProgress(long nCurrent, long nMax);
}
public static boolean downloadFile(String sFileURL, File whereToSave, DownloadFileData fileData, ProgressCallback progressCallback)
{
InputStream inputStream = null;
FileOutputStream fileOutput = null;
try
{
URL url = new URL(sFileURL);
URLConnection connection = url.openConnection();
//set up some things on the connection
connection.setDoOutput(true);
connection.connect();
fileOutput = new FileOutputStream(whereToSave);
inputStream = connection.getInputStream();
fileData.nTotalSize = connection.getContentLength();
fileData.nDownloadedSize = 0;
byte[] buffer = new byte[1024];
int bufferLength = 0; //used to store a temporary size of the buffer
// now, read through the input buffer and write the contents to the file
while ((bufferLength = inputStream.read(buffer)) > 0)
{
// if interrupted, don't download the file further and return
// also restore the interrupted flag so that the caller stopped also
if (Thread.interrupted())
{
Thread.currentThread().interrupt();
return false;
}
// add the data in the buffer to the file in the file output stream
fileOutput.write(buffer, 0, bufferLength);
// add up the size so we know how much is downloaded
fileData.nDownloadedSize += bufferLength;
if (null != progressCallback && fileData.nTotalSize > 0)
{
progressCallback.onProgress(fileData.nDownloadedSize, fileData.nTotalSize);
}
}
return true;
}
catch (FileNotFoundException e)
{
return false; // swallow a 404
}
catch (IOException e)
{
return false; // swallow a 404
}
catch (Throwable e)
{
return false;
}
finally
{
// in any case close input and output streams
if (null != inputStream)
{
try
{
inputStream.close();
inputStream = null;
}
catch (Exception e)
{
}
}
if (null != fileOutput)
{
try
{
fileOutput.close();
fileOutput = null;
}
catch (Exception e)
{
}
}
}
}
Here is the piece of code which processes the downloads. Since sometimes the number of read bytes is incorrect (it is > 0 and the real file has the size 0 bytes) I check the size of the downloaded file with outputFile.length(). But this again gives a value > 0 even though the file is really 0 byte. I tried to also just create a new file and read its size with recheckSizeFile.length(). Still the size is determined as > 0 while it's really 0 byte.
Utils.DownloadFileData fileData = new Utils.DownloadFileData();
boolean bDownloadedSuccessully = Utils.downloadFile(app.sCurrenltyDownloadedFile, outputFile, fileData, new Utils.ProgressCallback()
{
... // progress bar is updated here
});
if (bDownloadedSuccessully)
{
boolean bIsGarbage = false;
File recheckSizeFile = new File(sFullPath);
long nDownloadedFileSize = Math.min(recheckSizeFile.length(), Math.min(outputFile.length(), fileData.nDownloadedSize));
// if the file is 0bytes, it's garbage
if (0 == nDownloadedFileSize)
{
bIsGarbage = true;
}
// if this is a video and if of suspiciously small size, it's
// garbage, too
else if (Utils.isStringEndingWith(app.sCurrenltyDownloadedFile, App.VIDEO_FILE_EXTENSIONS) && nDownloadedFileSize < Constants.MIN_NON_GARBAGE_VIDEO_FILE_SIZE)
{
bIsGarbage = true;
}
if (bIsGarbage)
{
++app.nFilesGarbage;
app.updateLastMessageInDownloadLog("File is fake, deleting: " + app.sCurrenltyDownloadedFile);
// delete the garbage file
if (null != outputFile)
{
if (!outputFile.delete())
{
Log.e("MyService", "Failed to delete garbage file " + app.sCurrenltyDownloadedFile);
}
}
}
else
{
... // process the normally downloaded file
}
I am not sure but I think there is a bug in Android with reading file size. Has anyone seen a similar problem? Or am I maybe doing something wrong here?
Thanks!
EDIT: how i determine that the files are 0-byte:
all the files which get downloaded go thru the described routines. When I then later view the download folder with a file browser (Ghost Commander), some of the files (like maybe 10%) are 0-byte. They can't be played by a video player (shown as "broken file" icon).
It looks to me like your problem is that you only check for "garbage" files if the Utils.downloadFile call returns true. If the download fails in the getInputStream call or the first read, you will have created a file with zero length which will never be deleted.
You should call flush() on your FileOutputStream to ensure that all data is written to the file. This should make your issue with 0-byte files occur less often.
To check for 0 byte files using File.length() should work properly. Can you open a shell (adb shell) on the device and run ls -l to see the byte count displayed by it is 0 (maybe your file manager has some weird issues). Also please debug (or put some log statements) that sFullPath contains the correct file paths. I can't see where sFullPath gets set in your code above and why you don't just use outputFile but recreate another File object.
Related
I made an app Tiddloid and it runs well on my phone (a Meizu with an os based on Android 8.1). It saves modified html files to the original file using the following code:
...
try (ByteArrayInputStream is = new ByteArrayInputStream(data.getBytes(StandardCharsets.UTF_8));
OutputStream os = getContentResolver().openOutputStream(uri)) {
if (os == null)
throw new FileNotFoundException(MainActivity.EXCEPTION_SAF_FILE_NOT_EXISTS);
int len = is.available();
int length, lengthTotal = 0;
byte[] b = new byte[MainActivity.BUF_SIZE];
while ((length = is.read(b)) != -1) {
os.write(b, 0, length);
lengthTotal += length;
}
os.flush();
if (lengthTotal != len)
throw new IOException(MainActivity.EXCEPTION_TRANSFER_CORRUPTED);
failed = false;
runOnUiThread(() -> {
if (tree != null && treeIndex != null) try {
syncTree(tree, id, treeIndex);
} catch (IOException e) {
e.printStackTrace();
}
getInfo(wv);
});
} catch (IOException e) {
e.printStackTrace();
Toast.makeText(TWEditorWV.this, R.string.failed, Toast.LENGTH_SHORT).show();
failed = true;
dumpOnFail(data.getBytes(StandardCharsets.UTF_8), uri);
}
...
Several weeks ago I'm testing it on an Android 11 AVD. I noticed that sometimes there are html codes at the end of the saved file, especially if the new file to be saved is smaller than original:
OOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOO <- this is the original
XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX <- this is the new file
XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXOOO <- the actual saved file
At that time I treated it as a bug of AVD, because this had never happened to other AVDs and my phone. But these days more and more people's posting issues on GitHub, complaining that the same thing's happening. Anyone have idea about this? I even made it throw exceptions if detected that written size not equal to the source bytearray, but it never throws, just making the file corrupted.
UPDATE:
I found this post, which details exactly the same problem I am seeing. It turns out that the fact I am using a Pipe approach in my DocumentsProvider to stream content from DropBox means that ExoPlayer doesn't know the size of the file ahead of time, and so by default was not saving it to the cache.
So I ended up doing what I presume the author did - I created a custom CacheDataSource for these situations that alters the DataSpec.flags variable in the open() method of that class:
public long open(DataSpec dataSpec) throws IOException {
try {
key = cacheKeyFactory.buildCacheKey(dataSpec);
uri = dataSpec.uri;
actualUri = getRedirectedUriOrDefault(cache, key, /* defaultUri= */ uri);
httpMethod = dataSpec.httpMethod;
if ( !dataSpec.isFlagSet(DataSpec.FLAG_ALLOW_CACHING_UNKNOWN_LENGTH) ) { // <-- update here
flags = (dataSpec.flags | DataSpec.FLAG_ALLOW_CACHING_UNKNOWN_LENGTH);
} else {
flags = dataSpec.flags;
}
readPosition = dataSpec.position;
Not the optimum solution, and I also chimed in on the other post with a request for a more supported way to indicate this flag should be set.
But at least now my streamed files are being saved in the cache.
I am implementing a customer CacheDataSourceFactory for ExoPlayer2, in order to implement a cache to store videos streamed to ExoPlayer.
I have reviewed several posts here, this one was helpful in getting the general approach right to have a video cached into the directory of my choice.
I noticed that when handling a URI that resolves to my custom DocumentsProvider, the Cache defined by the CacheDataSourceFactory is only used to store what looks like a "pointer" or "index" file ("cached_content_index.exi"). Looking in that file I see the URI of the video streamed by my custom DocumentsProvider. However the actual video is not in the cache.
Here is the relevant portion of my Provider, it's quite straight forward:
// Return a descriptor that will stream the file
Timber.d("In openDocument of DropboxProvider for Id: %s, streaming from source", documentId);
ParcelFileDescriptor[] pipe;
try {
pipe = ParcelFileDescriptor.createPipe();
// Get input stream for the pipe
DbxDownloader downloader = mDbxClient.files().download(fileMetadata.getPathLower(), fileMetadata.getRev());
new TransferThread(downloader.getInputStream(), new ParcelFileDescriptor.AutoCloseOutputStream(pipe[1]), signal, fileMetadata.getSize()).start();
return pipe[0];
} catch (DbxException dbe) {
Timber.d("Got IDbxException when streaming content: %s", dbe.getMessage());
} catch (IOException ioe) {
Timber.d("Got IOException when streaming content: %s", ioe.getMessage());
} catch (Exception e) {
Timber.d("Got Exception when streaming content: %s", e.getMessage());
}
return null;
And the TransferThread:
private static class TransferThread extends Thread {
final InputStream in;
final OutputStream out;
final CancellationSignal signal;
final long size;
TransferThread(InputStream in, OutputStream out, CancellationSignal signal, long size) {
this.in = in;
this.out = out;
this.signal = signal;
this.size = size;
}
#Override
public void run() {
int biteSize = (8*1024);
if ( size <= (biteSize * 8) ) {
biteSize = Math.max( ((int)(size / (biteSize*2))) * (biteSize * 2), biteSize);
}
Timber.d("TransferThread: File size is: %s, buffer biteSize set to: %d", InTouchUtils.getFormattedFileSize(size), biteSize);
byte[] buf = new byte[biteSize];
int len;
try {
while ( ((len=in.read(buf)) >= 0) && (signal == null || !signal.isCanceled()) ) {
out.write(buf, 0, len);
}
} catch (IOException e) {
// When Glide is used to request a URI where this provider resolves the query,
// it will close the stream out from under us once it has fetched enough bytes
// to render a single frame as an image if the if it is to a video, so
// we swallow that exception here, only logging the error if it isn't that EPIPE
// (broken pipe due to one end being closed) exception.
if ( !(e.getMessage().contains("EPIPE"))) {
Timber.d("TransferThread: Got IOException transferring file: %s", e.getMessage());
}
} finally {
try {
if (in != null) {
in.close();
}
if ( out != null ) {
out.flush();
out.close();
}
Timber.d("TransferThread: Finished streaming file.");
} catch (IOException ioe) {
Timber.d("TransferThread: Got IOException closing file: %s", ioe.getMessage());
}
}
}
}
Again - ExoPlayer seems quite happy with the ParcelFileDescriptor it receives from the DocumentsProvider in this case - it takes the bytes streamed to it and plays the video. I am just not seeing the video file end up in the cache.
I also tried an example streaming a video from my Google Drive (which uses the out-of-the-box documents provider from the SAF), and this time the video did wind up in the cache.
Since they both use the same MediaSource instance - there must be an approach that the Google Docs provider takes so that ExoPlayer knows to place the resulting streamed video in the cache that my custom Dropbox DocumentsProvider is not doing.
Does anyone know how to get to the source code of the DocumentsProvider that ships with the SAF that manages access to Google Docs? I'd like to see what it is doing in its openDocument() method.
Is the fact that the Dropbox provider is utilizing a Pipe in its ParcelFileDescriptor something that ExoPlayer doesn't handle?
Other Ideas?
From my Android app I try to download from the windows Azure blob storage using the following URL: http://iclyps.blob.core.windows.net/broadcasts/23_6.mp4
The resulting file is corrupt when I download it from within my app. Same error occurs when I download it using the default Browser or Chrome. Also from the Easy Downloader app, the same error occurs. Only a download from my PC or using Firefox Beta from the Android device (or emulator), the file is retrieved correctly.
I use the following code (snippet):
try {
HttpURLConnection urlConnection = (HttpURLConnection) url.openConnection();
//set up some things on the connection
urlConnection.setRequestMethod("GET");
urlConnection.setDoOutput(true);
//and connect!
urlConnection.connect();
bis = new BufferedInputStream(urlConnection.getInputStream(), BUFSIZE);
bos = new BufferedOutputStream(
context.openFileOutput(TMPFILE, Context.MODE_PRIVATE), BUFSIZE);
/*
* Read bytes to the buffer in chunks of BUFSIZE bytes until there is nothing more to read.
* Each chunk is written to the output file.
*/
byte[] buf = new byte[BUFSIZE];
int nBytes = 0;
int tBytes = 0;
while ((nBytes = bis.read(buf, 0, BUFSIZE)) > 0) {
bos.write(buf, 0, nBytes);
tBytes += nBytes;
}
if (tBytes == 0) throw new Exception("no bytes received");
bos.flush();
MobyLog.d(TAG, "download succeeded: #bytes = " + Integer.toString(tBytes));
return true;
} catch (Exception e) {
MobyLog.e(TAG, "download failed: " + e);
context.deleteFile(TMPFILE); // remove possibly present partial file.
return false;
} finally {
if (bis != null) try { bis.close(); } catch (IOException e) {MobyLog.e(TAG, "bis close exception: " + e); };
if (bos != null) try { bos.close(); } catch (IOException e) {MobyLog.e(TAG, "bos close exception: " + e); };
}
Analyzing the files shows that the first part (about 700K) of the original file is repeated a number of times in the corrupted files, resulting in an invalid mp4 file.
Putting the file on another webserver (Apache/IIS), and downloading the file from that location does result in a correct download.
Has anyone experienced a similar problem performing a download from Azure? Can someone provide a solution?
Cheers,
Harald...
Have you tried using the azure-sdk-for-java in your android app?
Our scenario is slightly different in that we using the sdk to pull and push images from blob storage to a custom android app. But the fundamentals should be the same.
I need some input about my code.
Basically, I have a method to load music from Class A
public void onListItemClick(ListView parent, View v, int position, long id){
musicIndex = cursor.getColumnIndexOrThrow(MediaStore.Audio.Media.DATA);
cursor.moveToPosition(position);
filePath = cursor.getString(musicIndex);
fileName = new File(filePath).getName();
playMusic();//Play the selected music
}
public void playMusic(){
if(mPlayer.isPlaying()){
mPlayer.reset();
}
try{
mPlayer.setDataSource(filePath);
mPlayer.prepare();
mPlayer.start();
BeatDetection beatDetect = new BeatDetection();
beatDetect.init();
}catch (Exception e){
}
}
That method will call the init() method in Class B
public void init() throws Exception{
energy = 0;
variance = 0;
constant = 0;
isBeat = false;
sensitivity = 0;
dBuffer = new float[sampleRate / bufferSize];
eBuffer = new float[sampleRate / bufferSize];
timer = System.currentTimeMillis();
MusicLoad msc = new MusicLoad();
totalMs = 0;
seeking = true;
//msc.printText();
decode(msc.fileName, 25, 40);
}
In that method, it initializes everything and call the decode() method
public void decode(String path, int startMs, int maxMs)
throws IOException, javazoom.jl.decoder.DecoderException {
debug();
File in = new File(path);
InputStream inStream = new BufferedInputStream(new FileInputStream(in), 8 * 1024);
ByteArrayOutputStream outStream = new ByteArrayOutputStream(1024);
try {
Bitstream bitstream = new Bitstream(inStream);
Decoder decoder = new Decoder();
boolean done = false;
while (! done) {
Header frameHeader = bitstream.readFrame();
if (frameHeader == null) {
done = true;
} else {
totalMs += frameHeader.ms_per_frame();
if (totalMs >= startMs) {
seeking = false;
}
if (! seeking) {
SampleBuffer output = (SampleBuffer) decoder.decodeFrame(frameHeader, bitstream);
if (output.getSampleFrequency() != 44100 || output.getChannelCount() != 2) {
throw new javazoom.jl.decoder.DecoderException("mono or non-44100 MP3 not supported", null);
}
short[] pcm = output.getBuffer();
for (short s : pcm) {
outStream.write(s & 0xff);
outStream.write((s >> 8 ) & 0xff);
}
}
if (totalMs >= (startMs + maxMs)) {
done = true;
}
}
bitstream.closeFrame();
}
byte[] abAudioData = outStream.toByteArray();
calculation(abAudioData);
} catch (BitstreamException e) {
throw new IOException("Bitstream error: " + e);
} catch (DecoderException e) {
Log.w("Decoder error", e);
throw new javazoom.jl.decoder.DecoderException("Error",e);
} finally {
inStream.close();
}
}
Don't mind reading all the code lines. If you guys notice I put debug() in the beginning to see whether the method is called or not. At this point, the debug() is properly called. However, if I put the debug() after the line File in = new File(path);, the debug() will not be called anymore. It seems like the code is stop running at that point.
The ultimate result is, I can load and play the song without any problem. However, the decode() is not called and there is no error whatsoever. I'm stuck at pointing out the problem at this point. So if there's any input please help me.
EDIT: After I tried tracing the "path" variable, it returns NULL so the error is NullPointerException. Seems like the "fileName" variable from Class A is not passed to Class B. Any suggestion?
If you are using Eclipse with ADT then it's very easy to debug your Android apps, just add a breakpoint (probably in the new File(...) line) and see what happens.
My guess here is that File in = new File(path); probably is throwing a IOException in your decode method, that exception is bubbling first to init() and then to playMusic(), where it is caught by try catch block. Your catch is empty so you are not seeing anything. Try debugging as I said or add some logging info in the catch block.
This is just something to look at, but from the doc page
http://developer.android.com/reference/java/io/File.html#File%28java.lang.String%29
"The actual file referenced by a File may or may not exist. It may also, despite the name File, be a directory or other non-regular file."
If you had the path wrong, it may be trying to create the file and you may not have the correct permission to do so. Perhaps: WRITE_EXTERNAL_STORAGE.
I know this post is old, but I just wanted to show how to get the file path to read/write files for others that come across this post as I have:
String filePath = myContext.getFilesDir().getPath().toString() + "/sysout.log";
File file = new File(filePath);
These two lines will create (open if it exists, and overwrite) a file named "sysout.log" in the folder /data/data/com.app.name/files/; myContext is just the current context. Using this technique alleviates problems with defining your own path name. Hope this helps someone.
I'm downloading sets of images in separated threads and saving them to the SD card. The problem is that when I run two or more downloading threads the saved images are corrupted. If just one thread is running the images are ok. I'm downloading it from the same domain but different url, e.g. www.test.com/set1/img1.jpg, www.test.com/set2/img1.jpg etc.
I'm saving them to different folders by the name of the set. I noticed that mostly larger images are corrupted (over 500 KB), smaller are usually ok, but not always.
Do you have any clue why the files get corrupted when multiple threads are running?
Here's a part of the code I'm using:
protected class DownloadTask extends DownloadRunnable {
#Override
public void run() {
InputStream is = null;
OutputStream os = null;
File bitmapFile = null;
/** some more declarations and preparations are here */
for (int pg=getDownloadedPages(); pg < numPages; ++pg) {
for (char ch='a'; ch <= 'e'; ++ch) {
/* check for pause */
synchronized (pauseLock) {
while (paused && !aborted) {
try {
pauseLock.wait();
} catch (InterruptedException e) {
}
}
}
fileName = "page-" + df.format(pg) + "-" + ch;
url = MainApp.getRestrictedUrl(MainApp.tstcode, urlFile + fileName+ ".jpg");
is = new BufferedInputStream(new URL(url).openStream());
if(android.os.Environment.getExternalStorageState().equals(android.os.Environment.MEDIA_MOUNTED)) {
bitmapFile = new File(pathToSave, fileName + MagazinePage.FILE_EXT);
MainApp.encryptToFile(bitmapFile, is);
dwnlSize += bitmapFile.length();
}
is.close();
}
}
}
public static void encryptToFile(File file, InputStream is) throws IOException {
BufferedOutputStream os = null;
try {
if (file.exists()) {
file.delete();
} else {
file.getParentFile().mkdirs();
}
file.createNewFile();
os = new BufferedOutputStream(new FileOutputStream(file));
IkioskContentProvider.getInstance().encrypt(is, os);
} finally {
os.close();
}
}
}
DownloadRunnable is custom abstract class implementing Runnable. And I'm using it in thread the regular way:
protected void downloadIssuePages() {
dwnlTask = new DownloadTask();
new Thread(dwnlTask).start();
}
I'm calling downloadIssuePages() on two different objects to download two sets for example.
Using SDK version 11 (Android 3.0), device Acer Iconia Tab A500 with Android 3.1
I've tried to disable writing for the second thread, only the first thread was saving files to find out if there's a problem in reading from the stream or writing. Apparently writing was the problem as data was correct in this case.
So I decided to use lock around writing to the file and looks like it's working fine:
synchronized (MainApp.fileWritingLockObj) {
while (MainApp.fileWritingLocked) {
try {
MainApp.fileWritingLockObj.wait();
} catch (InterruptedException e) {
}
}
MainApp.fileWritingLocked = true;
if(android.os.Environment.getExternalStorageState().equals(android.os.Environment.MEDIA_MOUNTED)) {
bitmapFile = new File(pathToSave, fileName + MagazinePage.FILE_EXT);
MainApp.encryptToFile(bitmapFile, is);
dwnlSize += bitmapFile.length();
}
is.close();
MainApp.fileWritingLocked = false;
MainApp.fileWritingLockObj.notifyAll();
}