I'm splitting an encrypted video into 4 parts using this code
public class SplitVideoFile {
private static String result;
static ArrayList<String>update=new ArrayList<>();
public static String main(File file) {
try {
// File file = new File("C:/Documents/Despicable Me 2 - Trailer (HD) - YouTube.mp4");//File read from Source folder to Split.
if (file.exists()) {
String videoFileName = file.getName().substring(0, file.getName().lastIndexOf(".")); // Name of the videoFile without extension
// String path = Environment.getDataDirectory().getAbsolutePath().toString() + "/storage/emulated/0/Videointegrity";
String path = "/storage/emulated/0/Videointegrity";
// File myDir = new File(getFile, "folder");
//myDir.mkdir();
File splitFile = new File(path.concat("/").concat(videoFileName));//Destination folder to save.
if (!splitFile.exists()) {
splitFile.mkdirs();
Log.d("Directory Created -> ", splitFile.getAbsolutePath());
}
int i = 01;// Files count starts from 1
InputStream inputStream = new FileInputStream(file);
String videoFile = splitFile.getAbsolutePath() +"/"+ String.format("%02d", i) +"_"+ file.getName();// Location to save the files which are Split from the original file.
OutputStream outputStream = new FileOutputStream(videoFile);
Log.d("File Created Location: ", videoFile);
update.add("File Created Location: ".concat(videoFile));
int totalPartsToSplit =4 ;// Total files to split.
int splitSize = inputStream.available() / totalPartsToSplit;
int streamSize = 0;
int read = 0;
while ((read = inputStream.read()) != -1) {
if (splitSize == streamSize) {
if (i != totalPartsToSplit) {
i++;
String fileCount = String.format("%02d", i); // output will be 1 is 01, 2 is 02
videoFile = splitFile.getAbsolutePath() +"/"+ fileCount +"_"+ file.getName();
outputStream = new FileOutputStream(videoFile);
Log.d("File Created Location: ", videoFile);
streamSize = 0;
}
}
outputStream.write(read);
streamSize++;
}
inputStream.close();
outputStream.close();
Log.d("Total files Split ->", String.valueOf(totalPartsToSplit));
result="success";
} else {
System.err.println(file.getAbsolutePath() +" File Not Found.");
result="failed";
}
} catch (Exception e) {
e.printStackTrace();
}
return result;
}
public ArrayList<String> getUpdate()
{
return update;
}
And in my activity file i call this using async task's doinbackground method like below
protected String doInBackground(Void...arg0) {
Log.d(TAG + " DoINBackGround", "On doInBackground...");
File encvideo=new File(epath.getText().toString());
SplitVideoFile split=new SplitVideoFile();
String result=split.main(encvideo);
publishProgress(1);
return result;
}
Even though it splits the video, it takes too much of time to do the process.
How can I speed them up. As I'm showing a progress bar in preexecute method it looks like the user sees the progress bar for a long time, which I don't want.
I am using android's thread pool executor framework (initialized as below).
BlockingQueue<Runnable> taskQueue = new LinkedBlockingQueue<>();
ExecutorService executorService = new ThreadPoolExecutor(totalCores, totalCores * 3, 10, TimeUnit.SECONDS, taskQueue);
Now, consider the following function onFrameProcessed -
public void onFrameProcessed(RenderedImage renderedImage) {
String timeNow = new SimpleDateFormat("d-M-Y_HH_mm_ss_SSS").format(new Date()).toString();
CustomRunnable3 customRunnable3 = new CustomRunnable3(renderedImage, timeNow);
executorService.execute(customRunnable3);
}
Definition of CustomRunnable3 is as follows:
class CustomRunnable3 implements Runnable {
RenderedImage renderedImageLocal;
String basePath, timeNowCopy;
int hashCode;
CustomRunnable3(RenderedImage renderedImage, String timeNow) {
renderedImageLocal = renderedImage;
this.basePath = Environment.getExternalStoragePublicDirectory(Environment.DIRECTORY_PICTURES).toString();
this.timeNowCopy = timeNow;
hashCode = renderedImageLocal.hashCode();
}
#Override
public void run() {
if (renderedImageLocal.imageType() == RenderedImage.ImageType.ThermalRadiometricKelvinImage) {
int[] thermalData = renderedImageLocal.thermalPixelValues();
String dataPath = basePath + "/" + this.timeNowCopy + ".csv";
try {
PrintWriter printWriter = new PrintWriter(dataPath);
int dataLen = thermalData.length;
for (int i = 0; i < dataLen; i++) {
printWriter.println(thermalData[i]);
}
printWriter.close();
} catch (IOException e) {
e.printStackTrace();
}
String imgPath = basePath + "/" + this.timeNowCopy + ".jpg";
try {
if (hashCode != renderedImageLocal.hashCode()) {
Log.e("Checking", "Hash code changed..");
}
renderedImageLocal.getFrame().save(new File(imgPath), frameProcessor);
if (hashCode != renderedImageLocal.hashCode()) {
Log.e("Checking", "Hash code changed after writing..");
}
} catch (IOException e) {
e.printStackTrace();
}
}
}
}
Usage Scenario : onFrameReceived is being called multiple times per second(like 4-5 times). In each call to onFrameReceived, I am saving two files from renderedImage object (1 csv file, 1 jpg file). Both of these files must be related to each other because both are created from one parent and have same name(except the extension).
Problem : But that is not happening and somehow I am ending up with jpg file content from 1 renderedImage and csv content from another renderedImage object.
What are the possible reasons for this problem, please share your opinion.
I need to store multimedia files in Amazons3.
I used the following code for uploading a the file.
Method 1:
import android.content.Context;
import android.content.SharedPreferences;
import android.content.SharedPreferences.Editor;
import android.os.StrictMode;
import android.util.Log;
import com.amazonaws.services.s3.AmazonS3Client;
import com.amazonaws.services.s3.model.AbortMultipartUploadRequest;
import com.amazonaws.services.s3.model.CannedAccessControlList;
import com.amazonaws.services.s3.model.CompleteMultipartUploadRequest;
import com.amazonaws.services.s3.model.CompleteMultipartUploadResult;
import com.amazonaws.services.s3.model.InitiateMultipartUploadRequest;
import com.amazonaws.services.s3.model.InitiateMultipartUploadResult;
import com.amazonaws.services.s3.model.ObjectMetadata;
import com.amazonaws.services.s3.model.PartETag;
import com.amazonaws.services.s3.model.ProgressEvent;
import com.amazonaws.services.s3.model.ProgressListener;
import com.amazonaws.services.s3.model.UploadPartRequest;
import com.amazonaws.services.s3.model.UploadPartResult;
import java.io.File;
import java.util.ArrayList;
import java.util.List;
public class AmazonUploader {
private static final long MIN_DEFAULT_PART_SIZE = 5 * 1024 * 1024;
private static final String TAG = "AmazonUploader";
private static final String PREFS_NAME = "preferences_simpl3r";
private static final String PREFS_UPLOAD_ID = "_uploadId";
private static final String PREFS_ETAGS = "_etags";
private static final String PREFS_ETAG_SEP = "~~";
private AmazonS3Client s3Client;
private String s3bucketName;
private String s3key;
private File file;
private SharedPreferences prefs;
private long partSize = MIN_DEFAULT_PART_SIZE;
private UploadProgressListener progressListener;
private long bytesUploaded = 0;
private boolean userInterrupted = false;
private boolean userAborted = false;
public AmazonUploader(Context context, AmazonS3Client s3Client, String s3bucketName, String s3key, File file) {
if (android.os.Build.VERSION.SDK_INT > 9) {
StrictMode.ThreadPolicy policy = new StrictMode.ThreadPolicy.Builder().permitAll().build();
StrictMode.setThreadPolicy(policy);
}
this.s3Client = s3Client;
this.s3key = s3key;
this.s3bucketName = s3bucketName;
this.file = file;
prefs = context.getSharedPreferences(PREFS_NAME, Context.MODE_PRIVATE);
}
/**
* Initiate a multipart file upload to Amazon S3
*
* #return the URL of a successfully uploaded file
*/
public String start() {
// initialize
List<PartETag> partETags = new ArrayList<PartETag>();
final long contentLength = file.length();
long filePosition = 0;
int startPartNumber = 1;
userInterrupted = false;
userAborted = false;
bytesUploaded = 0;
// check if we can resume an incomplete download
String uploadId = getCachedUploadId();
if (uploadId != null) {
// we can resume the download
Log.i(TAG, "resuming upload for " + uploadId);
// get the cached etags
List<PartETag> cachedEtags = getCachedPartEtags();
partETags.addAll(cachedEtags);
// calculate the start position for resume
startPartNumber = cachedEtags.size() + 1;
filePosition = (startPartNumber - 1) * partSize;
bytesUploaded = filePosition;
Log.i(TAG, "resuming at part " + startPartNumber + " position " + filePosition);
} else {
// initiate a new multi part upload
Log.i(TAG, "initiating new upload");
InitiateMultipartUploadRequest initRequest = new InitiateMultipartUploadRequest(s3bucketName, s3key);
// ObjectMetadata obj = new ObjectMetadata();
// obj.setContentType("image/jpeg");
// obj.setHeader(Constants.APP_HEADER_REFERER, Constants.APP_REFERER_URL);
// initRequest.setObjectMetadata(obj);
configureInitiateRequest(initRequest);
InitiateMultipartUploadResult initResponse = s3Client.initiateMultipartUpload(initRequest);
uploadId = initResponse.getUploadId();
}
final AbortMultipartUploadRequest abortRequest = new AbortMultipartUploadRequest(s3bucketName, s3key, uploadId);
for (int k = startPartNumber; filePosition < contentLength; k++) {
long thisPartSize = Math.min(partSize, (contentLength - filePosition));
Log.i(TAG, "starting file part " + k + " with size " + thisPartSize);
UploadPartRequest uploadRequest = new UploadPartRequest().withBucketName(s3bucketName)
.withKey(s3key).withUploadId(uploadId)
.withPartNumber(k).withFileOffset(filePosition).withFile(file)
.withPartSize(thisPartSize);
ProgressListener s3progressListener = new ProgressListener() {
public void progressChanged(ProgressEvent progressEvent) {
// bail out if user cancelled
// TODO calling shutdown too brute force?
if (userInterrupted) {
s3Client.shutdown();
throw new UploadIterruptedException("User interrupted");
} else if (userAborted) {
// aborted requests cannot be resumed, so clear any cached etags
clearProgressCache();
s3Client.abortMultipartUpload(abortRequest);
s3Client.shutdown();
}
bytesUploaded += progressEvent.getBytesTransfered();
//Log.d(TAG, "bytesUploaded=" + bytesUploaded);
// broadcast progress
float fpercent = ((bytesUploaded * 100) / contentLength);
int percent = Math.round(fpercent);
if (progressListener != null) {
progressListener.progressChanged(progressEvent, bytesUploaded, percent);
}
}
};
uploadRequest.setProgressListener(s3progressListener);
UploadPartResult result = s3Client.uploadPart(uploadRequest);
partETags.add(result.getPartETag());
// cache the part progress for this upload
if (k == 1) {
initProgressCache(uploadId);
}
// store part etag
cachePartEtag(result);
filePosition += thisPartSize;
}
CompleteMultipartUploadRequest compRequest = new CompleteMultipartUploadRequest(
s3bucketName, s3key, uploadId,
partETags);
CompleteMultipartUploadResult result = s3Client.completeMultipartUpload(compRequest);
bytesUploaded = 0;
Log.i(TAG, "upload complete for " + uploadId);
clearProgressCache();
return result.getLocation();
}
private String getCachedUploadId() {
return prefs.getString(s3key + PREFS_UPLOAD_ID, null);
}
private List<PartETag> getCachedPartEtags() {
List<PartETag> result = new ArrayList<PartETag>();
// get the cached etags
ArrayList<String> etags = SharedPreferencesUtils.getStringArrayPref(prefs, s3key + PREFS_ETAGS);
for (String etagString : etags) {
String partNum = etagString.substring(0, etagString.indexOf(PREFS_ETAG_SEP));
String partTag = etagString.substring(etagString.indexOf(PREFS_ETAG_SEP) + 2, etagString.length());
PartETag etag = new PartETag(Integer.parseInt(partNum), partTag);
result.add(etag);
}
return result;
}
private void cachePartEtag(UploadPartResult result) {
String serialEtag = result.getPartETag().getPartNumber() + PREFS_ETAG_SEP + result.getPartETag().getETag();
ArrayList<String> etags = SharedPreferencesUtils.getStringArrayPref(prefs, s3key + PREFS_ETAGS);
etags.add(serialEtag);
SharedPreferencesUtils.setStringArrayPref(prefs, s3key + PREFS_ETAGS, etags);
}
private void initProgressCache(String uploadId) {
// store uploadID
Editor edit = prefs.edit().putString(s3key + PREFS_UPLOAD_ID, uploadId);
AmazonSharedPreferencesCompact.apply(edit);
// create empty etag array
ArrayList<String> etags = new ArrayList<String>();
SharedPreferencesUtils.setStringArrayPref(prefs, s3key + PREFS_ETAGS, etags);
}
private void clearProgressCache() {
// clear the cached uploadId and etags
Editor edit = prefs.edit();
edit.remove(s3key + PREFS_UPLOAD_ID);
edit.remove(s3key + PREFS_ETAGS);
AmazonSharedPreferencesCompact.apply(edit);
}
public void interrupt() {
userInterrupted = true;
}
public void abort() {
userAborted = true;
}
/**
* Override to configure the multipart upload request.
* <p/>
* By default uploaded files are publicly readable.
*
* #param initRequest S3 request object for the file to be uploaded
*/
protected void configureInitiateRequest(InitiateMultipartUploadRequest initRequest) {
initRequest.setCannedACL(CannedAccessControlList.PublicRead);
ObjectMetadata obj = new ObjectMetadata();
obj.setContentType("image/jpeg");
obj.setHeader(Constants.APP_HEADER_REFERER, Constants.APP_REFERER_URL);
initRequest.withObjectMetadata(obj);
}
public void setPrefs(SharedPreferences prefs) {
this.prefs = prefs;
}
public long getPartSize() {
return partSize;
}
public void setPartSize(long partSize) {
if (partSize < MIN_DEFAULT_PART_SIZE) {
throw new IllegalStateException("Part size is less than S3 minimum of " + MIN_DEFAULT_PART_SIZE);
} else {
this.partSize = partSize;
}
}
public void setProgressListener(UploadProgressListener progressListener) {
this.progressListener = progressListener;
}
public interface UploadProgressListener {
public void progressChanged(ProgressEvent progressEvent, long bytesUploaded, int percentUploaded);
}
}
Method 2:
TransferObserver transferObserver = transferUtility.upload(
Constants.S3_BUCKET_NAME, /* The bucket to upload to */
fileName, /* The key for the uploaded object */
new File(imagePath), /* The file where the data to upload exists */
objectMetadata);
transferObserverListener(transferObserver);
in both method i got the following error
com.amazonaws.services.s3.model.AmazonS3Exception: Forbidden (Service: Amazon S3; Status Code: 403; Error Code: 403 Forbidden; Request...
Here i must pass customer header parameters, so i add like following
ObjectMetadata objectMetadata = new ObjectMetadata();
HashMap<String, String> mMetaMap = new HashMap<String, String>();
mMetaMap.put("content-type", "image/jpeg");
mMetaMap.put(Constants.APP_HEADER_REFERER, Constants.APP_REFERER_URL);
objectMetadata.setUserMetadata(mMetaMap);
But still i got the above error.
Is i'm passing the header parameters in correct way either i need to do changes. Kindly advise on this. Thanks
I have a photo editing android app that users can choose the output directory of the the result photos. Problem is Google made a change on sdcard write permission with the KITKAT version and devices with Android KITKAT version won't allow apps to write secondary sdcards. Now I need to check if the choosen directory by user has granted the permission and won't throw EACCES error. I am already checking canRead and canWrite but these won't help. Could you please tell me how can I check if the choosen directory won't throw EACCES. My only solution is trying to write a file in a try catch, however I am hoping there is better way to do it.
[update k3b 2016-09-19]
i tried this on my android-4.4 but without success
Uri uri = Uri.fromFile(file);
int permissionCode =
context.checkCallingOrSelfUriPermission(uri,
Intent.FLAG_GRANT_WRITE_URI_PERMISSION);
if (permissionCode == PackageManager.PERMISSION_DENIED) {
// on my android-4.4 i always get PERMISSION_DENIED even
// if i can overwrite the file
return false;
}
try {
Process p = new ProcessBuilder("ls", "-l", "-s", dir.getCanonicalPath()).start();
String line;
ArrayList<String> lineOut = new ArrayList<>();
BufferedReader error = new BufferedReader(new InputStreamReader(p.getErrorStream()));
while ((line = error.readLine()) != null) {
Log.e(TAG, "ls error = "+line);
}
error.close();
BufferedReader input = new BufferedReader(new InputStreamReader(p.getInputStream()));
while ((line = input.readLine()) != null) {
lineOut.add(line);
}
input.close();
String[] strings = lineOut.toArray(new String[]{});
List<FilesLS.FileEntry> fileEntries = FilesLS.processNewLines(strings);
for(FilesLS.FileEntry file : fileEntries){
Log.d(TAG, file.name +" = " + file.permissions);
}
} catch (IOException e) {
e.printStackTrace();
}
And some edits to this class
import java.util.ArrayList;
import java.util.List;
import java.util.regex.Matcher;
import java.util.regex.Pattern;
public final class FilesLS {
/**
* Entry type: File
*/
public static final int TYPE_FILE = 0;
/**
* Entry type: Directory
*/
public static final int TYPE_DIRECTORY = 1;
/**
* Entry type: Directory Link
*/
public static final int TYPE_DIRECTORY_LINK = 2;
/**
* Entry type: Block
*/
public static final int TYPE_BLOCK = 3;
/**
* Entry type: Character
*/
public static final int TYPE_CHARACTER = 4;
/**
* Entry type: Link
*/
public static final int TYPE_LINK = 5;
/**
* Entry type: Socket
*/
public static final int TYPE_SOCKET = 6;
/**
* Entry type: FIFO
*/
public static final int TYPE_FIFO = 7;
/**
* Entry type: Other
*/
public static final int TYPE_OTHER = 8;
/**
* Device side file separator.
*/
public static final String FILE_SEPARATOR = "/"; //$NON-NLS-1$
/**
* Regexp pattern to parse the result from ls.
*/
private static Pattern sLsPattern = Pattern
.compile("^([bcdlsp-][-r][-w][-xsS][-r][-w][-xsS][-r][-w][-xstST])\\s+(\\S+)\\s+ (\\S+)\\s+(\\d{4}-\\d\\d-\\d\\d)\\s+(\\d\\d:\\d\\d)\\s+(.*)$"); //$NON-NLS-1$ \s+([\d\s,]*)
public static List<FileEntry> processNewLines(String[] lines) {
List<FileEntry> listOfFiles = new ArrayList<FileEntry>();
for (String line : lines) {
// no need to handle empty lines.
if (line.length() == 0) {
continue;
}
// run the line through the regexp
Matcher m = sLsPattern.matcher(line);
if (m.matches() == false) {
continue;
}
// get the name
String name = m.group(6);
// get the rest of the groups
String permissions = m.group(1);
String owner = m.group(2);
String group = m.group(3);
// String size = m.group(4);
String date = m.group(4);
String time = m.group(5);
String info = null;
// and the type
int objectType = TYPE_OTHER;
switch (permissions.charAt(0)) {
case '-':
objectType = TYPE_FILE;
break;
case 'b':
objectType = TYPE_BLOCK;
break;
case 'c':
objectType = TYPE_CHARACTER;
break;
case 'd':
objectType = TYPE_DIRECTORY;
break;
case 'l':
objectType = TYPE_LINK;
break;
case 's':
objectType = TYPE_SOCKET;
break;
case 'p':
objectType = TYPE_FIFO;
break;
}
// now check what we may be linking to
if (objectType == TYPE_LINK) {
String[] segments = name.split("\\s->\\s"); //$NON-NLS-1$
// we should have 2 segments
if (segments.length == 2) {
// update the entry name to not contain the link
name = segments[0];
// and the link name
info = segments[1];
// now get the path to the link
String[] pathSegments = info.split(FILE_SEPARATOR);
if (pathSegments.length == 1) {
// the link is to something in the same directory,
// unless the link is ..
if ("..".equals(pathSegments[0])) { //$NON-NLS-1$
// set the type and we're done.
objectType = TYPE_DIRECTORY_LINK;
} else {
// either we found the object already
// or we'll find it later.
}
}
}
// add an arrow in front to specify it's a link.
info = "-> " + info; //$NON-NLS-1$;
}
FileEntry entry = new FileEntry();
entry.permissions = permissions;
entry.name = name;
// entry.size = size;
entry.date = date;
entry.time = time;
entry.owner = owner;
entry.group = group;
if (objectType == TYPE_LINK) {
entry.info = info;
}
listOfFiles.add(entry);
}
return listOfFiles;
}
public final static class FileEntry {
String name;
String info;
String permissions;
String size;
String date;
String time;
String owner;
String group;
int type;
}
}
Add the permission(s) you need to the array:
private static final int REQUEST_CODE_PERMISSION = 2;
String[] mPermission = {
Manifest.permission.INTERNET,
Manifest.permission.CHANGE_WIFI_STATE,
Manifest.permission.CHANGE_NETWORK_STATE,
Manifest.permission.ACCESS_WIFI_STATE
};
Add this to onCreate or where you want it to be:
try {
if (
ActivityCompat.checkSelfPermission(this, mPermission[0])
!= MockPackageManager.PERMISSION_GRANTED ||
ActivityCompat.checkSelfPermission(this, mPermission[1])
!= MockPackageManager.PERMISSION_GRANTED ||
ActivityCompat.checkSelfPermission(this, mPermission[2])
!= MockPackageManager.PERMISSION_GRANTED ||
ActivityCompat.checkSelfPermission(this, mPermission[3])
!= MockPackageManager.PERMISSION_GRANTED
) {
Log.e("TAGTAG", "DENIED");
ActivityCompat.requestPermissions(
this, mPermission, REQUEST_CODE_PERMISSION
);
// 'Will execute recursively if any of the permissions was not granted.
} else {
Log.e("TAGTAG", "GRANTED");
}
} catch (Exception e) {
e.printStackTrace();
}
Don't forget to declare the permissions in AndroidManifest.xml.
In my Photo Collage app for Android I'm using Volley for loading images.
I'm using the DiskBasedCache (included with volley) with 50 mb storage to prevent re-downloading the same images multiple times.
Last time I checked the DiskBasedCache contained about 1000 cache entries.
When my app starts Volley calls mCache.initialize() and it will spend about 10 seconds (!) on my Galaxy S4 to do the following:
List all files in cache folder
Open each and every file and read the header section.
I find that reading 1000+ files at startup is not a very efficient way to load the cache index! :-)
From volley/toolbox/DiskBasedCache.java:
#Override
public synchronized void initialize() {
if (!mRootDirectory.exists()) {
if (!mRootDirectory.mkdirs()) {
VolleyLog.e("Unable to create cache dir %s", mRootDirectory.getAbsolutePath());
}
return;
}
File[] files = mRootDirectory.listFiles();
if (files == null) {
return;
}
for (File file : files) {
FileInputStream fis = null;
try {
fis = new FileInputStream(file);
CacheHeader entry = CacheHeader.readHeader(fis);
entry.size = file.length();
putEntry(entry.key, entry);
} catch (IOException e) {
if (file != null) {
file.delete();
}
} finally {
try {
if (fis != null) {
fis.close();
}
} catch (IOException ignored) { }
}
}
}
I'm looking for a fast and scalable solution. Perhaps an alternative DiskBasedCache implementation or suggestions on how to improve the volley library.
Update: (2014-01-06)
Noticing that the Volley cache used a lot of small (1 byte) IO read/writes. I cloned DiskBasedCache.java and encapsulating all FileInputStreams and FileOutputStreams with BufferedInputStream and BufferedOutputStreams. I found that that this optimization gave me a 3-10 times speed up.
This modification has a low risks of bugs compared to writing a new disk cache with a central index file.
Update: (2014-01-10)
Here is new class BufferedDiskBasedCache.java that I'm using now.
package no.ludde.android.ds.android.volley;
/*
* Copyright (C) 2011 The Android Open Source Project
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
import android.os.SystemClock;
import com.android.volley.Cache;
import com.android.volley.VolleyLog;
import java.io.BufferedInputStream;
import java.io.BufferedOutputStream;
import java.io.EOFException;
import java.io.File;
import java.io.FileInputStream;
import java.io.FileOutputStream;
import java.io.FilterInputStream;
import java.io.IOException;
import java.io.InputStream;
import java.io.OutputStream;
import java.util.Collections;
import java.util.HashMap;
import java.util.Iterator;
import java.util.LinkedHashMap;
import java.util.Map;
/**
* Cache implementation that caches files directly onto the hard disk in the specified
* directory. The default disk usage size is 5MB, but is configurable.
*/
public class BufferedDiskBasedCache implements Cache {
/** Map of the Key, CacheHeader pairs */
private final Map<String, CacheHeader> mEntries =
new LinkedHashMap<String, CacheHeader>(16, .75f, true);
/** Total amount of space currently used by the cache in bytes. */
private long mTotalSize = 0;
/** The root directory to use for the cache. */
private final File mRootDirectory;
/** The maximum size of the cache in bytes. */
private final int mMaxCacheSizeInBytes;
/** Default maximum disk usage in bytes. */
private static final int DEFAULT_DISK_USAGE_BYTES = 5 * 1024 * 1024;
/** High water mark percentage for the cache */
private static final float HYSTERESIS_FACTOR = 0.9f;
/** Magic number for current version of cache file format. */
private static final int CACHE_MAGIC = 0x20120504;
/**
* Constructs an instance of the DiskBasedCache at the specified directory.
* #param rootDirectory The root directory of the cache.
* #param maxCacheSizeInBytes The maximum size of the cache in bytes.
*/
public BufferedDiskBasedCache(File rootDirectory, int maxCacheSizeInBytes) {
mRootDirectory = rootDirectory;
mMaxCacheSizeInBytes = maxCacheSizeInBytes;
}
/**
* Constructs an instance of the DiskBasedCache at the specified directory using
* the default maximum cache size of 5MB.
* #param rootDirectory The root directory of the cache.
*/
public BufferedDiskBasedCache(File rootDirectory) {
this(rootDirectory, DEFAULT_DISK_USAGE_BYTES);
}
/**
* Clears the cache. Deletes all cached files from disk.
*/
#Override
public synchronized void clear() {
File[] files = mRootDirectory.listFiles();
if (files != null) {
for (File file : files) {
file.delete();
}
}
mEntries.clear();
mTotalSize = 0;
VolleyLog.d("Cache cleared.");
}
/**
* Returns the cache entry with the specified key if it exists, null otherwise.
*/
#Override
public synchronized Entry get(String key) {
CacheHeader entry = mEntries.get(key);
// if the entry does not exist, return.
if (entry == null) {
return null;
}
File file = getFileForKey(key);
CountingInputStream cis = null;
try {
cis = new CountingInputStream(new BufferedInputStream(new FileInputStream(file)));
CacheHeader.readHeader(cis); // eat header
byte[] data = streamToBytes(cis, (int) (file.length() - cis.bytesRead));
return entry.toCacheEntry(data);
} catch (IOException e) {
VolleyLog.d("%s: %s", file.getAbsolutePath(), e.toString());
remove(key);
return null;
} finally {
if (cis != null) {
try {
cis.close();
} catch (IOException ioe) {
return null;
}
}
}
}
/**
* Initializes the DiskBasedCache by scanning for all files currently in the
* specified root directory. Creates the root directory if necessary.
*/
#Override
public synchronized void initialize() {
if (!mRootDirectory.exists()) {
if (!mRootDirectory.mkdirs()) {
VolleyLog.e("Unable to create cache dir %s", mRootDirectory.getAbsolutePath());
}
return;
}
File[] files = mRootDirectory.listFiles();
if (files == null) {
return;
}
for (File file : files) {
BufferedInputStream fis = null;
try {
fis = new BufferedInputStream(new FileInputStream(file));
CacheHeader entry = CacheHeader.readHeader(fis);
entry.size = file.length();
putEntry(entry.key, entry);
} catch (IOException e) {
if (file != null) {
file.delete();
}
} finally {
try {
if (fis != null) {
fis.close();
}
} catch (IOException ignored) { }
}
}
}
/**
* Invalidates an entry in the cache.
* #param key Cache key
* #param fullExpire True to fully expire the entry, false to soft expire
*/
#Override
public synchronized void invalidate(String key, boolean fullExpire) {
Entry entry = get(key);
if (entry != null) {
entry.softTtl = 0;
if (fullExpire) {
entry.ttl = 0;
}
put(key, entry);
}
}
/**
* Puts the entry with the specified key into the cache.
*/
#Override
public synchronized void put(String key, Entry entry) {
pruneIfNeeded(entry.data.length);
File file = getFileForKey(key);
try {
BufferedOutputStream fos = new BufferedOutputStream(new FileOutputStream(file));
CacheHeader e = new CacheHeader(key, entry);
e.writeHeader(fos);
fos.write(entry.data);
fos.close();
putEntry(key, e);
return;
} catch (IOException e) {
}
boolean deleted = file.delete();
if (!deleted) {
VolleyLog.d("Could not clean up file %s", file.getAbsolutePath());
}
}
/**
* Removes the specified key from the cache if it exists.
*/
#Override
public synchronized void remove(String key) {
boolean deleted = getFileForKey(key).delete();
removeEntry(key);
if (!deleted) {
VolleyLog.d("Could not delete cache entry for key=%s, filename=%s",
key, getFilenameForKey(key));
}
}
/**
* Creates a pseudo-unique filename for the specified cache key.
* #param key The key to generate a file name for.
* #return A pseudo-unique filename.
*/
private String getFilenameForKey(String key) {
int firstHalfLength = key.length() / 2;
String localFilename = String.valueOf(key.substring(0, firstHalfLength).hashCode());
localFilename += String.valueOf(key.substring(firstHalfLength).hashCode());
return localFilename;
}
/**
* Returns a file object for the given cache key.
*/
public File getFileForKey(String key) {
return new File(mRootDirectory, getFilenameForKey(key));
}
/**
* Prunes the cache to fit the amount of bytes specified.
* #param neededSpace The amount of bytes we are trying to fit into the cache.
*/
private void pruneIfNeeded(int neededSpace) {
if ((mTotalSize + neededSpace) < mMaxCacheSizeInBytes) {
return;
}
if (VolleyLog.DEBUG) {
VolleyLog.v("Pruning old cache entries.");
}
long before = mTotalSize;
int prunedFiles = 0;
long startTime = SystemClock.elapsedRealtime();
Iterator<Map.Entry<String, CacheHeader>> iterator = mEntries.entrySet().iterator();
while (iterator.hasNext()) {
Map.Entry<String, CacheHeader> entry = iterator.next();
CacheHeader e = entry.getValue();
boolean deleted = getFileForKey(e.key).delete();
if (deleted) {
mTotalSize -= e.size;
} else {
VolleyLog.d("Could not delete cache entry for key=%s, filename=%s",
e.key, getFilenameForKey(e.key));
}
iterator.remove();
prunedFiles++;
if ((mTotalSize + neededSpace) < mMaxCacheSizeInBytes * HYSTERESIS_FACTOR) {
break;
}
}
if (VolleyLog.DEBUG) {
VolleyLog.v("pruned %d files, %d bytes, %d ms",
prunedFiles, (mTotalSize - before), SystemClock.elapsedRealtime() - startTime);
}
}
/**
* Puts the entry with the specified key into the cache.
* #param key The key to identify the entry by.
* #param entry The entry to cache.
*/
private void putEntry(String key, CacheHeader entry) {
if (!mEntries.containsKey(key)) {
mTotalSize += entry.size;
} else {
CacheHeader oldEntry = mEntries.get(key);
mTotalSize += (entry.size - oldEntry.size);
}
mEntries.put(key, entry);
}
/**
* Removes the entry identified by 'key' from the cache.
*/
private void removeEntry(String key) {
CacheHeader entry = mEntries.get(key);
if (entry != null) {
mTotalSize -= entry.size;
mEntries.remove(key);
}
}
/**
* Reads the contents of an InputStream into a byte[].
* */
private static byte[] streamToBytes(InputStream in, int length) throws IOException {
byte[] bytes = new byte[length];
int count;
int pos = 0;
while (pos < length && ((count = in.read(bytes, pos, length - pos)) != -1)) {
pos += count;
}
if (pos != length) {
throw new IOException("Expected " + length + " bytes, read " + pos + " bytes");
}
return bytes;
}
/**
* Handles holding onto the cache headers for an entry.
*/
// Visible for testing.
static class CacheHeader {
/** The size of the data identified by this CacheHeader. (This is not
* serialized to disk. */
public long size;
/** The key that identifies the cache entry. */
public String key;
/** ETag for cache coherence. */
public String etag;
/** Date of this response as reported by the server. */
public long serverDate;
/** TTL for this record. */
public long ttl;
/** Soft TTL for this record. */
public long softTtl;
/** Headers from the response resulting in this cache entry. */
public Map<String, String> responseHeaders;
private CacheHeader() { }
/**
* Instantiates a new CacheHeader object
* #param key The key that identifies the cache entry
* #param entry The cache entry.
*/
public CacheHeader(String key, Entry entry) {
this.key = key;
this.size = entry.data.length;
this.etag = entry.etag;
this.serverDate = entry.serverDate;
this.ttl = entry.ttl;
this.softTtl = entry.softTtl;
this.responseHeaders = entry.responseHeaders;
}
/**
* Reads the header off of an InputStream and returns a CacheHeader object.
* #param is The InputStream to read from.
* #throws IOException
*/
public static CacheHeader readHeader(InputStream is) throws IOException {
CacheHeader entry = new CacheHeader();
int magic = readInt(is);
if (magic != CACHE_MAGIC) {
// don't bother deleting, it'll get pruned eventually
throw new IOException();
}
entry.key = readString(is);
entry.etag = readString(is);
if (entry.etag.equals("")) {
entry.etag = null;
}
entry.serverDate = readLong(is);
entry.ttl = readLong(is);
entry.softTtl = readLong(is);
entry.responseHeaders = readStringStringMap(is);
return entry;
}
/**
* Creates a cache entry for the specified data.
*/
public Entry toCacheEntry(byte[] data) {
Entry e = new Entry();
e.data = data;
e.etag = etag;
e.serverDate = serverDate;
e.ttl = ttl;
e.softTtl = softTtl;
e.responseHeaders = responseHeaders;
return e;
}
/**
* Writes the contents of this CacheHeader to the specified OutputStream.
*/
public boolean writeHeader(OutputStream os) {
try {
writeInt(os, CACHE_MAGIC);
writeString(os, key);
writeString(os, etag == null ? "" : etag);
writeLong(os, serverDate);
writeLong(os, ttl);
writeLong(os, softTtl);
writeStringStringMap(responseHeaders, os);
os.flush();
return true;
} catch (IOException e) {
VolleyLog.d("%s", e.toString());
return false;
}
}
}
private static class CountingInputStream extends FilterInputStream {
private int bytesRead = 0;
private CountingInputStream(InputStream in) {
super(in);
}
#Override
public int read() throws IOException {
int result = super.read();
if (result != -1) {
bytesRead++;
}
return result;
}
#Override
public int read(byte[] buffer, int offset, int count) throws IOException {
int result = super.read(buffer, offset, count);
if (result != -1) {
bytesRead += result;
}
return result;
}
}
/*
* Homebrewed simple serialization system used for reading and writing cache
* headers on disk. Once upon a time, this used the standard Java
* Object{Input,Output}Stream, but the default implementation relies heavily
* on reflection (even for standard types) and generates a ton of garbage.
*/
/**
* Simple wrapper around {#link InputStream#read()} that throws EOFException
* instead of returning -1.
*/
private static int read(InputStream is) throws IOException {
int b = is.read();
if (b == -1) {
throw new EOFException();
}
return b;
}
static void writeInt(OutputStream os, int n) throws IOException {
os.write((n >> 0) & 0xff);
os.write((n >> 8) & 0xff);
os.write((n >> 16) & 0xff);
os.write((n >> 24) & 0xff);
}
static int readInt(InputStream is) throws IOException {
int n = 0;
n |= (read(is) << 0);
n |= (read(is) << 8);
n |= (read(is) << 16);
n |= (read(is) << 24);
return n;
}
static void writeLong(OutputStream os, long n) throws IOException {
os.write((byte)(n >>> 0));
os.write((byte)(n >>> 8));
os.write((byte)(n >>> 16));
os.write((byte)(n >>> 24));
os.write((byte)(n >>> 32));
os.write((byte)(n >>> 40));
os.write((byte)(n >>> 48));
os.write((byte)(n >>> 56));
}
static long readLong(InputStream is) throws IOException {
long n = 0;
n |= ((read(is) & 0xFFL) << 0);
n |= ((read(is) & 0xFFL) << 8);
n |= ((read(is) & 0xFFL) << 16);
n |= ((read(is) & 0xFFL) << 24);
n |= ((read(is) & 0xFFL) << 32);
n |= ((read(is) & 0xFFL) << 40);
n |= ((read(is) & 0xFFL) << 48);
n |= ((read(is) & 0xFFL) << 56);
return n;
}
static void writeString(OutputStream os, String s) throws IOException {
byte[] b = s.getBytes("UTF-8");
writeLong(os, b.length);
os.write(b, 0, b.length);
}
static String readString(InputStream is) throws IOException {
int n = (int) readLong(is);
byte[] b = streamToBytes(is, n);
return new String(b, "UTF-8");
}
static void writeStringStringMap(Map<String, String> map, OutputStream os) throws IOException {
if (map != null) {
writeInt(os, map.size());
for (Map.Entry<String, String> entry : map.entrySet()) {
writeString(os, entry.getKey());
writeString(os, entry.getValue());
}
} else {
writeInt(os, 0);
}
}
static Map<String, String> readStringStringMap(InputStream is) throws IOException {
int size = readInt(is);
Map<String, String> result = (size == 0)
? Collections.<String, String>emptyMap()
: new HashMap<String, String>(size);
for (int i = 0; i < size; i++) {
String key = readString(is).intern();
String value = readString(is).intern();
result.put(key, value);
}
return result;
}
}
Yes, the way DiskBasedCache works it needs to open all the files in initialize(). Which is simply.... not a good idea :-(
You need to make a different implementation that doesent open all the files at startup.
Take a copy of DiskBasedCache and change initialize() to
#Override
public synchronized void initialize() {
if (!mRootDirectory.exists()) {
if (!mRootDirectory.mkdirs()) {
VolleyLog.e("Unable to create cache dir %s", mRootDirectory.getAbsolutePath());
}
}
}
And change get() so it makes an additional check for if the file exists on the file system, like
#Override
public synchronized Entry get(String key) {
CacheHeader entry = mEntries.get(key);
File file = getFileForKey(key);
if (entry == null && !file.exists()) { // EXTRA CHECK
// if the entry does not exist, return.
VolleyLog.d("DrVolleyDiskBasedCache miss for " + key);
return null;
}
...
I use this approach in https://play.google.com/store/apps/details?id=dk.dr.radio and it works fine - its robustness have been tested by ~300000 users :-)
You can download a full version of the file from https://code.google.com/p/dr-radio-android/source/browse/trunk/DRRadiov3/src/dk/dr/radio/net/volley/DrDiskBasedCache.java (you'll have to delete some DR Radio specific stuff)
In the streamToBytes(), first it will new bytes by the cache file length, does your cache file was too large than application maximum heap size ?
private static byte[] streamToBytes(InputStream in, int length) throws IOException {
byte[] bytes = new byte[length];
...
}
public synchronized Entry get(String key) {
CacheHeader entry = mEntries.get(key);
File file = getFileForKey(key);
byte[] data = streamToBytes(..., file.length());
}
If you want to clear the cache, you could keep the DiskBasedCache reference, after clear time's came, use ClearCacheRequest and pass that cache instance in :
File cacheDir = new File(context.getCacheDir(), DEFAULT_CACHE_DIR);
DiskBasedCache cache = new DiskBasedCache(cacheDir);
RequestQueue queue = new RequestQueue(cache, network);
queue.start();
// clear all volley caches.
queue.add(new ClearCacheRequest(cache, null));
this way will clear all caches, so I suggest you use it carefully. of course, you can doing conditional check, just iterating the cacheDir files, estimate which was too large then remove it.
for (File cacheFile : cacheDir.listFiles()) {
if (cacheFile.isFile() && cacheFile.length() > 10000000) cacheFile.delete();
}
Volley wasn't design as a big data cache solution, it's common request cache, don't storing large data anytime.
------------- Update at 2014-07-17 -------------
In fact, clear all caches is final way, also isn't wise way, we should suppressing these large request use cache when we sure it would be, and if not sure? we still can determine the response data size whether large or not, then call setShouldCache(false) to disable it.
public class TheRequest extends Request {
#Override
protected Response<String> parseNetworkResponse(NetworkResponse response) {
// if response data was too large, disable caching is still time.
if (response.data.length > 10000) setShouldCache(false);
...
}
}
My initial thought was to use the DiskLruCache written by Jake Wharton by writing a com.android.volley.Cache wrapper over it.
But I finally implemented a singleton pattern for the Volley, combined with the cache creation in an AsyncTask called from the Application context
public static synchronized VolleyClient getInstance(Context context)
{
if (mInstance == null)
{
mInstance = new VolleyClient(context);
}
return mInstance;
}
private VolleyClient(Context context)
{
this.context = context;
VolleyCacheInitializer volleyCacheInitializer = new VolleyCacheInitializer();
volleyCacheInitializer.execute();
}
private class VolleyCacheInitializer extends AsyncTask<Void, Void, Boolean>
{
#Override
protected Boolean doInBackground(Void... params)
{
// Instantiate the cache with 50MB Cache Size
Cache diskBasedCache = new DiskBasedCache(context.getCacheDir(), 50 * 1024 * 1024);
// Instantiate the RequestQueue with the cache and network.
mRequestQueue = new RequestQueue(diskBasedCache, network);
// Start the queue which calls the DiskBasedCache.initialize()
mRequestQueue.start();
return true;
}
#Override
protected void onPostExecute(Boolean aBoolean)
{
super.onPostExecute(aBoolean);
if(aBoolean)
Log.d(TAG, "Volley request queue initialized");
else
Log.d(TAG, "Volley request queue initialization failed");
}
}
Inside MyApplication class
#Override
public void onCreate()
{
super.onCreate();
// Initialize an application level volley request queue
VolleyClient volleyHttpClient = VolleyClient.getInstance(this);
}