My interface:
public interface Downloader {
#Streaming #GET Call<ResponseBody> get(#Url final String url);
}
My download method:
private void download(final String url, final File zip) throws IOException {
AsyncHelper.assertNotMainThread();
final Call<ResponseBody> call = downloaderInstance.get(url);
final Response<ResponseBody> response = call.execute();
if (!response.isSuccessful()) {
downloadFailed(response.code());
return;
}
final ResponseBody body = response.body();
downloadedBytes = 0;
totalBytes = body.contentLength();
final InputStream in;
final OutputStream fout;
in = body().byteStream();
fout = new FileOutputStream(zip);
int read;
while ((read = in.read(BUFFER)) != -1) {
fout.write(BUFFER, 0, read);
downloadedBytes += read;
LOG.d("wrote %d bytes (total: %d)", downloadedBytes, totalBytes);
}
body.close();
fout.close();
}
This whole thing runs on a background thread and I don't get any log lines with "wrote x bytes (total: y)" in the log, which tells me that it's not streaming.
In a previous implementation I was running this stream directly into a ZipInputStream to unzip on the fly, I split the process into download and decompress steps thinking that the ZipInputStream might be the problem. The process fails during download, which means once I fix this I can revert back to unzip on the fly.
What am I doing wrong?
I've also tried final InputStream in = response.body().source().inputStream(); with the same results. I've also tried enqueue instead of execute with the same results.
So, this is embarrassing, but I'll leave it here in case others come across a similar scenario.
The answer to my question is: "Nothing". I'm doing everything right. At least according to the above code.
What I were doing wrong though was that for debugging purposes I had logging level set to BODY, which means it has to put the whole thing in a buffer. That's also why no incremental progress was being reported.
Related
I am using Cronet API with our current API stack, specifically UploadDataProvider, there is a ByteBuffer with preset limit, seems like the limit size is fixed and we need to pass the data chunk by chunk. Our current API uses InputStream, and write chunk to OutputStream. We're using following code to work with infinite size of file:
byte[] buf = new byte[16 * 1024];
int bytesRead;
while ((bytesRead =inputStream.read(buffer)) != -1) {
outputStream.write(buffer, 0, bytesRead);
}
I'd like to achieve the same for this Cronet API, UploadDataProvider. My plan was in its read(UploadDataSink, ByteBuffer) method, whenever this read() method was called, read ByteBuffer's limit from inputStream, but my following code not working as expected.
public class MyUploadDataProvider extends UploadDataProvider {
private ReadableByteChannel byteChannel;
MyUploadDataProvider(InputStream inputStream) {
byteChannel = Channels.newChannel(inputStream);
}
#Override
public void read(UploadDataSink uploadDataSink, ByteBuffer byteBuffer) throws IOException {
boolean finalChunk = false;
int read = this.byteChannel.read(byteBuffer);
if (read == -1) {
finalChunk = true;
}
uploadDataSink.onReadSucceeded(finalChunk);
}
}
Not sure why it read failed, can anyone please help me fix this? Thanks!
I have to upload big video files to a server, but it's taking too long to upload, so I decided to split/chunk the files and then send them to the server
After splitting my files, I get a response like the following:
[ /storage/emulated/0/1493357699.mp4.001, /storage/emulated/0/1493357699.mp4.002, /storage/emulated/0/1493357699.mp4.003, /storage/emulated/0/1493357699.mp4.004, /storage/emulated/0/1493357699.mp4.005, /storage/emulated/0/1493357699.mp4.006, /storage/emulated/0/1493357699.mp4.007, /storage/emulated/0/1493357699.mp4.008 ]
My thought is what is the use to upload spitting/chunk file to server?
My code for splitting files:
public static List<File> splitFile(File f) {
try {
int partCounter = 1;
List<File> result = new ArrayList<>();
int sizeOfFiles = 1024 * 1024;// 1MB
byte[] buffer = new byte[sizeOfFiles];
// create a buffer of bytes sized as the one chunk size
BufferedInputStream bis = new BufferedInputStream(new FileInputStream(f));
String name = f.getName();
int tmp = 0;
while ((tmp = bis.read(buffer)) > 0) {
File newFile = new File(f.getParent(), name + "." + String.format("%03d", partCounter++));
// naming files as <inputFileName>.001, <inputFileName>.002, ...
FileOutputStream out = new FileOutputStream(newFile);
out.write(buffer, 0, tmp);//tmp is chunk size. Need it for the last chunk,
// which could be less then 1 mb.
result.add(newFile);
}
return result;
} catch (Throwable throwable) {
throwable.printStackTrace();
}
return null;
}
I have implemented in one of my projects. I see two primary reasons:
To achieve multi-threaded / multiple connection for uploading chunks. You can upload multiple chunks at the same time.
Stop/Resume uploading of rest of the chunks if either of the chunk fails to upload (depending on server response)
I am trying to post a large video (nearly 1 GB).
I am using FTP to send video to a server, but the upload stops after a while. On the server the video crashes, but I am able to upload a smaller sized video.
I've also used HTTP to send video to the server, sent as a Base64 enoded string, but there is an out of memory exception while encoding.
I've tried to upload the video as a file, but without success. What is the best way to upload a large video to a server?
Use HTTP POST, and post content as Form-based File Upload (mime type: multipart/form-data). This system is standard on web for sending forms and/or uploading files.
Use HTTP chunked post mode, so that size doesn't need to be known beforehand, and you can stream any file in small parts. You still have to make some code on server (e.g. in PHP) to accept the file and do what is needed.
Use HttpURLConnection to initiate connection. Then use my attached class to send the data. It will create proper headers, etc, and you'll use it as OutputStream to write your raw data to it, then call close, and you're done. You can overrite its onHandleResult to handle resulting error code.
public class FormDataWriter extends FilterOutputStream{
private final HttpURLConnection con;
/**
* #param formName name of form in which data are sent
* #param fileName
* #param fileSize size of file, or -1 to use chunked encoding
*/
FormDataWriter(HttpURLConnection con, String formName, String fileName, long fileSize) throws IOException{
super(null);
this.con = con;
con.setDoOutput(true);
String boundary = generateBoundary();
con.setRequestProperty(HTTP.CONTENT_TYPE, "multipart/form-data; charset=UTF-8; boundary="+boundary);
{
StringBuilder sb = new StringBuilder();
writePartHeader(boundary, formName, fileName==null ? null : "filename=\""+fileName+"\"",
"application/octet-stream", sb);
headerBytes = sb.toString().getBytes("UTF-8");
sb = new StringBuilder();
sb.append("\r\n");
sb.append("--"+boundary+"--\r\n");
footerBytes = sb.toString().getBytes();
}
if(fileSize!=-1) {
fileSize += headerBytes.length + footerBytes.length;
con.setFixedLengthStreamingMode((int)fileSize);
}else
con.setChunkedStreamingMode(0x4000);
out = con.getOutputStream();
}
private byte[] headerBytes, footerBytes;
private String generateBoundary() {
StringBuilder sb = new StringBuilder();
Random rand = new Random();
int count = rand.nextInt(11) + 30;
int N = 10+26+26;
for(int i=0; i<count; i++) {
int r = rand.nextInt(N);
sb.append((char)(r<10 ? '0'+r : r<36 ? 'a'+r-10 : 'A'+r-36));
}
return sb.toString();
}
private void writePartHeader(String boundary, String name, String extraContentDispositions, String contentType, StringBuilder sb) {
sb.append("--"+boundary+"\r\n");
sb.append("Content-Disposition: form-data; charset=UTF-8; name=\""+name+"\"");
if(extraContentDispositions!=null)
sb.append("; ").append(extraContentDispositions);
sb.append("\r\n");
if(contentType!=null)
sb.append("Content-Type: "+contentType+"\r\n");
sb.append("\r\n");
}
#Override
public void write(byte[] buffer, int offset, int length) throws IOException{
if(headerBytes!=null) {
out.write(headerBytes);
headerBytes = null;
}
out.write(buffer, offset, length);
}
#Override
public void close() throws IOException{
flush();
if(footerBytes!=null) {
out.write(footerBytes);
footerBytes = null;
}
super.close();
int code = con.getResponseCode();
onHandleResult(code);
}
protected void onHandleResult(int code) throws IOException{
if(code!=200 && code!=201)
throw new IOException("Upload error code: "+code);
}
}
I guess it failed because of a timeout by the big size.
Since
Small size video uploaded successfully
, My suggestion is
Split one big file to several small files.
Upload one by one or several together based on the condition of network.
Join all of parts (after all of those uploaded successfully) at server.
Because of small size, Re-upload failed part will be easy.
Just a theroy.
This site may help .
Added 08.01.2013
It has been a while, don't know if you still need this. Anyway, I wrote some simple codes implement the theory above, because of interest mainly.
Split one big file to several small files. Read the big file into several small parts.
ByteBuffer bb = ByteBuffer.allocate(partSize);
int bytesRead = fc.read(bb);
if (bytesRead == -1) {
break;
}
byte[] bytes = bb.array();
parts.put(new Part(createFileName(fileName, i), bytes));
Upload one by one or several together based on the condition of network.
Part part = parts.take();
if (part == Part.NULL) {
parts.add(Part.NULL);// notify others to stop.
break;
} else {
uploader.upload(part);
}
Join all of parts (after all of those uploaded successfully) at server.
Because it is via HTTP, so it can be in any language, such as Java, PHP, Python, etc. Here is a java example.
...
try (FileOutputStream dest = new FileOutputStream(destFile, true)) {
FileChannel dc = dest.getChannel();// the final big file.
for (long i = 0; i < count; i++) {
File partFile = new File(destFileName + "." + i);// every small parts.
if (!partFile.exists()) {
break;
}
try (FileInputStream part = new FileInputStream(partFile)) {
FileChannel pc = part.getChannel();
pc.transferTo(0, pc.size(), dc);// combine.
}
partFile.delete();
}
statusCode = OK;// set ok at last.
} catch (Exception e) {
log.error("combine failed.", e);
}
I put all codes on GitHub. And made a Android example too.
Please have a look if you still need.
private HttpsURLConnection conn = null;
conn.setDoInput(true);
conn.setDoOutput(true);
conn.setUseCaches(false);
conn.setChunkedStreamingMode(1024);
Is there any way to execute Binary file in an android application. without JNI wrapper approach.
plz give me sample codes.
try this
public void pump(InputStream in, OutputStream out, int size) {
byte[] buffer = new byte[4096]; // Or whatever constant you feel like using
int done = 0;
while (done < size) {
int read = in.read(buffer);
if (read == -1) {
throw new IOException("Something went horribly wrong");
}
out.write(buffer, 0, read);
done += read;
}
// Maybe put cleanup code in here if you like, e.g. in.close, out.flush, out.close
}
from this link Reading and writing binary file in Java (seeing half of the file being corrupted)
I am trying to upload a file (20MB of size) but while uploading, logcat shows
Out of Memory Exception
So I thought to use NDK for this. But i dont know how to proceed. So help me on this
static int chunkSize = 512;
static final byte[] chunks = new byte[chunkSize];
.....
......
while (true)
{
synchronized (chunks)
{
int amountRead = fileInputStream.read(chunks);
System.out.println("========amount read========="+amountRead);
if (amountRead == -1)
{
break;
}
bufferOutputStream.write(chunks, 0, amountRead);
bufferOutputStream.flush();
}
}
When you upload, you need to use an InputStream rather than loading the whole file into memory.