Some header parameters should be help to detect correct content length.
/* Get download file size returned from http server header. */
public static long getDownloadUrlFileSize(String downloadUrl) {
long ret = 0;
HttpURLConnection con = null;
try {
if (downloadUrl != null && !TextUtils.isEmpty(downloadUrl)) {
URL url = new URL(downloadUrl);
con = (HttpURLConnection) url.openConnection();
con.setRequestProperty("Accept-Encoding", "identity;q=1, *;q=0");
System.out.println("Length : " + con.getContentLength());
ret = con.getContentLength();
}
} catch (Exception ex) {
Log.e(TAG_DOWNLOAD_MANAGER, ex.getMessage(), ex);
} finally {
if (con != null)
con.disconnect();
return ret;
}
}
Related
I want to download a file with his url.
I use an AsyncTask with HttpURLConnection but when I get response code, server return error 403.
I use the HttpURLConnection in doInBackground.
Code :
#Override
protected String doInBackground(String... sUrl) {
InputStream input = null;
OutputStream output = null;
HttpURLConnection connection = null;
try {
ext = FilenameUtils.getExtension(sUrl[0]);
fileName = FilenameUtils.getBaseName(sUrl[0]);
Log.i("Brieg", "storage : /storage/emulated/0/" + fileName + "." + ext);
URL url = new URL(sUrl[0]);
connection = (HttpURLConnection) url.openConnection();
connection.connect();
if (connection.getResponseCode() != HttpURLConnection.HTTP_OK) {
return "Server returned HTTP " + connection.getResponseCode() + " " + connection.getResponseMessage();
}
int fileLength = connection.getContentLength();
input = connection.getInputStream();
output = new FileOutputStream("/storage/emulated/0/" + fileName + "." + ext);
byte data[] = new byte[4096];
long total = 0;
int count;
while ((count = input.read(data)) != -1) {
if (isCancelled()) {
input.close();
return null;
}
total += count;
if (fileLength > 0)
publishProgress((int) (total * 100 / fileLength));
output.write(data, 0, count);
}
}
catch (MalformedURLException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
catch (IOException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
finally {
try {
if (output != null)
output.close();
if (input != null)
input.close();
}
catch (IOException ignored) {
}
if (connection != null)
connection.disconnect();
}
return null;
}
Where is the problem ?
Knowing that when I get URL in a browser, the download file starts up.
Thank you in advance.
The cause should be you are not setting User-Agent:
connection = (HttpURLConnection) url.openConnection();
connection.addRequestProperty("User-Agent", "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:221.0) Gecko/20100101 Firefox/31.0"); // add this line to your code
connection.connect();
HTTP 403 Forbidden
Where is the problem
Error code says it clearly - you are forbidden from accessing the resource on the server. Maybe you need to authenticate first, maybe you are simply banned. Many possibilities.
public class PreviewDownload extends AsyncTask<String, Void, String> {
public static final String TAG = "PreviewDownload";
public String inputPath = null;
public String outputFolder = null;
public IRIssue issue = null;
#Override
protected String doInBackground(String... parms) {
InputStream input = null;
OutputStream output = null;
HttpURLConnection connection = null;
issue = Broker.model.issueDataStore.getIRIssue(parms[0]);
outputFolder = IRConstant.issueFolder(issue.year, issue.month, issue.day, issue.pubKey);
try {
inputPath = IRConstant.downloadFile(issue.year, issue.month, issue.day, issue.pubKey, "preview", "0");
URL url = new URL(inputPath);
Log.d (TAG,"input: " + inputPath);
connection = (HttpURLConnection) url.openConnection();
connection.connect();
if (connection.getResponseCode() != HttpURLConnection.HTTP_OK)
return null;
// return "Server returned HTTP " + connection.getResponseCode()
// + " " + connection.getResponseMessage();
// download the file
input = connection.getInputStream();
output = new FileOutputStream(outputFolder + "/preview.zip");
Log.d (TAG,"output: " + output);
byte data[] = new byte[1024];
int count;
while ((count = input.read(data)) != -1) {
output.write(data, 0, count);
}
} catch (Exception e) {
// return e.toString();
return null;
} finally {
try {
if (output != null)
output.close();
if (input != null)
input.close();
} catch (IOException ignored) {
}
if (connection != null)
connection.disconnect();
}
return outputFolder;
}
#Override
protected void onPostExecute(String outputFolder) {
// TODO Auto-generated method stub
super.onPostExecute(outputFolder);
if (outputFolder != null) {
File zipFile = new File (outputFolder + "/preview.zip");
if (Utils.unzip(outputFolder,outputFolder + "/preview.zip" )) {
zipFile.delete();
issue.isThumbDownloaded = 1;
} else {
issue.isThumbDownloaded = 0;
}
} else {
Toast.makeText(Broker.launcherActivity.getBaseContext(), R.string.wordCantDownload, Toast.LENGTH_LONG).show();
issue.isThumbDownloaded = 0;
}
issue.updateProgress(issue.progress);
}
}
Here is the downloader I implemented , the problem is , when the network lost, the output become null and show error message, however, if I would like to retry two times before showing error message, are there any way to do this? If I perfer not to pass in an object instead of string ,is it not recommended? thanks
What prevents you from re-instanciating and re-executing a "Downloader" from your catch blocks in case of errors ?
You could use a single common shared object between dowloader instances to count the attempts, or better, pass a parameter to each of them. In the catch block, you would then retry if you didn't reach the limit, and increase the value passed to a new downloader... Something recursive.
int expectedLength = connection.getContentLength();
can you compare with the expectedLength & downloaded length and retry?
I have a problem. I met only once. My code algoritm is designed requesting only one time.
But a request was received twice by server. I must produce a solution.
I think, this problem likes this url But I cant find anything(bug or question) about httpurlconnection?
Device's Android Version is 4.0.3 and Connection Method is GET.
try {
String encoding = session.getPlatform().getEncodingIse();
if (TradeUtil.isApachiClientActive()) {
HttpResponse hr = XHttpClientFactory.getConnectionResponse(session.getPlatform(), TradeServiceType.ISE, command);
if (!session.getPlatform().isUseStaticEncodingForIse()) {
encoding = XHttpClientFactory.getEncoding(hr, encoding);
}
strRtn = TradeUtil.getResponse(hr.getEntity().getContent(), encoding);
} else {
con = TradeUtil.createConnection(session.getPlatform(), TradeServiceType.ISE, command);
if (!session.getPlatform().isUseStaticEncodingForIse()) {
encoding = TradeUtil.getEncoding(con, session.getPlatform().getEncodingIse());
}
strRtn = TradeUtil.getResponse(con.getInputStream(), encoding);
}
} catch (Exception e) {
TradeUtil.toConsole(e);
response.getResponseResult().setResponseCode(ResponseResult.CONNECTION_ERROR);
response.getResponseResult().setDescription(e.getMessage());
try {
if (con != null)
con.disconnect();
} catch (Exception ex) {
}
return response;
}
public static HttpURLConnection createConnection(Platform platform,
TradeServiceType serviceType, String command) throws Exception {
command = command + "&MTXTimeAnd=" + Calendar.getInstance().getTimeInMillis(); // Cachlemeyi
// Engellemek
// için
TradeUtil.initSSLContext(); // Zoom ve ustundeki makinalar için SSL
// handshake için trust manager set etmek
// gerekiyor.
System.setProperty("http.keepAlive", "false");// Yatirim finansman +
// Android 2.2 Bug için
// gerekli
System.setProperty("https.keepAlive", "false"); // Yatirim finansman +
// Android 2.2 Bug için
// gerekli
HttpURLConnection con;
String methodType;
String encoding;
String strAdd = null;
if (serviceType.equals(TradeServiceType.ISE)) {
strAdd = platform.getIseServiceAdress();
methodType = platform.getIseConnectionMethodType();
encoding = platform.getEncodingIse();
} else if (serviceType.equals(TradeServiceType.TURKDEX)) {
strAdd = platform.getTurkdexServiceAdress();
methodType = platform.getTurkdexConnectionMethodType();
encoding = platform.getEncodingTurkdex();
} else {
throw new IllegalArgumentException("hatali service tipi");
}
if (methodType.equalsIgnoreCase("GET")) {
strAdd = strAdd + "?" + command;
}
URL adress = new URL(strAdd);
if (adress.getProtocol().equalsIgnoreCase("https")) {
con = (HttpsURLConnection) adress.openConnection();
((HttpsURLConnection) con).setHostnameVerifier(new HostnameVerifier() {
public boolean verify(String hostname, SSLSession session) {
return true;
}
});
} else {
con = (HttpURLConnection) adress.openConnection();
}
con.setUseCaches(false);
con.setReadTimeout(platform.getConnectionTimeout());
con.setConnectTimeout(platform.getConnectionTimeout());
if (methodType.equalsIgnoreCase("POST")) {
con.setDoOutput(true);
con.setDoInput(true);
con.setRequestMethod("POST");
OutputStreamWriter osw = null;
try {
osw = new OutputStreamWriter(con.getOutputStream(), encoding);
osw.write(command);
osw.flush();
osw.close();
osw = null;
} catch (Exception ex) {
TradeUtil.toConsole(ex);
try {
if (osw != null)
osw.close();
} catch (Exception ex2) {
TradeUtil.toConsole(ex);
}
throw ex;
}
} else {
}
return con;
}
I want to print the InputStream in logcat(for testing/later I will use it), my current code is as follows.
#Override
protected Void doInBackground(Void... params) {
try {
InputStream in = null;
int response = -1;
URL url = new URL(
"http://any-website.com/search/users/sports+persons");
URLConnection conn = null;
HttpURLConnection httpConn = null;
conn = url.openConnection();
if (!(conn instanceof HttpURLConnection))
throw new IOException("Not an HTTP connection");
httpConn = (HttpURLConnection) conn;
httpConn.setAllowUserInteraction(false);
httpConn.setInstanceFollowRedirects(true);
httpConn.setRequestMethod("GET");
httpConn.connect();
response = httpConn.getResponseCode();
if (response == HttpURLConnection.HTTP_OK) {
in = httpConn.getInputStream();
}
if (in != null) {
Log.e(TAG, ">>>>>PRINTING<<<<<");
Log.e(TAG, in.toString());
// TODO: print 'in' from here
}
in.close();
in = null;
} catch (Exception e) {
e.printStackTrace();
}
return null;
}
But I am not able to do this, so please check the code and add/modify the code to do this.
String convertStreamToString(java.io.InputStream is) {
try {
return new java.util.Scanner(is).useDelimiter("\\A").next();
} catch (java.util.NoSuchElementException e) {
return "";
}
}
And in your code:
Log.e(TAG, ">>>>>PRINTING<<<<<");
Log.e(TAG, in.toString());
Log.e(TAG, convertStreamToString(in));
Your TAG should be a constant like:
public final String TAG = YourActivity.class.getSimpleName();
Then you would do something like:
Log.e(TAG, "You're message here:", e)
e is your error from print stack.
You also want to make sure you import the Log in the Android library.
Also, after looking at your code, you might want to surround your httpConnection() with try/catch statement, that way you can catch the error, and put it in your Log file. You have the Log printing if your stream has something, or isn't null, but you want to know if you don't have a connection, and that would give you a null value.
Hope that helps.
Just add this code to if(in != null):
byte[] reqBuffer = new byte[1024];
int reqLen = 1024;
int read = -1;
StringBuilder result = new StringBuilder();
try {
while ((read = is.read(reqBuffer, 0, reqLen)) >= 0)
result.append(new String(reqBuffer, 0, read));
} catch (IOException e) {
e.printStackTrace();
}
Log.d("TAG", result.toString());
My code (reproduced below), connects to a url and downloads the file to disk on android. All standard stuff. When I try using this code on a file on S3 accessed via a subdomain on our server mapped to a bucket (e.g. foo.example.com => bucket called foo.example.com), it often fails. Turns out (using the handy curl command..
"curl -v -L -X GET http://foo.example.com/f/a.txt")
.. that there's a redirect going on here.
The file download works ok, as HttpURLConnection will follow redirects by default, but the calls that require the header infomation (getContentLength, getHeaderFieldDate("Last-Modified", 0 ) etc) are returns the headers from the 307 redirect, and not the actual file thats downloaded.
Anyone know how to get around this?
Thanks
File local = null;
try {
Log.i(TAG, "Downloading file " + source);
conn = (HttpURLConnection) new URL(source).openConnection();
fileSize = conn.getContentLength(); // ** THIS IS WRONG ON REDIRECTED FILES
out = new BufferedOutputStream(new FileOutputStream(destination, false), 8 * 1024);
conn.connect();
stream = new BufferedInputStream(conn.getInputStream(), 8 * 1024);
byte[] buffer = new byte[MAX_BUFFER_SIZE];
while (true) {
int read = stream.read(buffer);
if (read == -1) {
break;
}
// writing to buffer
out.write(buffer, 0, read);
downloaded += read;
publishProgress(downloaded, fileSize);
if (isCancelled()) {
return "The user cancelled the download";
}
}
} catch (Exception e) {
String msg = "Failed to download file " + source + ". " + e.getMessage();
Log.e(TAG, msg );
return msg;
} finally {
if (out != null) {
try {
out.flush();
out.close();
} catch (IOException e) {
Log.e(TAG, "Failed to close file " + destination);
e.printStackTrace();
}
}
if (stream != null) {
try {
stream.close();
} catch (IOException e) {
Log.e(TAG, "Failed to close file " + destination);
e.printStackTrace();
}
} else {
long dateLong = conn.getHeaderFieldDate("Last-Modified", 0 ); // ** THIS IS WRONG ON REDIRECTED FILES
Date d = new Date(dateLong);
local.setLastModified(dateLong);
}
have you tried to set redirects to false and try to manually capture the redirected URL and associated header fields with it?
For example something like this:
URL url = new URL(url);
HttpURLConnection ucon = (HttpURLConnection) url.openConnection();
ucon.setInstanceFollowRedirects(false);
URL secondURL = new URL(ucon.getHeaderField("Location"));
URLConnection conn = secondURL.openConnection();
This example captures the redirected URL, but you could easily tweak this to try for any other header field. Does this help?
Consider using httpclient-android. You should get the right headers after redirection with this:
HttpClient client = HttpClientBuilder.create().build();
HttpGet request = new HttpGet(YOUR_URL);
HttpResponse response = client.execute(request);
response.getAllHeaders()
Note that android comes with an older version of httpclient, but it has the same problem as you reported. You actually need to import "httpclient-android" for a newer version.
Note: The code snippet is for v4.3. For other versions, look for how to do it in regular apache HttpClient.
Well, I've been playing a bit and this code, which uses the HttpClient library rather than HttpUrlConnection works fine. The headers it returns are those of the final redirect hop.
At least on the devices I've tested it on.
HttpClient client = null;
HttpGet get = null;
HttpResponse response = null;
try {
client = new DefaultHttpClient();
get = new HttpGet(source);
response = client.execute(get);
Header contentSize = response.getFirstHeader("Content-Length");
if (contentSize != null) {
String value = contentSize.getValue();
fileSize = Long.parseLong(value);
}
if (fileSize == -1) {
Log.e(TAG, "Failed to read the content length for the file " + source);
}
Header lastModified = response.getFirstHeader("Last-Modified");
lastModifiedDate = null;
if (lastModified != null) {
lastModifiedDate = DateUtils.parseDate(lastModified.getValue());
}
if (lastModifiedDate == null) {
Log.e(TAG, "Failed to read the last modified date for the file " + source);
}
out = new BufferedOutputStream(new FileOutputStream(destination, false), 8 * 1024); // false means don't append
stream = new BufferedInputStream(response.getEntity().getContent(), 8 * 1024);
byte[] buffer = new byte[MAX_BUFFER_SIZE];
int count = 0;
while (true) {
int read = stream.read(buffer);
if (read == -1) {
break;
}
// writing to buffer
out.write(buffer, 0, read);
downloaded += read;
publishProgress(downloaded, fileSize);
if (isCancelled()) {
Log.w(TAG, "User Cancelled");
return; // NOTE that onPostExecute is not called here..
}
}// end of while
publishProgress(downloaded, fileSize);
} catch (Exception e) {
String msg = "Failed to download file " + source + ". " + e.getMessage();
Log.e(TAG, msg );
return msg;
} finally {
if (out != null) {
try {
out.flush();
out.close();
} catch (IOException e) {
Log.e(TAG, "Failed to close file " + destination);
e.printStackTrace();
}
}
if (stream != null) {
try {
stream.close();
} catch (IOException e) {
Log.e(TAG, "Failed to close file " + destination);
e.printStackTrace();
}
}
}