How to get content from website with HttpURLConnection for android APIs 18 and above? The code is working fine for API 23, but inputstream is returning odd values for API 18. This is what I get when I try to read data from URL with API 18:
TTP/1.1 200 OK Content-Type: application/json; charset=utf-8
Access-Control-Allow-Origin: * Cache-Control: no-cache, no-store,
max-age=0, must-revalidate Pragma: no-cache Expires: Mon, 01 Jan 1990
00:00:00 GMT Date: Wed, 03 Aug 2016 19:01:33 GMTContent-Encoding: gzip
P3P: CP="This is not a P3P policy! See
https://support.google.com/accounts/answer/151657?hl=en for more
info." P3P: CP="This is not a P3P policy! See
https://support.google.com/accounts/answer/151657?hl=en for more
info." X-Content-Type-Options: nosniff X-Frame-Options: SAMEORIGIN
X-XSS-Protection: 1; mode=block Server: GSE Set-Cookie:
NID=83=Nb29w9eQo3Fdx_bQuj6YbdLwSfxjuQT4f1Lcb87IbTXQqdGGh6OyuxxB0XGWxNIiAfMdCePDtDb5P9vMYQvbln7svacSJjFkWnU6-B4AN9vLHHY4RUdL3Xny7zSmE8Lm;Domain=.googleusercontent.com;Path=/;Expires=Thu,
02-Feb-2017 19:01:33 GMT;HttpOnly Set-Cookie:
NID=83=Z9EmVPVCfKYu4FrAHTVHDPMNM80s23cO6P1VqJAocZHnrQb8QFPKW9BLjQGu5xKOwtqNaT38gTZVJm1zmbT7tVhZAYCQlaSb7dRiSTcqQ71a41cIs4l67RxEkOjXfttC;Domain=.googleusercontent.com;Path=/;Expires=Thu,
02-Feb-2017 19:01:33 GMT;HttpOnly Alternate-Protocol: 443:quic
Alt-Svc: quic=":443"; ma=2592000; v="36,35,34,33,32,31,30"
Transfer-Encoding: chunked 00000001 00000001 � 00000001 00000001 ��
00000001 �� 00000001 �� 00000001 �� �� ��
What would be the reasoning behind this? I can provide the code if needed. I'm fairly new with this and can't find the answer anywhere.
To get response from URL I use
private String downloadUrl(String urlString) throws IOException {
InputStream is = null;
try {
URL url = new URL(urlString);
HttpURLConnection conn = (HttpURLConnection) url.openConnection();
conn.setReadTimeout(10000 /* milliseconds */);
conn.setConnectTimeout(15000 /* milliseconds */);
conn.setRequestMethod("GET");
conn.setDoInput(true);
// Starts the query
conn.connect();
int responseCode = conn.getResponseCode();
is = conn.getInputStream();
String contentAsString = convertStreamToString(is);
return contentAsString;
} finally {
if (is != null) {
is.close();
}
}
}
and the method which converts the stream into string
private String convertStreamToString(InputStream is) {
BufferedReader reader = new BufferedReader(new InputStreamReader(is));
StringBuilder sb = new StringBuilder();
String line = null;
try {
while ((line = reader.readLine()) != null) {
sb.append(line + "\n");
}
} catch (IOException e) {
e.printStackTrace();
} finally {
try {
is.close();
} catch (IOException e) {
e.printStackTrace();
}
}
return sb.toString();
}
I have had a similar problem.. The problem is that you are using an HttpUrlConnection on an https request. So all of your data is going to be encrypted. I would recommend using some other HttpRequest library like http-request by kevinsawicki: https://github.com/kevinsawicki/http-request. This will have easy support for getting headers, bodies (Json, XML, Plaintext) and other meta data about the request.
How to add the maven
In android studio open up the project viewer and open build.gradle (app)
and add this line in the dependencies list
compile 'com.github.kevinsawicki:http-request:6.0'
Now open up the (project) build.gradle and make sure under repositories you have this
mavenCentral()
Example get request
HttpRequest req = HttpRequest.get("https://google.com");
req.trustAllCerts();
req.trustAllHosts(); //If you are having certificate problems
int code = req.code();
String body = req.body();
Log.d("CODE:", String.valueOf(code));
Log.d("BODY:", body);
Best regards, David
you might need
in = new InputStreamReader(
httpResponseCode == HttpURLConnection.HTTP_OK ? conn.getInputStream() : conn.getErrorStream(),
"UTF-8");
Related
String url = "https://maps.googleapis.com/maps/api/distancematrix/json?origins="+expedTextView.getText().toString()+"&destinations="+destTextView.getText().toString()+"&sensor=false&mode=driving&key=AIzaSyC9BT3f2jmmPSJh2ewGfGiuVz1zeIfPX4A";
HttpClient httpClient = new DefaultHttpClient();
HttpContext localContext = new BasicHttpContext();
HttpGet httpGet = new HttpGet(url);
HttpResponse response = null;
try {
response = httpClient.execute(httpGet, localContext);
} catch (IOException e) {
e.printStackTrace();
}
Toast.makeText(SecondScreen.this, ""+response, Toast.LENGTH_LONG).show();
}
This is my code. I've tried to make an HTTP request suing this URL which constructs just fine, but, when I read the response, it doesn't look like JSON at all. Below is the code from response. Notice that the link works well when I put it in my browser.
HTTP/1.1 200 OK [Content-Type: application/json; charset=UTF-8, Date: Fri, 11 Jan 2019 14:26:57 GMT, Expires: Sat, 12 Jan 2019 14:26:57 GMT, Cache-Control: public, max-age=86400, Server: mafe, X-XSS-Protection: 1; mode=block, X-Frame-Options: SAMEORIGIN, Server-Timing: gfet4t7; dur=91, Alt-Svc: quic=":443"; ma=2592000; v="44,43,39,35", Accept-Ranges: none, Vary: Accept-Language,Accept-Encoding, Transfer-Encoding: chunked] cz.msebera.android.httpclient.conn.BasicManagedEntity#2bffe96
D/HwRTBlurUtils: check blur style for HwToast-Toast, themeResId : 0x7f0d0005, context : com.example.rarestaciu.bikemessengerapp.SecondScreen#309bb49, Nhwext : 6, get Blur : disable with , android.graphics.drawable.NinePatchDrawable#aadbe04
I am building an android app that pulls in and displays data from a JSON api.
Specifically, the JSON api that I am pulling from is here
URL blogFeedUrl = new URL("http://www.johncorser.com/?json=1&count=15/");
HttpURLConnection connection = (HttpURLConnection)blogFeedUrl.openConnection();
connection.connect();
responseCode = connection.getResponseCode();
Log.i(TAG, "Code: " + responseCode);
Log.i(TAG, "Length " + connection.getContentLength());
if (responseCode == HttpURLConnection.HTTP_OK){
InputStream inputStream = connection.getInputStream();
Reader reader = new InputStreamReader(inputStream);
int contentLength = connection.getContentLength();
char[] charArray = new char[contentLength]; //Line 97, where the error is thrown because contentLength == -1
reader.read(charArray);
String responseData = new String(charArray);
jsonResponse = new JSONObject(responseData);
}
else {
Log.i(TAG, "Unsuccessful HTTP response code: " + responseCode);
}
When I run this, the logcat gives me this stacktrace:
10-15 11:36:00.512 8704-8718/com.johncorser.johncorser I/MainListActivity﹕ Code: 200
10-15 11:36:00.512 8704-8718/com.johncorser.johncorser I/MainListActivity﹕ Length -1
10-15 11:36:00.512 8704-8718/com.johncorser.johncorser E/MainListActivity﹕ Exception caught:
java.lang.NegativeArraySizeException: -1
at com.johncorser.johncorser.MainListActivity$GetBlogPostTask.doInBackground(MainListActivity.java:97)
at com.johncorser.johncorser.MainListActivity$GetBlogPostTask.doInBackground(MainListActivity.java:80)
at android.os.AsyncTask$2.call(AsyncTask.java:288)
at java.util.concurrent.FutureTask.run(FutureTask.java:237)
at android.os.AsyncTask$SerialExecutor$1.run(AsyncTask.java:231)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1112)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:587)
at java.lang.Thread.run(Thread.java:841)
The strangest part is, if I swap the api with one structured in a nearly identical way, like the one here, the code executes without errors.
Any idea why I can't load this specific JSON data?
The issue is that service returns data in chunks (Transfer-Encoding:chunked in response headers), so it does not know content-length at the time it sends headers.
To handle such situation you can just wrap your InputStreamReader in BufferedReader and read it line by line:
BufferedReader br = new BufferedReader(new InputStreamReader(inputStream));
StringBuilder responseBuilder = new StringBuilder();
String chunk;
while ((chunk = br.readLine()) != null) {
responseBuilder.append(chunk);
}
String responseData = responseBuilder.toString();
See the api description: http://developer.android.com/reference/java/net/URLConnection.html#getContentLength()
"Returns the content length in bytes specified by the response header field content-length or -1 if this field is not set."
The http response is not sending content length header back
If the content length isn't known when the server sends the headers, then it doesn't tell the client the length; that means that the client doesn't know the length, and the call returns -1. Details are in the Javadoc for URLConnection.
What this means in practical terms is that you can't do it with a fixed length char array. You have to read the response line by line, into a StringBuilder, until you've exhausted the input. Then you can put the result into your JSONObject constructor.
Remember that some things on the web stream live content, so it's impossible for them to know the total length at the point when they start transmitting. That means you can't ever rely on being told the length; and because of that, some things are lazy and don't bother telling you the length even when they do know it.
Here are the headers your johncorser.com web site returns for that page:
HTTP/1.1 200 OK
Date: Wed, 15 Oct 2014 15:40:15 GMT
Server: Apache
Set-Cookie: PHPSESSID=82ivean17opsfkqjf9n4itdog2; path=/
Expires: Thu, 19 Nov 1981 08:52:00 GMT
Cache-Control: no-store, no-cache, must-revalidate, post-check=0, pre-check=0
Pragma: no-cache
X-Pingback: http://www.johncorser.com/xmlrpc.php
Content-Type: application/json; charset=UTF-8
Note there's nothing about length there. But the teamtreehouse.com site you also linked to, with a similar structure, returns this:
HTTP/1.1 200 OK
Server: nginx
Date: Wed, 15 Oct 2014 15:46:59 GMT
Content-Type: application/json; charset=UTF-8
Content-Length: 4422
Connection: keep-alive
Keep-Alive: timeout=20
X-Pingback: http://blog.teamtreehouse.com/xmlrpc.php
X-Cacheable: SHORT
Vary: Accept-Encoding,Cookie
Cache-Control: max-age=600, must-revalidate
Accept-Ranges: bytes
X-Cache: HIT: 2
X-Cache-Group: normal
X-Type: default
Note that this one includes a Content-Length: 4422.
In my app , i have implemented HttpResponseCache to cache the responses so that it could be used instead of hitting the server. For a particular api , the server returns the header Cache-Control as no-cache;must-revalidate. It has the header ETag also. The problem is the response of this api is not cached. As a result each time I request the api , server returns 200.
Does no-cache,must-revalidate mean the response won't/shouldn't be cached ?
Please find below the request and response headers of the http request :
Request Headers :
GET HTTP/1.1
User-Agent
Accept application/json
Cache-Control max-age=0
Cookie
Host
Connection Keep-Alive
Accept-Encoding gzip
Response Headers :
HTTP/1.1 200 OK
Server Apache-Coyote/1.1
Cache-Control no-cache, must-revalidate
ETag "c683a0301c68c566fcc706f5cd82f1f8"
Content-Type application/json;charset=UTF-8
Transfer-Encoding chunked
Content-Encoding gzip
Vary Accept-Encoding
Date Mon, 24 Feb 2014 04:44:03 GMT
Sending HTTP_GET request :
URL url = new URL(this.url);
HttpURLConnection conn = null;
conn = (HttpURLConnection) url.openConnection();
conn.setRequestMethod("GET");
conn.setReadTimeout(timeout);
conn.setConnectTimeout(timeout);
conn.setInstanceFollowRedirects(true);
conn.setRequestProperty("User-Agent", this.userAgent);
conn.setUseCaches(true);
conn.setDefaultUseCaches(true);
conn.connect();
this.responseCode = conn.getResponseCode();
this.extractResponseHeaders(conn.getHeaderFields());
InputStream inputStream = null;
if (this.responseCode >= 400 ) {
inputStream = conn.getErrorStream();
} else {
inputStream = conn.getInputStream();
}
try {
if(null != inputStream){
this.response = convertToString(inputStream);
}
} catch (JSONException e) {
e.printStackTrace();
}
I try to send POST request to some server with json parameters and I wait json response.
From Android it works fine but from Linux I get status 302 redirect. I really don't know where is my problem.
Here is command from linux:
curl -i -X POST --data "id=105&json={\"orderBy\":0\"maxResults\":50}" "http://mysite.com/ctlClient/"
And I get response:
HTTP/1.1 302 Found
Date: Thu, 04 Jul 2013 12:22:06 GMT
Server: Apache
X-Powered-By: PHP/5.3.19
Set-Cookie: PHPSESSID=dqblvijvttgsdrv2u5tn72p9d6; path=/
Expires: Thu, 19 Nov 1981 08:52:00 GMT
Cache-Control: no-store, no-cache, must-revalidate, post-check=0, pre-check=0
Pragma: no-cache
location: http://mysite.com/fwf/online/
Content-Length: 0
Connection: close
Content-Type: text/html
From Server access log:
"POST /ctlClient/ HTTP/1.1" 302 - "-" "curl/7.15.5 (x86_64-redhat-linux-gnu) libcurl/7.15.5 OpenSSL/0.9.8b zlib/1.2.3 libidn/0.6.5"
Android:
String data = "{\"orderBy\":0\"maxResults\":50}";
String WEFI_MAIL_URL = "http://mysite.com/ctlClient/";
DefaultHttpClient httpclient = null;
String out = null;
try {
httpclient = new DefaultHttpClient();
HttpPost httpost = new HttpPost(WEFI_MAIL_URL);
httpost.setHeader("User-Agent", "Apache-HttpClient/4.1 (java 1.5)");
List <NameValuePair> nvps = new ArrayList <NameValuePair>();
nvps.add(new BasicNameValuePair("id", "105"));
nvps.add(new BasicNameValuePair("json", data));
httpost.setEntity(new UrlEncodedFormEntity(nvps, HTTP.UTF_8));
HttpResponse response = httpclient.execute(httpost);
HttpEntity entity = response.getEntity();
if (entity != null) {
InputStream is = entity.getContent();
InputStreamReader isr = new InputStreamReader(is);
BufferedReader br = new BufferedReader(isr);
String line = null;
while ( (line = br.readLine()) != null) {
out = line;
}
is.close();
if(isTimeOut == false){
_loadActual();
}
else{
return;
}
} else {
return;
}
} catch (Exception e) {
}
if (httpclient != null) {
httpclient.getConnectionManager().shutdown();
}
}
access log on Android response:
"POST /ctlClient/ HTTP/1.1" 302 - "-" "Apache-HttpClient/4.1 (java 1.5)"
"GET /fwf/online/ HTTP/1.1" 200 13981 "-" "Apache-HttpClient/4.1 (java 1.5)"
We can see that on POST server redirects me to /fwf/online/
From Wireshark i get the same results from both methods:
POST /ctlClient/ HTTP/1.1
User-Agent: Apache-HttpClient/4.1 (java 1.5)
Content-Type: application/x-www-form-urlencoded
Content-Length: 301
Host: mysite.com
Connection: Keep-Alive
Why it works from Android but not from Linux with CURL?
Can anybody help me or give me direction how can i solve it?
Thank you
You want to add -L to your curl command. See http://curl.haxx.se/docs/faq.html#How_do_I_tell_curl_to_follow_HTT.
I check this Post. But I don't still understand what the logical error could be. I am still getting this error. I tried to dump the emulator traffic. But I still don't get my head around what the problem could be.
From the traffic dump, this is what Android is sending as request to the server. You can see the response too:
GET /Authenticate/ HTTP/1.1
Authorization: Basic asdfasdfasdfas
Accept-Charset: UTF-8
Host: www.domain.com
User-Agent: Dalvik/1.4.0 (Linux; U; Android 2.3.3; sdk Build/GRI34)
Connection: Keep-Alive
Accept-Encoding: gzip
neQPˆ? 6 6 RT 4VRT 5 E (
» #ÍCl¦'
PÙ[ ˜ároP"8" neQPI " " RT 4VRT 5 E
¼ #ËVl¦'
PÙ[ ˜ároP"8«‹ HTTP/1.1 400 Bad Request
Date: Thu, 13 Sep 2012 04:47:42 GMT
Server: Apache/2.2.15 (CentOS)
Content-Length: 310
Connection: close
Content-Type: text/html; charset=iso-8859-1
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>400 Bad Request</title>
</head><body>
<h1>Bad Request</h1>
<p>Your browser sent a request that this server could not understand.<br />
</p>
<hr>
<address>Apache/2.2.15 (CentOS) Server at www.domain.com Port 80</address>
</body></html>
neQPé¬ 6 6 RT 4VRT 5 E (
½ #ÍAl¦'
PÙ[ ™îároP"8 , neQPéË # # RT 5RT 4V E (çØ# #³-
l¦'Ù[ Páro ™îP )E neQPéË # # RT 5RT 4V E (çÙ# #³,
l¦'Ù[ Páro ™ïP )D neQPö“
© © RT 5RT 4V E ›k‹# #¶Á
³ër,9Jr ‘Pÿÿ6B WRTE w [ * ¨«º[ 09-13 04:47:41.822 446:0x1c7 D/400 ]
text/html; charset=iso-8859-1Bad Request
I don't know what those extra characters mean. But I was trying to identify the problem from it.
This is the basic code:
String credentials = username + ":" + password;
byte[] toencode = null;
try {
toencode = credentials.getBytes("UTF-8");
} catch (UnsupportedEncodingException e1) {
e1.printStackTrace();
}
HttpURLConnection conn = null;
try {
//Utilities.isNetworkAvailable(context);
URL url = new URL(params[0]);
Log.d(params[0],"UR");
conn = (HttpURLConnection) url.openConnection();
conn.setRequestProperty("Authorization", "Basic " + Base64.encodeToString(toencode, Base64.DEFAULT));
conn.setRequestProperty("Accept-Charset", "UTF-8");
conn.setRequestProperty("Host", "www.domain.com");
conn.setRequestMethod("GET");
conn.setDoOutput(true);
String data = conn.getInputStream().toString();
return data;
}
Any ideas?
Update
I checked Webserver Logs to see if the requests are hitting the server and if there was any problem with the request. This is what I see from error logs:
[Thu Sep 13 10:05:24 2012] [error] [client 91.222.195.132] client sent HTTP/1.1 request without hostname (see RFC2616 section 14.23): /Authenticate/
[Thu Sep 13 23:11:57 2012] [error] [client 91.222.195.132] client sent HTTP/1.1 request without hostname (see RFC2616 section 14.23): /Authenticate/
[Thu Sep 13 23:12:03 2012] [error] [client 91.222.195.132] client sent HTTP/1.1 request without hostname (see RFC2616 section 14.23): /Authenticate/
However I am setting the header property for the request.
Any ideas?
I figured out this myself. Its an issue with the order of setting headers.
Edit: Order I used.
URL url = new URL(strUrl);
conn = (HttpURLConnection) url.openConnection();
conn.setRequestMethod("GET");
conn.setRequestProperty("Host", "myhost.com");
conn.setRequestProperty("Authorization", "Basic " + Base64.encodeToString(toencode, Base64.DEFAULT));
conn.setRequestProperty("User-Agent", "Mozilla/5.0 (Macintosh; U; PPC; en-US; rv:1.3.1)");
conn.setRequestProperty("Accept-Charset", "UTF-8");
conn.setConnectTimeout (5000) ;
conn.setDoOutput(true);
conn.setDoInput(true);
I was also facing this issue, but i fixed it by changing my codes, now i am using following lines of codes
BufferedReader reader;
StringBuffer buffer;
String res = null;
try {
URL url = new URL(request_url);
HttpURLConnection con = (HttpURLConnection) url.openConnection();
con.setReadTimeout(40000);
con.setConnectTimeout(40000);
con.setRequestMethod("GET");
con.setRequestProperty("Accept","application/json");
con.setRequestProperty("Content-Type", "application/x-www-form-urlencoded");
con.setDoInput(true);
con.setDoOutput(true);
int status = con.getResponseCode();
InputStream inputStream;
if (status == HttpURLConnection.HTTP_OK) {
inputStream = con.getInputStream();
} else {
inputStream = con.getErrorStream();
}
reader = new BufferedReader(new InputStreamReader(inputStream));
buffer = new StringBuffer();
String line = "";
while ((line = reader.readLine()) != null) {
buffer.append(line);
}
res = buffer.toString();
} catch (MalformedURLException e) {
e.printStackTrace();
} catch (ProtocolException e) {
e.printStackTrace();
} catch (IOException e) {
e.printStackTrace();
}
return res;
Try this
static final String _url = "http://www.google.com";
static final String charset = "UTF-8";
// to build the query string that will send the message
private static String buildRequestString(String param1,
String param2, String param3, String param4, String param5)
throws UnsupportedEncodingException {
String[] params = new String[5]; //customize this as per your need
params[0] = param1;
params[1] = param2;
params[2] = param3;
params[3] = param4;
params[4] = param5;
String query = String.format(
"uid=%s&pwd=%s&msg=%s&phone=%s&provider=%s",
URLEncoder.encode(params[0], charset),
URLEncoder.encode(params[1], charset),
URLEncoder.encode(params[2], charset),
URLEncoder.encode(params[3], charset),
URLEncoder.encode(params[4], charset));
return query;
}
public static void doSomething(String param1, String param2,
String param3, String param4, String param5) throws Exception {
// To establish the connection and perform the post request
URLConnection connection = new URL(_url
+ "?"
+ buildRequestString(param1, param2, param3, param4,
param5)).openConnection();
connection.setRequestProperty("Accept-Charset", charset);
// This automatically fires the request and we can use it to determine
// the response status
InputStream response = connection.getInputStream();
BufferedReader br = new BufferedReader(new InputStreamReader(response));
// This stores the response code.
// Any call to br.readLine() after this is null.
responsecode = br.readLine();
// And this logs the already stored response code
Log.d("ServerResponse", responsecode);
responseInt = Integer.valueOf(responsecode).intValue();
}