So I'm trying to load this JSON in Android from here, and have tried both Volley and regular HTTP requests. The page (eventually) loads fine as UTF-8 JSON and it looks fine. However, in android, I get garbage like this:
Checked the document.characterSet, it's UTF-8.
Example as Volley (trimmed out some code, so brackets may not be exact):
final JsonObjectRequest jsonObjectRequest = new JsonObjectRequest(url, new Response.Listener<JSONObject>() {
#Override
public void onResponse(JSONObject response) {
}
}, new Response.ErrorListener() {
#Override
public void onErrorResponse(VolleyError ex) {
Log.e("LOG", ex.toString());
}
});
requestQueue.add(jsonObjectRequest);
Example as regular HTTP GET:
HttpURLConnection urlConnection = null;
URL url = new URL(urlString);
urlConnection = (HttpURLConnection) url.openConnection();
urlConnection.setRequestMethod("GET");
urlConnection.setReadTimeout(10000 /* milliseconds */);
urlConnection.setConnectTimeout(2500 /* milliseconds */);
urlConnection.setDoOutput(true);
urlConnection.connect();
String UTF8 = "UTF-8";
int BUFFER_SIZE = 8192;
BufferedReader br = new BufferedReader(new InputStreamReader(url.openStream(), UTF8), BUFFER_SIZE);
StringBuilder sb = new StringBuilder();
String line;
while ((line = br.readLine()) != null) {
sb.append(line + "\n");
}
br.close();
String jsonString = sb.toString();
urlConnection.disconnect();
urlConnection = null;
br = null;
sb = null;
if (jsonString != null &&
jsonString.length() > 0) {
return new JSONObject(jsonString);
}
Both give garbage responses. What am I missing? I'm able to access other data on other sites.
The content is compressed using zlib. See header:
Content-Encoding: deflate
You'll have to read the raw bytes and decompress them before attempting to parse as JSON. Looks like Android provides native support for zlib via the Deflater class.
Note that further, readers by default use the system default character encoding. Unless your system default happens to match that of the delivered content, you'll need to tell the system how to decode the charset. The correct thing is to read raw bytes from a stream, then turn the bytes into a string using the proper character encoding,
byte[] buffer = new byte[1024];
int c;
ByteArrayOutputStream baos = new ByteArrayOutputStream();
while ((c = inputStream.read(buffer)) > 0) {
baos.write(buffer, 0, c);
}
String json = new String(baos.toByteArray(), "UTF-8"); // assuming the encoding if UTF-8
You can either know the encoding ahead of time, or parse it from the Content-Type header. I looked at the response from the provided URL and it does not specify a charset, so you'll have to hardcode the known value.
EDIT: apparently you can do this with a reader, although I haven't tried it:
Reader r = new InputStreamReader(inputStream, "UTF-8");
Related
I am running a flask server at the backend. I want to read the image from mobile app and send it to the server to detect faces.
This is the java code(client side) for sending image as bytes -
public class client {
public static void main(String [] args) throws Exception{
String url = "http://127.0.0.1:8080/facial";
// 2. create obj for the URL class
URL obj = new URL(url);
// 3. open connection on the url
HttpURLConnection con = (HttpURLConnection) obj.openConnection();
con.setRequestMethod("POST");
con.setRequestProperty("Content-Type","image/jpeg");
con.setDoInput(true);
con.setDoOutput(true);
try {
System.out.println("Reading image from disk. ");
ByteArrayOutputStream baos = new ByteArrayOutputStream();
baos.flush();
File file = new File("jpeg.jpg");
BufferedImage image1 = ImageIO.read(file);
ImageIO.write(image1, "jpg", baos);
baos.flush();
System.out.println(baos.size());
byte[] bytes = baos.toByteArray();
baos.close();
System.out.println("Sending image to server. ");
OutputStream out = con.getOutputStream();
DataOutputStream image = new DataOutputStream(out);
image.writeInt(bytes.length);
image.write(bytes, 0, bytes.length);
System.out.println("Image sent to server. ");
image.close();
// close the output stream
out.close();
}catch (Exception e) {
System.out.println("Exception: " + e.getMessage());
}
// define object for the reply from the server
BufferedReader in = new BufferedReader(new InputStreamReader(con.getInputStream()));
//Get response from server
int responseCode = con.getResponseCode();
System.out.println("Response Code : " + responseCode);
// read in the response from the server
String inputLine;
StringBuffer response = new StringBuffer();
while ((inputLine = in.readLine()) != null) {
response.append(inputLine);
System.out.println(inputLine);
}
// close the input stream
in.close();
}
}
This is my server code -
def get_facial(data):
face_api_url = 'https://southeastasia.api.cognitive.microsoft.com/face/v1.0/detect'
# Set image_url to the URL of an image that you want to analyze.
headers = {'Ocp-Apim-Subscription-Key': subscription_key,
"Content-Type":"application/octet-stream"
}
params = {
'returnFaceId': 'true',
'returnFaceLandmarks': 'false',
'returnFaceAttributes': 'age,gender,headPose,smile,facialHair,glasses,' +
'emotion,hair,makeup,occlusion,accessories,blur,exposure,noise'
}
response = requests.post(face_api_url, params=params, headers=headers, data=data)
faces = response.json()
res={}
import pdb; pdb.set_trace()
res["status"] = '200'
res["num"] = str(len(faces))
return res
#app.route('/facial',methods=['POST'])
def facial():
import pdb; pdb.set_trace()
data=bytes(request.get_data())
res={}
try:
res = get_facial(data)
except:
res['status'] = '404'
print(res)
return json.dumps(res)
After examining - I sent the same image from another python file and checked the size of the data. It was 102564 bytes and it works but
the same image read and sent from java code is 106208 bytes. I don't know where exactly the mistake is.
Any help is appreciated !!:-)
I found a quick fix to this problem -
Path path = Paths.get("jpeg.jpg");
byte[] fileContents = Files.readAllBytes(path);
image.write(fileContents, 0, fileContents.length);
I don't exactly know why reading from imageio fails. My guess is that its also reading the file headers of the jpg file.
I am trying to get a (JSON formatted) String from a URL and consume it as a Json object. I lose UTF-8 encoding when I convert the String to JSONObject.
This is The function I use to connect to the url and get the string:
private static String getUrlContents(String theUrl) {
StringBuilder content = new StringBuilder();
try {
URL url = new URL(theUrl);
URLConnection urlConnection = url.openConnection();
BufferedReader bufferedReader = new BufferedReader(new InputStreamReader(urlConnection.getInputStream()));
String line;
while ((line = bufferedReader.readLine()) != null) {
content.append(line + "\n");
}
bufferedReader.close();
} catch(Exception e) {
e.printStackTrace();
}
return content.toString();
}
When I get data from server, the following code displays correct characters:
String output = getUrlContents(url);
Log.i("message1", output);
But when I convert the output string to JSONObject the Persian characters becomes question marks like this ??????. (messages is the name of array in JSON)
JSONObject reader = new JSONObject(output);
String messages = new String(reader.getString("messages").getBytes("ISO-8859-1"), "UTF-8");
Log.i("message2", messages);
You're telling Java to convert the string (with key message) to bytes using ISO-8859-1 and than to create a new String from these bytes, interpreted as UTF-8.
new String(reader.getString("messages").getBytes("ISO-8859-1"), "UTF-8");
You could simply use:
String messages = reader.getString("messages");
You can update your code as the following:
private static String getUrlContents(String theUrl) {
StringBuilder content = new StringBuilder();
try {
URL url = new URL(theUrl);
URLConnection urlConnection = url.openConnection();
BufferedReader bufferedReader = new BufferedReader(new InputStreamReader(urlConnection.getInputStream(), "utf-8"));
String line;
while ((line = bufferedReader.readLine()) != null) {
content.append(line).append("\n");
}
bufferedReader.close();
} catch(Exception e) {
e.printStackTrace();
}
return content.toString().trim();
}
You've got two encoding issues:
The server sends text encoded in a character set. When you setup your InputStreamReader, you need to pass the encoding the server used so it can be decoded properly. The character encoding is usually given in the Content-type HTTP response, in the charset field. JSON is typically UTF-8 encoded, but can also be legally UTF-16 and UTF-32, so you need to check. Without a specified encoding, your system environment will be used when marshalling bytes to Strings, and vice versa . Basically, you should always specify the charset.
String messages = new String(reader.getString("messages").getBytes("ISO-8859-1"), "UTF-8"); is obviously going to cause issues (if you have non-ascii characters) - it's encoding the string to ISO-8995-1 and then trying to decode it as UTF-8.
A simple regex pattern can be used to extract the charset value from the Content-type header before reading the inputstream. I've also included a neat InputStream -> String converter.
private static String getUrlContents(String theUrl) {
try {
URL url = new URL(theUrl);
URLConnection urlConnection = url.openConnection();
InputStream is = urlConnection.getInputStream();
// Get charset field from Content-Type header
String contentType = urlConnection.getContentType();
// matches value in key / value pair
Pattern encodingPattern = Pattern.compile(".*charset\\s*=\\s*([\\w-]+).*");
Matcher encodingMatcher = encodingPattern.matcher(contentType);
// set charsetString to match value if charset is given, else default to UTF-8
String charsetString = encodingMatcher.matches() ? encodingMatcher.group(1) : "UTF-8";
// Quick way to read from InputStream.
// \A is a boundary match for beginning of the input
return new Scanner(is, charsetString).useDelimiter("\\A").next();
} catch(Exception e) {
e.printStackTrace();
}
return null;
}
Not sure if this will help, but you might be able to do something like this:
JSONObject result = null;
String str = null;
try
{
str = new String(output, "UTF-8");
result = (JSONObject) new JSONTokener(str).nextValue();
}
catch (Exception e) {}
String messages = result.getString("messages");
I saw this post How to send unicode characters in an HttpPost on Android but I usaully do request in this way in AsyncTask class.My log is also printing local language in urlParameters but server is returning no result while it is perfect for english Strings:
#Override
protected String doInBackground(String... URLs) {
StringBuffer response = new StringBuffer();
try {
URL obj = new URL(URLs[0]);
HttpURLConnection con = (HttpURLConnection) obj.openConnection();
// add request header
con.setRequestMethod("POST");
if (URLs[0].equals(URLHelper.get_preleases)) {
urlCall = 1;
} else
urlCall = 2;
// String urlParameters = "longitude=" + longitude + "&latitude="+latitude;
// Send post request
con.setDoOutput(true);
DataOutputStream wr = new DataOutputStream(con.getOutputStream());
wr.writeBytes(urlParameters);
wr.flush();
wr.close();
int responseCode = con.getResponseCode();
if (responseCode == 200) {
BufferedReader in = new BufferedReader(new InputStreamReader(con.getInputStream()));
String inputLine;
while ((inputLine = in.readLine()) != null) {
response.append(inputLine);
}
in.close();
}
} catch (MalformedURLException e) {
e.printStackTrace();
} catch (IOException e) {
e.printStackTrace();
}
return response.toString();
}
Is there a way to set character set UTF-8 to request parameters coding this way?
String urlParameters = "longitude=" + longitude + "&latitude="+latitude;
You need to URL-encode components you are injecting into an application/x-www-form-urlencoded context. (Even aside from non-ASCII characters, characters like the ampersand will break otherwise.)
Specify the string-to-bytes encoding that you are using for your request in that call, for example:
String urlParameters = "longitude=" + URLEncoder.encode(longitude, "UTF-8")...
...
DataOutputStream wr = new DataOutputStream(con.getOutputStream());
A DataOutputStream is for sending struct-like Java-typed binary data down a stream. It doesn't give you anything you need for writing HTTP request bodies. Maybe you meant OutputStreamWriter?
But since you already have the string all in memory you could simply do:
con.getOutputStream().write(urlParameters.getBytes("UTF-8"))
(Note the UTF-8 here is somewhat superfluous. Because you will already have URL-encoded all the non-ASCII characters into %xx escapes, there will be nothing to UTF-8-encoded. However in general it is almost always better to specify a particular encoding than omit it and revert to the unreliable system default encoding.)
new InputStreamReader(con.getInputStream())
is also omitting the encoding and reverting to the default encoding which is probably not the encoding of the response. So you will probably find non-ASCII characters get read incorrectly in the response too.
I want to send an XML message to a server from my Android Mobile app via HTTP post.
I tried it with HttpUrlConnection, following these steps:
URL url = new URL(vURL);
HttpUrlConnection conn = (HttpURLConnection) url.openConnection();
conn.setDoInput(true);
conn.setDoOutput(true);
// Adding headers (code removed)
conn.setRequestProperty("Content-Type", "text/xml; charset=utf-16");
OutputStream out = new BufferedOutputStream(conn.getOutputStream());
// Adding XML message to the connection output stream
// I have removed exception handling to improve readability for posting it here
out.write(pReq.getBytes()); // here pReq is the XML message in String
out.close();
conn.connect();
Once I get the response, the stream reading part is in done this manner:
BufferedReader in = null;
StringBuffer sb;
String result = null;
try {
InputStreamReader isr = new InputStreamReader(is);
// Just in case, I've also tried:
// new InputStreamReader(is, "UTF-16");
// new InputStreamReader(is, "UTF-16LE");
// new InputStreamReader(is, "UTF-16BE");
// new InputStreamReader(is, "UTF-8");
in = new BufferedReader(isr);
sb = new StringBuffer("");
String line = "";
while ((line = in.readLine()) != null)
sb.append(line);
in.close();
result = sb.toString();
} catch (Exception e) {
e.printStackTrace();
}
Now the result string I get is in some unreadable format/encoding.
When I try the same thing with HttpClient it works correctly. Here is the streaming reading part once I get an HttpResponse after the HttpClient.execute call:
BufferedReader in = null;
InputStream is;
StringBuffer sb;
String decompbuff = null;
try {
is = pResponse.getEntity().getContent();
InputStreamReader isr = new InputStreamReader(is);
in = new BufferedReader(isr);
// Prepare the String buffer
sb = new StringBuffer("");
String line = "";
while ((line = in.readLine()) != null)
sb.append(line);
in.close();
// gZip decompression of response. Note: message was compressed before
// posting it via HttpClient (Posting code is not mentioned here)
decompbuff = Decompress(sb.toString());
} catch (Exception e) {
e.printStackTrace();
}
return decompbuff;
Some help is appreciated in understanding the problem.
One (severe) problem could be that you're ignoring the encoding of input and output.
Input
conn.setRequestProperty("Content-Type", "text/xml; charset=utf-16");
OutputStream out = new BufferedOutputStream(conn.getOutputStream());
// Adding XML message to the connection output stream
// I have removed exception handling to improve readability for posting it here
out.write(pReq.getBytes()); // <-- you use standard platform encoding
out.close();
better:
out.write(pReq.getBytes("UTF-16"));
Output
You probably ignored compression, which would better look like this (taken from DavidWebb):
static InputStream wrapStream(String contentEncoding, InputStream inputStream)
throws IOException {
if (contentEncoding == null || "identity".equalsIgnoreCase(contentEncoding)) {
return inputStream;
}
if ("gzip".equalsIgnoreCase(contentEncoding)) {
return new GZIPInputStream(inputStream);
}
if ("deflate".equalsIgnoreCase(contentEncoding)) {
return new InflaterInputStream(inputStream, new Inflater(false), 512);
}
throw new RuntimeException("unsupported content-encoding: " + contentEncoding);
}
// ...
InputStream is = wrapStream(conn.getContentEncoding(), is);
InputStreamReader isr = new InputStreamReader(is, "UTF-16");
in = new BufferedReader(isr);
sb = new StringBuffer("");
String line = "";
while ((line = in.readLine()) != null)
sb.append(line); // <-- you're swallowing linefeeds!
in.close();
result = sb.toString();
It would be better to let the XML-Parser consume your InputStream directly. Don't create a JAVA string, but let the parser scan the bytes. It will automatically detect the encoding of the XML.
Generally there might be still an issue, because we don't know what type of UTF-16 you use. Can be BigEndian or LittleEndian. That's why I asked, if you really need UTF-16. If you don't have to treat with some asian languages, UTF-8 should be more efficient and easier to use.
So the "solution" I gave you is not guaranteed to work - you have to fiddle with UTF-16 BE/LE a bit and I wish you good luck and patience.
Another remark: in your example above you first construct the String and then Decompress it. That is the wrong order. The stream comes compressed (gzip, deflate) and must be decompressed first. Then you get the String.
I make a GET request to a server using HttpUrlConnection.
After connecting:
I get response code: 200
I get response message: OK
I get input stream, no exception thrown but:
in a standalone program I get the body of the response, as expected:
{"name":"my name","birthday":"01/01/1970","id":"100002215110084"}
in a android activity, the stream is empty (available() == 0), and thus I can't get
any text out.
Any hint or trail to follow? Thanks.
EDIT: here it is the code
Please note: I use import java.net.HttpURLConnection; This is the standard
http Java library. I don't want to use any other external library. In fact
I did have problems in android using the library httpclient from apache (some of their anonymous .class can't be used by the apk compiler).
Well, the code:
URLConnection theConnection;
theConnection = new URL("www.example.com?query=value").openConnection();
theConnection.setRequestProperty("Accept-Charset", "UTF-8");
HttpURLConnection httpConn = (HttpURLConnection) theConnection;
int responseCode = httpConn.getResponseCode();
String responseMessage = httpConn.getResponseMessage();
InputStream is = null;
if (responseCode >= 400) {
is = httpConn.getErrorStream();
} else {
is = httpConn.getInputStream();
}
String resp = responseCode + "\n" + responseMessage + "\n>" + Util.streamToString(is) + "<\n";
return resp;
I see:
200
OK
the body of the response
but only
200
OK
in android
Trying the code of Tomislav I've got the answer.
My function streamToString() used .available() to sense if there is any data received,
and it returns 0 in Android. Surely, I called it too soon.
If I rather use readLine():
class Util {
public static String streamToString(InputStream is) throws IOException {
StringBuilder sb = new StringBuilder();
BufferedReader rd = new BufferedReader(new InputStreamReader(is));
String line;
while ((line = rd.readLine()) != null) {
sb.append(line);
}
return sb.toString();
}
}
then, it waits for the data to arrive.
Thanks.
You can try with this code that will return response in String:
public String ReadHttpResponse(String url){
StringBuilder sb= new StringBuilder();
HttpClient client= new DefaultHttpClient();
HttpGet httpget = new HttpGet(url);
try {
HttpResponse response = client.execute(httpget);
StatusLine sl = response.getStatusLine();
int sc = sl.getStatusCode();
if (sc==200)
{
HttpEntity ent = response.getEntity();
InputStream inpst = ent.getContent();
BufferedReader rd= new BufferedReader(new InputStreamReader(inpst));
String line;
while ((line=rd.readLine())!=null)
{
sb.append(line);
}
}
else
{
Log.e("log_tag","I didn't get the response!");
}
} catch (ClientProtocolException e) {
e.printStackTrace();
} catch (IOException e) {
e.printStackTrace();
}
return sb.toString();
}
The Stream data may not be ready, so you should check in a loop that the data in the stream is available before attempting to access it.
Once the data is ready, you should read it and store in another place like a byte array; a binary stream object is a nice choice to read data as a byte array. The reason that a byte array is a better choice is because the data may be binary data like an image file, etc.
InputStream is = httpConnection.getInputStream();
byte[] bytes = null;
ByteArrayOutputStream baos = new ByteArrayOutputStream();
byte[] temp = new byte[is.available()];
while (is.read(temp, 0, temp.length) != -1) {
baos.write(temp);
temp = new byte[is.available()];
}
bytes = baos.toByteArray();
In the above code, bytes is the response as byte array. You can convert it to string if it is text data, for example data as utf-8 encoded text:
String text = new String(bytes, Charset.forName("utf-8"));