Develop Reference

Converting String to Android JSONObject loses utf-8

I am trying to get a (JSON formatted) String from a URL and consume it as a Json object. I lose UTF-8 encoding when I convert the String to JSONObject.
This is The function I use to connect to the url and get the string:
private static String getUrlContents(String theUrl) {
StringBuilder content = new StringBuilder();
try {
URL url = new URL(theUrl);
URLConnection urlConnection = url.openConnection();
BufferedReader bufferedReader = new BufferedReader(new InputStreamReader(urlConnection.getInputStream()));
String line;
while ((line = bufferedReader.readLine()) != null) {
content.append(line + "\n");
}
bufferedReader.close();
} catch(Exception e) {
e.printStackTrace();
}
return content.toString();
}
When I get data from server, the following code displays correct characters:
String output = getUrlContents(url);
Log.i("message1", output);
But when I convert the output string to JSONObject the Persian characters becomes question marks like this ??????. (messages is the name of array in JSON)
JSONObject reader = new JSONObject(output);
String messages = new String(reader.getString("messages").getBytes("ISO-8859-1"), "UTF-8");
Log.i("message2", messages);

You're telling Java to convert the string (with key message) to bytes using ISO-8859-1 and than to create a new String from these bytes, interpreted as UTF-8.
new String(reader.getString("messages").getBytes("ISO-8859-1"), "UTF-8");
You could simply use:
String messages = reader.getString("messages");

You can update your code as the following:
private static String getUrlContents(String theUrl) {
StringBuilder content = new StringBuilder();
try {
URL url = new URL(theUrl);
URLConnection urlConnection = url.openConnection();
BufferedReader bufferedReader = new BufferedReader(new InputStreamReader(urlConnection.getInputStream(), "utf-8"));
String line;
while ((line = bufferedReader.readLine()) != null) {
content.append(line).append("\n");
}
bufferedReader.close();
} catch(Exception e) {
e.printStackTrace();
}
return content.toString().trim();
}

You've got two encoding issues:
The server sends text encoded in a character set. When you setup your InputStreamReader, you need to pass the encoding the server used so it can be decoded properly. The character encoding is usually given in the Content-type HTTP response, in the charset field. JSON is typically UTF-8 encoded, but can also be legally UTF-16 and UTF-32, so you need to check. Without a specified encoding, your system environment will be used when marshalling bytes to Strings, and vice versa . Basically, you should always specify the charset.
String messages = new String(reader.getString("messages").getBytes("ISO-8859-1"), "UTF-8"); is obviously going to cause issues (if you have non-ascii characters) - it's encoding the string to ISO-8995-1 and then trying to decode it as UTF-8.
A simple regex pattern can be used to extract the charset value from the Content-type header before reading the inputstream. I've also included a neat InputStream -> String converter.
private static String getUrlContents(String theUrl) {
try {
URL url = new URL(theUrl);
URLConnection urlConnection = url.openConnection();
InputStream is = urlConnection.getInputStream();
// Get charset field from Content-Type header
String contentType = urlConnection.getContentType();
// matches value in key / value pair
Pattern encodingPattern = Pattern.compile(".*charset\\s*=\\s*([\\w-]+).*");
Matcher encodingMatcher = encodingPattern.matcher(contentType);
// set charsetString to match value if charset is given, else default to UTF-8
String charsetString = encodingMatcher.matches() ? encodingMatcher.group(1) : "UTF-8";
// Quick way to read from InputStream.
// \A is a boundary match for beginning of the input
return new Scanner(is, charsetString).useDelimiter("\\A").next();
} catch(Exception e) {
e.printStackTrace();
}
return null;
}

Not sure if this will help, but you might be able to do something like this:
JSONObject result = null;
String str = null;
try
{
str = new String(output, "UTF-8");
result = (JSONObject) new JSONTokener(str).nextValue();
}
catch (Exception e) {}
String messages = result.getString("messages");

Encoding Issue with HttpUrlConnection in Android

I want to send an XML message to a server from my Android Mobile app via HTTP post.
I tried it with HttpUrlConnection, following these steps:
URL url = new URL(vURL);
HttpUrlConnection conn = (HttpURLConnection) url.openConnection();
conn.setDoInput(true);
conn.setDoOutput(true);
// Adding headers (code removed)
conn.setRequestProperty("Content-Type", "text/xml; charset=utf-16");
OutputStream out = new BufferedOutputStream(conn.getOutputStream());
// Adding XML message to the connection output stream
// I have removed exception handling to improve readability for posting it here
out.write(pReq.getBytes()); // here pReq is the XML message in String
out.close();
conn.connect();
Once I get the response, the stream reading part is in done this manner:
BufferedReader in = null;
StringBuffer sb;
String result = null;
try {
InputStreamReader isr = new InputStreamReader(is);
// Just in case, I've also tried:
// new InputStreamReader(is, "UTF-16");
// new InputStreamReader(is, "UTF-16LE");
// new InputStreamReader(is, "UTF-16BE");
// new InputStreamReader(is, "UTF-8");
in = new BufferedReader(isr);
sb = new StringBuffer("");
String line = "";
while ((line = in.readLine()) != null)
sb.append(line);
in.close();
result = sb.toString();
} catch (Exception e) {
e.printStackTrace();
}
Now the result string I get is in some unreadable format/encoding.
When I try the same thing with HttpClient it works correctly. Here is the streaming reading part once I get an HttpResponse after the HttpClient.execute call:
BufferedReader in = null;
InputStream is;
StringBuffer sb;
String decompbuff = null;
try {
is = pResponse.getEntity().getContent();
InputStreamReader isr = new InputStreamReader(is);
in = new BufferedReader(isr);
// Prepare the String buffer
sb = new StringBuffer("");
String line = "";
while ((line = in.readLine()) != null)
sb.append(line);
in.close();
// gZip decompression of response. Note: message was compressed before
// posting it via HttpClient (Posting code is not mentioned here)
decompbuff = Decompress(sb.toString());
} catch (Exception e) {
e.printStackTrace();
}
return decompbuff;
Some help is appreciated in understanding the problem.

One (severe) problem could be that you're ignoring the encoding of input and output.
Input
conn.setRequestProperty("Content-Type", "text/xml; charset=utf-16");
OutputStream out = new BufferedOutputStream(conn.getOutputStream());
// Adding XML message to the connection output stream
// I have removed exception handling to improve readability for posting it here
out.write(pReq.getBytes()); // <-- you use standard platform encoding
out.close();
better:
out.write(pReq.getBytes("UTF-16"));
Output
You probably ignored compression, which would better look like this (taken from DavidWebb):
static InputStream wrapStream(String contentEncoding, InputStream inputStream)
throws IOException {
if (contentEncoding == null || "identity".equalsIgnoreCase(contentEncoding)) {
return inputStream;
}
if ("gzip".equalsIgnoreCase(contentEncoding)) {
return new GZIPInputStream(inputStream);
}
if ("deflate".equalsIgnoreCase(contentEncoding)) {
return new InflaterInputStream(inputStream, new Inflater(false), 512);
}
throw new RuntimeException("unsupported content-encoding: " + contentEncoding);
}
// ...
InputStream is = wrapStream(conn.getContentEncoding(), is);
InputStreamReader isr = new InputStreamReader(is, "UTF-16");
in = new BufferedReader(isr);
sb = new StringBuffer("");
String line = "";
while ((line = in.readLine()) != null)
sb.append(line); // <-- you're swallowing linefeeds!
in.close();
result = sb.toString();
It would be better to let the XML-Parser consume your InputStream directly. Don't create a JAVA string, but let the parser scan the bytes. It will automatically detect the encoding of the XML.
Generally there might be still an issue, because we don't know what type of UTF-16 you use. Can be BigEndian or LittleEndian. That's why I asked, if you really need UTF-16. If you don't have to treat with some asian languages, UTF-8 should be more efficient and easier to use.
So the "solution" I gave you is not guaranteed to work - you have to fiddle with UTF-16 BE/LE a bit and I wish you good luck and patience.
Another remark: in your example above you first construct the String and then Decompress it. That is the wrong order. The stream comes compressed (gzip, deflate) and must be decompressed first. Then you get the String.

Simplest way to download text from URL

Is there a simplest way to download small text string from URL like this one:"http://app.georeach.com/ios/version.txt"
In iOS its pretty simple. But for android em not finding something good. what is the method for getting text like that from the above URL??
I used this code in onCreate of hello app,n app crashed:
try {
// Create a URL for the desired page
URL url = new URL("http://app.georeach.com/ios/version.txt");
// Read all the text returned by the server
BufferedReader in = new BufferedReader(new InputStreamReader(url.openStream()));
String str;
StringBuilder sb = new StringBuilder(100);
while ((str = in.readLine()) != null) {
sb.append(str);
// str is one line of text; readLine() strips the newline character(s)
}
in.close();
tv.setText(sb.toString());
} catch (MalformedURLException e) {
tv.setText("mal");
} catch (IOException e) {
tv.setText("io");
}

You have to create a new class extended from AsyncTask. You can't do network stuff in the main thread. It could work but you may not want to do that. Take a look at this link : http://developer.android.com/reference/android/os/AsyncTask.html
Also don't forget to add Internet permissions to your AndroidManifest.xml.

Try this:
URL url = new URL("http://bla-bla...");
HttpURLConnection connection = (HttpURLConnection) url.openConnection();
InputStream in = connection.getInputStream();
BufferedReader br = new BufferedReader(new InputStreamReader(in));
StringBuilder sb = new StringBuilder();
String line;
while ((line = br.readLine()) != null) {
sb.append(line);
}
// your text is here
String text = sb.toString()
Do not forget to catch and handle IOException and close all streams.

An "easier" way would be this:
String url2txt = null;
try {
// Being address an URL instance
url2txt = new Scanner(address.openStream(), "UTF-8").useDelimiter("\\A").next();
} catch (IOException e) { ... }
The thing is what you consider "easier". As far as code goes, probably this is the shortest way, but it depends on what you want to do afterwards with the obtained text.

Android - Parse text from website

I have webpage with this simple text, which is changeable.
<html><head><style type="text/css"></style></head><body>69766</body></html>
I need parse only number 69766 and save it to variable as String or int. It's possible to parse this number without adding libraries?
Thanks for your questions !

You can do like this
URL url = new URL("http://url for your webpage");
URLConnection yc = url.openConnection();
BufferedReader in = new BufferedReader(
new InputStreamReader(
yc.getInputStream()));
String inputLine;
StringBuilder builder = new StringBuilder();
while ((inputLine = in.readLine()) != null)
builder.append(inputLine.trim());
in.close();
String htmlPage = builder.toString();
String yourNumber = htmlPage.replaceAll("\\<.*?>","");

For your basic need you should take a lot at Html class.

this link shows how to parse the xml with the SAX parser. Its pretty straight forward.
http://www.codeproject.com/Articles/334859/Parsing-XML-in-Android-with-SAX

how I get the html source with utf-8 format?

I write this code to get html source from a site.
HttpURLConnection connection;
OutputStreamWriter request = null;
URL url = null;
String response = null;
String parameters = "aranan="+et.getText();
try
{
url = new URL("http://www.fragmanfan.com/arama.asp");
connection = (HttpURLConnection) url.openConnection();
connection.setDoOutput(true);
connection.setRequestProperty("Content-Type", "application/x-www-form-urlencoded");
request = new OutputStreamWriter(connection.getOutputStream());
request.write(parameters);
request.flush();
String line = "";
InputStreamReader isr = new InputStreamReader(connection.getInputStream());
BufferedReader reader = new BufferedReader(isr);
StringBuilder sb = new StringBuilder();
while ((line = reader.readLine()) != null)
{
sb.append(line + "\n");
}
// Response from server after login process will be stored in response variable.
response = sb.toString();
// You can perform UI operations here
browser.loadDataWithBaseURL(null, response,"text/html", "UTF-8", null);
isr.close();
reader.close();
}
catch(IOException e)
{
// Error
}
}
});
But there is a one problem.It is : response (the variable that have html source) is not utf-8 format.
How I can fix this?
Thanks.

.
.
.
InputStreamReader isr = new InputStreamReader(connection.getInputStream(),"ISO-8859-9");
.
.
.

Since your response seems to be your HTML webpage in a single String, you should make sure that the head tag of your page cointains the label that defines the codification.. if not you can append it yourself to your StringBuilder.
Here is how you can do it:
final StringBuilder sb =
new StringBuilder("<html><head>"+ "<meta http-equiv=\"content-type\"content=\"text/html;charset=utf-8\" />"+ "</head><body>");
while ((line = reader.readLine()) != null) {
sb.append(line + "\n");
}
response = sb.toString();
sb.append(response);
sb.append("</body></html>");
and then you can properly load your HTML to your webview / browser. (this worked for me so I know for sure that it actually works =] )
p.d. make sure to accept the answer that properly answer your question so people keep answering your future questions.
https://meta.stackexchange.com/questions/5234/how-does-accepting-an-answer-work