Develop Reference

Converting String to Android JSONObject loses utf-8

I am trying to get a (JSON formatted) String from a URL and consume it as a Json object. I lose UTF-8 encoding when I convert the String to JSONObject.
This is The function I use to connect to the url and get the string:
private static String getUrlContents(String theUrl) {
StringBuilder content = new StringBuilder();
try {
URL url = new URL(theUrl);
URLConnection urlConnection = url.openConnection();
BufferedReader bufferedReader = new BufferedReader(new InputStreamReader(urlConnection.getInputStream()));
String line;
while ((line = bufferedReader.readLine()) != null) {
content.append(line + "\n");
}
bufferedReader.close();
} catch(Exception e) {
e.printStackTrace();
}
return content.toString();
}
When I get data from server, the following code displays correct characters:
String output = getUrlContents(url);
Log.i("message1", output);
But when I convert the output string to JSONObject the Persian characters becomes question marks like this ??????. (messages is the name of array in JSON)
JSONObject reader = new JSONObject(output);
String messages = new String(reader.getString("messages").getBytes("ISO-8859-1"), "UTF-8");
Log.i("message2", messages);

You're telling Java to convert the string (with key message) to bytes using ISO-8859-1 and than to create a new String from these bytes, interpreted as UTF-8.
new String(reader.getString("messages").getBytes("ISO-8859-1"), "UTF-8");
You could simply use:
String messages = reader.getString("messages");

You can update your code as the following:
private static String getUrlContents(String theUrl) {
StringBuilder content = new StringBuilder();
try {
URL url = new URL(theUrl);
URLConnection urlConnection = url.openConnection();
BufferedReader bufferedReader = new BufferedReader(new InputStreamReader(urlConnection.getInputStream(), "utf-8"));
String line;
while ((line = bufferedReader.readLine()) != null) {
content.append(line).append("\n");
}
bufferedReader.close();
} catch(Exception e) {
e.printStackTrace();
}
return content.toString().trim();
}

You've got two encoding issues:
The server sends text encoded in a character set. When you setup your InputStreamReader, you need to pass the encoding the server used so it can be decoded properly. The character encoding is usually given in the Content-type HTTP response, in the charset field. JSON is typically UTF-8 encoded, but can also be legally UTF-16 and UTF-32, so you need to check. Without a specified encoding, your system environment will be used when marshalling bytes to Strings, and vice versa . Basically, you should always specify the charset.
String messages = new String(reader.getString("messages").getBytes("ISO-8859-1"), "UTF-8"); is obviously going to cause issues (if you have non-ascii characters) - it's encoding the string to ISO-8995-1 and then trying to decode it as UTF-8.
A simple regex pattern can be used to extract the charset value from the Content-type header before reading the inputstream. I've also included a neat InputStream -> String converter.
private static String getUrlContents(String theUrl) {
try {
URL url = new URL(theUrl);
URLConnection urlConnection = url.openConnection();
InputStream is = urlConnection.getInputStream();
// Get charset field from Content-Type header
String contentType = urlConnection.getContentType();
// matches value in key / value pair
Pattern encodingPattern = Pattern.compile(".*charset\\s*=\\s*([\\w-]+).*");
Matcher encodingMatcher = encodingPattern.matcher(contentType);
// set charsetString to match value if charset is given, else default to UTF-8
String charsetString = encodingMatcher.matches() ? encodingMatcher.group(1) : "UTF-8";
// Quick way to read from InputStream.
// \A is a boundary match for beginning of the input
return new Scanner(is, charsetString).useDelimiter("\\A").next();
} catch(Exception e) {
e.printStackTrace();
}
return null;
}

Not sure if this will help, but you might be able to do something like this:
JSONObject result = null;
String str = null;
try
{
str = new String(output, "UTF-8");
result = (JSONObject) new JSONTokener(str).nextValue();
}
catch (Exception e) {}
String messages = result.getString("messages");

Not able to POST to REST API

Today I'm making my first attempt of sending a POST request with a JSON to save some data, and I'm not being able to do so.
My app works by signing in, and then save, modify and delete data. It's already done in iOS, but since I'm new to Android, I'm not sure how to do it.
Here's my POST function:
public String POST(String targetURL, String urlParameters, String user, String pwd) {
URL url;
String u = targetURL;
HttpURLConnection connection = null;
try {
// Create connection
// u=URLEncoder.encode(u, "UTF-8");
url = new URL(u);
connection = (HttpURLConnection) url.openConnection();
// cambiarlo luego al usuario q esta logeado
String login = user + ":" + pwd;
String encoding = new String(org.apache.commons.codec.binary.Base64.encodeBase64(org.apache.commons.codec.binary.StringUtils.getBytesUtf8(login)));
connection.setRequestMethod("POST");
connection.setRequestProperty("Authorization", "Basic " + encoding);
connection.setRequestProperty("Content-Type", "plain/text");// hace q sirva con el string de json
connection.setRequestProperty("Content-Length", "" + Integer.toString(urlParameters.getBytes().length));
connection.setRequestProperty("Content-Language", "en-US");
connection.setUseCaches(false);
connection.setDoInput(true);
connection.setDoOutput(true);
connection.setReadTimeout(120000);
// Send request
DataOutputStream wr = new DataOutputStream(connection.getOutputStream());
wr.writeBytes(urlParameters);
wr.flush();
wr.close();
// Get Response
InputStream is = connection.getInputStream();
BufferedReader rd = new BufferedReader(new InputStreamReader(is));
String line;
StringBuffer response = new StringBuffer();
this.setResponseCode(connection.getResponseCode());
while ((line = rd.readLine()) != null) {
response.append(line);
response.append('\r');
}
rd.close();
return response.toString();
} catch (Exception e) {
e.printStackTrace();
return null;
} finally {
if (connection != null) {
connection.disconnect();
}
}
}
The method above is executed with Asynctask, and even if I use it to Login using Spring security, it works, and even I can save for internal usage the username, password, and secret token.
I dunno if I need to put the token in a header or something, because I already did that, with no positive results.
I'm supposing that the only permission I need to execute this is the internet one, so in my manifest file I specified that permission.
I'm going crazy with this issue, please help!
Thanks in advance.
EDIT:
Sorry guys, I'm kinda new to this way of asking, and also, not an English native speaker :P
The output I receive after sending the request, is the HTML of the page that handles logging in into the web app... I need like a json response or something like that to make sure the request was saved correctly

Try handling your cookies
CookieManager cookieManager = new CookieManager();
CookieHandler.setDefault(cookieManager);
This should be a singleton.

issue in httpurlconnection getting status 500

I am trying to login through url and i am getting status code 500 in httpurlconnevtion
public static String excutePost(String targetURL, String urlParameters)
{
URL url;
HttpURLConnection connection = null;
try {
//Create connection
url = new URL(targetURL);
connection = (HttpURLConnection)url.openConnection();
connection.setRequestMethod("POST");
connection.setRequestProperty("Content-Type",
"application/x-www-form-urlencoded");
connection.setRequestProperty("Connection", "Keep-Alive");
connection.setRequestProperty("Content-Length", "" +
Integer.toString(urlParameters.getBytes().length));
connection.setRequestProperty("Content-Language", "en-US");
connection.setRequestProperty("Accept-Encoding", "");
connection.setUseCaches (false);
connection.setDoInput(true);
connection.setDoOutput(true);
//Send request
DataOutputStream wr = new DataOutputStream (
connection.getOutputStream ());
wr.writeBytes (urlParameters);
wr.flush ();
wr.close ();
System.out.println("status :"+connection.getResponseCode());
System.out.println("getErrorStream() :"+connection.getErrorStream());
//Get Response
InputStream is = connection.getInputStream();
BufferedReader rd = new BufferedReader(new InputStreamReader(is));
String line;
StringBuffer response = new StringBuffer();
while((line = rd.readLine()) != null) {
response.append(line);
response.append('\r');
}
rd.close();
return response.toString();
} catch (Exception e) {
e.printStackTrace();
return null;
} finally {
if(connection != null) {
connection.disconnect();
}
}
}
and my params are
String urlParameters =
"pwd1=" + URLEncoder.encode("DEMO123", "UTF-8") +
"&badge=" + URLEncoder.encode("1233", "UTF-8");
i am getting logcat
status :500
getErrorStream() :libcore.net.http.FixedLengthInputStream#417bc5c0
thank you
**EDITED 2**
I have also try with
DataOutputStream dos = new DataOutputStream(connection.getOutputStream());
// Add badge
dos.writeBytes(LINE_START + BOUNDRY + LINE_END);
dos.writeBytes("Content-Disposition: form-data; name='badge';");
dos.writeBytes(LINE_END + LINE_END);
dos.writeBytes("1233");
dos.writeBytes(LINE_END);
// Add password
dos.writeBytes(LINE_START + BOUNDRY + LINE_END);
dos.writeBytes("Content-Disposition: form-data; name='pwd1';");
dos.writeBytes(LINE_END + LINE_END);
dos.writeBytes("DEMO123");
dos.writeBytes(LINE_END);

500 denotes an Internal Server Error. There is probably no error on your side, it's on the server. Even if you are sending something incorrectly and it's causing the server to return 500, it's still a server problem.
Edit:
Okey, the server should rather return something like 400 Bad Request instead of 500 Internal Server Error but I found your error now:
connection.setRequestProperty("Content-Length", Integer.toString(urlParameters.getBytes().length));
...
//Send request
DataOutputStream wr = new DataOutputStream (connection.getOutputStream ());
wr.writeBytes (urlParameters);
The problem here is that your first get the the bytes from urlParameters using getBytes which (quoting javadoc):
Encodes this String into a sequence of bytes using the platform's default charset
And then you write the bytes using DataOutputStream.writeBytes which (quoting javadoc):
Each character in the string is written out, in sequence, by discarding its high eight bits.
In summary, your Content-Length doesn't match the data. So the server returns you the
java.io.IOException: exceeded content-length limit of 20 bytes
Solution:
//consider urlParameters.getBytes("UTF-8") method instead of using default encoding
byte[] bodyData = urlParameters.getBytes();
connection.setRequestProperty("Content-Length", Integer.toString(bodyData.length));
...
//Send request
InputStream out = connection.getOutputStream();
out.write(bodyData);
Edit 2:
Edit 1 is definitely valid, however, looking over the problem again, I believe the error is definitely caused by the server. I think the server is returning a bad Content-Length header and, when the data is read on Android, the system realizes there is more data coming from the server than it should be by the Content-Length and throws an exception, also replacing the status code by 500 because it really is a server error.
Edit 3:
connection.setRequestProperty("Content-Language", "en-US");
connection.setRequestProperty("Accept-Encoding", "");
Instead of setting Content-Language which is not neccessary here, you should set Content-Encoding to UTF-8 and instead of empty Accept-Encoding, you should add the real expected MIME-type. I believe this is a server error, but you maybe it won't appear if you set the request headers correctly.

Status code 500 means Internal Server Error. Why this is thrown at you, only the server behind targetURL knows.
Verify that you're making correct usage of the API. Taking a look at the response's body (besides the status code) may provide a hint.

Sending special characters (ë ä ï) in POST body with DataOutputStream in Android

Im currently working on an Android app with heavy server side communication. Yesterday I got a bug report saying that the users aren't able to send (simple) special characters such as ëäï.
I searched but didn't find anything helpful
Possible duplicate ( without answer ):
https://stackoverflow.com/questions/12388974/android-httpurlconnection-post-special-charactes-to-rest-clint-in-android
My relevant code:
public void execute(String method) {
HttpsURLConnection urlConnection = null;
try {
URL url = new URL(this.url);
urlConnection = (HttpsURLConnection) url.openConnection();
urlConnection.setRequestMethod(method);
urlConnection.setReadTimeout(30 * 1000);
urlConnection.setDoInput(true);
if (secure)
urlConnection.setRequestProperty("Authorization", "Basic " + getCredentials());
if (body != null) {
urlConnection.setRequestProperty("Content-Type", "application/json; charset=UTF-8");
urlConnection.setFixedLengthStreamingMode(body.length());
urlConnection.setDoOutput(true);
DataOutputStream dos = new DataOutputStream(urlConnection.getOutputStream());
dos.writeBytes(body);
dos.flush();
dos.close();
}
responseCode = urlConnection.getResponseCode();
message = urlConnection.getResponseMessage();
InputStream in = null;
try {
in = new BufferedInputStream(urlConnection.getInputStream(), 2048);
} catch (Exception e) {
in = new BufferedInputStream(urlConnection.getErrorStream(), 2048);
}
if (in != null)
response = convertStreamToString(in);
} catch (UnknownHostException no_con) {
responseCode = 101;
}catch (ConnectException no_con_2){
responseCode = 101;
}catch(IOException io_ex){
if(io_ex.getMessage().contains("No authentication challenges found")){
responseCode = 401;
}else
responseCode = 101;
} catch (Exception e) {
e.printStackTrace();
} finally {
if (urlConnection != null)
urlConnection.disconnect();
}
}
body is a String ;-)
Hope we can solve this together
UPDATE:
Tried:
writeUTF()
need a server capable of understanding the modified UTF-8
byte[] buf = body.getBytes("UTF-8");
dos.write(buf, 0, buf.length);
strings work but no special chars
update: Got it working with StringEntity(* string, "UTF-8") then parse the result to a byte[] and write it with dos.write(byte[])!
--

Setting the encoding of the StringEntity did the trick for me:
StringEntity entity = new StringEntity(body, "UTF-8");
seen here: https://stackoverflow.com/a/5819465/570168

i am not totally sure buy try this utility for your case
URLEncoder.encode(string, "UTF-8")

I faced this problem in android while passing a json with special char (ñ).
In my WebApi method, [FromBody] param is giving null, it seems it can't parse the json.
I got it working by getting bytes as UTF-8 then writing it in DataOutputStream (Client-side fix).
byte[] b = jsonString.getBytes("UTF-8");
os.write(b, 0, b.length);

POST with Basic Auth fails on Android but works in C#

I have an app I am developing that requires me to post data to a 3rd party API. I have been struggling with authentication since the beginning and kept putting off further and further, and now I'm stuck.
I have tried using an Authenticator, but have read all about how there appears to be a bug in certain Android versions: Authentication Example
I have tried several different options, including the Apache Commons HTTP Library with no success. After all of this, I decided to make sure that the API wasn't the pain point. So I wrote a quick WinForms program to test the API, which worked perfectly on the first try. So, the idea that I'm working from and the API I working with both seem fine, but I am in desperate need of some guidance as to why the Java code isn't working.
Examples follow:
C# Code that works everytime:
System.Net.ServicePointManager.Expect100Continue = false;
// Create a request using a URL that can receive a post.
WebRequest request = WebRequest.Create(addWorkoutUrl);
// Set the Method property of the request to POST.
request.Method = "POST";
// Create POST data and convert it to a byte array.
string postData = "distance=4000&hours=0&minutes=20&seconds=0&tenths=0&month=08&day=01&year=2011&typeOfWorkout=standard&weightClass=H&age=28";
byte[] byteArray = Encoding.UTF8.GetBytes(postData);
// Set the ContentType property of the WebRequest.
request.Headers["X-API-KEY"] = apiKey;
request.Headers["Authorization"] = "Basic " + Convert.ToBase64String(Encoding.Default.GetBytes("username:password"));
request.ContentType = "application/x-www-form-urlencoded";
// Set the ContentLength property of the WebRequest.
request.ContentLength = byteArray.Length;
// Get the request stream.
Stream dataStream = request.GetRequestStream();
// Write the data to the request stream.
dataStream.Write(byteArray, 0, byteArray.Length);
// Close the Stream object.
dataStream.Close();
// Get the response.
WebResponse response = request.GetResponse();
// Display the status.
MessageBox.Show(((HttpWebResponse)response).StatusDescription);
// Get the stream containing content returned by the server.
dataStream = response.GetResponseStream();
// Open the stream using a StreamReader for easy access.
StreamReader reader = new StreamReader(dataStream);
// Read the content.
string responseFromServer = reader.ReadToEnd();
// Display the content.
MessageBox.Show(responseFromServer);
// Clean up the streams.
reader.Close();
dataStream.Close();
response.Close();
Java code for Android that currently returns a 500:Internal Server Error, though I believe this is my fault.
URL url;
String data = "distance=4000&hours=0&minutes=20&seconds=0&tenths=0&month=08&day=01&year=2011&typeOfWorkout=standard&weightClass=H&age=28";
HttpURLConnection connection = null;
//Create connection
url = new URL(urlBasePath);
connection = (HttpURLConnection)url.openConnection();
connection.setConnectTimeout(10000);
connection.setUseCaches(false);
connection.setRequestProperty("User-Agent","Mozilla/5.0 ( compatible ) ");
connection.setRequestProperty("Accept","*/*");
connection.setRequestProperty("X-API-KEY", apiKey);
connection.setRequestProperty("Authorization", "Basic " +
Base64.encode((username + ":" + password).getBytes("UTF-8"), Base64.DEFAULT));
connection.setRequestProperty("Content-Type", "application/x-www-form-urlencoded");
connection.setRequestProperty("Content-Length", "" + Integer.toString(data.getBytes("UTF-8").length));
DataOutputStream wr = new DataOutputStream(connection.getOutputStream());
wr.write(data.getBytes("UTF-8"));
wr.flush();
wr.close();
statusCode = connection.getResponseCode();
statusReason = connection.getResponseMessage();
//At this point, I have the 500 error...

I figured out the problem, and the solution finally after stumbling across the root cause as mentioned in the comment above.
I was using Base64.encode() in my example, but I needed to be using Base64.encodeToString().
The difference being that encode() returns a byte[] and encodeToString() returns the string I was expecting.
Hopefully this will help somebody else who is caught by this.

Here's a nicer method to do to the POST.
install-package HttpClient
Then:
public void DoPost()
{
var httpClient = new HttpClient();
var creds = string.Format("{0}:{1}", _username, _password);
var basicAuth = string.Format("Basic {0}", Convert.ToBase64String(Encoding.UTF8.GetBytes(creds)));
httpClient.DefaultRequestHeaders.Add("Authorization", basicAuth);
var post = httpClient.PostAsync(_url,
new FormUrlEncodedContent(new Dictionary<string, string>
{
{ "name", "Henrik" },
{ "age", "99" }
}));
post.Wait();
}

I have tried this in java
import java.io.*;
import java.net.*;
class download{
public static void main(String args[]){
try{
String details = "API-Key=e6d871be90a689&orderInfo={\"booking\":{\"restaurantinfo\":{\"id\":\"5722\"},\"referrer\":{\"id\": \"9448476530\" }, \"bookingdetails\":{\"instructions\":\"Make the stuff spicy\",\"bookingtime\": \"2011-11-09 12:12 pm\", \"num_guests\": \"5\"}, \"customerinfo\":{\"name\":\"Ramjee Ganti\", \"mobile\":\"9345245530\", \"email\": \"sajid#pappilon.in\", \"landline\":{ \"number\":\"0908998393\",\"ext\":\"456\"}}}}";
Authenticator.setDefault(new Authenticator() {
protected PasswordAuthentication getPasswordAuthentication() {
return new PasswordAuthentication("admin", "1234".toCharArray());
}
});
HttpURLConnection conn = null;
//URL url = new URL("http://api-justeat.in/ma/orders/index");
URL url = new URL("http://api.geanly.in/ma/order_ma/index");
conn = (HttpURLConnection) url.openConnection();
conn.setDoOutput(true);
conn.setDoInput (true);
conn.setRequestMethod("POST");
//conn.setRequestMethod(HttpConnection.POST);
DataOutputStream outStream = new DataOutputStream(conn.getOutputStream());
outStream.writeBytes(details);
outStream.flush();
outStream.close();
//Get Response
InputStream is = conn.getInputStream();
BufferedReader rd = new BufferedReader(new InputStreamReader(is));
String line;
StringBuffer response = new StringBuffer();
while((line = rd.readLine()) != null) {
System.out.println(line);
}
rd.close();
System.out.println(conn.getResponseCode() + "\n\n");
}catch(Exception e){
System.out.println(e);
}
}
}
this could help.