Develop Reference

How to decode json string as UTF-8?

I've been working with json for some time and the issue is the strings I decode are encoded as Latin-1 and I cannot get it to work as UTF-8. Because of that, some characters are shown incorrectly (ex. ' shown as ').
I've read a few questions here on stackoverflow, but they doesn't seem to work.
The json structure I'm working with look like this (it is from YouTube API):
...
"items": [
{
...
"snippet": {
...
"title": "Powerbeats Pro “Totally Wireless” Except when you need a wire",
...
}
}
]
I encode it with:
response = await http.get(link, headers: {HttpHeaders.contentTypeHeader: "application/json; charset=utf-8"});
extractedData = json.decode(response.body);
dataTech = extractedData["items"];
And then what I tried was changing the second line to:
extractedData = json.decode(utf8.decode(response.body));
But this gave me an error about wrong format. So I changed it to:
extractedData = json.decode(utf8.decode(response.bodyBytes));
And this doesn't throw the error, but neither does it fix the problem. Playing around with headers does neither.
I would like the data to be stored in dataTech as they are now, but encoded as UTF-8. What am I doing wrong?

Just an aside first: UTF-8 is typically an external format, and typically represented by an array of bytes. It's what you might send over the network as part of an HTTP response. Internally, Dart stores strings as UTF-16 code points. The utf8 encoder/decoder converts between internal format strings and external format arrays of bytes.
This is why you are using utf8.decode(response.bodyBytes); taking the raw body bytes and converting them to an internal string. (response.body basically does this too, but it chooses the bytes->string decoder based on the response header charset. When this charset header is missing (as it often is) the http package picks Latin-1, which obviously doesn't work if you know that the response is in a different charset.) By using utf8.decode yourself, you are overriding the (potentially wrong) choice being made by http because you know that this particular server always sends UTF-8. (It may not, of course!)
Another aside: setting a content type header on a request is rarely useful. You typically aren't sending any content - so it doesn't have a type! And that doesn't influence the content type or content type charset that the server will send back to you. The accept header might be what you are looking for. That's a hint to the server of what type of content you'd like back - but not all servers respect it.
So why are your special characters still incorrect? Try printing utf8.decode(response.bodyBytes) before decoding it. Does it look right in the console? (It very useful to create a simple Dart command line application for this type of issue; I find it easier to set breakpoints and inspect variables in a simple ten line Dart app.) Try using something like Wireshark to capture the bytes on the wire (again, useful to have the simple Dart app for this). Or try using Postman to send the same request and inspect the response.
How are you trying to show the characters. If may simply be that the font you are using doesn't have them.

just add the header : 'Accept': 'application/json; charset=UTF-8';
it worked for me

My header looks like :
final response = await http.get(url, headers: {
'Content-Type': 'application/json; charset=UTF-8',
'Accept': 'application/json',
'Authorization': 'Bearer $token',
});
And the response is handled like:
Map<String, dynamic> data = json.decode(utf8.decode(response.bodyBytes));

Sending a post in android with bad encoding

I'm sending post content to a server from android.
The problem is that the data at the server arrives wrong, with encoding problems, for example "{" arrives as "%7B%".
This is the code from android:
RequestParams params = new RequestParams();
params.put("alta", "{s}");
String ruta = "http://www.something.com/receive";
client.post(ruta, params,
new AsyncHttpResponseHandler() {
#Override
public void onSuccess(String response) {
}
}
The server part is just receiving this data, like:
$data = $this->request->data;
$data =file_get_contents('php://input');

This issue is not directly related to text encoding per se.
As can be seen from the docs for RequestParams, text values are directly included in the url. As all text that is included in URLs has to be encoded to only include characters that are allowed in URLs (ASCII), text is url encoded.
AsyncHttpClient automatically does that encoding in the background, so you receive the strings in encoded form on the php side.
In order to get the original text you sent, you can use the rawurldecode() or urldecode() function on the php side to decode the encoded string you receive.

You need to use URLEncoder.encode(...) the the data part of you request.
At the server URL decode it.
You should be fine.

Hostname consist of hyphen is giving IllegalArgumentException in android

I have an url like http://ashok-reddy:8080/hyd which consists hyphen. While making Http Post request, I am getting IllegalArgumentException saying Host name may not be null . I have tried with replacing the hyphen with its hexadecimal value and also tried converting using URLEncoder/Uri.encode(). But nothing has been worked till now.
mHttpPost = new HttpPost("ashok-reddy:8080/hyd");
mEnvelope = new SoapSerializationEnvelope(SoapEnvelope.VER11);
mEnvelope.encodingStyle = SoapSerializationEnvelope.ENC;
mStringEntity = new StringEntity(soapData, HTTP.UTF_8);
mStringEntity.setContentType(mContext.getString(R.string.text_xml_content));
mHttpPost.setEntity(mStringEntity);
mHttpResponse = mHttpClient.execute(mHttpPost);
Can anyone please help on this?
Thanks in advance.
Arindam

If you are using a method like URLEncoder, you should not pass the full URL, because it will escape even '//' symbols in url. For example, it will encode :// into %3A%2F%2F
Pass to the function just the parameters list you need to encode to escape special characters.
EDIT:
As I can see you are using: mHttpPost = new HttpPost("ashok-reddy:8080/hyd");
instead of: mHttpPost = new HttpPost("http;//ashok-reddy:8080/hyd");

Changing UTF encoding in android

I'm sending data between my Android device and server and for some reason, all of my é gets displayed as é. But it gets properly displayed on my browser (same data). Through some searching, I discovered that the UTF encoding may be different from my browser and my device.
How can I display TextViews with a certain UTF encoding? (UTF-8). Or is there an app-wide setting that I can set? Thanks in advance.

Sorry, found a duplicate and solution here:
UTF8 Encoding in Android when invoking REST webservice
HttpResponse response = client.execute(request);
... // probably some other code to check for HTTP response status code
HttpEntity responseEntity = response.getEntity();
String xml = EntityUtils.toString(responseEntity, HTTP.UTF_8);

Android default charset when sending http post/put - Problems with special characters

I have configured the apache httpClient like so:
HttpProtocolParams.setContentCharset(httpParameters, "UTF-8");
HttpProtocolParams.setHttpElementCharset(httpParameters, "UTF-8");
I also include the http header "Content-Type: application/json; charset=UTF-8" for all http post and put requests.
I am trying to send http post/put requests with a json body that contains special characters (ie. chinese characters via the Google Pinyin keyboard, symbols, etc.) The characters appear as gibberish in the logs but I think this is because DDMS does not support UTF-8, as descibed in this issue.
The problem is when the server receives the request, it sometimes doesn't see the characters at all (especially the Chinese characters), or it becomes meaningless garbage when we retrieve it through a GET request.
I also tried putting 250 non-ascii characters in a single field because that particular field should be able to take up to 250 characters. However, it fails to validate at the server side which claims that the 250 character limit has been exceeded. 250 ASCII characters work just fine.
The server dudes claim that they support UTF-8. They even tried simulating a post request that contains Chinese characters, and the data was received by the server just fine. However, the guy (a Chinese guy) is using a Windows computer with the Chinese language pack installed (I think, because he can type Chinese characters on his keyboard).
I'm guessing that the charsets being used by the Android client and the server (made by Chinese guys btw) are not aligned. But I do not know which one is at fault since the server dudes claim that they support UTF-8, and our rest client is configured to support UTF-8.
This got me wondering on what charset Android uses by default on all text input, and if it can be changed to a different one programatically. I tried to find resources on how to do this on input widgets but I did not find anything useful.
Is there a way to set the charset for all input widgets in Android? Or maybe I missed something in the rest client configuration? Or maybe, just maybe, the server dudes are not using UTF-8 at their servers and used Windows charsets instead?

Apparently, I forgot to set the StringEntity's charset to UTF-8. These lines did the trick:
httpPut.setEntity(new StringEntity(body, HTTP.UTF_8));
httpPost.setEntity(new StringEntity(body, HTTP.UTF_8));
So, there are at least two levels to set the charset in the Android client when sending an http post with non-ascii characters.
The rest client itself itself
The StringEntity
UPDATE: As Samuel pointed out in the comments, the modern way to do it is to use a ContentType, like so:
final StringEntity se = new StringEntity(body, ContentType.APPLICATION_JSON);
httpPut.setEntity(se);

I know this post is a bit old but nevertheless here is a solution:
Here is my code for posting UTF-8 strings (it doesn't matter if they are xml soap or json) to a server. I tried it with cyrillic, hash values and some other special characters and it works like a charm. It is a compilation of many solutions I found through the forums.
HttpParams httpParameters = new BasicHttpParams();
HttpProtocolParams.setContentCharset(httpParameters, HTTP.UTF_8);
HttpProtocolParams.setHttpElementCharset(httpParameters, HTTP.UTF_8);
HttpClient client = new DefaultHttpClient(httpParameters);
client.getParams().setParameter("http.protocol.version", HttpVersion.HTTP_1_1);
client.getParams().setParameter("http.socket.timeout", new Integer(2000));
client.getParams().setParameter("http.protocol.content-charset", HTTP.UTF_8);
httpParameters.setBooleanParameter("http.protocol.expect-continue", false);
HttpPost request = new HttpPost("http://www.server.com/some_script.php?sid=" + String.valueOf(Math.random()));
request.getParams().setParameter("http.socket.timeout", new Integer(5000));
List<NameValuePair> postParameters = new ArrayList<NameValuePair>();
// you get this later in php with $_POST['value_name']
postParameters.add(new BasicNameValuePair("value_name", "value_val"));
UrlEncodedFormEntity formEntity = new UrlEncodedFormEntity(postParameters, HTTP.UTF_8);
request.setEntity(formEntity);
HttpResponse response = client.execute(request);
in = new BufferedReader(new InputStreamReader(response.getEntity().getContent()));
StringBuffer sb = new StringBuffer("");
String line = "";
String lineSeparator = System.getProperty("line.separator");
while ((line = in.readLine()) != null) {
sb.append(line);
sb.append(lineSeparator);
}
in.close();
String result = sb.toString();
I hope that someone will find this code helpful. :)

You should set charset of your string entity to UTF-8:
StringEntity stringEntity = new StringEntity(urlParameters, HTTP.UTF_8);

You can eliminate the server as the problem by using curl to send the same data.
If it works with curl use --trace to check the output.
Ensure you are sending the content body as bytes. Compare the HTTP request from Android with the output from the successful curl request.