Develop Reference

How to decode json string as UTF-8?

I've been working with json for some time and the issue is the strings I decode are encoded as Latin-1 and I cannot get it to work as UTF-8. Because of that, some characters are shown incorrectly (ex. ' shown as ').
I've read a few questions here on stackoverflow, but they doesn't seem to work.
The json structure I'm working with look like this (it is from YouTube API):
...
"items": [
{
...
"snippet": {
...
"title": "Powerbeats Pro “Totally Wireless” Except when you need a wire",
...
}
}
]
I encode it with:
response = await http.get(link, headers: {HttpHeaders.contentTypeHeader: "application/json; charset=utf-8"});
extractedData = json.decode(response.body);
dataTech = extractedData["items"];
And then what I tried was changing the second line to:
extractedData = json.decode(utf8.decode(response.body));
But this gave me an error about wrong format. So I changed it to:
extractedData = json.decode(utf8.decode(response.bodyBytes));
And this doesn't throw the error, but neither does it fix the problem. Playing around with headers does neither.
I would like the data to be stored in dataTech as they are now, but encoded as UTF-8. What am I doing wrong?

Just an aside first: UTF-8 is typically an external format, and typically represented by an array of bytes. It's what you might send over the network as part of an HTTP response. Internally, Dart stores strings as UTF-16 code points. The utf8 encoder/decoder converts between internal format strings and external format arrays of bytes.
This is why you are using utf8.decode(response.bodyBytes); taking the raw body bytes and converting them to an internal string. (response.body basically does this too, but it chooses the bytes->string decoder based on the response header charset. When this charset header is missing (as it often is) the http package picks Latin-1, which obviously doesn't work if you know that the response is in a different charset.) By using utf8.decode yourself, you are overriding the (potentially wrong) choice being made by http because you know that this particular server always sends UTF-8. (It may not, of course!)
Another aside: setting a content type header on a request is rarely useful. You typically aren't sending any content - so it doesn't have a type! And that doesn't influence the content type or content type charset that the server will send back to you. The accept header might be what you are looking for. That's a hint to the server of what type of content you'd like back - but not all servers respect it.
So why are your special characters still incorrect? Try printing utf8.decode(response.bodyBytes) before decoding it. Does it look right in the console? (It very useful to create a simple Dart command line application for this type of issue; I find it easier to set breakpoints and inspect variables in a simple ten line Dart app.) Try using something like Wireshark to capture the bytes on the wire (again, useful to have the simple Dart app for this). Or try using Postman to send the same request and inspect the response.
How are you trying to show the characters. If may simply be that the font you are using doesn't have them.

just add the header : 'Accept': 'application/json; charset=UTF-8';
it worked for me

My header looks like :
final response = await http.get(url, headers: {
'Content-Type': 'application/json; charset=UTF-8',
'Accept': 'application/json',
'Authorization': 'Bearer $token',
});
And the response is handled like:
Map<String, dynamic> data = json.decode(utf8.decode(response.bodyBytes));

Is OkHttp equivalent code really OK in Postman?

I'm trying to send a json post request to some API which in response sends a binary file back.
I'm doing well in Postman:
Header:
Body and result:
And I get the following code from Code section in Postman for Java/OKHTTP
OkHttpClient client = new OkHttpClient();
MediaType mediaType = MediaType.parse("application/json");
RequestBody body = RequestBody.create(mediaType, "{\r\n \"Text\":\"Hello\",\r\n \"APIKey\":\"MY_API_KEY\",\r\n \"Speaker\":\"Female1\",\r\n \"Format\":\"mp3/32/m\",\r\n \"Quality\":\"quality/normal\"\r\n}");
Request request = new Request.Builder()
.url("http://url/CloudService/ReadText")
.post(body)
.addHeader("content-type", "application/json")
.addHeader("cache-control", "no-cache")
.addHeader("postman-token", "0a1ce7c9-7a95-a2b9-7cde-8a7e6ce58386")
.build();
Response response = client.newCall(request).execute();
But when I use the above code in android it fails, I'm sure that I got Internet permission and the code is executed within an AsyncTask.
I'm not asking about the API or how to send json Post request to some API and get a binary file in response. I've used client.newCall(request).enqueue(new Callback(){//stuff here}); but none works. In response I got a 307 status code (instead of 200 in Postman) and no binary data at all. The API is very unclear and said nothing about the failure and I'm still working on that.
All I'm asking is that does Postman generates equivalent code for OkHttp correctly? and if not what is your suggestion for equivalent of this request in Java/OkHttp?
Just to provide another example, the following is also a working Python requests script to do the same job:
url = 'http://url/CloudService/ReadText'
api_key = 'MY_API_KEY'
body = {
'Text': 'Hello',
'Speaker': 'Female1',
'Format': 'mp3/32/m',
'Quality': 'quality/normal',
'APIKey': api_key
}
header = {
'Content-type': 'application/json'
}
r = requests.post(url, data=json.dumps(body), headers=header)

So after 3 days I found the problem, the server API did not mention that the URL endpoint must be ended with an / and even in their sample code they didn't use one.
It seems that both Postman and Python requests use an / at the end of URL in case of need, but Postman at least does not mention that in the generated equivalent code. Also OkHttp does not operate in the same manner.
However using a trailing / solved the problem.

How to avoid Retrofit to encode single quotation mark in HTTP Url?

I want to send a request to an api with a URL that includes single quotation mark characters. I'm using retrofit2.
I want this url to reach to api: http://someapp.somecompany.com/api/contents/inc()?$filter=_Id%20eq%20'57fb60534421dd35544b27a9'
I tried #Query and #QueryMap annotations with (encoded=true) and also (encoded=false) properties but none of my tries worked.
MyClient (Interface)
#PUT("api/contents/inc()")
Call<Object> addLike(#QueryMap(encoded =true) Map<String,String> options);
MainActivity (Activity)
Map<String,String> likeMap = new HashMap<>();
likeMap.put("$filter","_Id%20eq%20'57fb60534421dd35544b27a9'");
Call<Object> addLikeCall = myClient.addLike(likeMap);
addLikeCall.enqueue(new Callback<Object>() {.....});
This code gives me this output:
--> PUT http://someapp.somecompany.com/api/contents/inc()?$filter=_Id%20eq%20%2757fb60534421dd35544b27a9%27 http/1.1 (0-byte body)
<-- 500 Internal Server Error http://someapp.somecompany.com/api/contents/inc()?$filter=_Id%20eq%20%2757fb60534421dd35544b27a9%27 (132ms, 36-byte body)
How can i avoid retrofit to encode my single quotation mark in url ?
Thanks.

You can replace the ' character in your url with %27 manually if it solves your proplem.
Other ascii encoding is in this link:
http://www.w3schools.com/TAGS/ref_urlencode.asp

Retrofit 2 Multipart requests

I'm migrating my existing codebase to Retrofit 2, but having some trouble understanding the new syntax for Multipart requests. I'm also using Kotlin, although apart from a few syntax changes I think it shouldn't matter for this particular question.
Here's what I have right now:
val audioDuration = RequestBody.create(null, audioDuration.toString())
val file = RequestBody.create(MediaType.parse("audio/mp4"),
File(context.filesDir, filename).absoluteFile)
sendAudioChunk(audioDuration, file).enqueue(callback)
And here's the definition of the API:
#Multipart
#POST("path_to_request")
fun sendAudioChunk(#Part("duration") audioDuration: RequestBody,
#Part("audio") audioBlob: RequestBody) : Call<ResponseObject>
On Retrofit 1.9 I used TypedString and TypedFile for the request parameters, and now it seems one need to use RequestBody from OkHttp but I must be missing something since the request does not execute correctly.

I eventually figured it out. My web-service expects a filename for file uploads. This seems to be a work in progress support in the new Retrofit 2, but it is possible to circumvent the problem by adding it to the named parameter definition.
More details here : https://github.com/square/retrofit/issues/1140

One thing that is different is that TypedString would have a Content-Type of "text/plain; charset=UTF-8", where you are not setting a Context-Type at all for your audioDuration parameter. Try setting it to text/plain to get the same behavior as TypedString (charset will be set to utf-8 by default).
val audioDuration = RequestBody.create(MediaType.parse("text/plain"), audioDuration.toString())
If that doesn't work, you'll need to provide more information about how the "request does not execute correctly". A working request that you are trying to replicate would also be helpful.

Trying to replicate a successful POST request with JSoup - data posted to server does not get decoded

Http request header:
Host: www.mysite.com
Content-Type: application/x-www-form-urlencoded
Cookie: bbuserid=XXX; bbpassword=YYY; bbsessionhash=ZZZ
Content-Length: 252
Http request body:
message=%E4%F6%F5%FC%E4%F6%F5%FC%E4%F6%F5%FC%E4%F6%F5%FC&securitytoken=XXX&do=postreply&t=483553
Working fine! Data posted to server gets decoded on the other end and user sees orginal message which is äöõüäöõüäöõüäöõü.
Now lets try to implement this excact example with JSoup:
//request body
Map<String, String> datamap = new HashMap<String, String>();
datamap.put(Session.SESSION_SECURITYTOKEN,"XXX");
datamap.put("message", URLEncoder.encode(finalText, "ISO-8859-1"));
datamap.put("do", "postreply");
datamap.put("t", "483553");
//make a post
Jsoup.connect(url)
.header("Content-Type","application/x-www-form-urlencoded")
.timeout(10000)
.cookie(Session.COOKIE_HASH_KEY,session.bbsessionhash)
.cookie(Session.COOKIE_PASSWORD_KEY,session.bbpassword)
.cookie(Session.COOKIE_USERID_KEY,session.bbuserid)
.data(datamap).post();
My message gets posted BUT it is not decoded by the server. So when user views the message he/she sees: %E4%F6%F5%FC%E4%F6%F5%FC%E4%F6%F5%FC%E4%F6%F5%FC
Note: I am doing the post request from Android and posting data to vBulletin forum software (replay to thread).
The problem: When I send the message with JSoup, server sees it like a plain text not a encoded text. How can I make the server to understand that the message parameter holds encoded text, not plain text?

Jsoup uses UTF-8 by default to URL-encode the query string. With the current API version, you cannot change it without rebuilding the source (it's the org.jsoup.helper.DataUtil#defaultCharset constant which is been used in org.jsoup.helper.HttpConnection class). Best what you can do is to post an issue report requesting the ability to preset the charset beforehand.
Until then, you could use HttpClient or URLConnection instead which allows for a more finer grained control over sending HTTP requests. You could finally feed its response as an InputStream to Jsoup#parse() method.
Update: if the target website supports it, you could try explicitly specifying the client's used charset in the Content-Type request header:
.header("Content-Type","application/x-www-form-urlencoded;charset=UTF-8")
Note that you should not use URLEncoder#encode() yourself; let Jsoup do its job.