I've been working with json for some time and the issue is the strings I decode are encoded as Latin-1 and I cannot get it to work as UTF-8. Because of that, some characters are shown incorrectly (ex. ' shown as ').
I've read a few questions here on stackoverflow, but they doesn't seem to work.
The json structure I'm working with look like this (it is from YouTube API):
...
"items": [
{
...
"snippet": {
...
"title": "Powerbeats Pro “Totally Wireless” Except when you need a wire",
...
}
}
]
I encode it with:
response = await http.get(link, headers: {HttpHeaders.contentTypeHeader: "application/json; charset=utf-8"});
extractedData = json.decode(response.body);
dataTech = extractedData["items"];
And then what I tried was changing the second line to:
extractedData = json.decode(utf8.decode(response.body));
But this gave me an error about wrong format. So I changed it to:
extractedData = json.decode(utf8.decode(response.bodyBytes));
And this doesn't throw the error, but neither does it fix the problem. Playing around with headers does neither.
I would like the data to be stored in dataTech as they are now, but encoded as UTF-8. What am I doing wrong?
Just an aside first: UTF-8 is typically an external format, and typically represented by an array of bytes. It's what you might send over the network as part of an HTTP response. Internally, Dart stores strings as UTF-16 code points. The utf8 encoder/decoder converts between internal format strings and external format arrays of bytes.
This is why you are using utf8.decode(response.bodyBytes); taking the raw body bytes and converting them to an internal string. (response.body basically does this too, but it chooses the bytes->string decoder based on the response header charset. When this charset header is missing (as it often is) the http package picks Latin-1, which obviously doesn't work if you know that the response is in a different charset.) By using utf8.decode yourself, you are overriding the (potentially wrong) choice being made by http because you know that this particular server always sends UTF-8. (It may not, of course!)
Another aside: setting a content type header on a request is rarely useful. You typically aren't sending any content - so it doesn't have a type! And that doesn't influence the content type or content type charset that the server will send back to you. The accept header might be what you are looking for. That's a hint to the server of what type of content you'd like back - but not all servers respect it.
So why are your special characters still incorrect? Try printing utf8.decode(response.bodyBytes) before decoding it. Does it look right in the console? (It very useful to create a simple Dart command line application for this type of issue; I find it easier to set breakpoints and inspect variables in a simple ten line Dart app.) Try using something like Wireshark to capture the bytes on the wire (again, useful to have the simple Dart app for this). Or try using Postman to send the same request and inspect the response.
How are you trying to show the characters. If may simply be that the font you are using doesn't have them.
just add the header : 'Accept': 'application/json; charset=UTF-8';
it worked for me
My header looks like :
final response = await http.get(url, headers: {
'Content-Type': 'application/json; charset=UTF-8',
'Accept': 'application/json',
'Authorization': 'Bearer $token',
});
And the response is handled like:
Map<String, dynamic> data = json.decode(utf8.decode(response.bodyBytes));
Related
I have a react-native app that needs to send video/images to my server. I already know that normal posts work but when I attempt to send a formData object, it seems to never leave the phone. Here is my code.
// method = 'POST';
// body = new formData();
// body contains text data and image/video file
const post = (url, body, token, method) => {
let xhr = new XMLHttpRequest();
xhr.open(method, url);
xhr.setRequestHeader('Authorization', 'Bearer' + token);
xhr.setRequestHeader('Content-Type', 'multipart/form-data; boundary=---------------------------7692764ac82');
xhr.send(body);
console.log(xhr);
return xhr.response;
}
body is a formData object that contains an image/video. In the object that xhr prints at the console log the _response contains "Binary FormData part needs a content-type header." But it seems I set it correctly right?
Please help, there are other similar questions but I have run out of ideas. I have also tried using fetch with no success.
The error message is not about the content-type header for the request (which you have set), it's about the content-type header for the part (which you did not show, so we have to assume it is missing).
When you add the parts to the FormData, don't forget to include a type. Example for an image:
const body = new FormData();
// ...
body.append('image', {
uri: 'file:///...',
type: 'image/jpeg', // <- Did you miss that one?
name: 'someName',
});
With the type properly set, the React Native runtime should add a content-type header for the part. This is done in FormData.js at line 79 in v0.46.0 (wherein value is the value for your type property):
if (typeof value.type === 'string') {
headers['content-type'] = value.type;
}
So, when type is missing, then content-type header for the part is missing, and then on Android you end up here, where you can see the origin of your error message.
This exact error and root cause is discussed in that GitHub issue.
I have an upstream server that accepts image submissions using rest. The submitted image is part of a JSON payload similar to this one
{
"name": "Blah.jpg",
"uploader": "user1",
"image": "<base64.....>"
}
Using this strategy works for small images but generates Out of Memory errors on larger images.
Is it possible to stream the base64 component of the image? Pass in something like an iterator that will be used to read chunks of the image, base64 them and send them directly to the network?
Not with Gson or Moshi. Both of these libraries require strings to be in memory before emitting them to a stream.
I solved this with the following, in a class that extends okhttp3.RequestBody:
private void writeFile(File file, BufferedSink sink) throws IOException {
byte buf[] = new byte[3000];
try (FileInputStream fin = new FileInputStream(file)) {
while (fin.read(buf) >= 0) {
byte encoded[] = Base64.encodeBase64(buf);
sink.write(encoded);
}
}
}
It uses Android's android.util.Base64 Apache Commons' org.apache.commons.codec.binary.Base64 to encode a buffered chunk of data.
I ended up writing the other json fields separately, with enough granularity that I could insert the file record exactly where I needed to.
EDIT:
As you can see in my edits above, I ended up switching to Apache commons-codec, via compile 'commons-codec:commons-codec:1.5' in my build.gradle file.
I didn't have time to investigate why the Android SDK solution didn't work. I tried their Base64.encode(buf, Base64.NO_WRAP) as suggested elsewhere - supposedly equivalent to Apache Commons' encodeBase64(byte[]) - but this did not work, hence the switch.
The problem could have been on our backend, so don't rule out Android SDK's solution based on my post alone - I just wanted to add this note so readers can see the code snippet that actually worked for me in the end.
I am calling a web service through async task which is returning text in strange format. here is sample string
dhmot_enot = Ï. ÎÎ®Î¼Î¿Ï ÎοÏλαÏ
zoe_name = Î.Î.Î: ÎÏÎ½ÎµÏ Î ÏοÏÏαÏÎ¯Î±Ï ÎÏοÏÏ Î¥Î¼Î·ÏÏοÏ
zones_zoe = ÎΩÎÎ Î: ÎÎ ÎÎΥΤΠΠΡÎΣΤÎΣÎΠΤÎΣ ΦΥΣÎΣ
zoe_fek = 187/Î/2011
fek_rel = 544/Î/1978
yphresia = Î¥.ÎÎÎ Î. ÎάÏηÏ-ÎοÏλαÏ-ÎοÏλιαγμÎνηÏ
How to find and resolve this ?
Update 1
Here is the actual service link that i am calling from server (works well in web browser ) but when i call from android it looks like above
http://geo-polis.gistemp.com/geoserver/wms?service=WMS&version=1.1.1&srs=EPSG:4326&bbox=23.733829893171787,37.75098946973509,23.839769437909126,37.89294194933182&styles=&&buffer=20&OUTPUTFORMAT=json&request=GetFeatureInfo&layers=geopolis:oria_eniaiou_dhmou&query_layers=geopolis:oria_eniaiou_dhmou&width=1080&height=1832&x=690&y=821
The response is a normal UTF-8 encoded stream of data. To see this, go to the URL you show in your post in the browser, and look at the encoding it picked automatically: it'll show unicode/utf-8 as character encoding for the response. If you change that, forcing the browser to decode it as if it's ANSI encoded (windows codepage 1252/ISO-8859-15) then the text turns into the gibberish you were showing in your question, so: you're not decoding the data correctly, and need to make sure to decode as utf8.
I try enable GZip compression for improve connection performance between client and server, but it seems that WebView doesn't decompress GZip-response.
Response Header(from server) contains:
Content-Encoding:gzip
Content-Type:application/gzip
Body contains compressed JSON
On the Desktop browsers all fine,
but if it is android device in Chrome DevTools I see compressed body.
Mobile app doesn't decompress response body.
Created issue https://issues.apache.org/jira/browse/CB-9427
When I have changed content type into 'application/json' problem was solved.
Here sample from Grape framework app:
module App
class Users < API
resource :users do
format :json
content_type :txt, "application/json"
parser :json, nil
desc "Get users"
params do
optional :role, type: String, desc: "User's role"
end
get '/all' do
users = User.all
path = "#{Rails.root.to_s}/tmp/user_response.gz"
FileUtils.rm_rf(path, secure: true)
Zlib::GzipWriter.open(path){|gz| gz.write(users.to_json) }
content_type "application/json"
header['Content-Encoding'] = 'gzip';
env['api.format'] = :json
File.open(path).read
end
end
end
end
Issue https://issues.apache.org/jira/browse/CB-9427 was closed.
Thanks to cordova support team!
Http request header:
Host: www.mysite.com
Content-Type: application/x-www-form-urlencoded
Cookie: bbuserid=XXX; bbpassword=YYY; bbsessionhash=ZZZ
Content-Length: 252
Http request body:
message=%E4%F6%F5%FC%E4%F6%F5%FC%E4%F6%F5%FC%E4%F6%F5%FC&securitytoken=XXX&do=postreply&t=483553
Working fine! Data posted to server gets decoded on the other end and user sees orginal message which is äöõüäöõüäöõüäöõü.
Now lets try to implement this excact example with JSoup:
//request body
Map<String, String> datamap = new HashMap<String, String>();
datamap.put(Session.SESSION_SECURITYTOKEN,"XXX");
datamap.put("message", URLEncoder.encode(finalText, "ISO-8859-1"));
datamap.put("do", "postreply");
datamap.put("t", "483553");
//make a post
Jsoup.connect(url)
.header("Content-Type","application/x-www-form-urlencoded")
.timeout(10000)
.cookie(Session.COOKIE_HASH_KEY,session.bbsessionhash)
.cookie(Session.COOKIE_PASSWORD_KEY,session.bbpassword)
.cookie(Session.COOKIE_USERID_KEY,session.bbuserid)
.data(datamap).post();
My message gets posted BUT it is not decoded by the server. So when user views the message he/she sees: %E4%F6%F5%FC%E4%F6%F5%FC%E4%F6%F5%FC%E4%F6%F5%FC
Note: I am doing the post request from Android and posting data to vBulletin forum software (replay to thread).
The problem: When I send the message with JSoup, server sees it like a plain text not a encoded text. How can I make the server to understand that the message parameter holds encoded text, not plain text?
Jsoup uses UTF-8 by default to URL-encode the query string. With the current API version, you cannot change it without rebuilding the source (it's the org.jsoup.helper.DataUtil#defaultCharset constant which is been used in org.jsoup.helper.HttpConnection class). Best what you can do is to post an issue report requesting the ability to preset the charset beforehand.
Until then, you could use HttpClient or URLConnection instead which allows for a more finer grained control over sending HTTP requests. You could finally feed its response as an InputStream to Jsoup#parse() method.
Update: if the target website supports it, you could try explicitly specifying the client's used charset in the Content-Type request header:
.header("Content-Type","application/x-www-form-urlencoded;charset=UTF-8")
Note that you should not use URLEncoder#encode() yourself; let Jsoup do its job.