I have made a HTTP-post inside my android application. Values are sent as strings from my app to my webserver. Problem is, the values are not in UTF-8 as I want them to be. My webserver has UTF-8 encoding so I know that there is code inside my app that I need to change.
See my snippet below:
private void sendPostRequest(String facebookId, String name, String email) {
class SendPostReqAsyncTask extends AsyncTask<String, Void, String>{
#Override
protected String doInBackground(String... bcs) {
String bcFacebookId = bcs[0];
String bcName = bcs[1];
String bcEmail = bcs[2];
HttpClient httpClient = new DefaultHttpClient();
HttpPost httpPost = new HttpPost("URL");
BasicNameValuePair facebookIdBasicNameValuePair = new BasicNameValuePair("bcFacebookId", bcFacebookId);
BasicNameValuePair nameBasicNameValuePair = new BasicNameValuePair("bcName", bcName);
BasicNameValuePair emailBasicNameValiePair = new BasicNameValuePair("bcEmail", bcEmail);
List<NameValuePair> nameValuePairList = new ArrayList<NameValuePair>();
nameValuePairList.add(facebookIdBasicNameValuePair);
nameValuePairList.add(nameBasicNameValuePair);
nameValuePairList.add(emailBasicNameValiePair);
try {
UrlEncodedFormEntity urlEncodedFormEntity = new UrlEncodedFormEntity(nameValuePairList);
httpPost.setEntity(urlEncodedFormEntity);
try {
HttpResponse httpResponse = httpClient.execute(httpPost);
InputStream inputStream = httpResponse.getEntity().getContent();
InputStreamReader inputStreamReader = new InputStreamReader(inputStream);
BufferedReader bufferedReader = new BufferedReader(inputStreamReader);
StringBuilder stringBuilder = new StringBuilder();
String bufferedStrChunk = null;
while((bufferedStrChunk = bufferedReader.readLine()) != null){
stringBuilder.append(bufferedStrChunk);
}
return stringBuilder.toString();
} catch (ClientProtocolException cpe) {
cpe.printStackTrace();
} catch (IOException ioe) {
System.out.println("Second Exception caz of HttpResponse :" + ioe);
ioe.printStackTrace();
}
} catch (UnsupportedEncodingException uee) {
System.out.println("An Exception given because of UrlEncodedFormEntity argument :" + uee);
uee.printStackTrace();
}
return null;
}
For an example, the letter 'ö' becomes a '?'. How do I fix this?
Cheers!
The biggest single reason that characters get converted into question marks is the conversion of characters to bytes, and then back into characters, not matching.
The code you have supplied has this line:
InputStreamReader inputStreamReader = new InputStreamReader(inputStream);
This is problematic because you are not specifying how to convert the bytes into characters. Instead you probably want this:
InputStreamReader inputStreamReader = new InputStreamReader(inputStream, "UTF-8");
What you specify for the character encoding will depend upon the character encoding that you have specified elsewhere. Without specifying the character encoding, you will get the "default" character encoding, and that depends upon settings in both the client and the server. Java uses Unicode, and UTF-8 is the only encoding that will preserve all the characters that Java allows.
For debugging, you may want to use the InputStream and retrieve bytes from that, and print out the byte values, in order to verify that they are indeed UTF-8 encoded representations of the original character values. The proper encoding of 'ö' (x00F6) is 'ö' (x00C3 x00B6).
You will also need to assure that the original POST request is properly UTF-8 encoded. The UrlEncodedFormEntity class also uses the default character encoding, which might not be UTF-8. Change this:
UrlEncodedFormEntity uefe = new UrlEncodedFormEntity(nameValuePairList);
to
UrlEncodedFormEntity uefe = new UrlEncodedFormEntity(nameValuePairList, "UTF-8");
if database coding is set properly + table coding is set properly + columns coding set properly, then all data are stored properly. That's the first part. Now the second, important part - make sure you have this command after your mysql connection : SET NAMES utf8
This was my case for the same issue. Hope this this will work for you as well.
Related
I am using HttpClient 4.3.6 to perform http GET and POST requests. Right now I am using multipartentity to send a few string parameters and an image in the form of a file. I am able to successfully post the data but my problem comes in when I get the HTTP response. The response contains json data.
What happens is the HTTP response is incomplete and when i try to create a json object with the data i get jsonexception error saying:
Unterminated object at character 407.
I noticed that the response does not contain closed braces. Is this a problem on android or should I check the server? Because I am able to see the data properly on postman and on ios. I have never faced this issue before and don't know how to solve this.
This is my code to post and get the response:
#Override
protected String doInBackground(String... params) {
try {
String url = params[0];
HttpClient httpclient = new DefaultHttpClient();
HttpPost httppost = new HttpPost(url);
MultipartEntity entity = new MultipartEntity();
ByteArrayOutputStream baos = new ByteArrayOutputStream();
bitmap.compress(Bitmap.CompressFormat.JPEG, 100, baos);
byte[] imageBytes = baos.toByteArray();
ByteArrayBody bab = new ByteArrayBody(imageBytes, "image.jpg");
entity.addPart("image_data", bab);
entity.addPart("action", new StringBody("1", "text/plain", Charset.forName("UTF-8")));
entity.addPart("name", new StringBody("asdfg", "text/plain", Charset.forName("UTF-8")));
entity.addPart("user_id", new StringBody("157", "text/plain", Charset.forName("UTF-8")));
entity.addPart("birthday", new StringBody("18-04-1995", "text/plain", Charset.forName("UTF-8")));
entity.addPart("gender", new StringBody("male", "text/plain", Charset.forName("UTF-8")));
entity.addPart("is_jlpt_student", new StringBody(String.valueOf(0), "text/plain", Charset.forName("UTF-8")));
entity.addPart("relationship", new StringBody("Father", "text/plain", Charset.forName("UTF-8")));
entity.addPart("relationship_id", new StringBody(String.valueOf(10002), "text/plain", Charset.forName("UTF-8")));
entity.addPart("is_creator", new StringBody(String.valueOf(1), "text/plain", Charset.forName("UTF-8")));
entity.addPart("email", new StringBody(email, "text/plain", Charset.forName("UTF-8")));
httppost.setEntity(entity);
HttpResponse resp = httpclient.execute(httppost);
String response = EntityUtils.toString(resp.getEntity());
Log.i("HttpResponse", response);
return response;
} catch (ClientProtocolException e) {
e.printStackTrace();
} catch (IOException e) {
e.printStackTrace();
}
return null;
}
#Override
protected void onPostExecute (String result) {
super.onPostExecute(result);
JSONObject jsonObject = null;
try {
jsonObject = new JSONObject(result);
JSONObject json_data = jsonObject.getJSONObject("data");
String json_userid = json_data.getString("user_id");
String json_username = json_data.getString("name");
String json_email = json_data.getString("email");
String json_country = json_data.getString("country_code");
String json_imagefilename = json_data.getString("image_filename");
String json_imgurl = json_data.getString("image_url");
Toast.makeText(ParentGuardianProfile.this, "ImageFile " + json_imagefilename, Toast.LENGTH_SHORT).show();
User new_user = userdao.createUser(json_userid, json_username, json_email,json_imagefilename,json_country,selectedImageUri.toString(), 1);
Log.i("SQLITE", "added user : " + new_user.getmUserName() + new_user.getmId());
} catch (JSONException e) {
e.printStackTrace();
}
}
And my json response is :
{"status":1,"message":"success","data":{"child_id":"381","name":"asdfg","image_filename":"C201603021734476.jpg","image_url":"https:\/\/innokid.blob.core.windows.net\/media\/child\/381.jpg","birthday":"18-04-1995","gender":"male","is_jltp_student":"0","relationship":"Father","relationship_id":"10002","is_creator":1,"rank":1,"qrcode_url":"http:\/\/innokid.azurewebsites.net\/uploads\/qrcode\/child_381.png"
I tried using String buffer as suggested in this post String is being truncated when its too long . But i still get the same result.
Code looks ok at first glance.
How do you got know that the json data is cut? Logcat can truncate text. Debugger should be more reliable in this case.
Try to generate this same request with some tools like curl / SoapUI and validate JSON you got with some formatter / validator (you'll easily find a few of such tools).
It's beyond the range of question, but using raw Android built-in communication libraries seems to be a little bit masochistic. Have you ever consider to use Retrofit?
I think this code is problematic String response = EntityUtils.toString(resp.getEntity());
may be you should use some other function to convert response toString...
Apparently the json is missing two curly brackets '}}' at the end, which can happen due to some bug in the toString code.
I pulled up an old project that was using the org.apache.http stuff and below is how I was parsing the response. As you can see it is rather cumbersome. There are many tested and maintained libraries out there that are better suited to this kind of heavy-lifting.
// Get hold of the response entity (-> the data):
HttpEntity entity = response.getEntity();
if (entity != null) {
// Read the content stream
InputStream instream = entity.getContent();
Header contentEncoding = response.getFirstHeader("Content-Encoding");
if (contentEncoding != null && contentEncoding.getValue().equalsIgnoreCase("gzip")) {
instream = new GZIPInputStream(instream);
}
// Convert content stream to a String
resultString = convertStreamToString(instream);
instream.close();
// Do stuff with resultString here
// Consume Content
entity.consumeContent();
}
And the convertStreamToString() method:
private static String convertStreamToString(InputStream is) {
/*
* To convert the InputStream to String we use the
* BufferedReader.readLine() method. We iterate until the BufferedReader
* return null which means there's no more data to read. Each line will
* appended to a StringBuilder and returned as String.
*
* (c) public domain:
* http://senior.ceng.metu.edu.tr/2009/praeda/2009/01/
* 11/a-simple-restful-client-at-android/
*/
BufferedReader reader = new BufferedReader(new InputStreamReader(is), 8192);
StringBuilder sb = new StringBuilder();
String line = null;
try {
while ((line = reader.readLine()) != null) {
sb.append(line + "\n");
}
} catch (IOException e) {
e.printStackTrace();
} finally {
try {
is.close();
} catch (IOException e) {
e.printStackTrace();
}
}
return sb.toString();
}
I finally solved this issue by replacing httpclient library with Android Asynchronous Http Client. Now it works fine. Thanks a lot for your help!
However, I still dont understand why the response was truncated when i used httpclient.
I am having the following problem and I am stuck in it from a while and don't know what is the problem
I am getting the error
org.json.JSONException: Unterminated object at character 1526, as following the print of the response string
and as following my method
public void getItemType(final Context context,final ItemTypeRequestUIListener uiListener) {
if (PhoneUtils.isNetworkAvailable(context) == true) {
new AsyncTask<Void, Void, Void>() {
protected Void doInBackground(Void... params) {
try {
DefaultHttpClient client = new DefaultHttpClient();
HttpPost request = new HttpPost(AppConstants.URL_DO_LIST_LOOKUP);
request.setHeader("Accept", "application/json");
request.setHeader("Content-type", "application/json");
request.setEntity(new StringEntity(AppConstants.URL_DO_LIST_LOOKUP_ITEM_JSON_PARAM_FORMAT));
HttpResponse response = client.execute(request);
HttpEntity entity = response.getEntity();
if (entity.getContentLength() != 0) {
Reader objectReader = new InputStreamReader(response.getEntity().getContent());
char[] buffer = new char[(int) response.getEntity().getContentLength()];
objectReader.read(buffer);
objectReader.close();
String str = new String(buffer);
str = str.replace("?", "");
Log.i("AMIRA", "TYPES RESPONSE : " + str);
JSONObject jsonResponse = new JSONObject(str);
uiListener.onRequestCompleted(ItemTypeResponse.parseJSONObject(jsonResponse),null);
}
} catch (Exception e) {
e.printStackTrace();
uiListener.onRequestCompleted(null,AppError.GENERAL_ERROR);
}
return null;
};
}.execute();
} else {
uiListener.onRequestCompleted(null, AppError.NO_INTERNET);
}
}
Can anyone please tell me what is the problem here ?
The most likely problem is that you are using the wrong character encoding to read the stream.
A line of code like this:
Reader objectReader = new InputStreamReader(response.getEntity().getContent());
is always dangerous because you are not explicitly specifying what character set to use.
You can try this:
Reader objectReader = new InputStreamReader(response.getEntity().getContent(), "UTF-8");
if you know your server is sending UTF-8 content.
The best way to handle this is to read the Content-Type header in the response which could be something like:
Content-Type: application/json; charset=UTF-8
Parse the string after "charset=" and pass that as the character set name.
In effect you are specifying the character set that the server said it is using.
But be careful: The charset parameter is optional and may not be present.
As an extreme fallback, you could catch the JSONException and retry the parse with a different encoding. You would have to buffer the response somewhere so that you could re-read it.
I'm sending images and json text from the android client to a tomcat server and the other way around by using Multipart HttpPost's. Sending a Multipart Entity to the server is no big deal, because you can process the parts easily using request.getPart(<name>). But at the client side you can only access the response as a Stream. So I end up appending both, the JSON string and the image to the same ServletOutputStream and have to parse them by hand on the client side. I found apache-mime4j in the web but its hardly documented and I cant find a single example how to use it.
On the server side I build the response like this:
ServletResponse httpResponse = ctx.getResponse();
ResponseFacade rf = (ResponseFacade) httpResponse;
rf.addHeader("Access-Control-Allow-Origin", "*");
rf.addHeader("Access-Control-Allow-Methods", "POST");
rf.addHeader("content-type", "multipart/form-data");
httpResponse.setCharacterEncoding("UTF-8");
MultipartResponse multi = new MultipartResponse((HttpServletResponse) httpResponse);
ServletOutputStream out = httpResponse.getOutputStream();
multi.startResponse("text/plain");
out.println(CMD + "#" + content);
multi.endResponse();
multi.startResponse("image/jpeg");
out.write(data);
multi.endResponse();
multi.finish();
ctx.complete();
And on the client side on Android I want to access the text and the image data:
InputStream is = response.getEntity().getContent();
MimeStreamParser parser = new MimeStreamParser();
MultipartContentHandler con = new MultipartContentHandler();
parser.setContentHandler(con);
try {
parser.parse(is);
String json = con.getJSON(); //get extracted json string
byte[] imgBytes = con.getBytes(); //get extracted bytes
} catch (MimeException e) {
e.printStackTrace();
} finally {
is.close();
}
class MultipartContentHandler implements ContentHandler{
public void body(BodyDescriptor bd, InputStream in) throws MimeException, IOException {
//if MIME-Type is "text/plain"
// process json-part
//else
// process image-part
}
In the method body(BodyDescriptor bd, InputStream in) my whole response is treated as text\plain mime type. So I finally have to parse every byte manually again and the whole apache-mime4j is useless. Can you tell me what I am doing wrong? Thanks!
Ok i finally solved it myself. No here's what i did:
First I need to create a multipart/mixed Response at the server side. It can be done using apache-mime-4j API:
ServletResponse httpResponse = ctx.getResponse();
ResponseFacade rf = (ResponseFacade) httpResponse;
httpResponse.setCharacterEncoding("UTF-8");
httpResponse.setContentType("multipart/mixed");
MultipartEntity entity = new MultipartEntity(HttpMultipartMode.BROWSER_COMPATIBLE, "SEPERATOR_STRING",Charset.forName("UTF-8"));
entity.addPart("json", new StringBody(CMD + "#" + content, "text/plain", Charset.forName("UTF-8")));
entity.addPart("image", new ByteArrayBody(data, "image/jpeg", "file"));
httpResponse.setContentLength((int) entity.getContentLength());
entity.writeTo(httpResponse.getOutputStream());
ctx.complete();
Now at the client side to access the MIME-Parts of the HttpResponse I use the javax.mail API.
ByteArrayDataSource ds = new ByteArrayDataSource(response.getEntity().getContent(), "multipart/mixed");
MimeMultipart multipart = new MimeMultipart(ds);
BodyPart jsonPart = multipart.getBodyPart(0);
BodyPart imagePart = multipart.getBodyPart(1);
But you can't use the native API, instead take this one http://code.google.com/p/javamail-android/
Now you can proceed handling your individual parts.
It is also possible with apache-mime-4j:
HttpURLConnection conn = ...;
final InputStream is = conn.getInputStream();
try {
final StringBuilder sb = new StringBuilder();
sb.append("MIME-Version: ").append(conn.getHeaderField("MIME-Version")).append("\r\n");
sb.append("Content-Type: ").append(conn.getHeaderField("Content-Type")).append("\r\n");
sb.append("\r\n");
parser.parse(new SequenceInputStream(new ByteArrayInputStream(sb.toString().getBytes("US-ASCII")), is));
} catch (final MimeException e) {
e.printStackTrace();
} finally {
is.close();
}
I'm getting back names of (Foursquare) venues from a server call where the names of the venues returned can be in English or non-English.
Assume the venue name is in a JSON object as follows:
{...
"name":"venue name which can be in any language"
...}
I'm creating a JSONObject from this response and then pulling out the name of the venue as follows:
String name = jsonObject.getString("name");
Lastly, I'm setting the TextView's text to show the name of the venue as follows:
myTextView.setText(name);
I'm finding however for Arabic names that where the Arabic characters are joined in the original JSON object (as they should be), the characters that show in the app (i.e. in the TextView) are disjoint. (I'm not too familiar with other languages so can't really tell if they're showing incorrectly too.)
Is there something additional I should be doing to pull out non-English names correctly from the JSON object and setting it as the text of a TextView, or is it down to the phone to decide how the text will be displayed?
Edit: I've tried parsing the server response (as suggested by #bbedward) explicitly specifying the content encoding as UTF-8 as follows...
HttpEntity httpEntity = httpResponse.getEntity();
String responseMessage = EntityUtils.toString(myHttpEntity, "UTF-8");
JSONObject jsonObject = new JSONObject(responseMessage);
... but still no joy. (Arabic characters appear, as before, disjoint in words where they should be joint up.) Could it be a phone thing or is there something extra needing to be done myself to get the words/characters to show proper in non-English languages? Perhaps the server needs to explicitly specify a "Content-Type" header with value "UTF-8"?
I'm going to answer anyway, I'm guessing you aren't getting your json in UTF-8 as i had a similar problem, I believe json won't come any other way.
Complete Example
The only things to concern yourself with this is setting the encoding for the InputStreamReader and creating the JSONObject
private DefaultHttpClient httpclient = new DefaultHttpClient(new BasicHttpParams());
HttpPost httppost = new HttpPost("http://myjsonurl.com/search?type=json");
// Depending on your web service
httppost.setHeader("Content-type", "application/json");
try
{
String result = null;
HttpResponse response = httpclient.execute(httppost);
HttpEntity entity = response.getEntity();
InputStream inputStream = entity.getContent();
BufferedReader reader = new BufferedReader(new InputStreamReader(inputStream, "utf-8"), 8);
StringBuilder sb = new StringBuilder();
String line = null;
while ((line = reader.readLine()) != null)
{
sb.append(line + "\n");
}
JSONObject myJObject = new JSONObject(sb.toString();
}
catch (Exception e)
{
}
finally
{
try{if(inputStream != null)inputStream.close();}catch(Exception none){}
}
add this line when you connect to mysql:
mysql_set_charset('utf8', $con);
ex:
$con = mysql_connect(DB_HOST, DB_USER, DB_PASSWORD);
mysql_set_charset('utf8', $con);
mysql_select_db(DB_DATABASE);
I am writing a small app that retrieves some html from a web server based on some variables in the http POST. The HTML data that comes back has a <pre> section in it with some words that are spaced out nicely using newline and tab characters but my app does not receive them. The code is as follows
HttpPost post = new HttpPost("REMOVED FOR PRIVACY");
try {
// Add your data
List<NameValuePair> nameValuePairs = new ArrayList<NameValuePair>();
//REMOVED FOR PRIVACY
post.setEntity(new UrlEncodedFormEntity(nameValuePairs));
// Execute HTTP Post Request
HttpResponse response = httpClient.execute(post, httpContext);
String htmlBrief = inputStreamToString(response.getEntity().getContent()).toString();
return htmlBrief;
}
I think it might be how I am reading the response by putting it through a BufferedReader like so
private StringBuilder inputStreamToString(InputStream is) throws IOException {
int c;
StringBuilder total = new StringBuilder();
// Wrap a BufferedReader around the InputStream
BufferedReader rd = new BufferedReader(new InputStreamReader(is));
// Read response until the endIndex
while ((c = rd.read()) != -1) {
total.append((char)c);
}
// Return full string
return total;
}
I thought it might be because I was reading in line by line in the buffered reader so I switched to one character at a time but it didn't help the problem.
Any help would be greatly appreciated.
Thanks, Ben
Use Charles or Fiddler to inspect what is actually sent in the HTTP request body.
Most likely problems:
Mismatching character sets for client & server;
Failure to decode the URL encoded body.