Im sending an XML with HttpPost to a server. This used to wotk fine, and im doing it succesfully in other parts of the project.
Im using a StringBuilder to create the xml request but since i am appending strings as Data to the nodes, i am getting an error response from the parser on the server:
Invalid byte 2 of 2-byte UTF-8 sequence.
When i log the request and check it in w3c xml validator there are no errors.
This is an excerpt (whole method would be to big and has sensitive Data) from my Stringbuilder Method:
StringBuilder baseDocument = new StringBuilder();
baseDocument.append("<?xml version=\"1.0\" encoding=\"UTF-8\"?><request><setDisposalRequest><customer><company><![CDATA[");
baseDocument.append(company);
baseDocument.append("]]></company>");
baseDocument.append("<firstName><![CDATA[");
baseDocument.append(name);
baseDocument.append("]]></firstName>");
...
As soon as i replace the String vars i append with hardcoded Strings, all works fine
i.e
baseDocument.append(name);
to
baseDocument.append("name");
All the strings have values, non of them a null or are empty!
Before the request i set the StringEntity to xml
se.setContentType("application/xml");
what am i missing?!?
Your XML header claims that it's UTF-8, yet you never mention if you actually write UTF-8. Make sure the actual bytes you send are UTF-8 encoded. The error message suggests that you're using another encoding (probably a ISO-8859-* variant).
This is another reason that manually constructing XML like this is dangerous: there are just too many corner cases to observe and it's much easier to use a real XML handling library. Those tend to get the corner cases correct ;-)
And no: StringBuilder certainly does not break UTF-8. The problem is somewhere else.
Related
I created XML string using XmlSerializer and it prints out as
<tag> string </tag>
I want to add the xml string to SOAPObject and get response from web service.
So I added and when I print out the requestDump the < and > are being replaced by < ; and > ;
I assume that it is being encoded by HttpTransportSE.
I read here Android Ksoap2 web service for download/Upload and it says I need to convert my string to binary before uploading? I am confused because the API tells us to upload xml. Currently, there is no error or exception but the result is empty. I think it's the encoding problem.
Any help will be appreciated. Thanks!
When using ksoap2 you don't have to handle the serialization yourself. If you want to add an entry to the xml like this:
<tag>string</tag>
Then you just have to call this when constructing the SoapObject you are sending to the server:
soapObject.addProperty("tag", "string");
Also modifying the requestDump is never a good idea. This output is just for debug purposes. That's the reason why you have to set transport.debug = true; for the requestDump or responseDump to even appear. As I said above just use the addProperty() and addSoapObject() methods to construct your request.
Use SoapObject.setInnerText("..."); to add CDATA text to a Ksoap2 request.
I have to read an html as a text file and I have implemented two different methods as described on SO threads. I clean the html leaving just the body content to put it into DOM with document.write(string) with both methods, but it only works if I use Bufferedreader.readLine() (appending a stringBuilder in a loop). In fact, if I print the stringBuilder.toString() in the Logcat window I see that when BufferedReader.readLine() is used the stringBuilder is displayed on a single line in the Logcat window, while with InputStreamReader.read(charArray,0,(int)numBytes) it is displayed multi-line (I am using a real device). Maybe this can be related and suggest which is the issue. My question is : how have I to clean or process that char Array read by read(charArray,0,(int)numBytes) to be the same of the stringBuilder appended in the readLine() loop?
The BufferedReader.readLine() method reads till the end of the line and returns the value excluding the newline character. Hence when you append it to a StringBuffer everything gets appended to a single line. However, in the case of InputStreamReader.read, the newline character is also included, causing the string to be displayed in multiple lines.
To make the behavior similar, you will have to skip newline character while using InputStreamReader.read.
I have a crash in an xml file. it occurs on a ë, in this case belgië (dutch for belgium).
I'm busy with searching for an answer but I just can't find a solution.
I'm using the sax parser under Android.
error: org.apache.harmony.xml.ExpatParser$ParseException: At line 2, column 204: not well-formed
xml source: http://biohorma.weatheronyoursite.com/villadm_hooikoortsverwachting_be.xml
Side note, i get the data via a stream, is the only option to put this stream to a temp value, replace the illegal character with a valid one and make a new stream of it or can you add something in the stream to do this?
It seems you should use the String (byte[] bytes, String enc) constructor, assuming what server sends you is encoded in UTF-8:
String properXml = new String(byteArrayIReceivedFromServer, "UTF-8");
The issue is not with the parser - it's acting correctly - but with whatever code is sending the XML. ë needs to be encoded and passed as ë. The same also must be done to other accented characters, ampersands and angle brackets.
You should replace special characters in the xml I think..
See a comprehensive list of chars here: http://www.w3schools.com/tags/ref_entities.asp
it says your umlaut e is like : Ë Ë Ë capital e, umlaut mark
Then also for a brief explanation if u feel like reading.
Hope it helps.
The server sends these headers:
Content-Type: text/xml
Content-Length: 124512
Since no charset is specified for content type, the normally correct assumption is US_ASCII. However, the XML payload seems to be encoded in ISO-8859-1
<?xml version="1.0" encoding="iso-8859-1"?>
and the 'ë' is encoded as 0xEB (235). It is very common for servers to encode text payload in ISO-8859-1, so this is something that one simply has to deal with.
My guess is that if you serve the parser with a byte stream directly, it will detect the encoding an act accordingly. If you use a character stream (not recommended), make sure to specify correct encoding.
I'm downloading HTML source code of remote page into String variable. Unfortunetely the page is encoded via iso-8859-2 and contains characters from polish alphabet. How can I convert this string to utf-8, so I can display it's parts in TextView?
Thanks
You shouldn't need to "convert" the string at all, if you obey the Content-Encoding header sent by the web server.
Right now, you probably ignore that header while reading the response from the server (some BufferedReader-to-StringBuffer/Builder loop I assume), try this in your download code instead:
HttpResponse response = ....
String text = EntityUtils.toString(response.getEntity());
EntityUtils will automagically use the content encoding specified by the server.
I am trying to parse a Rss2.0 feed on Android using a Pull parser.
XmlPullParser parser = Xml.newPullParser();
parser.setInput(url.open(), null);
The prolog of the feed XML says the encoding is "utf-8". When I open the remote stream and pass this to my Pull Parser, I get invalid token, document not well formed exceptions.
When I save the XML file and open it in the browser(FireFox) the browser reports presence of Unicode 0x12 character(grave accent?) in the file and fails to render the XML.
What is the best way to handle such cases assuming that I do not have any control over the XML being returned?
Thanks.
Where did you find that 0x12 is the grave accent? UTF-8 has the character range 0x00-0x7F encoded the same as ASCII, and ASCII code point 0x12 is a control character, DC2, or CTRL+R.
It sounds like an encoding problem of some sort. The simplest way to resolve that is to look at the file you've saved in a hex editor. There are some things to check:
the byte order mark (BOM) at the beginning might confuse some XML parsers
even though the XML declaration says the encoding is in UTF-8, it may not actually have that encoding, and the file will be decoded incorrectly.
not all unicode characters are legal in XML, which is why firefox refuses to render it. In particular, the XML spec says that that 0x9, 0xA and 0xD are the only valid characters less than 0x20, so 0x12 will definitely cause compliant parsers to grumble.
If you can upload the file to pastebin or similar, I can help find the cause and suggest a resolution.
EDIT: Ok, you can't upload. That's understandable.
The XML you're getting is corrupted somehow, and the ideal course of action is to contact the party responsible for producing it, to see if the problem can be resolved.
One thing to check before doing that though - are you sure you are getting the data undisturbed? Some forms of communication (SMS) allow only 7-bit characters. This would turn 0x92 (ASCII forward tick/apostrophe - grave accent?) into 0x12. Seems like quite a coincidence, particularly if these appear in the file where you would expect an accent.
Otherwise, you will have to try to make best do with what you have:
although not strictly necessary, be defensive and pass "UTF-8" as the second paramter to setInput, on the parser.
similarly, force the parser to use another character encoding by passing a different encoding as the second parameter. Encodings to try in addtion to "UTF-8" are "iso-8859-1" and "UTF-16". A full list of supported encodings for java is given on the Sun site - you could try all of these. (I couldn't find a definitive list of supported encodings for Android.)
As a last resort, you can strip out invalid characters, e.g. remove all characters below 0x20 that are not whitespace (0x9,0xA and 0xD are all whitepsace.) If removing them is difficult, you can replace them instead.
For example
class ReplacingInputStream extends FilterInputStream
{
public int read() throws IOException
{
int read = super.read();
if (read!=-1 && read<0x20 && !(read==0x9 || read==0xA || read==0xB))
read = 0x20;
return read;
}
}
You wrap this around your existing input stream, and it filters out the invalid characters. Note that you could easily do more damage to the XML, or end up with nonsense XML, but equally it may allow you to get out the data you need or to more easily see where the problems lie.
I use to filter it with a regex, but the trick is not trying to get and replace the accents. It depends on the encode and you don't want to change the content.
Try to insert the content of the tags into this tags
Like this
<title>My title</title>
<link>http://mylink.com</link>
<description>My description</description>
To this
<title><![CDATA[My title]]></title>
<link><![CDATA[http://milynk.com]]></link>
<description><![CDATA[My Description]]></description>
The regex shouldn't be very hard to figure out. It works for me, hope it helps for you.
The problem with UTF-8 is that it is a multibyte encoding. As such it needs a way to indicate when a character is formed by more than one byte (maybe two, three, four, ...). The way of doing this is by reserving some byte values to signal multibyte characters. Thus encoding follows some basic rules:
One byte characters have no MSB set (codes compatible with 7-bit ASCII).
Two byte characters are represented by sequence: 110xxxxx 10xxxxxx
Three bytes: 1110xxxx 10xxxxxx 10xxxxxx
Four bytes: 11110xxx 10xxxxxx 10xxxxxx 10xxxxxx
Your problem is that you may be reading some character string supposedly encoded as UTF-8 (as the XML encoding definition states) but the byte chunk might not be really encoded in UTF-8 (it is a common mistake to declare something as UTF-8 but encoding text with a different encoding such as Cp1252). Your XML parser tries to interpret byte chunks as UTF-8 characters but finds something that does not fit the encoding rules (illegal character). I.e. two bytes with two most significate bytes set would bring an illegal encoding error: 110xxxxx must be always followed by 10xxxxxx (values such as 01xxxxxx 11xxxxxx 00xxxxxx would be illegal).
This problem does not arise when non-variable length encodings are used. I.e. if you state in your XML declaration that your file uses Windows-1252 encoding but you end up using ANSI your only problem will be that non-ASCII characters (values > 127) will render incorrectly.
The solution:
Try to detect encoding by other means.
If you will always be reading data from same source you could sample some files and use an advanced text editor that tries to infer actual encoding of the file (i.e. notepad++, jEdit, etc.).
Do it programatically. Preprocess raw bytes before doing any actual xml processing.
Force actual encoding at the XML processor
Alternatively if you do not mind about non-ASCII characters (no matter if strange symbols appear now and then) you could go directly to step 2 and force XML processing to any ASCII compatible 8-byte fixed length encoding (ANSI, any Windows-XXXX codepage, Mac-Roman encoding, etc.). With your present code you just could try:
XmlPullParser parser = Xml.newPullParser();
parser.setInput(url.open(), "ISO-8859-1");
Calling setInput(istream, null) already means for the pull parser to try to detect the encoding on its own. It obviously fails, due to the fact that there is an actual problem with the file. So it's not like your code is wrong - you can't be expected to be able to parse all incorrect documents, whether ill-formed or with wrong encodings.
If however it's mandatory that you try to parse this particular document, what you can do is modify your parsing code so it's in a function that takes the encoding as a parameter and is wrapped in a try/catch block. The first time through, do not specify an encoding, and if you get an encoding error, relaunch it with ISO-8859-1. If it's mandatory to have it succeed, repeat for other encodings, otherwise call it quits after two.
Before parsing your XML, you may tweak it, and manually remove the accents before you parse it.
Maybe not the best solution so far, but it will do the job.