I am trying to get compressed data from server. The guy that programmed the server told me, that he uses ZLIB library on his iPhone, and the gzcompress on server. I was trying to find any suitable way to get that data, but it ends up with info "java.io.IOException: unknown format (magic number 9c78)" while creating GZIPInputStream object. Finally I've reached point, where I had data as a String. It was compressed, so I used that answer to decompress: https://stackoverflow.com/a/6963668/419308 . But that code doesn't work. "in.read()" returns -1 at the beginning.
Anyone has any idea why there's -1 ? Or maybe a better way to get the compressed data?
EDIT:
I tried adding file to project and reading from that file. in.read() didn't return -1
EDIT2: According to jJ's answer I've tried this code:
HttpGet request = new HttpGet( urlTeam );
HttpResponse response = new DefaultHttpClient().execute( request );
HttpEntity entity = response.getEntity();
InputStream stream = AndroidHttpClient.getUngzippedContent( entity );
InputStreamReader reader = new InputStreamReader( stream );
BufferedReader buffer = new BufferedReader( reader );
StringBuilder sb = new StringBuilder();
sb.delete( 0, sb.length() );
String input;
while ( ( input = buffer.readLine() ) != null )
{
sb.append( input );
}
But the answer is still compressed (or unreadable)
on android you should use either HttpURLConnection which handles decompression (and http encoding headers) for you from GingerBread on or AndroidHttpClient (for older android versions) that has helper methods like getUngzippedContent and modifyRequestToAcceptGzipResponse.
This is a good summary Android HTTP clients
Related
I receive a file using the following code:
byte[] fileBytes;
....
JSONObject postJSON = new JSONObject();
postJSON.put("file_name", filename);
postJSON.put("client_id", clientID);
HttpPost post = new HttpPost(fileURL);
StringEntity se = new StringEntity( postJSON.toString(), "UTF-8");
se.setContentType(new BasicHeader(HTTP.CONTENT_TYPE, "application/json"));
post.setEntity(se);
response = httpClient.execute(post);
fileBytes = EntityUtils.toByteArray(response.getEntity());
Using the debugger, I see that the response gets an entity 27136 bytes in length, which is the correct length of the test file, but the fileBytes array is only 11470 bytes long. Can anyone tell my why this truncation is taking place? When I try to get other files, a similar truncation takes place, so it is not a function of the specific file or a specific file length.
Using the following code, I get 11997 bytes for the same file:
StringBuilder stringBuilder = new StringBuilder("");
stringBuilder.append(EntityUtils.toString(response.getEntity()));
fileBytes = stringBuilder.toString().getBytes();
Reading from an InputStream, I get 12288 bytes:
fileBytes = new byte[1024];
InputStream inputStream = response.getEntity().getContent();
int bytesRead = 0;
while(true){
bytesRead = inputStream.read(fileBytes);
if (bytesRead <= 0)
break;
....
}
Changing the encoding to UTF-16 gets me an internal server error.
I also tried the following:
InputStream inputStream = response.getEntity().getContent();
response.getEntity().getContentLength()];
while ((getByte = inputStream.read()) != -1) {
bos.write(getByte);
}
bos.close();
This also gave me a file of 11470.
In all cases, the files are corrupted, and cannot be opened. When compared in a binary file viewer, the firs 11 bytes match, and then the files diverge. I could not find any pattern in the corrupted file.
OK, the answer is apparently that all of the above are fine. The problem was with the server, which was not configuring the data stream correctly: Content-type was text/plain for all files, rather than application/pdf, and so on as appropriate.
My first clue was when we put a text file on the server, and it came over successfully. At that point I started working with the server side, and we figured it out pretty quickly.
Bottom line, if you are working on a server/client application, the problem might not be on your side.
I should have mentioned various posts which helped my construct the various versions that I collected above:
including this
and this
My apologies to various other helpful people whose posts I also looked at and up-voted.
I have a big JSON input (download the file) API and I don´t know how to parse this data. I need:
Save this data (entire JSON input) to text file or database. What is the best way for this?
Load this data from text file or database and create JSONArray from JSON tag "list" (first tag)
The solution should be fast and support Android 2.3. What you have recomend for this? Any ideas?
My code:
HttpClient httpClient = new DefaultHttpClient();
HttpGet httpGet = new HttpGet(urls[0]);
HttpResponse httpResponse;
httpResponse = httpClient.execute(httpGet);
HttpEntity httpEntity = httpResponse.getEntity();
... and what next ?...
FYI:
EntityUtils throws OutOfMemoryException
EDIT:
I try to save data to file like this:
InputStream inputStream = httpEntity.getContent();
FileOutputStream output = new FileOutputStream(Globals.fileNews);
int bufferSize = 1024;
byte[] buffer = new byte[bufferSize];
int len = 0;
while ((len = inputStream.read(buffer)) != -1) {
output.write(buffer, 0, len);
}
And it´s OK. I load data:
FileInputStream fis = null;
StringBuffer fileContent = new StringBuffer("");
fis = new FileInputStream(Globals.fileNews);
byte[] buffer = new byte[1024];
while (fis.read(buffer) != -1) {
fileContent.append(new String(buffer));
}
But how convert StringBuffer to JSONObject? fileContent.ToString() is not ideal, sometimes I get OutOfMemoryException.
First of all: Dispose the HttpClient. Google discourages it:
Unfortunately, Apache HTTP Client does not, which is one of the many
reasons we discourage its use.
Source: developer.android.com
A good replacement is Google Volley. You have to build the JAR yourself but it just works like charm. I use for my setups Google Volley with OkHttp-Stack and GSON requests.
In your case you would write another Request which just writes the response out to the SD-card chunk by chunk. You don't buffer the string before! And some logic to open an input-stream from the file you wrote and give it to your JSON Deserializer. Jackson and GSON are able to handle streams out of the box.
Of course everything works with Android 2.3.
Don't, I repeat, don't try to dump the whole serialized stuff into a string or something. That's almost a OutOfMemoryException guarantee.
I have a REST service I can't alter, with methods for uploading an image, encoded as a Base64 string.
The problem is that the images can go up to sizes of 5-10MB, perhaps more. When I try to construct a Base64 representation of an image of this size on the device, I get an OutOfMemory exception.
I can however encode chunks of bytes at a time (3000 let's say), but this is useless as I would need the whole string to create a HttpGet/HttpPost object:
DefaultHttpClient client = new DefaultHttpClient();
HttpGet httpGet = new HttpGet("www.server.com/longString");
HttpResponse response = client.execute(httpGet);
Is there a way of going around this?
Edit: trying to use Heiko Rupp's suggestions + the android doc, I get an exception ("java.io.FileNotFoundException: http://www.google.com") at the following line: InputStream in = urlConnection.getInputStream();
try {
URL url = new URL("http://www.google.com");
HttpURLConnection urlConnection = (HttpURLConnection) url.openConnection();
urlConnection.setDoOutput(true);
urlConnection.setChunkedStreamingMode(0);
OutputStream out = new BufferedOutputStream(urlConnection.getOutputStream());
out.write("/translate".getBytes());
InputStream in = urlConnection.getInputStream();
BufferedReader r = new BufferedReader(new InputStreamReader(in));
StringBuilder total = new StringBuilder();
String line;
while ((line = r.readLine()) != null) {
total.append(line);
}
System.out.println("response:" + total);
} catch (MalformedURLException e) {
e.printStackTrace();
} catch (IOException e) {
e.printStackTrace();
}
Am I missing something? The GET request that I need to execute looks like this:
"http://myRESTService.com/myMethod?params=LOOONG-String", so the idea was to connect to http://myRESTService.com/myMethod and then output a few characters of the long string at a time. Is this correct?
You should try to use the URLConnection instead of the apache http client, as this does not require you to hold the object to send in memory, but instead you can do something like:
pseudocode!
HttpUrlConnection con = restUrl.getConnection();
while (!done) {
byte[] part = base64encode(partOfImage);
con.write (part);
partOfImage = nextPartOfImage();
}
con.flush();
con.close();
Also in Android after 2.2 Google recommends the URLConnection over the http client. See the description of DefaultHttpClient.
The other thing you may want to look into is the amount of data to be sent. 10 MB + base64 will take quite a while to transfer (even with gzip compression, which the URLConnection transparently enables if the server side accepts it) over a mobile network.
You must read docs for this REST service, no such service will require you to send such long data in GET. Images are always sent as POST. POST data is always at the end of request and allows to be added iteratively.
I have a RESTful WCF service that I am using to retrieve encoded photos and display them in android (trying to anyway). The problem I am having is that the InputStream or possibly something else stops reading the characters before the end.
The response is just an XML string, I intend to parse it myself so no need to worry about that. What I need to know is what in the following code is stopping the input stream from reading characters into my buffer.
HttpEntity responseEntity = response.getEntity();
char[] buffer = new char[(int)responseEntity.getContentLength()];
InputStream stream = responseEntity.getContent();
InputStreamReader reader = new InputStreamReader(stream);
reader.read(buffer);
stream.close();
Have you implemented the HttpURLConnection class.. this would cause this behavior.
http://developer.android.com/reference/java/net/HttpURLConnection.html
What is the maximum size of android Post method? When I am getting responses from server, parts of the message are missing. I think it may have reached the maximum size of post method.
If there is no limit for post method, did I need to change my server specification for this?
I had the same problem, I used HttpPost and the response got from server but a part of data missed because of their very big size. That's why I used an other way : HttpURLConnection with OuputStream to send request to the server and BufferedReader/InputStream to get responses.
HttpURLConnection my_httpConnection = (HttpURLConnection) new URL("https://integrator-ut.vegaconnection.com/Authentication.svc?wsdl").openConnection();
my_httpConnection.setRequestMethod("POST");
my_httpConnection.setDoInput(true);
my_httpConnection.setDoOutput(true);
my_httpConnection.setRequestProperty("Content-type", "text/xml; charset=utf-8");
OutputStream my_outPutStream = this.my_httpConnection.getOutputStream();
Writer my_writer = new OutputStreamWriter(my_outPutStream);
my_writer.write(YOUR_REQUEST); //YOUR_REQUEST is a String
my_writer.flush();
my_writer.close();
BufferedReader my_bufferReader = new BufferedReader(new InputStreamReader(this.my_httpConnection.getInputStream()));
char[] buffer = new char[10000];
int nbCharRead=0;
try
{
while((nbCharRead = my_bufferReader.read(buffer, 0, 10000)) != -1)
{
/* Your treatement : saving on a file/arraylist/etc
}
}
Theoretically there isn't a limit. The POST response size is limited to Java VM Heap size which is device independent. It is probably more than your post response consumes.
How do you verify that part of your response is missing? If you print it out with LogCat or view it in debug mode, then you can see only the start of the message ending with three dots (all of the message is there, it isn't just displayed to you).