I have an application where i need to download a large amount of data via a SOAP call to a webservice into the application when it is first run. The response is then sent to a function which converts the XML and stores the data in a db file.
The data is more than 16MB in size and i have a java.lang.OutOfMemoryError everytime.
Modifying the webservice to give out smaller amounts of data is not an option.
Is there a way to be able to download the large data? Something like an InputStream perhaps?
This is my code
public Protocol[] getProtocols() {
String METHOD_NAME = "GetProtocols";
String SOAP_ACTION = "urn:protocolpedia#GetProtocols";
Log.d("service", "getProtocols");
SoapObject response = invokeMethod(METHOD_NAME, SOAP_ACTION);
return retrieveProtocolsFromSoap(response);
}
private SoapObject invokeMethod(String methodName, String soapAction) {
Log.d(TAG, "invokeMethod");
SoapObject request = GetSoapObject(methodName);
SoapSerializationEnvelope envelope = getEnvelope(request);
return makeCall(envelope, methodName, soapAction);
}
Can anyone suggest what should be done in this case?
Thanks and regards
Mukul
Just an update, I found that the "call" method in AndroidHttpTransport was running out of memory at this line -
if (debug) {
ByteArrayOutputStream bos = new ByteArrayOutputStream();
byte[] buf = new byte[256];
while (true) {
int rd = is.read(buf, 0, 256);
if (rd == -1)
break;
bos.write(buf, 0, rd);
}
bos.flush();
buf = bos.toByteArray(); //Goes out of memory here
responseDump = new String(buf);
is.close();
is = new ByteArrayInputStream(buf);
the call to toByteArray takes a lot of memory, so to overcome this, instead of converting the response to a byte array, i now directly write it to an XML file, and this is saved at a location of my choice. Here -
if (debug) {
FileOutputStream bos = new FileOutputStream("/data/data/com.mypackage.myapp/response.xml");
byte[] buf = new byte[1048576];
int current = 0; int i=0; int newCurrent = 0;
while ((current = inputStream.read(buf)) != -1) {
newCurrent = newCurrent + current;
Log.d("current", "Current = " + current + " total = "+newCurrent+" i = "+i++);
bos.write(buf, 0, current);
}
bos.flush();
}
The device no longer runs out of memory, and i have a custom parse method that takes this XML and writes it to the DB.
Two strategies to help you solve this problem:
Save your SOAP XML stream directly to disk as you download it. Don't store it in memory.
Parse it using a SAX-style parser, where you don't load the whole DOM in memory, but rather parse it in chunks.
Depending on the kind of XML you are handling, using SAX parsers is usually harder in code; you will have to keep track of many things yourself, and you won't be able to "jump" from section to section of your DOM tree. But the memory consumption will be way lower.
Take note, however, that many "high-level" network communication libraries usually load the whole XML DOM in memory, which might be the case here. You will probably have to create and manage the HTTP connection yourself, and then manually parse the result.
Fixed!
I downloaded/copied HttpTransportSE java class from here (after copied, some code errors can occur, but they are all quick fixable) and added to my package:
https://github.com/mosabua/ksoap2-android/blob/master/ksoap2-j2se/src/main/java/org/ksoap2/transport/HttpTransportSE.java
removed from my Connection class this row:
import org.ksoap2.transport.HttpsTransportSE;
and substituted this code in my new HttpTransportSE.java file:
if (debug) {
ByteArrayOutputStream bos = new ByteArrayOutputStream();
byte[] buf = new byte[256];
while (true) {
int rd = is.read(buf, 0, 256);
if (rd == -1)
break;
bos.write(buf, 0, rd);
}
bos.flush();
buf = bos.toByteArray(); //Goes out of memory here
responseDump = new String(buf);
is.close();
is = new ByteArrayInputStream(buf);
}
with this
if (debug) {
FileOutputStream bos = new FileOutputStream(file);
byte[] buf = new byte[256];
while (true) {
int rd = is.read(buf, 0, 256);
if (rd == -1) {
break;
}
bos.write(buf, 0, rd);
}
bos.flush();
}
where "file" is a simple file object like new File("/sdcard/","myFile.xml") for example
Related
This question already has answers here:
Java multiple file transfer over socket
(3 answers)
Closed 5 years ago.
I have been spending forever on this and can not seem to work it out. I am self taught and not very familiar with this so forgive me if it is a remedial question. I am sending data from Android to .Net server. Data is getting corrupt on encoding, I know this I am just not sure how to fix. I am using the .Net Async server sample code found here: Microsoft Async Sample
My Android client code is:
try {
final Socket sock = new Socket();
final int timeOut = (int) TimeUnit.SECONDS.toMillis(5); // 5 sec wait period
sock.connect(new InetSocketAddress("localhost", 11000), timeOut);
if (sock.isConnected()==true){
BufferedReader in = new BufferedReader(new InputStreamReader(sock.getInputStream()));
BufferedWriter out = new BufferedWriter(new OutputStreamWriter(sock.getOutputStream()));
String FileName = "myfile.jpg";
StringBuilder hd = new StringBuilder();
try {
String FilePath= Environment.getExternalStorageDirectory() + "/mydir/" + FileName;
File file = new File(FilePath);
FileInputStream is = new FileInputStream(file);
byte[] chunk = new byte[40960];
int chunkLen = 0;
while ((chunkLen = is.read(chunk)) != -1) {
//String str = new String(Base64.encodeToString(chunk, Base64.NO_WRAP));
//String str = new String(chunk, "ASCII");
String str = new String(chunk, "UTF-8");
out.write(str);
}
//out.write(hd.toString());
} catch (FileNotFoundException fnfE) {
// file not found, handle case
} catch (IOException ioE) {
// problem reading, handle case
}
out.write("<EOF>");
out.flush();
StringBuilder returnString = new StringBuilder();
String line;
while ((line = in.readLine()) != null) {
returnString.append(line).append('\n');
}
out.close();
in.close();
sock.close();
}else{
sock.close();
}
} catch (IOException e) {
e.printStackTrace();
}
As you can see in my comments, I have tried base64 and UTF-8. I get all kinds of errors on the server when I do that. If I use Base64 I get not part of Base64 error (extra padding etc.). UTF8 writes the file but it is corrupt. When I send it all as one Base64 string it works fine as I use 'Dim data As Byte() = Convert.FromBase64String(FileData)' but as expected it throws memory errors in Android for large files hence the chunking. I am sending some plain ASCII text along with it so I parse out the non-ASCII stuff to write the file. I am super stuck, any help would be much appreciated. Thanks in advance.
You don't have to encode it at all. Just write it directly as bytes using an OutputStream. Simpler, quicker, better.
I found the answer. It was so weird but makes sense now.
byte[] chunk = new byte[30000];
int chunkLen = 0;
while ((chunkLen = is.read(chunk)) != -1) {
String str = new String(Base64.encodeToString(chunk, Base64.NO_WRAP));
out.write(str);
}
I had to change the chunk size to a multiple of 3 then my base64 encoding worked great. Found it here and gave an up vote. Thank you 'mjv'. Link to the answer
In my android application , user can upload a 300kb image;
I'm going to use This ( Android Asynchronous Http Client ) which I think is great and also Whatsapp is one of it's users.
In this library , I can use a RequestParams ( which is provided by apache I think) , and add either a file to it or an string ( lots of others too).
here it is :
1- Adding a file which is my image ( I think as a multipart/form-data)
RequestParams params = new RequestParams();
String contentType = RequestParams.APPLICATION_OCTET_STREAM;
params.put("my_image", new File(image_file_path), contentType); // here I added my Imagefile direcyly without base64ing it.
.
.
.
client.post(url, params, responseHandler);
2- Sending as string ( So it would be base64encoded)
File fileName = new File(image_file_path);
InputStream inputStream = new FileInputStream(fileName);
byte[] bytes;
byte[] buffer = new byte[8192];
int bytesRead;
ByteArrayOutputStream output = new ByteArrayOutputStream();
try {
while ((bytesRead = inputStream.read(buffer)) != -1) {
output.write(buffer, 0, bytesRead);
}
} catch (IOException e) {
e.printStackTrace();
}
bytes = output.toByteArray();
String encoded_image = Base64.encodeToString(bytes, Base64.DEFAULT);
// then add it to params :
params.add("my_image",encoded_image);
// And the rest is the same as above
So my Question is :
Which one is better in sake of Speed and Higher Quality ?
What are the differences ?
NOTE :
I've read many answers to similar questions , but none of them actually answers this question , For example This One
Don't know if params.put() and params.add would cause for a change of multipart encoding.
The base64 endoded data would transfer 30% slower as there are 30% more bytes to transfer.
What you mean by quality i do not know. The quality of the uploaded images would be equal as they would be byte by byte the same to the original.
I am sending images from a server script using image/jpeg type response. Here is the server code (php using CodeIgniter):
$file = file_get_contents("http://www.menucool.com/slider/prod/image-slider-5.jpg");
$this->output->set_header("Content-Type: image/jpeg; charset=UTF-8");
$this->output->set_header("HTTP/1.1 200 OK");
$this->output->set_header("Accept-Charset", "utf-8");
$this->output->set_header("Accept-Encoding", "");
$this->output->set_output($file);
And this script sends the image to android, where I try to decode using the following script:
Log.e("response",response);
try {
String decompressedResponse = decompress(response.getBytes("UTF-8"));
InputStream instream = new ByteArrayInputStream(decompressedResponse.getBytes("UTF-8"));
Log.e("inputstream",instream.toString());
ByteArrayOutputStream baos = new ByteArrayOutputStream();
int bufferSize = 1024;
byte[] buffer = new byte[bufferSize];
int len = 0;
try {
while ((len = instream.read(buffer)) != -1) {
baos.write(buffer, 0, len);
}
baos.close();
} catch (IOException e) {
}
byte[] b = baos.toByteArray();
Log.e("bytearray",b.toString());
imageBitmap = BitmapFactory.decodeByteArray(b, 0, b.length);
image.setImageBitmap(imageBitmap);
} catch (UnsupportedEncodingException e) {
Log.e("UnsupportedEcondingEception","UNSUPPORTEDENCODINGEXCEPTIOn");
} catch (IOException e1) {
return;
}
Logcatting the response shows characters like this:
���8�#
The really weird thing here is that if I call this script from a web browser, it displays the image but apparently android and eclipse cannot decode this encoding.
I think it is an android problem rather than the server's. Any suggestions, hints are really appreciated as I have been stuck for a while.
As per this link below:
http://java.sun.com/developer/technicalArticles/Programming/PerfTuning/
You can speed up loading of bitmaps (or any files) if you do the buffering yourself (i.e., instead of using BufferedInputStream, you handle the buffering yourself).
In particular, Approach 4 looks promising (slurp whole file at a time). However, I have no idea how to implement that in android. Here's the Java code:
import java.io.*;
public class readfile {
public static void main(String args[]) {
if (args.length != 1) {
System.err.println("missing filename");
System.exit(1);
}
try {
int len = (int)(new File(args[0]).length());
FileInputStream fis =
new FileInputStream(args[0]);
byte buf[] = new byte[len];
fis.read(buf);
fis.close();
int cnt = 0;
for (int i = 0; i < len; i++) {
if (buf[i] == '\n')
cnt++;
}
System.out.println(cnt);
}
catch (IOException e) {
System.err.println(e);
}
}
}
This technique is not optimized for Android and will likely run poorly. The convention is to use AndroidHttpClient:
Subclass of the Apache DefaultHttpClient that is configured with reasonable default settings and registered schemes for Android, and also lets the user add HttpRequestInterceptor classes.
If you really want to use Sun's code above, you should be careful because you will likely exceed the VM heap budget when the size of the file exceeds the amount of heap space available to the application.
It would be wise to first check if there is sufficient heap space left using ActivityManager. See also the elaborate answer to this question.
Edit:
I've found an example of sending an InputStream via POST. Here a file is being read from a resource (res/data.xml), but you could replace the InputStream with the FileInputStream from your snippet. Converting the InputStream to a byte array does essentially the same as your code: read the entire file into memory and push it into the request. This is a notorious cause of OutOfMemoryErrors, so take care that you don't read files that are too large (I would suggest less than 1 MB).
public void executeMultipartPost() throws Exception {
try {
InputStream is = this.getAssets().open("data.xml");
HttpClient httpClient = new DefaultHttpClient();
HttpPost postRequest = new HttpPost("http://w3mentor.com/Upload.aspx");
byte[] data = IOUtils.toByteArray(is);
InputStreamBody isb = new InputStreamBody(new ByteArrayInputStream(data),"uploadedFile");
StringBody sb1 = new StringBody("someTextGoesHere");
StringBody sb2 = new StringBody("someTextGoesHere too");
MultipartEntity multipartContent = new MultipartEntity();
multipartContent.addPart("uploadedFile", isb);
multipartContent.addPart("one", sb1);
multipartContent.addPart("two", sb2);
postRequest.setEntity(multipartContent);
HttpResponse res = httpClient.execute(postRequest);
res.getEntity().getContent().close();
} catch (Throwable e) {
// handle exception here
}
}
I want to consume a BasicHttp WCF web service with ksoap2 that is compressed by GZIP.
Is there a way to do this in the Android version of ksoap2 (http://code.google.com/p/ksoap2-android/) or is there another way?
I have create a class that extend the HTTPTransportSe and overrode the call() method and added this line to the code (taken from here https://github.com/mosabua/ksoap2-android/blob/master/ksoap2-j2se/src/main/java/org/ksoap2/transport/HttpTransportSE.java)
connection.setRequestProperty("Accept-Encoding","[Here you have to put the encoding]");
Then when I get the InputStream I use the retHeaders variable to check if there's the encoding.
if (retHeaders != null){
Iterator it = retHeaders.iterator();
boolean found = false;
String encoding = "";
while (it.hasNext()){
HeaderProperty temp = (HeaderProperty) it.next();
if (temp.getKey().contentEquals("content-encoding")){
found = true;
encoding = temp.getValue();
}
}
if (found){
is = new GZIPInputStream(is, new Inflater(true));
ByteArrayOutputStream bos = new ByteArrayOutputStream();
byte[] buf = new byte[256];
while (true) {
int rd = is.read(buf, 0, 256);
if (rd == -1)
break;
bos.write(buf, 0, rd);
}
bos.flush();
buf = bos.toByteArray();
is.close();
is = new ByteArrayInputStream(buf);
}
}
parseResponse(envelope, is);
And then you have to pass the "is" to the parser.
If there's a better way to code it I will happy to know about it. .))