Develop Reference

Android : java.lang.OutOfMemoryError: Failed to allocate with free bytes and 70MB until OOM when using gson.toJson()

In my Android application I am getting below exception when I try to sync my data to the server which is large in size. I get this exception when data size is more than 20 MB I think. I am saving bitmap images as string using base64 encoding after sacle down the image size which makes such huge data.
04-18 13:51:51.957 16199-16816/com.example.myproject.app E/art﹕ Throwing OutOfMemoryError "Failed to allocate a 128887990 byte allocation with 16777216 free bytes and 70MB until OOM"
04-18 13:51:52.037 16199-16816/com.example.myproject.app E/AndroidRuntime﹕ FATAL EXCEPTION: Thread-4482
Process: com.example.myproject.app, PID: 16199
java.lang.OutOfMemoryError: Failed to allocate a 128887990 byte allocation with 16777216 free bytes and 70MB until OOM
at java.lang.AbstractStringBuilder.enlargeBuffer(AbstractStringBuilder.java:95)
at java.lang.AbstractStringBuilder.append0(AbstractStringBuilder.java:146)
at java.lang.StringBuffer.append(StringBuffer.java:219)
at java.io.StringWriter.write(StringWriter.java:167)
at com.google.gson.stream.JsonWriter.string(JsonWriter.java:570)
at com.google.gson.stream.JsonWriter.value(JsonWriter.java:419)
at com.google.gson.internal.bind.TypeAdapters$16.write(TypeAdapters.java:426)
at com.google.gson.internal.bind.TypeAdapters$16.write(TypeAdapters.java:410)
at com.google.gson.internal.bind.TypeAdapterRuntimeTypeWrapper.write(TypeAdapterRuntimeTypeWrapper.java:68)
at com.google.gson.internal.bind.ReflectiveTypeAdapterFactory$1.write(ReflectiveTypeAdapterFactory.java:112)
at com.google.gson.internal.bind.ReflectiveTypeAdapterFactory$Adapter.write(ReflectiveTypeAdapterFactory.java:239)
at com.google.gson.internal.bind.TypeAdapterRuntimeTypeWrapper.write(TypeAdapterRuntimeTypeWrapper.java:68)
at com.google.gson.internal.bind.CollectionTypeAdapterFactory$Adapter.write(CollectionTypeAdapterFactory.java:97)
at com.google.gson.internal.bind.CollectionTypeAdapterFactory$Adapter.write(CollectionTypeAdapterFactory.java:61)
at com.google.gson.internal.bind.TypeAdapterRuntimeTypeWrapper.write(TypeAdapterRuntimeTypeWrapper.java:68)
How to solve this problem? I know that this comes when I convert the data from class to json using Gson. Below is my code:
SimpleDateFormat dtf=new SimpleDateFormat("yyyy-MM-dd HH:mm:ss",Locale.ENGLISH);
GsonBuilder builder = new GsonBuilder();
builder.registerTypeAdapter(Date.class, new JsonDeserializer<Date>() {
#Override
public Date deserialize(JsonElement json, Type type, JsonDeserializationContext deserializationContext) throws JsonParseException {
String frStr = json.getAsJsonPrimitive().getAsString();
Date retDate =null;
try {
retDate = dtf.parse(frStr);
} catch (ParseException e) {
e.printStackTrace();
}
return retDate;
}
});
builder.registerTypeAdapter(Date.class, new JsonSerializer<Date>() {
#Override
public JsonElement serialize(Date src, Type typeOfSrc, JsonSerializationContext context) {
String jsDate = dtf.format(src);
return new JsonPrimitive(jsDate);
}
});
builder.registerTypeAdapter(byte[].class, new JsonDeserializer<byte[]>() {
#Override
public byte[] deserialize(JsonElement json, Type type, JsonDeserializationContext deserializationContext) throws JsonParseException {
return Base64.decode(json.getAsString(), Base64.NO_WRAP);
}
});
gson = builder.create();
attDataAcc.setAttList(attList);
String jsonAttAccts = gson.toJson(attDataAcc, AttachmentDataList.class);
HttpEntity<String> entityAtt = new HttpEntity<String>(jsonAttAccts,headers);
ResponseEntity<String> restResA = restTemplate.exchange(strUrl+"/saveAttToServer", HttpMethod.POST, entityAtt, String.class);
public class Attachment implements Serializable {
#DatabaseField(columnName = "id",id = true)
private String id;
#DatabaseField(columnName = "user_id")
private Integer userId;
#DatabaseField(columnName = "attachment_id")
private String attachmentId;
#DatabaseField(columnName = "file_name")
private String fileName;
#DatabaseField(columnName = "file_data")
private String fileData;
#DatabaseField(columnName = "date",dataType=DataType.DATE)
private Date date;
public Attachment() {
super();
// TODO Auto-generated constructor stub
}
public Attachment(String id, Integer userId, String attachmentId, String fileName, String fileData, Date date) {
this.id = id;
this.userId = userId;
this.attachmentId = attachmentId;
this.fileName = fileName;
this.fileData = fileData;
this.date = date;
}
public String getId() {
return id;
}
public void setId(String id) {
this.id = id;
}
public Integer getUserId() {
return userId;
}
public void setUserId(Integer userId) {
this.userId = userId;
}
public String getAttachmentId() {
return attachmentId;
}
public void setAttachmentId(String attachmentId) {
this.attachmentId = attachmentId;
}
public String getFileName() {
return fileName;
}
public void setFileName(String fileName) {
this.fileName = fileName;
}
public String getFileData() {
return fileData;
}
public void setFileData(String fileData) {
this.fileData = fileData;
}
public Date getDate() {
return date;
}
public void setDate(Date date) {
this.date = date;
}
#Override
public boolean equals(Object o) {
if (this == o) return true;
if (o == null || getClass() != o.getClass()) return false;
Attachment that = (Attachment) o;
if (id != null ? !id.equals(that.id) : that.id != null) return false;
if (userId != null ? !userId.equals(that.userId) : that.userId != null) return false;
if (attachmentId != null ? !attachmentId.equals(that.attachmentId) : that.attachmentId != null) return false;
if (fileName != null ? !fileName.equals(that.fileName) : that.fileName != null) return false;
if (fileData != null ? !fileData.equals(that.fileData) : that.fileData != null) return false;
if (date != null ? !date.equals(that.date) : that.date != null) return false;
}
#Override
public int hashCode() {
int result = id != null ? id.hashCode() : 0;
result = 31 * result + (userId != null ? userId.hashCode() : 0);
result = 31 * result + (attachmentId != null ? attachmentId.hashCode() : 0);
result = 31 * result + (fileName != null ? fileName.hashCode() : 0);
result = 31 * result + (fileData != null ? fileData.hashCode() : 0);
result = 31 * result + (date != null ? date.hashCode() : 0);
return result;
}
#Override
public String toString() {
return userFileName;
}
}
public class AttachmentDataList implements Serializable {
private ArrayList<Attachment> attList;
public ArrayList<Attachment> getAttList() {
return attList;
}
public void setAttList(ArrayList<Attachment> attList) {
this.attList = attList;
}
}

You're running into OutOfMemoryError because you're using inefficient and very memory-consuming Base64 transformations. Another hit here is Gson: it does not provide any raw-write methods for JsonWriter and JsonReader classes: the most you can do here is writing/reading a single string value. Collecting a huge input to a single string is another very memory-consuming operation: check your stacktrace to make sure that a string builder instance is used under the very hood -- and this is just to write a single value to the output stream. In short, it all looks like this (if I'm not mistaken about your code, because it seems to lack really important parts, so I'm just trying to reconstruct your scenario):
Obtaining a byte array (that would be a new object, probably a clone of another byte array);
Converting the byte array to a Base64 encoded string (it also hits performance because the would clone the byte array to create a defensive copy);
Converting ALL to a String gson.toJson(attDataAcc, AttachmentDataList.class); -- another huge hit.
All of this is extremely memory consuming. And it would be nice if Gson could support raw write to the output stream, but currently it lacks any of it.
In theory, you might overcome this issue by just writing to the underlying streams (probably directly from your byte array source without any massive transformations because Base64 can be streamed as well thus consuming memory minimum). You mentioned Gson 2.6.2, but I'm working with Gson 2.8.0, so the below solution can 100% work with Gson 2.8.0 only, and may not work even for any other minor Gson versions because it uses reflection in order to "hack" the JsonWriter class.
final class ByteArrayTypeAdapter
extends TypeAdapter<byte[]> {
// These two methods and one field from the super class privates are necessary to make it all work
private static final Method writeDeferredNameMethod;
private static final Method beforeValueMethod;
private static final Field writerField;
static {
try {
writeDeferredNameMethod = JsonWriter.class.getDeclaredMethod("writeDeferredName");
writeDeferredNameMethod.setAccessible(true);
beforeValueMethod = JsonWriter.class.getDeclaredMethod("beforeValue");
beforeValueMethod.setAccessible(true);
writerField = JsonWriter.class.getDeclaredField("out");
writerField.setAccessible(true);
} catch ( final NoSuchMethodException | NoSuchFieldException ex ) {
throw new RuntimeException(ex);
}
}
// This type adapter is effectively a singleton having no any internal state
private static final TypeAdapter<byte[]> byteArrayTypeAdapter = new ByteArrayTypeAdapter();
private ByteArrayTypeAdapter() {
}
// But making the constructor private and providing access to the instance via the method, we make sure that the only instance exists and it's safe
static TypeAdapter<byte[]> getByteArrayTypeAdapter() {
return byteArrayTypeAdapter;
}
#Override
public void write(final JsonWriter out, final byte[] bytes)
throws IOException {
try {
// Since we're writing a byte[] array, that's probably a field value, make sure that the corresponding property name has been written to the output stream
writeDeferredNameAndFlush(out);
// Now simulate JsonWriter.value(byte[]) if such a method could exist
writeRawBase64ValueAndFlush(bytes, (Writer) writerField.get(out));
} catch ( IllegalAccessException | InvocationTargetException ex ) {
throw new IOException(ex);
}
}
#Override
public byte[] read(final JsonReader in) {
// If necessary, requires more hacks...
// And this is crucial for the server-side:
// In theory, the client can generate HUGE Base64 strings,
// So the server could crash with OutOfMemoryError too
throw new UnsupportedOperationException();
}
private static void writeDeferredNameAndFlush(final Flushable out)
throws IOException, IllegalAccessException, InvocationTargetException {
writeDeferredNameMethod.invoke(out);
beforeValueMethod.invoke(out);
// Flush is necessary: the JsonWriter does not know that we're using its private field intruding to its privates and may not flush
out.flush();
}
private static void writeRawBase64ValueAndFlush(final byte[] bytes, final Writer writer)
throws IOException {
// Writing leading "
writer.write('\"');
// This comes from Google Guava
final BaseEncoding baseEncoding = BaseEncoding.base64();
final OutputStream outputStream = baseEncoding.encodingStream(writer);
// This too
// Note that we just r_e_d_i_r_e_c_t streams on fly not making heavy transformations
ByteStreams.copy(new ByteArrayInputStream(bytes), outputStream);
// This is necessary too
outputStream.close();
// Writing trailing "
writer.write('\"');
// Flush again to keep it all in sync
writer.flush();
}
}
I know it's a hack, but it's better than just getting OutOfMemoryError constantly.
Now, just make it work with Spring RestTemplates:
// Gson is thread-safe and can be re-used
private static final Gson gson = new GsonBuilder()
// SimpleDateFormat may be NOT thread-safe so you should not share the single SimpleDateFormat between threads
// However Gson supports date/time formats out of box
.setDateFormat("yyyy-MM-dd HH:mm:ss")
// Registering byte[] to the type adapter
.registerTypeAdapter(byte[].class, getByteArrayTypeAdapter())
.create();
private static final RestTemplate restTemplate = new RestTemplate();
private static final String URL = "http://localhost";
public static void main(final String... args) {
sendPostRequest("hello world".getBytes(), byte[].class);
}
private static void sendPostRequest(final Object object, final Type type) {
// This is where we're binding the output stream I was asking in the question comments
final RequestCallback requestCallback = request -> gson.toJson(object, type, new OutputStreamWriter(request.getBody()));
// Spring RestTemplates stuff here...
final SimpleClientHttpRequestFactory requestFactory = new SimpleClientHttpRequestFactory();
requestFactory.setBufferRequestBody(false);
final ResponseExtractor<String> responseExtractor = new HttpMessageConverterExtractor<>(String.class, restTemplate.getMessageConverters());
restTemplate.setRequestFactory(requestFactory);
// Let it fly
restTemplate.execute(URL, POST, requestCallback, responseExtractor);
}
Note that you might write a specialized type adapter for special types that could write directly to output stream so you could not get rid of byte[] at all. You could also vote up for this issue at the official Gson issue tracker: https://github.com/google/gson/issues/971 and probably there could no any need in using any Java Reflection API hacks in a future version of Gson.

why is the internal file of my android app doubling in size when adding an item to an Arraylist?

Situation: I have an android application. In this android application I have an internal file containing a "User" class (see code below) this user class has an array-list comprised of spendings, of the "Spending" class, comprised of several different basic attributes.
Problem: When I get this user from the internal file, add a "Spending" Object to the array-list of the User, then re-save this user (delete file and recreate) it doubles in size the Spendings array-list.
I even observed it by looking at the files themselves and can see clearly that the entire array-list is doubled every time a spending is added. the user does not seem to be duplicated.
I tried the same process only without adding the spending and it saves just fine without duplication.
the User class:
public class User implements Serializable {
private int id_utilisateur;
private String mail;
private String motDePasse;
private ArrayList<Spending> mySpendings;
public Utilisateur(int id_utilisateur, String mail, String motDePasse) {
this.id_utilisateur = id_utilisateur;
this.mail = mail;
this.motDePasse = motDePasse;
this.mySpendings= new ArrayList<>();
}
//Getters and Setters of all attributes here//
public void addSpending(Spending mySpending) {
mySpendings.add(mySpending);
}
}
My Spending class :
public class Spendingimplements Serializable {
private Integer idSpending;
private Date dateSpending;
private double montant;
private String pieceJoint;
private Magasin magasin;
private String domaine;
private Date garantieDebut;
private Date garantieFin;
private User user;
public Spending(Integer idSpending, Date dateSpending, double montant, User user, String domaine, Magasin magasin, String pieceJoint, Date garantieDebut, Date garantieFin) {
this.idSpending= idSpending;
this.dateSpending= dateSpending;
this.montant = montant;
this.user= user;
this.domaine = domaine;
this.magasin = magasin;
this.pieceJoint = pieceJoint;
this.garantieDebut = garantieDebut;
this.garantieFin = garantieFin;
}
public Spending(Integer idSpending, Date dateSpending, double montant, User user, String domaine, Magasin magasin, String pieceJoint) {
this.idSpending= idSpending;
this.dateSpending= dateSpending;
this.montant = montant;
this.user= user;
this.domaine = domaine;
this.magasin = magasin;
this.pieceJoint = pieceJoint;
this.garantieDebut = null;
this.garantieFin = null;
}
//geters and setters here//
}
My class StorageHelper:
public final class StorageHelper {
public StorageHelper() {}
public static void storeObject(Context context, Object object) {
try {
File dir = context.getFilesDir();
File file = new File(dir, "UserData.data");
file.delete();
FileOutputStream fos = context.openFileOutput("UserData.data", Context.MODE_PRIVATE);
ObjectOutputStream os = new ObjectOutputStream(fos);
os.writeObject(object);
os.close();
fos.close();
} catch (Exception e) {
Log.e("userFile", "Error: Failed to save User into internal storage - \n" + e.toString());
}
}
public static User getUser(Context context) {
User mainUser = null;
try {
FileInputStream fis = context.openFileInput("UserData.data");
ObjectInputStream is = new ObjectInputStream(fis);
mainUser = (User) is.readObject();
is.close();
fis.close();
} catch (Exception e) {
Log.e("userFile", "Error: loading from the internal storage failed - \n" + e.toString());
} finally {
return mainUser;
}
}
}
MainActivity :
StorageHelper storageHelper;
User mainUser = storageHelper.getUser(this.getBaseContext());
mainUser.addSpending(mySpending);
storageHelper.storeObject(this.getBaseContext(), mainUser);

How to save Object (includes Array) in Preferences?

How to save Object in Preferences which looks like this:
public class ToDoList {
public String name;
public String date;
public ArrayList<Product> products = new ArrayList<Product>();
public boolean isChecked;
}
and then loads its values?

You could do it with serialization. Serialization of an object is a short unique String format of an object that can be serialized. Particularly almost every object can be serialized in java except from the View object. You won't have any problem in your case.
How to do it:
You should make class ToDoList implement Serializable and all classes that are used inside your object, ex Product. String, boolean ArrayList are serializable so you don't have to do anything.
When implementing serialization in an object you have to supply a serial version UID which would be then used to serialize.
So ToDoList would be something like:
public class ToDoList implelements Serializable {
private static final long serialVersionUID = //generate random by eclipse
.....
public ArrayList<Product> products = new ArrayList<Product>();
}
and Product:
public class Product implelements Serializable {
private static final long serialVersionUID = //generate random by eclipse
.....
}
then include this static helper class:
public class ObjectSerializeDeserialize {
public static String ObjectSerialization(Object obj)
{
ByteArrayOutputStream byteArray = new ByteArrayOutputStream();
try
{
ObjectOutputStream objectOutputStream = new ObjectOutputStream(byteArray);
objectOutputStream.writeObject(obj);
objectOutputStream.close();
}
catch (Exception e) {
e.printStackTrace();
return "";
}
return new String(Base64.encode(byteArray.toByteArray(), 0));
}
public static Object ObjectDeserialization(String str)
{
byte[] byteArray = Base64.decode(str,0);
Object o;
try
{
ObjectInputStream ois = new ObjectInputStream(new ByteArrayInputStream(byteArray));
o = ois.readObject();
ois.close();
}
catch(Exception e)
{
e.printStackTrace();
return null;
}
return o;
}
}
and simply use the following code:
String todolistSer = ObjectSerializeDeserialize.ObjectSerialization(todolistObj);
the above line of code will return an empty String if something goes wrong and will print the detailed message in the log cat.
Then simply save the todolistSer as a String in preferences and reclaim your object like this:
ToDoList todolistObj = (ToDoList) ObjectSerializeDeserialize.ObjectDeserialization(todolistString);
suppress any warnings that are issued by the above method and you are done!
P.S. you can use the above solution whenever you have complicated structures that can not be saved as raw variables and you still don't want to use a database

Preferences are simple key ,value pairs. In your case better use SQLite.

Saving Parcelable data

I have a class which implements Parcelable, and could not implement Serializable because it contains some basic Android classes that I can not modify.
Some of the objects in this class are for example Location and PendingIntent (which are all conveniently Parcelable).
My problem is saving this information between the instances of my main Activity.
Currently, I'm holding a static reference to this class, which works well. But I assume that when I re-install the app, and probably when updates will come around, I won't be able to trust that this static member won't be re-initialized.
I tried to write this Parcelable to a file, but using marshall() is not always working (I'm getting Binder can't be marshalled error).
How can I safely save this information?
Thanks

Using static in your example leads to memory leaks and is not a good way to do anything.
I suggest using static only in 3 cases:
static final String or int - constants
on inner classes (so that they don't contain reference to outer class)
on util or in some cases (like CustomFragment.newInstance) factory methods
The question is why would you want to persist PendingIntent? Its usecase is for inter-process-communication.

I use a StateControl class to handle reading/writing to disc:
public class StateControl {
Context mContext;
Thread worker;
WriteObjectToFile writer;
// StateControl Constructor
public StateControl(Context context) {
mContext = context;
// Construct a writer to hold and save the data
writer = new WriteObjectToFile();
// Construct a worker thread to handle the writer
worker = new Thread(writer);
}// end of StateControl constructor
// Method to save the global data
public void saveObjectData(Object object, String key) {
if (object == null){
// I had a different action here
} else {
// Write the data to disc
writer.setParams(new WriteParams(object, key));
worker.run();
}
}// end of saveGlobalData method
// Method to read the Global Data
public Object readObjectData(String key){
Object returnData = (Object) readObjectFromFile(key);
if (returnData == null){
// I had a different action here
} else {
return returnData;
}
}// end of readGlobalData method
// Method to erase the Global data
public void clearObjectData(String key){
writer.setParams(new WriteParams(null, key));
worker.run();
}// end of clearGlobalData method
private class WriteObjectToFile implements Runnable {
WriteParams params;
public void setParams(WriteParams params) {
this.params = params;
}
public void run() {
writeObjectToFile(params.getObject(), params.getFilename());
}
private boolean writeObjectToFile(Object object, String filename) {
boolean success = true;
ObjectOutputStream objectOut = null;
try {
FileOutputStream fileOut = mContext.openFileOutput(filename, Activity.MODE_PRIVATE);
objectOut = new ObjectOutputStream(fileOut);
objectOut.writeObject(object);
fileOut.getFD().sync();
} catch (IOException e) {
success = false;
e.printStackTrace();
} finally {
if (objectOut != null) {
try {
objectOut.close();
} catch (IOException e) {
// do nothing
}
}// end of if
}// End of try/catch/finally block
return success;
}
}// end of writeObjectToFile method
private Object readObjectFromFile(String filename) {
ObjectInputStream objectIn = null;
Object object = null;
try {
FileInputStream fileIn = mContext.getApplicationContext().openFileInput(filename);
objectIn = new ObjectInputStream(fileIn);
object = objectIn.readObject();
} catch (FileNotFoundException e) {
// Do nothing
} catch (IOException e) {
e.printStackTrace();
} catch (ClassNotFoundException e) {
e.printStackTrace();
} finally {
if (objectIn != null) {
try {
objectIn.close();
} catch (IOException e) {
// do nowt
}
}
}
return object;
}
private static class WriteParams {
Object object;
String filename;
public WriteParams(Object object, String filename) {
super();
this.object = object;
this.filename = filename;
}
public Object getObject() {
return object;
}
public String getFilename() {
return filename;
}
}
}
Then invoke the public methods to kick off the writing/reading. For this version, I also having it taking place in a separate thread, but you could modify that if you needed to.

Binder
Most developers will not implement this class directly, instead using
the aidl tool to describe the desired interface, having it generate
the appropriate Binder subclass.
from the official documentation
Do you need to store the Binder object with the rest of your object? Maybe you can save your object without the Binder instance, and re-create the Binder object with aidl after you restore the object

Android getResource in non activity class from an Non Activity Class?

I got this following method in an non Activity class, My code is below.
public class ReadTextByLineNo {
public void setContext(Context _context) {
if (context == null) {
context = _context;
}
}
public String getTextByLine(int Filename,int LineNumber)
{
String output="";
String line="";
int counter=1;
try
{
InputStream in = context.getResources().openRawResource(Filename);
//InputStream in = assetManager.open(Filename);
if(in!=null)
{
InputStreamReader input = new InputStreamReader(in);
BufferedReader buff = new BufferedReader(input);
while((line=buff.readLine())!=null)
{
if(counter ==LineNumber){
output=line;
}counter++;
}in.close();
}else{
Log.e("Input STREAM PROBLEM", "TEXT IS NULL NULL NULL NULL NULL");
}
}catch(Exception e)
{
//log
}
return output;
}
**I am calling this method from an NON_ACTIVITY CLASS LIKE THIS **
class sample implements Isample
{
ReadTextByLineNo read = new ReadTextByLineNo();
String subMsg = read.getTextByLine(R.raw.subtitle, storySceneId);
//the above string is to called from an activity called Layout
}
How do I use resources/context from an non activity class? I cannot use the context in constructor since I'm also calling the method from an non Activity class.
so I can't set read.setContent(this); where I got setContext method in my ReadtextByLineNo class, thanks for the help .
Please help me to get the context/resourse in the class sample and example by code is appreciated

public class ReadTextByLineNo {
private static Context context;
public static void setContext(Context mcontext) {
if (context == null)
context = mcontext;
}
}
when your application start, just initialize this context, by calling
ReadTextByLineNo.setContext(getApplicationContext());
from your main activity..
Enjoy...