I have to migrate a 5 million tuples sqlite database to realm. The database will be for consultation only. I started converting the 5M tuples from sqlite to json and importing the data with createAllFromJson (). It was taking too long, so I decided to use FlatBuffer instead of json and do the import manually. The problem is that after several import iterations it returns the following exception: io.realm.exceptions.RealmError: Unrecoverable error. Mmap () failed: Out of memory size.
I learned that closing realm is really important for this kind of issues. So I end up building and closing a realm every time I iterate, but I still get that issue.
Doing some research I learned that realm passes really badly with long files. I assume that I have to split the file into several an query them separately.
My question is this:
How do I separate a large realm file into different small files, query them and keep the incredibly fast response?
Is splitting the right aproach?
I have only 1 RealmModel:
public class Owner extends RealmObject {
#PrimaryKey
private long id;
#Index
private String name;
#Index
private String mix;
#Index
private String entry;
#Index
private String value;
// ... Generated getters and setters ...
}
And I insert this way:
for (int i=0; i<flat_buffer_list.lenght(); i++){
RealmConfiguration config = new RealmConfiguration.Builder()
.name("owner.realm")
.build();
Realm realm=Realm.getInstance(config);
OwnerList list= flatbuffer.OwnerList.getElementAsRoot(flat_buffer_list.get(i));
for (int i=0; i<list.lenght(); i++){
//create the realm object an perform a realm.insert(object)
}
realm.close();
}
Related
I have an app that stores lots of data to work offline as well.
I have three classes, in a hierarchy like;
public class MainGroup
{
private UUID Oid;
private String name;
private Date CreatedOn;
}
-
public class Group
{
private UUID Oid;
private String name;
private Date CreatedOn;
private MainGruop MainGroup;
}
-
public class Product
{
private UUID Oid;
private String name;
private Date CreatedOn;
private MainGruop MainGroup;
private Group Group;
}
( Oid fields are selected as PrimaryKey with realm attribute. )
Let's say, all MainGroup objects were stored in Realm DB. Then, when i'm trying to insert Group objects, with nested MainGroup object but with only its Oid field to link its master, Realm updates the MainProduct record (with given Oid), and clear the other fields as nulls.
In same way, when i'm inserting Product objects and nested objects are includes only Oid, realm updates all fields with nulls.
So, there are more complex and deeply related objects and when i make a request to get JSON from server, i must produce a very big JSON response to keep data.
And mention to insert method; I'm creating java objects with JSON response via GSON and i'm using Realm.copyToRealmOrUpdate(obj); method to insert.
To reduce payload (JSON size, serialize and insertion process), i need to find a way to fix this issue.
Basically, there are two things I don't understand: objects with objects and objects with lists of objects
Say I receive a list of objects from the server. Each of them looks like this:
#Entity
public class BigObject {
#PrimaryKey
private int id;
private User user;
private List<SmallObject> smallObjects;
}
with these two objects as fields:
#Entity
public class User {
#PrimaryKey
private int id;
private String name;
#TypeConverters(GenderConverter.class)
public MyEnums.Gender gender;
}
#Entity
public class SmallObject {
#PrimaryKey (autoGenerate = true)
private int id;
private String smallValue;
}
They are more complicated than this, so I can't use #TypeConverters as Room suggests:
error: Cannot figure out how to save this field into database. You can consider adding a type converter for it.
How do I store this data structure in Room?
I think the best way to answer this is a breif overview in storing structures...
Lists
Room does not support storing lists that are nested inside of a POJO. The recommended way to store lists is to use the foreign key approach. Store the List of objects in a seperate table (in this case a smallObjects table) with a foreign key to their related parent object (in this case "big_object_id"). It should look something like this...
#Entity
public class BigObject {
#PrimaryKey
private int id;
private User user;
#Ignore
private List<SmallObject> smallObjects;
}
#Entity(foreignKeys = {
#ForeignKey(
entity = BigObject.class,
parentColumns = "id",
childColumns = "big_object_fk"
)})
public class SmallObject {
#PrimaryKey (autoGenerate = true)
private int id;
private String smallValue;
#ColumnInfo(name = "big_object_fk")
private int bigObjectIdFk
}
Note that we have added the #Ignore annotaiton to List<SmallObject> as we want to ignore the field during Room persistance (as lists are not supported). It now exists so that when we request our list of related small objects from the DB we can still store them in the POJO.
To my knowledge this will mean you are making two queries.
BigObject b = db.BigObjectDao.findById(bOId);
List<SmallObject> s = db.smallObjectDao.findAllSOforBO(bOId);
b.setsmallObjects(s);
It appears that there is a short hand for this in the form of #Relation
Type Converters
These are for cases where you have a complex data structure that can be flattend without losing information, and stored in a single column. A good example of this is the Date object. A Date object is complex and holds a lot of values, so storing it in the database is tricky. We use a type converter to extract the milli representation of a date object and store that. We then convert the millis to a date object on the way out thus keeping our data intact.
Embedded
This is used when you want to take the fields of all nested POJOs in your parent POJO and flatten them out to store in one table. an example :
- name
- age
- location
- x
- y
- DOB
..when embedded this structure would be stored in the database as :
- name
- age
- location_x
- location_y
- DOB
In a sense Embedded exists to save you time creating type converters for every nested object that contains primary type fields like String, int, float, etc...
Convert Object/List<Object> to String and then, Store it.
You can store the objects in Room Library as String. For that, you can serialize the object and store it as String in the Room Database.
Store to Room
Object -> Serialize -> String -> Store
Read from Room
String -> Deserialize ->Object -> Read
How to Serialize/Deserialize?
There are many options available. You can either do it manually or you can use a library for this. You can use Google's GSON library. It is pretty easy to use.
Code: Object -> String
public String stringFromObject(List<YourClass> list){
Gson gson = new Gson();
String jsonString = gson.toJson(list);
return jsonString;
}
Code: String-> Object
public List<YourClass> getObjectFromString(String jsonString){
Type listType = new TypeToken<ArrayList<YourClass>>(){}.getType();
List<YourClass> list = new Gson().fromJson(jsonString, listType);
return list;
}
I am working on some chat - app whith complex messages.
Messages from API stored in realm DB.
When i getting messages: call after API response, deserialization finished, messages correct:
realm.beginTransaction();
realm.copyToRealmOrUpdate(listOfBaseMessageResponses);
realm.commitTransaction();
messages in db replaced and i forced to make a List in my activity and try to handle changes by myself.
Changes tracked in activity by:
messagesChangedListener = new RealmChangeListener() {
...
}
mBaseMessageResponsesRealm = realm.allObjects(BaseMessageResponse.class);
mBaseMessageResponsesRealm.addChangeListener(messagesChangedListener);
Way to get messages:
return realm.allObjects(BaseMessageResponse.class);
BaseMessageResponse class(getters and setters exists. for better understanding not show):
#SerializedName("clr")
#Expose
#PrimaryKey
private String clr;
#SerializedName("cap")
#Expose
private String cap;
#SerializedName("eoc")
#Expose
private int eoc;
#SerializedName("list")
#Expose
private RealmList<MessageInResponse> list = new RealmList<MessageInResponse>();
...
Also MessageInResponse contain nested classes.
Why messages replaced? I try many ways to figure it out but no result.
May be some problems with PK and logic for copyToRealmOrUpdate method?
I am currently using Sugar ORM and Android Async Http Client for my Android application.
I read through the documentation of Sugar ORM and did exactly what is written there.
My HttpClient is using the singleton pattern and provides methods for calling some APIs.
Now comes the bad part about it. I am not able to save the data persistently into my database which is created by Sugar ORM.
Here is the method, that is calling an API:
public void getAvailableMarkets(final Context context, final MarketAdapter adapter) {
String url = BASE_URL.concat("/markets.json");
client.addHeader("Content-Type", "application/json");
client.addHeader("Accept", "application/json");
client.get(context, url, null, new JsonHttpResponseHandler() {
#Override
public void onSuccess(int statusCode, Header[] headers, JSONArray response) {
Log.i(TAG, "Fetched available markets from server: " + response.toString());
Result<Markets> productResult = new Result<Markets>();
productResult.setResults(new Gson().<ArrayList<Markets>>fromJson(response.toString(),
new TypeToken<ArrayList<Markets>>() {
}.getType()));
ArrayList<Markets> marketsArrayList = productResult.getResults();
// This lines tells me that there are no entries in the database
List<Markets> marketsInDb = Markets.listAll(Markets.class);
if(marketsInDb.size() < marketsArrayList.size() ||
marketsInDb.size() > marketsArrayList.size()) {
Markets.deleteAll(Markets.class);
for(Markets m : marketsArrayList) {
Markets market = new Markets(m.getId(), m.getName(), m.getChainId(), m.getLat(),
m.getLng(), m.getBusinessHourId(), m.getCountry(), m.getZip(), m.getCity(),
m.getStreet(), m.getPhoto(), m.getIcon(), m.getUrl());
market.save();
adapter.add(market);
}
adapter.notifyDataSetChanged();
}
List<Markets> market = Markets.listAll(Markets.class);
// This lines proves that Sugar ORM is not saving the entries
Log.i(TAG, "The market database list has the size of:" + market.size());
}
});
}
This is what Logcat is printing:
D/Sugar: Fetching properties
I/Sugar: Markets saved : 3
I/Sugar: Markets saved : 5
I/RestClient: The market database list has the size of:0
Also I took a look at the Sugar ORM tag here at stackoverflow, but no answers or questions could give me a hint on how to solve that problem.
I am a newbie to the android ecosystem and would love any help of you guys to solve this problem.
Thanks in advance
I just solve it the same problem as you have.
It was a pain in the neck but after few hours I find out what caused this problem.
Using Sugar ORM you must not set id property as it's belongs to SugarRecord class,
otherwise ORM will try to update objects instead of insert them.
As I need to have field with my object id, I used json annotation to assign it to another field.
Last step was configure GSON to exclude fields without Expose annotation.
So my class looks like one below now:
public class MyClass
{
#Expose
#SerializedName("id")
private long myId;
#Expose
private String field1;
#Expose
private String field2;
#Expose
private byte[] field3;
#Expose
private double field4;
public MyClass() { }
// parametrized constructor and more logic
}
Cheers!
I'm writing an Android application which will occasionally need to download a json string of around 1MB and containing around 1000 elements, and parse each of these into an SQLite database, which I use to populate a ListActivity.
Even though the downloading and parsing isn't something that needs to be done on every interaction with the app (only on first run or when the user chooses to refresh the data), I'm still concerned that the parsing part is taking too long, at around two to three minutes - it seems like an eternity in phone app terms!
I'm currently using Gson to parse each json object into a custom object that I've defined, and then using an SQLiteOpenHelper to enter it into the database.
My question is - is there a faster way of implementing this? Would it be noticeably faster to interact with the json directly, without using Gson? Or am I doing something stupid in the code below that's slowing things down?
Here's the method I'm using in my AsyncTask to parse the json to SQLite:
protected Boolean doInBackground(Integer... bType) {
InputStream source = getJsonInputStream(bTypeString);
VegDataHandler db = new VegDataHandler(mainActivity, bTypeString);
Gson gson = new Gson();
Reader reader = new InputStreamReader(source);
JsonParser jParser = new JsonParser();
JsonArray jArray = jParser.parse(reader).getAsJsonArray();
aLength = jArray.size();
mCurrProgress = 1;
publishProgress(mCurrProgress, 0, aLength);
/* Each array element is of the form { company: {...} } */
int i = 0;
mCurrProgress = 2;
for (JsonElement obj : jArray) {
Company c = gson.fromJson(obj.getAsJsonObject().getAsJsonObject("company"), Company.class);
db.addCompany(c);
i++;
publishProgress(mCurrProgress, i);
}
}
This is the addCompany method from my VegDataHandler class, which extends SQLiteOpenHelper:
public void addCompany(Company c) {
SQLiteDatabase db = this.getWritableDatabase();
ContentValues values = new ContentValues();
values.put(KEY_ID, c.getCompanyId());
values.put(KEY_NAME, c.getCompanyName());
values.put(KEY_RYG, c.getCompanyRedYellowGreen());
values.put(KEY_COUNTRY, c.getCompanyCountry());
values.put(KEY_URL, c.getCompanyUrl());
values.put(KEY_NOTES, c.getCompanyNotes());
values.put(KEY_EMAIL, c.getCompanyEmail());
db.insertWithOnConflict(TABLE_COMPANY, null, values, SQLiteDatabase.CONFLICT_REPLACE);
db.close();
}
This is the class that holds each json element before adding to the SQLite (I've omitted the getters and setters for brevity).
public class Company {
public Company() {
}
#SerializedName("id")
public int companyId;
#SerializedName("company_name")
public String companyName;
#SerializedName("red_yellow_green")
public String companyRedYellowGreen;
#SerializedName("country")
public String companyCountry;
#SerializedName("url")
public String companyUrl;
#SerializedName("notes")
public String companyNotes;
#SerializedName("email")
public String companyEmail;
}
Thanks in advance for any replies.
First you need to determine the portion(s) of the process that are eating up the most time. From your comment above it sounds like the JSON parsing is the culprit.
If JSON parsing is the issue:
Research and consider a faster JSON parser. Perhaps something like json-smart.
If SQLite/DB bulk inserts are the issue:
See my answer here
General tips:
Recycle objects as much as possible (keep new to a minimum)
Always use transactions in DB bulk inserts at the very least
Don't open/close the database. Do this once at the start/finish of your processing
Use pre-compiled statements!