storing large data in android

storing large data in android - android

I am pulling a large amount of json from a restful server. I use the GSON library from google to traverse this json and it works great. Now I want to save all of the json objects in my sqlite db, however I want to make use of a transaction to add all of them at once. This is difficult if you dont have all the objects ready in one datastructe. Since I am traversing the json one object at a time, I guess I would have to store that in a data structure such as an arraylist or hashmap and then afterwards use a database transaction to do the inserts fast. However... Storing a large amount of data aka 200 000 json objects into a structure in memory can take up a lot of memory and wil probably run out as well. What would be the best way to get all of that json objects into my sqlite db and at the same time not use up a lot of menory in otherwords storing and inserting in a way that allows for a lot of recycling.

If you want to add a large amount of data at an unique moment : it will take a lot of memory anyway. 200 000 large JSON objects take a certain amount of memory and you will not be able to change it.
You can keep this behavior, but I think it's not a great solution because you create a huge memory consumption on both Android device and server. It will be better if you receive data part by part and adding them this way : but you need to have control on the server code.
If you are absolutely forced to keep this behavior, maybe you should receive all the data at the same time, parse them on a huge JSON object, then make multiple transactions. Check if every transaction was executed correctly and put back your database in a good state if not. It's a really bad way to do it, IMHO... but I don't know all your constraints.
To finish : avoid receiving a large amount of data at only one time. It will be better to make multiple requests to get partial data set. It will make your app less network dependant : if you loose the network for 2 seconds, maybe only one request will fail. So you will have to retry only one request and received again a small part of data. With only one huge request : if you loose the network, you will have to retry the entire request...

I know this is not the best implementation of handling large json input in Android, but it certainly works great.
So my solution is pretty simple:
While parsing the JSON code, make use of prepared statements and a db transaction to execute the inserts.
So in short, as soon as a JSON object is parsed, take that info, insert it into the db using the prepared statement and the db transaction, then move on to the next object.
I have just pulled 210 000 rows with 23 fields each (thats 4.6 million values) from a remote server via JSON and GSON and at the same time inserting all those values into my SQLite3 db on the device, in less than 5 minutes and without wasting any memory. Each iteration of parsing/inserting uses the same amount of memory and is cleaned on the next iteration.
So yeah, there's the solution. Obviously, this is not the best solution for commercial applications or tables with 1000 000 + records, but it works great for my situation.

Have you ever tried to add a lot of data (and I really mean a lot, in my case 2600 rows of data) into the Android-internal database (SQLite)?
If so, you propably went the same road as I did.
I tried a normal InsertStatement which was way to slow (10 sec. in my emulator). Then I tried PreparedStatements. The time was better but still unacceptable. (6 sec.). After some frustrating hours of writing code and then throwing it away, I finally found a good solution.
The Android-OS provide the InsertHelper as a fast way to do bulk inserts.
To give you an overview of the performance (measured with an emulator on a crap computer, trying to insert 2600 rows of data):
Insert-Statement
10 seconds
Prepared-Statements
6 seconds
InsertHelper
320 ms
You can speed up the insertion even more with temporarily disable thread locks. This will gain about 30 % more performance. However it’s important to be sure that only one thread per time is using the database while inserting data due to it’s not threadsafe anymore.
public void fillDatabase(HashMap<String, int[]> localData){
//The InsertHelper needs to have the db instance + the name of the table where you want to add the data
InsertHelper ih = new InsertHelper(this.db, TABLE_NAME);
Iterator<Entry<String, int[]>> it = localData.entrySet().iterator();
final int firstExampleColumn = ih.getColumnIndex("firstExampleColumn");
final int secondExampleColumn = ih.getColumnIndex("secondExampleColumn");
final int thirdExampleColumn = ih.getColumnIndex("thirdExampleColumn");
try{
this.db.setLockingEnabled(false);
while(it.hasNext()){
Entry<String, int[]> entry = it.next();
int[] values = entry.getValue();
ih.prepareForInsert();
ih.bind(firstExampleColumn, entry.getKey());
ih.bind(secondExampleColumn, values[0]);
ih.bind(thirdExampleColumn, values[1]);
ih.execute();
}
}catch(Exception e){e.printStackTrace();}
finally{
if(ih!=null)
ih.close();
this.db.setLockingEnabled(true);
}
}

Related

DBFlow - update multiple items efficiently

ArrayList list =... --> Get data from database
//do some changes in the values of the objects in the list
for(MyObject object:list){
object.save();
}
Instead of this for loop is there a way to save multiple items that is more efficient than calling save() a bunch of times?

Finaly found it
FlowManager.getModelAdapter(MyTable.class).saveAll(myObjectsList);
and it's faster than calling save multiple times. From some quick testing for 15 items the multiple save took an average of 120 milliseconds while saveAll took about 100 milliseconds. percentage wise it's quite a big difference.

Calling save() multiple times is ok. If there is a thousand entries, you can do it with asynchronous transaction to avoid blocking UI.
If you really want to do save one time, create a new table(model) name ListObject, then use the relationship one-to-many to store it as a List<Object>. So have to save one time only.
https://agrosner.gitbooks.io/dbflow/content/Relationships.html

String Array in SharedPreferences

I need to work with a persistent String Array (n Rows, 1 column).
* On first running the app, the String Array needs to be created empty.
* On subsequent app executions the Array will be populated from a File and the contents need to be available throughout the rest of the app.
* As the app is executed, the Array needs to be able to 'grow' in row count
* As the app is executed, the Array rows need to be able to grow in length
* The user will have the option to Clear the Array of previous entries.
* At the end, the String Array contents will be written back to a File.
I find a lot of references to Putting and Getting from an existing SharedPreferences String[] but, in the newness of my Android development, I am struggling with how to proceed.
EDIT Follows...
The data itself suggests using an Array
Example:
MAIN ST. F55 63 KY08:57 12142015--------KY11:24 12142015345TMH KY13:57 12142015
MAIN ST. F56 WYE123 IN08:57 12142015--------KY11:24 12142015--------KY13:57 12142015
1ST ST. F57 --------KY08:57 12142015--------KY11:24 12142015789FPF KY13:57 12142015
1ST ST. F58 456FPF KY08:57 12142015998FPF KY11:24 12142015--------KY13:57 12142015
1ST ST. F59 789TTM KY08:57 12142015--------KY11:24 121420151234DG KY13:57 12142015
I first need to have this data in a File
Then in one GUI I check for the existence of the file.
If one exists, fine
If none exists, I create one.
Then, in subsequent GUI's, I must check for the existence of parameters
If they do not already exist, add them to the existing data lines.
If they already exist, notify the user
And so on and on.
Then when all of the current 'pass' data has been collected via multiple, separate GUI's, I have to write out the whole data-set into the file.
My reason for thinking that I need a SharedPreference approach is the need to find and check data from GUI to GUI as the user progresses through the app.
If that 'belief' is wrong, I am open to better approach suggestions.
EDIT 2 follows....
On further study of web references, I am beginning to think that perhaps the best approach for this data and how the data needs to change might be to use a SQLite approach. Any ideas about this?
Any assistance/suggestions you might have would be greatly appreciated.

i would discourage you from using sharedpreferences for anything else than preferences. means things that change rarely - really rarely and are really lightweight. do not put much data in there. less is better. the data structures underlying sharedpreferences are not a database.
another note. it is not a string list, but it would be a string set. sets are not necessarily ordered, nor do they necessarily keep their order. means - it is not rows. its a collection of strings that can come back in any fun order (usually there is some, but that depends on the implementation which i do not know)
now you could go and make your own list, your own data structure, save it into a string and read it out, use json to do exactly that or something similar, or better - use a database, which would exactly do that.
http://developer.android.com/training/basics/data-storage/databases.html
explains it, but as you'll see its something that might take some time.
now dont get me wrong, but i have to warn you about the following approach. it is valid, but has many problems and is far from thread safe. it will not be a problem as long as you only open it from the ui thread and do not keep anything in memory to cache - if you do it will create lots of problems.
your problem of adding a line and clearing can be solved by using a file. just a simple file
look here
http://developer.android.com/training/basics/data-storage/files.html#WriteInternalStorage
the change is to append when writing:
openFileOutput("filename", Context.MODE_APPEND);
see the context mode? now you can basically just write one line and append every time.
if you wanna clear the file, just deleteFile("filename")
this is as said not threadsafe, but can be a viable option if used carefully.

Please follow this step to achieve what you want with sharedPreference
create the class Parent for SharePreference
Create your empty Array
Convert Your empty array to String and put it on SharedPreference
to call your empty array from sharedPreference
Call your sharedPreference using your key
Convert the String to array
You get your array from the sharePreference
Hope it helps, and maybe this link will help you :
http://www.androidhive.info/2012/08/android-session-management-using-shared-preferences/

You can use my open-source library called prefser, which solves this problem and uses SharedPreferences and Gson under the hood. It's basically wrapper for default Android mechanism.
You can use it in the following way:
Prefser prefser = new Prefser(context); // create Prefser object
prefser.put("key", Arrays.asList("one", "two", "three")); // save array of Strings
String[] value = prefser.get("key", String[].class, new String[]{}); // read array of Strings
For more information check repository of the project, tests and README.md file.
Link: https://github.com/pwittchen/prefser
Please note, SharedPreferences have some limitations and shouldn't be used for storing large amount of data. If you expect a lot of data, consider using SQLite database or another type of database (e.g. with NoSQL or similar approach if you strive for simplicity).

OK, based on the data, how it needs to be manipulated and the pros and cons of using a SharedPreferences approach, I have decided to go with a SQLite approach.
With that approach I should be able to readily check:
* if the necessary table exists (if not create it)
* if the necessary Field1 + Field2 exists (if not create a new record)
* and I will be able to modify the record's Field3 contents as needed
Then when the user's actions are complete I can convert the SQLite table 'records' into strings and write them out as a File and then either DROP or PURGE the associated SQLite table (until needed next time).
I sincerely appreciate all of your suggestions.
Thank you.

What are ways to cache queries from SQLite database?

I am developing the following functionality:
an user picks a date and gets ListView populated by SimpleCoursorLoader (queries are executed in the background).
User frequently choices adjacent dates and there might be a lot of duplicate queries.
I tested the application and discovered that in case of high frequency requests - it runs very slow.
In order to speedup my application I decided to implement cache where results of queries will be stored. Key - date and value-?
Is it worth doing and what techniques could you advice?

1) Yes, it's really worth doing since DB access is relatively slow (even with such a great thing like SQLite)
2) Considering what I've got from your post I'd suggest using LongSparseArray: key will be date from database (long), stored value - your cached data object (Bundle etc). The reasons are it's:
naturally sorted
sort order is maintained on changes
fast
memory efficient
3) When you need to load overlapping/adjacent interval you have to check bounds and load only absent part
4) If a situation is possible when you cache non-adjacent intervals - you need to manage loaded intervals bounds as well. But if you do it only for list scroll purposes you may omit this (if you don't stop loading data on fling gesture)
About my experience: I've got about 3 times payoff using caching. But actual results depends on database scheme etc. You may get even more

I found MatrixCursor useful for the purpose of caching. I keep HashMap.
Logic: if no request has been done - issue it, get Cursor, convert it to MatrixCursor and write to cache.
Here is the snippet for convertion:
private MatrixCursor cursorToMatrixCursor(Cursor c) {
MatrixCursor result = new MatrixCursor(c.getColumnNames());
if (c.moveToFirst()) {
do {
ArrayList<String> columnValues = new ArrayList<>();
final int nOfColumns = c.getColumnCount();
for(int col = 0; col < nOfColumns; ++col)
columnValues.add(c.getString(col));
result.addRow(columnValues);
} while (c.moveToNext());
}
return result;
}

Android: Best technique to keep data, among Activities, consistent?

The user of my application will be able to change(add,edit,delete) the following data:
name
number1
number2
number3
All those data are parallel. For example:
George 200 100 50
Andreas 450 205 190
John 800 230 180
The user will be able to change the first three (name, time1, time2) from an Activity. And the last one (time3) from another Activity.
Upon change on those data a service would like to know about the changes immediately.
The solutions I am aware of are:
Storing the arrays using shared preferences (Solution) but this would be too time consuming for my application as I may have up to (maximum) 200 data entries. (so 200x4 data to store).
Upon change on the tables I can keep my data consistent by storing them in a database and sending a broadcast to my service with (putExtra) the new data. The service will locally update those arrays. The problem with this solution is that each time I want to make a change on the data I have to change my database too, which is time consuming.
Are there any other solutions?
Is any of the above a good solution for my problem?

How to store xml data into sqlite database

I have a xml format data which is came from server. Now i want to store it into database and it should load on button click. How should i do this?
enter code here
<qst_code> 7 </qst_code>
<qst_prg_code> 1 </qst_prg_code>
<qst_mod_code> 2 </qst_mod_code>
<qst_Question>What is not true about left dominant cardiology circulation? </qst_Question>
<qst_opt1>It is seen in 20% of the population</qst_opt1>
<qst_opt2>Left circumflex artery supplies the Posterior descending artery</qst_opt2>
<qst_opt3>Left circumflex artery terminates as obtuse marginal branch</qst_opt3>
<qst_opt4>Left circumflex artery may originate from right coronary sinus</qst_opt4>
<qst_opt01>1</qst_opt01>
<qst_opt02>1</qst_opt02>
<qst_opt03>1</qst_opt03>
<qst_opt04>1</qst_opt04>
<qst_CorctOpt>1</qst_CorctOpt>
<qst_Marks>10</qst_Marks>
<qst_company_code>1</qst_company_code>
<user_code>1</user_code>

One option is to store it as a string if the data is not too large, else break it into a schema that maps to sqlite and recreate it while loading.

If your XML data is large, I would rather change the data exchange type to json. XML parsing and then insert is a very expensive operation and is time-consuming.
Some issues which you will face with XML parsing and insert.
a. XML parsing is memory intensive and so you heap size will grow, you need to keep an eye on this as this might cause crash.
b. Inserts in SQLite DB will take around ~100ms per tuple (row), so you can calculate the time it will required to pump in thousands of rows of data.
If you data is not too large don't bother about using SQLite.

Develop Reference

The Android operating system is a mobile operating system that was developed by Google (GOOGL?) to be primarily used for touchscreen devices, cell phones, and tablets.