How to bulk insert from Json to Sqlite in Android [duplicate] - android

This question already has answers here:
Android SQLite database: slow insertion
(5 answers)
Closed 8 years ago.
How is it possible to insert bulk json data coming from server into Sqlite database in Android very efficiently. The method I use now is very inefficient and it takes almost a minute to complete insertion of about 2000 records. The method I am following is :
for (int i = 0; i < jsonObj.length(); i++)
{
JSONObject itemObj = (JSONObject) jsonObj.get(i);
ContentValues value1 = new ContentValues();
ContentValues value2 = new ContentValues();
value2.put(DbHelper.BusinessID,itemObj.getString("BusinessID"));
value2.put(DbHelper.LocationID,itemObj.getString("LocationID"));
JSONArray list = itemObj.getJSONArray("OfficeDetails");
if (list != null)
{
for (int k = 0; k < list.length(); k++)
{
JSONObject elem = list.getJSONObject(k);
if (elem != null)
{
try
{
value1.put(DbHelper.Office_Code,elem.getInt("Office_Code"));
value1.put(DbHelper.Office_District,elem.getInt("Office_District"));
db.insert(DbHelper.MessageDetail,null, value1);
}
catch (Exception e)
{
e.printStackTrace();
}
}
db.insert(DbHelper.Message,null, value2);
}
The input that is coming is a nested Json array, which itself is nested. Is there a better way to fastly insert huge amount of data in very short time ?

You could try bulkInsert as the following:
ContentResolver.bulkInsert (Uri url, ContentValues[] values); //Array of rows to be inserted
This only fits if you are going to use only 1 URI, if you are going to use multiple uris, you should use applyBatch method in your ContentResolver.
Hope it helps

First create the model class of your json data. Using Gson you can get data inside an arraylist. then you can insert data into sqlite using that arraylist. GreenDao is your best option for fast performance.
when you receive json data in a stream, use following code :
Type collectionType = new TypeToken<ArrayList<your_model>>() {}.getType();
Gson gson = new Gson();
ArrayList<your_model> yourModelList = gson.fromJson(stream, collectionType);

There is an excellent library called JSQL available in Github.
https://github.com/wenchaojiang/JSQL
Its makes it so easy to save Persist JSON string and JSON Objects to your SQLite on Database.

Create a list of your json data and then used this custom query for bulk data insertion:
/**
* insert the bulk data into database
*
* #param query to insert data into table
* #param parameter data to be inserted into table
* #return number of rows got inserted
*/
protected int insertBulk(String query, String[][] parameter) throws SQLiteConstraintException, SQLiteException {
int rowCount = -1;
SQLiteStatement statement = mSqldb.compileStatement(query);
mSqldb.beginTransaction();
try {
for (int index = 0; index < parameter.length; index++) {
statement.bindAllArgsAsStrings(parameter[index]);
statement.execute();
statement.clearBindings();
}
rowCount = parameter.length;
} finally {
mSqldb.setTransactionSuccessful();
mSqldb.endTransaction();
}
return rowCount;
}

Related

Retrieve Specific Columns from SQLite Database

I have an app that gets all the data from the sqlite database and converts it into JSON, but I was wondering how would I be able to do it if I only want to get the data of specific columns?
This is the code I use to get all the data from the SQLite databse and convert it to JSON:
Cursor data = db.getCartItems();
orderNameArray = new JSONArray();
data.moveToFirst();
while(data.isAfterLast() == false)
{
int totalColumn = data.getColumnCount();
orderNameObject = new JSONObject();
for (int i = 0; i < totalColumn; i++)
{
try {
orderNameObject.put(data.getColumnName(i), data.getString(i));
} catch (JSONException e) {
e.printStackTrace();
}
}
orderNameArray.put(orderNameObject);
data.moveToNext();
}
data.close();
Thanks in advance for all the insights or help in advance! :D
If you want to fetch only selected fields of table, then use the following query:
SELECT Coulmn1, Coulmn2, Coulmn3 FROM TABLENAME;
or u want all data
SELECT * FROM TABLENAME;
for more info https://www.tutorialspoint.com/sqlite/sqlite_select_query.htm

How to convert Records From sqlite to Jsonobject?

This may be Dumb question but i have tried hard i couldn't do here let me explain my problem i have retrieve all records from sqlite and converted using Gson library but it is converting as jsonarray not as jsonobject but i need jsonobject to send to the server how can i do this here let me post my code:
This is the db code where i retrieve records:
*public List<Model_Account> toServer() {
String countQuery = " SELECT * FROM " + Model_Account.Accunt_Table;
SQLiteDatabase db = this.getReadableDatabase();
Cursor cursor = db.rawQuery(countQuery, null);
List<Model_Account> listobj = new ArrayList<Model_Account>();
if (cursor.moveToFirst()) {
do {
Model_Account modelobj = new Model_Account();
modelobj.setCompany_group(cursor.getString(cursor.getColumnIndex(Model_Account.company_groups)));
modelobj.setParent_company(cursor.getString(cursor.getColumnIndex(Model_Account.Parent_company)));
modelobj.setCompany_name_id(cursor.getInt(cursor.getColumnIndex(Model_Account.companyname_id)));
modelobj.setCompany_type(cursor.getInt(cursor.getColumnIndex(Model_Account.Company_type)));
modelobj.setAddrss_line1(cursor.getString(cursor.getColumnIndex(Model_Account.Address_line1)));
modelobj.setCityid(cursor.getInt(cursor.getColumnIndex(Model_Account.CityID)));
modelobj.setPincode(cursor.getString(cursor.getColumnIndex(Model_Account.Pincode)));
modelobj.setLandline1(cursor.getString(cursor.getColumnIndex(Model_Account.Landline1)));
modelobj.setUrl(cursor.getString(cursor.getColumnIndex(Model_Account.Url)));
modelobj.setEmailid(cursor.getString(cursor.getColumnIndex(Model_Account.Email_id)));
modelobj.setIndustryid(cursor.getInt(cursor.getColumnIndex(Model_Account.IndustryID)));
modelobj.setAcm_id(cursor.getInt(cursor.getColumnIndex(Model_Account.Account_managerid)));
modelobj.setRegionID(cursor.getInt(cursor.getColumnIndex(Model_Account.regionid)));
modelobj.setMulti_location(cursor.getInt(cursor.getColumnIndex(Model_Account.mutilocationid)));
modelobj.setStateid(cursor.getInt(cursor.getColumnIndex(Model_Account.State_id)));
modelobj.setAddrss_line2(cursor.getString(cursor.getColumnIndex(Model_Account.Address_line2)));
modelobj.setAddrss_line3(cursor.getString(cursor.getColumnIndex(Model_Account.Address_line3)));
modelobj.setLandline2(cursor.getString(cursor.getColumnIndex(Model_Account.Landline2)));
listobj.add(modelobj);
}
while (cursor.moveToNext());
}
return listobj;
}
Here is the code where i try to convert to jsonobject:
listobj = account_sf_db.list();
Gson gson1=new Gson();
String s=gson1.toJson(listobj);
AM getting like this
[{"AddressLine1":"purasai","AddressLine2":"otteri","AddressLine3":"t nagar","CompanyGroup":"yogangroups","CompanyName":"yogan inc","IsIndividual":1,"IsMulitLocation":1,"LandLine1":"landline1","LandLine2":"Landline2","PinCode":"pincode","WebSiteContactEmailID":"vbbb","WebsiteURL":"ghb","company_name_id":68,"AccountManagerID":185,"CityID":165,"IndustryID":4,"RegionID":24,"StateID":129}]
But i must get like this:
{"AccountName":"Example"}
#Yog Instead of supplying listobj (which is a list of Model_Account objects built in the cursor loop) to gson1, perhaps you want to supply an individual Model_Account object. I don't know supplying which Model_Account object would meet your application requirement though. Maybe you need to narrow your SQL query to get just that unique Model_Account object which you want to convert to JSON format.

Parse get multiple objects using their objectIds

I have a ArrayList<String> -named listObjectId below- of objectIds. I'm trying to get all the objects that have an objectId contained in the ArrayList.
The solution I have right now, I think, is very bad from a performance point of view:
for (int i = 0; i < listObjectId.size(); i++) {
ItemModel mItemModelRetrieved = null;
ParseQuery<ItemModel > query = ParseQuery.getQuery(ItemModel .class);
try {
mItemModelRetrieved = query.get(listObjectId.get(i));
subscriber.onNext(mItemModelRetrieved ); //-- I'm using RxJava
} catch (ParseException e) {
Log.e(TAG, "Error Local " + e.getMessage());
}
}
You're using the wrong method. You have the object ids, so create a ParseObject with them using ParseObject.createWithoutData and then fetch the object. Try the following:
List<ParseObject> parseObjects = new ArrayList<>();
for (String objectId : listObjectId) {
parseObjects.add(ParseObject.createWithoutData(ItemModel.class, objectId));
}
ParseObject.fetchAll(parseObjects);
// parseObjects will now contain all data retrieved from Parse.
The error you're getting tells you that the data type of the column you query must be of type Array, not the value you pass into the method.

How to post multiple rows (or a LIST object) to PHP using JSON (Android)

I am creating an android application in which I am using SQLite. I have one table in which I have 5 fields. There are many rows in that table and I want to send all those rows using JSON to my PHP code so I can receive the object and insert it in the MySQL database.
What is the best way to do it. I am using LIST object but don't know how I can use POST it using JSON.
Please guide!
i am using this code
mCursor =db.rawQuery("SELECT * FROM customer", null);
while (mCursor.moveToNext()) {
// In one loop, cursor read one undergraduate all details
// Assume, we also need to see all the details of each and every undergraduate
// What we have to do is in each loop, read all the values, pass them to the POJO class
//and create a ArrayList of undergraduates
customer = new Customer();
customer.CustomerName = mCursor.getString(0).toString();
customer.Phone = mCursor.getString(1).toString();
customer.BrandName = mCursor.getString(2).toString();
customer.City = mCursor.getString(3).toString();
customer.FamilyMembers = mCursor.getString(4).toString();
customerList.add(customer);
}
for (int i = 0; i < customerList.size(); i++)
{
JSONObject row = new JSONObject();
row.put("Customer", customerList.get(i));
json.put(row);
}
Don't know how to post it to PHP code.
You have to iterate through your List and fill an JSONArray with JSONObjects for each row. In the JSONObject you can have the key as your column-name and the value with the type you need.
JSONArray json = new JSONArray();
for (int i = 0; i < list.size(); i++) {
JSONObject row = new JSONObject();
row.put("key1", list.get(i).value1);
...
json.put(row);
}
(not tested). Then you can post the json.toString() to the server and retrieve it in PHP with
$jsonstring = file_get_contents('php://input', true);

Saving ArrayLists in SQLite databases

So I want to save an ordered set of double values, and I want to be able to insert, retrieve or delete any value from this easily. As of such, I'm using a an ArrayList, where I define a class called Doubles to store the double values.
How do I store this arraylist in a record in an SQLite database? I mean...what should the columns type be? Can it be done?
You cannot insert ArrayList directly into Sqlite. Instead, you could use JSONObject (org.json.JSONObject) to insert the ArrayList. Please check below snippet, you can try something like below....
To insert,
JSONObject json = new JSONObject();
json.put("uniqueArrays", new JSONArray(items));
String arrayList = json.toString();
Insert the string into db.
To Read,
Read the string from db as String,
JSONObject json = new JSONObject(stringreadfromsqlite);
ArrayList items = json.optJSONArray("uniqueArrays");
To Insert :
ArrayList<String> inputArray=new ArrayList<String>();
Add Values to inputArray
Gson gson = new Gson();
String inputString= gson.toJson(inputArray);
System.out.println("inputString= " + inputString);
Use "inputString" to save the value of ArrayList<String> in SQLite Database
To retreive:
Get the String from the SQLiteDatabse what you saved and changed into ArrayList type like below:
outputarray is a String which is get from SQLiteDatabase for this example.
Type type = new TypeToken<ArrayList<String>>() {}.getType();
ArrayList<String> finalOutputString = gson.fromJson(outputarray, type);
In my case it was ArrayList of POJO classes Note
private String mNoteTitle;
private int mFingerIndex;
private Point mNoteCoordinates;
public Note(String noteTitle, int fingerIndex, Point noteCoordinates) {
this.mNoteTitle = noteTitle;
this.mFingerIndex = fingerIndex;
this.mNoteCoordinates = noteCoordinates;
}
As manual says JSONObject supports only following types: Object: a JSONObject, JSONArray, String, Boolean, Integer, Long, Double, NULL, or null. May not be NaNs or infinities. So, I should break my Note class into supported objects.
JSONObject json = new JSONObject();
JSONArray jsonArray = new JSONArray();
for(Note note: chordShape.getNotes()){
JSONObject singleNoteJsonObject = new JSONObject();
try {
singleNoteJsonObject.put(SHAPE_NOTE_TITLE, note.getNoteTitle());
singleNoteJsonObject.put(SHAPE_NOTE_FINGER_INDEX, note.getFingerIndex());
singleNoteJsonObject.put(SHAPE_NOTE_X, note.getNoteCoordinates().x);
singleNoteJsonObject.put(SHAPE_NOTE_Y, note.getNoteCoordinates().y);
} catch (JSONException e){
e.printStackTrace();
}
jsonArray.put(singleNoteJsonObject);
}
Pack created array into JSONObject.
try {
json.put(SHAPE_NOTES, jsonArray);
Log.i(TAG, json.toString());
} catch (JSONException e){
e.printStackTrace();
}
Create String.
String notesList = json.toString();
Put created String in ContentValues, cause in my case it's Android app
if(notesList.length() > 0){
contentValues.put(DatabaseHelper.SHAPE_NOTES_LIST, notesList);
}
And when i should read values from SQLite database.
ArrayList<Note> notes = new ArrayList<>();
while(cursor.moveToNext()){
JSONObject jsonNotes = null;
try {
jsonNotes = new JSONObject(cursor.getString(cursor.getColumnIndex(DatabaseHelper.SHAPE_NOTES_LIST)));
} catch (JSONException e){
e.printStackTrace();
}
if(jsonNotes != null){
Log.i(TAG, jsonNotes.toString());
JSONArray jsonArray = jsonNotes.optJSONArray(SHAPE_NOTES);
for(int i = 0; i < jsonArray.length(); i++){
Note note = null;
JSONObject arrayObject = null;
try {
arrayObject = jsonArray.getJSONObject(i);
} catch (JSONException e){
e.printStackTrace();
}
if(arrayObject != null){
try {
note = new Note(
arrayObject.getString(SHAPE_NOTE_TITLE),
arrayObject.getInt(SHAPE_NOTE_FINGER_INDEX),
new Point(
arrayObject.getInt(SHAPE_NOTE_X),
arrayObject.getInt(SHAPE_NOTE_Y)
)
);
} catch (JSONException e){
e.printStackTrace();
}
}
if(note != null){
notes.add(note);
}
}
}
}
cursor.close();
I suggest going through all 3 Notepad tutorials you want to store the values your storing to a database table. you don't store the actual array directly into the database just the data. but you shouldn't actually need to use an array at all instead of adding a new item to the array instead call your db insert method
I've needed to do something similar in my application, where I have a custom class (Foo, Bar, etc.) and I have an ArrayList of foo, bar, etc. that I persist to SQL. My knowledge of SQL isn't strong, but I'll explain my approach here in case it helps.
My understanding is that to store any kind of object, you need to define a particular table for that object type, where the table has separate columns representing the primitive types within that object. Furthermore, to persist and retrieve an ArrayList of those objects, you'll use one table row per ArrayList entry, and iterate over in a loop to store and retrieve.
There are ArrayLists of several custom classes in my application that I wanted to persist to DB. So, to make things tidy (well, to me at least -- I'm still a relatively new Java / Android programmer, so take this with a pinch of salt) I decided to implement a kind of "SQL Serializable Interface" that my DB-persistable objects must implement. Each object (Foo, Bar, etc.) that can be persisted to DB must implement:
A public static final TABLE_NAME string, the name of the SQL DB table used for this object type.
A public static final TABLE_CREATE_STRING, a complete SQL instruction to create the table for this object.
A constructor method to populate its member variables from a ContentValues object.
A 'get' method to populate a ContentValues from its member variables.
So, say I have ArrayLists of objects Foo and Bar. When the DB is first created, within my DB helper class I call Foo.TABLE_CREATE_STRING, Bar.TABLE_CREATE_STRING, etc. to create the tables for those objects.
To populate my ArrayList, I use something like:
cursor = dbh.retrieve(Foo.TABLE_NAME);
if(!cursor.moveToFirst()){
return false
}
do{
DatabaseUtils.cursorRowToContentValues(cursor, vales);
FooArrayList.add( new Foo(values) );
} while( cursor.moveToNext() );
Create a dbHelper class which has an inner class and pretty much whatever the notepad tutorial says. The class must be having an insertion method somthing like this :-
public long insertRows(ContentValues values, String tableName) {
long val = myDatabase.insert(tableName, null, values);
return val;
}
This method will then add values into the table row.
After that you can call this method from your main activity and since you are using cursor i believe you will call the method in a for loop
for(i=0;list.length();i++) // or may be its list.size :P
{
// Call the method here
}
and keep adding value in the database by calling the method in for loop

Categories

Resources