I ran into trouble trying to send SQLite data to a web server using json (gson).
Everything was fine until the table came round 6000 rows.
I ran into Out of Memory errors.
In my datahelper I have:
public String DonneesToJson (SQLiteDatabase db, int chantier, int chantier_serveur )
{
Cursor c = db.rawQuery("select * from "+P_TABLE+" where "+P_CHANTIER+"="+chantier+ " order by "+P_TIME+" desc ", null);
List<Donnees> donnees = new ArrayList<Donnees>();
c.moveToFirst();
while (c.moveToNext())
{
Donnees d = new Donnees (
c.getInt(c.getColumnIndex(Datahelper.P_ID)),
chantier_serveur,
offset,
c.getInt(c.getColumnIndex(Datahelper.P_PLAN)),
c.getString(c.getColumnIndex(Datahelper.P_TIME)),
c.getInt(c.getColumnIndex(Datahelper.P_PRESSION)),
c.getInt(c.getColumnIndex(Datahelper.P_PROFONDEUR)),
c.getInt(c.getColumnIndex(Datahelper.P_PROFONDEUR_TOTALE)),
c.getInt(c.getColumnIndex(Datahelper.P_ANGLE_X)),
c.getInt(c.getColumnIndex(Datahelper.P_ANGLE_Y)),
c.getString(c.getColumnIndex(Datahelper.P_PIEU)),
c.getInt(c.getColumnIndex(Datahelper.P_NO_RALLONGE)),
c.getString(c.getColumnIndex(Datahelper.P_RALLONGE)),
c.getInt(c.getColumnIndex(Datahelper.P_MOTEUR)),
c.getString(c.getColumnIndex(Datahelper.P_SERIE)),
c.getDouble(c.getColumnIndex(Datahelper.P_COEFF_A)),
c.getDouble(c.getColumnIndex(Datahelper.P_COEFF_B))
);
donnees.add(d);
}
c.close();
Gson gson = new Gson();
return gson.toJson(donnees);
}
Basically I call this like that:
String resultat = dbHelper.DonneesToJson(db,i, chantier_serveur);
HttpPost post2 = new HttpPost("http://www.zzzzzzz.com/test.php");
StringEntity se = new StringEntity(resultat);
se.setContentEncoding( new BasicHeader(HTTP.CONTENT_TYPE, "application/json"));
post2.setEntity(se);
response = client.execute(post2);
On the server side, it's quite basic php to store data in a big sql DB and then do analysis.
ie:
$decoded = json_decode($HTTP_RAW_POST_DATA,true);
foreach ( $decoded as $key => $value)
{
$query = ...
While doing OOM erros, it's slow. I mean getting sql data to json is slow.
I tried to go the jackson route, faster, no out of memory error, but... it can only write to a file or a stream.
I'd try to avoid writing to a file then send the file trough http post.
So I decided to open an http stream to send json data and I'm stuck.
I did not find any example on how to open an output stream to the web server using apache.
Any help appreciated.
Despite the answer from dmon to split the data into batches, you should work on other glitches. Try this code:
public String DonneesToJson (SQLiteDatabase db, int chantier, int chantier_serveur ) {
Cursor c = db.rawQuery("select * from "+P_TABLE+" where "+P_CHANTIER+"="+chantier+ " order by "+P_TIME+" desc ", null);
List<Donnees> donnees = new ArrayList<Donnees>();
if (c != null) { // nullcheck as rawQuery can return null!
Donnees d; // reuse variables for loops
while (c.moveToNext()) { // was buggy before, read comment below code
d = new Donnees (
c.getInt(c.getColumnIndex(Datahelper.P_ID)),
chantier_serveur,
offset,
c.getInt(c.getColumnIndex(Datahelper.P_PLAN)),
c.getString(c.getColumnIndex(Datahelper.P_TIME)),
c.getInt(c.getColumnIndex(Datahelper.P_PRESSION)),
c.getInt(c.getColumnIndex(Datahelper.P_PROFONDEUR)),
c.getInt(c.getColumnIndex(Datahelper.P_PROFONDEUR_TOTALE)),
c.getInt(c.getColumnIndex(Datahelper.P_ANGLE_X)),
c.getInt(c.getColumnIndex(Datahelper.P_ANGLE_Y)),
c.getString(c.getColumnIndex(Datahelper.P_PIEU)),
c.getInt(c.getColumnIndex(Datahelper.P_NO_RALLONGE)),
c.getString(c.getColumnIndex(Datahelper.P_RALLONGE)),
c.getInt(c.getColumnIndex(Datahelper.P_MOTEUR)),
c.getString(c.getColumnIndex(Datahelper.P_SERIE)),
c.getDouble(c.getColumnIndex(Datahelper.P_COEFF_A)),
c.getDouble(c.getColumnIndex(Datahelper.P_COEFF_B)));
donnees.add(d);
}
c.close();
}
return new Gson().toJson(donnees);
}
Your current implementation is buggy and you will never get the first entry. The reason for that is your call to moveToFirst() which moves you to the first. The while loop will move the internal pointer to the second entry with moveToNext() which will result in starting with the second element and completely ignoring the first one. Always and forever...
Related
I have an SQLite Db which stores call logs. Each entry has a number, date, and duration. I have seen that there are many ways we can send the data to the app server. As a JSON String and send one by one from an ArrayList of model class objects.
Which is the correct way to approach this? And how can I create a JSON from these data, I have done as much as getting these data to an ArrayList of objects. since each entry has many data, I am confused about how to do this.
public ArrayList<PhNumber> getCallLogs() {
ArrayList<PhNumber> callLogList;
callLogList = new ArrayList<PhNumber>();
String selectQuery = "SELECT * FROM callInfo where syncStatus = '" + "no" + "'";
SQLiteDatabase database = this.getWritableDatabase();
Cursor cursor = database.rawQuery(selectQuery, null);
if (cursor.moveToFirst()) {
do {
callLogList.add(new PhNumber(Integer.parseInt(cursor.getString(0)), cursor.getString(1),
cursor.getString(2),
cursor.getString(3),
cursor.getString(4)));
Log.e("DbHelper:getCallLogs", callLogList.toString());
} while (cursor.moveToNext());
}
return callLogList;
}
Which is the correct way to approach this?
There is not any one solution . It depends on your use case scenario.
For making JSON , you can do this after getting your data in ArrayList<PhNumber> callLogList
JSONArray jsonArray = new JSONArray();
for (int i = 0; i < callLogList.length(); i++) {
JSONObject jsonobject= new JSONObject();
jsonobject.put("data1",callLogList.get(i).get); //I dont know the field name of your PhNumber so fill accordingly
jsonobject.put("data2",callLogList.get(i).get);
jsonArray.put(jsonobject);
}
I would suggest you to use Retrofit for this, a library that is very easy to use, saves lot of time and code. Serialization and HTTP requests are handled seamlessly using Retrofit.
please refer to this article for simple understanding of Retrofit
I am using a string buffer to read to store values being read from an SQLite database. Inside the loop, the buffer always starts reading from the first database reference.
Eg: The database has the values apple,orange,banana.
Everytime I call the funtion,the string buffer stores the items from the beginning:
first function call: apple
second function call: apple apple orange
third function call: apple apple orange apple orange banana
On the third call i need it just to be: apple orange banana
How can I do this?
public void db{
Cursor res= databaseHelper.getAllData();
if(res!=null){
res.moveToFirst();
try{
while(res.moveToNext())
{
if(res.getString(0)!=null){
stringBuffer.append(res.getString(0)+"\n");
Log.i("TAG",stringBuffer.toString());
stringBuffer1.append(res.getString(1)+"\n");
stringBuffer2.append(res.getString(2)+"\n");
stringBuffer.setLength(0);
stringBuffer1.setLength(0);
stringBuffer2.setLength(0);
}catch(Exception e){}}}
If you need just print "apple orange banana", then use as follow
public void db(){
Cursor res = databaseHelper.getAllData();
if (res != null) {
while (res.moveToNext()) {
String value = res.getString(0) + " ";
stringBuffer.append(value);
Log.d(TAG, stringBuffer.toString())
...
}
res.close()
}
}
Just put StringBuffer stringBuffer = new StringBuffer(""); after public void db{ or clear the stringBuffer by stringBuffer.delete(0, stringBuffer.length()); before you start the loop.
I don't know where you declared stringBuffer but it keeps the previous values. Also you may lose the 1st item in db because you do moveFirst and immediately moveNext, so drop moveFirst
int i =0;
if (cursor.moveToFirst()){
do{
stringBuffer.append.(cursor.getString (i));
i++;
}
while(cursor.moveToNext());
}
cursor.close();
}
Try this instead of that, may this helps you.
I am trying upload some message data to server from my android app.it uploaded the server data well, But problem is, it is upload all message data every time when app started.I want to upload only those message data which would not uploaded previously.How to do this?
Here is my code of fetching and uploading the message data on server:
class SmsContent {
public List<SmsInfo> getSmsInfo() {
String[] projection = new String[] { "_id", "address", "person", "body", "date", "type" };
ContentResolver cr = activity.getContentResolver();
Cursor cursor = cr.query(uri, projection, "body"+" LIKE '%"+smsBuilder1+"%'or+" + "body"+" LIKE '%"+smsBuilder2+"%'", null, "date desc");
assert cursor != null;
int nameColumn = cursor.getColumnIndex("person");
smsbodyColumn = cursor.getColumnIndex("body");
int i = 0;
while (cursor.moveToNext() && i++ < 20) {
SmsInfo smsInfo = new SmsInfo();
smsInfo.setName(cursor.getString(nameColumn));
smsInfo.setSmsbody(cursor.getString(smsbodyColumn).trim());
per = (cursor.getString(smsbodyColumn));
smsInfo.setName(null == personName ? smsInfo.getPhoneNumber() : personName);
prepareListData(per);
infos.add(smsInfo);
}
cursor.close();
return infos;
}
private void prepareListData(final String per) {
// Volley's json array request object
StringRequest stringRequest = new StringRequest(Request.Method.POST, REGISTER_URL,
new Response.Listener<String>() {
#Override
public void onResponse(String response) {
VolleyLog.d(TAG, "Error: " + response);
}
}, new Response.ErrorListener() {
#Override
public void onErrorResponse(VolleyError error) {
VolleyLog.d(TAG, "Error: " + error.getMessage());
// hidePDialog();
}
}){
#Override
protected Map<String, String> getParams() {
Map<String, String> params = new HashMap<String, String>();
params.put(VENDOR, "test");
params.put(ORGMESSAGE, per);
return params;
}
};
// Adding request to request queue
MyApplication.getInstance().addToRequestQueue(stringRequest);
}
}
If anyone want to know more info please ask.
Save recently received message date in SharedPreferences and have a condition in query so as to select the messages which are received after that date.
String dateFromSharedPreferences="2000-01-01";
try{
String dateFromSharedPreferences=getActivity().getPreferences(Context.MODE_PRIVATE);
dateFromSharedPreferences=sharedPref.getString("LastMsgDate", "2000-01-01");
catch(Exception ex){}
Cursor cursor = cr.query(uri, projection, "(body"+" LIKE '%"+smsBuilder1+"%'or+" + "body"+" LIKE '%"+smsBuilder2+"%') and date>'"+dateFromSharedPreferences+"'", null, "date desc");
Dont forget to save the current last received sms date in the same SharedPreferences
while (cursor.moveToNext() && i++ < 20) {
if(i==1) saveDateToSharedPreferences(cursor.getString(cursor.getColumnIndex("date")))
//other codes here
}
saveDateToSharedPreferences function will be as follows
function saveDateToSharedPreferences(String LastMsgDate){
SharedPreferences sharedPref = getActivity().getPreferences(Context.MODE_PRIVATE);
SharedPreferences.Editor editor = sharedPref.edit();
editor.putString("LastMsgDate", LastMsgDate);
editor.commit();
}
Make another column in your table and store data like whether it's uploaded or not. And upload the data which are not uploaded.
So, what you're dealing with here is local cache, mind you that this answer will give you the flow of a way to handle it, the coding is up to you, by the code you posted it seems your up to the task, here are your options:
SharedPreferences.
Android's built in sqlite.
SharedPreferences is an option, but - it would be more suitable for single variables, or for example, an object serialized into a JSON string.
In your case, creating and maintaining a local (on the phone itself, client-side) database would probably be better - since you're maintaining a messaging record, which is big and might reach numbers in the tens, you mentioned that you're not creating a custom table - you should, and since your code suggests that there IS a db somewhere in there, this is generally how I would go about this:
create a new table, holding all the known values plus 2 new columns (e.g. is_sent, is_received).
think of two situations: msg sent, vs. msg received - you might have sent it, the other side (server) might not get it.
Pull the msgs using the cursor (like you did), save them in an ArrayList, and their info in your new table, marking them all as false on the sent and received columns.
Send that list to your server (marking all of them as sent = true in the table).
make sure your server response returns a confirmation (either per id, or just a general "success:true"), and then in the onResponse callback - update your table (use you local list for id comparison if you need to, marking what is relevant with received = true).
modify the method that sends stuff on startup to pull the messages, and first compare them to your table,only send the msgs that are marked as sent=false && received=false, if in doubt - check the id, if it's not in your table - you don't know about it, so send it and make sure your server knows to compare msg id's cross-referenced with user id's to avoid duplicates.
All this requires both client side work and server side work, and possibly some help from your server-side/db guy, and again - this is a possible flow similar to chat message caching, and probably could be greatly tweaked into efficiency when coding.
Hope this Helps.
my app retrieves VCards as Strings and puts all together to a big string via StringBuilder
public String getVcardStrings() throws Exception {
Cursor cursor = context.getContentResolver().query(ContactsContract.CommonDataKinds.Phone.CONTENT_URI, null, null, null, null);
StringBuilder builder = new StringBuilder();
if (cursor.moveToFirst()) ;
do {
String s = getVCardStringFromContact(cursor);
L.d("VCARD", s);
builder.append(s);
} while (cursor.moveToNext());
cursor.close();
L.d("VCARD", "Output: \n" + builder.toString());
return builder.toString();
}
The output of each getVCardAsString() call is correct. It contains all data. but calling builder.toString() onlly returns the vcard data of 1,5 Contacts, which is about 4kb of text data and the rest is simply missing!
There is no exception and I´ve read that StringBuilders capacity theoretically reaches up to 4GB. Now my question is, what is happening here?
I'm pretty sure this is a false positive:
In other words the data is there but is being truncated by logcat.
This observation would seem to be validated by this define:
#define LOGGER_ENTRY_MAX_LEN (4*1024)
Which is bang-on 4KB and relates directly to what you are experiencing.
Write your output to a file to check if its actually there.
I am creating local database I want to send all data sent to web service.
For example product name one column. Lots of product name is there. I want to send it.
& Product name = briyani,egg,rice
I got all details from database below i have mention code:
public String fetchMyRowid(String column_name)
{
String query = "select "+column_name+" From " + TABLErestaurant;
mCursor =db.rawQuery(query, null);
StringBuffer buf = new StringBuffer();
if (mCursor.moveToNext()) {
buf.append(mCursor.getString(0));
String str = buf.toString();
System.out.println("**************"+str);
}
return buf.toString();
}
}
return buf.toString();
}
In class :
HashMap<String, String> paramsvalue = new HashMap<String, String>(); paramsvalue.put("product_name", dataBase.fetchMyRowid(DatabaseHelper.columnproductname));
But I have some issue. I got only one product name. I need all product name. Can any one suggest solution for this.
wel come to stackoveflow,
Please check below link in that i have first select all table's recode then i have created one another method for get all columns value by row. after that i have marge all data in to JSON. this is idea you have to do similar way...
https://stackoverflow.com/a/10600440/1168654