I'm using the external storage for storing events in a database while they are waiting to be sent to the server.
I'm seeing really bad performance when inserting records.
I know the external memory can be slow but I wanted to see some number so I wrote a small app which tests it.
Here is the code:
public static final int INSERTS = 100;
File dbFile = new File(Environment.getExternalStorageDirectory(), "test.sqlite3");
// File dbFile = new File(getFilesDir(), "test.sqlite3");
dbFile.delete();
SQLiteDatabase db = SQLiteDatabase.openOrCreateDatabase(dbFile, null);
db.execSQL("CREATE TABLE events (_id integer primary key autoincrement, event_type TEXT NOT NULL, timestamp BIGINT, data TEXT);");
db.execSQL("CREATE INDEX mainIndex ON events (event_type, timestamp ASC);");
InsertHelper helper = new InsertHelper(db, "events");
final int eventTypeCol = helper.getColumnIndex("event_type");
final int timestampCol = helper.getColumnIndex("timestamp");
final int dataCol = helper.getColumnIndex("data");
long start = System.currentTimeMillis();
String eventType = "foo", data = "bar";
long timestamp = 4711;
for(int i = 0; i < INSERTS; ++i) {
helper.prepareForInsert();
helper.bind(eventTypeCol, eventType);
helper.bind(timestampCol, timestamp);
helper.bind(dataCol, data);
helper.execute();
}
long end = System.currentTimeMillis();
Log.i("Test", String.format("InsertHelper, Speed: %d ms, Records per second: %.2f", (int)(end-start), 1000*(double)INSERTS/(double)(end-start)));
db.close();
dbFile.delete();
db = SQLiteDatabase.openOrCreateDatabase(dbFile, null);
db.execSQL("CREATE TABLE events (_id integer primary key autoincrement, event_type TEXT NOT NULL, timestamp BIGINT, data TEXT);");
db.execSQL("CREATE INDEX mainIndex ON events (event_type, timestamp ASC);");
start = System.currentTimeMillis();
ContentValues cv = new ContentValues();
for(int i = 0; i < INSERTS; ++i) {
cv.put("event_type", eventType);
cv.put("timestamp", timestamp);
cv.put("data", data);
db.insert("events", null, cv);
}
end = System.currentTimeMillis();
Log.i("Test", String.format("Normal, Speed: %d ms, Records per second: %.2f", end-start, 1000*(double)INSERTS/(double)(end-start)));
db.close();
dbFile.delete();
The database is exactly as the one my real app is using, I tried removing the index but it made no difference.
Here are the results:
Nexus One, Internal memory
Method | Records | Time (ms) | Records per second
-------------+---------+-----------+--------------------
Normal | 100 | 2072 | 48.26
InsertHelper | 100 | 1662 | 60.17
Nexus One, External memory:
Method | Records | Time (ms) | Records per second
-------------+---------+-----------+--------------------
Normal | 100 | 7390 | 13.53
InsertHelper | 100 | 7152 | 13.98
Emulator, Internal memory:
Method | Records | Time (ms) | Records per second
-------------+---------+-----------+--------------------
Normal | 100 | 1803 | 55.46
InsertHelper | 100 | 3075 | 32.52
Emulator, External memory:
Method | Records | Time (ms) | Records per second
-------------+---------+-----------+--------------------
Normal | 100 | 5742 | 17.42
InsertHelper | 100 | 7164 | 13.96
As you can see the emulator cannot be trusted, InsertHelper should be faster if anything.
This is, of course, to be expected, the test was mostly done out of curiosity.
What have me concerned however is the bad performance on my phone when using external memory, have I missed some crucial aspect of SQLiteDatabase or is it simply so that the SD card will be slow?
I can add that in my real app I've disabled locking and it makes little difference.
CommonsWare is correct in his comment. Something that makes a big difference for db performance is using transactions. Wrap your insert loop in a transaction. I'm not 100% sure if it would work with the InsertHelper but you can try replacing your for loop with this:
db.beginTransaction();
try {
for(int i = 0; i < INSERTS; ++i) {
helper.prepareForInsert();
helper.bind(eventTypeCol, eventType);
helper.bind(timestampCol, timestamp);
helper.bind(dataCol, data);
helper.execute();
}
db.setTransactionSuccessful();
} finally {
db.endTransaction();
}
I have some db performance issues so I used your code to measure the inserts per second on my system. But I also added wrapping in {begin,end}Transaction().
In the emulator. I got:
InsertHelper-Internal-Trans, Speed: 67 ms, Records per second: 1492.54
InsertHelper-External-Trans, Speed: 70 ms, Records per second: 1428.57
Normal-Internal-Trans, Speed: 148 ms, Records per second: 675.68
Normal-External-Trans, Speed: 152 ms, Records per second: 657.89
InsertHelper-Internal-NoTrans, Speed: 514 ms, Records per second: 194.55
Normal-Internal-NoTrans, Speed: 519 ms, Records per second: 192.68
InsertHelper-External-NoTrans, Speed: 590 ms, Records per second: 169.49
Normal-External-NoTrans, Speed: 618 ms, Records per second: 161.81
And on a Samsung Galaxy Note:
InsertHelper-External-Trans, Speed: 52 ms, Records per second: 1923.08
InsertHelper-Internal-Trans, Speed: 52 ms, Records per second: 1923.08
Normal-External-Trans, Speed: 77 ms, Records per second: 1298.70
Normal-Internal-Trans, Speed: 121 ms, Records per second: 826.45
Normal-External-NoTrans, Speed: 4562 ms, Records per second: 21.92
Normal-Internal-NoTrans, Speed: 4855 ms, Records per second: 20.60
InsertHelper-External-NoTrans, Speed: 5997 ms, Records per second: 16.68
InsertHelper-Internal-NoTrans, Speed: 8361 ms, Records per second: 11.96
Related
I have payment analysis app with the following simplified structure:
Payment
CREATE TABLE payments (
payment_Id TEXT NOT NULL,
payment_type INTEGER NOT NULL,
payment_name TEXT NOT NULL,
payment_category INTEGER NOT NULL,
payment_date TEXT NOT NULL,
payment_cost TEXT NOT NULL, PRIMARY KEY(payment_Id))
Category
CREATE TABLE categories (
category_Id INTEGER NOT NULL,
idRoot TEXT NOT NULL,
category_Name TEXT NOT NULL,
category_Type INTEGER NOT NULL, PRIMARY KEY(category_Id))
Foreign key: categories.category_Id = payment_category
Categories are structured in a tree hierarchy:
IdRoot
category_Id
Name
-
0
Unknown
0.
1
Expenses
0.1.
2
House
0.1.
3
Transport
0.1.
4
Travel
0.1.2.
5
Internet
0.1.2.
6
Groceries
0.1.2.
7
Water
0.1.3.
8
Car
0.1.3.
9
Public Transport
0.1.3.8.
10
Gasoline
0.1.3.8.
11
Tolls
...
...
....
Payments can be assigned to any category (except category_Id = 0).
I want to know if it is possible using just SQLite (compatible with Android is appreciated) to get all the children categories of a parent category and the sum of the payment_cost assign to every category in that level and its children.
Example: given category Transport(category_Id: 3), sqlite will return Car and Public Transport and their sums but the sum of Car will also include the payment_cost assign to Gasoline and Tolls something like this
Input: category_Id: 3
Data:
| payment_name | payment_cost| payment_category |
| ------------ | ---------- | ----------------- |
| Gas Station | 50,3 | 10 |
| Metro | 2,4 | 9 |
| Tool NY - Phil | 21 | 11 |
| Car cleaning | 11 | 8 |
Result
category_name
Total
Car
82,3 (50,3 + 21 + 11)
Public Transport
2,4
Thank you in advance for your time and knowledge
You can use like to get the matching roots:
select sum(payment_cost)
from payments p join
categories c
on p.payment_category = c.category_id cross join
categories c3
on c3.category_id = 3
where p.idroot like c3.idroot || '%'
The app im working with is getting data from a .csv (20k-30k records) from a server and it needs to persist the data into an SQLiteDatabase.
It works but some records are missing and appeared that it have been skipped.
I/Choreographer( 2555): Skipped 46 frames! The application may be doing too much work on its main thread.
I know that this error says that the memory consumption is very high due to heavy load. Is there a more efficient way in persisting data in SQLiteDatabase rather than the classic accessing of CSV and processing it from there?
Code for writing in DB
String sql = "INSERT INTO " + tableName
+ " VALUES (?,?,?,?,?,?);";
SQLiteDatabase db = openHelper.getWritableDatabase();
SQLiteStatement statement = db.compileStatement(sql);
try {
db.beginTransaction();
String[] sa = null;
for (final String csvline : arrCSV) {
statement.clearBindings();
sa = csvline.split(",");
if(sa.length==6){
statement.bindString(1, sa[0]);
statement.bindString(2, sa[1]);
statement.bindString(3, sa[2]);
statement.bindString(4, sa[3]);
statement.bindString(5, sa[4]);
statement.bindString(6, sa[5]);
}
statement.execute();
}
db.setTransactionSuccessful();
Log.d("Transaction", "Successful");
}catch(Exception e){
e.printStackTrace();
}finally {
statement.releaseReference();
statement.close();
db.endTransaction();
db.releaseMemory();
}
UPDATE
The missing records were not loaded in the Collection.
Is the skipping of frames the culprit here?
The loading in the collection is just a simple parsing of a csv file and
non replicable at times so Im assuming it is due to the skipping of frames.
I believe the issue is not linked to skipping frames and < 100 frames is considered a small/insignificant number. At least according to The application may be doing too much work on its main thread.
I frequently see it and has never been the cause of any issues. I've even seen it basically doing nothing other than returning a result from an activity to an activity that just starts the second activity.
As you have commented, the number of elements that result from the split is on occasion not 6. The issue is likely that the insert is not happening on such an occasion, perhaps due to constraints (without seeing how the columns are defined only guesses could be made).
However, you appear to consider that each line in csvline should be split into 6 elements. You should thus investigate as to why not?
To investigate I'd suggest getting details of the original data before the split and the resultant data after the split whenever the number of elements created by the split is not 6. e.g. by changing :-
sa = csvline.split(",");
if(sa.length==6){
statement.bindString(1, sa[0]);
statement.bindString(2, sa[1]);
statement.bindString(3, sa[2]);
statement.bindString(4, sa[3]);
statement.bindString(5, sa[4]);
statement.bindString(6, sa[5]);
}
statement.execute();
to
sa = csvline.split(",");
if(sa.length==6){
statement.bindString(1, sa[0]);
statement.bindString(2, sa[1]);
statement.bindString(3, sa[2]);
statement.bindString(4, sa[3]);
statement.bindString(5, sa[4]);
statement.bindString(6, sa[5]);
} else {
Log.d("NOT6SPLIT","CSVLINE WAS ===>" + csvline + "<===");
Log.d("NOT6SPLIT","CSVLINE WAS SPLIT INTO " + Integer.toString(sa.length) + " ELEMENTS :-");
for(String s: sa) {
Log.d("NOT6SPLIT","\tElement Value ===>" + s + "<===");
}
}
statement.execute();
Changing statement.execute() to :-
if (statement.executeInsert() < 1) {
Log.d("INSERTFAIL","Couldn't insert where CSVLINE was ===>" + csvline + "<===");
}
May also assist ('executeInsert' returns the rowid of the inserted record, else -1, not sure of the consequences of a table defined with WITHOUT ROWID).
It wouldn't surprise me at all if the issue boils down to your data containing characters that split considers special or metacharaceters:-
there are 12
characters with
special meanings:
the backslash \,
the caret ^,
the dollar sign $,
the period or dot .,
the vertical bar or pipe symbol |,
the question mark ?,
the asterisk or star *,
the plus sign +,
the opening parenthesis (,
the closing parenthesis ),
the opening square bracket [,
and the opening curly brace {,
These special characters are often called "metacharacters". Most
of them are errors when used alone.
My task is to export data to CSV file. I am trying to generate a query before exporting data to a CSV File. Each student has a unique ID. As for now, I'm able to export the data in different tables separately and its working well.
Here is my current schema :
1) Student Table
studID | name | teacher | activity
SQ202 Mich Lisa Hearing Test
FG91A Lim Hanson Behavioural Test
2) Session Table (Each student can have more than one session - not fixed)
studID | startTime | endTime | sessionNo | status
SQ202 10:00 10:40 1 Completed
SQ202 13:00 13:50 2 Completed
FG91A 14:20 15:00 1 Completed
3) Interval Table (Each session can have more than four intervals - not fixed)
studID |sessionNum |intervalNo | score
SQ202 1 1 10/10
SQ202 1 2 7/10
.
.
.
This is how I exported data from a specific table :
File exportDir = new File(Environment.getExternalStorageDirectory(), "");
if (!exportDir.exists()) {
exportDir.mkdirs();
}
file = new File(exportDir, “ChildObservation" + " (D)" + date + " (T)" + time +".csv");
try {
file.createNewFile();
CSVWriter csvWrite = new CSVWriter(new FileWriter(file));
// sqlite core query
SQLiteDatabase db = myNewGradingDB.getReadableDatabase();
Cursor curChild = db.rawQuery("SELECT * FROM Student ORDER BY name", null);
csvWrite.writeNext(“Student ID”, “Name”, “Teacher”, “Activity");
while (curChild.moveToNext()) {
String arrStr[] = {
curChild.getString(0), curChild.getString(1),
curChild.getString(2), curChild.getString(3),
};
csvWrite.writeNext(arrStr);
}
csvWrite.writeNext();
csvWrite.close();
return true;
}
catch (IOException e) {
Log.e(“Child", e.getMessage(), e);
return false;
}
However, my goal is to retrieve and display all the data tied to the student in a single row. Each row in the exported Excel sheet will contain all data from the Student table & Session table as well as from the respective interval number and scores from the Interval table. I have tried using inner joins and different kind of queries but there are still some duplicated values in those rows. Each cell must only contain one value, so in this case I won't be able to concatenate the columns in my query.
Here is what I want to achieve :
studID | name | teacher | activity | startTime | endTime | sessionNo | status | intervalNo | score | intervalNo | score ... (repeat intervalNo and score)
SQ202 Mich Lisa Hearing Test 10:00 10:40 1 Completed 1 10/10 2 7/10 ...
FG91A Lim Hanson Behavioural Test 12:30 1:40 2 Completed 1 8/10 2 6/10 ...
At this stage, its impossible for me to alter the database design. Perhaps there are other ways to resolve this, whether by SQL or another. Any help or suggestions will be greatly appreciated. Thank you.
if you want a sql only solution then
the trick to solve you problem is to join the interval table several times using an alias
select
st.studID, st.name,
se.startTime,
i1.intervalNo as intervalNo1, i1.score as score1,
i2.intervalNo as intervalNo2, i2.score as score2,
i3.intervalNo as intervalNo3, i3.score as score3,
i4.intervalNo as intervalNo4, i4.score as score4
from Student as st
left join Session as se on st.studID = se.studID
left join Interval as i1 on st.studID = i1.studID and i1.sessionNo=1
left join Interval as i2 on st.studID = i2.studID and i2.sessionNo=2
left join Interval as i3 on st.studID = i3.studID and i3.sessionNo=3
left join Interval as i4 on st.studID = i4.studID and i4.sessionNo=4
but this only works if you know in advance how many interval-s there are
I have a question.
Have a table like this:
(table name: times)
+-------+-----------+---------+
| block | startTime | endTime |
+-------+-----------+---------+
| 1 | 08:00 | 10:00 |
+-------+-----------+---------+
when I use rawquery like this:
String ctime = "'09:45'";
cursor = sdb.rawQuery(String.format("select block from times where startTime < time(%s) and endTime> time(%s)", ctime,ctime), null);
I Can get the correct data "1".
But when I use query like this:
cursor = sdb.query("times", new String[]{"block"}, "startTime <= time(?) and EndTime >= time(?)", new String[]{ctime,ctime}, null, null, null);
I can't get the correct data.
The Cursor count is 0.
Why??
A valid time string must not contain quotes.
The quotes in your ctime variable are correct when you are constructing the SQL statement by hand, but they are used to delimit the SQL string and are not part of the value of the SQL string.
When you are using parameters, you do not need to put quotes around strings (unless you actually want them to be part of the string's value).
In my Database, I have the follwoing data stored:
_id | NumericalEquivalents | ValTo | valFrom | Section
------------------------------------------------------
1 1.00 100 99 a
2 1.25 98 96 a
3 1.75 95 93 a
.....
.....
10 5.00 74 0 a
Now in my app, the grade will be computed programatically, and I have the code below to get the Numerical Equivalent of the computed grade:
public float getNumEqui(String sect, float grade)
{
String q = "SELECT * FROM NumericalEquivalents where Section='"+sect+"' and valTo>="+grade+" and valFrom<="+grade;
Cursor cursor = mDb.rawQuery(q, null);
if(cursor.moveToFirst())
{
float reset= cursor.getFloat(cursor.getColumnIndex("NumericalEquivalent"));
return reset;
}
//return sometotal;
return -1;
}
My problem now is that it always return 5.00, I was wondering if something was wrong with the query that I used. Any help will be appreciated.
Use BETWEEN :
SELECT *
FROM NumericalEquivalents
where Section = 'a' and #grade between ValFrom and ValTo
Demo here