I am developing the following functionality:
an user picks a date and gets ListView populated by SimpleCoursorLoader (queries are executed in the background).
User frequently choices adjacent dates and there might be a lot of duplicate queries.
I tested the application and discovered that in case of high frequency requests - it runs very slow.
In order to speedup my application I decided to implement cache where results of queries will be stored. Key - date and value-?
Is it worth doing and what techniques could you advice?
1) Yes, it's really worth doing since DB access is relatively slow (even with such a great thing like SQLite)
2) Considering what I've got from your post I'd suggest using LongSparseArray: key will be date from database (long), stored value - your cached data object (Bundle etc). The reasons are it's:
naturally sorted
sort order is maintained on changes
fast
memory efficient
3) When you need to load overlapping/adjacent interval you have to check bounds and load only absent part
4) If a situation is possible when you cache non-adjacent intervals - you need to manage loaded intervals bounds as well. But if you do it only for list scroll purposes you may omit this (if you don't stop loading data on fling gesture)
About my experience: I've got about 3 times payoff using caching. But actual results depends on database scheme etc. You may get even more
I found MatrixCursor useful for the purpose of caching. I keep HashMap.
Logic: if no request has been done - issue it, get Cursor, convert it to MatrixCursor and write to cache.
Here is the snippet for convertion:
private MatrixCursor cursorToMatrixCursor(Cursor c) {
MatrixCursor result = new MatrixCursor(c.getColumnNames());
if (c.moveToFirst()) {
do {
ArrayList<String> columnValues = new ArrayList<>();
final int nOfColumns = c.getColumnCount();
for(int col = 0; col < nOfColumns; ++col)
columnValues.add(c.getString(col));
result.addRow(columnValues);
} while (c.moveToNext());
}
return result;
}
Related
I have one Android project where I need to query nearby items & these items should be sorted by time.
Basically, I need docs that are in 100KM. Sorted by time (Field).
So I have checked Firestore docs for this & I got solution (Add geoHash to docs & then query them by geoHasBounds) But there is an issue what if there are 1k docs in 100km then it will load all which is not good, so how can I limit those different queries & gets only 25-30 docs then next 25-30 docs ??
In short, this is what I need-
How can I query the latest 25 docs in 100KM radius & when the user scroll down the next 25 docs?
this is my code of query-
List<GeoQueryBounds> bounds = GeoFireUtils.getGeoHashQueryBounds(center, radiusInM);
final List<Task<QuerySnapshot>> tasks = new ArrayList<>();
for (GeoQueryBounds b : bounds) {
Query newQuery = itemQuery.orderBy("geoHash").startAt(b.startHash).endAt(b.endHash);
tasks.add(newQuery.get());
}
// Collect all the query results together into a single list
Tasks.whenAllComplete(tasks).........
What you are looking for is called pagination. I have answered a question here on Stackoverflow, where I have explained a recommended way in which you can paginate queries by combining query cursors with the "limit() method". I also recommend you take a look at this video for a better understanding.
If you are willing to try Paging 3 library, please see below an article that will help you achieve that.
How to paginate Firestore using Paging 3 on Android?
Edit:
The Tasks.whenAllComplete() method:
Returns a Task with a list of Tasks that completes successfully when all of the specified Tasks complete.
So you can then simply convert each object to a type of object that you need and paginate that list accordingly. Unfortunately, this implies getting all the objects in the first place. Otherwise, you can divide those queries into separate queries and treat them accordingly, by using separate paginantion.
I have a simple firebase database: /rides is a list of simple objects like this
{
car: "Toyota"
minutes: 15
}
and I need to display sum of minutes of all the rides. The obvious solution is to load all the rides and calculate the sum. But if I have several hundreds of rides this is very slow, up to several seconds.
So it seems I have to maintain a separate field /totalMinutesin the database for this. But thus I will have to manually update /totalMinutes every time I add/remove/change a ride. Anyway this is not a big deal of work.
But what if I need to calculate total minutes only for a subset of rides? For instance only for "Toyota" cars or "Ford" cars? Manual maintaining /totalMinutesFord, /totalMinutesToyota now doesn't seem so easy.
So what is the correct way to maintain such dynamic values in firebase?
Firebase has no way to get automatically calculate values based on the data in your database.
So your two options are:
calculate the value whenever you update the data
retrieve all the data and calculate the value on the client
You already (wisely) decided that retrieving all data is not a good idea. Your users will be grateful for that.
So that leaves calculating the derived values whenever you update the data of a ride. I'm not sure why doing that for multiple values would be more difficult than doing it for a single value. It may be more code, but it's pretty much the same code:
var ride = { car: "Toyota", minutes: 15 };
ref = new Firebase('https://yours.firebaseio.com/');
ref.child('rides').push(ride);
ref.child('totalMinutes').transaction(function(current_value) {
return (current_value || 0) + ride.minutes;
});
ref.child('totalMinutes'+ride.car).transaction(function(current_value) {
return (current_value || 0) + ride.minutes;
})
I am pulling a large amount of json from a restful server. I use the GSON library from google to traverse this json and it works great. Now I want to save all of the json objects in my sqlite db, however I want to make use of a transaction to add all of them at once. This is difficult if you dont have all the objects ready in one datastructe. Since I am traversing the json one object at a time, I guess I would have to store that in a data structure such as an arraylist or hashmap and then afterwards use a database transaction to do the inserts fast. However... Storing a large amount of data aka 200 000 json objects into a structure in memory can take up a lot of memory and wil probably run out as well. What would be the best way to get all of that json objects into my sqlite db and at the same time not use up a lot of menory in otherwords storing and inserting in a way that allows for a lot of recycling.
If you want to add a large amount of data at an unique moment : it will take a lot of memory anyway. 200 000 large JSON objects take a certain amount of memory and you will not be able to change it.
You can keep this behavior, but I think it's not a great solution because you create a huge memory consumption on both Android device and server. It will be better if you receive data part by part and adding them this way : but you need to have control on the server code.
If you are absolutely forced to keep this behavior, maybe you should receive all the data at the same time, parse them on a huge JSON object, then make multiple transactions. Check if every transaction was executed correctly and put back your database in a good state if not. It's a really bad way to do it, IMHO... but I don't know all your constraints.
To finish : avoid receiving a large amount of data at only one time. It will be better to make multiple requests to get partial data set. It will make your app less network dependant : if you loose the network for 2 seconds, maybe only one request will fail. So you will have to retry only one request and received again a small part of data. With only one huge request : if you loose the network, you will have to retry the entire request...
I know this is not the best implementation of handling large json input in Android, but it certainly works great.
So my solution is pretty simple:
While parsing the JSON code, make use of prepared statements and a db transaction to execute the inserts.
So in short, as soon as a JSON object is parsed, take that info, insert it into the db using the prepared statement and the db transaction, then move on to the next object.
I have just pulled 210 000 rows with 23 fields each (thats 4.6 million values) from a remote server via JSON and GSON and at the same time inserting all those values into my SQLite3 db on the device, in less than 5 minutes and without wasting any memory. Each iteration of parsing/inserting uses the same amount of memory and is cleaned on the next iteration.
So yeah, there's the solution. Obviously, this is not the best solution for commercial applications or tables with 1000 000 + records, but it works great for my situation.
Have you ever tried to add a lot of data (and I really mean a lot, in my case 2600 rows of data) into the Android-internal database (SQLite)?
If so, you propably went the same road as I did.
I tried a normal InsertStatement which was way to slow (10 sec. in my emulator). Then I tried PreparedStatements. The time was better but still unacceptable. (6 sec.). After some frustrating hours of writing code and then throwing it away, I finally found a good solution.
The Android-OS provide the InsertHelper as a fast way to do bulk inserts.
To give you an overview of the performance (measured with an emulator on a crap computer, trying to insert 2600 rows of data):
Insert-Statement
10 seconds
Prepared-Statements
6 seconds
InsertHelper
320 ms
You can speed up the insertion even more with temporarily disable thread locks. This will gain about 30 % more performance. However it’s important to be sure that only one thread per time is using the database while inserting data due to it’s not threadsafe anymore.
public void fillDatabase(HashMap<String, int[]> localData){
//The InsertHelper needs to have the db instance + the name of the table where you want to add the data
InsertHelper ih = new InsertHelper(this.db, TABLE_NAME);
Iterator<Entry<String, int[]>> it = localData.entrySet().iterator();
final int firstExampleColumn = ih.getColumnIndex("firstExampleColumn");
final int secondExampleColumn = ih.getColumnIndex("secondExampleColumn");
final int thirdExampleColumn = ih.getColumnIndex("thirdExampleColumn");
try{
this.db.setLockingEnabled(false);
while(it.hasNext()){
Entry<String, int[]> entry = it.next();
int[] values = entry.getValue();
ih.prepareForInsert();
ih.bind(firstExampleColumn, entry.getKey());
ih.bind(secondExampleColumn, values[0]);
ih.bind(thirdExampleColumn, values[1]);
ih.execute();
}
}catch(Exception e){e.printStackTrace();}
finally{
if(ih!=null)
ih.close();
this.db.setLockingEnabled(true);
}
}
My application records user movement with Geofence boundaries, if the user exits the Geofence, alerts are appropriately escalated. These alert are counted and displayed in a summary at the end of the activity. However I would like to create a stats page where it displays the last week or month of activities as well as the number of alerts so that I can display these in a chart. Is there anyway to do this effectively without using a database?
I had thought of writing data to a log file and reading it but curious as to if there is a better option.
You can use SharedPreferences but it will require a lot of controls, probably more then creating a database. If you insist not to use a database, put an integer to your shared preferences saving the count of your data, also that integer will become your id. Then you can store your data with a loop depending on your data.
Here is to write your data to shared preferences
SharedPreferences mSharedPrefs = getSharedPreferences("MyStoredData",
MODE_PRIVATE);
private SharedPreferences.Editor mPrefsEditor = mSharedPrefs.edit();
int count = mSharedPrefs.getInt("storedDataCount", 0);
for(int i = 0 ; i < yourCurrentDataCount ; i++) {
mPrefsEditor.putInt("data" + count, yourData.get(i));
count++;
}
mPrefsEditor.putInt("storedDataCount", count);
And to get your data,
int count = mSharedPrefs.getInt("storedDataCount", 0);
for(int i = 0 ; i < count ; i++) {
yourData.add(mSharedPrefs.getString("data" + i, "defaultData"));
count++;
}
Edit:
I should have added some explaining. The idea is to save the count of your data to generate an id, and save the tag according to it. This code will work like this, lets say you have 5 strings. Since you don't have a MyStoredData xml, it will get created. Then since you don't have the "storedDataCount" tag, you will get 0 as a count. Your loop will iterate 5 times and in each iteration, you will add a tag to your xml like "<.data0>your first data<./data0><.data1>your second data <./data1>... After your loop is done, you will modify your storedDataCount and it will become <.storedDataCount>5<./ storedDataCount>. And the next time you use your app, your count will start from 5 so your tag will start from <.data5>. For reading, you will iterate through tags by checking "data0", "data1" and so on.
You can use java serialization if you dont want to use database.
You can also use XML/JSON for storing data.
I support already mentioned favoritism towards using a DB for this task. Nevertheless, if I were to do it via FS, I would use a transactional async library like Square's tape is.
In your case I would keep the data during a session in JSON object (structure) and persist it (in onPause()) and restore it (in onRestore()) with tape's GSON Object Converter.
Should be easy out of the box, I believe.
Tape website: http://square.github.io/tape/
Alternatively to manually persisting a file or using a 3rd party library like tape, you could always (de)serialize your JSON to SharedPreferences.
What is the best way to keep a List<String[]> while my application runs? I am having problems with my approach. It most of the time gives me an OutOfMemory Error since the list is too big.
The List<String[]> is the result of parsing a csv file that I have downloaded online. What I do is parse the csv in an activity then save its result in a static class member like:
String url = "http://xxx/pdf/pms/pms_test.csv";
try {
InputStream input = new URL(url).openStream();
CSVReader reader = new CSVReader(new InputStreamReader(input));
SchedController.sched = reader.readAll();
input.close();
}
...then access ClassName.sched on different activities.
I am doing this so that the parsed data will be available in every activity... And I don't have to parse again. What can I do to improve it?
I think you can have 2 approaches.
You save the file and parse it in a lazy loading way
You create a database and save your data.
I suggest you to create a database, this is not difficult and let you to manage well your data. You can do easily lazyLoading with cursor, or use a ORM (ORMLite / Greendao)... I think this is the best way and the fastest to load your data.
Hope this will help you.
Just to add to Paresh's comment.
I can suggest something similar I used in my app.
For example I need to display list of Items in a store. (Each item will have ID, name and cost).
To achieve this I first make a request to server to get number of items, say itemCount.
Now I set a limit to number of items displayed in a page say 100.
If the itemCount is greater than 100, I display an alert saying only 100 items will be displayed and a next button can be added to download next set of items.
Or if it is a search you can ask user to go back and refine the search.
If itemCount is less than 100 then you will not have any issues
This way Paging can be implemented to avoid OutOfMemory issues