I have been reading through parse forums and I gather that as of 2 years ago there was no equivalent of SQL Group By.
I'm wondering if there have been any developments on this? I have thousands of records and I need to pull down all records in descending order of a value rating and then group them by name.
If this isn't available at present perhaps someone could suggest something I could work on instead.
Regards.
See https://parse.com/docs/android_guide#queries-basic , you can have secondary sorting key, which would be equivalent of "group by".
query.orderByDescending("rating");
query.addAscendingOrder("name");
could do the trick for you .
You have two options generally:
Refactor your schema to accommodate the model
Run a job to aggregate your data into "snapshots"
For option 1 it means creating "afterSave" handlers that update counters etc as data changes (expensive writes, cheap reads), here's a sample in the documentation that saves a "comments" count for a Post object:
https://parse.com/docs/cloud_code_guide#functions-aftersave
You would need to run a once-off job to set your initial counts since you already have data there (see option 2).
For option 2 you can learn more about background jobs by reading up here:
https://parse.com/docs/cloud_code_guide#jobs
Your job is allowed to run for up to 15 minutes (e.g. if it is processing insane amounts of data, or doing many things in sequence).
Related
I see that Firebase website has a lot of documentation to help us optimize the usage of resources, however, I have not found a detailed example of the resources used.
This could be useful to me to understand how to build my applications and to better choose the strategy in terms of performance and cost.
TAKING ANDROID AS EXAMPLE
I understand that when i do a:
query.addListenerForSingleValueEvent(...);
all the reference is "queried" to the database so that is a single query but takes down all the object.
if i do:
query.addValueEventListener(...);
the connection is kept open, but will it keep making connections on time intervals?
Or maybe is considered like a single connection in terms of billing?
And after a change on the database, will it query all the object down again?
In general how much is heavier and expensive to make a single request vs using the realtime-db feature of listening to a reference?
Maybe there is a section in the docs that explain this but I didn't found it.
query.addListenerForSingleValueEvent(...);
all the reference queried" to the database so that is a single
query but takes down all the object.
It will listen once to the objects within inside the child you are querying, not all of the objects inside the database.
query.addValueEventListener(...);
the connection is kept open, but will it keep making connections on
time intervals?
It has no intervals, instead it listens whenever a change is made into your database, lets say you change certain value from your database and that will trigger your addValueEventListener. This will only consume resources when some value changes into your database, so the usage will be a variant with your database usage, instead , addListenerForSingleValue will fire just once to query your data and we can assure that it will consume less network resources than a listener that is always listen to some changes to bring into your app
Check this usefull link : https://www.firebase.com/docs/java-api/javadoc/com/firebase/client/ValueEventListener.html
I've got a table with about 7 million rows in it. I'm inserting on average about one row every second into the database. When I do this, I am noticing that it is taking an incredibly long time (as much as 15 seconds) to run a simple SELECT against the database, e.g. something like:
SELECT * FROM table WHERE rowid > 7100000
This select often returns no rows of data as sometimes no data has been inserted in this particular table. It is often happening even when the table I'm writing to isn't even actually inserting rows into the table I am reading.
The idea is that there are two separate processes, one is adding data, the other is trying to get all new data that has not yet been read. But the read side is connected to a UI and any noticable lag is intolerable, much less 15 seconds. This is being run under Android and the the UI thread doesn't like being blocked for that long either and it is wreaking havoc.
My initial thought was maybe the insert is requiring an update to the indicies as originally I had the index on a different field (a time field). This seems at least partially confirmed because if I use a database with only a few rows each select completes in a few milliseconds. But when I re-created the table to only have the rowid as primary key it actually got slower. I would expect inserting a new row at the end would always result in very fast reads when just comparing on the rowid as primary key.
I have tried enabling write ahead logging, but it appears that SQLCipher doesn't support this, at least not directly, as it doesn't adhere to the lastest API for android.database.sqlite.SQLiteDatabase. Even using "PRAGMA journal_mode = WAL" in the postKey hook hasn't made any difference.
What's going on here? How can I speed up my selects?
Update: I tried getting rid of sqlcipher and just using plain sqlite to see if that was a factor. I used sqlcipher_export to export to a plaintext database, and then used the default android.database.sqlite.SQLCipher. The delay time dropped from 10-20s to 1.8-2.8s. I then removed write-ahead and it dropped further to 1.3-2.7s. So the issue is still noticably there, although it did get a lot better.
SQLite is ultimately file-based, and there is no portable mechanism to communicate to another process which part of a file has changed. So when one process has written something, all other processes must drop their caches when they access the database file the next time.
If possible, modify your architecture that both parts of the code are in the same process and share the same database connection. (With multiple threads, this requires locking, but SQLite has not much concurrency anyway.)
Alternatively, write the new data into a separate database, and let the UI app move it to its own database.
I don't know why SQLCipher is so much slower (it's unlikely to be the CPU overhead of the decryption).
I think its well known that in list of worst-documented topics, SyncAdapter shines bright like a diamond !
acording to http://udinic.wordpress.com/2013/07/24/write-your-own-android-sync-adapter/ SyncAdapter brings 4 main benefits :
A) Battery efficiency
B) Interface C) Content awareness D) Retry mechanism;
if in any case there's a need to sync an sqlite DB with remote SQL DB, and none of these benefits is needed, what other alternatives are there**?** its easy to manage a service in-between the DBs with php, I did that for Uploading part of syncing process,but for the downloading part I feel silly if I use the query filling method,cause in near future remote db might get large and larger.the only solution that comes to my mind is to write my own sync activity/service, but I dont know how to access the last update date to SQLite db/table (other than specifying a _date in every table,) to check if it is necessary to sync again ? I feel my head is between two places!
You are mixing the problem.
1- Do you really have to use sync Adapter ??? So if yes, you are gonna have a Sync call per table and no needs to save the last call date. Android will do it for you. Just setup your sync timers properly
2- other solution is to do a simple AsyncTask and do your job here. (For exemple if you have to do it only once per week)
For your date problem, the thing is if you really wants to know if you are up to date you got many solutions. On your server save the date, or increment a version and compare these when you call a sync from your device to know if you have to sync or not.
An other solution is to simply just refresh your db wherever it is updated or not(for exemple you got a small db, so no need to create an optimized system).
I faced the same problem months ago and hoped this helped you.
You might want to consider this article:
https://www.bignerdranch.com/blog/choosing-the-right-background-scheduler-in-android/
It makes it clear how syncadapter is a good choice as a result of lesser convenient options when needing to utilize the battery well and go out to the network.
I don't recommend Asyntask for theses reasons:
http://blog.danlew.net/2014/06/21/the-hidden-pitfalls-of-asynctask/
If syncadapter is really not working for you there is
android's best practices which suggests to use an IntentService and WakefulBroadcastReceiver with partial wake lock when doing long-running operations. It says "the Android framework offers several classes that help you off-load operations onto a separate thread that runs in the background. The most useful of these is IntentService."
https://developer.android.com/training/run-background-service/index.html
https://developer.android.com/training/scheduling/wakelock.html
there must be some truth to it since they wrote it.
Android Jetpack includes WorkManager which is a valid alternative to syncadapters.
Main features:
Schedule a job according to network availablity or device charging status
Backward compatiblity up to api 14
Ensures task execution, even if the app or device restarts
Intended for deferrable tasks (E.g periodically syncing application data with a server)
In alternative, something similar is Android-Job library by Evernote
I'm sort of lost on this. I have an application that is reading from a static SQLite database that has 439397 records (~32MB).
I am querying the database on a column that is indexed, but the it takes ~8-12 seconds to finish the query. The current query I am using is to do database.query(tableName, columnHeaders, "some_id=" + id) for a list of ids.
I tried doing the "WHERE some_id IN (id1, id2, id3)" approach, but that took over twice as long. I have a feeling that I might be doing it wrong.
The query is done in an AsyncTask, so I am at a lost at what other thing I could do to improve the performance.
UPDATE:
I resolved the problem by changing the behavior of the application.
You can use EXPLAIN QUERY PLAN to confirm that your index is indeed being properly used.
You can try running your query once with a COUNT(*) instead of the real column list, to see if the issue is the act of actually reading the row data off of flash storage (which is possible if there are lots of matches and lots of big columns).
You can try running your query to match on a single ID (rather than N of them), to start to try to get a handle on whether the issue is too many comparisons.
However, please bear in mind that AsyncTask does not somehow make things magically faster. It makes things magically not run on the main application thread.
Looks like you don't have an index on that field. Note that the index might need to cover several fields if you use them for filtering/sorting/grouping in your real query (can't tell in more details because I haven't seen it).
In my Android app, I need to get 50,000 database entries (text) and compare them with a value when the activity starts (in onCreate()). I am doing this with the simplest way: I get the whole table from db to a cursor. However this way is too laggy. Are there any other ways to do it more effectively ?
Edit: The app is "scrabble solver" that is why I am not using WHERE clause in my query (Take the whole data and compare it with combination of the input letters). At first I was using a big table which contains whole possible words. Now I am using 26 tables. This reduced the lag and I am making database calls on a thread - that solved a lot problems too. It is still little bit laggy but much better.
To summarize and add a bit more
Let the database do the work of searching for you, use a WHERE clause when you perform a query!
If your where clause does not operate on the primary key column or a unique column you should create an index for that column. Here is some info on how to do that: http://web.utk.edu/~jplyon/sqlite/SQLite_optimization_FAQ.html#indexes
Google says: "Use question mark parameter markers such as 'phone=?' instead of explicit values in the selection parameter, so that queries that differ only by those values will be recognized as the same for caching purposes."
Run the query analysis using EXPLAIN QUERY PLAN http://www.sqlite.org/lang_explain.html and look for any scan operations, these are much slower than search operations. Uses indexes to avoid scan operations.
Don't perform any time consuming tasks in onCreate(), always use an AsyncTask, a Handler running on a background thread or some other non-main thread.
If you need to do full text search please read: http://www.sqlite.org/fts3.html
You should never read from the database in the UI thread. Use a background thread via AsyncTask or using regular threading. This will fix the UI lag issue your having.
Making the database read faster will help with bringing the data faster to the user but it's even more important that the fetching of the data does not block the user from using the app.
Check out the Following Links:
Painless Threading in Android
YouTube: Writing Zippy Android Apps
Use a WHERE clause in your SQL rather than reading the whole thing in. Then add an index on the columns in the WHERE clause.
At least you can put index on the field you compare and use WHERE clause. If you are comparing numerics Sqlite (db engine used by Android) supports functions such as MIN and MAX. Also if you are comparing partial strings you can use LIKE. For query optimization there are many resources such as this