My goal is to effectively get a list of children (ordered* and indexed**) with the lowest number of data transfer.
* ordered: ordered by points for each user / database child
** indexed: 2 or less ranks behind/after the current user [A specific child] (further elaborated below)
My database structure is as follows:-
I basically want to get the first 3 users ordered by points (simple):-
val usersRef = FirebaseDatabase.getInstance(DB_LINK).getReference("users").orderByChild("points")
usersRef.limitToFirst(3).addValueEventListener(
object : ValueEventListener {
override fun onDataChange(snapshot: DataSnapshot) {
for (ds in snapshot.children) {
val points: String = snapshot.child("points").getValue(String::class.java)!!
val firstName: String = snapshot.child("firstName").getValue(String::class.java) ?: ""
val uid: String = snapshot.key!!
// Update View
}
}
override fun onCancelled(error: DatabaseError) {}
}
)
Then, provided that the currently logged in user isn't one of the first three, I want to get his rank (order according to points in the whole db), 2 users' before him, and 2 users' after him without querying the whole database (it's a user database that can get up to 50K unique users) because querying the whole database is a really expensive client-side task.
I checked firebase data filtering page but found nothing useful about limiting results according to a certain child.
This answer doesn't satisfy my needs, as it loops over the whole database (in my case, 50K records). I need an effective method as I need to really save these firebase bills.
Moreover, I check this answer but it didn't meet my needs because it still queries the whole database, meaning it is not effective and will be billed for each node before the current user. (Maybe he is number 40,000 in the db, so I shouldn't query the whole db each time to get his rank and get billed for 39,999 reads)
I searched for a way to somehow use booleans to filter queries but again found nothing useful. Here is my not-effective code:-
// Gets all children.
usersRef.addValueEventListener(
object : ValueEventListener {
override fun onDataChange(snapshot: DataSnapshot) {
for (ds in snapshot.children) {
val points: String = snapshot.child("points").getValue(String::class.java)!!
val firstName: String = snapshot.child("firstName").getValue(String::class.java) ?: ""
val uid: String = snapshot.key!!
// Update View only if user is `2 <= usersRank - theirRank <= -2`
}
}
override fun onCancelled(error: DatabaseError) {}
}
)
I want to achieve something like this:- (Styling already done, logic remaining)
Is there a way to achieve this? Any alternatives?
EDIT: I found out that firestore offers aggregation queries that may help in this situation. Doing more research to further narrow down the costs.
This operation is not available on a Firebase Realtime Database. A better option would be Firestore.
Why?
Well, A fire-store database can give you the count of objects in a certain query. This is a new feature added by firebase. You basically type the query you want, then add .count() before .get(); that way it'll return the count of objects only. This is called aggregation queries. Learn more about them here.
Cloud Functions - Why isn't it appropriate here?
Using a Cloud Function for aggregations avoids some of the issues with client-side transactions, but comes with a different set of limitations:
Cost - Each rating added will cause a Cloud Function invocation, which may increase your costs. For more information, see the Cloud Functions pricing page.
Latency - By offloading the aggregation work to a Cloud Function, your app will not see updated data until the Cloud Function has finished executing and the client has been notified of the new data. Depending on the speed of your Cloud Function, this could take longer than executing the transaction locally.
Write rates - this solution may not work for frequently updated aggregations because Cloud Firestore documents can only be updated at most once per second. Additionally, If a transaction reads a document that was modified outside of the transaction, it retries a finite number of times and then fails.
Combining this with other methods
Now that you're using COUNT() for this system, there is one more method to help further narrow down the costs. That is Periodic Updates.
Periodic Updates
Who would care about a live ranking of all users? You can make the leaderboard update each minute, hour, or day. For example, stack overflow's leaderboard is updated once a day!
This approach would really work for any number of players and any write rate. However, you might need to adjust the frequency though as you grow depending on your willingness to pay.
Costs Side
For each normal read, you are charged for one read. Very simple. However, for one count, you're charged for 0.001 reads (meaning 1000 counts = 1 read). For more information about costs, check this article by firebase.
Final Thoughts
To connect everything up, we shall now apply this on our problem. Firstly, we'll need to keep the first portion of the code as it is. (The portion that grabs the first 3 users), though with some changes to port it to firebase.
NOTICE: Don't forget to setup a composite index because we're ordering by multiple fields at the same time.
val top3 = HashMap<Int, HashMap<String, String>>()
Firebase.firestore.collection("users").orderBy("points", Query.Direction.DESCENDING)
.orderBy("firstName", Query.Direction.ASCENDING)
.get().addOnSuccessListener {
for ((index, doc) in it.documents.withIndex()) {
val firstName = doc.getString("firstName")!!
val points = doc.getString("points")!!
top3[index+1] = hashMapOf("first" to firstName, "points" to points, "uid" to doc.id)
}
}
More about ordering and limiting here.
Then, we'll need to implement the COUNT() feature.
Firebase.firestore.collection("users")
.whereGreaterThan("points", UserProfile.getInstance()!!.getProfile().points)
.count().get(AggregateSource.SERVER).addOnSuccessListener {
println("Your rank is: ${it.count+1}")
}
Basically what I did here was:-
Selecting the Collection
Ordered Ascending by first name so no duplicate ranks.
Count them, and pass onto the function.
The final step is just updating the hash map top3 and rank of user each hour/day/minute/...
Related
I need to track only Realm INSERT operations. Is there a listener for something like this?
I use a Realm-java for Android and work with a server that sends data in independent parts. For example, a Person or a Pet can be received via websocket in any order. For example, the server can first send me a Pet and then after a few minutes a Person. Or vice versa. I can't control it. I want to save data from the server to the database without any logic. At the same time, the listener, who reacts ONLY to the insertion, puts a link to the Pet in the Person after the recording. For example, it connects a newly received Person with a Pet already in the database, whose ownerId is equal to the inserted Person's id.
I found a solution: to track only the insert, you need to use DynamicRealm and OrderedRealmCollectionChangeListener on the search result.
val dRealm = DynamicRealm.getInstance(conf)
dRealm
.where('Person')
.findAllAsync()
.apply {
addChangeListener { results: RealmResults<DynamicRealmObject>, changeSet: OrderedCollectionChangeSet ->
if (changeSet.insertions.isNotEmpty()) {
// do what you need
}
}
}
.asFlowable()
.subscribe()
Lets say i have a list of Objects (Accounts) in my RoomDatabase. And when the application starts i request the Accounts from Server in order to Update the Accounts Table.
So i query the database, getting a
current: List<Account> which is the existing accounts in the database
From the Server we have the news: List<Account> list which is the updated Accounts
I want to find
The elements from the news that exist in the current, in order to update them
The elements from the news that don't exist in the current, in order to insert them
The elements from the current that don't exist in the news, in order to delete them
Is there any extension function that can do that Job with an elegant and fast way, except the obvious that is to do one step at a time with the already known technique? (Loop in each list every time)
You could use partition I suppose
val current = listOf("hey", "hi", "wow")
val news = listOf("hi", "hey", "sup")
fun main() {
val (retain, discard) = current.partition { it in news }
val insert = news.filterNot { it in current }
println("Retain: $retain\nDiscard: $discard\nInsert: $insert")
}
>> Retain: [hey, hi]
>> Discard: [wow]
>> Insert: [sup]
"Elegant and fast" is subjective - is this less elegant?
val retain = current.filter { it in news }
val discard = current.filter { it !in news }
val insert = news.filter { it !in current }
it's arguably neater and easier to read - three conditions, described the exact same way.
Faster? Well that depends on the size of your lists, and you'd have to benchmark the solutions to see how much difference it makes (and if it's worth prioritising it over readability). Converting the lists to sets would give you a performance boost for the in checks, but the conversion itself might be slower than the gains in lookup speed
According on your answer - the most efficient way in your case is just drop your previous accounts from db and write new.
Your backend data always a priority.
Instead of many data manipulations(get from db, compare lists, insert and delete to new list), do just
delete old accounts with delete method in your room dao.
insert new accounts from your backend using insert method in dao.
Extension for this with coroutines:
suspend fun List<Account>.insertActual() = withContext(IO) {
yourDao.deleteAccounts()
yourDao.insertAccounts(this)
}
So suppose I am building an app that lets users manage trips. When a trip is created , any number of users can be added in it. I want this data to be stored in a single place and then for each user be able to retrieve the trips that that person is included in. My data looks like this:
-trips
-<trip-id>
-title
-budget
-etc
-people
-<person-id>
-name
-uid
-<person-id>
-name
-uid
-<trip-id>
-
-
-
This trip will contain all the trips created by all the users. To show any person their trips, I want to retrieve only the lists that person exists in.
This is what I've tried to do including other similar approaches.
rootReference.child("trips").orderByChild("uid").equalTo(FirebaseAuth.instance.currentUser.uid).addValueEventListener(object:ValueEventListener){
override fun onDataChange(snapshot: DataSnapshot) {
//this should only return the trips that current user exists in.
}
}
I have checked the documentation for searching and filtering on firebase but there is nothing that show filtering based nested keys. One particular example is this. I understand it perfectly. If for example I try to filter my trips based on the main attributes like title, budget, it works, but not when I use an attribute of a nested child.
What other approach can I use to filter based to nested keys or should I structure the data differently? Any help is greatly appreciated.
Firebase Realtime Database queries operate on a flat list of child nodes directly under the path that you query.
So the value you order/filter on has to be at a fixex path under each immediate child node. Since that isn't the case for your uid, you can't query across all trips for the UID of all users of those trips.
You can query across one trip for a UID of a user (and then get back that user), or you can query across all trips for properties of the trip itself, such as its title or budget.
If you want to query across all users on all trips, consider keeping an additional list where you have the UID of the user as the key, and then all their trips under there:
"user_trips": {
"uid1": {
"tripid1": true,
"tripid2": true
},
"uid2": {
"tripid2": true,
"tripid3": true
}
}
Also see:
Firebase Query Double Nested
Firebase query if child of child contains a value
Many to Many relationship in Firebase
whats up?
I have an app that displays a list of items on Firestore using Kotlin and RecyclerView (from FirebaseUI api).
The DB is structured like this:
Users/userID/document_month_year
I need to query the data from the current user.
Each user has his own document_month_year document.
I read a lot of posts here, but each one tell one thing.. there´s no consense and nothing seems to work.
This query just sends me all documents from all users, how can I fix this?
private val queryFilteredByPaidStatus = db.collectionGroup(collectionName).whereEqualTo("users", userId)
Like this is an important question, here is the awnser that I
private val queryTest = db.document("users/"+userId).collection(collectionName)
fun getData() : FirestoreRecyclerOptions<Expense> {
return FirestoreRecyclerOptions
.Builder<Expense>()
.setQuery(queryTest, Expense::class.java)
.build()
}
Create a separate collection for documents to be read i.e month_year and for each document, add a field inside it which tells you the uid of the authenticated user, to which the document belongs to. Now you can query the collection like:
firestoreDB.collections("month_year").whereEqualTo("uid",auth.currentUser.uid)
Currently I am just using FirebaseListAdapter in Android to display a list of Orders.
My Orders are stored like this within firebase database.
-completed
-KsRHO1sLcVxKtVMec3o
dateCreated: 1503713136950
orderItems: 0
description: “Fries”
id: 101
1
description: “Small Burger“
id: 1023
I would like to create a chart of items sold
How could i go through the list of completed orders and count the amount of fries that have been sold on a certain date?
I would like the data to be returned like this
{
“Item”: “fries”,
“amountSold”: 34
}
{
“Item”: “Small Burger”,
“amountSold”: 4
}
Firebase function makes it easy to do that. You would need to go through your data and count the sold items (children count essentially) and then write the sum to your destination location.
See Here for an example of such Firebase function for counting children. The code that counts the correct value depends on the structure of your data but essentially is the same. You have to get the data, count children and upload the sum back to database. All happens within one Firebase function. Your function can be triggered by onWrite so that any new write or update triggers it.
Make sure to use transactions when writing the sum back to database because the value of that sum can change as you are writing it to database.
All You can do is create filed for date like created date and end date make save same date in both and then query on child according to date.!
var startDate = 14-02-17;
var endDate = 14-02-17;
ref.orderByChild("childID").startAt(startDate).endAt(endDate)
.addValueEventListener(new ValueEventListener() {
}
var query = firebase.database().ref('child').orderByChild('startDate').equalTo(startDate);