I have a list of users, assume 10k.
Now if I run a query like below:
query = FirebaseDatabase.getInstance().getReference().child("users")
.orderByChild(SELECTED_QUERY)
.limitToFirst(150);
Will it consume bandwidth of 150 users' data or 10k users' data?
This question may be silly, but I haven't found expected answer.
Whether the SDK retrieves all users or just the first 150 depends on whether you've defined an index on SELECTED_QUERY. Without an index the SDK will retrieve all of users and filter locally on the client; with the index the filtering will happen on the server.
Given your comment, you'll need to add an index to your rules like this:
{
"rules": {
...
"users": {
...
".indexOn": "name"
}
}
}
Related
My goal is to effectively get a list of children (ordered* and indexed**) with the lowest number of data transfer.
* ordered: ordered by points for each user / database child
** indexed: 2 or less ranks behind/after the current user [A specific child] (further elaborated below)
My database structure is as follows:-
I basically want to get the first 3 users ordered by points (simple):-
val usersRef = FirebaseDatabase.getInstance(DB_LINK).getReference("users").orderByChild("points")
usersRef.limitToFirst(3).addValueEventListener(
object : ValueEventListener {
override fun onDataChange(snapshot: DataSnapshot) {
for (ds in snapshot.children) {
val points: String = snapshot.child("points").getValue(String::class.java)!!
val firstName: String = snapshot.child("firstName").getValue(String::class.java) ?: ""
val uid: String = snapshot.key!!
// Update View
}
}
override fun onCancelled(error: DatabaseError) {}
}
)
Then, provided that the currently logged in user isn't one of the first three, I want to get his rank (order according to points in the whole db), 2 users' before him, and 2 users' after him without querying the whole database (it's a user database that can get up to 50K unique users) because querying the whole database is a really expensive client-side task.
I checked firebase data filtering page but found nothing useful about limiting results according to a certain child.
This answer doesn't satisfy my needs, as it loops over the whole database (in my case, 50K records). I need an effective method as I need to really save these firebase bills.
Moreover, I check this answer but it didn't meet my needs because it still queries the whole database, meaning it is not effective and will be billed for each node before the current user. (Maybe he is number 40,000 in the db, so I shouldn't query the whole db each time to get his rank and get billed for 39,999 reads)
I searched for a way to somehow use booleans to filter queries but again found nothing useful. Here is my not-effective code:-
// Gets all children.
usersRef.addValueEventListener(
object : ValueEventListener {
override fun onDataChange(snapshot: DataSnapshot) {
for (ds in snapshot.children) {
val points: String = snapshot.child("points").getValue(String::class.java)!!
val firstName: String = snapshot.child("firstName").getValue(String::class.java) ?: ""
val uid: String = snapshot.key!!
// Update View only if user is `2 <= usersRank - theirRank <= -2`
}
}
override fun onCancelled(error: DatabaseError) {}
}
)
I want to achieve something like this:- (Styling already done, logic remaining)
Is there a way to achieve this? Any alternatives?
EDIT: I found out that firestore offers aggregation queries that may help in this situation. Doing more research to further narrow down the costs.
This operation is not available on a Firebase Realtime Database. A better option would be Firestore.
Why?
Well, A fire-store database can give you the count of objects in a certain query. This is a new feature added by firebase. You basically type the query you want, then add .count() before .get(); that way it'll return the count of objects only. This is called aggregation queries. Learn more about them here.
Cloud Functions - Why isn't it appropriate here?
Using a Cloud Function for aggregations avoids some of the issues with client-side transactions, but comes with a different set of limitations:
Cost - Each rating added will cause a Cloud Function invocation, which may increase your costs. For more information, see the Cloud Functions pricing page.
Latency - By offloading the aggregation work to a Cloud Function, your app will not see updated data until the Cloud Function has finished executing and the client has been notified of the new data. Depending on the speed of your Cloud Function, this could take longer than executing the transaction locally.
Write rates - this solution may not work for frequently updated aggregations because Cloud Firestore documents can only be updated at most once per second. Additionally, If a transaction reads a document that was modified outside of the transaction, it retries a finite number of times and then fails.
Combining this with other methods
Now that you're using COUNT() for this system, there is one more method to help further narrow down the costs. That is Periodic Updates.
Periodic Updates
Who would care about a live ranking of all users? You can make the leaderboard update each minute, hour, or day. For example, stack overflow's leaderboard is updated once a day!
This approach would really work for any number of players and any write rate. However, you might need to adjust the frequency though as you grow depending on your willingness to pay.
Costs Side
For each normal read, you are charged for one read. Very simple. However, for one count, you're charged for 0.001 reads (meaning 1000 counts = 1 read). For more information about costs, check this article by firebase.
Final Thoughts
To connect everything up, we shall now apply this on our problem. Firstly, we'll need to keep the first portion of the code as it is. (The portion that grabs the first 3 users), though with some changes to port it to firebase.
NOTICE: Don't forget to setup a composite index because we're ordering by multiple fields at the same time.
val top3 = HashMap<Int, HashMap<String, String>>()
Firebase.firestore.collection("users").orderBy("points", Query.Direction.DESCENDING)
.orderBy("firstName", Query.Direction.ASCENDING)
.get().addOnSuccessListener {
for ((index, doc) in it.documents.withIndex()) {
val firstName = doc.getString("firstName")!!
val points = doc.getString("points")!!
top3[index+1] = hashMapOf("first" to firstName, "points" to points, "uid" to doc.id)
}
}
More about ordering and limiting here.
Then, we'll need to implement the COUNT() feature.
Firebase.firestore.collection("users")
.whereGreaterThan("points", UserProfile.getInstance()!!.getProfile().points)
.count().get(AggregateSource.SERVER).addOnSuccessListener {
println("Your rank is: ${it.count+1}")
}
Basically what I did here was:-
Selecting the Collection
Ordered Ascending by first name so no duplicate ranks.
Count them, and pass onto the function.
The final step is just updating the hash map top3 and rank of user each hour/day/minute/...
I'm developing an app that has a video and an article feed. I implemented swipe refresh layout. Whenever user swipes it, it loads last 5 video datas from Firebase.
My database is like this (it's not much data) =>
Swipe refresh layout listener =>
mSwipeRefreshLayout.setOnRefreshListener(() -> {
manager.scrollToPosition(0);
refreshVideoFeed();
});
And refreshVideoFeed method triggers this =>
Query query = myRef.orderByChild("videoDate").limitToLast(5);
query.addListenerForSingleValueEvent(new ValueEventListener() {
#Override
public void onDataChange(#NonNull DataSnapshot dataSnapshot) {
..................
It works fine. However, I realized that app is using lots of data when i looked at Realtime Database dashboard (even app is not in product)
Then I opened Profiler in Android Studio to see what is going on. I ordered by child 'videoDate' and got last 5 videos. It costs 76 KB for refreshing video feed.
Then again I ordered by child 'videoDate' and got last 20 videos. Again it costs 77KB !
Also, I implemented the same mechanism for article feed. I have 236 articles in my realtime database. Whenever user swipes, it loads 10 articles and it costs 3,6 MB!
I wonder why this is happening and how to avoid that. It seems Firebase SDK fetchs all Videos to client and then filters them. But I want to fetch only specific range of data that I specified.
Ordering and filtering data is only done on the Firebase servers if there's an index defined on the property/value you filter on. If there's no index, the server sends all data to the client, which then orders and filters it. There should a quite explicit message in the log output when this happens.
To define an index, you go to the security rules panel in your Firebase console and on the node that myRef points to add an .indexOn property. Say myRef refers to /articles, it'd look something likeL
{
"rules": {
"articles": {
".indexOn": "videoDate"
}
}
}
If you order/filter on different properties, you can add multiple indexes:
".indexOn": [ "videoDate", "category" ]
I am making a challenges between users in my app .I am trying to get the last 15 users who enters in challenges. I store the time each time the users enter a new challenge. The query was working well in the begging but after that it stops showing new users and only old users appears in it.
this is the query code :
usersReference.orderByChild("lastChallengeDate")
.limitToLast(15)
.addListenerForSingleValueEvent(new ValueEventListener()
and this is the database structre of the user child :
When I opened the log I found this warn although I am using index on in my rules
W/PersistentConnection: pc_0 - Using an unspecified index. Consider adding '".indexOn": "lastChallengeDate"' at users to your security and Firebase Database rules for better performance
If you perform a query on a location, Firebase sorts the children under that location on the property you specify. There is no value in lastChallengeDate directly under each child of users. Instead the property is under lastChallengeDate/time, so you should order on that
usersReference.orderByChild("lastChallengeDate/time")
.limitToLast(15)
.addListenerForSingleValueEvent(new ValueEventListener()
You also need to define an index on users (or whatever the name is of the node you query):
{
"rules": {
"users": {
".indexOn": "lastChallengeDate/time"
}
}
}
Be sure to also study:
the documentation on queries, which includes an example of querying such a nested property
the documentation on defining indexes
some of the many questions with the same error message
I use Firebase database on my Android app. Normally, it works fine. But when the database is getting larger, the query performance is getting worse. I added about 5k record on database (under "elk" and "su" nodes), then I queried on database (on "cut" and "user" nodes) but all the queries are very very slow. I defined data index on database rules but it did not work. How can I solve that problem?
Here are my queries :
// query to get the zones followed by user
FirebaseDatabase.getInstance()
.getReference()
.child("user")
.child(userID)
.child("zones");
// query to get cuts on a zone
FirebaseDatabase.getInstance()
.getReference()
.child("cut")
.child(cutType)
.orderByChild("zoneID")
.equalTo(zoneID);
If you want to continue expanding the best thing to do would be to duplicate your data in a zone reference where it knows which elk/su are a part of it. Something like this:
{
zones: {
elk: {
"istan-besik": {
"-KSp)bL5....": true,
...: true
}
}
}
}
That way when you want to search for all you would just do:
...child('zones').child(zoneId).child(cutType)
And then loop through those to go get each elk/su directly
While firebase recommends use of indexes to keep the data structure flat, I am running into an issue here.
Consider this users entity (from Firebase docs sample)
{
"users": {
"mchen": {
"name": "Mary Chen",
// index Mary's groups in her profile
"groups": {
// the value here doesn't matter, just that the key exists
"alpha": true,
"charlie": true
"delta" : true
...
"10000th entry":true
}
},
...
}
Now I can read a user from the database like this.
List<User> users = new Select().from(User.class).limit(5).execute();
But, my question is if the "groups" key has say 10,000 entries what would happen? Will it fetch the whole list? Wouldn't it crash the app, due to lack of memory.
In general, if I am trying to fetch a high level node (which has nested nodes), what part of it will be fetched?