Basically
Incrementing values seems like it should be so simple but it seems like it's not in my case.. I know I've still got so much to learn especially with VTL which is entirely new to me so please excuse my ignorance.
All I want is to be able to add to/subtract from an existing value possibly offline and for the server to capture all those changes when coming online, keeping multiple (intermittently online) devices consistent.
Use cases:
Keep count of inventory/
Keep balances for Accounts/
Votes/
Visits/
etc/
Scenario
device 1 makes n changes to Item A.quantityOnHand, ( i.e: A delivery is made from a driver using the app)
device 2 makes n changes to Item A.quantityOnHand,( i.e: Some Sales are made of the item over time)
device n makes n changes to Items/Customers/Cash/Votes/Visits/Some other Counting operation.
All changes need to be captured and at any time devices could go offline.
So Far..
I have looked at a custom resolver.. something like this simple change in my resolver:
#if ( $entryKeyAttributeName == "transacted")
##perhaps use regex and filter a set prefix
$util.qr($expAdd.put("#$entryKeyAttributeName", ":$entryKeyAttributeName"))
#else
$util.qr($expSet.put("#$entryKeyAttributeName", ":$entryKeyAttributeName"))
#end
I found that this only works once, for the most current version of the model. The result is only the latest update is synced, depending on current server versions, resulting in inconsistent data.
I thought the custom conflict handling could help, Perhaps If I have a locally stored 'last-synced-value' and then use the differences to create a modified model corrected with the changed local values taken into account.
Could this work?
What is the downside of creating _increment logic locally in a DB?
It seems like it would be a simple process?
if amplify.isConnected -> save normally, immediate update using model.
if amplify.isNotConnected:
->diff model saved and queued separately in a local DB (with a 'incremented value' field)
->query(model) finds and sums any relevant model during unSynced/offline State
upon Amplify.isNowSynced && connected
->save the now synced and updated models with the relevant increments
->delete the increment rows once response from server is received and checked for consistency
I did something similar in the below extension... i used another SQflite db with one table..
Can you see any downside to something like the followingextension?
extension AmplifyHelper on DataStoreCategory {
Future<bool> incrementValue<T extends Model>(T model, incrementValue, QueryField incField,
{QueryPredicate? where}) async {
/// If online and connected, just save in the normal way
if (_networkService.connectivityResult.value != ConnectivityResult.none) {
try {
save(model, where: where);
return true;
} catch (e) {
debugPrint('*** An error occurred trying to save while online ***');
return false;
}
} else {
/// Otherwise, create a map to save in a sqflite DB
var map = <String, dynamic>{};
map['dyn_tableName'] = model.getInstanceType().modelName();
map['dyn_rowId'] = model.getId();
map['dyn_increment'] = incrementValue;
map['dyn_field'] = incField.fieldName;
map['dyn_fromValue'] = model.toJson()[incField.fieldName];
return _dbService.updateOrInsertTable(map);
}
}
Future<List> queryWithOffline<T extends Model>(ModelType<T> modelType,
{QueryPredicate? where, QueryPagination? pagination, List<QuerySortBy>? sortBy}) async {
/// Get normal results ( basically unchanged from increments, contains the last-synced value for any increment)
List<T> amplifyResult = await query(modelType, where: where, pagination: pagination, sortBy: sortBy);
/// Get any increments from other DB
List<Map<String, dynamic>> offlineList = await _dbService.getOfflineTableRows(tableName: modelType.modelName());
if (offlineList.isNotEmpty && amplifyResult.isNotEmpty) {
/// If there is something in there SUM the relevant fields for each row and return.
List<T> listWithOffline = [];
for (var rowMap in offlineList) {
ModelField field = modelProvider.modelSchemas
.firstWhere((mdl) => mdl.name == rowMap['dyn_tableName'])
.fields![rowMap['dyn_field']]!;
Map<String, dynamic> modelMap =
amplifyResult.firstWhere((item) => item.getId() == rowMap['dyn_rowId']).toJson();
modelMap[field.name] = modelMap[field.name] + rowMap['dyn_increment'];
listWithOffline.add(modelType.fromJson(modelMap));
}
return listWithOffline;
} else {
/// There is nothing in the sync DB, just return data.
return amplifyResult;
}
}
Future<bool> returnToOnline<T extends Model>() async {
/// Called when Amplify is synced /ready after coming online
/// Check if Amplify is resynced
if (!_amplifySyncService.amplifyHasDirt) {
List<Map<String, dynamic>> offlineList = await _dbService.getOfflineTableRows();
if (offlineList.isNotEmpty) {
List<T> listWithOffline = [];
ModelType<T>? modelType;
List<T> amplifyResult = [];
for (var rowMap in offlineList) {
///Basically the same process of match and sum as above
if (modelType == null || modelType.modelName() != rowMap['dyn_tableName']) {
modelType = modelProvider.getModelTypeByModelName(rowMap['dyn_tableName']) as ModelType<T>?;
amplifyResult = await Amplify.DataStore.query(modelType!);
}
ModelField field = modelProvider.modelSchemas
.firstWhere((mdl) => mdl.name == rowMap['dyn_tableName'])
.fields![rowMap['dyn_field']]!;
Map<String, dynamic> modelMap = amplifyResult.firstWhere((item) => item.getId() == rowMap['dyn_rowId']).toJson();
modelMap[field.name] = modelMap[field.name] + rowMap['dyn_increment'];
listWithOffline.add(modelType.fromJson(modelMap));
}
for (var mdl in listWithOffline) {
/// Saving the updated model with the increments added
await Amplify.DataStore.save(mdl);
if (await _dbService.deleteRow(mdl.getId())) {
debugPrint('${mdl.getId()} has been processed from the jump queue');
// TODO: final looks...
// if(isNowSynced(mdl){
listWithOffline.remove(mdl);
// }else{
// rollback(mdl);
// }
}
} else {
debugPrint('No jump queue to process');
}
} else {
print('*** Amplify Had Dirt! ***');
}
return true;
}
}
Thanks for reading
On a project i am working on i list all contacts with checkboxlist. When you tap one it adds to selectedContact list and Database, tap again and it deletes from both so far so good. My problem was when app restarts selected Contacts also zeroed so i used database. I save phone numbers and query all contacts with that phone number and keep adding it to selectedContacts. The problem is even though selectedContact items has the same values as _contacts items selectedContact.contains(c) returns false.
Any ideas?
c is the 'Contact' class object
return ListView.builder(
itemCount: _contacts?.length ?? 0,
itemBuilder: (BuildContext context, int index) {
Contact c = _contacts?.elementAt(index);
//print(snapshot.data[index]);
//selectedContacts.forEach((element) {print(element.phones.first.value);});
//print(selectedContacts.contains(c).toString() +" " +c.phones.first.value);
return CheckboxListTile(
value: selectedContacts.contains(c),
onChanged: (bool value) {
setState(() {
value
? addDb(c)
: deleteDb(c);
});
},
title: Text(c.displayName ?? ""),
secondary: CircleAvatar(child: Text(c.initials())),
);
void addDb(Contact c) {
selectedContacts.add(c);
DatabaseHelper.addContact(c);
}
void deleteDb(Contact c) {
selectedContacts.remove(c);
DatabaseHelper.deleteContact(c);
}
var selectedContacts = List<Contact>();
void importFromDatabase()async{
var import = await DatabaseHelper.importContacts();
import.forEach((element)async {
var contacts = (await ContactsService.getContactsForPhone(element,withThumbnails: false,iOSLocalizedLabels: true)).toList();
selectedContacts.addAll(contacts);
});
}
I have some kind of debugging print statement below which prints selectedContacts and prints if it contains the C
but even though their variables are same it returns false.
EDIT:
Contact class is from contacts_service package
// Name
String displayName, givenName, middleName, prefix, suffix, familyName;
// Company
String company, jobTitle;
// Email addresses
Iterable<Item> emails = [];
// Phone numbers
Iterable<Item> phones = [];
// Post addresses
Iterable<PostalAddress> postalAddresses = [];
// Contact avatar/thumbnail
Uint8List avatar;
List.contains checks the items in the list for strict equality. The Contact class from the referenced package defines equality as two objects having equal values in 16 different fields. If even one of them is different in the object you are checking with, contains will return false. Since you are only saving phone numbers in your database, you are probably not saving all the fields that a Contact object has, so when you recreate that object, it won't "equal" the object it originally referred to.
You need to store the whole object in your database. Relational databases aren't ideal for storing objects with a complex structure, so instead I would recommend using a NoSQL database such as hive, which comes with the added bonus of being able to operate exponentially faster than sqflite due to its use of a memory buffer.
Fortunately, the Contact class defines toMap and fromMap methods, so both storage and retrieval are easy:
// Storage
var box = Hive.box('contactsBox');
var payload = contact.toMap();
await box.put(contact.phones.first, payload);
// Retrieval
var box = Hive.box('contactsBox');
var stored = box.get(phoneNumber);
var contact = Contact.fromMap(stored);
(Note: box.put returns a future, but awaiting it isn't strictly necessary. Storage of data into the memory buffer is synchronous and the future only resolves when the data is successfully written to the file storage. If you don't care about waiting for that to happen and want to access the data immediately, you can skip the await here.)
I am wondering if it's possible to get multiple documents by a list of ids in one round trip (network call) to the Firestore database.
if you're within Node:
https://github.com/googleapis/nodejs-firestore/blob/master/dev/src/index.ts#L978
/**
* Retrieves multiple documents from Firestore.
*
* #param {...DocumentReference} documents - The document references
* to receive.
* #returns {Promise<Array.<DocumentSnapshot>>} A Promise that
* contains an array with the resulting document snapshots.
*
* #example
* let documentRef1 = firestore.doc('col/doc1');
* let documentRef2 = firestore.doc('col/doc2');
*
* firestore.getAll(documentRef1, documentRef2).then(docs => {
* console.log(`First document: ${JSON.stringify(docs[0])}`);
* console.log(`Second document: ${JSON.stringify(docs[1])}`);
* });
*/
This is specifically for the server SDK
UPDATE: Cloud Firestore Now Supports IN Queries!
myCollection.where(firestore.FieldPath.documentId(), 'in', ["123","456","789"])
In practise you would use firestore.getAll like this
async getUsers({userIds}) {
const refs = userIds.map(id => this.firestore.doc(`users/${id}`))
const users = await this.firestore.getAll(...refs)
console.log(users.map(doc => doc.data()))
}
or with promise syntax
getUsers({userIds}) {
const refs = userIds.map(id => this.firestore.doc(`users/${id}`))
this.firestore.getAll(...refs).then(users => console.log(users.map(doc => doc.data())))
}
They have just announced this functionality, https://firebase.googleblog.com/2019/11/cloud-firestore-now-supports-in-queries.html .
Now you can use queries like, but mind that the input size can't be greater than 10.
userCollection.where('uid', 'in', ["1231","222","2131"])
With Firebase Version 9 (Dec, 2021 Update):
You can get multiple documents by multiple ids in one round-trip using "documentId()" and "in" with "where" clause:
import {
query,
collection,
where,
documentId,
getDocs
} from "firebase/firestore";
const q = query(
collection(db, "products"),
where(documentId(), "in",
[
"8AVJvG81kDtb9l6BwfCa",
"XOHS5e3KY9XOSV7YYMw2",
"Y2gkHe86tmR4nC5PTzAx"
]
),
);
const productsDocsSnap = await getDocs(q);
productsDocsSnap.forEach((doc) => {
console.log(doc.data()); // "doc1", "doc2" and "doc3"
});
You could use a function like this:
function getById (path, ids) {
return firestore.getAll(
[].concat(ids).map(id => firestore.doc(`${path}/${id}`))
)
}
It can be called with a single ID:
getById('collection', 'some_id')
or an array of IDs:
getById('collection', ['some_id', 'some_other_id'])
No, right now there is no way to batch multiple read requests using the Cloud Firestore SDK and therefore no way to guarantee that you can read all of the data at once.
However as Frank van Puffelen has said in the comments above this does not mean that fetching 3 documents will be 3x as slow as fetching one document. It is best to perform your own measurements before reaching a conclusion here.
If you are using flutter, you can do the following:
Firestore.instance.collection('your_collection_name')
.where(FieldPath.documentId, whereIn:["list", "of", "document", "ids"])
.getDocuments();
This will return a Future containing List<DocumentSnapshot> which you can iterate as you feel fit.
Surely the best way to do this is by implementing the actual query of Firestore in a Cloud Function? There would then only be a single round trip call from the client to Firebase, which seems to be what you're asking for.
You really want to be keeping all of your data access logic like this server side anyway.
Internally there will likely be the same number of calls to Firebase itself, but they would all be across Google's super-fast interconnects, rather than the external network, and combined with the pipelining which Frank van Puffelen has explained, you should get excellent performance from this approach.
You can perform an IN query with the document IDs (up to ten):
import {
query,
collection,
where,
getDocs,
documentId,
} from 'firebase/firestore';
export async function fetchAccounts(
ids: string[]
) {
// use lodash _.chunk, for example
const result = await Promise.all(
chunk(ids, 10).map(async (chunkIds) => {
const accounts = await getDocs(
query(
collection(firestore, 'accounts'),
where(documentId(), 'in', chunkIds)
));
return accounts.docs.filter(doc => doc.exists()).map(doc => doc.data());
})
);
return result.flat(1);
}
Here's how you would do something like this in Kotlin with the Android SDK.
May not necessarily be in one round trip, but it does effectively group the result and avoid many nested callbacks.
val userIds = listOf("123", "456")
val userTasks = userIds.map { firestore.document("users/${it!!}").get() }
Tasks.whenAllSuccess<DocumentSnapshot>(userTasks).addOnSuccessListener { documentList ->
//Do what you need to with the document list
}
Note that fetching specific documents is much better than fetching all documents and filtering the result. This is because Firestore charges you for the query result set.
For some who are stucked in same problem
here is a sample code:
List<String> documentsIds = {your document ids};
FirebaseFirestore.getInstance().collection("collection_name")
.whereIn(FieldPath.documentId(), documentsIds).get().addOnCompleteListener(new OnCompleteListener<QuerySnapshot>() {
#Override
public void onComplete(#NonNull Task<QuerySnapshot> task) {
if (task.isSuccessful()) {
for (DocumentSnapshot document : Objects.requireNonNull(task.getResult())) {
YourClass object = document.toObject(YourClass.class);
// add to your custom list
}
}
}
}).addOnFailureListener(new OnFailureListener() {
#Override
public void onFailure(#NonNull Exception e) {
e.printStackTrace();
}
});
For the ones who want to do it using Angular, here is an example:
First some library imports are needed: (must be preinstalled)
import * as firebase from 'firebase/app'
import { AngularFirestore, AngularFirestoreCollection } from '#angular/fire/firestore'
Some configuration for the collection:
yourCollection: AngularFirestoreCollection;
constructor(
private _db : AngularFirestore,
) {
// this is your firestore collection
this.yourCollection = this._db.collection('collectionName');
}
Here is the method to do the query: ('products_id' is an Array of ids)
getProducts(products_ids) {
var queryId = firebase.firestore.FieldPath.documentId();
this.yourCollection.ref.where(queryId, 'in', products_ids).get()
.then(({ docs }) => {
console.log(docs.map(doc => doc.data()))
})
}
I hope this helps you, it works for me.
getCartGoodsData(id) {
const goodsIDs: string[] = [];
return new Promise((resolve) => {
this.fs.firestore.collection(`users/${id}/cart`).get()
.then(querySnapshot => {
querySnapshot.forEach(doc => {
goodsIDs.push(doc.id);
});
const getDocs = goodsIDs.map((id: string) => {
return this.fs.firestore.collection('goods').doc(id).get()
.then((docData) => {
return docData.data();
});
});
Promise.all(getDocs).then((goods: Goods[]) => {
resolve(goods);
});
});
});
}
Yes, it is possible. Sample in .NET SDK for Firestore:
/*List of document references, for example:
FirestoreDb.Collection(ROOT_LEVEL_COLLECTION).Document(DOCUMENT_ID);*/
List<DocumentReference> docRefList = YOUR_DOCUMENT_REFERENCE_LIST;
// Required fields of documents, not necessary while fetching entire documents
FieldMask fieldMask = new FieldMask(FIELD-1, FIELD-2, ...);
// With field mask
List<DocumentSnapshot> documentSnapshotsMasked = await FirestoreDb.GetAllSnapshotsAsync(docRefList, fieldMask);
// Without field mask
List<DocumentSnapshot>documentSnapshots = await FirestoreDb.GetAllSnapshotsAsync(docRefList);
Documentation in .NET:
Get all snapshots
Field mask
This doesn't seem to be possible in Firestore at the moment. I don't understand why Alexander's answer is accepted, the solution he proposes just returns all the documents in the "users" collection.
Depending on what you need to do, you should look into duplicating the relevant data you need to display and only request a full document when needed.
if you are using the python firebase admin sdk this is how you query for multiple documents using their uids
from firebase_admin import firestore
import firebase_admin
from google.cloud.firestore_v1.field_path import FieldPath
app = firebase_admin.initialize_app(cred)
client = firestore.client(app)
collection_ref = client.collection('collection_name')
query = collection_ref.where(FieldPath.document_id(), 'in', listOfIds)
docs = query.get()
for doc in docs:
print(doc.id, doc.to_dict())
Instead of importing FieldPath you can also simply use the string __name__. Now your query will be collection_ref.where('__name__', 'in', listOfIds)
The best you can do is not use Promise.all as your client then must wait for .all the reads before proceeding.
Iterate the reads and let them resolve independently. On the client side, this probably boils down to the UI having several progress loader images resolve to values independently. However, this is better than freezing the whole client until .all the reads resolve.
Therefore, dump all the synchronous results to the view immediately, then let the asynchronous results come in as they resolve, individually. This may seem like petty distinction, but if your client has poor Internet connectivity (like I currently have at this coffee shop), freezing the whole client experience for several seconds will likely result in a 'this app sucks' experience.