Be kind this is my first question on StackOverflow :p. I am hoping its specific enough.
Here is how the project is structured
REST API built using NodeJS and MongoDB (mongoose has been used for modelling the database schemas) with a Express server.
Android app.
Web app using AngularJS
My question is regarding how I should structure synchronisation of data between the Android app and the REST API. The following points bring clarity to the entire scenario -
The database model (on the server) is pretty complex with each database model having multiple subdocuments.
There are about 6 - 7 models which reference each other.
I am currently using Volley to get the data from the remote server.
I am also considering adding a SyncAdapter for syncing the data regularly and am not sure as to how to incorporate this with a local database. Should I have a different ContentProvider for every table / model in the database? and how should I handle nested schemas (that are in the remote server) locally?
To summarise my question what I exactly want to know is considering that there will be about 15-20 requests per user per day for about 100,000 users daily, would the best approach be to -
Use volley to do all my database work.
Use a local database (SQLite) along with a SyncAdapter to keep the data in sync automatically? If this is the case could you recommend some resources to better understand how to integrate a complex database with SyncAdapter.
Anything else you suggest for structuring this app.
To demonstrate the complexity of the app please have a look at the models below
This is the user model
var AddressSchema = new Schema({
name: String,
address: String,
mobile: String,
pincode: String,
city: String,
state: String
});
var CartSchema = new Schema({
book: { type: Schema.Types.ObjectId, ref: 'Book' },
quantity: {
type: Number,
default: 1
},
dateAdded: Date
});
var WishlistSchema = new Schema({
book: { type: Schema.Types.ObjectId, ref: 'Book' },
dateAdded: Date
});
var OrderSchema = new Schema({
orderNumber: String,
cart: [CartSchema],
totalAmount: Number,
deliveryCharge: Number,
discountAmount: Number,
address: [AddressSchema],
date: Date,
deliveryDate: Date,
deliveryStatus: String
});
var SellOrderSchema = new Schema({
orderNumber: String,
bookDetails: [{
isbn: String,
title: String,
quantity: {
type: Number,
default: 1
}
}],
address: [AddressSchema],
date: {
type: Date,
default: Date.now()
}
});
var ReceivedOrdersSchema = new Schema({
orderNumber: String,
bookDetails: [{
book: { type: Schema.Types.ObjectId, ref: 'Book' },
quantity: Number,
price: Number
}],
dueDate: Date,
status: {
type: String,
default: 'Pending'
}
});
var CouponSchema = new Schema({
coupon: [{ type: Schema.Types.ObjectId, ref: 'Coupon' }],
used: Number,
totalDiscount: Number
});
var UserSchema = new Schema({
name: String,
email: { type: String, lowercase: true },
role: {
type: String,
default: 'user'
},
hashedPassword: String,
provider: String,
salt: String,
facebook: {},
twitter: {},
google: {},
github: {},
usedCoupons: [CouponSchema],
cart: [CartSchema],
wishlist: [WishlistSchema],
orders: [OrderSchema],
sellOrders: [SellOrderSchema],
addresses: [AddressSchema],
booksToSell: [{ type: Schema.Types.ObjectId, ref: 'Book' }],
receivedOrders: [ReceivedOrdersSchema]
});
This is the books model
var BookSchema = new Schema({
_creator : {type: Schema.Types.ObjectId, ref: 'User'},
title: String,
subtitle: String,
author: String,
language: String,
pages: Number,
publisher: String,
publishedDate: String,
isbn10: String,
isbn13: String,
description: String,
dimensions: {
height: String,
width: String,
thickness: String
},
category: String,
rating: String,
bookType: String,
imageLinks: {
extraLarge: String,
large: String,
medium: String,
small: String,
smallThumbnail: String,
thumbnail: String,
uploaded: String
},
uploadedImageLink: String,
supplierData: {
mrp: Number,
supplierPrice: Number,
quantity: Number
},
pricing: {
salesPrice: Number,
deliveryCharge: Number
},
dateAdded: Date,
isFeatured: Boolean,
isBestseller: Boolean
});
There are 5-6 other such models that are dependent on each other
Static Data
Data like your UserTable is mostly static so you can store it in SQLite (or even Shared Preferences) for fast local data access. Also the stored data about books(title, author etc) is pretty much static (I don't think the author changes that often) so caching it in a SQLite is easy and straight-forward.
Dynamic Data
Caching dynamic data? Hard, useless but if you have to use Volley, it helps you a lot. Volley caches your responses using HTTP tags like Expire and Last-Modified. Your app will send a HTTP Request with an If-Modified-Since tag that contains a timestamp. If the data from your server has changed since that timestamp you will receive a normal response. Otherwise, you will receive a HTTP 304 Not Modified and you can use your old cached data. In both cases you app will save the current timestamp and use that next time. All of this is handled by Volley and hidden from you, but is useful to know. You will have to figure out how to handle this on the server side. If you don't want your server to handle to many connections, you could show your user cached data first and implement a Pull to Request gesture that would send a request.
Offline Access
Volley gives you an abstraction layer for times when you app isn't able to access the internet, but you should add a clear visual indicator for the user so they know that whatever you have showed them could have changed in the meantime.
All in all, you should divide your data based on how often it changes and according to that employ different strategies. Specify an expiration period or date to volley when you know it.
Related
I am parsing the below sample json using retrofit in android:
{
"success": true,
"timestamp": 1664080564,
"base": "EUR",
"date": "2022-09-25",
"rates": {
"AED": 3.559105,
"AFN": 86.151217,
"ALL": 116.321643,
"AMD": 404.265711
}
}
As you can see there is no array in this json data, but I want the values of rates as a list or a map so that I can get "AED, AFN, ALL, AMD" as an array too. How can i achieve that using retrofit?
You can define rates as a Map<String, Double> in your data class and Retrofit will automatically parse the rates in form of a Map.
data class MyModel(
val success: Boolean,
val timestamp: Long,
val base: String,
val date: String,
val rates: Map<String, Double>
)
so that I can get "AED, AFN, ALL, AMD" as an array too.
For this you can simply use rates.keys to get all the keys.
If I have a one-to-many Android Room relationship between two tables, expressed like this (using an example cut and paste from the Android developers site):
#Entity
data class User(
#PrimaryKey val userId: Long,
val name: String,
val age: Int
)
#Entity
data class Playlist(
#PrimaryKey val playlistId: Long,
val userCreatorId: Long,
val playlistName: String
)
data class UserWithPlaylists(
#Embedded val user: User,
#Relation(
parentColumn = "userId",
entityColumn = "userCreatorId"
)
val playlists: List<Playlist>
)
then when I use Retrofit2 to get the nested one-to-many object UserWithPlaylists from an API it's expecting JSON that looks more or less like this:
{
"user" : {
"userID": 1,
"name": "whatever",
"age": 10
},
"playlists": [
{
"playlistId": 0,
"playlistName": "whatever"
},
{
"playlistId": 1,
"playlistName": "whatever"
}
]
}
But my api (and as far as I can tell, most APIs) doesn't embed the user details in "user {}" like that. It just shows the user details unembedded, like this:
{
"userID": 1,
"name": "whatever",
"age": 10,
"playlists": [
{
"playlistId": 0,
"playlistName": "whatever"
},
{
"playlistId": 1,
"playlistName": "whatever"
}
]
}
So Retrofit2 would be able to create the playlists, but it won't find a user{} object so I would end up with empty user objects but populated playlist objects.
How would I force Retrofit2 look for the user details unembedded?
Apologies for my roundabout way of asking this question. I don't know the proper terminology for all of this, which has made it difficult for me to search.
thanks!
John
I'm trying to send some data from my node.js server to an android client via FCM(Firebase Cloud Messaging). I get the following error: "data must only contain string values" when sending it. My data contains 2 JSONObjects. Do I have to convert them to strings or what is the way to go here? Thanks.
var message = {
notification:{
"title": "Alert!",
"body": position[0] + " has left Area: " + activeGeofences[i][1].name
},
data:{
Geofence: activeGeofences[i][1],
Position: position[1]
},
token: activeGeofences[i][0]
};
To convert any JSON objects to a string, you can use JSON.stringify(). On the receiving side you can then use JSON.parse() (or your platform's equivalent) to parse the string back into a tree structure.
You can also do this, since it seemed more accurate to me
data: {
key1: "value1",
key2: "value2",
}
you just have to make sure that value1 or value2 or any n key's value pair is string if its a int or anything else it would throw the error. This could save you from parsing thing.
Inside data object data type should be always strings.
If you will pass type a number data type an error will occur.
let message = {
notification: {
title: payload.title,
body: payload.message,
},
data: {
name: 'sds',
type: '2',
_id: 'sdsd',
},
token: deviceToken,
};
Warning: My query would be more theoretical (sorry programmers, please bear with me). I am trying to get some idea on how to define the database structure for use in Firebase.
I am exploring the use of Firebase as a backend for a Review app (in Android) I am trying to build.
The app provides product details and review for products of different kinds. So here is an example use case.
The products displayed in the app are of same type (say smartphones). In this use case, defining the database structure is easier. For every phone, I simply need to save the phone specs to Firebase and retrieve them into my app.
Root
|
+--Smartphone
|
+--Manufacturer Name
+--Screen Size
+--Screen Density
+--Processor
+--RAM,...
The products displayed in the app are of different type (say smartphones, Car, Book,...). In this use case, defining the database structure becomes complex. I can simply define the data structure like
Root
|
+--Product
|
+--Manufacturer Name
+--Screen Size
+--Screen Density
+--Processor
+--RAM
+--Fuel type (Petrol/Diesel/Electric)
+--Vehicle Type (Sedan/Hatchback)
+--Vehicle Price,...
However, the problem with above data structure is, when I am trying to make a product review for a smartphone, the data related to Car will remain blank. Same will be the case for a product review of a Car.
This problem can be solved by using Flattening the data structure. This is where I am confused.
Root
|
+--Smartphone
| |
| +--Manufacturer Name
| +--Screen Size
| +--Screen Density
| +--Processor
| +--RAM
|
+--Car
|
+--Fuel type (Petrol/Diesel/Electric)
+--Vehicle Type (Sedan/Hatchback)
+--Vehicle Price,...
However, all product reviews will be displayed in a single activity/fragment. Hence, there will not be different activities/fragments for every product type. Could someone provide me a clear picture of using flattened data structures in my use case?
Regards
You can structure your database like this:
products: {
smartphones: {
smartphone1: {
name: "Best Phone",
ram: "6 GB",
screen: "5.5 inch"
reviews: {
review1: true,
review2: true
}
}
},
cars: {
car1: {
name: "Lightning"
reviews: {
review3: true,
review4: true,
review5: true
}
}
}
},
product-review: {
review1: {
submittedBy: "Conqueror",
message: "Best phone at this price",
timestamp: 1472405901
},
review2: {
submittedBy: "Magic Blaster",
message: "Pros: RAM, Cons: Everything else.",
timestamp: 1472405901
},
review3: {
submittedBy: "Boss",
message: "Excellent Car",
timestamp: 1472405901
},
...
}
Every product(smartphone1, car1 etc..) contains a reviews node, so you can easily load the linked reviews of a particular product.
You can generate timestamps for the nodes you want. You can add it for smartphone1, car1 etc.. but it is not required. But reviews by user require timestamp because you should display when the user has posted a review.
Have a look at this answer to know how to generate timestamps and to convert timestamp back into displayable date:
DateFormat sdf = new SimpleDateFormat("MM-dd-yyyy hh:mm:ss a");
Date someDate = (new Date(timestamp));
return sdf.format(someDate);
You can store user reviews in their individual nodes.
users: {
abcdefghijklm: { // User UID
name: "Conqueror",
reviews: {
review1: smartphone1,
review7: car1
}
},
nopqrstuvwxyz: {
name: "Magic Blaster",
reviews: {
review2: smartphone1
}
}
}
To get the reviews of a specific user, you can get the user's uid and use it to display reviews. Unfortunately, I have never used Firebase on Android, in Javascript, I would do it like this:
firebase.database().ref('users/'+userUID+'/reviews').on('value', function(snapshot) {
firebase.database().ref('product-review/'+snapshot.key).on('value', function(reviewSnapshot) {
console.log(reviewSnapshot.val().message);
});
});
If you want to only see the review of user "Conqueror", set userUID to his unique id. Here is the official documentation on Firebase for Android to retrieve data from database.
Here is the flattest database structure that I can think of. For the products node, you can also use the third structure in your question, it will only affect the logic on how to map the item in your app.
products: {
item1: {
type: "smartphone",
manufacturer_name: "some value",
screen_size: "some value",
screen_density: "some value",
processor: "some value",
ram: "some value"
},
item2: {
type: "car",
fuel_type: "some value",
vehicle_type: "some value",
vehicle_price: "some value"
}
},
users: {
user1: {
name: "some value",
email: "some value"
},
user2: {
name: "some value",
email: "some value"
},
},
products_reviews: {
item1: {
user1: ewview1,
user2: review2
},
item2: {
user2: review3
}
},
users_reviews: {
user1: {
item1: review1
},
user2: {
item1: review2,
item2: review3
}
},
reviews: {
review1: {
text: "this is my review",
timestamp: 1472488486000
},
review2: {
text: "this is my review",
timestamp: 1472488486000
},
review3: {
text: "this is my review",
timestamp: 1472488486000
}
}
Now you should be able to retrieve all reviews from each user and also retrieve all reviews for each product.
Comment here if you have questions, hope this helps :)
I need to get back the contact id after it is saved in order to save it to my online database. However the cordova contact.save() method does not return an id after execution.
Here is my logic:
if ($scope.contact.id === undefined) {
contact.save();
console.log("Contact ID is:", savedContact.id);
table.insert({ id: contact.id.value, firstname: name.givenName, lastname: name.familyName, homephone: phoneNumbers[0].value, mobilephone: phoneNumbers[1].value, email: emails[0].value });
}
This does not work.
Is there any way to retrieve the id for the contact without having to search the phones contact list using a phone number like this:
if ($scope.contact.id === undefined) {
contact.save();
var savedContact = navigator.contacts.find({ "phoneNumbers[0]": phoneNumbers[0].value });
console.log("Contact ID is:", savedContact.id);
table.insert({ id: contact.id.value, firstname: name.givenName, lastname: name.familyName, homephone: phoneNumbers[0].value, mobilephone: phoneNumbers[1].value, email: emails[0].value });
}
The above seems like way too much overhead. Not to mention it may not even return the correct contact as a phone number may not be unique.(If someone saves the contact twice with different information)
contact.save() can take two callbacks, success and failure. The success callback should return your newly saved contact (which would include the id.)
if ($scope.contact.id === undefined) {
contact.save(contactSuccess, contactFailure);
}
function contactSuccess(newContact) {
console.log("Contact ID is:", newContact.id);
table.insert({ id: contact.id.value, firstname: name.givenName, lastname: name.familyName, homephone: phoneNumbers[0].value, mobilephone: phoneNumbers[1].value, email: emails[0].value });
}
function contactError(err) {
//bb10 fires multiple error callbacks with empty errors
if (err) {
console.log(err);
}
}
Since it looks like you are using Angular, check out the ngCordova project. It provides some nice wrappers around some plugins that make everything a bit more readable. Here is the relevant excerpt from their contacts docs:
$cordovaContacts.save($scope.contactForm).then(function(savedContact) {
console.log("Contact ID is:", newContact.id);
table.insert({ id: contact.id.value, firstname: name.givenName, lastname: name.familyName, homephone: phoneNumbers[0].value, mobilephone: phoneNumbers[1].value, email: emails[0].value });
}, function(err) {
if (err) {
console.log(err);
}
});