I'm having sync problems on my mobile application. The app works on-line and off-line.
The main issues consists on sync time, data crush and sync interrupts.
I have an REST API that serves data and business logic to mobile device. The Rest API has also the authentication mechanism of the app.
To sync, the mobile app has an background job (battery consumption problem) that runs every 30 seconds. This process the data stored on local database (SQLLite) and makes some rest calls for each operation. Some operations requires to write on different entities which it could originate several calls for each operation since the rest api it is entity oriented based on ODATA protocol.
At the moment, I'm studying new approaches to solve some of this problems.
I think, I want to entry on real-time database on the current the architecture...
I have some scenarios in mind, but my lack of knowledge on this kind of architectures makes me had some doubts about the best scenario.
My doubts are:
If I add an real-time database (like firebase, realm or others...),
should I keep the SQLLite database on the mobile app?
Since I already have an rest api to manage the data how this should communicate with the real-time db? should be my current rest
api "ask" to the real-time db the updates?
It is possible to implement an action on the real-time db (firebase or others) that notifies my current API?
Should the web evolve to use the real-time db also?
For what I found on the web, exists a lot of scenarios that can be followed depending on each case. However I don't find any clear scenario for my case.
In abstract, my app needs to work on-line and off-line and syncs fully with an server api.
The current implemented sync mechanism on the mobile app it may be not the best. Some operations has to make some calls to api and wait for the answer to process the next calls. Exists some business logic to process these calls. On server side some requests can be a little slow depending on the volume of the data to process.
Initial, my point is to improve the performance of the app with no sync issues.
Related
So I have a project that I'm making which is An Android Application, which takes provides information taken from API Server (Laravel 5.x) and also provides live-chat feature
What I'm not sure of doing is how to handle the live-chat feature. Currently here's how the chat system works (keep in mind, there is also a Web App which provides the same feature, that I'm willing to change) :
User type chat and sends it. The client (Android app or Web App) sends it to the API Server. which then saves it to the database.
Then the API Server (which is connected to mysql database server) sends in email / notification which based on the other user's configuration.
When the other User opens the chat, it will call the API Server (which is connected to mysql database server) asking if there's an update every 3 seconds.
if there is, then (Android App or Web App) will show the new chat.
Now first of all I know that this is not efficient, bad on performance, etc.
Now I want to try and use firebase database for realtime chat.
Now, I really want to use the API Server for logging and storing Chat info inside my own database because of some legacy Android App which takes a long time to work on.
which is the better scenario :
The Android App directly sends new chat information to Firebase Database
The Android App sends new chat information to API Server. Then the API Server sends it to Firebase Database
Is there a better way to do this
so which is the better way to do this?
I would highly recommend using the firebase firestore. It sounds like it would solve the best of both worlds. Nosql json storage format is great for a chat app. Also the data is "live" so it will loosen for a change and update the user sides data. Saving you the 3 sec call resources. Last I checked it was still in beta but I used it top make an app that lets you create plus and live pools and it wad pretty smooth. It does take a bit to wrap your head around though.
I am currently using Firebase Firestore as a primary backend that retrieves data from a variety of sources. I also use Android's Room for my mobile backend. When the phone receives data it is stored in the Room database in the event the user will not go online again for days even weeks.
After looking through the device files, I see firestore saves the data in files under the /data/data/<your-app>/databases directory.
The file looks something like this
I have read the offline persistence docs on the firestore and there is no indication on how durable the offline persistence is It mentions that the data is cached but not for how long. My question is, what is the durability of Firestore's offline persistence. Would one recommend using it instead of having a fully-fledged local DB to store data that may not be synced over long periods of time (days,weeks)?
It seems to already handle syncing data well once a connection is re-established. Im just worried that after some point that file may be deleted by the system and the user loses everything.
On Android (as of this writing) Firestore uses SQLite as a persistence mechanism. So for intermittent periods of offline activity you should have no problems with performance or durability.
However if you are going to be offline for days or weeks (as you said) there are some things you should be aware of:
Performance
Because Cloud Firestore is meant to be used mostly online, pending writes that have not yet been synced to the server are held in a queue. If you do many pending writes without going online to resolve them, that queue will grow and it will slow down your overall read/write performance. Most of Cloud Firestore's performance guarantees come from indexing and replication on the backend, but most of those optimizations don't exist when you're operating offline-only.
Conflicts
Firestore's basic conflict resolution model is "last write wins". So if you have many offline clients writing to the same document, only the last one to come online will actually "win" and persist their change.
Features
Most of Firestore's features work offline with one major exception: transactions. Transactions can only execute when you are online. So if your app uses transactions it will not work properly offline without some special handling.
There is no indication in offical documentation on how durable the offline persistence is because it cannot be predicted. This question cannot have an exact answer, like 4 weeks or something like this because it depends on how many write operations take place while you are offline.
I recommend you not to use Cloud Firestore as an offline-only database. It's really designed as an online realtime database that can work for short to intermediate periods of being disconnected.
While offline, it will keep queue of all your write operations. As this queue grows, local operations and app startup will slow down. But you need to know that these operation will persist even if you restart the device. You not gonna lose any data.
It seems that mobile local databases with sync capabilities are gaining some momentum in the mobile scene. That kind of approach can replicate data from the backend to the app (and vice-versa) without much effort. You won't have to write all that code in the app and backend to exchange information. It sounds awesome, right?
But, I was wondering in which cases the replication won't be efficient. What are the pros and cons of that approach? I'm particularly worried about too much data consumption on the mobile side and searching for info that's not on the local database.
I would argue that the replication approach available in CouchDB, Couchbase Mobile, PouchDB…actually saves on data consumption because there is a local database on each end of the replication. Instead of storing the data in a cache that would potentially be invalidated in time, the Sync approach looks for the diffs on each side and works out what’s missing and should be sent.
However, the sync approach doesn’t fit all use cases. If we imagine the example of a location based mobile application. There may be a feature to search for content around the user’s location. It wouldn’t make sense to replicate the entire server side database to the device to make that query. You can use REST API calls for those type of queries. And once a user has "bookmarked" locations, use the sync approach to make them accessible to all logged in devices.
In the case of Couchbase Mobile, the filtered sync is handled by the Sync Function and the REST API can be used for queries on the server side database.
I am in the process of developing an app with the help of someone else for a site I am developing. This app being developed will require access to the database on the web server. So my question here is typically I would want to keep my servers access limited to specific IP's however. Mobile devices all have there own ip's from what I hear. So if this is the case I essentially have to open up my database access to allow any inbound remote connection. Which is more than possible, but I forsee a potential security risk with that. namely in the notion of someone attempting to brute force the DB to gain root access or something to the effect there of. With that my question comes down to is there a more secure way of doing what I need in the overall? Is there any methods I can apply that would allow me a little more peace of mind. The DB I am currently using is mySQL, and will generally remain that type til later in the future.
Your concerns about security risks are valid. You should avoid a situation where countless mobile apps are making database connections to your MySQL database. A better solution would be to host a REST API layer in front of the database. This would allow you to control what portions of your database are accessed, but potentially include authentication routines as well. The REST web services you publish will give you the opportunity to produce permitted representations of your data rather than raw access to the data. That means in addition to producing web services, you'll need to develop the app to be a REST client that consumes these services or makes "requests" to them on an as needed basis. Obviously, this requires planning on both sides of development (back-end and front-end).
One possible solution to consider for producing REST web services is here:
http://phprestsql.sourceforge.net/
There are additional frameworks available for PHP, Java, ASP.NET and other platforms.
http://peej.github.com/tonic/
http://jersey.java.net/
Good luck!
Why not access the database via an API and then secure the API? So in essence, you should have a REST(or anything) API on your web server which your mobile application talks to which subsequently talks to the database. Accessing and securing your API will make a more cleaner and leaner design.
I'm currently developing my first Android application and still in the designing stage trying to come up with a solid model.
My application will use the GCal data from a users Google calendar and sync it up with one or more other users to determine common meeting times between all without the tedious back and forth of scheduling over email.
I vision this working by storing each user and their calendar data in a database that will be refreshed daily. When a query to determine the optimal meeting times between a group is issued, I want to select the calendar data of each user from the database, perform the computation to find optimal times, and display the results back to the user who made the query.
The AWS SDK for Android supports Amazon SimpleDB and S3, in which case I would use SimpleDB for my database. Where I am getting lost is using the Amazon EC2 web service in concert with the SimpleDB to perform the computation.
First off, any feedback on my approach and/or design is appreciated.
Second, how does using non-Android, but Java based APIs/SDKs effect applications, or is it even possible to do so?
The API typica for Java looks interesting and useful if it is possible to use with Android for instance.
Thanks!
So, I think its important to note a couple of things.
What you are describing is not an 'android application'. Its a web service application with an android client. The reason I'm being pedantic is that many of the design decisions you need to make are completely besides the fact that your primary client will run on android.
I'm concerned about the viability of storing the users calendar in a non-relation database. I don't know if you've already looked through this, but the problem you are trying to solve (calendaring) seems like it would benefit from the relational benefits of a relational database. For instance, i'm not sure how you would structure for storage the data of past, present and future events/meetings in a non-relational. Its probably possible, but i'm not sure if its optimal. Depending on the amount of data you may also need to consider the maximum record size.
While its true that AWS SDK for android supports writing to S3 or SimpleDB, I think there is a lot to consider. The reason you are confused about the interaction with EC2 is that normally, your EC2 web service will be interacting with S3 or SimpleDB. By using the AWS SDK you can, in theory, remove the requirement for a web service. My main issue with that is that you're now forced to do lots more on each client because there is no common access pattern. Your ios client or web client needs to have all the same logic that your android client has to make sure its accessing your s3 and simple db data the same. If that doesn't make sense i can elaborate.
Using non-android api's and sdks is a mixed bag. Sometimes it works fine if the classes compile to Davlik. If they don't it doesn't work.
One thing I might point out, since you'll already possibly be tied to a Google technology is Google App Engine. The nice part about it is that there is a free level of service which lets you get your app up and running without cost. Based on the technologies you are suggesting, it might be something for you to look into. Other than that, my other strong suggestion is that you focus on building out the web service first and independently of the android client. Take the time to model what the client server interaction would be and move as much of the 'logic' to the server as is possible. Thats what I felt like was missing from your initial description. Where the crunching would be.
my solution is that you use O-O principles. store your db on amazon dynamoDB and then sync user data with the mobile app. then you do processing of the data/computation on the device before displaying the results