I have an Android app, where a part of the app is a list of data which is currently contained in a string-array (in an xml resource). I currently release updates every so often to the actual app, which do nothing more than update this list of data. (yes, in hindsight this method was a bad idea to start with).
My goal now is to change this so that I will be able to only update that one part of the app that needs to be changed. I have a webserver, and am now serving a JSON version of the data off a URL. So all the app has to do is hit that URL, check if it changed (perhaps using a version number), and then update.
My problem lies in the actual implementation:
Where/how should I store this data? As a raw file? SharedPrefs? Database? [i.e. what are the pros and cons of each]
How can I preform a seamless upgrade where even if something devastating happens during the update [such as a user pulling a battery...], it still won't break the app?
Should the updating code live in a service?
I would separate your data crud into a small background service. Use the provided SQLLite. To verify data consistency you could use md5 checks, database rollback features and most importantly design a small set of tests. One only sending a partial file, i.e. "the broken transmission test", garbage file, etc. Keep it a separate and testable component of your app.
Related
As I plan for how to structure my data in Firestore to optimize for my use-case, I can't help but wonder what would happen if in the future I need to update the path of my data in Firestore?
Will I need to implement some mechanism of notifying all older versions of my mobile app to stop starting and require upgrades? Or a safety check when the app starts that reads a location in Firestore to decide whether the client needs to upgrade or can continue using the current version?
It seems like paths to data are supposed to be hardcoded in the client code, which can pose risk, especially for improvements in the product when data structure in Firestore is complicated (subcollections, etc)
Is there a way to better plan for this kind of a disaster in Firestore?
The first goal should be to make any database changes backwards compatible. It's possible a lot more often than you may think now.
But if you really want to prepare for incompatible changes you should include a version number in your database. In Cloud Firestore you'd typically put that in a global collection in a known document name, such as /Globals/VersionNumber. Note that this is the version number of the data model, not necessarily the version number of the app.
Now when the app starts, it reads that document first and checks if the version number matches with what it was made for. If it doesn't, tell the user to upgrade. Otherwise continue as usual.
The Setup
I have native iOS and Android apps which sync data to and from my webserver. A requirement of the apps is that they work offline so data is stored on the apps in sqlite databases.
The apps communicate with the server with a series of REST calls which send JSON from the server for the apps to store in their databases.
My Problem
The scale of this data is very large, some tables can have a million records, and the final size of the phone databases can approach 100mb.
The REST endpoints must limit their data and have to be called many times with different offsets for a whole sync to be achieved.
So I'm looking for ways to improve the efficiency of this process.
My Idea
An idea I had was to create a script which would run on the server which would create an sqlite file from the servers database, compress it and put it somewhere for the apps to download. Effectively creating a snapshot of the server's current data.
The apps would download this snapshot but still have to call their REST methods in case something had changed since the snapshot happened.
The Question
This would add another level of complexity to my webapp and I'm wondering if this is the right approach. Are there other techniques that people use when syncing large amounts of data?
This is a complex question, as the answer should depend on your constraints:
How often will data change? If it is too often, then the snapshot will get out of date really fast, thus apps will be effectively updating data a lot. Also, with the big volume of data, an application will waste CPU time on synchronization (even if user is not actively using all of that data!), or may become quickly out of sync with the server - this is especially true for iOS where Applications have very limited background capabilities (only small window, which is throttled) compared to Android apps.
Is that DB read-only? Are you sending updates to the server? If so, then you need to prepare conflict resolution techniques and cover cases, in which data is modified, but not immediately posted to the server.
You need to support cases when DB scheme changes. Effectively in your approach, you need to have multiple (initial) databases ready for different versions of your application.
Your idea is good in case there are not too many updates done to the database and regular means of download are not efficient (which is what you generally described: sending millions of records through multiple REST calls is quite a pain).
But, beware of hitting a wall: in case data changes a lot, and you are forced to update tens/hundreds of thousands of records every day, on every device, then you probably need to consider a completely different approach: one that may require your application to support only partial offline mode (for most recent/important items) or hybrid approach to data model (so live requests performed for most recent data in case user wants to edit something).
100mb is not so big. My apps have been synching many GBs at this point. If your data can be statically generated and upated , then one thing you can do is write everything to the server, (json, images, etc...) and then sync all on your local filesystem. In my case I use S3. At a select time or when the user wants to, they sync and it only pulls/updates what's changed. AWS actually has an API call called sync on a local/remote folder or bucket. A single call. I do mine custom, but essentially it's the same, check the last update date and file size locally and if it's different, you add that to the download queue.
So the app is for a Parrot Rescue. It will contain profiles for each of the birds that we currently have for adoption. So a picture of the bird, and then basic info about the bird. It needs to be regularly updated as the birds are adopted out and new ones come in. What route should I go? My main concern is that I want it to be relatively easy to update so that some of the less technically inclined could use it, as I won't always be available to help them.
Instead of loading a new version of the application every time, you should implement a database on a parrot information server. Then the application (the client) will ask the server to receive parrot information.
Just implement a mySQL database and a PHP page that returns the information in JSON format. Then through the application you will make a request to the server using the volley library or okHTTP and enter the information into a list.
As there are also pictures, use the glide library to download the images.
Also remember to upload small images to the server.
This way the app is always up to date
The easiest way is to update the app on the play store with the newest db, and let auto-update update the app with the new db.
Next easiest is to have a nightly (or whatever frequency) service download a new db and replace the old one from your servers
If you want it to be as easy as possible to update for everyone I would recommend a little app that has two main activities. One for displaying the entries of the database (and maybe also giving the ability to delete the entry) and another with a template to fill with all information and the image.
Then like Raffaele D'Arco explained use a little server. You could also delay the updates to be in a 5 minute routine or the like so that you put as little stress as possible on the server.
There are multiple good ways to update the interface without updating the app constantly.
Use a database which the app will download after a fixed amount of time which will be located on the server.
Secondly, if you have a website you can convert the web pages to XML or JSON or something like that and can update the app UI as per the data downloaded from the website.
I would recommend the second option as it is quite easy than the first one...
Background: we are porting an enterprise system to have android clients. The architecture for windows and html is based around a core library that does the hard business logic but no user interaction at all, and we use programs or single page web apps to provide the user interface and simply call the core API library to actually do stuff.
The "core" is implemented as a shared library on windows and built into each app. If we mirror this and use a java library, we need to share files using external storage, which is a not permitted as data needs to be reasonably secure. (Nb data is binary data, not Sql database, in case that is relevant)
So we thought about using a bound service, and using intents, content provider etc, but it seems (from googling) we must then distribute the background service separately the user interface app, but this seems terrible experience for new users. However, a bound service seems ideal from all other angles.
We also cannot guarantee which apps a user might download, we will have at least 10 individual apps all doing logically different things, but referencing similar data.
In brief:
lots of individual apps all wanting access to same data
no control over which apps are downloaded
using external data is not permitted as data should be semi secure
using sqllite might not work as data is long binary chunks ( eg 3Mb plus ). (Ref: How to share data across a group of applications in Android )
some data files are big and do not want every app to download a private copy
some data changes dynamically, say every 15min
core business logic is big and complex, cannot be distributed in source form, lib/jar ok though.
the windows solutions all use network IO to an application server, but we want to avoid as much network traffic as possible by storing data locally.
How can we bundle a bound service in each and every user interface app we distribute? Or is there a different way to approach this whole design?
I think that there is a few number of options that you can explore:
1) I never have done this before though this seems possible as Android is package based.
First you need to use the same main package across all your apps though each app must be in a separated sub package, e.g. : main -> au.com.myapp.main and the app actually have it's first screen on app1 -> au.com.myapp.main.app1 .
Create on your main app a method(s) that will look for those extra packages (within your project), as it find something you create a trigger that will display a item on the menu. Each app should have the same main packages and main activity, as it will be responsible for enable the user have access to the others and all of them can share the same preferences, files folders and Database.
When installing the same packages should be overrides though those different ones should keep intact. You should have all the 'main' classes for each app, not the real main one declared on your manifest (that will be quite big depending on the amount of activities in all your apps) with those packages.
2) You can using Spongy Castle, create a shared zone (folder) where you create the DB and write your settings or files, encrypting everything with a key (strong one or using RSA) that might be made by the user or provided once for your company at the very first run. You must decide how to handle this.
3) You also can use the share id in all your apps and each app before run perform look up for all packages (it's possible to do) to know and if and what packages exist to check if there is a DB with data in that package.
4) Not really nice though create a background service that keep updated all tables in all apps (sharing id or using content provider), you can user AlarmManager to trigger it in intervals rather keep it on at all times and you have all apps.
My last project had a similar requirement though as the user had to login to do anything, I opted for the option 3 and each data pertinent exclusively to each app went the app DB.
I hope this helps.
I am trying to decide on what data storage methods to implement. Here is the situation. Whatever method I choose, it is going to be updated once week (can I update a SQLite db without putting out an update in the market?). The user cannot add or remove items from this ListActivity, they can only pick the ones they want. This data method should be able to remember the selected items during any given week. Let me know what method you would use and why. Thanks so much in advance.
A webservice would allow you to update the data whenever you want without having to push updates to the market. And updating your app in the market doesn't guarantee that users will apply the update. Ofcourse the downside here is that your users would need to be connected to the internet while using the app.
Moving your database to remote server will give you freedom to manipulate data without actual application being updated, thus no need to update on the market. If it is a matter of access to internet, you can still use this practice, just more work has to be done (adding Broadcasters that will listen to connectivity than update the local database with global one on your server, or something similar).
If you want to update the data on the device once a week, then you will need to use the local SQLite database and interact with a web service that provides the updates. You will not need to go through the market to do this. However, if you need to update the structure of the database (add, remove, or change columns or tables for example), then you will need to update your app on the market.
I highly recommend watching the Google IO 2010 talk Developing Android REST client applications. The speaker is the author original author of the Twitter app for Android, and talks about the design patterns and best practices that he uses.