So I have an application that logs medical data entered by user. I need to add functionality for backing up and restoring that data in case user changes their Android phone OR switches to and iPhone (there is an iPhone version of this app, which I have limited visibility into).
The data is currently stored inside a SQLite DB on the device.
I need help figuring out cleanest/easiest way to do this. So far I am thinking about these options:
add UI to email DB file to + UI to accept a DB file and copy it over current DB. Seems hacky.
create a web service which will store user data and sync on the background. I would need to build sync process and introduce some kind of account system. This seems like quite a bit of work, though likely the most flexible solution in the long run.
switch to Google Calendar as my data storage (data is essentially a set of event entries anyway). This would probably be most seamless, but iPhone option is out of the window.
Are there other pros/cons to these options that I am missing? Or perhaps there are some standard solutions to this?
Option #1 is filled with a lot of problems. Sideloading databases from files coming from external sources is no easy task. I wouldn't recommend it.
Option #3 should be a possibility - but it require the user to set up a Google account which (imho) shouldn't be required. Furthermore the addition of a new calendar and/or a lot of events wouldn't be many users favorite feature. Personally I would hate it.
The second option seems like to most promising. You can build it as simple as you like. You could even make it require only a simple identification code for backing up - or only provide "live" synchronization. It is true that you need some sort of server and accounting system - but these things come as standards and servers are cheap. This option would be the one of my choice.
Related
The Setup
I have native iOS and Android apps which sync data to and from my webserver. A requirement of the apps is that they work offline so data is stored on the apps in sqlite databases.
The apps communicate with the server with a series of REST calls which send JSON from the server for the apps to store in their databases.
My Problem
The scale of this data is very large, some tables can have a million records, and the final size of the phone databases can approach 100mb.
The REST endpoints must limit their data and have to be called many times with different offsets for a whole sync to be achieved.
So I'm looking for ways to improve the efficiency of this process.
My Idea
An idea I had was to create a script which would run on the server which would create an sqlite file from the servers database, compress it and put it somewhere for the apps to download. Effectively creating a snapshot of the server's current data.
The apps would download this snapshot but still have to call their REST methods in case something had changed since the snapshot happened.
The Question
This would add another level of complexity to my webapp and I'm wondering if this is the right approach. Are there other techniques that people use when syncing large amounts of data?
This is a complex question, as the answer should depend on your constraints:
How often will data change? If it is too often, then the snapshot will get out of date really fast, thus apps will be effectively updating data a lot. Also, with the big volume of data, an application will waste CPU time on synchronization (even if user is not actively using all of that data!), or may become quickly out of sync with the server - this is especially true for iOS where Applications have very limited background capabilities (only small window, which is throttled) compared to Android apps.
Is that DB read-only? Are you sending updates to the server? If so, then you need to prepare conflict resolution techniques and cover cases, in which data is modified, but not immediately posted to the server.
You need to support cases when DB scheme changes. Effectively in your approach, you need to have multiple (initial) databases ready for different versions of your application.
Your idea is good in case there are not too many updates done to the database and regular means of download are not efficient (which is what you generally described: sending millions of records through multiple REST calls is quite a pain).
But, beware of hitting a wall: in case data changes a lot, and you are forced to update tens/hundreds of thousands of records every day, on every device, then you probably need to consider a completely different approach: one that may require your application to support only partial offline mode (for most recent/important items) or hybrid approach to data model (so live requests performed for most recent data in case user wants to edit something).
100mb is not so big. My apps have been synching many GBs at this point. If your data can be statically generated and upated , then one thing you can do is write everything to the server, (json, images, etc...) and then sync all on your local filesystem. In my case I use S3. At a select time or when the user wants to, they sync and it only pulls/updates what's changed. AWS actually has an API call called sync on a local/remote folder or bucket. A single call. I do mine custom, but essentially it's the same, check the last update date and file size locally and if it's different, you add that to the download queue.
My Android app is fetching data from the web (node.js server).
The user create a list of items (usually 20-30 but it can be up to 60+). For each item I query the server to get information for this item. Once this info is fetched (per item), it won't change anymore but new records will be added as time go by (another server call not related to the previous one).
My question is about either storing this info locally (sqlite?) or fetching this info from the server every time the user asks for it (I remind you the amount of calls).
What should be my guidelines whether to store it locally or not other than "speed"?
You should read about the "offline first" principles.
To summarize, mobile users won't always have a stable internet connection (even no connection at all) and the use of your application should not be dependant on a fulltime internet access.
You should decide which data is elligible for offline storage.
It will mainly depend on what the user is supposed to access most often.
If your Items don't vary, you should persist them locally to act as a cache. Despite the fact that the data mayn't be really big, users will welcome it, as your app will need less Internet usage, which may lead to long waits, timeouts, etc.
You could make use of Retrofit to make the calls to the web service.
When it comes to persisting data locally within an Android application, you can store it in several ways.
First one, the easiest, is to use Shared Preferences. I wouldn't suggest you this time, as you're using some objects.
The second one is to use a raw SQLite database.
However, I'd avoid making SQL queries and give a try to ORM frameworks. In Android, you can find several, such as GreenDAO, ORMLite, and so on. This is the choice you should take. And believe me, initially you might find ORMs quite difficult to understand but, when you learn how do they work and the benefits they provide us, you'll love them.
Background: we are porting an enterprise system to have android clients. The architecture for windows and html is based around a core library that does the hard business logic but no user interaction at all, and we use programs or single page web apps to provide the user interface and simply call the core API library to actually do stuff.
The "core" is implemented as a shared library on windows and built into each app. If we mirror this and use a java library, we need to share files using external storage, which is a not permitted as data needs to be reasonably secure. (Nb data is binary data, not Sql database, in case that is relevant)
So we thought about using a bound service, and using intents, content provider etc, but it seems (from googling) we must then distribute the background service separately the user interface app, but this seems terrible experience for new users. However, a bound service seems ideal from all other angles.
We also cannot guarantee which apps a user might download, we will have at least 10 individual apps all doing logically different things, but referencing similar data.
In brief:
lots of individual apps all wanting access to same data
no control over which apps are downloaded
using external data is not permitted as data should be semi secure
using sqllite might not work as data is long binary chunks ( eg 3Mb plus ). (Ref: How to share data across a group of applications in Android )
some data files are big and do not want every app to download a private copy
some data changes dynamically, say every 15min
core business logic is big and complex, cannot be distributed in source form, lib/jar ok though.
the windows solutions all use network IO to an application server, but we want to avoid as much network traffic as possible by storing data locally.
How can we bundle a bound service in each and every user interface app we distribute? Or is there a different way to approach this whole design?
I think that there is a few number of options that you can explore:
1) I never have done this before though this seems possible as Android is package based.
First you need to use the same main package across all your apps though each app must be in a separated sub package, e.g. : main -> au.com.myapp.main and the app actually have it's first screen on app1 -> au.com.myapp.main.app1 .
Create on your main app a method(s) that will look for those extra packages (within your project), as it find something you create a trigger that will display a item on the menu. Each app should have the same main packages and main activity, as it will be responsible for enable the user have access to the others and all of them can share the same preferences, files folders and Database.
When installing the same packages should be overrides though those different ones should keep intact. You should have all the 'main' classes for each app, not the real main one declared on your manifest (that will be quite big depending on the amount of activities in all your apps) with those packages.
2) You can using Spongy Castle, create a shared zone (folder) where you create the DB and write your settings or files, encrypting everything with a key (strong one or using RSA) that might be made by the user or provided once for your company at the very first run. You must decide how to handle this.
3) You also can use the share id in all your apps and each app before run perform look up for all packages (it's possible to do) to know and if and what packages exist to check if there is a DB with data in that package.
4) Not really nice though create a background service that keep updated all tables in all apps (sharing id or using content provider), you can user AlarmManager to trigger it in intervals rather keep it on at all times and you have all apps.
My last project had a similar requirement though as the user had to login to do anything, I opted for the option 3 and each data pertinent exclusively to each app went the app DB.
I hope this helps.
Consider the following scenario. A company releases many apps. And they want some data to be shared across all these apps. Any of these app can create or read these data, just like a common database. So company decided to create an android library which does this purpose. I searched for a few days and my analysis are given below.
SharedPreferences- not recommended and is deprecated. It does not serve the purpose too. All other apps need to know the package name of the app that created the data to create PackageContext. Here this is impractical as any app can create/update/read data and it is not possible to say who is who.
ContentProviders - This does not work for me. The reason being ContentProviders has to be present in each app. There can not be 2 content providers with same name in a device. In addition to that, ContentProviders are basically meant for one app creates data and other apps subscribe to it using Content_Uri.
Network connection - We do not want to do store data in any server.
External storage - This is the only option remaining. Should I go for this?
And interestingly the data has to be secured as well which is nowhere supported in any of the storage options.
Note: For iOS, we use keychain to implement the same functionality
Understanding the problem on Android
Ironically, due to the intense sandboxing on iOS, there's a straightforward way to make this happen there (App Groups) provided the apps that need to share data are all by the same developer. Likely because Android is more flexible on security, this actually ends up being a more difficult problem there. The Android team have so far not seen fit to provide a convenient and secure way to specifically share this kind of data because there's a low security workaround.
That said, there are plenty of ways to share data between applications without involving the cloud.
SharedPreferences
The original question states that SharedPreferences are deprecated. This isn't true, as far as I can tell, however the MODE_WORLD_READABLE and MODE_WORLD_WRITABLE contexts are deprecated which makes this approach subject to not working in the future. The mode has been deprecated for quite some time, though - since Android 4.2 (2012). There's no threat in the current Android docs to suggest they're actually phasing it out (sometimes deprecation just means "this isn't a great idea" not "this is going to be removed"). I suspect the lack of a more secure OS-level direct alternative for application data sharing at the settings level is probably the reason for preserving it in a state of deprecation for the last 5 years.
File Access
The simplest and most common way I'm aware of to implement data sharing between applications on Android is to simply request file access on the device and create a shared location on external storage for this data. (Don't be confused by the "external storage" designation - this is just how Android refers to shared data. It doesn't necessarily refer to an SD card.) You give the file a unique name, and you store it somewhere that your apps know where to look for it. Best way to get that path is something like:
Environment.getExternalStoragePublicDirectory(Environment.DIRECTORY_DOCUMENTS)
The obvious problem with this is security. While not deprecated by the OS, it introduces the same problem that the Android docs list as the reason for deprecating MODE_WORLD_* - it's inherently insecure and opens up potential exploits in your application.
You're placing your information where everything has ready access to
it.
You're asking for permissions that your app may not otherwise need.
You're reading files that you can't verify the origin of.
If your application isn't handling any sensitive data, maybe this doesn't matter to you (it might to your users). If you're planning to read data from those files, you should ensure you're providing maximum validation for that data before parsing. Check the size of the file, validate the formatting, etc.
Creating your own service
You could always create a Service or an IntentService. (There are subtle differences between the two, but IntentService is a subclass of Service that runs in a Worker thread while Service interrupts the main thread. IntentService also implements Intent support which provides the most straightforward interapplication communication on Android).
This service has its own private storage, for which it has full read/write access, but nothing else does. This service then provides an interface to receive Intents from other apps, and to return results (as Intents) to those apps. This is an extremely friendly way to implement interapplication data while maximizing data privacy and security of that data. If outlying apps mostly need to request very basic information from a central application, this is your entry-level option.
Implementing a BroadcastReceiver
Along the same lines is the BroadcastReceiver class. Depending on what sort of data you're intending to share between applications, and how familiar those applications may be with your specific approach, this another possibility. Again, you'll be managing the shared data under one application's private storage. Communication is done by Intents, so it's similar to an IntentService - except that applications may communicate with a BroadcastReceiver by issuing systemwide events (that is, they don't need to be explicitly communicating with your app or service - they're shouting out the world for a piece of info, and expecting an answer.)
Creating a ContentProvider
The Original Post seems to misunderstand what a ContentProvider is and how it works. You have to think of this type of item like you would a cloud solution - even though it's local to your device. Every app doesn't need a ContentProvider - they all need to communicate with a ContentProvider, and that ContentProvider maintains, updates and returns data.
This is probably the most "Android-y" solution for this particular usecase and offers the greatest expandability. You implement an independent process that handles data storage and responds to other applications. It is, however, a more evolved solution - and as such may be more a more challenging endeavor. If you need a real database service, rather than a fairly simple request/response type service, ContentProvider seems to be the best choice.
Seems like what you need is shared user id.
It allows application sandbox to be shared across multiple android applications if they are all signed by the same signature.
But, watch out for gotchas!
Yes. Probably using same path in external storage for all applications is the best way. A common portion of code could be used to know whether database there exists or not and therefore open or create new one. For security I recommend always to use 'user' and 'password' when connecting to DB, but if you think it is not sufficient I advise you to see this: http://www.hwaci.com/sw/sqlite/see.html
I'm about to build a GPS Spot Finder application with Android and I am trying to decide what requirements are feasible and what aren't. The app would enable users to essentially add different spots on a Google Map. One of the problems would be fetching the data, adding new spots, etc, etc. This, of course would mean the database would have to be online and it would have to be central. My question is, what kind technologies would I need to make this happen? I am mostly familiar with XAMPP, PHPMyAdmin and the like. Can I just use that and connect Android to the database? I assume I would not need to create a website...just the database?
What different approaches can I take with this? Be great if people can point me in the right direction.
Sorry if I don't make any sense and if this type of question is inappropriate for Stackoverflow :S
Create a website to access the database locally, and have Android send requests to the website.
If users are adding spots to a map that only they see, then it makes sense to keep the data local to Android using a built-in database (SQLite). That looks like
ANDROID -> DATABASE
You can read up about SQLite options here.
If users need to see all the spots added by all other users, or even a subset of spots added by users, then you need a web service to handle queries to the database: Connect to a remote database...online database
ANDROID -> HTTP -> APPLICATION SERVER -> DATABASE
Not only is trying to interface directly to a database less stable, but it may pose risks in terms of security and accessibility.
Never never use a database driver across an Internet connection, for any database, for any platform, for any client, anywhere. That goes double for mobile. Database drivers are designed for LAN operations and are not designed for flaky/intermittent connections or high latency.
Additionally, Android does not come with built in clients to access databases such as MySQL. So while it may seem like more work to run a web service somewhere, you will actually be way better off than trying to do things directly with a database. Here is a tutorial showing how to interface these two.
There is a hidden benefit to using html routes. You will need a programming mindset to think through what type of data is being sent in the POST and what is being retrieved in the GET. This alone will improve your application architecture and results.
Why not try using something that is already built into android like SQLite? Save the coordinates of these "spots" into a database through there. This way, everything is local, and should be speedy. Unless, one of your features is to share spots with other users? You can still send these "spots" through different methods other than having a central database.
And yes, you just need an open database, not a website, exactly. You could technically host a database from your home computer, but I do not suggest it.
If you are looking at storing the data in your users mobile nothing better than built in SQLLite.
If you are looking at centralized database to store information, Parse.com is a easy and better way to store your user application data in centralized repository.
Parse.com is not exactly a SQL based database, However you can create table , insert / update and retrieve rows from android.
Best part is it is free upto 1GB. They claim 400,000 apps are built on Parse.com. I have used few of my application typically for user management worked great for me.