I'm currently facing a problem where I should store my object structure on the android device.
The usecase: I'm starting a call to an applicationserver (with the great help of AsyncTask), get a well known response (xml-response) from the server, parse the data and transform it finally into my object structure (highly complex class diagram with many associations between the classes). So far it's working, thanks to the great XMLPullParser ;)
I'm wondering where to store (and of course share) the fetched data between my activities... I already know that I can use sqlite, but I do not have an or-mapper (like hibernate in the j2ee environment). I'm also not allowed to store this sensitive data on the device (in sqlite or file system), so my first approach was to store this data in a singleton (which is of cource being held in memory...). But what happens when system is getting on low memory, can android "destroy" the data stored in my singleton? I already read about extending the android.app.Application class... So what is the best way to securely store object data (called from "webservice") on android devices?
BTW: Android development is that cool! We are currently porting a Windows Mobile 6.5 App to Android and iPhone, and my colleague (reponsible for iPhone-dev) is complaining all the time^^
Regarding an OR mapper, I came across OrmLite the other day. It is a general ORM tool for Java, but it also has some special adaptations that makes it work for Android. I haven't had time to test it myself yet, but it looks promising :)
As for storing sensitive data on the phone, you really don't have the option of storing it only in memory (using some kind of singleton as you suggested). As soon as your application goes to the background, it can be killed instantaneously, so you have to persist what data you want to keep in some way. That being said, if you save data to Internal Storage, this will not be available for any other app on the phone (given that the phone is not rooted, because if is rooted this is easy to get around). I do believe that this same goes for data you store using SQLite, but I'm not 100% certain, so I won't guarantee it.
But basically, if you are sure that your app will only run on non-rooted devices, you should be pretty safe saving your data to Internal Storage. And if that isn't good enough, there there is the javax.crypto package, but I've never used that so I can't really say anything about it.
Related
My app are sometime needed syncing with web servers and pull the data in mobile sqlite database for offline usages, so database size is keep growing exponentially.
I want to know how the professional app like whatsapp,hike,evernote etc manage their offline sqlite database.
Please suggest me the steps to solve this problem.
PS: I am asking about offline database (i.e growing in the size after syncing) management do not confuse with database syncing with web servers.
I do not know how large is your data size is. However, I think it should not be a problem storing reasonably large data into the internal memory of an application. The internal memory is shared among all applications and hence it can grow until the storage getting filled.
In my opinion, the main problem here is the query time if you do not have the proper indexing to your database tables. Otherwise, keeping the databases in your internal storage is completely fine and I think you do not have to be worried about the amount of data which can be stored in the internal storage of an application as the newer Android devices provide better storage capability.
Hence, if your database is really big, which does not fit into the internal memory, you might consider having the data only which is being used frequently and delete otherwise. This highly depends on the use case of your application.
In one of the applications that I developed, I stored some large databases in the external memory and copied them into the internal memory whenever it was necessary. Copying the database from external storage into internal storage took some time (few seconds) though. However, once the database got copied I could run queries efficiently.
Let me know if you need any help or clarification for some points. I hope that helps you.
For max size databases. AFAIK You don't want to loose what's on the device and force a reload.
Ensure you don't drop the database with each new release of your app when a simple alter table add column will work.
What you do archive and remove from the device give the user a way to load it in the background.
There might be some Apps / databases where you can find a documentation, but probably this case is limited and an exception.
So to know exactly what's going on you need to create some snapshots of the databases. You can start with that of one app only, or do it directly with several, but without analyzing you won't get a reliable statement.
The reasons might be even different for each app as databases and app-features differ naturally too.
Faster growth in size than amount of incoming content might be related to cache-tables or indexing for searches, but perhaps there exist other reasons too. Without verification and some important basic-info about it, it's impossible to tell you a detailed reason.
It's possible that table-names of a database give already some hints, but if tablenames or even fields just use meaningless strings, then you've to analyze the data inside including the changes between snapshots.
The following link will help in understanding what exactly Whatsapp is using,
https://www.quora.com/How-is-the-Whatsapp-database-structured
Not really sure if you have to keep all the data all the time stored on the device, but if you have a choice you can always use cloud services (like FCM, AWS) to store or backup most of the data. If you need to keep all the data on the device, then perhaps one way is to use Caching mechanisms in your app.
For Example - Using LRU (Least Recently Used) to cache/store the data that you need on the device, while storing the rest on the cloud, and deleting whats unneeded from the device. If needed you can always retrieve the data on demand (i.e. if the user tries to pull to refresh or on a different action) and delete it whenever its not being used.
I'd like to develop an app which need some data persistency. For some reason, I don't want to use database. And I'd like something similar to the shared preference. But only have one more requirement:
I hope my data can be persistent across installation. That means even the app uninstalled the data still be stored in Android system safely.
Are there any suggestions? Thanks
P.S.: In iOS, I use the keychain.It works perfectly.
The only solution I can think of to ensure non-removal on uninstallation is to write a file somewhere on the external storage. However this is (a) insecure (b) prone to other issues such as users deleting it.
If you are worried about persisting data on upgrades then using a database with a ContentProvider you can achieve this fairly easily. Shared Prefs are only really meant for a small number of small values/primitives.
There is also the option of storing something remotely, with a web service if it fits with whatever need you have to retain data when uninstalled/reinstalled.
The majority of questions like yours indicate an issue somewhere else in the way your application is architected, if you provide more detail people here may give you a better solution
The only way I can think of doing it is syncing the data with the cloud and restoring it if the user reinstalls the app. You may want to look at leveraging the BackupManager.
I am new to Android Application Development and a new member at stackoverflow. I am currently trying to design a recipe application. I have decided upon the features of the app and the scope it will cover. The scope is very vast for me in terms of covering all the recipes from all over the world. I am to deal with a lot of data in this process.
I am currently trying to figure a good and efficient way of handling the data in my app. So far, as per what I have read in different forums, I believe that I have two options in terms of a database choice : 1) SQLite 2) Database on remote server (MySql/Postgre)
Following are some of the thoughts that have been going on in my mind when it comes to taking a decision between the two :
1) SQLite : This could be a good option but would be slow as it would need to access the file system. I could eliminate the slowness by performing DB data fetch tasks in the AsyncTask. But then there could be a limitation of the storage on different phones. Also I believe using SQLite would be easier as compared to using a remote DB.
2) Remote Database : The issue that I can see here is the slowness with multiple DB requests coming at the same time. Can I use threads here in some way to queue multiple requests and handle them one by one ? Is there an efficient way to do this.
Also I have one more question in terms of the formatting of my data once I pull it out from the above DB's. Is there a way I could preserve the formatting of my data ?
I would be more than thankful if someone could share their knowledgeable and expert comments on the above scenario. Also this is not a homework for me and I am not looking for any ready made code solutions. I am just looking for hints/suggestions that would help me clear my thoughts and help me take a decision. I have been looking for this for sometime now but was not able to find concrete information. I hope I will get some good advice here from the experienced people who might have encountered similar situation.
Thanks for reading this long post.
What about combining both approaches?
A local SQLite database that has the least recently used receipes so you don't need network all the time. Network is way slower than accessing the filesystem.
Some remote database accessed via some HTTP interface where you can read / write the whole database. And if you want users to be able to add receipes for other users to see you'll need an external database anyways.
SQLite : This could be a good option but would be slow as it would need to access the file system.
Accessing a local database is pretty fast, 5ms or so if it's just a simple read only query on a small database.
But then there could be a limitation of the storage on different phones
Depends on your definition of huge database. It is okay if it is only 2MB which would be enough to store lots of text-only receipes.
Also I believe using SQLite would be easier as compared to using a remote DB.
Yes, Android has a nice built-in SQLite API but no remote database API. And you don't need to setup a database server & interface.
The issue that I can see here is the slowness with multiple DB requests coming at the same time.
A decent database server can handle thousands of requests. Depends on your server hardware & software. https://dba.stackexchange.com/ should have more info on that. Required performance depends on how much users you have / expect.
I'd suggest a simple REST interface to your database since it's pretty lightweight but does not expose your database directly to the web. There are tons of tutorials and books about creating such interfaces to databases. There are even hosted database services like nextDb that do most of the work for you.
Is there a way I could preserve the formatting of my data ?
You could store HTML formatted data in your database and display it in a WebView or a TextView (via Html#fromHtml()) - both can display formatted text.
Databases don't care what type of text you store, for transfer over the internets you may need to encode the text so it does not interfere with the transport formatting (XML, JSON, ...).
A simple way is to integrate Parse into your app. They have a nice framework that easily integrates into iOS and Android. Their plan is freemium, so you'll be able to use up to 1 million API request for no charge, and then its 7 cents for every request after that.
You'll have 1gb to store all your data sets / images, etc.
I don't use parse for everything, but I HIGHLY recommended it for large data schemes because they do all the scaling for you. Check out the API, I think it would be worth your time.
I just started to work on a few of my own projects, and I'm using Parse again. I have to say it's improved a lot over the last 6-8 months. Especially with the Twitter and Facebook integration.
The key issue here is the size of the data - any significant database of recipes would be too large to store on the phone imho,thus you seem stuck with the remote database solution.
As opposed to trying access the remote database from android I suggest you use a a go between web application that will process requests from the application and return JSON objects that you need.
It totally depends on your software requirements. If you need to deal with a small amount of data then you may choose SQLite, but for a huge amount to data better use a remote DB.
SQLite: It works fine with little amount of data & I experienced it response time is good.
Remote DB: I think you may use small server side app to submit the data to your client app. It will solve/reduce your thread related issues/complexities.
My app needs to store data on the phone, but I'm not sure what's the more efficient method. I don't need to search through the data or anything like that. I just need to be able to save the app's current state when it closes and restore when it's back up. There is between 1mb and 10mb worth of data that will need saving.
There are basically a bunch of custom classes with data in them, and right now I have them as Serializable, and just save each class to a file. Is there any reason for me to change that to store it in SQLite?
If you where to use sqlite you could save as you go, and know that whats in the DB is pretty much uptodate if the app/activity holding the data is suddenly killed by the os. Other that that I cant see and obvious reason to use sqlite for your use-case.
Also for the sql approach you have a clear cut way to change the structure of your domain objects at a later time and to migrate the data from a old to a new version of your database. This can be done using serialized objects as-well, but then the objects needs to be duplicated, both new and old at the same time. And that to me sounds very very scary and messy, especially considering that you might need to go from version x to y, so you might end up with some pretty tricky problems if you ever need to update the domain objects.
And I can honestly not see any benefits of using the flat-file/serialized approach.
You mention in your question that the data is only meant to save the state of the app, therefore my initial response would be to keep it on the devices especially since you mention that the file size would not be much more than 10MB, which is quite reasonable.
So my answer to you would be to keep it as is on the device. If your usage of the information changes in the future, you should then reconsider this approach, but for now it's totally logical.
If you're only saving serialized classes, you could use an ORM mapper as discussed in this thread . This saves you the inconvenience of writing your own mapper and is easily extendable to new classes. Also, if your requirements change, you COULD lookup data.
The only reasons for changing your system to SQLite would be more comfort and maybe a more foolproof system. E.g. now you have to check if the file exists, parse the contents etc. and if you'd use SQLite, you don't have to verify the integrity of the data and Android also helps you a little. And you could use the data for other causes, like displaying them in a ListView.
I have an Android app that needs to work offline and requires a lot of static data.
Currently I'm using a JSON file in the /res/raw and loading it with the Jackson parser into my POJO scheme. It works really well since I have an external program that will be generating this data and once in a while when there is a change I'll just publish new version to the Market so I don't have to deal with running an update server and so on.
However, right now my JSON file is about 2.5MB with limited dataset for testing, in the end it'll be about 5-10MB.
The issue is that it already takes about 3-5 seconds to parse the file and this needs to be done every time the application is restarted.
So, what are my options here? I could put the data to a sqlite database, but that would require rewriting the external application and changing the data structure quite a bit. But then I could only query the things I need at the moment and not loading the entire thing at once.
Is there some easier/better way? Also, is there a good way to publish the app with the sqlite database? All the articles I've found talk about creating the database for user data at first startup, but this is not user data and I need it to be deployed from the Market.
JSON feels like the wrong approach for this - it's a good way to encode data to transfer, but that's pretty much it.
It'd be nice to have a bit more info on what exactly your app does, but I'm struggling to imagine a use-case where having several MB of POJOs in memory is an efficient solution. I think it'd be much better to use SQLite, and this is why:
I could put the data to a sqlite database, but that would require rewriting the external application and changing the data structure quite a bit.
You can still use your other program's JSON output, but instead of loading everything into POJOs with Jackson, you could populate the database on first app launch. This way, the app boot time is negligible if the dataset is unchanged.
If you still want to work with POJOs in the rest of your app, it'd be trivial to write a query that retrieved data from the database, and created objects in the same manner as Jackson.
But then I could only query the things I need at the moment and not loading the entire thing at once.
What're you doing that requires access to all the data at once? Searching or ordering a set of objects is always going to be slower than a SQL query to achieve the same thing.
Also, is there a good way to publish the app with the sqlite database?
You can definitely ship your app with a database, though I've not done so personally. This is a relevant question:
By Default load some data into our database sqlite
Hope that's of some help.
There's an excellent API called JExcel (just google it) that works with .xls spreadsheets. If you're not going to be doing any selecting and just loading data from a source, I like to use JExcel because it's more manageable from a desktop and produces easier-to-read code.
Not sure of any performance differences, however. Just throwing in my 2 cents :p