I have an Android app that needs to work offline and requires a lot of static data.
Currently I'm using a JSON file in the /res/raw and loading it with the Jackson parser into my POJO scheme. It works really well since I have an external program that will be generating this data and once in a while when there is a change I'll just publish new version to the Market so I don't have to deal with running an update server and so on.
However, right now my JSON file is about 2.5MB with limited dataset for testing, in the end it'll be about 5-10MB.
The issue is that it already takes about 3-5 seconds to parse the file and this needs to be done every time the application is restarted.
So, what are my options here? I could put the data to a sqlite database, but that would require rewriting the external application and changing the data structure quite a bit. But then I could only query the things I need at the moment and not loading the entire thing at once.
Is there some easier/better way? Also, is there a good way to publish the app with the sqlite database? All the articles I've found talk about creating the database for user data at first startup, but this is not user data and I need it to be deployed from the Market.
JSON feels like the wrong approach for this - it's a good way to encode data to transfer, but that's pretty much it.
It'd be nice to have a bit more info on what exactly your app does, but I'm struggling to imagine a use-case where having several MB of POJOs in memory is an efficient solution. I think it'd be much better to use SQLite, and this is why:
I could put the data to a sqlite database, but that would require rewriting the external application and changing the data structure quite a bit.
You can still use your other program's JSON output, but instead of loading everything into POJOs with Jackson, you could populate the database on first app launch. This way, the app boot time is negligible if the dataset is unchanged.
If you still want to work with POJOs in the rest of your app, it'd be trivial to write a query that retrieved data from the database, and created objects in the same manner as Jackson.
But then I could only query the things I need at the moment and not loading the entire thing at once.
What're you doing that requires access to all the data at once? Searching or ordering a set of objects is always going to be slower than a SQL query to achieve the same thing.
Also, is there a good way to publish the app with the sqlite database?
You can definitely ship your app with a database, though I've not done so personally. This is a relevant question:
By Default load some data into our database sqlite
Hope that's of some help.
There's an excellent API called JExcel (just google it) that works with .xls spreadsheets. If you're not going to be doing any selecting and just loading data from a source, I like to use JExcel because it's more manageable from a desktop and produces easier-to-read code.
Not sure of any performance differences, however. Just throwing in my 2 cents :p
Related
I am currently learning android programming and creating an app that will store some integers representing user choices (values inserted several times a day, must be displayed in the results activity) and steps data collected Google Fit HISTORY Android APIs, also displayed in the results activity. I am looking for the most efficient way to store this data. I know that it might be possible to insert the custom data types in the GOOGLE fit database. However, I am not sure if it is a good idea if the app mostly works offline, and it needs to immediately represent only a small set of results, for example, the values inserted in the last 2 weeks, with step counts. On the other hand, I am not sure if it is ok to have two databases storing the data.
My apologies if the question sounds a bit too amateur, I am doing my best to find an optimal solution in terms of performance.
Thank you for your answers.
So, to give you my opinion and answer (mainly opinion)
Android has 3 ways (mainly) for storing data:
Files
Online database/API
Local database
for this specific scenario you have listed, wanting the data to be available offline, you should probably be looking at using Room: https://developer.android.com/training/data-storage/room, as it supports storing primitive types without having to write any type converters, you can store models and custom data as well, it uses very basic SQL (because it's a wrapper for the older Sqlite database methods) and is part of android (not an external 3rd party library). Room also requires most operations to be done off of threads, instead of main threads and this will improve your performance as well (also has support for livedata/rxjava to observe straight onto any changes as they happen)
However, as I told this user here:
Should i store one arrayList per file or should i store all my arrayList in the same file?
When starting out, don't worry about the best way for doing something, instead, try something out and learn from it, worrying about the best solution now is rather pointless, either way, happy learning and coding :P
I have been asked to create a tiny android app.
In everyday work i code for .NET and I have no experience connected with Android, but as it is a really small app I guess it's going to be a good experience rather than something hard.
The core of the app would be a small database (probably XML, unless somebody suggest better solution) that would contain categories, names of the institutions assigned with each category and logo (not very high resolution I guess a single file would be <100kB) of the institution.
The database also would not be very big - I expect not more than 1000 records in total. The DB has to be totally offline and local, it cannot require Internet access when operating.
The model I assume would be to ship new version of the application when the database changes (which is not going to be very frequent).
What is the best way to deal with these requirements?
My first idea was to create an XML file that would contain the records and link to the image. The XML and all the images linked to it would be stored in single file (preferably zip) that would be stored in app resources. This is very good as it is going to be very easy to update the database.
The second idea that somebody suggested me would be to use SQLite and store images in BLOB. In general I have read that it isn't a good idea to store images in database directly, and I am afraid if it's going to be possible to meet all requirements mentioned above.
Mostly I have no idea how to update the database easily and attach it to new version of application.
Any suggestions?
I would be grateful for any response.
I wouldn't go about using XML to save your data and by no means zip anything.
I think your way of thinking is ok, but you're making things really complicated for yourself.
Seeing as you're used to .NET I suppose you're also pretty confident with SQL, so I'd suggest you have a look at how to use the built-in SQLite database in Android.
If you would go the XML route you'd have to serialize and de-serialize the XML file over and over again and then parse the XML. Ok you don't have a lot of data, but searching inside an XML file with at least 1000 nodes would be slow in comparison to the performance of a database.
Also upgrading an existing SQLite database is not that hard - Android has methods for that (onUpgrade coming from the SQLiteOpenHelper).
As to saving images I'm assuming that you won't fetch new pictures from the Internet, so it would be best just to store them in the drawable folder of your app (be mindful of different screensizes) and then reading them into an ImageView when needed. To figure out what image should go for what institution I would store either the image name of each image in the SQLite database or store the resource id for each image in the database - for instance R.drawable.myawesomepictureformyinstitution.
I know my answer is somewhat "superficial", but your question is also somewhat "broad" and hard to answer without me actually writing most of the code, and that's not my intention ;-)
Hope this helps - let me know if anything is unclear.
I am looking to use an XML file to store the data my Android app generates. With that in mind, I have two questions for the community:
Is XML the best way to store data on Android and most efficient in cases where data may be added or altered every second or less then a second.
If XML is indeed the best for the scenario described in #1, how do I go about setting it up?
1.) Is XML the best way to database data on android and most efficient in cases where data may be added or altered every second or less then a second.
Definitely not.
2.) If XML is indeed the best for the scenario described in #1, how do I go about setting it up?
If you plan to store data just locally, the best way would be SQLite which works as a local database on every device.
If you later plan to synchronize this data with a central database, you may do this asynchronously within an AsyncTask or a Thread which would run periodically, but writing each second into a XML file is a bad idea as far as performance goes.
It's probably also a bad idea synchronizing a remote database at each insert/modification/deletion operation as if you had many users you could collapse the remote database.
I think the best approach is (as previously said) having a local database where you would store that data, and implement a webservice in the remote side if needed and use it to periodically synchronize both databases.
I would use JSON over XML and I would highly consider using GSON from Google. You maybe want to consider writing directly to a database with it's own structure and use transactions and sets. Is there are reason you want to go through JSON/XML?
XML is one of the worst ideas to keep local data in Android.
Most common used is SQLite available on the Android platform, but it all depends on what data and how you want to use.
In many mobile applications you don't need the relational database for one of the following reasons:
You have no relational data (i.e. settings) => no point in making relational tables with 1 record each
You have small, and dynamically changed data (like cache for downloaded content)
You don't need to search for data (using indexes etc.)
What alternatives can be used?
Shared preferences - simple key/value storage of primitive objects
Data serialization - for your consideration - binary (native java), JSON, parcelable (can be combined with the shared preferences)
For most of my app I'm currently using the binary serialization for "local storage".
- It's fast enough (usually much faster than starting the local SQLite engine)
- It's extremely easy and quick to implement, especially when you are using it for json/xml downloaded data parsed to POJO objects. All you need to do is just put "extends serializable" and put few lines of code to serialize/deserialize whole structure
- You can use those same classes for keeping data locally and communication with backend
Of course - it all depends from the situation - if you want to keep locally log of data from some sensor, or allow others apps to use this data, have to quick filter 1k+ records, or you really like to write few hundreds lines of code SQLite will be the best option for you. But most of mobile applications has no clear reason to use the relational (and trust me - not perfect one) engine.
I am new to Android Application Development and a new member at stackoverflow. I am currently trying to design a recipe application. I have decided upon the features of the app and the scope it will cover. The scope is very vast for me in terms of covering all the recipes from all over the world. I am to deal with a lot of data in this process.
I am currently trying to figure a good and efficient way of handling the data in my app. So far, as per what I have read in different forums, I believe that I have two options in terms of a database choice : 1) SQLite 2) Database on remote server (MySql/Postgre)
Following are some of the thoughts that have been going on in my mind when it comes to taking a decision between the two :
1) SQLite : This could be a good option but would be slow as it would need to access the file system. I could eliminate the slowness by performing DB data fetch tasks in the AsyncTask. But then there could be a limitation of the storage on different phones. Also I believe using SQLite would be easier as compared to using a remote DB.
2) Remote Database : The issue that I can see here is the slowness with multiple DB requests coming at the same time. Can I use threads here in some way to queue multiple requests and handle them one by one ? Is there an efficient way to do this.
Also I have one more question in terms of the formatting of my data once I pull it out from the above DB's. Is there a way I could preserve the formatting of my data ?
I would be more than thankful if someone could share their knowledgeable and expert comments on the above scenario. Also this is not a homework for me and I am not looking for any ready made code solutions. I am just looking for hints/suggestions that would help me clear my thoughts and help me take a decision. I have been looking for this for sometime now but was not able to find concrete information. I hope I will get some good advice here from the experienced people who might have encountered similar situation.
Thanks for reading this long post.
What about combining both approaches?
A local SQLite database that has the least recently used receipes so you don't need network all the time. Network is way slower than accessing the filesystem.
Some remote database accessed via some HTTP interface where you can read / write the whole database. And if you want users to be able to add receipes for other users to see you'll need an external database anyways.
SQLite : This could be a good option but would be slow as it would need to access the file system.
Accessing a local database is pretty fast, 5ms or so if it's just a simple read only query on a small database.
But then there could be a limitation of the storage on different phones
Depends on your definition of huge database. It is okay if it is only 2MB which would be enough to store lots of text-only receipes.
Also I believe using SQLite would be easier as compared to using a remote DB.
Yes, Android has a nice built-in SQLite API but no remote database API. And you don't need to setup a database server & interface.
The issue that I can see here is the slowness with multiple DB requests coming at the same time.
A decent database server can handle thousands of requests. Depends on your server hardware & software. https://dba.stackexchange.com/ should have more info on that. Required performance depends on how much users you have / expect.
I'd suggest a simple REST interface to your database since it's pretty lightweight but does not expose your database directly to the web. There are tons of tutorials and books about creating such interfaces to databases. There are even hosted database services like nextDb that do most of the work for you.
Is there a way I could preserve the formatting of my data ?
You could store HTML formatted data in your database and display it in a WebView or a TextView (via Html#fromHtml()) - both can display formatted text.
Databases don't care what type of text you store, for transfer over the internets you may need to encode the text so it does not interfere with the transport formatting (XML, JSON, ...).
A simple way is to integrate Parse into your app. They have a nice framework that easily integrates into iOS and Android. Their plan is freemium, so you'll be able to use up to 1 million API request for no charge, and then its 7 cents for every request after that.
You'll have 1gb to store all your data sets / images, etc.
I don't use parse for everything, but I HIGHLY recommended it for large data schemes because they do all the scaling for you. Check out the API, I think it would be worth your time.
I just started to work on a few of my own projects, and I'm using Parse again. I have to say it's improved a lot over the last 6-8 months. Especially with the Twitter and Facebook integration.
The key issue here is the size of the data - any significant database of recipes would be too large to store on the phone imho,thus you seem stuck with the remote database solution.
As opposed to trying access the remote database from android I suggest you use a a go between web application that will process requests from the application and return JSON objects that you need.
It totally depends on your software requirements. If you need to deal with a small amount of data then you may choose SQLite, but for a huge amount to data better use a remote DB.
SQLite: It works fine with little amount of data & I experienced it response time is good.
Remote DB: I think you may use small server side app to submit the data to your client app. It will solve/reduce your thread related issues/complexities.
My app needs to store data on the phone, but I'm not sure what's the more efficient method. I don't need to search through the data or anything like that. I just need to be able to save the app's current state when it closes and restore when it's back up. There is between 1mb and 10mb worth of data that will need saving.
There are basically a bunch of custom classes with data in them, and right now I have them as Serializable, and just save each class to a file. Is there any reason for me to change that to store it in SQLite?
If you where to use sqlite you could save as you go, and know that whats in the DB is pretty much uptodate if the app/activity holding the data is suddenly killed by the os. Other that that I cant see and obvious reason to use sqlite for your use-case.
Also for the sql approach you have a clear cut way to change the structure of your domain objects at a later time and to migrate the data from a old to a new version of your database. This can be done using serialized objects as-well, but then the objects needs to be duplicated, both new and old at the same time. And that to me sounds very very scary and messy, especially considering that you might need to go from version x to y, so you might end up with some pretty tricky problems if you ever need to update the domain objects.
And I can honestly not see any benefits of using the flat-file/serialized approach.
You mention in your question that the data is only meant to save the state of the app, therefore my initial response would be to keep it on the devices especially since you mention that the file size would not be much more than 10MB, which is quite reasonable.
So my answer to you would be to keep it as is on the device. If your usage of the information changes in the future, you should then reconsider this approach, but for now it's totally logical.
If you're only saving serialized classes, you could use an ORM mapper as discussed in this thread . This saves you the inconvenience of writing your own mapper and is easily extendable to new classes. Also, if your requirements change, you COULD lookup data.
The only reasons for changing your system to SQLite would be more comfort and maybe a more foolproof system. E.g. now you have to check if the file exists, parse the contents etc. and if you'd use SQLite, you don't have to verify the integrity of the data and Android also helps you a little. And you could use the data for other causes, like displaying them in a ListView.