I'm writing an application that communicates with a web API, which responds with JSON. Currently, I'm translating the JSON objects to Java objects using gson (which is awesome, by the way).
Now, I want to store some of these objects in an SQLite database. However, they have lots of properties that would never be used in queries (i.e. I won't be ORDERing, WHEREing, or anything like that with those properties), so I feel it's unnecessary to create columns for all of them. What I'm thinking of doing is:
Only have columns for the essential data that will be used when querying the database
Have one TEXT or BLOB column (which one do you recommend?) that stores the actual JSON, so I can recreate my Java object from it and access all the data.
This would both make my life easier and streamline my code (I would not have to write very different code when dealing with data from the API vs. data from the database).
However, although I see no downsides, it feels a bit fishy.
What kind of trouble do you think I would run into if I use this technique?
The main thing I wouldn't like about it is relying on the structure of the stored/retrieved JSON to be valid, since it's completely out of the hands of the database. Not that you can't take precautions against possible issues, but if the JSON is somehow truncated or otherwise compromised in a way that trips up the parser, you're then missing the entire object instead of just one invalid or truncated property. If that's an acceptable risk, then it's probably a reasonable technique.
Related
I have been looking at different ways to hold onto some predefined character data, however I am having a hard time nailing down which would be the best solution.
An example of data would be 10 strings, 5 int arrays (of size 10 each). There would be 10+ set's of this data. The application would load in the information and inflate generic "character" objects.
Possible solutions:
XML: Due to Android's structured XML requirements it can be hard to use without making a different XML file for every character, and even then it would have ID overlapping for similar named data values.
SQLite: Wouldn't be a huge database, but databases are ugly version controlled unless it is done with a create-database script which has its own downsides (such as making sure DB is up to date between builds).
Hard-coded Objects: By far my least favorite solution, using polymorphism to hardcode all of the objects. Too dirty, not nearly as dynamic as it should be.
I would like to consider things such as version controlling the files, ease of updating (due to them only being inflated, never changed by the app).
If this data is baked i would suggest to use harcoded data.
Reasons.
In those three solution you save the data in the application.
If you use XML-data, you have to consume the time while code parsing inside the code. And you have to write the code that parses your xml.
If you use SQLite, your data will be doubled because of you have to store this database in raw or assest directory, copied in the /data/data folder. Futhermore, if you use Strings and SQLite by default the data will be doubled again (due to UTF-16 encoding).
If think, if only you manage the data this is more usefull to store directly inside the code. Obviously, if you do not use tons of content:)
You might want to use the Realm framework, which is comparatively faster than SQLite and easy to implement inside in your current code.
It handles large data too and it feels like you're using only native android classes.
I work on an application which contains complex data, like user list, images, long ArrayList etc. I look for advices to save properly the data. I wonder if marshalling was safe or do you suggest me to use SQLite ? Because if I save an user created object in a file, I think that I couldn't open it if the object class changed (for exemple if you add a field to your class).
I have no idea what to do exactly.
SQLite is very powerful, but not always necessary. You can get along just fine by serializing your data to disk using something like Gson in many cases. If you're dealing with lots of data, SQLite probably the best bet, but as you observed, it's inflexible at times. You'll have to map your model objects to tables and then write upgrade logic when the structure of your models change.
I recommend starting with serializing/de-serializing your data to/from the disk to start. If the amount of data you need to handle makes this unwieldily, migrate to SQLite. If you do go that direction, look into GreenDAO or OrmLite.
Use SQLite. It will process the data in the fastest way possible while creating the possibility to execute complex queries.
Long story short: I'm working on refactoring an old Android project of mine. Previously, it was using serialization, which was painfully slow and, from what I'm reading, a pretty lousy idea in general for Android apps. I'm looking for another way to persist both user-specific data as well and read-only data for the application.
There is going to be a good deal of data on both sides and I'm not sure if there's a "good" way to store it. Basically, the app is a small RPG. There are a number of "maps" that are represented as 2D arrays of Tiles. Each Tile will have a number of attributes, some simple primitives or enums, others additional objects, such as Events, which will also potentially contain various objects, etc. With 400 Tiles in a 20x20 map alone, there's a LOT of data to store. In addition to storing that data, it would need to store a lot of user-specific data, such as which Tiles have been visited, which Events have been successfully run, etc.
I've been examining methods of saving this data out and I just can't seem to settle on something. I guess it boils down to XML or JSON vs SQLite. XML or JSON would be more flexible in terms of future changes, which is good as I want flexibility in the data, ie, adding new attributes to existing objects, adding new objects as the need arises, etc. SQLite isn't as easily malleable as you have to change up the schema, perhaps adjust queries and indexes, etc, but I haven't really used SQLite in the past, so maybe there are some features that help to simplify that process. However, I would also like fast random access to data to avoid loading everything into memory at once if it can be helped. For example, when moving from one map to another, I'd much rather load the next map only when needed rather than having everything held in memory, which is where SQLite would shine as I'd be able to directly query the data rather than traversing a JSON/XML file to find potentially scattered data, ie, we load the map, but Events and objects contained in the Events may not be unique to that map and could easily lie elsewhere in the file or in another file entirely. However, normalizing the data in SQLite would mean a lot of tables and quite a bit of deconstructing/reconstructing of objects.
Writing user data would only occur when the user manually saves the game, so write performance is not a big issue.
I sometimes have a tendency to overanalyze and get hung up on stuff like this. Maybe neither case is necessarily "wrong" and I'm worrying about things that are infinitesimal. Maybe there are other cases that I haven't considered. I've used Hibernate and have considered something like ORMLite to handle a lot of the database nitty-gritty, but that would require a lot of retrofitting, likely much more than what I would need to do for other options.
I'd suggest you use SQLite. I think it makes the most sense considering the amount of data you're trying to store.
As far as your concerns with it not being as flexible, I would argue that point. Just use a ContentProvider. ContentProviders make it pretty easy to update the db schema and queries without affecting your existing functionality. If you use a ContentProvider, you could even swap out persistent data strategies in the future as well as use different ones simultaneously.
http://developer.android.com/guide/topics/providers/content-providers.html
I have an Android app that needs to work offline and requires a lot of static data.
Currently I'm using a JSON file in the /res/raw and loading it with the Jackson parser into my POJO scheme. It works really well since I have an external program that will be generating this data and once in a while when there is a change I'll just publish new version to the Market so I don't have to deal with running an update server and so on.
However, right now my JSON file is about 2.5MB with limited dataset for testing, in the end it'll be about 5-10MB.
The issue is that it already takes about 3-5 seconds to parse the file and this needs to be done every time the application is restarted.
So, what are my options here? I could put the data to a sqlite database, but that would require rewriting the external application and changing the data structure quite a bit. But then I could only query the things I need at the moment and not loading the entire thing at once.
Is there some easier/better way? Also, is there a good way to publish the app with the sqlite database? All the articles I've found talk about creating the database for user data at first startup, but this is not user data and I need it to be deployed from the Market.
JSON feels like the wrong approach for this - it's a good way to encode data to transfer, but that's pretty much it.
It'd be nice to have a bit more info on what exactly your app does, but I'm struggling to imagine a use-case where having several MB of POJOs in memory is an efficient solution. I think it'd be much better to use SQLite, and this is why:
I could put the data to a sqlite database, but that would require rewriting the external application and changing the data structure quite a bit.
You can still use your other program's JSON output, but instead of loading everything into POJOs with Jackson, you could populate the database on first app launch. This way, the app boot time is negligible if the dataset is unchanged.
If you still want to work with POJOs in the rest of your app, it'd be trivial to write a query that retrieved data from the database, and created objects in the same manner as Jackson.
But then I could only query the things I need at the moment and not loading the entire thing at once.
What're you doing that requires access to all the data at once? Searching or ordering a set of objects is always going to be slower than a SQL query to achieve the same thing.
Also, is there a good way to publish the app with the sqlite database?
You can definitely ship your app with a database, though I've not done so personally. This is a relevant question:
By Default load some data into our database sqlite
Hope that's of some help.
There's an excellent API called JExcel (just google it) that works with .xls spreadsheets. If you're not going to be doing any selecting and just loading data from a source, I like to use JExcel because it's more manageable from a desktop and produces easier-to-read code.
Not sure of any performance differences, however. Just throwing in my 2 cents :p
In my app I have to download JSON data from numerous web services. The data classes I use are fairly complex ones (lots of properties, quite deep inheritance tree, etc.).
I intend to do caching, using a single db table, where I'd store the downloaded JSON data in a VARCHAR column (along with other meta-data containing columns). JSON serialization is being done with the Gson library.
It seems quite convenient to just dump the instances into JSON, and parse them again later when I need them. No need to create custom tables for every class, or write loads of custom serialization code. Also, I can do queries on the cache table this way.
The question: Is this approach an anti-pattern by any means?
There is absolutely nothing wrong with this approach; however, I am going to recommend that you instead use the built in caching storage. See the section called "Saving cache files" in Data Storage for more details.This way you don't hog any precious space if your JSON objects are large in the event of a low memory situation.