We are currently writing an app for mobile platforms which is going to use a quite sophisticated data structure.
The data structure is organized in a tree structure and should contain both data and text.
Of course XML is a really good choice to store data organized in a tree structure. Our problem is that we need raw data blobs (a couple of MBs in size) to be written into the file.
When using XML, the blobs need to be encoded in base64 or similar which will significantly increase the file size ...
Thus, we considered creating our own binary format.
We don't have any experience in designing file formats and we are having a hard time figuring out how to get a tree structure and also variable length data blocks in a self-designed file format.
We need this to run on iOS and Android.
Does anyone have experience in how to do this or what to use for this?
Related
I am developing an Android app that will display a list of PDF documents. The document list is a JSON file (which is currently used in a web application). My question is: does it make more sense to store the JSON file as an asset or store the JSON as stings in a variable in a class? The JSON is static and I only read from the file, there's no writing.
I don't think it really matters - an assets file is a little more complicated to read, but it's not a huge deal.
But it might be easier to work with - you can just replace the file, edit it in whatever external editor you want, maybe generate it automatically from some other tool. That's easier with a separate file than something that has to edit a string within a Java file or whatever, you know? Plus JSON can be awkward to work with without a nice editor checking it for you
And it makes testing easier - you can write it so you provide a different source file, have a different set of assets etc, which you can't do so easily if you hardcode the JSON.
And you'll probably get better build times if you're just modifying an external assets file - no changes to the actual code files means they don't need to be rebuilt.
Most of this stuff probably doesn't matter! Just laying out the stuff that could be a consideration - but if the string form is working out fine for you, why complicate things?
I am about to implement backup for my Android app, and my issue is: The data resides in an sqlite db. Some of the data are just there for user convenience, and can be recreated from other sources. So in order to minimize the size, I wish to export relevant data only (the limit for backup using the Google API is 1mb).
All the data has class equivalents, which are populated via my SQLiteOpenHelper implementation. This means I can implement serialization.
So far I can see the following options:
Serialization using Java Serializable, and write all objects into a binary chunk and pass it to writeEntityData()
Serialization using XML or JSON, perhaps together with the zip API and dump the file as a binary chunk
Clone the database with relevant objects only. Probably a lot of work.
So far, using XML or JSON seems to be the best option, as I can reuse that for data sharing across users/devices. Java Serializable seems to bloat the size..
Would like to hear your opinions on this !
I recommend you to use vacuum() in order to shrink the db size, or use the auto_vacuum pragma.
If your db is something big, you can try to zip it.
Delete all unnecessary libs (compat_v7, for example, if you don't need it).
Try to compress images with optipng.
Try to convert your wav or mp3 to aac.
And... we are talking about how many Mb?
I have more then 50 fields that is continiously writing in the .proto file but my query is that
1) if I need to read only 10 fields then how can this be achieve.
2) If I need to read partial data from the particular field then how can I achieve.
this should be done without loading all the data from the .proto file.
Thanks for your concern.
This is not really possible with Protobufs. In theory you could write a streaming parser that might be able to extract part of the message without parsing the whole thing, but it would only work if the fields you needed happened to be located towards the front of the message, since you'd at least have to parse through everything before the fields you want. In any case, none of the standard protobuf implementations provide an easy way to do streaming parses, because this isn't the way protobuf is designed to be used. Some third-party implementations, such as upb, might help.
On the other hand, Cap'n Proto, an alternative to Protocol Buffers, does support reading just one field out of a large file, without having to parse the fields before it. It does this by placing fields at fixed offsets and taking advantage of mmap() for large files.
Disclosure: I am the author of both Cap'n Proto and Protocol Buffers v2 (the version open sourced by Google).
I want to give backup facility in my android application. So for that purpose i don't know which which format will be suitable. I am thinking either XML or CSV. Please tell me which is efficient.
It's my opinion that you're probably better off using JSON, as it has many great advantages as listed below and given your data, wont be considerably larger than CSV or Binary.
Take a look at this post for details on how to implement it:
How to parse JSON in Android
The following is a general breakdown of the different data format options:
XML
This format is the least efficient (file size and time to parse), but comes with the advantage that it can be easily debugged or modified/read by a person. In general, use this if you are going to be reading the content, displaying it in some other program or the file will be small enough that it's size and processing disadvantages don't have a significant effect.
JSON
More concise than XML, while still maintaining it's human readability. It's syntax isn't quite as simple as XML but it's still very simple. I would recommend this over XML.
CSV
This format is much more efficient than XML, but is prone to errors if modified manually and can be very hard to read. You will likely require special care in dealing with the delimiting character so you'll want to find yourself a simple CSV library. It's disadvantages are that although
Binary
These formats will be read/written to a file as bytes. They are structured in such a way that only your specific application/reader will know how how the bytes are structured. This format is the most concise and has the best read/write performance, but of course, it's practically impossible to modify or read.
Edit: Also worth considering is your ability to modify the format of the data, for the purposes of supporting future version changes. Using JSON or XML allows you to easily add new fields or ignore existing ones and so can be easier to maintain backwards compatibility for existing applications without breaking their functionality. A similar solution for CSV or Binary would require that you store and check the data format version number with the files, and then manually switch between loaders in code.
I'd go (and I use them in my apps) for CSV files, since the data are crude and concise (i.e.: small file size and fast to read/write).
I won't choose XML files, which put a lot of garbage in the file, bloating them ridicolously.
I need to download onto my devices some data in multiple files.
Then this data will be copied to application's local db (this is SQLite db, however in future this may be Compact SQL on WInPhone).
What is the best format for such files?
I am considering such possibilities:
SQLited db file - possibly this will be easy to copy to my db. My current prefferance.
JSON format. Maybe not enough compact because column name will be repeating.
CSV - it allows to store only one table but I would prefer have few tables in one file
XML - I do not see any prefferaces over json.
JSON is the most popular, human-readable, easy to use format. There's tons of supporting libraries, native and not, for all OSs. It's fast and reliable. You can easily update the data you pass with it without updating the apps (which you cannot by passing an SQLite database and would be difficult with a CSV file). XML is being slowly deprecated for data communications... but if you see some special advantage with XML (parsing the XML directly, which is not as effective with JSON yet, for example), go for it. I'd choose JSON anyway, it's the current standard and will still be for a long time.