I have more then 50 fields that is continiously writing in the .proto file but my query is that
1) if I need to read only 10 fields then how can this be achieve.
2) If I need to read partial data from the particular field then how can I achieve.
this should be done without loading all the data from the .proto file.
Thanks for your concern.
This is not really possible with Protobufs. In theory you could write a streaming parser that might be able to extract part of the message without parsing the whole thing, but it would only work if the fields you needed happened to be located towards the front of the message, since you'd at least have to parse through everything before the fields you want. In any case, none of the standard protobuf implementations provide an easy way to do streaming parses, because this isn't the way protobuf is designed to be used. Some third-party implementations, such as upb, might help.
On the other hand, Cap'n Proto, an alternative to Protocol Buffers, does support reading just one field out of a large file, without having to parse the fields before it. It does this by placing fields at fixed offsets and taking advantage of mmap() for large files.
Disclosure: I am the author of both Cap'n Proto and Protocol Buffers v2 (the version open sourced by Google).
Related
I want to give backup facility in my android application. So for that purpose i don't know which which format will be suitable. I am thinking either XML or CSV. Please tell me which is efficient.
It's my opinion that you're probably better off using JSON, as it has many great advantages as listed below and given your data, wont be considerably larger than CSV or Binary.
Take a look at this post for details on how to implement it:
How to parse JSON in Android
The following is a general breakdown of the different data format options:
XML
This format is the least efficient (file size and time to parse), but comes with the advantage that it can be easily debugged or modified/read by a person. In general, use this if you are going to be reading the content, displaying it in some other program or the file will be small enough that it's size and processing disadvantages don't have a significant effect.
JSON
More concise than XML, while still maintaining it's human readability. It's syntax isn't quite as simple as XML but it's still very simple. I would recommend this over XML.
CSV
This format is much more efficient than XML, but is prone to errors if modified manually and can be very hard to read. You will likely require special care in dealing with the delimiting character so you'll want to find yourself a simple CSV library. It's disadvantages are that although
Binary
These formats will be read/written to a file as bytes. They are structured in such a way that only your specific application/reader will know how how the bytes are structured. This format is the most concise and has the best read/write performance, but of course, it's practically impossible to modify or read.
Edit: Also worth considering is your ability to modify the format of the data, for the purposes of supporting future version changes. Using JSON or XML allows you to easily add new fields or ignore existing ones and so can be easier to maintain backwards compatibility for existing applications without breaking their functionality. A similar solution for CSV or Binary would require that you store and check the data format version number with the files, and then manually switch between loaders in code.
I'd go (and I use them in my apps) for CSV files, since the data are crude and concise (i.e.: small file size and fast to read/write).
I won't choose XML files, which put a lot of garbage in the file, bloating them ridicolously.
I have one file which is create in iPhone using serialization object. and i want to do read in android. Is it possible or not? please reply fast in advance thanks
I am assuming that the serialized data is in a pList.
See this http://code.google.com/p/plist/
You should be able to read any file generated by another language. It's better if there's a library out there for it. It'll get complicated if the file uses arrays, especially dynamic ones. This may require some fun time with a hex editor. I can't guarantee reliability with this method though if the data isn't simple. Just realize that this may end up being a tricky task
It's better to have a file spec made. You don't have to use a binary file, as you can use something like JSON or XML if it's all text-based information.
I would like some advice. I'm going to be using an sqlite database that will be pulling down information from my server and then saving it in the DB then displaying it. Could someone advise me of the best way to populate the DB, should I...
Use a http request and return a string de-liminated with say a | and use a loop to write to the data base.
Use a JSON to retrieve the information and then store it in the database.
The information is going to be just text and some fields will contains links to images I want to then download (get to that later). Just wanted some advice on best practices. I have done some searches on SO and other sites but can't find much advice. Also as a side note any examples you know of that are good for noobs :)
Based on what you write here I would pick JSON.
To core points:
JSON is a standard format.
Android ships with a JSON lib (org.json) making it easy to handle it (encode / decode data).
JSON is known by a large community so you can ask questions and get them answered rather easily. With a custom format you cannot tag the question as 'json' here at SO... ;-)
Using standard formats and libraries helps you to avoid designing and implementing this stuff yourself, which makes your software more robust.
Sometime later you might need to add more complex data to your project. By that time it will be rather straightforward to use JSON's array and objects. With your private scheme you will have to add this capability to it and extend your parsing code. That can easily introduce subtle bugs. Or you might decide at that point that it is too hard with your custom format and decide to move over to a standard like JSON, XML, etc. At that time it costs you much more to shift over than if you start with a standard format. Consider time invested to write and test the current code and then the extra time to change to the standard format for the current system.
I am a php/mysql developer learning android. I am creating an android app that receives info from my php app to create list views of different products which will open a web view of that product's detail.
Currently my php cms web application outputs xml lists for an iphone app.... (also, separately outputs html). I have full control of the php app so if there is a better way to output the data for the android app please let me know.
I have created code that reads the xml from the web and creates the list view. The list can be refreshed daily, so the data does not need to be read from the online xml every time the app starts.
So I was thinking to store the data retrieved locally to improve my apps responsiveness. there may be up to 500 product descriptions to be stored at any given time in up to 30 different xml lists. I am starting development with one xml list with about 30 products.
For best performance should i store the product info in a sqlLite db or should i store the actual xml file in the cache/db or some other method like application cache.
I also was think to create the update of the data as a service, would this be a good idea?
The most efficient way to store data is RAM. But if you want to cache it, then the most efficient way is Database.
I recommend you store your data in sqlite android database.
You could also consider zipping you xml for faster network transfer and unzipping through java.util.zip package classes. You could even consider a simpler format for transmitting data, less verbose than xml, using a datainput/outputstream.
(I do that in of my apps and it works great)
Here are some details on data input / output stream method :
imagine a proprietary protocol for your data, only what you need. No tags, no attributes, just raw values in order.
on the client side, get an input stream on your data using URL.getContent() and cast it in input stream.
on the client side still, build a data input stream encapsulating your socket input stream and read data in order. Use readInt, readDouble, readUTF, and so on.
on the client side, from php, you need to find a way to save your data in a format that is compatible with the data format expected by the client. I can't tell much about PHP, I only program using java.
The advantage of this technique is that you save bandwith as there is only data and no verbose decoration due to xml. You should read about java specs to understand how double, int, strings are written in data output stream. But it can be hard using two languages to get the data right.
If php can't save format in a suitable way, use xml, it will be much simpler. First try with just plain xml, then give a try using a zip or tarball or xml file.
But all this is about speed gain during network connection.
The second part of what you have to do is to store each row of your list in a SQL table. Then you can retrieve it pretty fast using a CursorAdapter for your list view (it breaks the charming MVC model but it is quite fast !).
Sorry about this, but it became too long to write as a comment. This is not intended to be an answer to your question, because in my opinion Stéphane answered very well. The best solution is indeed to store the data in an sqlite database. Then you need to create the class to be used as a connection between the data, the database and the app. I don't want to take credit for what is said here already (I, too, voted it up).
I'm concerned with the other suggestion (use of low level raw streams for data manipulation, the list steps on that answer). I strongly recommend you to avoid creating your own proprietary protocol. It goes like this:
I need to exchange data.
I don't want to deal with the hassle of integrating external APIs into my code.
I know I can write two 5 minute routines to read and write the data back and forth.
Therefore, I just created my own proprietary format for exchanging data!
It makes me cry whenever I need to deal with unknown, obscure and arbitrary sequence of data blobs. It's always good to remember why we should not use unknown formats:
Reinventing the wheel is counter-productive. It seems not, but on the middle term it is. You can adapt your project to other mediums (server-side, other platforms) easily.
Using off-the-shelf components help you scale your code later.
Whenever you need to adapt your solution to other technologies and mediums, you'll work faster. Otherwise, you would probably end up with ad hoc code solutions that are not (easily) extensible and interoperable.
Using off the shelf components enables you to leverage advances in that particular technology. That's particularly important when you are using Android APIs, as they are frequently optimized for performance later down the road (See Android's Designing for Performance). Rolling your own standards may result in a performance penalty.
Unless you document your protocol, it's extremely easy to forget the protocol you created yourself. Just give it enough time and it will happen: you'll need to relearn/remember. If you document, then you are just wasting the computational time of your brain.
You think you don't need to scale your work, but chances are you will most of the time.
When you do, you will wish you had learned how to easily and seamlessly integrate well known formats.
The learning curve is needed anyway. In my experience, when you learn, you actually integrate well known formats faster than imagining your own way of doing things.
Finally, trust your data to geniuses that take their lives into creating cohesive and intelligent standards. They know it better!
Finally, if the purpose is to avoid the typical verbosity of XML, for whatever reasons, you have several options. Right now I can think of CSV, but I'm no expert in data storage, so if you're not confortable with it, I'm sure you can find good alternatives with plenty of ready to use APIs.
Good luck!
I have a client server program to communicate the pc with an android phone using usb, based on http://www.anothem.net/archives/2010/10/15/android-usb-connection-to-pc/
Now I need to design a protocol that facilitates information exchange from the pc to the phone. How should I go about this ? Should we use XML for this? I was looking at google protocol buffer.. Is that the right direction ?
Thanks
Caroline
I've been using protocol buffers on Android. I avoided it for a long time, reasoning that it was overkill. That was before I took half a day to try it out.
Here were my results, after spending an afternoon on it. At first it increased my APK size from 2.89megs to 3.1 megs. I consider that inconsequential. Then I found that I was able to delete code I had all over the place, where I had been doing parsing manually. I was able to delete code that required the data to come in in a specific sequence.
Then I was able to completely delete a few classes I had in my application, whose purpose was to serve as temporary, lightweight information-only "model" classes that represented data coming to or from the data stream. In short, it started making things way easier and smaller, and more reliable.
Protocol Buffers may well be not the best thing for your situation. But I do recommend that you take a few hours and try it out. That way you'll be making your decision to leave it behind from a position of strength and knowledge.
Be sure to use the lite version of protocol buffers. The .jar file is 160k, but the amount of functionality it brings is huge. I'll be using it all the time from here on out.
I do have some concerns about dynamic memory allocation / garbage collection when using it in a game context long term. But for now the serialization happens infrequently enough that it's a non-issue for me.
Another bonus: I have some Python code that generates data files that the application reads. That python code processes some XML and then generates binary files. I think I'll be able to completely eliminate that code by using protocol buffer's text mode, then using protoc directly to create binary files.
You need first to design the higher layer of your protocol. Xml and protocol buffers have to do with how the data are formatted. Depending on the data you need to exchange they may or not be suitable. From what you way in your comment, it seems that you want to develop something like a remote control for your PC. In that case both XML and Google Protocol Buffers will be an overdesign. Simply create a text protocol, allocate a byte or two for the command type and some bytes for the data body.
Have a look at the AT commands structure. The structure of an HDLC frame could also give you some ideas. This has some things like error checking, which you don't need, but other than that a simple text protocol will use similar fields.