I am looking to develop a transit app using GTFS static data. One of the constraints I've set to myself is that the app should use minimal mobile data transfers. Therefore, I would like to embed all the data in the app.
My issue is that GTFS data sets are usually quite large (85MB uncompressed for the city of Sydney for example). I've done a bit of reverse engineering on other apps out there and found out that some of them have managed to compress all that data into a much smaller file (I'm talking about a few MB at most).
Using 7zip, I've managed to compress my 85MB data set down to 5MB which is acceptable for me. The next step is for me to use that 7z file into my app and that's where I'm stuck. There's no way I'm going to uncompress it and put it in a SQL database as that will use too much space on the phone. So I was wondering what are my other options.
Thanks
First, for embedding, I recommend using the Embedded XZ library (similar to 7zip). I have embedded this in a project and had good luck with it. Just be sure to compress data using 'xz --check=crc32' so it's compatible with Embedded XZ, and remember to initialize the CRC table.
As for a decompression strategy, you may need to segment the data in such a way that you can decompress different parts of it on demand (i.e., a tree of databases). I'm not familiar with your data's characteristics. Will a user need it all loaded at the same time? Or can it easily be compartmentalized?
Also, XZ can be a bit slow, even to decode. Have you evaluated how well regular gzip performs? That tends to be A) very fast; and B) available as a standard part of all embedded and mobile frameworks.
Use protocol binary format (pbf) formely google and now open source. It is compact and very fast searchable, so no need to decompress it on a device and load it into a database on that device because pbf acts as a database. Just include pbf library in your code to query it. Of course you have to compress it once before distributing the data online.
Related
I am building an android app which requires to have >300 images and ca. 100 audio files for an apk size of around 50mb (after webp compression and proguard).
It is not huge and I could probably live with that. But since I am planning to add other features the size will get bigger and bigger.
I am still interested though, since I'm quite new to android development, if there is a better way to store all this files, perhaps remotely and access them when required.
When the app starts I would have to load all the images into a list and once an element of the list is tapped I would need to open a separate activity and load the sound. So there is no upload from the App, just a resource gathering.
I do not know if it is more efficient this way or to store all the files locally.
Either way I would like to know what my options are. what are the pros and cons of a server (and if it would be a viable solution for me at all) and what is the advantage of storing them locally instead.
Please keep in mind that I working by myself and haven't got money to invest on premium servers or stuff like that.
FILE STORAGE will be best for you. Performance depends on the type and amount of data you are using. You do not need too much of data manipulation so go for file storage if privacy is not your concern for the data as it will be available for all the applications.
Use SQLite Database if the files needs to be protected from other applications.
Use File Storage(internal/external memory) if other applications can also access your files.
Avoid Fetching data from server using JSON parsing/Http requests it will make your app rely on the internet all the time. Unless you are using it to update your database or file storage.
I need to download large audio files from a web server from within a corona sdk app, and have been looking at using network.download() or possibly network.request() for this purpose.
Because much of the user base will be in areas that have poor or intermittent network coverage, I would like to improve robustness of the download process by allowing for resuming download from where it left off if the network drops out.
From the documentation neither network.download or network.request functions seem to support this directly, but is there a way that I can use either of these functions to achieve what I'm looking for? If not, is there another technique I can use?
My app will eventually be for both iOS and Android, but for now I am developing the iOS version first. Therefore, I am ok with having to use a different solution for each platform if there is not an easy solution that covers both platforms. However, I would prefer not to have to use native code if possible as I don't currently have an Enterprise subscription for Corona.
I don't think you can do this using Corona without native plugins.
One way to go around this problem is to split the large files in smaller ones, and I don't mean creating multiple audio files just splitting the large audio files in smaller files.
Then you can download one chunk at a time and when one fails, by getting an error in the handler or waiting a reasonable timeout and then checking to see if the file is present in the file system, you can start downloading it again.
After all chunks are download you can recreate the large file by using the ltn library.
You can read some more about the ltn12 library here and I think you need to take a close look to the Pumps method.
On your server-side, create a simple program that splits an audio file in multiple sub-files of the max size you would like to specify.
On your client-side, create a function that collides multiple chunks in one single audio files.
For example if you would like to limit your file size to 1MB, create a server side program that splits any audio file above 1MB in chunks: a 4.5MB file would be split in part1 1MB, part2 1MB, part3 1MB, part4 1MB, part5 0.5MB.
Glue the chunks together in one single file in your lua code when you have fetched that with network.request.
I am writing an Android app for my local transit authority. They publish their data in the the Google Transit Feed Specification (GTFS). I'd need to download this feed and save it to the device and allow my application to access the contents. Since the feed data has a standard specification, I figured I'd store behind a general ContentProvider serving general GTFS type data, that I could then access in a standard way.
I figure, this is a Google standard, I can't be the first to want to store and provide GTFS data on an Android phone; why reinvent the wheel, right? But, I haven't been able to find any such ContentProvider.
Does anyone know if there is an Android ContentProvider, or something similar, for storing and providing GTFS data? If not, I'll write my own; if I have to write code to store and provide transit data, I may as well make it standard; but since I'm still very much green to Android programing, I'd prefer to use a more professionally written ContentProvider, if one exists.
Use protocol binary format (pbf) formely google and now open source. It is compact and very fast searchable, so no need to decompress it on a device and load it into a database on that device because pbf acts as a database. Just include pbf library in your code to query it. Of course you have to compress it once before distributing the data online. Once downloaded you can use the dats in pbf offline
I am a php/mysql developer learning android. I am creating an android app that receives info from my php app to create list views of different products which will open a web view of that product's detail.
Currently my php cms web application outputs xml lists for an iphone app.... (also, separately outputs html). I have full control of the php app so if there is a better way to output the data for the android app please let me know.
I have created code that reads the xml from the web and creates the list view. The list can be refreshed daily, so the data does not need to be read from the online xml every time the app starts.
So I was thinking to store the data retrieved locally to improve my apps responsiveness. there may be up to 500 product descriptions to be stored at any given time in up to 30 different xml lists. I am starting development with one xml list with about 30 products.
For best performance should i store the product info in a sqlLite db or should i store the actual xml file in the cache/db or some other method like application cache.
I also was think to create the update of the data as a service, would this be a good idea?
The most efficient way to store data is RAM. But if you want to cache it, then the most efficient way is Database.
I recommend you store your data in sqlite android database.
You could also consider zipping you xml for faster network transfer and unzipping through java.util.zip package classes. You could even consider a simpler format for transmitting data, less verbose than xml, using a datainput/outputstream.
(I do that in of my apps and it works great)
Here are some details on data input / output stream method :
imagine a proprietary protocol for your data, only what you need. No tags, no attributes, just raw values in order.
on the client side, get an input stream on your data using URL.getContent() and cast it in input stream.
on the client side still, build a data input stream encapsulating your socket input stream and read data in order. Use readInt, readDouble, readUTF, and so on.
on the client side, from php, you need to find a way to save your data in a format that is compatible with the data format expected by the client. I can't tell much about PHP, I only program using java.
The advantage of this technique is that you save bandwith as there is only data and no verbose decoration due to xml. You should read about java specs to understand how double, int, strings are written in data output stream. But it can be hard using two languages to get the data right.
If php can't save format in a suitable way, use xml, it will be much simpler. First try with just plain xml, then give a try using a zip or tarball or xml file.
But all this is about speed gain during network connection.
The second part of what you have to do is to store each row of your list in a SQL table. Then you can retrieve it pretty fast using a CursorAdapter for your list view (it breaks the charming MVC model but it is quite fast !).
Sorry about this, but it became too long to write as a comment. This is not intended to be an answer to your question, because in my opinion Stéphane answered very well. The best solution is indeed to store the data in an sqlite database. Then you need to create the class to be used as a connection between the data, the database and the app. I don't want to take credit for what is said here already (I, too, voted it up).
I'm concerned with the other suggestion (use of low level raw streams for data manipulation, the list steps on that answer). I strongly recommend you to avoid creating your own proprietary protocol. It goes like this:
I need to exchange data.
I don't want to deal with the hassle of integrating external APIs into my code.
I know I can write two 5 minute routines to read and write the data back and forth.
Therefore, I just created my own proprietary format for exchanging data!
It makes me cry whenever I need to deal with unknown, obscure and arbitrary sequence of data blobs. It's always good to remember why we should not use unknown formats:
Reinventing the wheel is counter-productive. It seems not, but on the middle term it is. You can adapt your project to other mediums (server-side, other platforms) easily.
Using off-the-shelf components help you scale your code later.
Whenever you need to adapt your solution to other technologies and mediums, you'll work faster. Otherwise, you would probably end up with ad hoc code solutions that are not (easily) extensible and interoperable.
Using off the shelf components enables you to leverage advances in that particular technology. That's particularly important when you are using Android APIs, as they are frequently optimized for performance later down the road (See Android's Designing for Performance). Rolling your own standards may result in a performance penalty.
Unless you document your protocol, it's extremely easy to forget the protocol you created yourself. Just give it enough time and it will happen: you'll need to relearn/remember. If you document, then you are just wasting the computational time of your brain.
You think you don't need to scale your work, but chances are you will most of the time.
When you do, you will wish you had learned how to easily and seamlessly integrate well known formats.
The learning curve is needed anyway. In my experience, when you learn, you actually integrate well known formats faster than imagining your own way of doing things.
Finally, trust your data to geniuses that take their lives into creating cohesive and intelligent standards. They know it better!
Finally, if the purpose is to avoid the typical verbosity of XML, for whatever reasons, you have several options. Right now I can think of CSV, but I'm no expert in data storage, so if you're not confortable with it, I'm sure you can find good alternatives with plenty of ready to use APIs.
Good luck!
I have a client server program to communicate the pc with an android phone using usb, based on http://www.anothem.net/archives/2010/10/15/android-usb-connection-to-pc/
Now I need to design a protocol that facilitates information exchange from the pc to the phone. How should I go about this ? Should we use XML for this? I was looking at google protocol buffer.. Is that the right direction ?
Thanks
Caroline
I've been using protocol buffers on Android. I avoided it for a long time, reasoning that it was overkill. That was before I took half a day to try it out.
Here were my results, after spending an afternoon on it. At first it increased my APK size from 2.89megs to 3.1 megs. I consider that inconsequential. Then I found that I was able to delete code I had all over the place, where I had been doing parsing manually. I was able to delete code that required the data to come in in a specific sequence.
Then I was able to completely delete a few classes I had in my application, whose purpose was to serve as temporary, lightweight information-only "model" classes that represented data coming to or from the data stream. In short, it started making things way easier and smaller, and more reliable.
Protocol Buffers may well be not the best thing for your situation. But I do recommend that you take a few hours and try it out. That way you'll be making your decision to leave it behind from a position of strength and knowledge.
Be sure to use the lite version of protocol buffers. The .jar file is 160k, but the amount of functionality it brings is huge. I'll be using it all the time from here on out.
I do have some concerns about dynamic memory allocation / garbage collection when using it in a game context long term. But for now the serialization happens infrequently enough that it's a non-issue for me.
Another bonus: I have some Python code that generates data files that the application reads. That python code processes some XML and then generates binary files. I think I'll be able to completely eliminate that code by using protocol buffer's text mode, then using protoc directly to create binary files.
You need first to design the higher layer of your protocol. Xml and protocol buffers have to do with how the data are formatted. Depending on the data you need to exchange they may or not be suitable. From what you way in your comment, it seems that you want to develop something like a remote control for your PC. In that case both XML and Google Protocol Buffers will be an overdesign. Simply create a text protocol, allocate a byte or two for the command type and some bytes for the data body.
Have a look at the AT commands structure. The structure of an HDLC frame could also give you some ideas. This has some things like error checking, which you don't need, but other than that a simple text protocol will use similar fields.