I am a php/mysql developer learning android. I am creating an android app that receives info from my php app to create list views of different products which will open a web view of that product's detail.
Currently my php cms web application outputs xml lists for an iphone app.... (also, separately outputs html). I have full control of the php app so if there is a better way to output the data for the android app please let me know.
I have created code that reads the xml from the web and creates the list view. The list can be refreshed daily, so the data does not need to be read from the online xml every time the app starts.
So I was thinking to store the data retrieved locally to improve my apps responsiveness. there may be up to 500 product descriptions to be stored at any given time in up to 30 different xml lists. I am starting development with one xml list with about 30 products.
For best performance should i store the product info in a sqlLite db or should i store the actual xml file in the cache/db or some other method like application cache.
I also was think to create the update of the data as a service, would this be a good idea?
The most efficient way to store data is RAM. But if you want to cache it, then the most efficient way is Database.
I recommend you store your data in sqlite android database.
You could also consider zipping you xml for faster network transfer and unzipping through java.util.zip package classes. You could even consider a simpler format for transmitting data, less verbose than xml, using a datainput/outputstream.
(I do that in of my apps and it works great)
Here are some details on data input / output stream method :
imagine a proprietary protocol for your data, only what you need. No tags, no attributes, just raw values in order.
on the client side, get an input stream on your data using URL.getContent() and cast it in input stream.
on the client side still, build a data input stream encapsulating your socket input stream and read data in order. Use readInt, readDouble, readUTF, and so on.
on the client side, from php, you need to find a way to save your data in a format that is compatible with the data format expected by the client. I can't tell much about PHP, I only program using java.
The advantage of this technique is that you save bandwith as there is only data and no verbose decoration due to xml. You should read about java specs to understand how double, int, strings are written in data output stream. But it can be hard using two languages to get the data right.
If php can't save format in a suitable way, use xml, it will be much simpler. First try with just plain xml, then give a try using a zip or tarball or xml file.
But all this is about speed gain during network connection.
The second part of what you have to do is to store each row of your list in a SQL table. Then you can retrieve it pretty fast using a CursorAdapter for your list view (it breaks the charming MVC model but it is quite fast !).
Sorry about this, but it became too long to write as a comment. This is not intended to be an answer to your question, because in my opinion Stéphane answered very well. The best solution is indeed to store the data in an sqlite database. Then you need to create the class to be used as a connection between the data, the database and the app. I don't want to take credit for what is said here already (I, too, voted it up).
I'm concerned with the other suggestion (use of low level raw streams for data manipulation, the list steps on that answer). I strongly recommend you to avoid creating your own proprietary protocol. It goes like this:
I need to exchange data.
I don't want to deal with the hassle of integrating external APIs into my code.
I know I can write two 5 minute routines to read and write the data back and forth.
Therefore, I just created my own proprietary format for exchanging data!
It makes me cry whenever I need to deal with unknown, obscure and arbitrary sequence of data blobs. It's always good to remember why we should not use unknown formats:
Reinventing the wheel is counter-productive. It seems not, but on the middle term it is. You can adapt your project to other mediums (server-side, other platforms) easily.
Using off-the-shelf components help you scale your code later.
Whenever you need to adapt your solution to other technologies and mediums, you'll work faster. Otherwise, you would probably end up with ad hoc code solutions that are not (easily) extensible and interoperable.
Using off the shelf components enables you to leverage advances in that particular technology. That's particularly important when you are using Android APIs, as they are frequently optimized for performance later down the road (See Android's Designing for Performance). Rolling your own standards may result in a performance penalty.
Unless you document your protocol, it's extremely easy to forget the protocol you created yourself. Just give it enough time and it will happen: you'll need to relearn/remember. If you document, then you are just wasting the computational time of your brain.
You think you don't need to scale your work, but chances are you will most of the time.
When you do, you will wish you had learned how to easily and seamlessly integrate well known formats.
The learning curve is needed anyway. In my experience, when you learn, you actually integrate well known formats faster than imagining your own way of doing things.
Finally, trust your data to geniuses that take their lives into creating cohesive and intelligent standards. They know it better!
Finally, if the purpose is to avoid the typical verbosity of XML, for whatever reasons, you have several options. Right now I can think of CSV, but I'm no expert in data storage, so if you're not confortable with it, I'm sure you can find good alternatives with plenty of ready to use APIs.
Good luck!
Related
I am currently learning android programming and creating an app that will store some integers representing user choices (values inserted several times a day, must be displayed in the results activity) and steps data collected Google Fit HISTORY Android APIs, also displayed in the results activity. I am looking for the most efficient way to store this data. I know that it might be possible to insert the custom data types in the GOOGLE fit database. However, I am not sure if it is a good idea if the app mostly works offline, and it needs to immediately represent only a small set of results, for example, the values inserted in the last 2 weeks, with step counts. On the other hand, I am not sure if it is ok to have two databases storing the data.
My apologies if the question sounds a bit too amateur, I am doing my best to find an optimal solution in terms of performance.
Thank you for your answers.
So, to give you my opinion and answer (mainly opinion)
Android has 3 ways (mainly) for storing data:
Files
Online database/API
Local database
for this specific scenario you have listed, wanting the data to be available offline, you should probably be looking at using Room: https://developer.android.com/training/data-storage/room, as it supports storing primitive types without having to write any type converters, you can store models and custom data as well, it uses very basic SQL (because it's a wrapper for the older Sqlite database methods) and is part of android (not an external 3rd party library). Room also requires most operations to be done off of threads, instead of main threads and this will improve your performance as well (also has support for livedata/rxjava to observe straight onto any changes as they happen)
However, as I told this user here:
Should i store one arrayList per file or should i store all my arrayList in the same file?
When starting out, don't worry about the best way for doing something, instead, try something out and learn from it, worrying about the best solution now is rather pointless, either way, happy learning and coding :P
We developing an app that will store data locally on phone, on server and share it between users. The access to store and retrieve data will be approximately 20 times a day for each user, it will consist mostly of strings. The app will be cross platform Android/iOS. So before beginning developing DB what you suggest Json vs SQlite or use HTTP Methods: GET, POST.
What are advantages and disadvantages of each, and what do you suggest.
I do not agree that sql is slow. It's not like these devices are using spinning disks. In my app, i've implemented a RESTful API that passes JSON data back. This is easy because on iOS, json can be easily converted to the iOS Native NSDictionary object.
I recommend storing data using sqlite or Core Data because you'll be happy for it later on if you need to query the data. That's what databases are used for, storing and retrieving data easily.
As an alternative, you can also consider building the SQLite database on the server, and actually downloading that file to the devices. This has it's benefits because you won't need to do the actual sqlite processing on device.
I recommend staying away from XML, not because it's bad, but because passing JSON data back has become standard, and a lot of large companies are moving away from XML formats because of the processing power required.
In designing your RESTful API, it's always good practice to use the already available HTTP request methods. Use GET to retrieve data, POST to create new data, PUT to update data and DELETE to remove data. Once you have a beautifully designed REST api, all things things should just fall into place and you'll have an app that's maintainable and scalable on the client and on the server side.
This question really depends on your own area of expertise. If you have good understanding of JSON, then use that. XML is another great way via SOAP (or REST) to distribute data to your platforms, Server -> Device -> Server. My own experience is that JSON with GZIP is small and fast and there exists lots of parsers that is fast. XML has the downside that it gets big quickly but is often easier to maintain over time since the syntax is set in the "namespace" of your XML. SQLite is slow, since its disk based and should only be used for persistent storage of data. Access it as little as possible during runtime. My suggestion: JSON+GZIP, REST-api on server/device, store as much as possible in memory on devices, SQLite for persistent storage on devices and lastly MySQL on server for persistent storage (free, fast enough if done correctly with views and methods).
But as said, this is all up to the implementation.
I am new to Android Application Development and a new member at stackoverflow. I am currently trying to design a recipe application. I have decided upon the features of the app and the scope it will cover. The scope is very vast for me in terms of covering all the recipes from all over the world. I am to deal with a lot of data in this process.
I am currently trying to figure a good and efficient way of handling the data in my app. So far, as per what I have read in different forums, I believe that I have two options in terms of a database choice : 1) SQLite 2) Database on remote server (MySql/Postgre)
Following are some of the thoughts that have been going on in my mind when it comes to taking a decision between the two :
1) SQLite : This could be a good option but would be slow as it would need to access the file system. I could eliminate the slowness by performing DB data fetch tasks in the AsyncTask. But then there could be a limitation of the storage on different phones. Also I believe using SQLite would be easier as compared to using a remote DB.
2) Remote Database : The issue that I can see here is the slowness with multiple DB requests coming at the same time. Can I use threads here in some way to queue multiple requests and handle them one by one ? Is there an efficient way to do this.
Also I have one more question in terms of the formatting of my data once I pull it out from the above DB's. Is there a way I could preserve the formatting of my data ?
I would be more than thankful if someone could share their knowledgeable and expert comments on the above scenario. Also this is not a homework for me and I am not looking for any ready made code solutions. I am just looking for hints/suggestions that would help me clear my thoughts and help me take a decision. I have been looking for this for sometime now but was not able to find concrete information. I hope I will get some good advice here from the experienced people who might have encountered similar situation.
Thanks for reading this long post.
What about combining both approaches?
A local SQLite database that has the least recently used receipes so you don't need network all the time. Network is way slower than accessing the filesystem.
Some remote database accessed via some HTTP interface where you can read / write the whole database. And if you want users to be able to add receipes for other users to see you'll need an external database anyways.
SQLite : This could be a good option but would be slow as it would need to access the file system.
Accessing a local database is pretty fast, 5ms or so if it's just a simple read only query on a small database.
But then there could be a limitation of the storage on different phones
Depends on your definition of huge database. It is okay if it is only 2MB which would be enough to store lots of text-only receipes.
Also I believe using SQLite would be easier as compared to using a remote DB.
Yes, Android has a nice built-in SQLite API but no remote database API. And you don't need to setup a database server & interface.
The issue that I can see here is the slowness with multiple DB requests coming at the same time.
A decent database server can handle thousands of requests. Depends on your server hardware & software. https://dba.stackexchange.com/ should have more info on that. Required performance depends on how much users you have / expect.
I'd suggest a simple REST interface to your database since it's pretty lightweight but does not expose your database directly to the web. There are tons of tutorials and books about creating such interfaces to databases. There are even hosted database services like nextDb that do most of the work for you.
Is there a way I could preserve the formatting of my data ?
You could store HTML formatted data in your database and display it in a WebView or a TextView (via Html#fromHtml()) - both can display formatted text.
Databases don't care what type of text you store, for transfer over the internets you may need to encode the text so it does not interfere with the transport formatting (XML, JSON, ...).
A simple way is to integrate Parse into your app. They have a nice framework that easily integrates into iOS and Android. Their plan is freemium, so you'll be able to use up to 1 million API request for no charge, and then its 7 cents for every request after that.
You'll have 1gb to store all your data sets / images, etc.
I don't use parse for everything, but I HIGHLY recommended it for large data schemes because they do all the scaling for you. Check out the API, I think it would be worth your time.
I just started to work on a few of my own projects, and I'm using Parse again. I have to say it's improved a lot over the last 6-8 months. Especially with the Twitter and Facebook integration.
The key issue here is the size of the data - any significant database of recipes would be too large to store on the phone imho,thus you seem stuck with the remote database solution.
As opposed to trying access the remote database from android I suggest you use a a go between web application that will process requests from the application and return JSON objects that you need.
It totally depends on your software requirements. If you need to deal with a small amount of data then you may choose SQLite, but for a huge amount to data better use a remote DB.
SQLite: It works fine with little amount of data & I experienced it response time is good.
Remote DB: I think you may use small server side app to submit the data to your client app. It will solve/reduce your thread related issues/complexities.
I am writing a social networking android application. I have create a .Net webservice with a database on Microsoft Azure, and I plan to call that web service to get data from the cloud and display it to the user. Similar to Facebook.
Now, I have two approaches in mind, and I'm not sure which one to implement. The approaches follow:
"Every time an activity loads, call the web service and reload all the data." This, of course, is the easiest approach, but is it right? I mean, I have around 30 activities, and half of them load data, while the other half post. As far as I see, this approach can be a problem, because it might slow down the application. It can also increase my cloud bill with so many requests. And I don't know if it's right to reload everytime.
"Call the webservice every 10 minutes, and store all the data in a SQLite database, and only update the data if it's been over 10 minutes, or possibly even have a refresh button." This approach is probably the better one, but I'm not sure if it is even worth writing so much code.
I need you advice in deciding on the right technique. Number 2 looks good, but what if there is something I don't know, and I'm writing all that extra code for no reason.
Please help me out here. If there is even a better approach, please do tell me.
It really depends on the sort of data, what sort of latency is required for the data and the quantity of data. Also the size of the project and benefit you will get from implementation as complexity will be increased. For a more precise answer provide further information.
Local caching can be a very good idea in this sort of situation. It is fairly common practice and there are multiple mechanisms which could be used. Depending on the format of data retrieved from your web service, you can store in a
Database, when data needs to processed (searched, queried etc) or there is a lot of data.
File(s), sometimes useful if you are working with formatted data such as xml or json as you can maintain the structure. You can use the native android caching to assist in managing the storage.
Preferences, when data is simple types (and strings) and there isn't much of it.
Local caching will reduce bandwidth consumption (will end up saving the user money which is always popular) and if implemented correctly memory and processing consumption. Possibly most important (depending on the data) it could allow for the effective use of the application when the user doesn't have connectivity.
By the way 30 activities sounds like a lot, you should really look at reducing that by sharing functionality across activities, this should improve navigation, code bulk and memory foot print.
UPDATE from comments
From the limited information available about your project I would suggest using the database as your data store (if any) as you don't want to be caching complete SOAP messages in files, and the quantity 40+ records may make preference storage difficult to manage.
However, as I have mentioned previously you need to consider complexity. When using the database you will have to create a method of construction (perhaps some sort of ORM) separate to your de-serialisation of SOAP objects because technically you will 2 separate persisted data formats.
I am not able to get a definitive answer because there is still very limited information but you need to evaluate the cost of adding such a feature to your project and the benefits you will receive.
I few other things worth mentioning when considering this sort of caching.
How you will manage the cache, it's size and data integrity.
When will you cache, as soon as you have de-serialised your SOAP objects? when you have finished with the data? etc..
How will decide when to use the cache and when to hit the network?
I have a client server program to communicate the pc with an android phone using usb, based on http://www.anothem.net/archives/2010/10/15/android-usb-connection-to-pc/
Now I need to design a protocol that facilitates information exchange from the pc to the phone. How should I go about this ? Should we use XML for this? I was looking at google protocol buffer.. Is that the right direction ?
Thanks
Caroline
I've been using protocol buffers on Android. I avoided it for a long time, reasoning that it was overkill. That was before I took half a day to try it out.
Here were my results, after spending an afternoon on it. At first it increased my APK size from 2.89megs to 3.1 megs. I consider that inconsequential. Then I found that I was able to delete code I had all over the place, where I had been doing parsing manually. I was able to delete code that required the data to come in in a specific sequence.
Then I was able to completely delete a few classes I had in my application, whose purpose was to serve as temporary, lightweight information-only "model" classes that represented data coming to or from the data stream. In short, it started making things way easier and smaller, and more reliable.
Protocol Buffers may well be not the best thing for your situation. But I do recommend that you take a few hours and try it out. That way you'll be making your decision to leave it behind from a position of strength and knowledge.
Be sure to use the lite version of protocol buffers. The .jar file is 160k, but the amount of functionality it brings is huge. I'll be using it all the time from here on out.
I do have some concerns about dynamic memory allocation / garbage collection when using it in a game context long term. But for now the serialization happens infrequently enough that it's a non-issue for me.
Another bonus: I have some Python code that generates data files that the application reads. That python code processes some XML and then generates binary files. I think I'll be able to completely eliminate that code by using protocol buffer's text mode, then using protoc directly to create binary files.
You need first to design the higher layer of your protocol. Xml and protocol buffers have to do with how the data are formatted. Depending on the data you need to exchange they may or not be suitable. From what you way in your comment, it seems that you want to develop something like a remote control for your PC. In that case both XML and Google Protocol Buffers will be an overdesign. Simply create a text protocol, allocate a byte or two for the command type and some bytes for the data body.
Have a look at the AT commands structure. The structure of an HDLC frame could also give you some ideas. This has some things like error checking, which you don't need, but other than that a simple text protocol will use similar fields.