I want to implement queues in one of my app. I've been investigating few options like file based queues and database queues. File based doesn't look like a right choice when data grows really big. I'm more inclined towards SQLite based queues now since it can support the large data set. Wondering if someone has already implemented queues in android and the approaches taken. What is the best way to implement queues in a multithreaded environment in android?
[EDIT]
I want to make a queue for messages, which can grow really big if device is offline for long time and unable to flush queues to server. I feel SQLite could be better solution, but afraid if all the database loads as a first thing into memory when an app starts. I want to support really huge backlogs queues, but don't want to consume too much memory to load them all at once.
Wondering if someone has already implemented queues in android and the approaches taken
Square released Tape, a persistent queue implementation for Android, that they use in their apps.
File based doesn't look like a right choice when data grows really big.
IMHO, a persistent queue should never "grow really big". If the data associated with the queue entry is large, that suggests that some of that data should not be considered part of the queue data model, but part of the app's overall data model, resident elsewhere (e.g., SQLite) and be pointed to from the queue.
For example, suppose the queue is for image uploads. IMHO, those images are not part of the queue. The queue represents commands to upload images. The queue entries should point to where the images are stored, as those images likely serve other roles (e.g., cache entries) beyond simply waiting to be uploaded.
Related
I don't know this title is clearly reflect the question content. Please help rephrase if you think it is confusing. Thanks!
This is a generic question about some common architecture. I am working on a Android app that can take and share photo just like Instagram. And I have a backend web service powered by Django.
The issue I am facing is, in the app, I will need to show different resolution of image (for example, thumbnails for profile image, mid-resolution for previews, and full resolution for expanded image views.) I want to ask a common pattern about how to make this happen.
I have two proposals on doing this task, and not sure which way I should go:
1.When user upload photos from the mobile app, I can compress it locally and send 3 different sizes (low res thumbnails, mid res, and high res). So server side can store them and return them back in different cases. The Con of this approach that I can think about is it will drain more user's data usage because now user will need to send multiple images. It may also cause uploading take more time and higher impact in user experience.
2.When user upload photos from the mobile app, they only upload the original image. The server side logic will do the compression for each income image, and store them accordingly. The Con of this approach is server may need to carry a lot more workload. In this case, if user increases a lot, it may crash the server.
I am hoping to get some pointer on this issue, or any reference about this topic will be helpful!
Upload the full size image to the server and have the server do the heavy lifting. Create three version of the image (small medium and large) and store them on the server (or content delivery network). Create a database table to keep track of the image id for each image and its various versions.
With android you wont know how powerful the phone running your app is, never assume it can handle whatever image manipulation you need, also try and keep network I/O as low as possible.
Alan - Think of smart phones as relatively dumb terminals made for purpose of consuming content. Most of the business logic and heavy processing should me implemented on server side.
In terms of architecture, you are facing scalability problem. You cannot expand cpu/memory or storage of a device to any level you want. However you can scale your servers horizontally or vertically by adding more ram/cores/disks etc. You can put cluster on server and have a farm of servers if data increases to that level.
So its always advisable to just upload the original image without locally processing. Locally processing also drains battery besides other downside you mentioned.
Again you business logic or processing technique changes, you have to redeploy all apps. Where as on server side such deployments are very well in your control.
What i would do is a first resize in the mobile, just one, because you dont one to transfer 13Mpx picture and also you dont want in your server a picture 2 screens wide (this is the first thing instagram does). After that, upload the file, and have the server doing the rest of the stuff.
this avoid a lot of that usage
ensures that will work fine in any device (every one has different capabilities, so you cant trust them for heavy work)
You can change some decisions about settings or configurations, and all this is centralized, if you change any common behavior, you don't need all the devices to get the last version of the app.
About crashing the server, there is one first measure i would take:
Don't do any operation, like re-sizing or making copies of the image in the moment of the upload, you can choose one of this approaches:
Do it when there is a request for the file. this is, dont do it when the user sends it, but when somebody needs it.
Do it in a continuos backgorund process, like a cronjob running every minute, for instance.
or a combination of the two, there is a process in the background doing the stuff for all the pending images, but if somebody enters in the website and they need an image that is not yet generated, it is generate in that moment.
I'm using Jake Wharton's DiskLruCache lib.
I'm curious about app performance, caching strategy, using caching both in a view and across a whole application. Most of the time, the image won't change.
Let's say for example I have a 320x320 photo on my server. I open the stream, save the image.
In my list views, I show bitmaps and in the detail, I show a larger image. Should I save a thumbnail bitmap too? Is that more efficient?
What is your experience with sharing the cache "object" across the entire app (let's say I have multiple views that might leverage the same data. What are the issues with this?
For the sake of performance and currency, what if the image changes on the server. What's the best strategy to know that it's changed? I don't have access to modified date. Only size and yet, I don't really want to query size every time either. Set a flag in the app on the server and then query the flag?
In a traditional application (if there is such a thing), what's the best practice for clearing the cache from time to time? (indent weirded out.)
(I was inspired to write this after seeing all of the performance improvements by Facebook in iOS. I don't have billions to do caching but I would like to at least be smart about it! LOL)
A lot of these answers depend on the type of app you're writing, how important image updates are (and how likely images will change, etc), and total images produced. Aside from disk caching, you should be using memory caching as well, especially in ListViews and other areas where the same images will be scrolled through repeatedly. Look at LruCache and read the Caching Bitmaps entry from Google.
320x320 is probably too big for a listview, you will likely want to create thumbnails (depending on device, and how you're implementing your listviews).
1) You should be using Disk caching fairly aggressively (how you define that is up to the app you're writing). Use the external storage directory, and if they have a few GB left over, it's not an issue if you take 100 mb for your app for instance. It can all be cleared away if it's ever needed anyway.
2) There shouldn't be an issue. Disk IO (even to a flash medium) should never be handled on the main thread. Use AsyncTasks to load the images. There can only be one main foreground activity at once anyway, and while an activity is sleeping, it shouldn't be trying to read from the disk anyway.
3) Again this depends on how you're implementing your app. You should be able to get additional information when retrieving the file (even Apache can tell your app the last modified date).
3.1) You can have a sqllite db that keeps track of how often certain images are used, and most recent read. If the most recent read is a few days old, let that image expire.
Edit: There's a library from Jake Wharton now called Picasso that I would recommend using, which handles the Network/local IO as well as memory and disk caching. Check it ou here: http://square.github.io/picasso/ . A lot of what you will need to do can be accomplished with one line: Picasso.with(this).load(imageFileURL).into(imageView);
I am writing a social networking android application. I have create a .Net webservice with a database on Microsoft Azure, and I plan to call that web service to get data from the cloud and display it to the user. Similar to Facebook.
Now, I have two approaches in mind, and I'm not sure which one to implement. The approaches follow:
"Every time an activity loads, call the web service and reload all the data." This, of course, is the easiest approach, but is it right? I mean, I have around 30 activities, and half of them load data, while the other half post. As far as I see, this approach can be a problem, because it might slow down the application. It can also increase my cloud bill with so many requests. And I don't know if it's right to reload everytime.
"Call the webservice every 10 minutes, and store all the data in a SQLite database, and only update the data if it's been over 10 minutes, or possibly even have a refresh button." This approach is probably the better one, but I'm not sure if it is even worth writing so much code.
I need you advice in deciding on the right technique. Number 2 looks good, but what if there is something I don't know, and I'm writing all that extra code for no reason.
Please help me out here. If there is even a better approach, please do tell me.
It really depends on the sort of data, what sort of latency is required for the data and the quantity of data. Also the size of the project and benefit you will get from implementation as complexity will be increased. For a more precise answer provide further information.
Local caching can be a very good idea in this sort of situation. It is fairly common practice and there are multiple mechanisms which could be used. Depending on the format of data retrieved from your web service, you can store in a
Database, when data needs to processed (searched, queried etc) or there is a lot of data.
File(s), sometimes useful if you are working with formatted data such as xml or json as you can maintain the structure. You can use the native android caching to assist in managing the storage.
Preferences, when data is simple types (and strings) and there isn't much of it.
Local caching will reduce bandwidth consumption (will end up saving the user money which is always popular) and if implemented correctly memory and processing consumption. Possibly most important (depending on the data) it could allow for the effective use of the application when the user doesn't have connectivity.
By the way 30 activities sounds like a lot, you should really look at reducing that by sharing functionality across activities, this should improve navigation, code bulk and memory foot print.
UPDATE from comments
From the limited information available about your project I would suggest using the database as your data store (if any) as you don't want to be caching complete SOAP messages in files, and the quantity 40+ records may make preference storage difficult to manage.
However, as I have mentioned previously you need to consider complexity. When using the database you will have to create a method of construction (perhaps some sort of ORM) separate to your de-serialisation of SOAP objects because technically you will 2 separate persisted data formats.
I am not able to get a definitive answer because there is still very limited information but you need to evaluate the cost of adding such a feature to your project and the benefits you will receive.
I few other things worth mentioning when considering this sort of caching.
How you will manage the cache, it's size and data integrity.
When will you cache, as soon as you have de-serialised your SOAP objects? when you have finished with the data? etc..
How will decide when to use the cache and when to hit the network?
I'm building an app that fetches XML from a server, then parses it (it's DIDL formatted in places). The goal is to load as much into memory as possible to make it easier for users to browse (so as to enable fast scrolling through results), but I keep getting OutOfMemoryErrors. I'm kind of new to Android, and it seems like the VMs are really finicky about things. I'm loading no more than a megabyte of XML, and pretty much discarding it right away. What should I do to prevent these errors from happening?
Should I load chunk by chunk of the file over the network, write it to disk, then load chunk by chunk back into and out of memory, parsing everything into POJOs? If it can be avoided, I'd like to not have to implement some form of pagination, as per the Twitter app (it used to load more entries when you hit the bottom, it now loads a lot all at once and likewise crashes with an OOM error.) I'm running a Nexus One if that helps, CM7/Android 2.3.3.
You can read this nice article about XML parsing on Android. Using pull parser would be good choice for you, since it does not need to read complete document into the memory, which is the problem in your case. I would suggest that you store parsed results into the database, since once they are there, you can quickly do list them and page them any way you want, and performance of the DB is great. This way, you need to do loading of complete data from server to DB just once (if data is not changing on the server) or to load it once and then to get updates from time to time if it is changing (like in case of twitter).
Move all your object declarations out of loops and nullify them after the use. And use System.gc() frequently (beleive me, it works). Use class level objects as less as you can. Execute your app and keep watch on the Logcat logs.
I had a question related to whether or not (and how) I should store images loaded from the web. Let's say I am calling a web service from my Android app. In this web service I get a URL for an image on the web. I download and show this image on the left side of a list item in a ListView. My question is, what method should I use for possibly storing the image? Should I:
Save it to the SDCard, checking if it exists when the ListView is created (on subsequent requests) and re-downloading as necessary (while updating the image occasionally, in case it changes).
Store it in the cache using Context.getCacheDir(), but possibly being forced to re-download it more often since I cannot rely on the image staying in the cache.
Always download it and never store the image.
The image files themselves are fairly small, but I expect some users could have dozens of these small images downloaded/stored. Which would work best, and/or what is the preferred method?
As a side question, should I load all of the images in my ListView first (and possibly lock the UI for some time) or load them asynchronously but show a placeholder graphic in the meantime (which might be a bit more "ugly")? What's the standard here?
About where to store:
The answer depends on what is being downloaded and how much. Then you make a choice.
For instance: If you are downloading something that is temporary, less in number(less remote fetches) and size(less memory) and is local to an Activity then you should consider holding the images in memory using SoftReferences. SoftReferences can lead to refetches but since the number of items is small it should be affordable.
However, if the number of items to be downloaded exceeds a certain threshold(meaning more fetches and memory) you should consider reducing the fetches and also the Runtime memory consumption by caching them. Here you can either choose to save them on Sdcard or on temporary storage(cache directories local to app). For items that are small and hold meaning only in the context of Application(e.g thumbnails), the user will mostly not use it outside of your application. So, you can store such things in the cache directory. The best part of using them is that you don't have to clean the mess. It is handled automatically. It might lead to re-fetches though.
However, if the downloaded items are large in size and can stand alone outside the context of the application such as pictures, video, audio clips then SDcard should be your option.
You should also read: Handling large Bitmaps to avoid OOM error during BitmapFactory.decodeStream(..)
Note that you can also check to see if using a database can help here.See this
Some considerations while lazy loading items in a ListView:
You should do the loading in the background and not block the UI thread.You should consider showing a temporary image while the item is being downloaded. This is apparent in many native applications. For an example implementation of lazy loading see this.
Additionally, for large lists, you can implement the SlowAdapter pattern(check API demos).
It basically stalls download while the list is scrolling.
Example Projects that can help you here:
Romain Guy's Shelves project uses two level of caching wherein he employs an in-memory cache(HashMap containing SoftReferences) and storage on Sdcard.Browse Source code here
There are also some open source libraries that Mark Murphy wrote(CWAC) and DroidFu that can help here.
Good Luck!
In regards to your "side question" -- I think that loading them asynchronously would be the preferred behavior, especially since you need to consider that with network transactions, it might not just be "lock the UI for some time", but "lock the UI permanently" in case it never loads.
If you consider this behavior ugly, though, you could set a timer (1 or 2 seconds) that gives a few of the images a chance to load, and if they have all loaded or the timer has expired, go ahead and show the UI anyway with placeholder images and let the rest load asynchronously. This would be the way to go to prevent permanent locks anyway.
As for the first part of your question, I think it is somewhat dependent on the context and type of images you are displaying. With most web assets, though, I think #2 would be the preferred approach, since you certainly don't want to download multiple times in the same session, but you probably don't want to eat up permanent storage either.
In regards to your "side question" -- I think that loading them asynchronously would be the preferred behavior, especially since you need to consider that with network transactions, it might not just be "lock the UI for some time", but "lock the UI permanently" in case it never loads.
To avoid this , If we are talking about WebImageView by droid-fu you can change the ImageLoader.java initialize() function to this
static int alive=-1;
public static synchronized void initialize(Context context) {
if (executor == null) {
executor = (ThreadPoolExecutor) Executors.newFixedThreadPool(DEFAULT_POOL_SIZE);
}
else if(alive==executor.getActiveCount()){
executor = (ThreadPoolExecutor) Executors.newFixedThreadPool(DEFAULT_POOL_SIZE);
}
alive=executor.getActiveCount();
if (imageCache == null) {
imageCache = new ImageCacheHope(context, 25, 5);
}
}
I wrote an Android Image Manager that handles caching transparently (memory and disk). The code is on Github https://github.com/felipecsl/Android-ImageManager