I'm building an app that fetches XML from a server, then parses it (it's DIDL formatted in places). The goal is to load as much into memory as possible to make it easier for users to browse (so as to enable fast scrolling through results), but I keep getting OutOfMemoryErrors. I'm kind of new to Android, and it seems like the VMs are really finicky about things. I'm loading no more than a megabyte of XML, and pretty much discarding it right away. What should I do to prevent these errors from happening?
Should I load chunk by chunk of the file over the network, write it to disk, then load chunk by chunk back into and out of memory, parsing everything into POJOs? If it can be avoided, I'd like to not have to implement some form of pagination, as per the Twitter app (it used to load more entries when you hit the bottom, it now loads a lot all at once and likewise crashes with an OOM error.) I'm running a Nexus One if that helps, CM7/Android 2.3.3.
You can read this nice article about XML parsing on Android. Using pull parser would be good choice for you, since it does not need to read complete document into the memory, which is the problem in your case. I would suggest that you store parsed results into the database, since once they are there, you can quickly do list them and page them any way you want, and performance of the DB is great. This way, you need to do loading of complete data from server to DB just once (if data is not changing on the server) or to load it once and then to get updates from time to time if it is changing (like in case of twitter).
Move all your object declarations out of loops and nullify them after the use. And use System.gc() frequently (beleive me, it works). Use class level objects as less as you can. Execute your app and keep watch on the Logcat logs.
Related
I have an app that creates custom files that are shown in a gallery. They are not images or other standard media files. Because the files are large, I ask users for access to a folder in which the files will be stored.
So far so good.
Reading those files or file info via DocumentFile is so much slower than via File. Operations like
DocumentsContract.findDocumentPath()
documentFile.isFile()
documentFile.getName()
take up to 15ms each, which accumulates to a large time when reading a large number of files.
What's the best way to handle this? Is there another way to do this? I don't want the files to be in internal app memory or users might lose their work when they uninstall the app.
Thanks in advance! This is stressing me out a lot.
Do it asynchronously. Do you need to get all of that data immediately? Probably not. So do it on a thread/in a coroutine, and just program your app to not display the data if you do not yet have it (and refresh itself when you do).
In addition, even if you do need it up front- do you need it for all files up front? In a gallery, you can make do with just the ones that are immediately displayed, and the next few which might be. So only fetch that, and get the others when you need them (or when its likely you'll need them soon). Treat it like fetching results from a web API- you don't fetch every post ever made, you fetch a few dozen, then you fetch a few dozen more when they're getting close to the bottom
In the end, DocumentFile is a wrapper around other APIs, a wrapper that is designed to provide a convenient, File-ish API for developers to use. It is not designed for bulk use. If you want to try to get more speed, you can look at the source code to DocumentFile and related classes used in its implementation, such as DocumentContract19, and work at a lower level.
For example, getName() winds up using this queryForString() method to do the real work. Making a query using ContentResolver adds IPC overhead, so if there are other values that you need that you can get in that same query, you could do the query yourself. For example, isFile() also winds up doing a query, so you could combine those two requests into one and cut the overall time in half (roughly).
Do not use DocumentFile class.
Only use DocumentsContract functions.
Its about twenty times as fast as DocumentFile and nearly as fast as classic file operations
So I've been struggling for a while with my Android application. I'm fairly new to Android programming and made the application for a customer. Back then I programmed it for new devices like the S2 and S3 from Samsung.
Now I got a problem: Garbage Collection. I've read some posts about the mechanics and how to get less GC, but I don't think it will help. I hope someone can hint me on how to take care of GC.
My code does the following: it downloads a lot of data from a online CRM called vTiger. It's a dynamic system so the application gets a lot of linked data, not just (for example) a Users table with username and password.
Now this goes into JSON objects and I split those out so I can use all the data, in a loop like this: for each module, get all records. Well those records combined are about 5000 (now, still growing with usage).
A lot of info, so I store all the information into ArrayList and ArrayList, which are simple objects I made to hold information. I remember I didn't just write the objects to the database instead of hold them in the ArrayList as the database got overloaded. So what I did is save all those objects (around 5000 - 10000) in the ArrayLists and then use Transactions on the SQLite database. This way it worked well.
BUT,
On the emulator / older devices I get a LOT of GC_concurrent logging, which means I got memory leaks in my code. I understand that the device cannot hold all those objects in it's memory but how would you take care of this? Due to GC, older devices cannot process through my 'sync' function. For example I let the emulator run for a few hours but it was almost stuck (very,very, very slow) halfway through the sync.
Hope someone is willing to give me some advice, if even possible.
Thanks in advance!
If you're loading a lot of data, you're going to get GC spikes. There's really no way around it. The only way to not have as big of spikes is to chunk the data--split it into smaller pieces (possibly per-device), and process them before loading more.
Instead of requesting all the data from the service at once, request the data in chunks. This will prevent memory from filling up and forcing small GCs.
I'm using Jake Wharton's DiskLruCache lib.
I'm curious about app performance, caching strategy, using caching both in a view and across a whole application. Most of the time, the image won't change.
Let's say for example I have a 320x320 photo on my server. I open the stream, save the image.
In my list views, I show bitmaps and in the detail, I show a larger image. Should I save a thumbnail bitmap too? Is that more efficient?
What is your experience with sharing the cache "object" across the entire app (let's say I have multiple views that might leverage the same data. What are the issues with this?
For the sake of performance and currency, what if the image changes on the server. What's the best strategy to know that it's changed? I don't have access to modified date. Only size and yet, I don't really want to query size every time either. Set a flag in the app on the server and then query the flag?
In a traditional application (if there is such a thing), what's the best practice for clearing the cache from time to time? (indent weirded out.)
(I was inspired to write this after seeing all of the performance improvements by Facebook in iOS. I don't have billions to do caching but I would like to at least be smart about it! LOL)
A lot of these answers depend on the type of app you're writing, how important image updates are (and how likely images will change, etc), and total images produced. Aside from disk caching, you should be using memory caching as well, especially in ListViews and other areas where the same images will be scrolled through repeatedly. Look at LruCache and read the Caching Bitmaps entry from Google.
320x320 is probably too big for a listview, you will likely want to create thumbnails (depending on device, and how you're implementing your listviews).
1) You should be using Disk caching fairly aggressively (how you define that is up to the app you're writing). Use the external storage directory, and if they have a few GB left over, it's not an issue if you take 100 mb for your app for instance. It can all be cleared away if it's ever needed anyway.
2) There shouldn't be an issue. Disk IO (even to a flash medium) should never be handled on the main thread. Use AsyncTasks to load the images. There can only be one main foreground activity at once anyway, and while an activity is sleeping, it shouldn't be trying to read from the disk anyway.
3) Again this depends on how you're implementing your app. You should be able to get additional information when retrieving the file (even Apache can tell your app the last modified date).
3.1) You can have a sqllite db that keeps track of how often certain images are used, and most recent read. If the most recent read is a few days old, let that image expire.
Edit: There's a library from Jake Wharton now called Picasso that I would recommend using, which handles the Network/local IO as well as memory and disk caching. Check it ou here: http://square.github.io/picasso/ . A lot of what you will need to do can be accomplished with one line: Picasso.with(this).load(imageFileURL).into(imageView);
I have an Android app that needs to work offline and requires a lot of static data.
Currently I'm using a JSON file in the /res/raw and loading it with the Jackson parser into my POJO scheme. It works really well since I have an external program that will be generating this data and once in a while when there is a change I'll just publish new version to the Market so I don't have to deal with running an update server and so on.
However, right now my JSON file is about 2.5MB with limited dataset for testing, in the end it'll be about 5-10MB.
The issue is that it already takes about 3-5 seconds to parse the file and this needs to be done every time the application is restarted.
So, what are my options here? I could put the data to a sqlite database, but that would require rewriting the external application and changing the data structure quite a bit. But then I could only query the things I need at the moment and not loading the entire thing at once.
Is there some easier/better way? Also, is there a good way to publish the app with the sqlite database? All the articles I've found talk about creating the database for user data at first startup, but this is not user data and I need it to be deployed from the Market.
JSON feels like the wrong approach for this - it's a good way to encode data to transfer, but that's pretty much it.
It'd be nice to have a bit more info on what exactly your app does, but I'm struggling to imagine a use-case where having several MB of POJOs in memory is an efficient solution. I think it'd be much better to use SQLite, and this is why:
I could put the data to a sqlite database, but that would require rewriting the external application and changing the data structure quite a bit.
You can still use your other program's JSON output, but instead of loading everything into POJOs with Jackson, you could populate the database on first app launch. This way, the app boot time is negligible if the dataset is unchanged.
If you still want to work with POJOs in the rest of your app, it'd be trivial to write a query that retrieved data from the database, and created objects in the same manner as Jackson.
But then I could only query the things I need at the moment and not loading the entire thing at once.
What're you doing that requires access to all the data at once? Searching or ordering a set of objects is always going to be slower than a SQL query to achieve the same thing.
Also, is there a good way to publish the app with the sqlite database?
You can definitely ship your app with a database, though I've not done so personally. This is a relevant question:
By Default load some data into our database sqlite
Hope that's of some help.
There's an excellent API called JExcel (just google it) that works with .xls spreadsheets. If you're not going to be doing any selecting and just loading data from a source, I like to use JExcel because it's more manageable from a desktop and produces easier-to-read code.
Not sure of any performance differences, however. Just throwing in my 2 cents :p
There is a web service that provides some data that my app makes use of. This data is fairly large and only changes VERY infrequently so I thought it would be nice if the app could cache it on the SD Card and only update it as needed.
Currently I'm grabbing the data (an XML file) and parsing it into an object tree using SAX. This process takes (at most) 2-3 seconds over my WIFI. However, serializing the resulting objects to the SDCard takes significantly longer (a minute or more) and deserializing it still takes longer than just download/parsing in the first place.
Does anyone have any recommendations for improving this or alternate ideas for persisting this data (other than just saving the XML file and reparsing every time)?
UPDATE: This is more than a trivial collection of records. The object-graph is actually ridiculously complex and storing it into a database would result in dozens of tables with only a single record in each one.
Android serialization is notoriously slow. I highly suggest switching to using XML or JSON (or whatever) and writing the file out that way. Since you've already got an XML parser, it may make the most sense just to cache the original XML file you downloaded and reparse it as necessary.
I have switched from Serializable to JSON file storage in an app before and the speed increase was incredible, at least one order of magnitude.
(I may be misunderstanding your question - I assume you are using Serializable for writing to the disc. If you are reproducing the XML, then I'm not sure why it is so much slower on the SD card. Also, I agree that the SQLite database makes the most sense typically, but as you've already stated it does not fit the needs of your application.)
Also unless your data is at least 100s of Kb, I would suggest just storing it in your private data storage instead of on the SD card. Keep in mind that you can't rely on the SD card being available.
I've just been writing an android application for the last week which basically does this. It fetches some (large) XML file online, and then displays part of the data in various views.
We do it by fetching and parsing the XML using SAX, and (while parsing) writing it all to a SQLite database. And then we are just querying the database each time we need to display some view of the dataset.
Works like a charm, and is fast enough for displaying a lot of data on a google map overlay, where we are querying the database on every single call to the draw method of our map overlay.
So I would definitely suggest going for a SQLite database, if the data in the XML document is easily represented in a database.
If the web service can give you just a specified number of results(something like: requestData between index 1 and 10 or give me first 25 results) try to use that (put a simple "Load more results" button or implement an auto-loading mechanism). If the web service not provide this feature then try to save your xml on sdcard and when you need the data try to parse just a specified number of results. Hope this help!
Why don't you use database? See Android Data Storage Guide