For my app I store map images in a cache on the external storage as to reduce calls to the API that I am using. Since this data is map data, it is subject to change over time. As a result these images should be updated every-so-often.
How can I programmatically delete the cache directory periodically? Say for example, every week.
Some extra thoughts:
Perhaps deleting the entire directory is not the best way to go about it. Perhaps I can check the "freshness" of each image, and delete the old ones?
Yes, it's usually a good idea to limit your cache with some method or another. Some caches have fixed sizes, and older content is removed once your cache size is exceeded. This size could be a count of the number of items or a size in disk space. Some caches have TTLs for each item (or the same TTL for all items) and so items expire after a certain time. Some caches may never expire. Perhaps it is bounded in size by the number of possible items that would be cached in the first place. Any of these approaches are valid, though some may be more appropriate for certain scenarios than others.
The "freshness" idea is probably an approach I'd consider. It's the same as the TTL. Basically you want to set a length of time that your cache item will live for. Store this cache time along with the cache data, then check it whenever reading the cache data. If it's past the expiration date, you can delete that cache file and retrieve the map data from the API again (and cache that). You could probably do something with just reading the file creation time too if you don't want to store a date separately and have a fixed lifespan hard-coded or configurable in your application.
Update to address comments:
I've used hashing the URL as the filename before too. Just be aware that there's a possibility of collisions (highly dependent on your hash algorithm and your data set of course). Also, if you're going through a lot of URLs, the performance of your hash algorithm might matter too.
Storing that cache metadata in a text file is fine, especially if you don't have a ton of URLs. You'll want to be careful about how you update that text file though. You could easily corrupt it if you're not careful and access it from multiple threads without proper synchronization. If you have a lot of data, another option you can consider is using a database. If you do store this cache metadata in a file--whether database or text file--you can avoid all the problems with hashing by using a different scheme for your filenames. You could just increment in hex or base 36 for example.
Related
My app are sometime needed syncing with web servers and pull the data in mobile sqlite database for offline usages, so database size is keep growing exponentially.
I want to know how the professional app like whatsapp,hike,evernote etc manage their offline sqlite database.
Please suggest me the steps to solve this problem.
PS: I am asking about offline database (i.e growing in the size after syncing) management do not confuse with database syncing with web servers.
I do not know how large is your data size is. However, I think it should not be a problem storing reasonably large data into the internal memory of an application. The internal memory is shared among all applications and hence it can grow until the storage getting filled.
In my opinion, the main problem here is the query time if you do not have the proper indexing to your database tables. Otherwise, keeping the databases in your internal storage is completely fine and I think you do not have to be worried about the amount of data which can be stored in the internal storage of an application as the newer Android devices provide better storage capability.
Hence, if your database is really big, which does not fit into the internal memory, you might consider having the data only which is being used frequently and delete otherwise. This highly depends on the use case of your application.
In one of the applications that I developed, I stored some large databases in the external memory and copied them into the internal memory whenever it was necessary. Copying the database from external storage into internal storage took some time (few seconds) though. However, once the database got copied I could run queries efficiently.
Let me know if you need any help or clarification for some points. I hope that helps you.
For max size databases. AFAIK You don't want to loose what's on the device and force a reload.
Ensure you don't drop the database with each new release of your app when a simple alter table add column will work.
What you do archive and remove from the device give the user a way to load it in the background.
There might be some Apps / databases where you can find a documentation, but probably this case is limited and an exception.
So to know exactly what's going on you need to create some snapshots of the databases. You can start with that of one app only, or do it directly with several, but without analyzing you won't get a reliable statement.
The reasons might be even different for each app as databases and app-features differ naturally too.
Faster growth in size than amount of incoming content might be related to cache-tables or indexing for searches, but perhaps there exist other reasons too. Without verification and some important basic-info about it, it's impossible to tell you a detailed reason.
It's possible that table-names of a database give already some hints, but if tablenames or even fields just use meaningless strings, then you've to analyze the data inside including the changes between snapshots.
The following link will help in understanding what exactly Whatsapp is using,
https://www.quora.com/How-is-the-Whatsapp-database-structured
Not really sure if you have to keep all the data all the time stored on the device, but if you have a choice you can always use cloud services (like FCM, AWS) to store or backup most of the data. If you need to keep all the data on the device, then perhaps one way is to use Caching mechanisms in your app.
For Example - Using LRU (Least Recently Used) to cache/store the data that you need on the device, while storing the rest on the cloud, and deleting whats unneeded from the device. If needed you can always retrieve the data on demand (i.e. if the user tries to pull to refresh or on a different action) and delete it whenever its not being used.
I have a list of about 88,000 surnames,
I need to read it every few minutes, so instead I read it once and store it in a list.
It that better or should I read every time.
Memory vs Run-Time
It that better or should I read every time
Neither, IMHO. Use a smarter cache.
Have a static thread-safe cache for this data.
Design the API around that cache such that the cache can be lazy-loaded. Whoever is seeking to examine the cache should get the results asynchronously, so if the cache is empty, you can load the cache contents on a background thread. This could be accomplished using a listener/callback pattern, RxJava/RxAndroid, LiveData, etc.
Have a custom subclass of Application, and in there, override onTrimMemory(). If onTrimMemory() is called with a value of TRIM_MEMORY_BACKGROUND or higher, empty the cache, so that your memory usage is lower when the user is not actively using your app.
This way, the code using the cache does not care whether the data needs to be loaded or not, and you can minimize memory consumption when it is unlikely that the cache needs to be used.
Or, if you never need all 88,000 surnames in memory at once, store them in a SQLite database and query that database as needed.
In general run-time uses memory so you should be on same. Difference is that you can store in memory as an object and remove it from memory as you need so some memory picks will be present instead of consistent memory load.
Did you consider using API or JavaScript for that logic and load JSON instead? Approach with different architecture using API or JavaScript as business logic to load JSON might help with loading and unloading object that contains data. JSON can be static stored locally or even dynamic (stored in remote server) which can help with easy update of your surnames. Additionally, this approach can query data as per parameters passed so you might need not to load all 80.0000 surnames at once. I see that question is about Android but architecture approach could be the same for any device.
I hope this might help you or somebody else to determine approach depending on architecture and mobile platform (is not really defined in question if any is used).
Cheers!
I am currently storing bitmap images inside my cache, each bitmap can be upto 3mb in size each. I am using getCacheDir() however, after reading androids documentation, I found this:
Note: you should not rely on the system deleting these files for you; you should always have a reasonable maximum, such as 1 MB, for the amount of space you consume with cache files, and prune those files when exceeding that space.
So I am considering switching my cache to using getExternalCacheDir(), but I am abit uncertain about this:
The platform does not always monitor the space available in shared storage, and thus may not automatically delete these files. Apps should always manage the maximum space used in this location. Currently the only time files here will be deleted by the platform is when running on JELLY_BEAN_MR1 or later and isExternalStorageEmulated(File) returns true.
I would like to have full control of the cache dir, because if files are randomly deleted this could really affect the running of my application. I have already set an upper limit for the size of the cache so it will never exceed a certain amount of space. How can I now prevent the system from monitoring and deleting from this cache as it pleases?
thanks
Don't think that cache is permanent storage. You mustn't store there files that you rely on. Cache used for storing data that you don't really want to reload or recreate, but you can do it if needed (as with loading pictures from web. Much faster to load from disk, if already done, but can be done without it). If you really depend on this files use getFilesDir() this will route you to specific app directory, that wouldn't be erased if no memory (but still can be erased by user in settings). Also, if you have data that is static and you need it always, you must store it in the assets folder, all other data must be considered as temporary (any time user or system can erase it) and checked for existence.
There are many similar questions about this issue but I have clear points about my question to ask you.
I am new at Android development and before only I developed small applications which store small sized data. For example country List, calendar, birthday reminder etc. I stored my small data in single XML file and I parsed it with easy methods. This was enough for me. But for my Mobile Application Development Course I took a project which will store huge static data.
Specifications of my project will like these:
There are about 200 entities.
Each entity has about 20 sub categories which they stored in text format.
Each sub category has about 30-sub categories which they stored again in text format.
Also for each parent entity I will have 2-3 image
If I calculate simply, I have to store 200 X 20 X 30 = 120.000 static data for my application and data does not change. This is only install and use application. Online data interaction is not necessary. (If there are some changes for data I will relase major updates in long periods of time.)
My question is about storing method.
Which way should I choose? SQLite or XML parsing? For your answer can you explain advantages / disadvantages for your choice?
Interesting project, although not necessarily realistic.
To manage a large amount of "static" data, you'll want a database. XML parsing forces you to store the data in memory, which means that you have to read it into memory on a regular basis. Remember that you can't count the in-memory data being around when the user goes to your app; Android may have destroyed your app previously.
On the other hand, you can use an SQLite database on disk directly from your app. It's persistent, even if your app goes away. You'll have to load the database once, when you install the app.
Consider wrapping your SQLite database in a content provider. This will, among other things, allow you to do asynchronous queries using a CursorLoader.
What I'm trying to do right now within my app is modify the ImageDownloader class that Google put out last year in one of their tutorials that asynchronously downloads and caches images for ImageViews without leaking the context.
In other words, since I'm using a global cache singleton object which provides references to my Bitmap HashMaps, I just need to know: since I have to cache images separately depending on certain aspects of my app (client requirement), should I have pairs of hard and soft HashMaps for each of those types of Bitmaps, or would it be more efficient to have only one soft HashMap in which the other hard caches move their files to when they are pressed for space?
I've encountered a similar problem on an app we worked on. We hard cache images we pull in an SQLite blob. The wrapper will check the cache's existance, or go pull over the network, so even if someone clears the app data, it'll work (a bit slower until images are cached again). Since it's in SQLite the app data can easily be moved around to the SD card and back without having to worry about file paths being changed.
Consider using something like:
ConcurrentHashMap<String, SoftReference<Bitmap>> image_cache =
newConcurrentHashMap<String, SoftReference<Bitmap>>( 1 );
For the memory portion of your cache. The SoftReference's will be garbage collected as the device needs memory. This will allow your application to keep as many images in memory as possible without causing memory issues.
You can choose to back this cache with a file cache either on the SD card with a root path using:
Environment.getExternalStorageDirectory();
or using the cache space provided by android using a path of:
context.getCacheDir();
The cache space is private whereby the SD card images could be retrieved by a user or modified by a 3rd party program. The cache space is on the internal storage of the device and it shows up in the application manager statistics. The user can also clear this cache easily from their settings->application manager screen.
You will need to fill in the algorithm that checks the memory cache first, checks the soft reference if found, checks the file system if not found, and then finally fetches from the network, saves to a file, and puts it in the memory cache. You can then add additional requirements for the cache based on your client requirements on top of this structure.