Develop Reference

How to Manage increasing size of sqlite in Mobile App?

My app are sometime needed syncing with web servers and pull the data in mobile sqlite database for offline usages, so database size is keep growing exponentially.
I want to know how the professional app like whatsapp,hike,evernote etc manage their offline sqlite database.
Please suggest me the steps to solve this problem.
PS: I am asking about offline database (i.e growing in the size after syncing) management do not confuse with database syncing with web servers.

I do not know how large is your data size is. However, I think it should not be a problem storing reasonably large data into the internal memory of an application. The internal memory is shared among all applications and hence it can grow until the storage getting filled.
In my opinion, the main problem here is the query time if you do not have the proper indexing to your database tables. Otherwise, keeping the databases in your internal storage is completely fine and I think you do not have to be worried about the amount of data which can be stored in the internal storage of an application as the newer Android devices provide better storage capability.
Hence, if your database is really big, which does not fit into the internal memory, you might consider having the data only which is being used frequently and delete otherwise. This highly depends on the use case of your application.
In one of the applications that I developed, I stored some large databases in the external memory and copied them into the internal memory whenever it was necessary. Copying the database from external storage into internal storage took some time (few seconds) though. However, once the database got copied I could run queries efficiently.
Let me know if you need any help or clarification for some points. I hope that helps you.

For max size databases. AFAIK You don't want to loose what's on the device and force a reload.
Ensure you don't drop the database with each new release of your app when a simple alter table add column will work.
What you do archive and remove from the device give the user a way to load it in the background.

There might be some Apps / databases where you can find a documentation, but probably this case is limited and an exception.
So to know exactly what's going on you need to create some snapshots of the databases. You can start with that of one app only, or do it directly with several, but without analyzing you won't get a reliable statement.
The reasons might be even different for each app as databases and app-features differ naturally too.
Faster growth in size than amount of incoming content might be related to cache-tables or indexing for searches, but perhaps there exist other reasons too. Without verification and some important basic-info about it, it's impossible to tell you a detailed reason.
It's possible that table-names of a database give already some hints, but if tablenames or even fields just use meaningless strings, then you've to analyze the data inside including the changes between snapshots.

The following link will help in understanding what exactly Whatsapp is using,
https://www.quora.com/How-is-the-Whatsapp-database-structured
Not really sure if you have to keep all the data all the time stored on the device, but if you have a choice you can always use cloud services (like FCM, AWS) to store or backup most of the data. If you need to keep all the data on the device, then perhaps one way is to use Caching mechanisms in your app.
For Example - Using LRU (Least Recently Used) to cache/store the data that you need on the device, while storing the rest on the cloud, and deleting whats unneeded from the device. If needed you can always retrieve the data on demand (i.e. if the user tries to pull to refresh or on a different action) and delete it whenever its not being used.

android application with huge database

Let me explain how my application is supposed to work:
Application will ship with a sqlite database in its assets folder which will be copied into databases folder later and it has some content in it(categories, sub categories, products and news) which they all have image. Then after download user can update the content via internet and the application store the new content in database so the application can perform offline.
So my question is, after a while this content will increase in size, is it gonna cause my application to crash? Lets say I release the application with 1 MB database and after 2 years of work the database size goes up around 120 MB. Is it gonna make the application to crash?
Also the other concern is that currently I'm storing the images in database and I load'em from there. Is it a good approach? Because I don't want user to be able to clear the cache or delete the images because later on updating the content it has to download those deleted images again and it will consume traffic imo.
please remember that the Application should be able to load content offline

No, applications don't just crash because they have a large database.
Part of the point of a Cursor is that it gives you a view into a large set of data, without having to load it all into memory at the same time.
If you follow best practices I see no problem - you're using a database. Forget for a second that it's on Android - you should optimize your table structure, indexes, etc, as best you can.
Also, large database or not, don't make any queries to it on the main thread. Use the Loader API if you need to show the result of a query in your UI.
Last, potentially most importantly, rethink why you even need such a large database. Is it really that common that a user will need to access all data ever while offline? Or might it make more sense for you to only store data from the last week or month, etc, and tell them that they need to be online to access older data.
Regarding your 2nd question - please in the future separate that into a separate question. But, no, storing binary blobs (images in this case) in a sqlite database is not good approach. Also, if they clear data on the app, everything is gone, so there's no advantage to using a database to avoid that. I would suggest storing images in a folder named after your app in external storage of the device, potentially storing image URIs/names in the database.

Any problem with database will cause SQLiteException which you are able to handle in your app to prevent the abnormal termination.
Having said that, a database of 120 MB seems to be too much, are you sure your users will want all that?

Android application's cache

What data is stored in application's cache directory? I'm not using it in my application and I still can see its size grows. Is it webview caching images? HttpClient storing some data? What else?
Can I erase it's contents safely from code at any time?
Thanks in advance

Android can clean the cache up whenever it's running low on space, so the assumption for whatever application using cache files should be that they can be deleted at any time. In other words, if you're using some other modules that create the cache in your application, you should be able to delete those files safely at any time, assuming those modules are well coded. What it will do to performance, that depends on many more things.
Maybe just keep the size under control, so delete the older files when you feel it gets too big? I think Android recommends 1MB for cache.

Android image caching - hard and soft HashMaps question

What I'm trying to do right now within my app is modify the ImageDownloader class that Google put out last year in one of their tutorials that asynchronously downloads and caches images for ImageViews without leaking the context.
In other words, since I'm using a global cache singleton object which provides references to my Bitmap HashMaps, I just need to know: since I have to cache images separately depending on certain aspects of my app (client requirement), should I have pairs of hard and soft HashMaps for each of those types of Bitmaps, or would it be more efficient to have only one soft HashMap in which the other hard caches move their files to when they are pressed for space?

I've encountered a similar problem on an app we worked on. We hard cache images we pull in an SQLite blob. The wrapper will check the cache's existance, or go pull over the network, so even if someone clears the app data, it'll work (a bit slower until images are cached again). Since it's in SQLite the app data can easily be moved around to the SD card and back without having to worry about file paths being changed.

Consider using something like:
ConcurrentHashMap<String, SoftReference<Bitmap>> image_cache =
newConcurrentHashMap<String, SoftReference<Bitmap>>( 1 );
For the memory portion of your cache. The SoftReference's will be garbage collected as the device needs memory. This will allow your application to keep as many images in memory as possible without causing memory issues.
You can choose to back this cache with a file cache either on the SD card with a root path using:
Environment.getExternalStorageDirectory();
or using the cache space provided by android using a path of:
context.getCacheDir();
The cache space is private whereby the SD card images could be retrieved by a user or modified by a 3rd party program. The cache space is on the internal storage of the device and it shows up in the application manager statistics. The user can also clear this cache easily from their settings->application manager screen.
You will need to fill in the algorithm that checks the memory cache first, checks the soft reference if found, checks the file system if not found, and then finally fetches from the network, saves to a file, and puts it in the memory cache. You can then add additional requirements for the cache based on your client requirements on top of this structure.

What is the ideal place to cache images?

I am fetching a lot of thumbnails in my application from a remote server and lazy-loading these in a list view.
The image fetches run in a background AsynTask and when completed the fetched images are stored in a HashMap using SoftReferences.
This works just fine but when the images in the cache gets GC’d, the fetches have to be rerun.
I could have stored them on the SD card so there would be no re-fetching.
But I did not take this approach because of the clutter it would create on the SD card.
Are there perhaps alternatives to these sorta like temporary folders that can be cleared when activity/app finishes/exits?
What is the "ideal place" to cache images? Any advice, example projects, or pointers would be appreciated.

You should check files data storage options:
http://developer.android.com/guide/topics/data/data-storage.html
Files are being saved in the applications directory, so there is no mess anywhere. Files will be finally removed with the app. You can also implement some method which will clear the cache when the app starts (for example remove files older than 2 weeks).

You should be able to locate a SQLite DB on the SD card. This would let you have a fairly clean interface/cache space while keeping data off the very limited internal phone memory. You can specify the location of the DB by calling openOrCreateDatabase(String path, SQLiteDatabase.CursorFactory factory). It may be more work than you'd like to put in, but you should be able to clear ou the DB when you application is unloaded with a line or two of code.
More from the documentation here:
http://developer.android.com/reference/android/database/sqlite/SQLiteDatabase.html