I will be making a mobile application in Android. My application is like Google Map's Get Direction feature, but a lot more complex, so I need to store data about points in the map. So I'm worried that SQLite may not be able to handle these large amount of data(or considering the limited storage of the phone). I have no background in SQLite so please bear with me.
SQLite can handle large amount of data, the problem here is the device's limits. If you are going to store 3MB or more you should consider saving that data in an external server and access it via the Internet. In fact, when you are building an application that use large amount of data, usually the application don't use all data all the time, so you can save in cache (in a local database) the data that the app is currently using or is about to use.
I think the best way to find out is to write a simple app that simulates the types of transactions you'll be doing and see how it does.
You might also want to compare how SQLite does to an object database like db4o, which is very performant and used very often as an embedded database (and can easily handle gigs of data).
Related
I'm working on an app and I have tried to use local storage but I'm not sure if it will be OK for my project.
I need to store really large data every second in my local database, and at the end of the day send to my server. This can be large data because every second write my GPS coordinates.
What is best: SQLite, local storage or websql?
I work using cordova.
I appreciate for your help.
Regards
I am an Android developer so this is how it works in Android :
Shared Preferences - simple key/value pairs specific to your application. This is probably the closest to WebStorage - only for small amount of data.
Internal Storage - read/write files(only the app can access it)
External Storage - SD card file storage(app + other apps including the user) can access it).
Databases (SQLite) - better for large amounts of structured data
Network - obviously, you can store/retrieve data remotely if needed (like Firebase).
What would I choose? if data is structured then I would choose SQLite.
More detailed information :
https://developer.android.com/training/data-storage
cordova-sqlite-storage is Native SQLite component with API based.
It will easily handle large data and easy for CRUD operation.
Yes you can use cordova-sqlite-storage for large data inputs. But I think it is also advisable not to stress your device with so much data in it because mobile is just mobile with so many limitations in terms of hardware. Maybe you can just set a treshold of data size and after reaching it, send it to your server then clear your database.
The point is don't stress you device, use minimum data as much as possible to provide a clean app for the users.
i have a question which i know its answers vary from one app to another.
consider an app like Telegram which you have a noticeably amount of data there like your contacts and chats. when you close the app and later you relaunch that, even if you don't have any connection to internet, app loads your contacts and chats and this is done in fraction of second. i want to know how these works are done and implemented?
for example they store all the data in disk in a file(storing that amount of data on disk can take long time moreover they cache images too), or we save each part of app in different files and we load them whenever user opens them.
You have two commonly used ways to store data in Android. SharedPreferences and local databases.
Before I continue, you should check out other articles regarding storing data on android. There are loads of them explaining it better and in-depth than what I can, I will just give some examples what you can use.
Start here or just google it!
SharedPreferences
SharedPreferences works in a Key -> Value, used for saving basic types of data. Good for storing user tokens, settings and similar things.
See more
Databases
Android uses SQLite for local data storage, and there are many libraries that can help you with this.
More notably Room, but there are other ones as well that are very popular to use. Databases are good for storing large amounts of data, like conversations and contacts in the example you are talking about.
Caching
You also have access to caching data, but keep in mind that it's meant as a temporary storage, and if you want to store something like user information or other complex data, you should stick to a database. Caching is useful for images.
This is called api data caching. It is important that we should know which data to be cached and which data should not be cached.
There is a library from NYtimes you can use it for api level caching with retrofit.
My app are sometime needed syncing with web servers and pull the data in mobile sqlite database for offline usages, so database size is keep growing exponentially.
I want to know how the professional app like whatsapp,hike,evernote etc manage their offline sqlite database.
Please suggest me the steps to solve this problem.
PS: I am asking about offline database (i.e growing in the size after syncing) management do not confuse with database syncing with web servers.
I do not know how large is your data size is. However, I think it should not be a problem storing reasonably large data into the internal memory of an application. The internal memory is shared among all applications and hence it can grow until the storage getting filled.
In my opinion, the main problem here is the query time if you do not have the proper indexing to your database tables. Otherwise, keeping the databases in your internal storage is completely fine and I think you do not have to be worried about the amount of data which can be stored in the internal storage of an application as the newer Android devices provide better storage capability.
Hence, if your database is really big, which does not fit into the internal memory, you might consider having the data only which is being used frequently and delete otherwise. This highly depends on the use case of your application.
In one of the applications that I developed, I stored some large databases in the external memory and copied them into the internal memory whenever it was necessary. Copying the database from external storage into internal storage took some time (few seconds) though. However, once the database got copied I could run queries efficiently.
Let me know if you need any help or clarification for some points. I hope that helps you.
For max size databases. AFAIK You don't want to loose what's on the device and force a reload.
Ensure you don't drop the database with each new release of your app when a simple alter table add column will work.
What you do archive and remove from the device give the user a way to load it in the background.
There might be some Apps / databases where you can find a documentation, but probably this case is limited and an exception.
So to know exactly what's going on you need to create some snapshots of the databases. You can start with that of one app only, or do it directly with several, but without analyzing you won't get a reliable statement.
The reasons might be even different for each app as databases and app-features differ naturally too.
Faster growth in size than amount of incoming content might be related to cache-tables or indexing for searches, but perhaps there exist other reasons too. Without verification and some important basic-info about it, it's impossible to tell you a detailed reason.
It's possible that table-names of a database give already some hints, but if tablenames or even fields just use meaningless strings, then you've to analyze the data inside including the changes between snapshots.
The following link will help in understanding what exactly Whatsapp is using,
https://www.quora.com/How-is-the-Whatsapp-database-structured
Not really sure if you have to keep all the data all the time stored on the device, but if you have a choice you can always use cloud services (like FCM, AWS) to store or backup most of the data. If you need to keep all the data on the device, then perhaps one way is to use Caching mechanisms in your app.
For Example - Using LRU (Least Recently Used) to cache/store the data that you need on the device, while storing the rest on the cloud, and deleting whats unneeded from the device. If needed you can always retrieve the data on demand (i.e. if the user tries to pull to refresh or on a different action) and delete it whenever its not being used.
I am building a mobile appfor Reporting and Business Intelligence. The app will show a live stream of billing, payments and other data to management of a company which is doing business across 70 different physical locations. The idea is to see what is happening in the business across all locations as a live stream on line graphs on your tablet.
However the question I have in mind is that with billing and payments data across 70 locations, this stream of data will be huge, and continuous. Over 1 year this dataset will exceed 10GB. Should so much of information be kept in a sqllite database on a mobile? Are there any best practises on handling such large datasets on a mobile app? Tx.
For large databases you should use Realm. It is noSql db and it is more efficient with large data sets. But, more importantly, you can store realm database as ordinary file on SD card.
As I understood from your description, you need to store very large volume of data, but you don't have to access any portion of it (access latest data immediately and previous data with some delay).
You can create system's architecture in that way, so there will be 'database adapter' for Realm, which will take one file, extract data from it, convert it to anther format (in background, of coarse) and then upload those data to AWS or any other cloud storage.
If you use Realm, you can just monitor file size (database is ordinary file), and, if it exceeds given value, create new database file and process previous file in the way I mentioned.
Another approach - interpolate your data and save interpolation factors, but there is math envolved and some data will be lost.
Yeh, all data is kept on remote server, and you uploading some pieces when you need it. Don't know if it's what you searching from.
I am developing an application that periodically sends information to an external server. I make a local copy of the data being sent, for backup purposes.
What is the best option to store the data in terms of saving battery life? Each data submission is a serialized object (the class has 5 fields, including a date, numbers and strings) of about 5K-10K.
Any other idea?
I don't believe it matters whether you use SQLite or a File, because the SQLite db is simply a file on the system (stored in /data/data/<your_package>/databases/). You'll need to commit to the db at the right times, just as much as you would need to save a file to the hard drive at the right times. In other words, one way or the other you can use just as many hard drive writes.
I think that what you choose depends more on what sort of data you are saving. If you need the powers that having a db can bestow (such as querying), then by all means use SQLite. However, if you don't need a db, or you've got data that varies wildly (and can't be easily setup in a relational database) then I'd go with files.
What I can tell you for sure is that you should not use serialization for saving a file, if that is the route you choose to go. Android serialization is slow, slow, slow and creates large files. It is much better to either write your own XML or JSON format for performance reasons.
I have no idea in terms of battery life directly but one criteria would be which is easier to manage? Fewer operations to manage the data would mean fewer CPU cycles and in turn longer battery life.
I would say the SQLite option is easier. You can put a date column in the SQLite table which stores your data which makes removing old submissions which you don't need any more very easy - and all handled via the native SQL library. Managing a whole load of file - or worse a single file - with your own Java code would be much more work.
Additionally, you can write data the to database and just forget about it until you need to read it again. If you're storing data in files, you'll need to work out when you should be reading and writing files in terms on the Android application life cycle. If you're worried about battery you probably wouldn't want to write files more often than you should, and cache data in memory, but you'd need to make sure you didn't lose any data when your app is Paused or Destroyed. In my opinion it's much easier to use an SQLite database and not worry about any of this.
Is your application multi-threaded? If you have multiple threads accessing the data store then I would go with SQLite. Let SQLite worry about locking issues.