I have this question on how is deletion done on Mobile development, in lieu with Database of course, which or what is the best practice in these field, Does deletion of records permanently in database recommended on mobile dev, or flagging in database that a record is deleted more efficient?... i know that in large scale DBMS u have to flag a record that is deleted so u can view past records i just read that on the DBMS book though, does the same principle apply on mobile development, considering that i'm only developing an application for a very small scale only
Any inputs form all you veteran DBAs and Mob. Devs out there are much Appreciated..
As databases on a mobile device are not likely to get large, deletion shouldn't be too inefficient.
But I would say it is useful to flag deleted rows instead, as usually databases in these scenarios are used to hold data locally when the device is off-line pending synchronization to the network, and it's much easier to query deletions and upload them rather than diffing the local rows with remote ones to find what needs to be deleted.
It depends on the design requirements of your application, so it has nothing to do with the efficiency of app. What you read is for large scale systems where audit is required. Since you are developing an application which will run on mobile platform and would probably not require auditing, so in that case its fine to delete such records (at least that's what I would prefer to do).
Flagging records would increase the size of database and is not a good idea to leave unwanted data (memory) on your device.
Related
My app are sometime needed syncing with web servers and pull the data in mobile sqlite database for offline usages, so database size is keep growing exponentially.
I want to know how the professional app like whatsapp,hike,evernote etc manage their offline sqlite database.
Please suggest me the steps to solve this problem.
PS: I am asking about offline database (i.e growing in the size after syncing) management do not confuse with database syncing with web servers.
I do not know how large is your data size is. However, I think it should not be a problem storing reasonably large data into the internal memory of an application. The internal memory is shared among all applications and hence it can grow until the storage getting filled.
In my opinion, the main problem here is the query time if you do not have the proper indexing to your database tables. Otherwise, keeping the databases in your internal storage is completely fine and I think you do not have to be worried about the amount of data which can be stored in the internal storage of an application as the newer Android devices provide better storage capability.
Hence, if your database is really big, which does not fit into the internal memory, you might consider having the data only which is being used frequently and delete otherwise. This highly depends on the use case of your application.
In one of the applications that I developed, I stored some large databases in the external memory and copied them into the internal memory whenever it was necessary. Copying the database from external storage into internal storage took some time (few seconds) though. However, once the database got copied I could run queries efficiently.
Let me know if you need any help or clarification for some points. I hope that helps you.
For max size databases. AFAIK You don't want to loose what's on the device and force a reload.
Ensure you don't drop the database with each new release of your app when a simple alter table add column will work.
What you do archive and remove from the device give the user a way to load it in the background.
There might be some Apps / databases where you can find a documentation, but probably this case is limited and an exception.
So to know exactly what's going on you need to create some snapshots of the databases. You can start with that of one app only, or do it directly with several, but without analyzing you won't get a reliable statement.
The reasons might be even different for each app as databases and app-features differ naturally too.
Faster growth in size than amount of incoming content might be related to cache-tables or indexing for searches, but perhaps there exist other reasons too. Without verification and some important basic-info about it, it's impossible to tell you a detailed reason.
It's possible that table-names of a database give already some hints, but if tablenames or even fields just use meaningless strings, then you've to analyze the data inside including the changes between snapshots.
The following link will help in understanding what exactly Whatsapp is using,
https://www.quora.com/How-is-the-Whatsapp-database-structured
Not really sure if you have to keep all the data all the time stored on the device, but if you have a choice you can always use cloud services (like FCM, AWS) to store or backup most of the data. If you need to keep all the data on the device, then perhaps one way is to use Caching mechanisms in your app.
For Example - Using LRU (Least Recently Used) to cache/store the data that you need on the device, while storing the rest on the cloud, and deleting whats unneeded from the device. If needed you can always retrieve the data on demand (i.e. if the user tries to pull to refresh or on a different action) and delete it whenever its not being used.
I'm really new to programming apps - so this question might sound a bit strange:
I'm trying to program an app in android studio, where people can upload different things (basically strings and links put together in some kind of "package") and other peoble can then decide what "packages" they want to add inside their app. However after downloading, this data should be stored on their device and not just in the memory of the phone so that they can use it after restarting the app (and also if theres no internet connection). Do you have any idea what would be the best way to store this data both on the phone and in a database and how to synchronize the data on the phone with the selected data from the database. I really dont want to know how to do this exactly but would rather like some basic ideas and maybe you could tell me what kinds of stuff i should learn in order to succeed and what kind of database would be best here (firebase, MYSQL,..)?
Thanks a lot,
Andi
First of all you should decide what DB you are going to use.
In my opinion all RDBMs are good, but using Sqlite in order to achieve best performance on android devices is a good idea.
In your case you need server-side DB and application too.
(Depend on the scenario and framework you use can be different (sql,mysql,PostgreSQL,oracle,...)).
About how to sync local database with server-side you can download new DB from server and replace it with previous one, if you need previous user data you can have 2 different table and update one by downloading it from server, and save id or any identical row from specific package that already saved by user.
These are some question has been already answered in Stackoverflow
java - How to update table in sqlite?
java - SQLite in Android How to update a specific row
Create SQLite database in android
If you are talking about local databases. Go for Realm or look up a good ORM on github (Object relational mapping, you dont have to write SQL queries explicitly) .
I would suggest Realm which is very fast and user friendly.
I'm asking for a piece of advice. Currently, we are developing android client for out service. Service produces like a lot of dynamic information, and it must be stored on users phone so it can be accessed without connection to the net. On iOS client we achieved this using restKit. On android I found that there is no tool like restKit. So there are 2 options - use sqlite db or cache last json response. I want to use sqlite db, but our android developer sad that it's not stable and slow. Is he right? What practice is better?
The second question is that I found a sqlite editor app, which allows to edit sqlite databases on phone. Is there any way to avoid this?
You can use SQLite because SQLite is capable of being extremely fast. If you are seeing speeds slower than other DB systems such as MySQL or PostGres, then you are not utilizing SQLite to its full potential. Optimizing for speed appears to be the second priority of D. Richard Hipp, the author and maintainer of SQLite. The first is data integrity and verifiability.
The first thing you should know is that most of the time spent by SQLite (and most other DB systems) is on disk access, which is slow compared to memory operations. So the key to optimizing SQLite is to minimize the amount of disk access required. This requires some understanding of when SQLite is accessing information on disk, and why. Because operating systems generally cache open disk files in memory, a carefully tuned use of SQLite can approach the speed of a pure in-memory database.
If you are new to optimization, you must know that the only reliable way to optimize for speed is to measure where time is being spent. Unless you are already familiar with doing this for SQLite, you will often guess wrong. Use quantitative tests on representative examples whenever possible. Unfortunately, reproducibly measuring performance on an application that does disk access isn't trivial.
However it is difficult to use cache no doubt cache is fast but its not for large data and data cannot stored on cache for long time. So if you want that user will use your app offline then you should place your data on SQLite in an optimized way.
Hope this will help you.
No according to experience SQLite is the most reliable database to use in android device itself. It doesn't have separate server process and it directly read/right to single disk file.
This link will provide more information
I have a huge database and I want my application to work with it as soon as possible. I'm using android so resources are more restricted. I know that its not a good idea to storage huge data in the sqlite database, but I need this.
Each database contain only ONE table and I use it READ only.
What advice can you give me to optimize databases as much as possible. I've already read this post, and except the PRAGMA commands what else can I use?
Maybe there are some special types of the tables which are restricted for read only queries, but principally faster then ordinary table types?
As long as your database fits on the device, there is no problem with that; you'll just have less space for other apps.
There is no special table type. However, if you have queries that use only a subset of a table's columns, and if you have enough space left, consider adding one or more covering indexes.
Being read-only allows the database to be optimized on the desktop, before you deploy it:
set page size, etc.;
create useful indexes;
ANALYZE
VACUUM
In your app, you might experiment with increasing the page cache size, but if your working set is larger than free memory, that won't help anyway. In any case, random reads from flash are fast, so that would not be much of a problem.
Huge is relative. But ultimately a device is constrained on storage and memory. So assuming that huge is beyond the typical constraints of a device, you have a few options.
The first option is to store your huge dataset in the cloud and the connected device can offer views into that data by offering cloud services with something like RESTful APIs from the coud to proffer the data to the device. If the device and app rely on always being connected, you don't need as much local storage unless you want to cache data.
Another approach is an occasionally connected device (sometimes offline) where you pull down a slice of the most relevant data to work on to the device. In that model, yo can work offline and push/pull back to the cloud. In this model, sqlite is the storage mechanism to hold that slice of relevant data.
EDIT based on comments:
Concerning optimizing what you have on the device, see the optimization FAQ here:
http://web.utk.edu/~jplyon/sqlite/SQLite_optimization_FAQ.html
(in rough order of effectiveness)
Use an in-memory database
Use BEGIN TRANSACTION and END TRANSACTION
Use indexes Use PRAGMA cache_size
Use PRAGMA synchronous=OFF
Compact the database
Replace the memory allocation library
Use PRAGMA count_changes=OFF
Maybe I'm stating the obvious but you should probably just open it with the SQLITE_OPEN_READONLY flag to sqlite3_open: I think that SQLite will take advantage of this fact and optimize the behaviour of the engine.
Note that all normal SQL(ite) optimization tips still apply (e.g. VACUUMing to finalize the database, setting the correct page size at database creation, proper indexes and so on...)
In addition, if you have multiple threads accessing the database in your application, you may want to try out also the SQLITE_OPEN_NOMUTEX and SQLITE_OPEN_SHAREDCACHE flags (they require sqlite3_open_v2, though)
Also you need journalling switch off, because data not change http://www.sqlite.org/pragma.html#pragma_journal_mode
PRAGMA journal_mode=OFF
I am new to Android Application Development and a new member at stackoverflow. I am currently trying to design a recipe application. I have decided upon the features of the app and the scope it will cover. The scope is very vast for me in terms of covering all the recipes from all over the world. I am to deal with a lot of data in this process.
I am currently trying to figure a good and efficient way of handling the data in my app. So far, as per what I have read in different forums, I believe that I have two options in terms of a database choice : 1) SQLite 2) Database on remote server (MySql/Postgre)
Following are some of the thoughts that have been going on in my mind when it comes to taking a decision between the two :
1) SQLite : This could be a good option but would be slow as it would need to access the file system. I could eliminate the slowness by performing DB data fetch tasks in the AsyncTask. But then there could be a limitation of the storage on different phones. Also I believe using SQLite would be easier as compared to using a remote DB.
2) Remote Database : The issue that I can see here is the slowness with multiple DB requests coming at the same time. Can I use threads here in some way to queue multiple requests and handle them one by one ? Is there an efficient way to do this.
Also I have one more question in terms of the formatting of my data once I pull it out from the above DB's. Is there a way I could preserve the formatting of my data ?
I would be more than thankful if someone could share their knowledgeable and expert comments on the above scenario. Also this is not a homework for me and I am not looking for any ready made code solutions. I am just looking for hints/suggestions that would help me clear my thoughts and help me take a decision. I have been looking for this for sometime now but was not able to find concrete information. I hope I will get some good advice here from the experienced people who might have encountered similar situation.
Thanks for reading this long post.
What about combining both approaches?
A local SQLite database that has the least recently used receipes so you don't need network all the time. Network is way slower than accessing the filesystem.
Some remote database accessed via some HTTP interface where you can read / write the whole database. And if you want users to be able to add receipes for other users to see you'll need an external database anyways.
SQLite : This could be a good option but would be slow as it would need to access the file system.
Accessing a local database is pretty fast, 5ms or so if it's just a simple read only query on a small database.
But then there could be a limitation of the storage on different phones
Depends on your definition of huge database. It is okay if it is only 2MB which would be enough to store lots of text-only receipes.
Also I believe using SQLite would be easier as compared to using a remote DB.
Yes, Android has a nice built-in SQLite API but no remote database API. And you don't need to setup a database server & interface.
The issue that I can see here is the slowness with multiple DB requests coming at the same time.
A decent database server can handle thousands of requests. Depends on your server hardware & software. https://dba.stackexchange.com/ should have more info on that. Required performance depends on how much users you have / expect.
I'd suggest a simple REST interface to your database since it's pretty lightweight but does not expose your database directly to the web. There are tons of tutorials and books about creating such interfaces to databases. There are even hosted database services like nextDb that do most of the work for you.
Is there a way I could preserve the formatting of my data ?
You could store HTML formatted data in your database and display it in a WebView or a TextView (via Html#fromHtml()) - both can display formatted text.
Databases don't care what type of text you store, for transfer over the internets you may need to encode the text so it does not interfere with the transport formatting (XML, JSON, ...).
A simple way is to integrate Parse into your app. They have a nice framework that easily integrates into iOS and Android. Their plan is freemium, so you'll be able to use up to 1 million API request for no charge, and then its 7 cents for every request after that.
You'll have 1gb to store all your data sets / images, etc.
I don't use parse for everything, but I HIGHLY recommended it for large data schemes because they do all the scaling for you. Check out the API, I think it would be worth your time.
I just started to work on a few of my own projects, and I'm using Parse again. I have to say it's improved a lot over the last 6-8 months. Especially with the Twitter and Facebook integration.
The key issue here is the size of the data - any significant database of recipes would be too large to store on the phone imho,thus you seem stuck with the remote database solution.
As opposed to trying access the remote database from android I suggest you use a a go between web application that will process requests from the application and return JSON objects that you need.
It totally depends on your software requirements. If you need to deal with a small amount of data then you may choose SQLite, but for a huge amount to data better use a remote DB.
SQLite: It works fine with little amount of data & I experienced it response time is good.
Remote DB: I think you may use small server side app to submit the data to your client app. It will solve/reduce your thread related issues/complexities.