Dealing with a large database in Android - android

I'm currently developing an app that has the potential to create a very large database. I had planned on installing the app on the SD card to allow for some extra room. Recent dealings here have warned me that that might not be a good idea. Are there any steps I can take to mitigate the danger of this course of action? Or are there any better alternatives?

This is a comprehensive post on the subject (I'm not the author).
I think, overall, it needs to be communicated that SQLite is just a SQL mechanism for accessing a file. It appears that the current market limit is 50mb for the entire APK. When installing to internal memory, you require 2x your APK size. Installing to sdcard requires just the stated APK size.
Here is what you will be working against:
1.) Since SQLite is just a abstraction over your file, when you do selects, inserts, updates, etc, you will be incurring sdcard read write costs
2.) I've seen mention of a soft limit of 10000 records based on performance. This article is a bit old, so its likely gotten better.
Other then that, you'll probably have to set up some tests to see what is feasible. Cursory search of google did not show any benchmarks to date.

As pointed out previously, SQLite has the functionality that you're looking for, packaged in a small library. It's designed to replace simple file access with reliable, recoverable, transactional data access via a SQL API. There's a great summary on their main page.
There are literally thousands of projects that are using SQLite. If your data set is going to be very large (more than 100-200MB), then you might want to consider using Berkeley DB as an option. Berkeley DB recently introduced support for a SQL API, which is completely SQLite compatible. In addition to the functionality that's provided by the SQLite SQL parser, query planner and executor, you also get the reliability and scalability that Berkeley DB is well known for. We have several customers who happily started out with SQLite. When they realized that they needed additional concurrency, scalability and reliability that not available in SQLite, they replaced the SQLite library with the BDB library, recompiled their application and had it tested and running on Berkeley within a few days.
I'm one of the Product Managers for Berkeley DB, so I'm a little biased. :-) However, we implemented the BDB SQL API so that we could offer users the best of both worlds: the ubiquity and ease-of-use of SQLite combined with the concurrency, reliability and scalability of Berkeley DB. Especially with larger data sets, Berkeley DB can make all the difference in application performance.

As mentioned above, you can use Berkeley DB (not the Java Edition) on Android. I described the steps of the cross-compiling process here since it requires minor tweaking.

Related

Alternative to core data in android?

I am building a social media application which requires local storage of table data entities. This data must also be connected to a server to retrieve and update information to and from users. Our team has built an iOS client using core data, though we are looking for storage options in android. Is using SQLite the way to go? Any thoughts would be greatly appreciated.
You should take a look at Realm, it has clients for Objective-C, Swift and Android.
Description from their GitHub repository:
Features
Mobile-first: Realm is the first database built from the ground up to run directly inside phones, tablets and wearables.
Simple: Data is directly exposed as objects and queryable by code, removing the need for ORM's riddled with performance & maintenance issues. Plus, we've worked hard to keep our API down to very few classes: most of our users pick it up intuitively, getting simple apps up & running in minutes.
Modern: Realm supports easy thread-safety, relationships & encryption.
Fast: Realm is faster than even raw SQLite on common operations, while maintaining an extremely rich feature set.
If you're familiar with RxJava, you will probably want to check SQLBrite, wich is Square's solution for this.
Yes, Sqlite is a default storage solution for android. Howevere there is a wrap around it called ContentProvider. ContentProvider can be used with Loaders and provide async data loading. ContentProvider may be used to modify contats and merge accounts, see this guide. However ContentProvider may seem tricky and if you prefer ORMs you can use ORMLite or GreenDAO which are using sqlite as well.

cross platform syncing solution with offline and relationship support

I am relatively new to app development and core data, so take it easy on me. I have been working on an app (currently for iOS, android in the future too) which stores the user's data locally using core data.
The data has relationships across entities. User can create, update, delete data.
Now I need to integrate some kind of syncing solution. My requirements are:
Data should be accessible offline (right now I am testing queue operations for that)
Data should sync to cloud storage when network is available (on iOS I have tested out reachability for this)
Cloud storage needs to be integrateble to both my current iOS and future android version.
Relationships (or some other way of linking parent-child, i talk about it later) need to be maintained.
Core data on ios should be used, not third party replacement.
I have messed around with many solutions so far:
I setup my own rethinkdb database on a server and used PHP and REST to get things going. This was very messy.
I tested parse.com and afnetworking http requests (instead of parse's library) - this was better, but i couldn't seem to be able to manage the relationships from core data. And my syncing algorithm is complicated (it works but I am not sure if there are holes in it when it may fail)
I tested dropbox datastore api. I have only tested the iOS sample app they provide, seemed pretty good (still need to understand the workings).
My questions are:
If I were to use dropbox datastore api, how does it work with android? of course core data is not available there, so how does that work (sorry I don't have android dev experience yet)? ALso how does it handle relationships between entities?
If I go with Parse.com, do you think my this idea will work-
Instead of using relationships, I can use identifiers? my relationships are all 1-to-many, so on the parent (1) I can have an id A. On all the children I can point their parentid to A. Also since my children can have grandchildren too, on each of the children I can have another id which the grandchildren can point to. So on... does this make sense as a replacement of relationships? If yes, then what's the point of relationships in xcode?? other than having automatic cascade option maybe.
Are there any better solutions available for syncing cross platform?
I know my question may seem a bit asking for opinion, but I would like to see what everyone else has already tried. Past week of switching from one solution to another and designing the syncing algorithm has fried my brain.
(I work at Dropbox and will address just that part of the question.)
The Dropbox Datastore API doesn't use core data and thus works exactly the same way on Android. Why is core data a requirement?
As to relationships, what you describe for Parse is exactly what I would suggest for use with the Datastore API. I believe you're right that the reason for modeling relationships in databases and in code is to get automatic cascading operations and enforcement of relationship invariants.

HTML5 Web SQL Database Concerns

My company is currently transitioning to a new architecture, as well as a model that provides services to mobile devices. Our application was traditionally web based (HTML5/CSS3/JS/PHP/MYSQL), so we want to be able to port it to mobile platforms without having to reinvent the wheel (Cordova), as well as to desktops in a standalone fashion (AppJS) so that we don't have to worry about browser-dependent bugs. We are also transitioning away from PHP to NodeJS to make this more feasible.
The problem is that our users NEED the ability to use our application offline, with no real limit on how long or how much data they can store before "syncing" it to our servers.
With AppJS, this isn't an issue because they can store as much data as needed within a sqlite database. However, I have discovered the issue regarding a 5MB quota for WebSQL data and cordova/phonegap. This presents obvious technical challenges regarding our business needs.
I am aware that there are a couple of plugins that enable you to use sqlite data. The original sqlite plugin (https://github.com/davibe/Phonegap-SQLitePlugin) lacks documentation for android, and the other (https://github.com/chbrody/Cordova-SQLitePlugin) requires me to specify a data limit which obviously isn't possible. I am therefore left with very few options, one being to split the databases into multiple 5MB segments which are controlled by a JS wrapper.
Can you feasibly split larger datastructures into multiple 5MB websql databases with no issues across iOS and Android?
Is there a limit on the total number of websql databases on iOS and android?
Are there any other options that you would recommend we look at? We need to be able to use our existing HTML5 and CSS for the frontend!
EDIT
Our original architecture never really worked with tablet devices. This is something we are looking to remedy with cordova, as well as building a more stable solution. Originally, our offline product ran on a Server2Go stack that had all sorts of platforming issues because of the various applications running on the clients machine
For the android platform, you could create a native app, and then use a webview to display the front end (if its really that important to keep it HTML), then use a javascript bridge to allow the content in the WebView to interact with your native app.
From what you have described about the application though, I think you are better off actually creating a native application. As you say in your comment, one of the reasons you are switching your platform is because of lack of control. Why then immediately re-open this can of worms by introducing all sorts of hybrid app framework into your stack.
Long story short, you will have full control (and quite likely, better product) if you just go with the native approach.
To get a truly cross-platform, cross-browser relational solution that will work offline and have the necessary storage capacity, I would suggest going with: SequelSphere
Since it is a 100% JavaScript solution, it should be usable in each of your configurations without requiring a different codebase for each. Furthermore, it uses IndexedDB (if available, and then LocalStorage) to store its data in relational tables. I believe most browsers do not have a size limit on the storage amount of IndexedDB. I think Firefox may have an initial 50MB limit per the following: Max size in IndexedDB.
Also, there is a new feature called "Change Trackers" that significantly help in the data synchronization process by tracking changes (inserts/updates/deletes) to tables, and then reporting them to you when you desire.
As an extra added bonus, unlike WebSQL, SequelSphere is VERY JSON friendly. Creating tables couldn't be easier:
db.catalog.createTable({
tableName: "EMPL",
columns: [ "EMPL_ID", "NAME", "AGE", "DEPT_ID" ],
primaryKey: [ "EMPL_ID" ],
data: [
[0,"Bob",32,0],
[1,"John",37,2],
[2,"Fred",28,1]
]
});
Querying the data is easy:
var res = db.query("SELECT name, age, dept_id FROM empl WHERE Dept_ID = 2");
Inserting / Updating / Deleteing data is easy:
db.insertRow("empl", [3, "Sue", 26, 2]);
db.updateRow("empl", [3, "Suzy", 26, 2]);
db.deleteRow("empl", [3, "Suzy", 26, 2]);
One thing to be aware of: Since this will be used in an offline application, make sure you set the "Persistence Scope" of either the entire Catalog, or each Table to be "SCOPE_LOCAL":
db.catalog.setPersistenceScope(db.SCOPE_LOCAL);
// -or-
db.catalog.getTable("empl").setPersistenceScope(db.SCOPE_LOCAL);
Documentation on the basics: SequelSphere Basics
Documentation on "Local Storage" support: Local Data Storage and Persistence
Here is the API: SequelSphere API
If you have any questions, just email the support for SequelSphere.
For complete transparency, I am part of SequelSphere, and it does seem to answer your question very well... ;)
Generally speaking, I recommend using lawnchair so that you can be implementation agnostic. http://brian.io/lawnchair/adapters/
Can you feasibly split larger datastructures into multiple 5MB websql
databases with no issues across iOS and Android?
Before I knew about lawnchair, I implemented this using WebSQL, which I kinda regret. WebSQL databases can grow beyond 5MBs. I'm not sure what the true max is, but I've set my Phonegap app to 100 * 1024 * 1024 and haven't had any issues on Honeycomb Android.
For iOS you could use SQLite database on PhoneGap / iOS - More than 5mb possible.

Does Android support any other database system other than SQLite?

I have been searching and have found out, that android supports only SQLite databse and no other. Is this true?
Yes. The Android library provides native support to only SQLite. Of course, this doesn't mean you absolutely can't use other databases on Android; if you need to use other databases, you'll have to either look for already-existing third-party libraries(1), or roll out your own API.
(1) Careful there: If you look for third-party libraries, make sure they're built specifically for Android, since Android includes only a subset of the Java standard library. If they're not specifically built for Android, there's a possibility that the libraries won't work due to missing classes.
Yes, Android Supports H2 Database too. please check it out with below link,
http://www.h2database.com/html/tutorial.html#android
Thanks,
Though SQLite is natively supported on Android and is most used database, there exists other options as well. Listing a few of them below;
Realm:
Reactive, concurrent, and lightweight, allowing you to work with live, native objects.
https://realm.io/docs/
H2:
Full Unicode support including UPPER() and LOWER().
Streaming API for BLOB and CLOB data.
Fulltext search.
Multiple connections.
http://www.h2database.com/html/tutorial.html#android
CouchDB:
Full CRUD and query functionality, NoSQL, lightweight, embedded, syncable
https://developer.couchbase.com/mobile/
LevelDB:
Lightweight and single purpose (not an SQL database)
http://leveldb.org/
Java wrapper: https://github.com/hf/leveldb-android
I am not including BerkeleyDB here as (AFAIK) it needs OS level changes to be made to replace the SQLite routines. More info here https://blogs.oracle.com/berkeleydb/now-you-can-build-berkeley-db-into-your-android-apps
SQLite is the only one I ever actually see being used. I would suggest using it because it is lightweight and free to use. When developing your App, lightweight is key, memory is limited on these devices! Is there any specific reason why you wouldn't want to use SQLite? Any feature that you're looking for that you don't see in SQLite?

Higher level database layer for Android?

Are there any good database abstraction layers/object relational mappers/ActiveRecord implementations/whatever they are called for Android? I'm aware that db4o is officially supported, but it has quite a large footprint and I'd rather use a more conventional database (SQLite).
I am the main author of ORMLite which was designed to be small[ish] but still provide higher level functionality. ORMLite makes calls to the native Android OS database APIs to support its ORM functionality. See the following for general information
http://ormlite.com/sqlite_java_android_orm.shtml
Here are some Android example applications:
http://ormlite.com/docs/android-examples
I tried the Sugar ORM, which is very basic (and easy to use) but it worked for my needs.
Sugar website
There is an 'android-active-record' project which provides ActiveRecord abstraction for accessing Android SQLite database.
It's available here: http://code.google.com/p/android-active-record
It allows to eliminate most of boilerplate coding when performing CRUD operations on database entities and also minimizes efforts for creating/maintaining a database structure
Try ActiveAndroid. It is free and open source (Apache Version 2.0).
From the website:
ActiveAndroid is an active record style ORM (object relational
mapper). [...] ActiveAndroid allows you
to save and retrieve SQLite database records without ever writing a
single SQL statement. Each database record is wrapped neatly into a
class with methods like save() and delete().
[...] Accessing the database is a hassle, to say the least, in Android.
ActiveAndroid takes care of all the setup and messy stuff, and all
with just a few simple steps of configuration.
If performance and size matter, you should have a look at our open source ORM tool greenDAO. We wrote it because we did not want to compromise on speed. Other tools heavily rely on reflection, which is very slow on Android. Despite the tiny size (<100k), it supports relations, query builders, etc.
Shameless plug, but I've been working on a new open source Android framework called Infinitum. One of its main features is an ORM which has a criteria API similar to Hibernate and a few other nifty features (associations, lazy loading, etc.). It's still in its early stages, but I think it's coming along pretty nicely.
I have written a new ORM, for android, that's aimed and being as easy as possible to implement. It support lists and SQL free migration a couple things which I always found had an overhead in other libraries.
http://www.rushorm.com/
I faced the same problem and looked at both android-active-record and ActiveAndroid. I found android-active-record didn't handle the things I cared about (relationships for example), and ActiveAndroid isn't free. Therefore, I decided to write my own library. It's called AndroidRecord and it's hosted on GitHub and you're free to do with it what you want (I think I'm going to go with the MIT license). I use this every day and I'm content with it, but I'd love to get feedback.
If you need to know how to use it, I'm working on the documentation. If you need it right away, you can check out this lame example project which should be enough to dip your toes in. You can also email me of course.
There's also Neodatis and Perst (Lite).
I've toyed with Perst a year ago and concluded it's not worth it.
After all, a) Android runs on a rather restricted device with ~16mb of heap space per app and b) You customers would really appreciate performance and low power consumption.
So my advice is to go with SQLite and hand-written SQL. It's not hard at all and the wrappers provided by Android SDK are really nice.
EDIT: In 2012 the advice would be to use the ORM component of DroidParts (which is my project).
I was comparing basics of ormlite and greendao some time ago. You might want to take a look there. I plan to write some follow up with more advanced stuff in the near future but for now it's only a basic stuff. In my own project I'm using GreenDAO.
Have a look at Androrm. It is open source and well documented (see here). If you ever worked with django, you will notice, that the syntax is very similar.
Androrm also supports abstraction classes for the most common field types, plus relational fields. This way it enables you to query for your data in an very easy manner with only very little effort on your side.
SQLite is explicitly part of Android:
http://developer.android.com/reference/android/database/sqlite/SQLiteDatabase.html
However you might have to create your own abstraction layer (query builder for simple queries), or otherwise deal with SQL.
Maybe http://developer.android.com/reference/android/database/sqlite/SQLiteQueryBuilder.html is what you need?

Categories

Resources