My company is currently transitioning to a new architecture, as well as a model that provides services to mobile devices. Our application was traditionally web based (HTML5/CSS3/JS/PHP/MYSQL), so we want to be able to port it to mobile platforms without having to reinvent the wheel (Cordova), as well as to desktops in a standalone fashion (AppJS) so that we don't have to worry about browser-dependent bugs. We are also transitioning away from PHP to NodeJS to make this more feasible.
The problem is that our users NEED the ability to use our application offline, with no real limit on how long or how much data they can store before "syncing" it to our servers.
With AppJS, this isn't an issue because they can store as much data as needed within a sqlite database. However, I have discovered the issue regarding a 5MB quota for WebSQL data and cordova/phonegap. This presents obvious technical challenges regarding our business needs.
I am aware that there are a couple of plugins that enable you to use sqlite data. The original sqlite plugin (https://github.com/davibe/Phonegap-SQLitePlugin) lacks documentation for android, and the other (https://github.com/chbrody/Cordova-SQLitePlugin) requires me to specify a data limit which obviously isn't possible. I am therefore left with very few options, one being to split the databases into multiple 5MB segments which are controlled by a JS wrapper.
Can you feasibly split larger datastructures into multiple 5MB websql databases with no issues across iOS and Android?
Is there a limit on the total number of websql databases on iOS and android?
Are there any other options that you would recommend we look at? We need to be able to use our existing HTML5 and CSS for the frontend!
EDIT
Our original architecture never really worked with tablet devices. This is something we are looking to remedy with cordova, as well as building a more stable solution. Originally, our offline product ran on a Server2Go stack that had all sorts of platforming issues because of the various applications running on the clients machine
For the android platform, you could create a native app, and then use a webview to display the front end (if its really that important to keep it HTML), then use a javascript bridge to allow the content in the WebView to interact with your native app.
From what you have described about the application though, I think you are better off actually creating a native application. As you say in your comment, one of the reasons you are switching your platform is because of lack of control. Why then immediately re-open this can of worms by introducing all sorts of hybrid app framework into your stack.
Long story short, you will have full control (and quite likely, better product) if you just go with the native approach.
To get a truly cross-platform, cross-browser relational solution that will work offline and have the necessary storage capacity, I would suggest going with: SequelSphere
Since it is a 100% JavaScript solution, it should be usable in each of your configurations without requiring a different codebase for each. Furthermore, it uses IndexedDB (if available, and then LocalStorage) to store its data in relational tables. I believe most browsers do not have a size limit on the storage amount of IndexedDB. I think Firefox may have an initial 50MB limit per the following: Max size in IndexedDB.
Also, there is a new feature called "Change Trackers" that significantly help in the data synchronization process by tracking changes (inserts/updates/deletes) to tables, and then reporting them to you when you desire.
As an extra added bonus, unlike WebSQL, SequelSphere is VERY JSON friendly. Creating tables couldn't be easier:
db.catalog.createTable({
tableName: "EMPL",
columns: [ "EMPL_ID", "NAME", "AGE", "DEPT_ID" ],
primaryKey: [ "EMPL_ID" ],
data: [
[0,"Bob",32,0],
[1,"John",37,2],
[2,"Fred",28,1]
]
});
Querying the data is easy:
var res = db.query("SELECT name, age, dept_id FROM empl WHERE Dept_ID = 2");
Inserting / Updating / Deleteing data is easy:
db.insertRow("empl", [3, "Sue", 26, 2]);
db.updateRow("empl", [3, "Suzy", 26, 2]);
db.deleteRow("empl", [3, "Suzy", 26, 2]);
One thing to be aware of: Since this will be used in an offline application, make sure you set the "Persistence Scope" of either the entire Catalog, or each Table to be "SCOPE_LOCAL":
db.catalog.setPersistenceScope(db.SCOPE_LOCAL);
// -or-
db.catalog.getTable("empl").setPersistenceScope(db.SCOPE_LOCAL);
Documentation on the basics: SequelSphere Basics
Documentation on "Local Storage" support: Local Data Storage and Persistence
Here is the API: SequelSphere API
If you have any questions, just email the support for SequelSphere.
For complete transparency, I am part of SequelSphere, and it does seem to answer your question very well... ;)
Generally speaking, I recommend using lawnchair so that you can be implementation agnostic. http://brian.io/lawnchair/adapters/
Can you feasibly split larger datastructures into multiple 5MB websql
databases with no issues across iOS and Android?
Before I knew about lawnchair, I implemented this using WebSQL, which I kinda regret. WebSQL databases can grow beyond 5MBs. I'm not sure what the true max is, but I've set my Phonegap app to 100 * 1024 * 1024 and haven't had any issues on Honeycomb Android.
For iOS you could use SQLite database on PhoneGap / iOS - More than 5mb possible.
Related
I am building a social media application which requires local storage of table data entities. This data must also be connected to a server to retrieve and update information to and from users. Our team has built an iOS client using core data, though we are looking for storage options in android. Is using SQLite the way to go? Any thoughts would be greatly appreciated.
You should take a look at Realm, it has clients for Objective-C, Swift and Android.
Description from their GitHub repository:
Features
Mobile-first: Realm is the first database built from the ground up to run directly inside phones, tablets and wearables.
Simple: Data is directly exposed as objects and queryable by code, removing the need for ORM's riddled with performance & maintenance issues. Plus, we've worked hard to keep our API down to very few classes: most of our users pick it up intuitively, getting simple apps up & running in minutes.
Modern: Realm supports easy thread-safety, relationships & encryption.
Fast: Realm is faster than even raw SQLite on common operations, while maintaining an extremely rich feature set.
If you're familiar with RxJava, you will probably want to check SQLBrite, wich is Square's solution for this.
Yes, Sqlite is a default storage solution for android. Howevere there is a wrap around it called ContentProvider. ContentProvider can be used with Loaders and provide async data loading. ContentProvider may be used to modify contats and merge accounts, see this guide. However ContentProvider may seem tricky and if you prefer ORMs you can use ORMLite or GreenDAO which are using sqlite as well.
I am relatively new to app development and core data, so take it easy on me. I have been working on an app (currently for iOS, android in the future too) which stores the user's data locally using core data.
The data has relationships across entities. User can create, update, delete data.
Now I need to integrate some kind of syncing solution. My requirements are:
Data should be accessible offline (right now I am testing queue operations for that)
Data should sync to cloud storage when network is available (on iOS I have tested out reachability for this)
Cloud storage needs to be integrateble to both my current iOS and future android version.
Relationships (or some other way of linking parent-child, i talk about it later) need to be maintained.
Core data on ios should be used, not third party replacement.
I have messed around with many solutions so far:
I setup my own rethinkdb database on a server and used PHP and REST to get things going. This was very messy.
I tested parse.com and afnetworking http requests (instead of parse's library) - this was better, but i couldn't seem to be able to manage the relationships from core data. And my syncing algorithm is complicated (it works but I am not sure if there are holes in it when it may fail)
I tested dropbox datastore api. I have only tested the iOS sample app they provide, seemed pretty good (still need to understand the workings).
My questions are:
If I were to use dropbox datastore api, how does it work with android? of course core data is not available there, so how does that work (sorry I don't have android dev experience yet)? ALso how does it handle relationships between entities?
If I go with Parse.com, do you think my this idea will work-
Instead of using relationships, I can use identifiers? my relationships are all 1-to-many, so on the parent (1) I can have an id A. On all the children I can point their parentid to A. Also since my children can have grandchildren too, on each of the children I can have another id which the grandchildren can point to. So on... does this make sense as a replacement of relationships? If yes, then what's the point of relationships in xcode?? other than having automatic cascade option maybe.
Are there any better solutions available for syncing cross platform?
I know my question may seem a bit asking for opinion, but I would like to see what everyone else has already tried. Past week of switching from one solution to another and designing the syncing algorithm has fried my brain.
(I work at Dropbox and will address just that part of the question.)
The Dropbox Datastore API doesn't use core data and thus works exactly the same way on Android. Why is core data a requirement?
As to relationships, what you describe for Parse is exactly what I would suggest for use with the Datastore API. I believe you're right that the reason for modeling relationships in databases and in code is to get automatic cascading operations and enforcement of relationship invariants.
I am building a PhoneGap application for Android and iOS.
I want to know which one is a better selection for database management: HTML5's SQLite or Native database of device.
In case of native use, i need to build a plugin for interaction.
Please suggest me a better option according to performance and speed.
i have tested with a POC that Native DB of Android/iOS is too much faster then WebSQL (of HTML5), in case of a PhoneGap App we should use a plugin for Native database.
in my test; i have run 999 insert query in a PhoneGap App with Plugin for native environment and the results are:
in Android queries are 558% faster in Emulator, 275% faster in htc droid hd and amazingly 2327% faster for htc wildfire.
in iOS the results are just hilarious queries are 2k+ percentage faster in Simulator/iPhone 4 and 5600% faster in iPhone 3GS.
The Web SQL Database standard is no longer being developed. It will probably be replaced by IndexedDB. I wouldn't rely on its existence if I were you.
From the spec:
Beware. This specification is no longer in active maintenance and the Web Applications Working Group does not intend to maintain it further.
I don't know about native database, but HTML5 WebSQL is very slow on iPhone when using tables with hundreds of rows, mainly 3G, 4G is little better. On Android there is no problem running pretty nice queries - we thought it would be the same on iPhone but we had to drop use of WebSQL in our project and rewrite everything to JSON and many more functions which had functionality of a few simple WebSQL queries.
It's a shame that support of WebSQL is officialy dropped, but currently, it's supported on both Android and iPhone and it's very handy for some cases. So I hope and I think WebSQL will be maintened "not-officially" but still in Android and iPhone in future.
As for your question - my experience tells me, that it depends on your tables structure, how complex queries you have and mainly - how many rows you have in tables. If you have tens of rows, its OK to use WebSQL, but if you hit 100 or more rows in a table, it will get slow on iPhone.
We are starting from scratch to develop a couple sophisticated mobile apps. We anticipate supporting iPad, iPhone, Android tablet and Android phones. However, our initial focus will be Android only.
The applications will be used to collect data and send that data to the backend. The whole scope of this project is data collection, data analysis, and data presentation. It's all about data.
What is the right backend technology?
EDIT: Sorry for not providing this info initially. The attraction of Python is the SciPy / NumPy libraries for data presentation and data analysis. I don't think any other platform has libraries that can match these. So Python/Django would seem ideal if there is no problem interfacing with mobile apps...
It will also be nice to use the same technology for the general website and I guess Python/Django is good in that regard too.
I don't see any difference between PHP, Python and Ruby in this case. As I understand, backend is going to be used as a database access layer and for nothing more complex. Am I right? If yes, go with technology you know better.
Why do you want to use Drupal for this task? From my point of view, it is better to use something more oriented on high performance and database interaction. Try Yii, Codeigniter or Fat Free PHP frameworks for example.
Django is a good choise as well because it has wonderful administration interface which is being generated by framework automatically. The only thing you should do is to set up models in your app and enable admin module.
Unfortunatelly, I have no RoR experience so I can say nothing about it except that a lot of people say that it is great.
Python/Django plus SciPy/NumPy proved to be the best option for our project.
The attraction of Python is the SciPy / NumPy libraries for data
presentation and data analysis. I don't think any other platform has
libraries that can match these.
Seriously: Java. You've got ease of use, statefulness and many very very good frameworks. And performance for large datasets is still very very good.
I'd especially recommend the Spring framework and hibernate.
I'm currently developing an app that has the potential to create a very large database. I had planned on installing the app on the SD card to allow for some extra room. Recent dealings here have warned me that that might not be a good idea. Are there any steps I can take to mitigate the danger of this course of action? Or are there any better alternatives?
This is a comprehensive post on the subject (I'm not the author).
I think, overall, it needs to be communicated that SQLite is just a SQL mechanism for accessing a file. It appears that the current market limit is 50mb for the entire APK. When installing to internal memory, you require 2x your APK size. Installing to sdcard requires just the stated APK size.
Here is what you will be working against:
1.) Since SQLite is just a abstraction over your file, when you do selects, inserts, updates, etc, you will be incurring sdcard read write costs
2.) I've seen mention of a soft limit of 10000 records based on performance. This article is a bit old, so its likely gotten better.
Other then that, you'll probably have to set up some tests to see what is feasible. Cursory search of google did not show any benchmarks to date.
As pointed out previously, SQLite has the functionality that you're looking for, packaged in a small library. It's designed to replace simple file access with reliable, recoverable, transactional data access via a SQL API. There's a great summary on their main page.
There are literally thousands of projects that are using SQLite. If your data set is going to be very large (more than 100-200MB), then you might want to consider using Berkeley DB as an option. Berkeley DB recently introduced support for a SQL API, which is completely SQLite compatible. In addition to the functionality that's provided by the SQLite SQL parser, query planner and executor, you also get the reliability and scalability that Berkeley DB is well known for. We have several customers who happily started out with SQLite. When they realized that they needed additional concurrency, scalability and reliability that not available in SQLite, they replaced the SQLite library with the BDB library, recompiled their application and had it tested and running on Berkeley within a few days.
I'm one of the Product Managers for Berkeley DB, so I'm a little biased. :-) However, we implemented the BDB SQL API so that we could offer users the best of both worlds: the ubiquity and ease-of-use of SQLite combined with the concurrency, reliability and scalability of Berkeley DB. Especially with larger data sets, Berkeley DB can make all the difference in application performance.
As mentioned above, you can use Berkeley DB (not the Java Edition) on Android. I described the steps of the cross-compiling process here since it requires minor tweaking.