I have zero experience with databases, and am looking for some direction in getting started. I am making an Android app that needs to read in a large amount of data quickly... Eventually this will be a stock market app that accesses real-time stock information. But before I spend money trying to get access to such real-time data, I'm working with a Microsoft excel file that has times and information on stocks from the past.
I've read up on databases and apparently Androids come with SQLite but I can't figure out how to start using it, or how to import my excel file.
Please help get me started or point me in the right direction. I've found some tutorials but they were difficult for me to follow with my complete lack of experience, and not necessarily related to my project; I'd need really basic step-by-step instructions...
Here is one way of doing this, think only in terms of data you need, rather than trying to get all spreadsheet data into your SQLite DB.
So, think in terms of learning basic SQL and SQL queries and create the tables (on a piece of paper) you need, how you will manipulate the data in these tables (updates, inserts, deletes etc).
Now that you have the table schema and your queries, the task remains of importing this data into the tables. For this I haven't done this but I am sure you will find a lot of resources on the Internet.
Now, as you can see, the seemingly complicated problem is broken into tangible actionable steps.
Here is a reasonable tutorial on SQLite. Here is another for SQLite and Android. You can google for more help.
Also take a look here. The blog talks about importing from excel into SQLite DB.
Related
I'm really new to programming apps - so this question might sound a bit strange:
I'm trying to program an app in android studio, where people can upload different things (basically strings and links put together in some kind of "package") and other peoble can then decide what "packages" they want to add inside their app. However after downloading, this data should be stored on their device and not just in the memory of the phone so that they can use it after restarting the app (and also if theres no internet connection). Do you have any idea what would be the best way to store this data both on the phone and in a database and how to synchronize the data on the phone with the selected data from the database. I really dont want to know how to do this exactly but would rather like some basic ideas and maybe you could tell me what kinds of stuff i should learn in order to succeed and what kind of database would be best here (firebase, MYSQL,..)?
Thanks a lot,
Andi
First of all you should decide what DB you are going to use.
In my opinion all RDBMs are good, but using Sqlite in order to achieve best performance on android devices is a good idea.
In your case you need server-side DB and application too.
(Depend on the scenario and framework you use can be different (sql,mysql,PostgreSQL,oracle,...)).
About how to sync local database with server-side you can download new DB from server and replace it with previous one, if you need previous user data you can have 2 different table and update one by downloading it from server, and save id or any identical row from specific package that already saved by user.
These are some question has been already answered in Stackoverflow
java - How to update table in sqlite?
java - SQLite in Android How to update a specific row
Create SQLite database in android
If you are talking about local databases. Go for Realm or look up a good ORM on github (Object relational mapping, you dont have to write SQL queries explicitly) .
I would suggest Realm which is very fast and user friendly.
I need to sync my MySQL database to android phone. I am new to this. So I am referring this tutorial from Programmer Guru.
The Problem is in this tutorial they have used 3 classes, DBController, SampleBC.java, MyService.java. I have almost 10 tables to sync. So how can I change this to a better one?
Is there a way to sync the whole database?
I couldn't see correct tutorial for this.
This really depends on what your app does, and how it uses the remote data.
I have written apps which have an 'update' feature - which basically serialises the whole mysql database as json, and then decodes it in to the sqlite database on the device.
Other apps will just fetch a smaller amount of data as and when they need it.
There is no better way per se. Just different options for different situations. You need to look at how frequently your mysql data will change, and what will the experience be for your app users. Nobody likes slow apps which spend 50% of their time displaying a progress dialog while they download data.
I am new to Android Application Development and a new member at stackoverflow. I am currently trying to design a recipe application. I have decided upon the features of the app and the scope it will cover. The scope is very vast for me in terms of covering all the recipes from all over the world. I am to deal with a lot of data in this process.
I am currently trying to figure a good and efficient way of handling the data in my app. So far, as per what I have read in different forums, I believe that I have two options in terms of a database choice : 1) SQLite 2) Database on remote server (MySql/Postgre)
Following are some of the thoughts that have been going on in my mind when it comes to taking a decision between the two :
1) SQLite : This could be a good option but would be slow as it would need to access the file system. I could eliminate the slowness by performing DB data fetch tasks in the AsyncTask. But then there could be a limitation of the storage on different phones. Also I believe using SQLite would be easier as compared to using a remote DB.
2) Remote Database : The issue that I can see here is the slowness with multiple DB requests coming at the same time. Can I use threads here in some way to queue multiple requests and handle them one by one ? Is there an efficient way to do this.
Also I have one more question in terms of the formatting of my data once I pull it out from the above DB's. Is there a way I could preserve the formatting of my data ?
I would be more than thankful if someone could share their knowledgeable and expert comments on the above scenario. Also this is not a homework for me and I am not looking for any ready made code solutions. I am just looking for hints/suggestions that would help me clear my thoughts and help me take a decision. I have been looking for this for sometime now but was not able to find concrete information. I hope I will get some good advice here from the experienced people who might have encountered similar situation.
Thanks for reading this long post.
What about combining both approaches?
A local SQLite database that has the least recently used receipes so you don't need network all the time. Network is way slower than accessing the filesystem.
Some remote database accessed via some HTTP interface where you can read / write the whole database. And if you want users to be able to add receipes for other users to see you'll need an external database anyways.
SQLite : This could be a good option but would be slow as it would need to access the file system.
Accessing a local database is pretty fast, 5ms or so if it's just a simple read only query on a small database.
But then there could be a limitation of the storage on different phones
Depends on your definition of huge database. It is okay if it is only 2MB which would be enough to store lots of text-only receipes.
Also I believe using SQLite would be easier as compared to using a remote DB.
Yes, Android has a nice built-in SQLite API but no remote database API. And you don't need to setup a database server & interface.
The issue that I can see here is the slowness with multiple DB requests coming at the same time.
A decent database server can handle thousands of requests. Depends on your server hardware & software. https://dba.stackexchange.com/ should have more info on that. Required performance depends on how much users you have / expect.
I'd suggest a simple REST interface to your database since it's pretty lightweight but does not expose your database directly to the web. There are tons of tutorials and books about creating such interfaces to databases. There are even hosted database services like nextDb that do most of the work for you.
Is there a way I could preserve the formatting of my data ?
You could store HTML formatted data in your database and display it in a WebView or a TextView (via Html#fromHtml()) - both can display formatted text.
Databases don't care what type of text you store, for transfer over the internets you may need to encode the text so it does not interfere with the transport formatting (XML, JSON, ...).
A simple way is to integrate Parse into your app. They have a nice framework that easily integrates into iOS and Android. Their plan is freemium, so you'll be able to use up to 1 million API request for no charge, and then its 7 cents for every request after that.
You'll have 1gb to store all your data sets / images, etc.
I don't use parse for everything, but I HIGHLY recommended it for large data schemes because they do all the scaling for you. Check out the API, I think it would be worth your time.
I just started to work on a few of my own projects, and I'm using Parse again. I have to say it's improved a lot over the last 6-8 months. Especially with the Twitter and Facebook integration.
The key issue here is the size of the data - any significant database of recipes would be too large to store on the phone imho,thus you seem stuck with the remote database solution.
As opposed to trying access the remote database from android I suggest you use a a go between web application that will process requests from the application and return JSON objects that you need.
It totally depends on your software requirements. If you need to deal with a small amount of data then you may choose SQLite, but for a huge amount to data better use a remote DB.
SQLite: It works fine with little amount of data & I experienced it response time is good.
Remote DB: I think you may use small server side app to submit the data to your client app. It will solve/reduce your thread related issues/complexities.
I'm creating my first android app that will make use of SQlite. I have zero experience with databases, except for creating a mysql database to use with wordpress...
Edit: After doing some research about rest, I'm still confused about how rest, sqlite, and android dev fit together. My goal is to access a rest-based web service through a url and access certain datasets, then store them in my SQlite database. Then I want to access the contents of the database through my java program, and use them accordingly.
The datasets can be downloaded individually in CSV format, but because I will be using so many of them, I don't want to go through every line individually and store them in the database. I'm hoping there's a more efficient way to store these datasets in the database.
My main questions are:
How can I copy the XML contents of a webpage from a url into my sqlite database? Can I do this with my java program, through the sqlite database, or a java library?
Do I only need to copy the contents of the webpages from the url into the sqlite database one time? If so, what can I do if any information is changed in the datasets?
You first need a schema for your sqllite DB. That schema should map to the objects behind the web service. For e.g, you need a Person table in your DB if there is a Person entity on the web. It depends on what all you want to capture.
When you are done designing the schema, you should start writing the code that help you create & manage DB on android. This is done with the help of SQLiteOpenHelper class:
http://developer.android.com/reference/android/database/sqlite/SQLiteOpenHelper.html
If you need to keep the DB synce'd with the data on the cloud (web services), you should implement sync. Android provides a very efficient sync framework.
Also, do watch this video from Android engineers explaining the best practices: http://www.youtube.com/watch?v=xHXn3Kg2IQE
Note, to actually fetch the data from the web service you would use UrlConnection API:
http://developer.android.com/reference/java/net/URLConnection.html
This sample probably captures most of it.
http://developer.android.com/resources/samples/SampleSyncAdapter/index.html
In terms of reading CSV files, there are some good resources here:
Can you recommend a Java library for reading (and possibly writing) CSV files?
Once you have read each CSV line into an object, then you can turn around and persist it to the database. I'm the author of ORMLite so I'll talk about using it. I don't believe there is a hibernate port for Android.
There are a number of Android examples to help you to get up to speed with ORMLite. Also some good tutorials. If you want to write a number of rows at once then I'd recommend using the batch tasks ORMLite feature. For information, see the discussion about creating lists of objects on the mailing list.
I can answer your first question about " I'm not sure how to add them efficiently"?
yes, SQlite is very powerful and intelligent, you can add thousand of records in one transaction, just like traditional database, It significantly improve performance.
about second question, as my understanding, because CVS file is very simple, so you can download and analyze it by yourself.
This is my first time trying to work with a database, so bear with me.
I need to write a program that will use a database that I do not have access to yet. I know there is MySQL, SQLite, and a bunch of other SQL things, but I'm not sure what the difference between them is. Do I need to know what kind of database it is before I can use it (i.e., is a MySQL .db file different from a SQLite .db file?), or is the file itself going to be the same and the difference is how it is accessed?
EDIT: I am programming for an Android tablet, that probably matters. But I will not be creating the database, it will be given to me and I have to work with it.
You might want to run through a view tutorials creating a MySQL database, and a SQLite instance to see how they work and how you can easily interact with them.
The access to each involves adapters, notwithstanding creating the tables you'll need, the user access (username / password / role to access the table).
Here's a straight-forward sqlite tutorial:
http://zetcode.com/db/sqlite/
https://zetcode.com/php/mysqli/
http://zetcode.com/php/pdo/
Here's a staight-forward sql with C# tutorial:
https://zetcode.com/csharp/mysql/
I remember being overwhelmed with databases my first time. My advice is to start with something well-supported, with a nice community, and search results a-plenty of tutorials, where you can grasp the fundamentals.
Then, based on your application requirements, and platform constraints, you can determine where to pivot. But reviewing the differences between databases without actually having worked with them is like trying pieces of sushi without ever eating fish.
EDIT:
If you're building on Android, yep, learn SQLite.
I highly recommend this tutorial to help you see SQLite used in an Android project:
https://developer.android.com/training/data-storage/sqlite
For data abstraction / ORM, you should familiarize yourself with Content Providers and Cursors (http://developer.android.com/guide/topics/providers/content-providers.html)
Good Querying!
First off, you'll need to know what type of database it is before you can connect to it as you'll need to use the appropriate database driver for your platform. mySQL, Oracle, Postgres, SQL Server, etc will all use different drivers as the binary protocol used to talk to them (authenticate and transfer information back and forth) is different.
As far as your actual SQL code, it depends. For the most part, you can count on SELECT * FROM Foo working with any SQL compliant database on the planet. However, once you start doing anything more complicated or using non-standard keywords, you might run into trouble. For example, some databases support the USING keyword for JOINs and some don't.
One thing you might look into is an ORM. This will allow you to abstract the actual SQL dialect from your program and then (for the most part) not have to worry about the actual SQL. Some popular ORMs are ActiveRecord (for Ruby on Rails) and Entity Framework for .NET. Hope this helps!
UPDATE:
Since the Android tag was just added to the post, I did a quick search for SqlLite ORMs that support Android and came up with this. Maybe worth checking into, or perhaps someone can comment on whether it's any good :)
Database Abstraction Layer is the strategy of having a smart middleman so you can program without knowing exactly which database system you have underneath. Database Abstraction isn't as utopian in practice as it sounds, and most people end up programming directly for a particular database. You will likely find that the more experience you get with your database of choice, the more you wish you knew its particular nuances better. So the short answer is there is enough difference to matter!
If you use a software architectural pattern like Model-View-Controller, it can enable you to do a lot of your programming irrespective of the database you use. Taking Model-View-Controller as an example, you could program your Controllers and your Views without knowing what database you are using.
The limitation to what you can do before deciding on a database system is the inconvenient fact that good programming practice calls for fat Models and skinny Controllers. So a big part of your actual programming effort is held hostage by your choice of a database solution.
If I didn't know what database would be used, I would start programming my View and then my Controller. If PHP is your language, you may find a CodeIgniter tutorial enlightening.
An article about "Appropriate Uses For SQLite"
http://www.sqlite.org/whentouse.html
Summary：Checklist For Choosing The Right Database Engine
Is the data separated from the application by a network? → choose client/server
Many concurrent writers? → choose client/server
Big data? → choose client/server
Otherwise → choose SQLite!