I have a question for you guys.
I have been working on a project application that in one part uses an SQLite database loaded from a txt file (it has about 100k-200k rows of 5 strings separated by the ^ sign).
Now my question is, since this is my first time working with databases, how does .txt import for modifiable databases work? If I understand right, it pulls all data from the txt file once and creates a database that it keeps to work on, so when I modify the database I modify the newly created one and not the txt? Does the code try to pull info again from the txt whenever the app loads, and would loading 200k 10char words every time be too much? :)
The database consists of music bands in this format: name/genre/popular[yes/no]/selected
The selected column is the only one being modified by the user (and the app for that matter). If I use the regular approach to databases with added implementation from a txt file will the selected column reset every time (do not want that)?
Don't distribute your app with a huge txt-file and import it on the users device. This takes time and is annoying.
Rather distribute your app with a pre-populated database and copy it over from the res-folder. You can use android-sqlite-asset-helper to automate this.
Also, yes. The Database is always stored on the internal memory and you can't access it on a non-rooted device (unless you're using the AVD).
To import your txt-contents into a database, create a script or something that parses the contents and executes the corresponding SQL-queries. Again, your App should ship with the database, not the raw-file!
I was a little bored and hacked together a short Python-Script to read all entries from your txt-file and insert them into a SQLite Database:
import sqlite3
import re
counter = 0;
pattern = re.compile('^([^\^]+)\^([\w\s]+)\^(yes|no)\^\w+$');
conn = sqlite3.connect("imported.db");
cursor = conn.cursor();
# Create the Table:
conn.execute('''
CREATE TABLE Bands (
name TEXT,
genre TEXT,
popular INTEGER,
selected INTEGER
);''');
# Now, insert:
with open('bands.txt', 'r') as f:
for line in f:
match = pattern.search(line);
if match:
cursor.execute('''
INSERT INTO Bands (name, genre, popular, selected)
VALUES (?,?,?,0)''',
(
match.group(1), match.group(2),
(1 if match.group(3) == 'yes' else 0)
)
);
counter+=1;
conn.commit();
conn.close();
print "Imported ", counter, " bands!";
This will assume that the txt-file is named bands.txt, each value is separated by a / and each entry will be on it's own line. The resulting database-file is imported.db.
Also, I use INTEGER for all True|False-fields (popular, selected). These will then hold a 0 for false and a 1 for true.
Last but not least, the RegEx only allows "yes" and "no" for the popular-value.
Related
I faced with the following situation. In my program I have to keep files in database. This database contains title of the article which keeps file and path to the file. All files are kept in Assets folder and are created manually. But what if I want to add files from the program itself. For example to create a special edittexts where user can write title and articles. How can I keep this data? I understand how to add title,entered by user,to database,it's easy. But what about the articles. I can't place them with file which were created manually,as Assets can't keep such files. I thought to add all full articles to database,but how can I add asset's files in such case?
Files (images, PDF'd. Word documents .........) are not structured data and serve little purpose being stored in the database (Full Text Search (FTS) an extension for SQLite is a different matter).
The recommended technique is to store some reference to the file (e.g. full path or perhaps file name (if all files are stored at a location that is discernible by the file name)) in the database. So when you locate/search and obtain a row you then use the file itself.
However, if the files average around about 100k or less then SQLite can actual be faster and thus it may performance wise, be better to store the files in the database.
Note the 100k based upon the link below.
35% Faster Than The Filesystem
You would store files as BLOB's (byte arrays). You read them into a byte array and on Android, assuming java and that the byte array is named my_byte_array and that the SQLiteDatabase is db then :-
ContentValues cv = new Contentvalues();
cv.put("FILE_COLUMN",my_byte_array);
........ other cv.put's for the other columns
long inserted_id = db.insert("The_Table",null,cv);
if (inserted_id < 1) {
.... code to handle row not inserted.
} else {
.... code to handle row inserted OK
}
WARNING
Files greater than 2M will be impossible to easily retrieve as a Cursor Window is limited to 2M of memory. Even 1M may well cause issues and may be unusable at some stage if the App needs to be backwardly compatible as I believe that Cursor window's were restricted to 1M (not sure when).
Retrieval from the database
To retrieve data from a BLOB you use the Cursor getBlob(column_offset) method, or generally better use the Cursor getColumnIndex(column_name) method to retrieve the column offset according to the column name. So if the Cursor is named csr then** my_other_byte_array = csr.getBlob(csr.getColumnIndex(column_name));**
Noting that you have to move to a valid row before using the getBlob method.
I am developing an android application which have sqlite db on my device and sql server db on the main machine, the android device contain very small part of main db, so i don't want to replicate whole db. and I have my own webserver that upload/download data between databases,
my question is how I know which part of main database is changed to only download changed entries?
One usefull technique is using rowversion data type. You can see it as the server change number. You can get the current change number by using the MIN_ACTIVE_ROWVERSION() function. Then query from changes occurred form last downloaded change number to current change number and finally store the current change number as the last downloaded change number in the local SQLite database.
Something like this:
DECLARE #CurrentServerChange binary(8)
SET #CurrentServerChange = MIN_ACTIVE_ROWVERSION()
SELECT * FROM Table1
WHERE
RecordVersion >= #LastDownloadedChangeNumber
AND RecordVersion < #CurrentServerChange
SELECT * FROM Table3
WHERE
RecordVersion >= #LastDownloadedChangeNumber
AND RecordVersion < #CurrentServerChange
SELECT #CurrentServerChange AS CurrentServerChange
I assume RecordVersion columns are of type rowversion
I have a .csv file that I turned into an SQLite database with the first column as a date and time (as a TEXT datatype because it's in the format 7/20/2011 12:00:00 PM), the second column as the subsecond (a REAL, like 0.142857), and the rest of the columns as data about that time. I will be using it as a read-only db in my Android app, in which the user selects a start time and an end time and then the data is displayed graphically for that time period.
This is my first time using a database and I'm not sure how to work with this. I believe I need a unique NUMERIC primary key for each row but simply labeling them by row number is a waste of space. The first column is not unique and is a TEXT datatype anyway, though if the subsecond was to be somehow incorporated then it would be unique. But I really can't re-write the database because it has 65534 rows... How do you suggest I access specific times in this database?
In Android you need a column named _ID in your database (else you'll face some issues later on). You will use that as the primary key.
Dates and times can be stored in SQLite in the form of a text column in the following format (See http://www.sqlite.org/datatype3.html):
TEXT as ISO8601 strings ("YYYY-MM-DD HH:MM:SS.SSS")
If your database is static, simply use a tool such as SQLite Database Browser to convert it to a format convenient for Android.
If your database is local and external(not remote), than you must have _id and another another table android_metadata which will hold the locale.
If your database was remote. Yes, you can it is only matter of speed if you are write, since you don't. Using WHERE clause will do the work.
Every date can be converted to numeric timestamp quite easy:
Date date = new Date();
long timestamp = date.getTime();
Date otherDate = new Date(timestamp);
Numbers are MUCH easier and faster to process than text fields. If you are completely sure, that you have unique data within column you can use it as primary key.
Importing csv file into table should be also easy:
//just open file in some known way and read it line by line
// we have declared String[] data, String line is line read from your csv somewhere earlier
// in code
data = line.split(",");
//here you have to process values, and execute insert statement
You have to create indexes on every column which will be used to search or order data. Please be also aware, that rows in table has no "default", "natural" or any other order. If you execute this same select statement twice you can get two totally different results in meaning of sorting. So simple select statement should look like that:
select
_id, other_colum_name, some_other_column
from
table_name
where
column_name = 5
and other_column_name = 3
order by
column_name;
I currently have a .csv file with several unlabeled columns of data, which to my knowledge translate to the following datatypes in sqlite:
datetime (in the format 7/19/2011 12:00:00 PM) -> numeric
double -> real
char(1) -> text
float -> real
I can create the database by doing the following:
sqlite> create table myTable (myVar1 numeric, myVar2 real, myVar3 text, myVar4 real);
sqlite> .separator ","
sqlite> .import myFile.csv myTable
Then I copy and paste the newly created myTable.db into the "assets" folder in my project in eclipse. I make a DatabaseHelper class that extends SQLiteOpenHelper, and then I can start using and reading from the database in my Android project.
Am I getting this right? I've never used a database before and I've seen so many vastly different instructions on doing this. Some of my questions are-- do I have to label the columns of my .csv file? Is my .csv file not "simple" enough to just use .import and I'll need to find a program to translate it? I've come across sites saying that I need to rename something (which I don't seem to have) to "_id", and I don't know what this is, where this is, or how to do this, or if it's even necessary, or what it's for. What else am I missing?
I think you are getting it "right" except for that first datetime column. You should use the TEXT type, not a numeric type.
Also, you can inspect your data after the import to see if all is well, especially with that datetime field:
SELECT * FROM myTable ORDER BY RANDOM() LIMIT 10;
UPDATE
In response to the OP's last comment: my understanding of how you store date(time) is that it depends on your context. So if the date in the flat file is in the format "7/19/2011 12:00:00 PM", then without any transformation it'll be imported as TEXT anyway.
Importing csv into database is 15 lines of code task and it gives you more control over this process.
Table columns names like "var3" are just terrible, however there is no need to rename it in database you can just use sql aliases:
select myVar1 _id, myVar2 from myTable
_id is common name for primary key column of table (it's usually numeric column witch must be unique). Every ADK class using datastore assumes relays on it, so it's nice to use this convention.
If you just want read only database you can prepare db locally and find some tutorials how to include it into your app.
I want to bulk insert about 700 records into the Android database on my next upgrade. What's the most efficient way to do this? From various posts, I know that if I use Insert statements, I should wrap them in a transaction. There's also a post about using your own database, but I need this data to go into my app's standard Android database. Note that this would only be done once per device.
Some ideas:
Put a bunch of SQL statements in a file, read them in a line at a time, and exec the SQL.
Put the data in a CSV file, or JSON, or YAML, or XML, or whatever. Read a line at a time and do db.insert().
Figure out how to do an import and do a single import of the entire file.
Make a sqlite database containing all the records, copy that onto the Android device, and somehow merge the two databases.
[EDIT] Put all the SQL statements in a single file in res/values as one big string. Then read them a line at a time and exec the SQL.
What's the best way? Are there other ways to load data? Are 3 and 4 even possible?
Normally, each time db.insert() is used, SQLite creates a transaction (and resulting journal file in the filesystem), which slows things down.
If you use db.beginTransaction() and db.endTransaction() SQLite creates only a single journal file on the filesystem and then commits all the inserts at the same time, dramatically speeding things up.
Here is some pseudo code from: Batch insert to SQLite database on Android
try
{
db.beginTransaction();
for each record in the list
{
do_some_processing();
if (line represent a valid entry)
{
db.insert(SOME_TABLE, null, SOME_VALUE);
}
some_other_processing();
}
db.setTransactionSuccessful();
}
catch (SQLException e) {}
finally
{
db.endTransaction();
}
If you wish to abort a transaction due to an unexpected error or something, simply db.endTransaction() without first setting the transaction as successful (db.setTransactionSuccessful()).
Another useful method is to use db.inTransaction() (returns true or false) to determine if you are currently in the middle of a transaction.
Documentation here
I've found that for bulk insertions, the (apparently little-used) DatabaseUtils.InsertHelper class is several times faster than using SQLiteDatabase.insert.
Two other optimizations also helped with my app's performance, though they may not be appropriate in all cases:
Don't bind values that are empty or null.
If you can be certain that it's safe to do it, temporarily turning off the database's internal locking can also help performance.
I have a blog post with more details.
This example below will work perfectly
String sql = "INSERT INTO " + DatabaseHelper.TABLE_PRODUCT_LIST
+ " VALUES (?,?,?,?,?,?,?,?,?);";
SQLiteDatabase db = this.getWritableDatabase();
SQLiteStatement statement = db.compileStatement(sql);
db.beginTransaction();
for(int idx=0; idx < Produc_List.size(); idx++) {
statement.clearBindings();
statement.bindLong(1, Produc_List.get(idx).getProduct_id());
statement.bindLong(2, Produc_List.get(idx).getCategory_id());
statement.bindString(3, Produc_List.get(idx).getName());
// statement.bindString(4, Produc_List.get(idx).getBrand());
statement.bindString(5, Produc_List.get(idx).getPrice());
//statement.bindString(6, Produc_List.get(idx).getDiscPrice());
statement.bindString(7, Produc_List.get(idx).getImage());
statement.bindLong(8, Produc_List.get(idx).getLanguage_id());
statement.bindLong(9, Produc_List.get(idx).getPl_rank());
statement.execute();
}
db.setTransactionSuccessful();
db.endTransaction();
Well, my solution for this it kind of weird but works fine...
I compile a large sum of data and insert it in one go (bulk insert?)
I use the db.execSQL(Query) command and I build the "Query" with the following statement...
INSERT INTO yourtable SELECT * FROM (
SELECT 'data1','data2'.... UNION
SELECT 'data1','data2'.... UNION
SELECT 'data1','data2'.... UNION
.
.
.
SELECT 'data1','data2'....
)
The only problem is the building of the query which can be kind of messy.
I hope it helps
I don't believe there is any feasible way to accomplish #3 or #4 on your list.
Of the other solutions you list two that have the datafile contain direct SQL, and the other has the data in a non-SQL format.
All three would work just fine, but the latter suggestion of grabbing the data from a formatted file and building the SQL yourself seems the cleanest. If true batch update capability is added at a later date your datafile is still usable, or at least easily processable into a usable form. Also, creation of the datafile is more straightforward and less error prone. Finally, having the "raw" data would allow import into other data-store formats.
In any case, you should (as you mentioned) wrap the groups of inserts into transactions to avoid the per-row transaction journal creation.