I currently have a .csv file with several unlabeled columns of data, which to my knowledge translate to the following datatypes in sqlite:
datetime (in the format 7/19/2011 12:00:00 PM) -> numeric
double -> real
char(1) -> text
float -> real
I can create the database by doing the following:
sqlite> create table myTable (myVar1 numeric, myVar2 real, myVar3 text, myVar4 real);
sqlite> .separator ","
sqlite> .import myFile.csv myTable
Then I copy and paste the newly created myTable.db into the "assets" folder in my project in eclipse. I make a DatabaseHelper class that extends SQLiteOpenHelper, and then I can start using and reading from the database in my Android project.
Am I getting this right? I've never used a database before and I've seen so many vastly different instructions on doing this. Some of my questions are-- do I have to label the columns of my .csv file? Is my .csv file not "simple" enough to just use .import and I'll need to find a program to translate it? I've come across sites saying that I need to rename something (which I don't seem to have) to "_id", and I don't know what this is, where this is, or how to do this, or if it's even necessary, or what it's for. What else am I missing?
I think you are getting it "right" except for that first datetime column. You should use the TEXT type, not a numeric type.
Also, you can inspect your data after the import to see if all is well, especially with that datetime field:
SELECT * FROM myTable ORDER BY RANDOM() LIMIT 10;
UPDATE
In response to the OP's last comment: my understanding of how you store date(time) is that it depends on your context. So if the date in the flat file is in the format "7/19/2011 12:00:00 PM", then without any transformation it'll be imported as TEXT anyway.
Importing csv into database is 15 lines of code task and it gives you more control over this process.
Table columns names like "var3" are just terrible, however there is no need to rename it in database you can just use sql aliases:
select myVar1 _id, myVar2 from myTable
_id is common name for primary key column of table (it's usually numeric column witch must be unique). Every ADK class using datastore assumes relays on it, so it's nice to use this convention.
If you just want read only database you can prepare db locally and find some tutorials how to include it into your app.
Related
Interesting issue while using SQLite in Android. I am seeing an inconsistency in the string length and quoting of a string between what is stored in the database and the materialized value seen in Java.
We are using an ORM called SugarORM to query the DB, but I've traced the offending code to the internal android.database.sqlite.SQLiteCursor class used within SugarORM, specifically the cursor.getString(columnIndex) method.
I have a string in the database that is an ISO data string 2019-03-25T19:19:39.664Z and is stored in a VARCHAR column . I have confirmed using DB Browser for SQLite that the length of the string as its stored in the database is indeed 24 characters. SELECT LENGTH(MyStringColumn) FROM MyTable WHERE ...
When I get the value of this string via cursor.getString(columnIndex), it is returning the string "2019-03-25T19:19:39.664Z". Notice the leading and trailing quotes. Java reports to me that the string is 26 characters long.
Any value that I store in this column that is not an ISO data does not have this behavior. I tried tracing the SQLiteCursor source back, but ultimately it ends up being a Native method and that's where my skill set stops.
Can anyone explain what might be going on here? I am probably just going to write a wrapper around my queries to get rid of the quotes, but its all very perplexing. The date string is being fed to a JavaScript interpreter and causing it to fail when creating a JavaScript Date object.
If it helps, I have replicated the behavior on both my S7 physical device and a Pixel 6 emulator.
As a quick get around you could use :-
SELECT length(replace(mystringcolumn,'"','')) FROM mytable;
or before using the original SELECT use :-
UPDATE mytable SET mystringcolumn = replace(mystringcolumn,'"','');
If this doesn't fix the issue, then for some reason it is the code that retrieves the data that is at fault.
e.g. consider :-
DROP TABLE IF EXISTS mytable;
CREATE TABLE IF NOT EXISTS mytable (mystringcolumn VARCHAR);
INSERT INTO mytable VALUES('2019-03-25T19:19:39.664Z'),('"2019-03-25T19:19:39.664Z"');
SELECT length(mystringcolumn), length(replace(mystringcolumn,'"','')) FROM mytable;
which results in :-
i.e. The 2nd row, 2nd column retrieves the appropriate value by using the replace function to strip of the quotes, if they exist.
As to why the quotes exist could depend upon either the way that the data is inserted (perhaps you have inadvertenly coded the quotes but the db being looked at isn't the actual database as copied from the App) or the way in which the data is being retrieved that for some reason adds them.
I don't believe it likely that the Cursor getString method has a bug in which the quotes are added, otherwise such an issue would likely be a recurring issue.
I am in a situation where the user has a sqlite database that has data that should not be tampered with at all. Essentially I want to import a table from a .csv file or something along those lines into their database without touching any of the data.
I notice that there is no library frmo what I can see that does explicitly this. My knowledge with SQLite isn't as comfortable as I'd like it to be so I'm unsure of where to go here.
Should I just read the file line per line copying the data and then inserting it into the created table? Each table will have 400 records, not too many so I figure it can't be that inefficient. My inexperience is what worries me thinking I will somehow damage the data. Hoping to prevent mistakes and liability here..
Here's one way:
Create a new temporary database table without a primary key (so you can verify it before copying it.)
e.g.
CREATE TABLE salespeopleTMP (
id INTEGER,
first_name TEXT NOT NULL,
last_name TEXT NOT NULL,
commission_rate REAL NOT NULL
);
and there is existing data in the table that looks like this:
sqlite> select * from salespeople;
1|Fred|Flinstone|10.0
2|Barney|Rubble|10.0
If I now have a CSV data file named people.txt that looks like this:
3,John,Doe,5.0
4,Jane,Smith,5.0
Import the CSV data into that temporary SQLite table
You can import the CSV data into my SQLite table with these two commands:
sqlite> .separator ','
sqlite> .import people.txt salespeopleTMP
Use the INSERT INTO command to import the data from your temporary
table into your actual table
insert into salespeople select * from salespeopleTMP
Delete your temporary table salespeopleTMP
based on and bug fixed from https://alvinalexander.com/android/sqlite-csv-import-data-table-primary-key
For my Android app, I want to save data using sqlite with this format:
name, date, attr1, attr2, attr3,...
These are the requirements:
each date can only contain each name once
there can be a variable number of attributes(numbers) for each name
each specific name has the same number of attributes
The app will be used to track events throughout the day. Events can have zero or more numeric properties.
The questions are: is sqlite the best way to store things here? If so how do I design my database? What other ways are there to store this kind of data?
is sqlite the best way to store things here?
This will depend on a number of other factors, such as how the data will be queried and used, the volume of transactions, data growth and retention, etc. From what you've described, though, SQLite is a great option, offering functionality out-of-the-box that supports some of your requirements directly, and is commonly used in such cases.
If you don't have much experience with relational databases, implementing this functionality may seem difficult at first, but like learning a new language or framework, it will get easier with time.
If so how do I design my database?
Let's step through each of your enumerated requirements...
each date can only contain each name once
SQLite supports the UNIQUE constraint. For example, if your columns were named name and date, you could add the following to your CREATE TABLE statement:
UNIQUE(name, date)
(A more complete CREATE TABLE statement is in the next example below, and it includes this constraint.)
This constraint prevents the insertion of rows with name/date pairs that already exist. Using android.database.sqlite.SQLiteDatabase, if you attempt to insert a row into the table with a duplicate name/date pair, a SQLiteConstraintException will be thrown at runtime. You will need to handle this exception in your java code.
there can be a variable number of attributes(numbers) for each name
This is a textbook case for normalizing the database, putting your data into multiple tables. For example:
CREATE TABLE names (
name_id INTEGER PRIMARY KEY AUTOINCREMENT NOT NULL,
name TEXT NOT NULL,
date DATETIME,
UNIQUE(name, date));
CREATE TABLE attrs (
name_id INTEGER NOT NULL,
attr_value INTEGER NOT NULL,
FOREIGN KEY(attr_value) REFERENCES names(name_id));
Your queries that retrieve attribute data would then JOIN the two tables. Since you indicated that "Events can have zero or more numeric properties", a LEFT OUTER JOIN might be most appropriate, as it will return names and dates even if there are no attributes.
Here's an example query, by name:
SELECT n.name, n.date, a.attr_value
FROM names AS n
LEFT OUTER JOIN attrs AS a
ON n.name_id = a.name_id
WHERE n.name = 'SMITH'
ORDER BY n.name, n.date, a.attr_value;
This query would return results like the following:
name date attr_value
--------------- ---------- ------------
SMITH 2015-02-13 1027
SMITH 2015-02-13 4426
SMITH 2015-02-13 8390
SMITH 2015-02-20 4426
SMITH 2015-02-20 8152
SMITH 2015-02-20 9328
You can then iterate through and process these results in java. If your results include multiple names and/or dates, then in your loop you should keep track of the last used name and date. If the name/date in the current record is the same, the attribute belongs to the current one. If the name/date is different, then this is a new one.
Note that this approach to your database design is flexible, allowing you to query on the attributes, for instance, to see what name/date pairs are associated.
Also note that there is a FOREIGN KEY constraint on the attrs table, meaning that if you attempt to insert a record into that table with a name_id that does not exist in the names table, a SQLiteConstraintException will be thrown at runtime. You will need to handle this exception in your java code.
each specific name has the same number of attributes
You will need to accommodate this requirement in your java code, probably doing some checks in the database prior to performing an INSERT.
What other ways are there to store this kind of data?
Flat files, JSON, XML, third-party data stores (with their own libraries), to name a few.
I'm not sure but I think the best way to achieve your requirement is to use sqlite and to solve your problem you can have 3 columns only. One for the name and one for the date and the other contains a JSON array that represents the rest of the attributes.
I currently want to dump a row from a SQLite database, transfer it over a network, and import it into the SQLite database on another phone. How would I properly dump the row and import it later? I looked around and I see a number of people mentioning .sql files, but is there an Android-specific way to do it? Thanks.
I assume that you have exact same table structure on both source and target sides (otherwise question would not make much sense), and you probably already know your table structure.
In that case, if you simply SELECT row you want to export (for example using Java code):
SELECT * FROM mytable WHERE id = 12345;
you would know row contents, so you can construct INSERT statement for target table, which should look like this:
INSERT INTO mytable ( col1, col2, ...)
VALUES ('val1', 'val2', ...);
All you need now is to execute this insert statement on target side (using Java or sqlite3 command line).
I have a .csv file that I turned into an SQLite database with the first column as a date and time (as a TEXT datatype because it's in the format 7/20/2011 12:00:00 PM), the second column as the subsecond (a REAL, like 0.142857), and the rest of the columns as data about that time. I will be using it as a read-only db in my Android app, in which the user selects a start time and an end time and then the data is displayed graphically for that time period.
This is my first time using a database and I'm not sure how to work with this. I believe I need a unique NUMERIC primary key for each row but simply labeling them by row number is a waste of space. The first column is not unique and is a TEXT datatype anyway, though if the subsecond was to be somehow incorporated then it would be unique. But I really can't re-write the database because it has 65534 rows... How do you suggest I access specific times in this database?
In Android you need a column named _ID in your database (else you'll face some issues later on). You will use that as the primary key.
Dates and times can be stored in SQLite in the form of a text column in the following format (See http://www.sqlite.org/datatype3.html):
TEXT as ISO8601 strings ("YYYY-MM-DD HH:MM:SS.SSS")
If your database is static, simply use a tool such as SQLite Database Browser to convert it to a format convenient for Android.
If your database is local and external(not remote), than you must have _id and another another table android_metadata which will hold the locale.
If your database was remote. Yes, you can it is only matter of speed if you are write, since you don't. Using WHERE clause will do the work.
Every date can be converted to numeric timestamp quite easy:
Date date = new Date();
long timestamp = date.getTime();
Date otherDate = new Date(timestamp);
Numbers are MUCH easier and faster to process than text fields. If you are completely sure, that you have unique data within column you can use it as primary key.
Importing csv file into table should be also easy:
//just open file in some known way and read it line by line
// we have declared String[] data, String line is line read from your csv somewhere earlier
// in code
data = line.split(",");
//here you have to process values, and execute insert statement
You have to create indexes on every column which will be used to search or order data. Please be also aware, that rows in table has no "default", "natural" or any other order. If you execute this same select statement twice you can get two totally different results in meaning of sorting. So simple select statement should look like that:
select
_id, other_colum_name, some_other_column
from
table_name
where
column_name = 5
and other_column_name = 3
order by
column_name;