I faced with the following situation. In my program I have to keep files in database. This database contains title of the article which keeps file and path to the file. All files are kept in Assets folder and are created manually. But what if I want to add files from the program itself. For example to create a special edittexts where user can write title and articles. How can I keep this data? I understand how to add title,entered by user,to database,it's easy. But what about the articles. I can't place them with file which were created manually,as Assets can't keep such files. I thought to add all full articles to database,but how can I add asset's files in such case?
Files (images, PDF'd. Word documents .........) are not structured data and serve little purpose being stored in the database (Full Text Search (FTS) an extension for SQLite is a different matter).
The recommended technique is to store some reference to the file (e.g. full path or perhaps file name (if all files are stored at a location that is discernible by the file name)) in the database. So when you locate/search and obtain a row you then use the file itself.
However, if the files average around about 100k or less then SQLite can actual be faster and thus it may performance wise, be better to store the files in the database.
Note the 100k based upon the link below.
35% Faster Than The Filesystem
You would store files as BLOB's (byte arrays). You read them into a byte array and on Android, assuming java and that the byte array is named my_byte_array and that the SQLiteDatabase is db then :-
ContentValues cv = new Contentvalues();
cv.put("FILE_COLUMN",my_byte_array);
........ other cv.put's for the other columns
long inserted_id = db.insert("The_Table",null,cv);
if (inserted_id < 1) {
.... code to handle row not inserted.
} else {
.... code to handle row inserted OK
}
WARNING
Files greater than 2M will be impossible to easily retrieve as a Cursor Window is limited to 2M of memory. Even 1M may well cause issues and may be unusable at some stage if the App needs to be backwardly compatible as I believe that Cursor window's were restricted to 1M (not sure when).
Retrieval from the database
To retrieve data from a BLOB you use the Cursor getBlob(column_offset) method, or generally better use the Cursor getColumnIndex(column_name) method to retrieve the column offset according to the column name. So if the Cursor is named csr then** my_other_byte_array = csr.getBlob(csr.getColumnIndex(column_name));**
Noting that you have to move to a valid row before using the getBlob method.
Related
I want to save image using my developed app into mobile what is the best way to it.
I was trying to save via sqlite but is there any other options?
The recommended way is to store the image as a file not in the database and then store the path or enough of the path to uniquely identify the image in the database.
To store an image you have to store it as a BLOB based upon a byte[] (byte array);
However, using the Android SDK, there is a limitation that only images less than 2MB can be retrieved even though they can be saved.
Technically you could store the image as a hex, binary, octal string but that would be more complex, less efficient and even more restrictive.. So really you have to store it as a BLOB is not completely factual.
It's also actually pretty useless storing images (or any large files) with SQLite or most structured databases as there is little that you can do with the data query wise.
You actually save a BLOB via SQL using something like :-
INSERT INTO your_table VALUES(x'00FF01EF');
i.e. 00 (0) is the first byte, FF (255) the 2nd, 01 (01) the 3rd .........
note the above would only work for a table with a single column
However, the SQLiteDatabase insert convenience method does the conversion from the byte[] to the correct SQL (hex string) on your behalf.
So you'd typically use :-
ContentValues cv = new ContentValues();
cv.put("your_column",your_image_as_a_byte_array);
your_sqlitedatabase_object.insert("your_table",null,cv);
You retrieve the byte array from a Cursor (result of a query) using something along the lines of
your_retrieved_image_byte_array = your_cursor.getBlob(your_column_offset_as_an_int); //<<<<<<<<< CursorWindow full error if the row cannot fit into the CursorWindow which has 2MB limit
You may wish to have a look at How can I insert image in a sqlite database, as this goes into more detail and has code that stores both images and paths (either one is stored based upon the image size, less then 100k, then the image is stored, else the path is stored) and additionally retrieves the images into a ListView. Additionally, although not recommended this is a way of getting around the 2Mb limit How to use images in Android SQLite that are larger than the limitations of a CursorWindow?
I have a question for you guys.
I have been working on a project application that in one part uses an SQLite database loaded from a txt file (it has about 100k-200k rows of 5 strings separated by the ^ sign).
Now my question is, since this is my first time working with databases, how does .txt import for modifiable databases work? If I understand right, it pulls all data from the txt file once and creates a database that it keeps to work on, so when I modify the database I modify the newly created one and not the txt? Does the code try to pull info again from the txt whenever the app loads, and would loading 200k 10char words every time be too much? :)
The database consists of music bands in this format: name/genre/popular[yes/no]/selected
The selected column is the only one being modified by the user (and the app for that matter). If I use the regular approach to databases with added implementation from a txt file will the selected column reset every time (do not want that)?
Don't distribute your app with a huge txt-file and import it on the users device. This takes time and is annoying.
Rather distribute your app with a pre-populated database and copy it over from the res-folder. You can use android-sqlite-asset-helper to automate this.
Also, yes. The Database is always stored on the internal memory and you can't access it on a non-rooted device (unless you're using the AVD).
To import your txt-contents into a database, create a script or something that parses the contents and executes the corresponding SQL-queries. Again, your App should ship with the database, not the raw-file!
I was a little bored and hacked together a short Python-Script to read all entries from your txt-file and insert them into a SQLite Database:
import sqlite3
import re
counter = 0;
pattern = re.compile('^([^\^]+)\^([\w\s]+)\^(yes|no)\^\w+$');
conn = sqlite3.connect("imported.db");
cursor = conn.cursor();
# Create the Table:
conn.execute('''
CREATE TABLE Bands (
name TEXT,
genre TEXT,
popular INTEGER,
selected INTEGER
);''');
# Now, insert:
with open('bands.txt', 'r') as f:
for line in f:
match = pattern.search(line);
if match:
cursor.execute('''
INSERT INTO Bands (name, genre, popular, selected)
VALUES (?,?,?,0)''',
(
match.group(1), match.group(2),
(1 if match.group(3) == 'yes' else 0)
)
);
counter+=1;
conn.commit();
conn.close();
print "Imported ", counter, " bands!";
This will assume that the txt-file is named bands.txt, each value is separated by a / and each entry will be on it's own line. The resulting database-file is imported.db.
Also, I use INTEGER for all True|False-fields (popular, selected). These will then hold a 0 for false and a 1 for true.
Last but not least, the RegEx only allows "yes" and "no" for the popular-value.
I stored chunks of binary data (protobufs) in the sqlite database of an Android app without realizing that Android's Cursor can only hold a maximum of 1MB of data. I now know that I should have stored these binary blobs in files and only referenced the files in the sqlite database entries.
I need to upgrade the database (the app has been in use for a while) in order to move these binary chunks to files. The problem is that some user's data may have already exceeded the 1MB limit and I'm not able to retrieve it from the database (accessing the resulting Cursor for a single row that contains a large blob throws an IllegalStateException: Couldn't read row 0, col 0 from CursorWindow. Make sure the Cursor is initialize before accessing data from it).
How do I retrieve the binary blobs that are greater than the 1MB Cursor limit that have been stored in the sqlite database?
You can read large blobs in pieces. First find out which ones need this treatment:
SELECT id, length(blobcolumn) FROM mytable WHERE length(blobcolumn) > 1000000
and then read chunks with substr:
SELECT substr(blobcolumn, 1, 1000000) FROM mytable WHERE id = 123
SELECT substr(blobcolumn, 1000001, 1000000) FROM mytable WHERE id = 123
...
You could also compile your own copy of SQLite and access either the BLOB stream I/O functions or the normal query functions of the C API with the NDK, but that would be too complex in this case.
CL. answer will only with blobs<5MB. If you tried to use it with blobs bigger than 5 megabytes you will still get the exception. To fetch large blobs you need to use a library called sqlite4java that uses native calls to the database without using cursors. Here is an example of using this library to fetch a large blob:
SQLiteConnection sqLiteConnection=null;
SQLiteStatement sqLiteStatement=null;
try
{
File databaseFile = context.getDatabasePath("database.db");
sqLiteConnection=new SQLiteConnection(databaseFile);
sqLiteConnection.open();
sqLiteStatement=sqLiteConnection.prepare("SELECT blob FROM table WHERE id=?");
sqLiteStatement.bind(1, id);
sqLiteStatement.step();
byte[] blob=sqLiteStatement.columnBlob(0);
}
finally
{
if(sqLiteStatement!=null)
sqLiteStatement.dispose();
if(sqLiteConnection!=null)
sqLiteConnection.dispose();
}
I am dealing with ECG signal processing using Android phone. I am getting the filtered ECG signal after the series of operations as my final output, and it is saved in text file in particular location. The text file contains the values of voltages in a single column, containing 30000 samples. This is done for number of patients.
Now I have to use these text files in my database with particular patients, as a entry in their row.
I am using SQLite database.
How can it be done?
please help me with this.
Read the file and save it as blob in sqlite, but if data is too large, you will have problems with size. Another way would b to keep the files in sdcard with some id as filename, you could save these ids in db which you can refer.
OK basically what you need to do is:
bulkinsert your data.
here is the link on how to do it:
http://notes.theorbis.net/2010/02/batch-insert-to-sqlite-on-android.html
if you need to populate your second table with this data then
update yourtable set blah=(select field from yourfreshlypopulatedtable)
I maintain an application that is collecting a lot of information and is storing these information in an ArrayList.
In detail this ArrayList is defined as ArrayList<FileInformation> which has some member like:
private File mFile;
private Long mSize;
private int mCount;
private Long mFilteredSize;
private int mFilteredCount;
private int mNumberOfFilters;
etc.
This approach is working but is not very flexible when I would like to introduce some new functionality. It also has some limitations in terms of memory usage and scale-ability. Because of this I did some tests if a database is the better approach. From the flexibility there is no question, but somehow I'm not able to make it running fast enough to become a real alternative.
Right now the database has just one table like this:
CREATE TABLE ExtContent (
"path" TEXT not null,
"folderpath" TEXT not null,
"filename" TEXT,
"extention" TEXT,
"size" NUMERIC,
"filedate" NUMERIC,
"isfolder" INTEGER not null,
"firstfound" NUMERIC not null,
"lastfound" NUMERIC not null,
"filtered" INTEGER not null
);
The performance issue is immense. Collecting and writing ~14000 items takes ~3mins! when writing into the database and just 4-5secs if written into the ArrayList.
Creating the database in-memory does not make a big difference.
As my experience in terms of SQLITE is rather limited, I started by creating the entries via the android.database.sqlite.SQLiteDatabase.insert methode.
As there was no meaningful difference between a file based and a in-memory database, I guess using BEGIN TRANSACTION and COMMIT TRANSACTION will not make any difference.
Is there some way to optimize this behavior?
Just for clarification, putting BEGIN TRANSACTION and END TRANSACTION will increase the performance greatly. Quoted from http://www.sqlite.org/faq.html#q19 :
SQLite will easily do 50,000 or more INSERT statements per second on an average desktop computer. But it will only do a few dozen transactions per second. By default, each INSERT statement is its own transaction...
I had a similar issue on an app I was coding on the weekend.
Is the data in the database to be included in the app when it's released? If so, bulk inserts aren't they way to go, instead you want to look at creating the database and including it in the assets directory and copying it over to the device. Here's a great link.
Otherwise I'm not sure you can do much to improve performance, this link explains methods on bulk inserting into an SqlLite Database.
Edit: You may also want to post your insert code too.
This is opretty obvious. Assuming you already allocated object to insert into. ( This is the same workload for bot solutions ) Let's compare alternatives:
Inserting in ArrayList does:
- (optional) allocate new chinks of cells for pointers if necessary
- insert object pointer into array list on the end
... really fast
INserting into sqlite:
-prepare insertion query ( I hope you use prepared query, and do not construct it from strings)
-perform database table insertion with modifications of indexes etc.
... a lot of work
Only advantage of database is that you can:
- query it later
- it handles external storage transparently allowing you to have much more entities
But it comes at cost of performance.
Depending on what you are for, there could be better alternatives.
For example, in my android games I store highscore entries in JSON file and utilise
GSON Pull parser / databinding layer ( https://github.com/ko5tik/jsonserializer ) to create objects out of it. Typical load time for 2000 entries from external storage is about 2-3 seconds