I am using SQLite in my RN apps with help of react-native-sqlite-storage. Then, one of my table has a column which contain a very long string. When I try to insert the string, no error occured. But, when I select the row, it return null. This open issue is similar with my problem.
Then, I found this documentation. It said that
During part of SQLite's INSERT and SELECT processing, the complete
content of each row in the database is encoded as a single BLOB. So
the SQLITE_MAX_LENGTH parameter also determines the maximum number of
bytes in a row.
The maximum string or BLOB length can be lowered at run-time using the
sqlite3_limit(db,SQLITE_LIMIT_LENGTH,size) interface.
My question is, if I have this piece of working code to select data from a table:
const db = await SQLite.openDatabase('databasename', 'default');
db.executeSql('SELECT * FROM tablename', [], (res) => {
console.log(res.rows.item(0));
}, err => console.log(err));
How to implement sqlite3_limit(db, SQLITE_LIMIT_LENGTH, size) to above code in order to increase the SQLITE_LIMIT_LENGTH?
You can't easily. Android's default sqlite implementation doesn't provide access to it. You would need to import your own version of the sqlite Java library (which is really a C library accessed via JNI), then provide your own RN module to access it in Java, then use only that module to access the db at all. Its a lot of work. How large is this string? You may want to consider storing it in a file and putting the filename in the db.
The issue you linked particularly mentioned images. Sqlite is not really a good solution for storing images, for a variety of reasons. Use a file and store the filename for that.
Related
I am trying to display images from Firebase's storage in my Android app inside a GridView.
I have done that, however, the images are displaying in an unknown order, and I have noticed that there is a 'Last modified' column inside Firebase's storage when I upload the images to it.
My question is: is there a way that I could sort the images inside the GridView in order to display them according to that date? (for example, the last one added, would have the latest 'Last modified' date and would be viewed first and such...)
This is my code:
val listRef : StorageReference = FirebaseStorage.getInstance().reference.child("images/posts/$userName")
val fileNameList: ArrayList<String> = ArrayList<String>()
listRef.listAll()
.addOnSuccessListener { it ->
it.items.forEach{
fileNameList.add(it.name)
}
gridView?.adapter = ImageRecyclerAdapter(activity, fileNameList,userName)
}
Note: the code is in Kotlin
I have looked everywhere and couldn't find anything that helps.
Any help is appreciated, thank you :)
According to the official documentation regarding StorageReference's listAll() method:
List all items (files) and prefixes (folders) under this StorageReference.
You might not be interested in listing folders within the reference you are pointing to.
That been said, the best option that you have is to store the URLs in a database. Such a database can be either Cloud Firestore or Firebase Realtime Database. This means that each object should have at least two fields, one for the actual URL and one for a timestamp. Please see in my answer from the following post how you can add a timestamp to Firestore:
ServerTimestamp is always null on Firebase Firestore
Is in Java, but you can simply convert it to Kotlin. Once you have all URLs in place, you can create a query and order the URLs according to the date. In Firebase Realtime Database the default order is ASCENDING, but below is how you can reverse the order:
How to arrange firebase database data in ascending or descending order?
While in Firestore, you can simply pass the desired direction to Query's orderBy(String field, Query.Direction direction) method.
I faced with the following situation. In my program I have to keep files in database. This database contains title of the article which keeps file and path to the file. All files are kept in Assets folder and are created manually. But what if I want to add files from the program itself. For example to create a special edittexts where user can write title and articles. How can I keep this data? I understand how to add title,entered by user,to database,it's easy. But what about the articles. I can't place them with file which were created manually,as Assets can't keep such files. I thought to add all full articles to database,but how can I add asset's files in such case?
Files (images, PDF'd. Word documents .........) are not structured data and serve little purpose being stored in the database (Full Text Search (FTS) an extension for SQLite is a different matter).
The recommended technique is to store some reference to the file (e.g. full path or perhaps file name (if all files are stored at a location that is discernible by the file name)) in the database. So when you locate/search and obtain a row you then use the file itself.
However, if the files average around about 100k or less then SQLite can actual be faster and thus it may performance wise, be better to store the files in the database.
Note the 100k based upon the link below.
35% Faster Than The Filesystem
You would store files as BLOB's (byte arrays). You read them into a byte array and on Android, assuming java and that the byte array is named my_byte_array and that the SQLiteDatabase is db then :-
ContentValues cv = new Contentvalues();
cv.put("FILE_COLUMN",my_byte_array);
........ other cv.put's for the other columns
long inserted_id = db.insert("The_Table",null,cv);
if (inserted_id < 1) {
.... code to handle row not inserted.
} else {
.... code to handle row inserted OK
}
WARNING
Files greater than 2M will be impossible to easily retrieve as a Cursor Window is limited to 2M of memory. Even 1M may well cause issues and may be unusable at some stage if the App needs to be backwardly compatible as I believe that Cursor window's were restricted to 1M (not sure when).
Retrieval from the database
To retrieve data from a BLOB you use the Cursor getBlob(column_offset) method, or generally better use the Cursor getColumnIndex(column_name) method to retrieve the column offset according to the column name. So if the Cursor is named csr then** my_other_byte_array = csr.getBlob(csr.getColumnIndex(column_name));**
Noting that you have to move to a valid row before using the getBlob method.
Is there any way to select specific properties from firebase Realtime Database? I know there is a way to retrieve selected properties from firestore but how can get via Realtime database using Node.js
I want only Notes from everyone nothing else.
Suppose i just want to select Notes from Allergy here is my sample code which i tried but not successes...
admin.database().ref(`vitals/Allergy`).select('Notes').then(result => {//here is my result.....})
But it shows me that select is not a function.......
Realtime Database doesn't support "projections" like this (neither does Cloud Firestore). If you are going to query across multiple child nodes, you are going to get each entire child node that matches the query. Even if you want just one property of each child, you can't avoid the cost of downloading the entire child.
If your app is very sensitive to performance on these types of queries, consider duplicating the data such that there is another branch of your database that contains only the "Notes" property, and query that branch alone. This duplication is common in NoSQL type databases, and is call the "fan out" technique.
If you want to save download band then i can't help you further, otherelse:
let ArrayOfAllDownloadedNotes = [];
firebase.database().ref('yourRootFolders/vitals/Allergy').once((snapshot)=>{
ArrayOfAllDownloadedNotes = [...ArrayOfAllDownloadedNotes, snapshot.val().Notes];
});
//Reapeat the firebase formula for every child you want retrieve the Notes from
//You may also use forEach((item)=>{}) function for each folder you want to retrieve the //notes from if you want
console.log(My Array of Notes:',ArrayOfAllDownloadedNotes );
I maintain an application that is collecting a lot of information and is storing these information in an ArrayList.
In detail this ArrayList is defined as ArrayList<FileInformation> which has some member like:
private File mFile;
private Long mSize;
private int mCount;
private Long mFilteredSize;
private int mFilteredCount;
private int mNumberOfFilters;
etc.
This approach is working but is not very flexible when I would like to introduce some new functionality. It also has some limitations in terms of memory usage and scale-ability. Because of this I did some tests if a database is the better approach. From the flexibility there is no question, but somehow I'm not able to make it running fast enough to become a real alternative.
Right now the database has just one table like this:
CREATE TABLE ExtContent (
"path" TEXT not null,
"folderpath" TEXT not null,
"filename" TEXT,
"extention" TEXT,
"size" NUMERIC,
"filedate" NUMERIC,
"isfolder" INTEGER not null,
"firstfound" NUMERIC not null,
"lastfound" NUMERIC not null,
"filtered" INTEGER not null
);
The performance issue is immense. Collecting and writing ~14000 items takes ~3mins! when writing into the database and just 4-5secs if written into the ArrayList.
Creating the database in-memory does not make a big difference.
As my experience in terms of SQLITE is rather limited, I started by creating the entries via the android.database.sqlite.SQLiteDatabase.insert methode.
As there was no meaningful difference between a file based and a in-memory database, I guess using BEGIN TRANSACTION and COMMIT TRANSACTION will not make any difference.
Is there some way to optimize this behavior?
Just for clarification, putting BEGIN TRANSACTION and END TRANSACTION will increase the performance greatly. Quoted from http://www.sqlite.org/faq.html#q19 :
SQLite will easily do 50,000 or more INSERT statements per second on an average desktop computer. But it will only do a few dozen transactions per second. By default, each INSERT statement is its own transaction...
I had a similar issue on an app I was coding on the weekend.
Is the data in the database to be included in the app when it's released? If so, bulk inserts aren't they way to go, instead you want to look at creating the database and including it in the assets directory and copying it over to the device. Here's a great link.
Otherwise I'm not sure you can do much to improve performance, this link explains methods on bulk inserting into an SqlLite Database.
Edit: You may also want to post your insert code too.
This is opretty obvious. Assuming you already allocated object to insert into. ( This is the same workload for bot solutions ) Let's compare alternatives:
Inserting in ArrayList does:
- (optional) allocate new chinks of cells for pointers if necessary
- insert object pointer into array list on the end
... really fast
INserting into sqlite:
-prepare insertion query ( I hope you use prepared query, and do not construct it from strings)
-perform database table insertion with modifications of indexes etc.
... a lot of work
Only advantage of database is that you can:
- query it later
- it handles external storage transparently allowing you to have much more entities
But it comes at cost of performance.
Depending on what you are for, there could be better alternatives.
For example, in my android games I store highscore entries in JSON file and utilise
GSON Pull parser / databinding layer ( https://github.com/ko5tik/jsonserializer ) to create objects out of it. Typical load time for 2000 entries from external storage is about 2-3 seconds
I want to bulk insert about 700 records into the Android database on my next upgrade. What's the most efficient way to do this? From various posts, I know that if I use Insert statements, I should wrap them in a transaction. There's also a post about using your own database, but I need this data to go into my app's standard Android database. Note that this would only be done once per device.
Some ideas:
Put a bunch of SQL statements in a file, read them in a line at a time, and exec the SQL.
Put the data in a CSV file, or JSON, or YAML, or XML, or whatever. Read a line at a time and do db.insert().
Figure out how to do an import and do a single import of the entire file.
Make a sqlite database containing all the records, copy that onto the Android device, and somehow merge the two databases.
[EDIT] Put all the SQL statements in a single file in res/values as one big string. Then read them a line at a time and exec the SQL.
What's the best way? Are there other ways to load data? Are 3 and 4 even possible?
Normally, each time db.insert() is used, SQLite creates a transaction (and resulting journal file in the filesystem), which slows things down.
If you use db.beginTransaction() and db.endTransaction() SQLite creates only a single journal file on the filesystem and then commits all the inserts at the same time, dramatically speeding things up.
Here is some pseudo code from: Batch insert to SQLite database on Android
try
{
db.beginTransaction();
for each record in the list
{
do_some_processing();
if (line represent a valid entry)
{
db.insert(SOME_TABLE, null, SOME_VALUE);
}
some_other_processing();
}
db.setTransactionSuccessful();
}
catch (SQLException e) {}
finally
{
db.endTransaction();
}
If you wish to abort a transaction due to an unexpected error or something, simply db.endTransaction() without first setting the transaction as successful (db.setTransactionSuccessful()).
Another useful method is to use db.inTransaction() (returns true or false) to determine if you are currently in the middle of a transaction.
Documentation here
I've found that for bulk insertions, the (apparently little-used) DatabaseUtils.InsertHelper class is several times faster than using SQLiteDatabase.insert.
Two other optimizations also helped with my app's performance, though they may not be appropriate in all cases:
Don't bind values that are empty or null.
If you can be certain that it's safe to do it, temporarily turning off the database's internal locking can also help performance.
I have a blog post with more details.
This example below will work perfectly
String sql = "INSERT INTO " + DatabaseHelper.TABLE_PRODUCT_LIST
+ " VALUES (?,?,?,?,?,?,?,?,?);";
SQLiteDatabase db = this.getWritableDatabase();
SQLiteStatement statement = db.compileStatement(sql);
db.beginTransaction();
for(int idx=0; idx < Produc_List.size(); idx++) {
statement.clearBindings();
statement.bindLong(1, Produc_List.get(idx).getProduct_id());
statement.bindLong(2, Produc_List.get(idx).getCategory_id());
statement.bindString(3, Produc_List.get(idx).getName());
// statement.bindString(4, Produc_List.get(idx).getBrand());
statement.bindString(5, Produc_List.get(idx).getPrice());
//statement.bindString(6, Produc_List.get(idx).getDiscPrice());
statement.bindString(7, Produc_List.get(idx).getImage());
statement.bindLong(8, Produc_List.get(idx).getLanguage_id());
statement.bindLong(9, Produc_List.get(idx).getPl_rank());
statement.execute();
}
db.setTransactionSuccessful();
db.endTransaction();
Well, my solution for this it kind of weird but works fine...
I compile a large sum of data and insert it in one go (bulk insert?)
I use the db.execSQL(Query) command and I build the "Query" with the following statement...
INSERT INTO yourtable SELECT * FROM (
SELECT 'data1','data2'.... UNION
SELECT 'data1','data2'.... UNION
SELECT 'data1','data2'.... UNION
.
.
.
SELECT 'data1','data2'....
)
The only problem is the building of the query which can be kind of messy.
I hope it helps
I don't believe there is any feasible way to accomplish #3 or #4 on your list.
Of the other solutions you list two that have the datafile contain direct SQL, and the other has the data in a non-SQL format.
All three would work just fine, but the latter suggestion of grabbing the data from a formatted file and building the SQL yourself seems the cleanest. If true batch update capability is added at a later date your datafile is still usable, or at least easily processable into a usable form. Also, creation of the datafile is more straightforward and less error prone. Finally, having the "raw" data would allow import into other data-store formats.
In any case, you should (as you mentioned) wrap the groups of inserts into transactions to avoid the per-row transaction journal creation.