Develop Reference

What is the best way to save a string-array in SQLite on Android?

One string size is about 200 bytes,
and it stores 10~20 size in a daily array.
(Store 10~20 strings of 200bytes, as array type)
I have found a way to convert an array to a string
and store it in SQLite.
However, I do not know it's a good idea
because the size of the string is large.
1.
If large arrays of strings,
is it a good idea to store arrays as a string?
2.
or is there a better way?
I would like advice. Thank you.

You're actually placing your concern onto the wrong part of your database design.
For SQLite, the maximum length of a String is 1 billion bytes, so your worries about your 10-20 strings of 200 bytes each actually isn't considered that large.
There's really no harm in storing your array as a single long String in your database. Especially when it's nowhere close to the maximum limit of a String.
Your database query time won't become longer due to your String being long. The real concern here is the processing you'll be doing on that String to turn it back into an Array. Typically, if the String is extremely long, the real performance hit is when you're flattening the array into a String and when you're transforming that String back into an Array.
However, typically, this is something you'll show a loading indicator for to your users.
For storing an Array into a database, there's really only two ways to do so:
Flatten array into a single String and store the String as TEXT
Create a table meant to store the individual elements of the string, and include a column for a Foreign Key that allows you to associate those rows with the same array. Then you'll store each element of your String arrays as a row in this table.
Depending on what you need, one design is better than the other.
For example, you would normally prefer the second implementation if your app requires you to constantly edit individual elements of an array.
For such an example, it wouldn't make much sense to use the first solution, because this means every time you want to edit the contents of an array, you'll be fetching back the complete array in it's entirety. This is impractical when you only want to fetch or edit a particular portion of that String.
Therefore, in such an example, it is much more practical to store the individual elements of the arrays into individual rows of a Table meant for this type of data. You'll be querying only the row you want and updating only the row you want.
So to answer your questions, there's really only two ways to store your String array as a TEXT type in your SQLite database. Both ways work and the real concern is to consider which solution fits your needs best.
If your app only requires you to store and fetch the array in it's entirety each time, then the single String method might be more preferable.
But if your app requires you to work with individual elements of your array, then using the table method would be more convenient.

storing large data in android

I am pulling a large amount of json from a restful server. I use the GSON library from google to traverse this json and it works great. Now I want to save all of the json objects in my sqlite db, however I want to make use of a transaction to add all of them at once. This is difficult if you dont have all the objects ready in one datastructe. Since I am traversing the json one object at a time, I guess I would have to store that in a data structure such as an arraylist or hashmap and then afterwards use a database transaction to do the inserts fast. However... Storing a large amount of data aka 200 000 json objects into a structure in memory can take up a lot of memory and wil probably run out as well. What would be the best way to get all of that json objects into my sqlite db and at the same time not use up a lot of menory in otherwords storing and inserting in a way that allows for a lot of recycling.

If you want to add a large amount of data at an unique moment : it will take a lot of memory anyway. 200 000 large JSON objects take a certain amount of memory and you will not be able to change it.
You can keep this behavior, but I think it's not a great solution because you create a huge memory consumption on both Android device and server. It will be better if you receive data part by part and adding them this way : but you need to have control on the server code.
If you are absolutely forced to keep this behavior, maybe you should receive all the data at the same time, parse them on a huge JSON object, then make multiple transactions. Check if every transaction was executed correctly and put back your database in a good state if not. It's a really bad way to do it, IMHO... but I don't know all your constraints.
To finish : avoid receiving a large amount of data at only one time. It will be better to make multiple requests to get partial data set. It will make your app less network dependant : if you loose the network for 2 seconds, maybe only one request will fail. So you will have to retry only one request and received again a small part of data. With only one huge request : if you loose the network, you will have to retry the entire request...

I know this is not the best implementation of handling large json input in Android, but it certainly works great.
So my solution is pretty simple:
While parsing the JSON code, make use of prepared statements and a db transaction to execute the inserts.
So in short, as soon as a JSON object is parsed, take that info, insert it into the db using the prepared statement and the db transaction, then move on to the next object.
I have just pulled 210 000 rows with 23 fields each (thats 4.6 million values) from a remote server via JSON and GSON and at the same time inserting all those values into my SQLite3 db on the device, in less than 5 minutes and without wasting any memory. Each iteration of parsing/inserting uses the same amount of memory and is cleaned on the next iteration.
So yeah, there's the solution. Obviously, this is not the best solution for commercial applications or tables with 1000 000 + records, but it works great for my situation.

Have you ever tried to add a lot of data (and I really mean a lot, in my case 2600 rows of data) into the Android-internal database (SQLite)?
If so, you propably went the same road as I did.
I tried a normal InsertStatement which was way to slow (10 sec. in my emulator). Then I tried PreparedStatements. The time was better but still unacceptable. (6 sec.). After some frustrating hours of writing code and then throwing it away, I finally found a good solution.
The Android-OS provide the InsertHelper as a fast way to do bulk inserts.
To give you an overview of the performance (measured with an emulator on a crap computer, trying to insert 2600 rows of data):
Insert-Statement
10 seconds
Prepared-Statements
6 seconds
InsertHelper
320 ms
You can speed up the insertion even more with temporarily disable thread locks. This will gain about 30 % more performance. However it’s important to be sure that only one thread per time is using the database while inserting data due to it’s not threadsafe anymore.
public void fillDatabase(HashMap<String, int[]> localData){
//The InsertHelper needs to have the db instance + the name of the table where you want to add the data
InsertHelper ih = new InsertHelper(this.db, TABLE_NAME);
Iterator<Entry<String, int[]>> it = localData.entrySet().iterator();
final int firstExampleColumn = ih.getColumnIndex("firstExampleColumn");
final int secondExampleColumn = ih.getColumnIndex("secondExampleColumn");
final int thirdExampleColumn = ih.getColumnIndex("thirdExampleColumn");
try{
this.db.setLockingEnabled(false);
while(it.hasNext()){
Entry<String, int[]> entry = it.next();
int[] values = entry.getValue();
ih.prepareForInsert();
ih.bind(firstExampleColumn, entry.getKey());
ih.bind(secondExampleColumn, values[0]);
ih.bind(thirdExampleColumn, values[1]);
ih.execute();
}
}catch(Exception e){e.printStackTrace();}
finally{
if(ih!=null)
ih.close();
this.db.setLockingEnabled(true);
}
}

How many strings is too many for an android app

I am creating a language learning app and I have a section in the app that will be dependent on fairly large amounts of text. about 800 words per activity, spread out over different text views. I am planning to store the paragraphs / sentences in the strings resource. which will more o less equate to 100 strings per activty and about 10 activties.
In total the R.jar file will have between 1000 -1500 seperate string id's.Is this a lot or too much for an app. Will this make my app really slow or is it normal practice for a text based app?
I know that I could store the data in sqlite database but prepoulating the database then filtering results and placing each string into its corresponding view programatically would be a real pain in the neck and I think would be even slower. and I would prefer to declare the string in the xml layout straight from the resource.

Android creates static references for all strings you place in Resources and application will load all instances in memory when app is launched.
Its unnecessary object creation.
You should go for creating a db for all words and use when they are required.

GreenDao - determine database size on runtime

I would like to know please how to get my database(which is off course *.sqlite file) size in bytes?
My current way to do it(which isn't working) is:
new File(DataManager.getInstance().db.getPath()).length()
but I'm just getting here the same number every time 53,676~ , which is irrelevant to the database's content, I'm getting this number even when it's empty.
Thank you.

OK the solution is pretty simple, my recent way to check the database file was good.
But I didn't take in account that greenDao adds to the database another 53 KB. So an empty DB size would be 53± KB and after some insertions it would get bigger and bigger.

Can large String Arrays freeze my program?

I recently created a program that gets medi-large amounts of xml data and converts it into arrays of Strings, then displays the data.
The program works great, but it freezes when it is making the arrays (for around 16 seconds depending on the size).
Is there any way I can optimize my program (Alternatives to string arrays etc.)

3 optimizations that should help:
Threading
If the program freezes it most likely means that you're not using a separate thread to process the large XML file. This means that your app has to wait until this task finishes to respond again.
Instead, create a new thread to process the XML and notify the main thread via a Handler when it's done, or use AsyncTask. This is explained in more detail here.
Data storage
Additionally, a local SQLite database might be more appropriate to store large amounts of data, specially if you don't have to show it all at once. This can be achieved with cursors that are provided by the platform.
Configuration changes
Finally, make sure that your data doesn't have to be reconstructed when a configuration change occurs (such as an orientation change). A persistent SQLite database can help with that, and also these methods.

You can use SAX to process the stream of XML, rather than trying to parse the whole file and generating a DOM in memory.

If you find that you really are using too much memory, and you have a reason to keep the string in memory rather than caching them on disk, there are certainly ways you can reduce the memory requirements. It's a sad fact that Java strings use a lot of space. They require two objects (the string itself and an underlying char array) and use two bytes per char. If your data is mostly 7-bit ASCII, you may be better of leaving it as a UTF-8 encoded byte stream, using 1 byte per character in the typical case.
A very effective scheme is to maintain an array of 32k byte buffers, and append the UTF-8 representation of each new string onto the first empty space in one of those arrays. Your reference to the string becomes a simple integer: PTR = (buffer index * 32k) + (buffer offset). "PTR/32k" yields the index of the desired byte buffer, and "PTR % 32k" yields the location within the buffer. Use either an initial length byte or a null terminator to keep track of how long the string is. When you need to access one of the strings, don't allocate a new String object: unpack it into a mutable StringBuilder or work directly with the UTF-8 byte representation.
The above approach is obviously a lot more work, but can save you between a factor of 2 and 6 in memory usage (depending on the length of your strings). However, you should beware of premature optimization. If your problem is with the processing time to parse your input, or is somewhere else in your program, you could find that you've done a lot of work to fix something that isn't your bottleneck and thus get no improvement at all.