I have an SQLite db, and it has audio files in it stored as blobs.
Is it possible in android (or anywhere) to stream media from a db?
I would recommend not storing the audio data in the database. The memory issues mentioned earlier can lead to huge amounts of GC thrashing which can make the system non-responsive for seconds or more at time.
The typical approach involves a handful of steps.
Store the audio in a file somewhere in your application's directory.
Create two columns in your database. One column (called anything you like) contains a "content://" URL that references the data. Seeing a "content://" URL is a trigger to the system to then look up the contents of the "_data" column in the same row. The contents of that column should be the full path to the file.
The system then transparently reads that file, and presents it to whichever code actually requested the content.
I've got some example code for doing this with images -- obviously, it's not quite the same, but I can walk through it here and you should get the gist.
The specific problem I was trying to solve was storing album artwork for a track that's stored off the device. I wanted to be able to show the album artwork in a list, and cache it locally on the device so that repeatedly scrolling through it is fast and does involve repeated network fetches for the same data.
I have an albums database, with various columns that get lazily populated from a remote server. I implement this database using the ContentProvider framework. There's a lot of great information about ContentProviders at http://developer.android.com/guide/topics/providers/content-providers.html, and you should read that first so that the rest of this makes sense.
The files involved are (note: I've linked to specific points in the tree because this is a work in progress and I want the line number references I give you to be stable):
https://github.com/nikclayton/android-squeezer/blob/02c08ace43f775412cc9715bf55aeb83e7b5f2dc/src/com/danga/squeezer/service/AlbumCache.java
This class defines various constants that are used elsewhere, and is pretty idiomatic for anything that's implemented as a ContentProvider.
In this class, COL_ARTWORK_PATH is the column that's going to contain the content:// URL.
https://github.com/nikclayton/android-squeezer/blob/02c08ace43f775412cc9715bf55aeb83e7b5f2dc/src/com/danga/squeezer/service/AlbumCacheProvider.java
This is the implementation of the ContentProvider. Again, this is pretty idiomatic for ContentProviders that are wrapping SQLite databases. Some points of interest:
429: albumListCallback()
This code is called whenever the app receives data about an album from the remote server (that's specific to my app, and not relevant to your problem). By this point the data has been wrapped up as a list of SqueezerAlbums, so this code has to unpack that data and turn it in to rows in the database.
456: Here we call updateAlbumArt with enough data that it can do a remote fetch of the album artwork (and I've just realised looking at this code that I can make this more efficient because it's updating the database more often than it should. But I digress).
475: updateAlbumArt()
This has to fetch the remote image, resize it, store both the original and resized versions in the filesystem (why both? Because I haven't finished this, and there will be code to select the correct cached size later).
This creates a cache directory as necessary, downloads the remote image, resizes it, and saves it to the files.
535: This is the bit you're probably particularly interested in. This creates a content:// URL (using the constants in AlbumCache.java) that references the data, and puts that in COL_ARTWORK_PATH. Then it puts the absolute path to the file in the _data column.
571: openFile()
You must implement this. Users of the ContentProvider will call openFile() when they want to open the file in the database. This implementation uses openFileHelper(), which is the code that looks up the value in the _data column, opens that file, and returns a ParcelFileDescriptor to the caller.
As you may just have realised, your open implementation of openFile() doesn't have to do this -- you could use another column name, or perhaps you have a way of going straight from the URL to the file in the filesystem. This does seem to be a very common idiom though.
Assuming you've done something like that, and now have a ContentProvider for your database, to actually access the image, your application will need to have generated a URI that references a given piece of content by it's ID. The code in the app to open the file looks like this:
Inputstream f = this.getContentResolver().openInputStream(theUri);
which ends up calling your implementation of openFile(), which ends up calling openFileHelper(), which finally gets to the right file in the filesystem. One other advantage of this approach is that openFile() is called in your application's security domain, so it can access the file, and, if implemented correctly, your ContentProvider can be called by completely different applications if you make the URLs that it responds to publicly known.
If you are storing the actual audio data in the database in a format that the Android audio system can natively interpret (i.e. mp3, 3gpp, ogg) then one way you could do it is to implement a web server in a Service, and have that Service open the SQLite database, fetch the blob using Cursor.getBlob and then feeding that Blob out to a MediaPlayer instance through the web server by wrapping the byte array with a ByteArrayInputStream. I've seen implementations like this done (in those cases it was from file, not db but the same principles apply).
Alternatively you could use AudioTrack, translate the audio if its not in PCM format and then play it and handle the audio management yourself: probably a lot more work but more efficient.
Note that this will be EXTREMELY memory-intensive and will probably perform poorly: for a 5mb MP3 you'd basically have to hold the entire thing in memory since it doesn't appear that Android's SQLite interface gives you a stream interface to blobs. If you are loading multiple media files, then...bad things happen.
Related
Main Question
How the cursor retrieves data from SQlite? does it refers to database file addresses dynamically? or loads it fully to the memory? though i know the dalvik virtual machine is address based and the the first assumption is more likely to be true, as the nature of RAM memory and phone storage are almost the same.
So my main question is to know how the data are load? from loading to memory? or just addressing to database file content?
To clarify: (the sample is just for clarifying. you can skip it)
The question is raised from the point that:
I have created an app which loads data from sqlite and displays them in listview. the databsase grows up using user data by time. Now, when the database goes larger, is it required to load data to listview in a manner like using load more or pagination? or its true to load them in one place?
although, pagination would be better for responsiveness but, when trying to export data to xls or pdf format, is it possible to retrieve a cursor to all the database and save data in xls or pdf?
the messaging app of android loads all messages in one place and has causes no problem even when i have 3000 messages in one thread.
Seems that data for Cursor is stored in memory or some kind of cache file (it's implementation details as I've properly mentioned).
There are two possible ways to proof / show why it's not original DB file. I'm sure there should be also some kind of more theoretical explanation.
Take a look at SQLiteCursor source (it's available in your sdk platform installation): it's based on CursorWindow which is
A buffer containing multiple cursor rows.
A CursorWindow is read-write when initially created and used locally.
When sent to a remote process (by writing it to a Parcel), the remote
process receives a read-only view of the cursor window. Typically the
cursor window will be allocated by the producer, filled with data, and
then sent to the consumer for reading.
Also, from the source it looks like that Window contains all data in that buffer.
Create test application with test DB with lot of records. Request all records and show in the list. While list is showing keep constantly changing db content and observe that list content is not changed (I assume eliminate usage of requery() and related deprecated stuff).
I have an external database on Azure that holds large amounts of information. My app needs to be able to synchronize a selection of this data and store it locally on the app's database. I currently use web services to make calls to the database. When the app starts for the first time, it will have a large amount of data that would need to be downloaded. What's the most efficient way of downloading a large number of rows from the external database? I was thinking an XML file might be the best way, but I'm not sure. There could be thousands of rows that need downloading so I'm not sure which method would be the most appropriate.
There isn't 300 ways to download data, so you'll basically have to call your API and get the data. For obvious performance reasons, I would avoid XML and prefere JSON instead. So once you've got your JSON file, you parse it and put it in the locale database. I also would suggest you to use service as it won't be interupt, and don't forget to warn the user that you gonna download massive data ;)
If you wan't to minimize the amount of data downloaded, you can store a JSON file for example in the assets that will contains all the things "static" that won't change online.
I would offload the work to some sort of service. Services have the benefit of running in the background without interacting with the application. You would still need to create another thread to do the work. When the download is finished, it will simply destroy itself.
Data should be downloaded anyway whatever method you use, maybe you may prepare the initial data as XML file, download it the first time and store the data in the database showing a progress bar to the user
How do I implement a ContentProvider with a JSON file? My goal is wanting the contents of a json file (or the file itself) from one app to be transferred to another app. Android documentation says it can be done with BLOBs but I have no idea what that means, and all of their examples are referring to SQL databases, which my app doesn't use (all data is stored in one JSON file).
My goal is wanting the contents of a json file (or the file itself) from one app to be transferred to another app.
To literally do this, use openFile() on the ContentProvider side and openInputStream()/openOutputStream() on the ContentResolver (client) side.
However, I suspect that this is not an especially good idea. JSON is not a data storage model that will work will with multiple simultaneous accessors. IOW, when both apps try to work with the JSON at the same time, who wins and loses?
Also, what happens if the app hosting the JSON file is uninstalled? Now the other app has no more access to the data.
We've got an android app and an iPhone app (same functionality) that use sqlite for local data storage. The apps initially come with no data, then on the first run they receive data from a remote server and store it in a sqlite database. The sqlite database is created by the server and the apps download it as one file, which is then used buy the apps. The database file is not very large by today's standards, but not a tiny one either - about 5-6 MB.
Now, once in a while, the apps need to refresh the data from the server. There a few approaches I can think of:
Download a new full database from the server and replace the existing one. This one sounds like the simplest way to deal with the problem were it not for a repeated 5-6 MB downloads. The apps do prompt the user whether they want to download the updates, so this may not be too much of a problem.
Download a delta database from the server, containing only the new/modified records and in some form information about what records to delete. This would lead to a much smaller download size, but the work on the client side is more complicated. I would need to read one database and, based on what is read, update another one. To the best of my knowledge, there's not way with sqlite to do something like insert into db1.table1 (select * from db2.table1) where db1 and db2 are two sqlite databases containing table1 of the same structure. (The full sqlite database contains about 10 tables with the largest one probably containing about 500 records or so.)
Download delta of the data in some other format (json, xml, etc.) and use this info to update the database in the app. Same as before: not to much problem on the server side, smaller download size than the full database, but quite a painful process to do the updates.
Which of the three approaches you recommend? Or maybe there's yet another way that I missed?
Many thanks in advance.
After much considerations and tries-and-errors, I went for a combination of options (2) and (3).
If no data is present at all, then the app downloads a full database file from the server.
If data is present and an update is required, the app downloads some database from the server. And checks the content of a particular value in a particular table. That value will state whether the new database is to replace the original or whether it contains deletions/updates/inserts
This turns out to be the fastest way (performance-wise) and leaves all the heavy lifting (determining whether to put everything into one database or just an update) to the server. Further, with this approach, if I need to modify the algorithm to, say, always download the full database, it would only be a change on the server without the need to re-compile and re-distribute the app.
Is there a way you can have a JSON field for each of the tables? For instance, if you got a table named users, have a column named "json" that stores the JSON for each of the users. In essence, it would contain the information the rest of the fields have.
So when you download the delta in JSON, all you got to do is insert the JSON's into the tables.
Of course with this method, you will need to do additional work in parsing the JSON and creating the model/object from it, but it's just an extra 3-4 small steps.
I will recommend approach 3, because app will download the json file more fast and local db will be updated more easily avoid overhead of more internet usages.
Just create a empty db initially according to server db and then regularly updated the same by fetching json
I am thinking about how to ideally implement a cache layer in my Android app.
Currently I have generic Activities which display data coming from a remote server. The data is represented by a DTO TemplateInstance. Each TemplateInstance has a Map with Components in it and each of the components can have child components. The components themselves can be Text (String), Image (ByteArray) or Time (or whatever by sub-classing Component).
Currently my app loads a TemplateInstance from the server each time an Activity is started.
I would now like to implement a cache layer in the app, so that
the time to display data is reduced to a minimum,
the data is refreshed when it is changed on the server.
My strategy for this looks like this:
The started Activity loads the TemplateInstance from a local storage by an ID (if exists)
A UpdateService checks in the background if the TemplateInstance has changed on the server (using a version field in the database)
If the server version is greater than the local one or there is no local TemplateInstance then retrieve the data from the server, update the local store and update the view
I implemented this already successfully with db4o. There are just two problems with this solution:
db4o is under GPL (I cannot use it)
db4o is really slow when I load TemplateInstances which have many images (4 seconds for a query)
Now I am looking for the best replacement for db4o. My ideas about that are until now:
SQLite is not suitable because of the structure of the data
I don´t need database functionality - retrieving objects by ID would be enough
Holding the objects in memory would be significantly faster
The memory state should be saved to disk when application exits, so the objects can be reinstantiated at startup
What do you think is the best solution for this?
My research on this brought me to EHCache and JCS, which I have never used. Do you think they are appropriate, also in respect of resources on an Android phone? Or do you have other suggestions?
If I understand your situation correctly, I think you should implement your own caching solution.
I would use an HashMap<id, TemplateInstance>. The HashMap is serializable and you could store/load it using ObjectOutputStream and ObjectInputStream, respectively.
db4o is not limited to GPL, via its dOCL you can opt for other open source licenses and, if you can't go open source at all, it's also totally free for Android apps
You can use my fork of
simple-disk-cache This is an easy to use disk cache which uses DiskLruCache under the hood.
I have replace Apache Commons IO dependence with google guava. And have add new public methods:
put(String key, String value, Object[] array) - the value I have use to put the timestamp for max TTL (after this time in ms the cache expired)
and: T getArray(String key, Class type)
You can put array from Serializable objects like this:
cache.put("key", String.valueOf(new Date().getTime() + 60000), Groups[] arrayGroups);
and get it: Groups[] cacheArray = getCacheArray("key", Groups[].class);
you can put in cache Bitmap images too.