I am populating a listview with data(Text,image Urls etc) fetched from server.
This data might be updated at server side weekly/monthly.
What i want is , *when my application loads the listview first time, the data fetched from server is stored at application/client side, so as to be re-used later in redrawing the listview.*How can i acheive that.
One Possible Solution is:
Use Application class to download this data first time, store it in SQLlite db and always fetch results from DB while drawing a listview.Later i can use GCM Send-To-Sync tickles to update the sqldb data.
-Querying db everytime a listview is to be drawn, is it not a good practice.??
Any common practice to achieve this.
Any views about using Loaders/Loading Manager.
EDIT
The data in questions is a jsonpackage. With structure like(consider array size as 100 at max.)
[ // JSON Array
{ // JSON Object
"rank":1,"country":"China",
"population":"1,354,040,000",
"flag":"http://www.androidbegin.com/tutorial/flag/china.png"
}, (
]
Yes, you need to store the data fetched from server to database itself and then, bind the data from SQLiteDatabase to ListView
It cannot be said that Querying db everytime a listview is to be drawn, is it not a good practice.., because you need to fetch data only once when the activity is started or when the data within database is changed.
Re-Using already fetched data from server... means that you need a persistent storage and the storage should be updated periodically or on some user request.
It depends on what kind of data you have. Generally if it in a json package coming from the server then you can just keep that raw json and parse and populate the data on every app launch. But this will only be helpful if the amount of data is not too large. If the amount of data you have at any given moment is huge then you will have to parse and store the formatted data to a SQL DB, query and fetch the data on every app launch.
Also in case of manageable amount of data you can keep the data in-memory for the duration of the app lifecycle using static data model instances.
There can be multiple ways in which you can solve this problem but it all depends on how much data do you have and in what format and accordingly you take a call on which methodology will fetch you the best performance.
Create a directory in sdcard/internal storage, whichever is available using this static method.
/**
* Cache class to create dir, save, retrieve and delete files
* File names are hashCode of the url json is downloaded from.
**/
public class FileCache {
static File cacheDir;
static final String DIRECTORY_ADDRESS = "/Android/data/<app_package>/.<directory name>";
public static createDirectory(Context context){
if (android.os.Environment.getExternalStorageState().equals(android.os.Environment.MEDIA_MOUNTED))
cacheDir=new File(Environment.getExternalStorageDirectory(),DIRECTORY_ADDRESS);
else
cacheDir=context.getCacheDir();
if(!cacheDir.exists())
cacheDir.mkdirs();
}
public static File getFile(String url){
String filename=String.valueOf(url.hashCode());
File f = new File(cacheDir, filename);
return f;
}
public static void saveFile(InputStream is, File file){
try {
OutputStream os = new FileOutputStream(file);
copyStream(is, os);
os.close();
} catch (FileNotFoundException e) {
} catch (IOException e) {
}
}
/**
* Clear all files at app uninstall or application or on
* low memory or some other event, using a boradcastreceiver/alarmmanager
**/
public static void clear(){
File[] files=cacheDir.listFiles();
if(files==null)
return;
for(File f:files)
f.delete();
}
}
Now create directory when application starts:
FileCache.createDirectory(context);
Once this is set, you have to fetch your JSONArray using an AsyncTask, store store it in a file
/**
* Add this in doinBackground()
**/
private File getJson(Stirng url) {
File f = FileCache.getFile(url);
if(f != null && f.isFile()) {
//file exists in directory, no need to download.
return f;
}
try {
URL fileUrl = new URL(url);
HttpURLConnection conn = (HttpURLConnection) fileUrl.openConnection();
conn.setConnectTimeout(120000);
conn.setReadTimeout(120000);
conn.setInstanceFollowRedirects(true);
InputStream is = conn.getInputStream();
FileCache.saveFile(is, f);
return f;
} catch (Exception ex) {
return null;
}
}
Now you have to run this thread every time you need to load and bind the data into listview. I am excluding the details, of how you will parse the json from file and convert it into List for the Adapter.
This AsyncTask will automatically fetch the data if not available in file, and load into listview, in onPostExecute(). So, from the 2nd time, it wont take much time, and data will be persisted in storage. You can also add pagination in the ListView, i.e if you have 1000 json hashes, you can read data from this file in batches, that will also be a good optimization.
Now, whenever the data gets updated (monthly/weekly), send a push notification using GCM, run a service, clear the old file using FileCache.clear(), and start this AsyncTask from the service, all in the background, so if the app is opened afterwards, you will get updated data from file storage.
Putting everything in sqlite is a similar alternative, then you have to use DbAdapters, CursorLoaders and CursorAdapters, and then using the service to update the db.
But since you dont have large data sets, I think file caching is sufficient.
Hope that helps. :)
I implemented as below:
(If you see any issues in the implementation,performance wise or any please comment)
Fetch Data(Json Array with a length of 50 ) from server for first time, store it as a string(JsonArray.toString()) in sharedpreferences and populate the listview.
When required to display the listview again, get stored jsonarraystring from sharedpreferences ,convert it back to jsonarray and populate the listview.
To get updated data from server, send a GCM tickle to update stored sharedpreferences array with new value in background.
This approach is working fine so far. Do you guys see any issues i can face at anystage.
ThankYou
Related
I am making an app that blocks inappropriate websites for parental control, I have the blocked websites in a text file which 50MB in size. I want to add them all to room database so that I can check if a url is blocked or not.
But reading and looping through each line in the text file taking forever, is there any better way I can read the file and add each line to room database?
FileInputStream inputStream = null;
Scanner sc = null;
try {
inputStream = new FileInputStream(path);
sc = new Scanner(inputStream, "UTF-8");
while (sc.hasNextLine()) {
String line = sc.nextLine();
// insert to room database
}
// note that Scanner suppresses exceptions
if (sc.ioException() != null) {
throw sc.ioException();
}
} finally {
if (inputStream != null) {
inputStream.close();
}
if (sc != null) {
sc.close();
}
}
Make it as json and place it your project..
And when starting the application show a syncing state and load data to db ..
Do this only once.
You have a few options. As previously stated you could load the websites into json. This website should help automate that a bit. https://pdfmall.com/txt-to-json. This approach would still lead to an O(n) time complexity for searching.
While it will take a bit of memory, the method I'd recommend would be to build a custom object that holds all of those lines as Strings. Every time you start the app you would need to rebuild this object, but that's better than scanning constantly. You could implement an Async thread to do this in the background.
If you take this approach I would recommend storing them in a Olog(n) data structure like a binary tree. You don't want to just throw them into an Arraylist of strings, because then you'd be doing the same thing and having to scan through the entire array. If you store them in a tree, ordered by alphabetical order or perhaps length - whatever you want to do- then you reduce your search complexity from O(n) to Olog(n). Keep in mind when using custom objects, if you intend to pass them between Activities, you will need to implement Parcelable or another method.
Third you could use an Sql database, but in your case I don't believe this would speed anything up, so I wouldn't recommend it.
Keep the urls in the text file sorted alphabetically.
Load the file in a string array list.
Do a simple binary search in the sorted list to check for an specific url.
There are lots of tutorials out there describing how to fetch JSON objects from the web and map them to Core Data.
I'm currently working on an iOS (later: Android as well) app which loads json objects from web and displays them to the user. In my opinion all this mapping from and to Core Data is an overhead in this case, it would be much easier to save the JSON objects directly and use them as "cache" in the app. Are there libraries/documented ways how to achieve fetching json objects, save them locally and fetch them with a predefined identifier?
I would love to fetch e.g. 10 objects, show them to the user and save the data locally. The next time the user is on that list the local data is shown and in the background the json-file is fetched again to be up-to-date. I guess this is a common use case but I didn't find any tutorials/frameworks enabling exactly this.
You can simply use NSURLCache to cache http responses instead saving JSONs
http://nshipster.com/nsurlcache/
There are many ways to implement this. You can implement cache using either file storage or database depending on the complexity as well as quantity of your data. If you're using files, you just need to store JSON response and load it whenever activity/fragment is crated. What I have done sometimes is store the JSON response in the form of string in a file, and then retrieve it on activity/fragment load. Here's an example of reading and writing string files:
Writing files:
FileOutputStream outputStream = context.openFileOutput("myfilename",Context.MODE_PRIVATE);
String stringToBeSaved = myJSONObject.toString();
outputStream.write(stringToBeSaved.getBytes());
Reading from files
FileInputStream inputStream= context.openFileInput("myfilename");
int c;
String temp="";
while( (c = inputStream.read()) != -1){
temp = temp + Character.toString((char)c);
You can convert this string to JSONObject using :
JSONObject jsonObject = new JSONObject(temp);
Or you can use the string according to your needs.
I want to get the data from a JSON URL. It contains 22 fields and it has above 6000 rows. I want to parse this JSON data and store it in an SQLite database.
I know the basics of JSON but when I use it I get an "out of memory" error.
After a lots of R&D and googling I found that I have to use GSON for Heavy URL, but I'm not familiar with it.
Can I implement paging in JSON? In other words, can I do something like getting partial data from JSON and storing it into a database and then request certain data and store it into a database, continuing the process until the full JSON file is parsed?
You must implement pagination mechanism on server side (the one, that serves you JSON data) and pass range parameters as part of the URL you mentioned.
If I knew I was going to have lot of rows/data, then I would run some simple SQL call to determine the size of the data. Then break up the original JSON within Android to deal with the data in smaller chunks, committing after each segment. So in your example, you know that you have 6000+ rows - store that in memory in Android as an int, then call 1000 rows at a time.
Some code after further queries:
String locationCountPath = "http://mywebserver/countAllMD5UserLocations";
try {
//My method to return a count of locations
int locCount = MyJSONUtilities.countAllUsersLocations(
locationCountPath, MyJSONUtilities.writeUserDetailsJSON(
username.toUpperCase(), password));
// Compare new count with count on phone
if (countLocs != locCount && locCount > 0) {
try {
// Grab the new set of locations
askToSyncLocations();
} catch (Exception e) {
e.printStackTrace();
}
}
} catch (Exception e) {
e.printStackTrace();
}
So, if you replace the askToSyncLocations() method with your own way of handling your 6000+ rows in a loop as batched of 1000 at a time, it should not encounter any memory problems?
I need some text data (street names of a 2 million town) to get into my android application. I think the best way to do this is to store it into a sqlite database read-only when it starts the first time. As I read, doing this with a pre-defined database is pretty tedious and not clean but you have to copy it from external storage or something and have the data twice then. So I thought about using a CSV file from raw resources and delete it after import, but this is not possible too because this data will be built into the sdk file and can't be deleted any more.
My target is it to make some kind of initial data transfer from local and delete this data source. Any ideas how to achieve that properly? I'd like to go without downloading the data from a server because this would mean that the only reason my application needs an internet connection is because of downloading the initial data. Otherwise it wouldn't need an internet connection.
What about putting the .csv file in you asset folder and read it.
String next[] = {};
List<String[]> list = new ArrayList<String[]>();
try {
CSVReader reader = new CSVReader(new InputStreamReader(getAssets().open("test.csv")));
for(;;) {
next = reader.readNext();
if(next != null) {
list.add(next);
} else {
break;
}
}
} catch (IOException e) {
e.printStackTrace();
}
I'm thinking I'd compress the data in the initial installation resource and expand it on install. This answer sees to reference a 7zip open source java api.
Data compression on Android (other than java.util.zip ?)
Meanwhile I know that the data I will use in my app is too complex to populate the data by hand. Moreover the app needs internet access at any rate, so I will use a Json string to get the data.
I've read many topic on stackoverflow,and google, about this argument but i can't orient myself with writing custom ArrayList into files !
I've an ArrayList of type "CustomType". "CustomType" have two String variable : "name" and "description".
I want to save this ArrayList for reading it after closing and reopening my app.
Does anyone help me doing this? and does anyone explain me what happen when i save/read?
you could write it simply as a csv file, like this,
1,name,desc
2,name2,desc2
3,name3,desc3
...
or you could use gson library - http://code.google.com/p/google-gson/, to convert it into a json string and then write it to a file, that way when you read it you could directly get the object from json,
i would personally use the second method,
Edit: for writing to csv
try
{
FileWriter writer = new FileWriter("<path>/MyFile.csv");
while(lst.next())
{
for(int i = 0; i < columnSize; i++)
{
writer.append(i);
writer.append(',');
writer.append(lst.get(i).getName());
writer.append(',');
writer.append(lst.get(i).getDesc());
}
writer.append('\n');
}
}
catch(Exception e)
{
e.printStackTrace();
}
(assuming the object in side the ArrayList has methods getName() and getDesc() )
this might be helpful in reading it again,
Get and Parse CSV file in android
To save any kind of data you have to choose any of the available data storage tequinue
1)shared preferences
2)internal storage
3)external storage
4)Sqlite
5)internet server
detailed docs here
throgh any one of these way you can store ArrayList or other data which you are using to populate the screen , so every time user will open the app data will be loaded from stored location
You could use XML files too:
http://androidideasblog.blogspot.com/2010/01/read-write-and-parse-xml-file-in.html
http://www.ibm.com/developerworks/opensource/library/x-android/
how to create ,read & write xml file in android by using DocumentBuilderFactory android application
http://www.anddev.org/viewtopic.php?p=30363
This way you can use all kind of text and symbols, if you are using a CSV you can not use the 'separator' in your content.