What is the best way to keep a List<String[]> while my application runs? I am having problems with my approach. It most of the time gives me an OutOfMemory Error since the list is too big.
The List<String[]> is the result of parsing a csv file that I have downloaded online. What I do is parse the csv in an activity then save its result in a static class member like:
String url = "http://xxx/pdf/pms/pms_test.csv";
try {
InputStream input = new URL(url).openStream();
CSVReader reader = new CSVReader(new InputStreamReader(input));
SchedController.sched = reader.readAll();
input.close();
}
...then access ClassName.sched on different activities.
I am doing this so that the parsed data will be available in every activity... And I don't have to parse again. What can I do to improve it?
I think you can have 2 approaches.
You save the file and parse it in a lazy loading way
You create a database and save your data.
I suggest you to create a database, this is not difficult and let you to manage well your data. You can do easily lazyLoading with cursor, or use a ORM (ORMLite / Greendao)... I think this is the best way and the fastest to load your data.
Hope this will help you.
Just to add to Paresh's comment.
I can suggest something similar I used in my app.
For example I need to display list of Items in a store. (Each item will have ID, name and cost).
To achieve this I first make a request to server to get number of items, say itemCount.
Now I set a limit to number of items displayed in a page say 100.
If the itemCount is greater than 100, I display an alert saying only 100 items will be displayed and a next button can be added to download next set of items.
Or if it is a search you can ask user to go back and refine the search.
If itemCount is less than 100 then you will not have any issues
This way Paging can be implemented to avoid OutOfMemory issues
Related
I am making an app that blocks inappropriate websites for parental control, I have the blocked websites in a text file which 50MB in size. I want to add them all to room database so that I can check if a url is blocked or not.
But reading and looping through each line in the text file taking forever, is there any better way I can read the file and add each line to room database?
FileInputStream inputStream = null;
Scanner sc = null;
try {
inputStream = new FileInputStream(path);
sc = new Scanner(inputStream, "UTF-8");
while (sc.hasNextLine()) {
String line = sc.nextLine();
// insert to room database
}
// note that Scanner suppresses exceptions
if (sc.ioException() != null) {
throw sc.ioException();
}
} finally {
if (inputStream != null) {
inputStream.close();
}
if (sc != null) {
sc.close();
}
}
Make it as json and place it your project..
And when starting the application show a syncing state and load data to db ..
Do this only once.
You have a few options. As previously stated you could load the websites into json. This website should help automate that a bit. https://pdfmall.com/txt-to-json. This approach would still lead to an O(n) time complexity for searching.
While it will take a bit of memory, the method I'd recommend would be to build a custom object that holds all of those lines as Strings. Every time you start the app you would need to rebuild this object, but that's better than scanning constantly. You could implement an Async thread to do this in the background.
If you take this approach I would recommend storing them in a Olog(n) data structure like a binary tree. You don't want to just throw them into an Arraylist of strings, because then you'd be doing the same thing and having to scan through the entire array. If you store them in a tree, ordered by alphabetical order or perhaps length - whatever you want to do- then you reduce your search complexity from O(n) to Olog(n). Keep in mind when using custom objects, if you intend to pass them between Activities, you will need to implement Parcelable or another method.
Third you could use an Sql database, but in your case I don't believe this would speed anything up, so I wouldn't recommend it.
Keep the urls in the text file sorted alphabetically.
Load the file in a string array list.
Do a simple binary search in the sorted list to check for an specific url.
I'm making an Android application and want to create a "Favorites" list for some objects in the app. I wanna make the list accessible and editable in all my activities and I can't really figure out the best way to do this.
Shared preferences? Writing a small txt file to the device? What's the fastest way to do this?
Thanks in advance!
dependencies {
compile 'com.google.code.gson:gson:2.3.1'
}
Then when you want to save, convert your array into String:
ArrayList<Type> yourData = new ArrayList<Type>();
String dataStr = new Gson().toJson(yourData);
//put this dataStr in your shared preferences as String
To retrieve and convert back to an object is also simple:
String str = "";
//you need to retrieve this string from shared preferences.
Type type = new TypeToken<ArrayList<Type>>() { }.getType();
ArrayList<Type> restoreData = new Gson().fromJson(str, type);
If you want to create a Favorites list, use a SQLite Database.
There's really only four ways to store data.
Shared Preferences
Databases
Local files
Remote Server - Slowest since it depends on network connection, so let's skip this.
Between the remaining 3, SharedPreferences is a great option when used to store a single value. However, it's not a good option for storing a Favorites list, mainly because you can only store a single String.
You can still store it by combining all items in your list into one string, then splitting it each time. However, as your Favorites list gets larger, this single long String will too. Splitting and combining all the time isn't efficient.
SharedPreferences is still a decent option if you only have to store the Favorite's list, but since you want to edit it too, it becomes a less attractive solution.
Local Files and Databases are the better options, however local files require you to read in the file each time you want to use it. This process of reading and writing to a file isn't as efficient as using a Database, especially if you want to edit. For example, let's say you want to remove an item from the middle of your Favorite's list. This would require you to read in the file, edit it, then write the change into the file again. Not too pleasant when compared with the ease of the final solution.
Databases are the best option for this, mainly because it's designed to manage data. You can create a single Favorite's table and add each item as it's own individual row. Fetching the entire table becomes quick and easy. Fetching a single item becomes quick and easy. Adding a new item or removing a new item is also quick and easy.
I need to work with a persistent String Array (n Rows, 1 column).
* On first running the app, the String Array needs to be created empty.
* On subsequent app executions the Array will be populated from a File and the contents need to be available throughout the rest of the app.
* As the app is executed, the Array needs to be able to 'grow' in row count
* As the app is executed, the Array rows need to be able to grow in length
* The user will have the option to Clear the Array of previous entries.
* At the end, the String Array contents will be written back to a File.
I find a lot of references to Putting and Getting from an existing SharedPreferences String[] but, in the newness of my Android development, I am struggling with how to proceed.
EDIT Follows...
The data itself suggests using an Array
Example:
MAIN ST. F55 63 KY08:57 12142015--------KY11:24 12142015345TMH KY13:57 12142015
MAIN ST. F56 WYE123 IN08:57 12142015--------KY11:24 12142015--------KY13:57 12142015
1ST ST. F57 --------KY08:57 12142015--------KY11:24 12142015789FPF KY13:57 12142015
1ST ST. F58 456FPF KY08:57 12142015998FPF KY11:24 12142015--------KY13:57 12142015
1ST ST. F59 789TTM KY08:57 12142015--------KY11:24 121420151234DG KY13:57 12142015
I first need to have this data in a File
Then in one GUI I check for the existence of the file.
If one exists, fine
If none exists, I create one.
Then, in subsequent GUI's, I must check for the existence of parameters
If they do not already exist, add them to the existing data lines.
If they already exist, notify the user
And so on and on.
Then when all of the current 'pass' data has been collected via multiple, separate GUI's, I have to write out the whole data-set into the file.
My reason for thinking that I need a SharedPreference approach is the need to find and check data from GUI to GUI as the user progresses through the app.
If that 'belief' is wrong, I am open to better approach suggestions.
EDIT 2 follows....
On further study of web references, I am beginning to think that perhaps the best approach for this data and how the data needs to change might be to use a SQLite approach. Any ideas about this?
Any assistance/suggestions you might have would be greatly appreciated.
i would discourage you from using sharedpreferences for anything else than preferences. means things that change rarely - really rarely and are really lightweight. do not put much data in there. less is better. the data structures underlying sharedpreferences are not a database.
another note. it is not a string list, but it would be a string set. sets are not necessarily ordered, nor do they necessarily keep their order. means - it is not rows. its a collection of strings that can come back in any fun order (usually there is some, but that depends on the implementation which i do not know)
now you could go and make your own list, your own data structure, save it into a string and read it out, use json to do exactly that or something similar, or better - use a database, which would exactly do that.
http://developer.android.com/training/basics/data-storage/databases.html
explains it, but as you'll see its something that might take some time.
now dont get me wrong, but i have to warn you about the following approach. it is valid, but has many problems and is far from thread safe. it will not be a problem as long as you only open it from the ui thread and do not keep anything in memory to cache - if you do it will create lots of problems.
your problem of adding a line and clearing can be solved by using a file. just a simple file
look here
http://developer.android.com/training/basics/data-storage/files.html#WriteInternalStorage
the change is to append when writing:
openFileOutput("filename", Context.MODE_APPEND);
see the context mode? now you can basically just write one line and append every time.
if you wanna clear the file, just deleteFile("filename")
this is as said not threadsafe, but can be a viable option if used carefully.
Please follow this step to achieve what you want with sharedPreference
create the class Parent for SharePreference
Create your empty Array
Convert Your empty array to String and put it on SharedPreference
to call your empty array from sharedPreference
Call your sharedPreference using your key
Convert the String to array
You get your array from the sharePreference
Hope it helps, and maybe this link will help you :
http://www.androidhive.info/2012/08/android-session-management-using-shared-preferences/
You can use my open-source library called prefser, which solves this problem and uses SharedPreferences and Gson under the hood. It's basically wrapper for default Android mechanism.
You can use it in the following way:
Prefser prefser = new Prefser(context); // create Prefser object
prefser.put("key", Arrays.asList("one", "two", "three")); // save array of Strings
String[] value = prefser.get("key", String[].class, new String[]{}); // read array of Strings
For more information check repository of the project, tests and README.md file.
Link: https://github.com/pwittchen/prefser
Please note, SharedPreferences have some limitations and shouldn't be used for storing large amount of data. If you expect a lot of data, consider using SQLite database or another type of database (e.g. with NoSQL or similar approach if you strive for simplicity).
OK, based on the data, how it needs to be manipulated and the pros and cons of using a SharedPreferences approach, I have decided to go with a SQLite approach.
With that approach I should be able to readily check:
* if the necessary table exists (if not create it)
* if the necessary Field1 + Field2 exists (if not create a new record)
* and I will be able to modify the record's Field3 contents as needed
Then when the user's actions are complete I can convert the SQLite table 'records' into strings and write them out as a File and then either DROP or PURGE the associated SQLite table (until needed next time).
I sincerely appreciate all of your suggestions.
Thank you.
i would like to ask, how to store json data. I have a JSON file, which i parse using JSON Library. Now i got the data from a file. But i want to store them and show them later again.
The question is, whats the best way to store data? And is it even worth to store them?
I'm thinking about sql database, because its simple and most used.
Official android docs have few examples, so far i searched but if u have better guide, let me know.
Thank you! :)
EDIT1:
Ok, i have json file with data, which i can add to my app using RAW resources. Those data wont change, its a list of recipes, i dont have to download it. I can read the data like this:
InputStream is = mContext.getResources().openRawResource(R.raw.package_01);
Writer writer = new StringWriter();
char[] buffer = new char[1024];
try {
Reader reader = new BufferedReader(new InputStreamReader(is, "UTF-8"));
int n;
while ((n = reader.read(buffer)) != -1) {
writer.write(buffer, 0, n);
}
is.close();
//catchblock
.....
}
and then i can parse the data trought JSONLibrary like this:
try {
//representing []JSON
JSONArray jsonArray = new JSONArray(writer.toString());
if(jsonArray != null){...}
...}
Im sending a HashMap to ListView, which includes name and id. And if the user clicks the ListView/GridView item, there is new Activity started, which shows all parsed data. I need to get match those parsed data with the id.
There are about 200 recipes in the file. The data are parsed on start of the Activity, while Splashscreen is displayed. I´m not sure, if its good idea to parse the data everytime when app starts.
So, is it effitient to parse data everytime the app starts? And if yes, how to keep the parsed data "together"? Should i use HashMap?
I hope i clarified my question :) Thanks for help.
EDIT2:
So after i knew what to do, i tried the suggested solution to use HashMap. Problem was there i got Failed Binder Exception. I have images encoded in Base64, that means i have a very long String, example here. That is a lot of data, there is limit:
The Binder transaction buffer has a limited fixed size, currently 1Mb, which is shared by all transactions in progress for the process.
I´ve tried to save it to
Set<String> titles = new HashSet<String>();
and then to SharedPreferences but the data for each recipe gets mixed.
**So, here it comes again, should i save the data to SQLite database or is there another effective option i can use? **
Thank you!
It really depends on a number of things like: How much data do you have now, how much will you have later, how complicated is the data. You could use something as simple as an array or hashmap; or something as complex as a database. You need to consider what you are trying to do , and find the simplest solution. If you are trying to persist data, you could use shared preferences, database, and internal/external storage (options outlined here).
Without more information it's hard to say what exactly to do. Keep it simple though. If you are getting JSON from a web service, I'd use an ArrayList or HashMap to store the data, rather than persisting it. It is simpler to implement and does the job.
EDIT:
To answer your question: Yes, using a HashMap and parsing each time is fine. You only have 200 fields, and you don't have images, so the time it will take to parse is minimal. Regardless of how you store the data, there is going to some level of "parsing" done. For example, if you store the data in a database, you are going to have to still pull the data, and put it into a HashMap.
I want to know a good way to create a list in my android app. I have all info in my DB and want to load data from it each time I start the app and make a list from it (id and title).
What is the best approach?
Should I make a PHP-script that responds with a JSON encoded array with all list items or should I make an XML-file that generates each time the data in the DB changes that I import to the app each time it starts? or any other good way to do it?
Since all stuff are made by XML-files in android it feels like importing a XML would be a good thing, is it? And how do I import an XML-file from a web server into the app?
// Daniel
You can use either JSON or XML.
You can use the web service approach or you can include your db with your application.
In fact, I most often choose to create a sqlite3 database of my data and include it in the assets folder, which can be copied to the app's data folder on startup.
As for copying your sqlite3 database from assets/ to the db data directory, I found these instructions helpful.
In your situation I would pick JSON over XML for all the reason's stated in the following post: http://ajaxian.com/archives/json-vs-xml-the-debate
Plus, in android, there are JSON Array's built in by default so you don't have to do any extra passing of the code.
return new JSONArray("my json string goes here...");
Since we are talking about a mobile device, I would always generate changes in your php script rather than have a full sync as this will be a lot smaller in size that a full sync. However, you will need to give your user a option to do a full re-sync if this is applicable to your app. I would use a SQLite database to store the data and only update the changes in that.
To also make the stream smaller, you can gzip compress your output from php as this can be natively read by the android device. In my app, I compress 500kb down to ~110kb before transmitting, a huge saving on performance. Here a partial example of how to read the stream:
InputStream in = null;
HttpURLConnection httpConn = null; // you will have to write your on code for this bit.
if (httpConn.getContentEncoding() != null)
{
String contentEncoding = httpConn.getContentEncoding().toString();
if (contentEncoding.contains("gzip"))
{
in = new GZIPInputStream(httpConn.getInputStream());
}
}
else
{
in = httpConn.getInputStream();
}
I hope that this all makes sense, it's been a long day programming :)
Stu