I am building an android application in which I need to parse the data from the Database through JSON using Volley. I have 17,000+ rows in the table, I need to fetch all the row from the table via JSON and I need to store it in SQLite. Once downloaded the user can modify the fields of all row. Once the fields have been modified. The whole data need to be sent back to the server(Like synchronize).
What my problem is, If I attempt to download this huge data to my app. It's getting crashed. Is there any way that I can do to make this app work. Also tell me how to parse huge data from the server using volley.
I think it crashes because of timeOutError. When volley has to parse a large amount of data, you should set the timeout for your volley request. Default timeout is 2500ms, set it to 10000ms or higher it'll work for ex.
strReq.setRetryPolicy(new DefaultRetryPolicy(10000,
DefaultRetryPolicy.DEFAULT_MAX_RETRIES, DefaultRetryPolicy.DEFAULT_BACKOFF_MULT));
It will not consume all the time that you allotted, but it will be disconnected once all the data is fetched
timeoutError will not give you crash, but onErrorResponse would be called, please make sure that tables that you are going to insert into were created before making insert operation.
1- Use Gson for parsing data and store in ormlite database.
2-After the modification to database tables you should be synchronize table row bye row using sync adapter to server.
3-Make Service for synchronize to server
I think you are getting OutOfMemoryException, am i right ?
Make paging for huge data, you can pull 100 by 100 or 1000 by 1000 from server,then you can store them in sqlite directly
Enable largheap in application manifest.
Use this link to know how to enable it
I am using retrofit to download large file.
Try to parse using jsonreader for large data.
I am not sure what type of server technology you are using. Downloading 17K + data and uploading the same amount overtime is I guess not the optimum use of technology.
Try some technology which reduces your burden about data sync and parsing. I am sure you are aware about Firebase by google. that one of the solution in this case.
Let me know if I misunderstood your question in this case.
Related
In my android application im using web-service to get information about food the problem that if i write appl instead of apple,row,skin it dosent fill automatically and an error message will be shown,, what i did i stored 2000 name from web service in sqlite and search through them is it wrong to do this ? and how can i search with misspelling in web-service without the error message ?
The api url http://api.nal.usda.gov/ndb/search/?format=json&q=apple&sort=n&max=25&offset=0&api_key=DEMO_KEY
Salam Enas,
If the data do change frequently (like a movies list) then it's not ok to store it in the sqlite and it's better to be stored in the server database.
If the data does not change frequently, then it's ok to store them on the sqlite database although the app size will increase by 2 Mb which is a drawback.
You may also want to consider other APIs if you are not committed to use this one.
When storing data on the server, HTTP requests burden can be alleviated by caching data on the mobile.
good luck :)
I'm trying to parse a JSON stream with Android to put all the elements as Markers in an ActivityMap.
The problem is the quantity of the data: json returns over 3300 rows of a mysql db, and every row has 3 objects. So when i try to fetch all the data with an async task the system after a while returns Application Not Responding.
Any suggestion?
Maybe you can use Service. You can find tutorial in here
Services is mostly using for big data transfer ,also regular methods for example connect server per 2 hours. Services working on background and user can't see , because services doesnt include a layout only work on background. you can set your service self destroy when your work finish.Just trigger your Service ,connect your stream and handle your big response . When finish self destroy your service.
I think you should do the parsing in a service not in your application Activities.
Also you can separate a call for a smaller calls for example get every 1000 row in a separate AsyncTask but it's not a better solution in a performance wise, I think the better way is to use services or loaders instead of the traditional AsyncTask way that it's have a big couple with the activity it self.
Loaders:loaders-and-loadermanager
Services:android_services
Also check your server side timeout config, it may be due a timeout from server side.
If you have json data, and corresponding Model class, and you want to map json data to Model class, I would recommend using Gson() library. It will parse your json data to your java class. Include following line to your gradle dependencies in order to include Gson() library.
compile 'com.google.code.gson:gson:2.3'
Check below links that will give you further idea of using this library.
Link1 and Link2
I am trying to import some data from server to my android app. Server gets data from sql server and returns it throw a HttpRequest. I parse the data from HttpResponse and store it in sqlite database upon the JSONObject is parsed.
However the number of objects to be inserted is high and the operation may be canceled intentional(by pressing pause) or unintentional(internet problem). So I have below choices:
1- Ignore inserting into database after parsing JSONObject and wait for complete successful response from server: This solution is highly bad because if a problem occurs the user should start importing data again.
2- Make a feedback to server when I insert a row to database. So if I resume importing data just new records are imported from server: It is good but imposes extra network communication and also may affect performance.
3- Get one data packed file and try to parse it: So I am sure pausing the operation will not cause to data lost. But I prefer another solution rather than working with file.
What is the best way to handle this issue?
Thanks
I went through this choice two weeks ago.
I found the better way to manage these data is to build a web service in the server (for example, in PHP; i was using a wordpress blog), send to the web service an offset and limit request and then download the data in JSON format.
In this way, if you have 100 rows to query, you can tell the web service to give you the first 10 results, then you can parse and store them, and then, you can go back with another query asking the results from the row 11 to the row 20, and so on.
In this way, the app will manage many sequential requests (and the related low-weight JSON answers), instead of an extremely big JSON file.
You can also manage all these operation in background or with AsyncTask so, in the meantime, the user can work on the foreground activity.
I'm building an Android app, which should:
show some data, loaded from a server in the Internet.
At the moment I have a local SQliteDB used in my app where the data is stored, which should be displayed. I use this, because I want to be able to show the data, even if there is temporarily no internet connection available.
Next step I will work on inserting data in the local SQliteDB from a internet server. I thought about doing it this way:
When app starts, check if internet is available. If yes, connect to a webservice (including username and password). The webservice should deliver the necessary data via json object to the app and I will update the local SQlite DB.
My questions:
Is this a good idea?
Are there any better ways to do this?
The data can be viewed (and edited) by a Zend Website, too.
Thanks in advance.
Best regards
Daniel
The way you put it seems optimal. Maybe you should set a flag or alert which is time or date related..in case the app starts too many times without internet.
>> For updates to your mobile app, you should consider the priority/urgency of having the same data on the server and your app.
> For the better ways to do it, you can opt the way which suits your requirement better.
To fetch the data in one thread and render it in another,
1. Write custom Asynctasks:
http://developer.android.com/reference/android/os/AsyncTask.html
AsyncTask Android example
OR
2. Use something like AsyncHttpClient: http://loopj.com/android-async-http/
where you get onSuccess and onFailure methods to work with the response.
The option 2. is better if you just want to fetch data without doing anything else, and work on it, or save it. For the same, you need to parse the response first.
To parse your data:
As your response is JSON format, you may be better off using Gson to map data and with custom model classes. Eg.:
Gson gson = new Gson();
ModelClass modelClass= new ModelClass();
modelClass= gson.fromJson(responseContent,ModelClass.class);
//where responseContent is your jsonString
Log.i("Web service response", ""+modelClass.toString());
More on: https://code.google.com/p/google-gson/
For Naming discrepancies(according to the variables in webservice), can use annotations like
#SerializedName.
Use a for each loop to verify/browse/access the data that would be populated in/as objects/fields of your model class:
Check these for doubts:
http://docs.oracle.com/javase/1.5.0/docs/guide/language/foreach.html
How does the Java 'for each' loop work?
Now about saving your data:
>> It depends a lot on what the data from server is and how much data do you want to store.
In Android Storage Options:
http://developer.android.com/guide/topics/data/data-storage.html
a. There's Shared Preferences:
These are good for saving/storing data which would be relatively small in size and could be overwritten and fetched frequently. Eg. username, current user's details, password
http://developer.android.com/reference/android/content/SharedPreferences.html
How to use SharedPreferences in Android to store, fetch and edit values
b. Maintaining a database is good for the larger chunk needed in your app.
You can store, update or over-write the data according to your need. There can be multiple tables or more data could be stored in various fields.
http://developer.android.com/reference/android/database/sqlite/SQLiteDatabase.html
http://www.vogella.com/tutorials/AndroidSQLite/article.html
While you use Gson, you also have the option of populating the objects of model class and then storing that response in a String(maybe in SharedPreferences, depending on length/size) using gsonToJson method. I have used that, cheeky but effective.
You need to consider other stuff too, pertaining to UI and memory optimization, ListViews or layouts etc depending on your app and its control flow.
You could start a thread and get new data on loading the app. If you decide this path we had nice results with JSON using the Volley Project.
You can see a walk through here http://www.javacodegeeks.com/2013/06/android-volley-library-example.html
and get the code here
https://android.googlesource.com/platform/frameworks/volley/
Try using this library:-
https://github.com/Abhishekpalodath/FetchServerUtil-Android.git.
You can also load real-time data from the server by using this.
First of all, the strategy that i have at the moment was achieved by searching here in stackoverflow, but by some reason, the "next step" was never mentioned here..
So i'm currently developing an android app that needs to fetch some data
from a remote database. I've searched in the web and at the moment i'm
able to get the data i want through a http request (rest webservice +
json response). My problem is, i can only fetch data of one database
query to my app in the same database session..
I wanted some strategy where i could send the http request, and them
perform 2 or 3 database querys and return them back to my app, without
the need to open 2 or 3 times a database connection, or sending one
http request for every query.
I know i can do some tricky manipulations with the json response, like
putting all the database query's response separated by some marks i
choose, but i was looking for some clean solution.
Can you help me achieve this pls?
Thanks
Unless you create some combined data structure which is the result of all the queries and then convert it to a JSON string, then your suggestion is the only other way to go, I feel. The combined data structure can then be recreated from the JSON received at the client end.