I am downloading a json file that countain 5000 objects. This is done in 10 seconds, but when i am trying to save all those elements in my database, that take a huge time, more than 5 minutes.
Did any one face this problem? any explanation, why it take all this time? How can i fix this problem?
I used this answer to fix my problem:
https://stackoverflow.com/a/3501572/833219
Try to use transactions to do multiple inserts.
You can only do very few commits per second but lots of inserts per transaction.
http://www.sqlite.org/faq.html#q19
Related
I have been reading through parse forums and I gather that as of 2 years ago there was no equivalent of SQL Group By.
I'm wondering if there have been any developments on this? I have thousands of records and I need to pull down all records in descending order of a value rating and then group them by name.
If this isn't available at present perhaps someone could suggest something I could work on instead.
Regards.
See https://parse.com/docs/android_guide#queries-basic , you can have secondary sorting key, which would be equivalent of "group by".
query.orderByDescending("rating");
query.addAscendingOrder("name");
could do the trick for you .
You have two options generally:
Refactor your schema to accommodate the model
Run a job to aggregate your data into "snapshots"
For option 1 it means creating "afterSave" handlers that update counters etc as data changes (expensive writes, cheap reads), here's a sample in the documentation that saves a "comments" count for a Post object:
https://parse.com/docs/cloud_code_guide#functions-aftersave
You would need to run a once-off job to set your initial counts since you already have data there (see option 2).
For option 2 you can learn more about background jobs by reading up here:
https://parse.com/docs/cloud_code_guide#jobs
Your job is allowed to run for up to 15 minutes (e.g. if it is processing insane amounts of data, or doing many things in sequence).
I am making an android app which requires 1200 records to insert and then fetch. I am using sqlite as back end. Everything is working fine but app takes lot of time to load the data at very first time for new installation. I am using SQLiteOpenHelper class to insert data. Please suggest ways to reduce the time to load the data at very first time.
EDIT :
I resolved my problem:) Here is how?
My app was taking much time because, I was inserting all the 1200 records at a time. Now i changed my code such that it should insert only required data one at a time so it saved my lot of time.
Use transactions as explained in this answer and make sure that you are only creating one connection to the SQL database for the transaction(s)
There should be a significant speed increase
I am implementing the client-server communication for an app on android. I have some high-level questions before I start to code. Apologize if the question turns out vague or confusing. I am really new to this (2 days of Google-ing is all I have), and I haven't coded anything because this design issue I am asking.
Background: Our android app records users activities. These activities are, for example, the user bookmarks an article. Eventually we want to synchronize these activities with server's database. However this sync is not time-critical, meaning it is not necessary that we send the activity to the server every time there is a new activity.
Current Design: Every time there is an activity, we get the activities and put it in a Java object, mark as "notSync", then make an update call to the database (SQLite) on the mobile. Every 5 minutes or so we make a call into the database, pull the nonSync items into an array of objects, convert it to JSON, and send it to the server.
Question 1: It seems rather inefficient to put an object into database, then every a few minutes pull it out from database as an array of objects, then convert to JSON. Would that be a better design, if we just append all the activities to an array of objects, then convert to JSON directly from that array and clear that array? My dilemma is that, If we implement the latter, there might potentially be a huge array floating around in phone's memory. How do we choose what is the best to implement?
Question 2 When sync mobile SQLite with server MySQL, is our design (using java to pull the rows out as an array of objects, then converting it to JSON) a reasonable practice?
Thanks in advance!
The first thing to keep in mind is where the bottlenecks really are. In my experience communicating with a MySQL server, the connection takes by far the longest, followed by uploading/downloading data, followed by anything I did on the phone. So even without knowing exactly what you're going to be recording, I think it's a safe bet that using SQLite and creating objects will not have a big time impact compared to your network activity.
I would run some basic tests to measure how long various operations take, but to answer your questions:
If you clear the array properly, you won't have to worry about it growing too big and, if your array ever did get that big, you probably have other things to worry about (uploading many MBs of data). However, if your user does something for 4 minutes, then quits the app or something, they'll lose their data. Storing it in the DB is safer.
JSON seems to be the best way to send your data, it's quite efficient for that, and as I said above, I don't thing the SQLite/object creation time will be too great.
I have some issues figuring out the use of aSynctask, threads and downloading images. I've made a class that downloads image but doesn't implement any thread, asynctask or whatsoever.
At the moment i'm only using one asynctask to accomplish steps 1 and 2. This takes around 10 seconds with my tablet.
I have thought of 3 steps how the application would work/proceed.
Download the most important information from url, which is in JSON format and save it into sqlite database. This is the most relevant information that other steps need to have, in order to proceed. This takes around 2 seconds at most at the moment and I am saving this into database.
Query into my database to get more urls from the previous JSON data and download even more information from the same site (30 different URL's) and save the JSON into my database. This takes around 5 seconds at the moment within the single asynctask. I was thinking about separating this into 3(?) threads if possible.
Query database again to get information that step 2 provided and download approx 200 from different URLS, images that are of size ~150px and save these images to device storage. All of the previous steps have to be done before I can do the downloading. This has to be split up into multiple threads also so I can download as many images fast as possible.
I was wondering what would be the smartest way to use asynctask for this task and the separate threads. How many threads would you think that I would have to use (of course, I'll figure this out later on when I'm optimizing). Should I first use the asynctask to download step 1 and on post execute start the other threads? Should I create the other threads within this class that has nested async class.
Thanks a lot at least for reading all this and possibly for your help.
Edit: oh I'm using a service to start the asynctask.
The only reason to make multiple threads downloading different images is if server would take some time to query for the image and as a result you are idling and not actively pumping data. But if it is not the case then your device is working as fast as your connection allows you. Hope it makes sense.
Why do we use the sqlite data base in android.I am developing an android application where the data is to be fetched from the server and do some data calculation and show it on the UI.
Is it good for me to fetch the data into the sqlite DB and update the UI on regular interval from the sqlite in evry 20 minutes or will it be good to sent the Http get request to the server and update the data from teh response on the UI.
I wanted to know which one will be better and why?Why to involve sqlite DB?
The data corresponds to some 40X40 table data on which some heavy mathematical processing is to be done and then displayed on the UI(similar to Stock market application) and data needs to be cleared after every 12 hours.
plz advice
Rgds,
Raul
It is good to use database in your case.
Pros:
If your application gets closed the in memory data will be lost, but after that you will be able to restore the state from the database if you have one
Especially for the case of complex calculations it is good to store the result once in the database and not recalculate it multiple times on demand
The database will untie your UI from the internet connection and thus you will be able to display results even if there is not internet connection
Using database you will be able to fetch the updated data from a background service, without impacting your UI
Organizing your data in database usually makes it a lot easier to manage all the application data.
Cons:
Adding database will require a bit of additional effort on your side
As you can see my list proves you SHOULD use database in your case. Maybe I am biased, but at least I provide you with things to consider.
It's really a design decision, SQLite offers a very robust way to organize and persist your data, you're only other options are to write to a file, or to save in SharedPrefs, both methods become a lot harder to manage once the size of your data begins to grow, as you must manually keep a list of objects and manage their names etc etc. 40 x 40 table data is large enough to justify using SQLite, even if you are dropping and recreating the table every 12 hours.
You might want to consider using an ORM library to make fetching and saving data from the DB simpler, ORMLite is good and compatible with Android
http://ormlite.com/
If your application relies heavily on an internet connection you don't need to buffer information in the database. However if you want to use the app where you have bad or no signal you might want to used cached values from the sqlite database.
With slow internet connection your application may be unresponsive so caching may be a good idea - but it doesn't necessarily be in the sqlite database. You should use the sqlite database for data that is required by the device frequently and that is irrelevant to your server component.
If the data is updated frequently but only while the application runs you might want to cache in the devices memory. I assume your app is not running all the time within the 12 hours but is called regularly instead to check something.
12hrs is a long time, so rather than leaving your data wander in RAM, i would suggest you to use database. Because you never know when you may need to read it again.
Otherwise, if your purpose is only to downloaded data, process it and display in activity, then you dont need to involve database because if your app is closed (due to user or low memory), in anyway your app will be downloading fresh data from server... am i right?
> update the UI on regular interval from the sqlite in evry 20 minutes
Dont expect your app to be open for such a long duration.
To precisely suggest to your case
Avoid DB
Fetch Data at app start or at appropriate time when app is opened
and save it in plain java objects.
Define Methods within it that perform operation in it.
Make map or list to save those POJO
Define Seprate Controller Classes within your project to update map of pojo at any
specific change to make fresh data available to UI.