I need to perform a series of http requests, each of which may depend on a previous http response. I have been able to achieve this using an AsyncTask "tree" of sorts, but as the decision tree grows, the AsyncTask technique grows more unwieldy.
I think that somehow using a SynchronousQueue (or other type of queue) is the best approach, but I can't seem to find any good guidance or tutorials on how to use a Queue for something like http requests.
Can anyone provide any guidance or point to any good tutorials on using SynchronousQueue or suggest the best kind of Queue?
Use a java.util.concurrent.SingleThreadExecutor and make a Runnable out of each HTTP operation and result-handler. You can submit subsequent tasks to it as you determine whether you need to continue progress.
For example, the HTTP "task" would run and submit the Result "task" on success, or the Error "task" on failure. The Result task would in-turn submit another HTTP task when it was done processing. Using SingleThreadExecutor ensures only one task runs at-a-time.
You could use a ThreadPoolExecutor if you can handle multiple operations in-flight at once.
Take all that, and wrap it in an AsyncTask that manages the top-level "kick-off" and waits for everything to complete. It would probably be useful to have a ConditionVariable or something to synchronize the "end" signal (using a Done "task") so you can safely tear down the Executor.
A SynchronousQueue doesn't do anything helpful for you here, because it leaves you to do all the tread management. If you use an Executor that is all handled and all you deal with is Runnables and Futures. That's probably why you are not finding any tutorials. Anyway, the Executors all use one of those queue implementations underneath!
As requested, here is some skeleton Java code. Unsupported untested as-is. This should get you started. You can use a different synchronization object if you don't like ConditionVariable.
This is a generic technique, not specific to Android, feel free to use it in other contexts.
This functions as a State Machine, with HttpTask et al forming the states, and the transitions are hard-coded by submitting the Next State to the ExecutorService. There's even a "Big Bang at the end, so everyone knows when to clap" in the form of the ConditionVariable.
Some may consider DoneTask and FailedTask overkill, but it keeps the Next State mechanism consistent, and lets Future<? extends ResultTask> function as a somewhat type-safe container for the results, and certainly keeps you from mis-assigning to it.
abstract class BasicTask {
final ExecutorService es;
final ConditionVariable cv;
public BasicTask(ExecutorService es, ConditionVariable cv) {
this.es = es;
this.cv = cv;
}
}
abstract class HttpTask extends BasicTask {
// source omitted.
// you should make a class to prepare e.g. Apache HTTP resources for specific tasks (see below).
}
abstract class ResultTask implements Runnable {
final ConditionVariable cv;
public ResultTask(ConditionVariable cv) {
this.cv = cv;
}
public void run() {
cv.open();
}
}
final class FailedTask extends ResultTask {
final Exception ex;
public FailedTask(ConditionVariable cv, Exception ex) {
super(cv);
this.ex = ex;
}
public Exception getError() { return ex; }
}
final class DoneTask<T> extends ResultTask {
final T results;
public DoneTask(ConditionVariable cv, T results) {
super(cv);
this.results = results;
}
public T getResults() { return results; }
}
class HttpSequence extends AsyncTask<Void,Void,Object> {
// this will capture the ending task
Future<? extends ResultTask> result;
// this is an inner class, in order to set Result. Refactor so these are small.
// if you don't like inner classes, you still need to arrange for capturing the "answer"
final class SomeHttpTask extends HttpTask implements Runnable {
public void run() {
try {
final SomeType thisStep = doTheStuff(lastStep);
if(thisStep.isDone()) {
// we are done here
result = es.submit(new DoneTask<SomeType>(cv, thisStep));
}
else if(thisStep.isFailed()) {
// not done: we can't proceed because of something in the response
throw thisStep.getError();
}
else {
// not done, everything is ok for next step
es.submit(new NextHttpTask(es, cv, thisStep));
}
}
catch(Exception ex) {
result = es.submit(new FailedTask(cv, ex));
}
}
}
final class TheFirstTask extends HttpTask implements Runnable {
// source omitted. just emphasizing you need one of these for each "step".
// if you don't need to set Result, this could be a static inner class.
}
#Override
public Object doInBackground(Void...) {
final ExecutorService es = Executors.newSingleThreadExecutor();
final ConditionVariable cv = new ConditionVariable(false);
try {
es.submit(new TheFirstTask(es, cv));
// you can choose not to timeout at this level and simply block until something happens...
final boolean done = cv.block(timeout);
if(!done) {
// you will need to account for unfinished threads, see finally section!
return new IllegalStateException("timed out waiting on completion!");
}
if(result != null) {
final ResultTask done = result.get();
if(done instanceof DoneTask) {
// pass SomeType to onPostExecute()
return ((DoneTask<SomeTYpe>)done).getResults();
}
else if(done instanceof FailedTask) {
// pass Exception to onPostExecute()
return ((FailedTask)done).getError();
}
else {
// something bad happened, pass it to onPostExecute()
return new IllegalStateException("something unexpected signalled CV!");
}
}
else {
// something bad happened, pass it to onPostExecute()
return new IllegalStateException("something signalled CV without setting result!");
}
}
catch(Exception ex) {
// something outside workflow failed, pass it to onPostExecute()
return ex;
}
finally {
// naive shutdown (doesn't interrupt running tasks): read JavaDoc on ExecutorService for details
es.shutdown();
}
}
#Override
public void onPostExecute(Object result) {
if(result instanceof SomeType) {
// success UI
}
else if(result instanceof Exception) {
// error UI
}
}
}
I can't say for sure without knowing the details of your use case, but you probably want to avoid the SynchronousQueue, as it will block the thread putting things into the queue until the listener thread takes it back out of the queue. If you were putting things in using the UI thread you'd be locking up the UI.
I think a BlockingQueue may suit your needs. The JavaDoc has a good producer-consumer example.
Related
I have an Android app from which I receive BLE data (every 62ms via notifications). The app can save data via a BufferedWriter to a file. Upon each onCharacteristicChanged() callback, I call either an AsyncTask, Thread or an IntentService to do a file write if the user enabled file save.
The AsyncTask seems to work fine. But the docs say execute must be invoked on the UI thread, and I'm calling it from the BLE callback. Is that a problem? And how should I fix it?
Using Thread causes this error: GKI_exception out of buffers https://code.google.com/p/android/issues/detail?id=65455 (except my code is not scanning but receiving notifications) and if the file save is long, I need to power cycle the Nexus 7 (the app and BLE become totally unresponsive). Why does the Thread not work and how can I fix it?
The IntentService never goes to the onHandleIntent(). What are the issues here?
Here is some code:
...
_context = this.getApplicationContext();
...
private BluetoothGattCallback mGattCallback = new BluetoothGattCallback() {
...
#Override
public void onCharacteristicChanged(BluetoothGatt gatt, BluetoothGattCharacteristic characteristic) {
...
int mode = 1;
if (mode==0) // Asynctask
new doFileWriteTask().execute(strBuild.toString());
else if (mode==1) // Thread
{
final String str = strBuild.toString();
new Thread(new Runnable() {
public void run() {
try {
_writer.write(str);
} catch (Exception e) {
e.printStackTrace();
}
}
}).start();
}
else if (mode==2) // intentService
{
Intent mServiceIntent = new Intent(_context, writeFileService.class);
mServiceIntent.putExtra("foo", strBuild.toString());
startService(mServiceIntent);
}
}
...
};
private class doFileWriteTask extends AsyncTask<String, Void, Void> {
#Override
protected Void doInBackground(String... strings) {
try {
_writer.write(strings[0]);
} catch (Exception e) {
e.printStackTrace();
}
return null;
}
private class writeFileService extends IntentService {
public writeFileService() {
super("writeFileService");
}
#Override
protected void onHandleIntent(Intent workIntent) {
String dataString = workIntent.getStringExtra("foo");
try {
_writer.write(dataString);
} catch (Exception e) {
e.printStackTrace();
}
}
}
...
But the docs say execute must be invoked on the UI thread, and I'm calling it from the BLE callback. Is that a problem? And how should I fix it?
The framework triggers the AsyncTask callback methods on the same thread it was called from (presumed to be the main thread). It doesn't really affect the background work, but you could see problems if you started trying to use onPostExecute() and the like. AsyncTask probably isn't the best choice to be called from a thread that you don't have control over.
Why does the Thread not work and how can I fix it?
I can't say exactly why you are still seeing errors, through spawning a series of private unsynchronized threads will probably lead to other headaches. If you want to use a single worker thread, a better choice would be to use a single HandlerThread that you can post to from your event callbacks using a Handler, something like:
…
_workerThread = new HandlerThread("Worker");
_workerThread.start();
_handler = new Handler(_workerThread.getLooper(), new Handler.Callback() {
#Override
public boolean handleMessage(Message msg) {
String str = (String) msg.obj;
_writer.write(str);
return true;
}
});
…
#Override
public void onCharacteristicChanged(BluetoothGatt gatt, BluetoothGattCharacteristic characteristic) {
…
Message msg = Message.obtain(_handler, 0, strBuild.toString());
_handler.sendMessage(msg);
…
}
That solution is quite a bit more code, but given the frequency of writes this is probably the most efficient choice.
The IntentService never goes to the onHandleIntent(). What are the issues here?
You should pretty much never implement a top level Android component (activity, service, content provider, receiver) as an inner class, because they have to be declared in your manifest as well (and the XML syntax for inner classes is ugly). If your service does not have a matching entry in the manifest, then you will never see it start. You might want to have a look at the docs on using services.
At a minimum, a Service written as an inner class must be public static to work. Otherwise the framework cannot see it and cannot instantiate it using a default constructor (non-static inner classes mess with the constructor). Unless you are calling startService() inside of a try/catch right now, I'm surprised it isn't crashing when you attempt this.
IntentService is probably the simplest of your three choices because it is the most decoupled and the framework will handle queueing up work and tearing down the threads when all the incoming work is done.
I think this is a quite common problem, but still I didn't find a satisfactory answer so I'm going to ask myself.
This is a piece of code:
// this is insine OnClickView
TextView status = (TextView) findViewById(R.id.status);
status.setText("Trying to connect to the server...");
try {
// this opens a socket and send a login request to the server.
int result = CommunicationManager.login(String email, String password);
switch (result) {
case CommunicationManager.SUCCESS:
// login ok, go on with next screen
break;
case CommunicationManager.WRONG_EMAIL:
status.setTextColor(Color.RED);
status.setText("Wrong Email!");
break;
case CommunicationManager.WRONG_PASSWORD:
status.setTextColor(Color.RED);
status.setText("Wrong Password!");
break;
}
} catch (CommunicationException e) {
status.setTextColor(Color.RED);
status.setText("Unable to estabilish a connection!");
} catch (ProtocolException e) {
status.setTextColor(Color.RED);
status.setText("Protocol error!");
}
This is what I would like to achieve:
User click Send button;
status textview shows "Trying to connect to the server...";
UI "waits" for communications to be over;
status textview shows result accordingly.
But instead when user clicks Send button, UI freezes (oddly before status text appears) until communication is done (I tried to connect to an unknown host).
A quick fix is to set a socket timeout, but I don't like this kind of solution: UI still freezes and which timeout should be set?
My first thought were Thread obviously, but as you can see I need to return a value, thing that in threading environment doesn't make much sense since threads run independently and asynchronously.
So what I need is definitely that UI waits for the service to be executed but without freezing.
By the way it seems to me that waiting for a return value means that UI has to wait for the task to be over, I just would not let it freeze.
I came across AsyncTask but I see two major disadvantages:
it seems to me that is too tightly coupled with UI;
what if I want to execute service with Integer, String and Boolean parameters? Should I extend AsyncTask<Object, Void, Void>?
Both leads to inextensibility.
What can I do to achieve my goal?
Please note that another request to the service will be a request for something that could not be ready yet, so I should automatically repeat request every few time (let's say ten minutes). So probably I'll need something I can use with TimerTask, but I'll still need to return a value to UI every time I execute that service (so I can update the status text and let the user know what's going on).
This is typical use case while dealing through external communication i.e. HTTP calls.
Best way is to use AsyncTask. To give you answers for your concerns for AsyncTask.
it seems to me that is too tightly coupled with UI;
Here good code design will play a role. You can write you own call back mechanism to get rid of tight coupling. Example can be below.
Create your version for request and response you need for WS call. It can be very simple primitive type or complex type parameters.
class Result{
//Define more para.
}
class Request{
//Deinf more para.
}
Write below callback interface.
public interface MyCallBack {
public void onComplete(Result result);}
Create AsyncTask and get above Interface object in constructor, same object can return Result object.
class LongRunningTask extends AsyncTask<Request, Integer, Long>{
private MyCallBack callback;
public LongRunningTask(MyCallBack callback) {
super();
this.callback = callback;
}
#Override
protected Long doInBackground(Request... params) {
// Perform your back ground task.
return null;
}
#Override
protected void onPostExecute(Long result) {
super.onPostExecute(result);
callback.onComplete(new Result()); //Here result is dummy but in real it should be contructred from doInBackground() method
}
}
Now last and important part to implement the interface and call asynctask. I am trying to reuse your code to have better clarity.
public class MainActivity extends Activity implements MyCallBack{
#Override
protected void onCreate(Bundle savedInstanceState) {
super.onCreate(savedInstanceState);
setContentView(R.layout.activity_main);
TextView status = (TextView) findViewById(R.id.status);
status.setText("Trying to connect to the server...");
}
private void onClick(){
//Similer to CommunicationManager.login(String email, String password); in your code.
LongRunningTask longRunningTask = new LongRunningTask(this);
longRunningTask.execute(new Request());
}
#Override
public void onComplete(Result result) {
try {
int result = result.getStatus
switch (result) {
case CommunicationManager.SUCCESS:
// login ok, go on with next screen
break;
case CommunicationManager.WRONG_EMAIL:
status.setTextColor(Color.RED);
status.setText("Wrong Email!");
break;
case CommunicationManager.WRONG_PASSWORD:
status.setTextColor(Color.RED);
status.setText("Wrong Password!");
break;
}
} catch (CommunicationException e) {
status.setTextColor(Color.RED);
status.setText("Unable to estabilish a connection!");
} catch (ProtocolException e) {
status.setTextColor(Color.RED);
status.setText("Protocol error!");
}
}
what if I want to execute service with Integer, String and Boolean parameters? Should I extend AsyncTask?
First Parameter is any user defined para. In case you need to pass multiple parameters then put them in to form of entity (i.e. - Class). Also, you can pass initial configuration parameter in constructor of AsyncTask i.e. - Communication URL.
Hope it will help.
Use multi threading, do all the communication in a different thread
Use worker thread, or AsyncTask for doing long-running operations.
Moreover, from Android Honeycomb, system throws exception, if you perform network operations on UI thread.
I have two simple classes:
public class MainActivity extends Activity {
NetworkTask task;
#Override
protected void onCreate(Bundle savedInstanceState) {
[...]
task = new NetworkTask();
task.execute();
}
public void myClickHandler(View view) {
switch(view.getId()) {
case R.id.button1:
// Why this line crash?
task.connection("127.0.0.1");
break;
}
}
}
and
public class NetworkTask extends AsyncTask<String, Void, String> {
Socket sock;
volatile boolean running = true;
public int connection(String url){
try{
sock = new Socket(url, 4567)
}
catch (IOException ex){
Logger.getLogger(NetworkTask.class.getName()).log(Level.SEVERE, null, ex);
return -1;
}
}
public String doInBackground(String... strings) {
// If I do this, it works well
//connection(127.0.0.1);
while(running)
{
[...]
}
return null;
}
}
As I commented when I call connection method from outside of the AsyncTask method, the app crashes more particulary « sock = new Socket(...) » line. But when connection call is done inside the AsynTask method socket is created.
I don't understand why.
What's happening?
Thanks.
It's because when you do
task.connection("127.0.0.1");
You are still working in the main (UI) Thread - you're not using the AsyncTask properly. Instead you're using it like a normal class, and so, you get a NetworkOnMainThreadException on the new Android versions.
However when you call from doInBackground(), it means you started the AsyncTask via execute and the work is done in a separate Thread, letting everything work as it should.
Keep in mind that if you are doing non-network stuff, you can still call from outside. However, I'd recommend keeping your AsyncTask depend on the outside as little as possible, since AsyncTasks only run once. You then have to make a new instance if you want to do more work, which means if you depend on setter methods or similar, you have to make sure you call those methods again, which makes this simple class more complex than needed.
For a good, to the point explanation of how to use an AsyncTask, this is a pretty good source. And of course the official documentation.
My app runs fine until I interrupt the initialization process at the very first start after installation by exiting and launching the app several times as long as the initialization process has not yet been finished. The processing logic and the AsyncTask can handle this pretty well, so I don't get any inconsistencies, but I have a problem with the heap. It increasing more and more while I do this disturbing exits and launches at app setup, which will lead to OutOfMemory error. I already found a leak by analyzing the heap with MAT but I still have another leak which I can't isolate yet.
For background info: I store the application context, a list and a timestamp in a static class to be able to access it from classes anywhere in my application without using tedious passing references by constructor.
Anyway, there must be something wrong with this static class (ApplicationContext) since it causes a memory leak due to the list of zones. Zone objects are processed GeoJSON data. This is how this class looks like:
public class ApplicationContext extends Application {
private static Context context;
private static String timestamp;
private static List<Zone> zones = new ArrayList<Zone>();
public void onCreate() {
super.onCreate();
ApplicationContext.context = getApplicationContext();
}
public static Context getAppContext() {
return ApplicationContext.context;
}
public static List<Zone> getZones() {
return zones;
}
public static void setData(String timestamp, List<Zone> zones) {
ApplicationContext.timestamp = timestamp;
ApplicationContext.zones = zones;
}
public static String getTimestamp() {
return timestamp;
}
}
I already tried to store the zones like this
ApplicationContext.zones = new ArrayList(zones);
but it had no effect. I already tried to put the zones attribute into another static class since ApplicationContext is loaded before all other classes (due to an entry in AndroidManifest) which could lead to such behavior but this isn't the problem too.
setData is invoked in my "ProcessController" twice. Once in doUpdateFromStorage, and once in doUpdateFromUrl(String). This class looks like this:
public final class ProcessController {
private HttpClient httpClient = new HttpClient();
public final InitializationResult initializeData() {
String urlTimestamp;
try {
urlTimestamp = getTimestampDataFromUrl();
if (isModelEmpty()) {
if (storageFilesExist()) {
try {
String localTimestamp = getLocalTimestamp();
if (isStorageDataUpToDate(localTimestamp, urlTimestamp)) {
return doDataUpdateFromStorage();
}
else {
return doDataUpdateFromUrl(urlTimestamp);
}
}
catch (IOException e) {
return new InitializationResult(false, Errors.cannotReadTimestampFile());
}
}
else {
try {
createNewFiles();
return doDataUpdateFromUrl(urlTimestamp);
}
catch (IOException e) {
return new InitializationResult(false, Errors.fileCreationFailed());
}
}
}
else {
if (isApplicationContextDataUpToDate(urlTimestamp)) {
return new InitializationResult(true, "");
}
else {
return doDataUpdateFromUrl(urlTimestamp);
}
}
}
catch (IOException e1) {
return new InitializationResult(false, Errors.noTimestampConnection());
}
}
private String getTimestampDataFromUrl() throws IOException {
if (ProcessNotification.isCancelled()) {
throw new InterruptedIOException();
}
return httpClient.getDataFromUrl(FileType.TIMESTAMP);
}
private String getJsonDataFromUrl() throws IOException {
if (ProcessNotification.isCancelled()) {
throw new InterruptedIOException();
}
return httpClient.getDataFromUrl(FileType.JSONDATA);
}
private String getLocalTimestamp() throws IOException {
if (ProcessNotification.isCancelled()) {
throw new InterruptedIOException();
}
return PersistenceManager.getFileData(FileType.TIMESTAMP);
}
private List<Zone> getLocalJsonData() throws IOException, ParseException {
if (ProcessNotification.isCancelled()) {
throw new InterruptedIOException();
}
return JsonStringParser.parse(PersistenceManager.getFileData(FileType.JSONDATA));
}
private InitializationResult doDataUpdateFromStorage() throws InterruptedIOException {
if (ProcessNotification.isCancelled()) {
throw new InterruptedIOException();
}
try {
ApplicationContext.setData(getLocalTimestamp(), getLocalJsonData());
return new InitializationResult(true, "");
}
catch (IOException e) {
return new InitializationResult(false, Errors.cannotReadJsonFile());
}
catch (ParseException e) {
return new InitializationResult(false, Errors.parseError());
}
}
private InitializationResult doDataUpdateFromUrl(String urlTimestamp) throws InterruptedIOException {
if (ProcessNotification.isCancelled()) {
throw new InterruptedIOException();
}
String jsonData;
List<Zone> zones;
try {
jsonData = getJsonDataFromUrl();
zones = JsonStringParser.parse(jsonData);
try {
PersistenceManager.persist(jsonData, FileType.JSONDATA);
PersistenceManager.persist(urlTimestamp, FileType.TIMESTAMP);
ApplicationContext.setData(urlTimestamp, zones);
return new InitializationResult(true, "");
}
catch (IOException e) {
return new InitializationResult(false, Errors.filePersistError());
}
}
catch (IOException e) {
return new InitializationResult(false, Errors.noJsonConnection());
}
catch (ParseException e) {
return new InitializationResult(false, Errors.parseError());
}
}
private boolean isModelEmpty() {
if (ApplicationContext.getZones() == null || ApplicationContext.getZones().isEmpty()) {
return true;
}
return false;
}
private boolean isApplicationContextDataUpToDate(String urlTimestamp) {
if (ApplicationContext.getTimestamp() == null) {
return false;
}
String localTimestamp = ApplicationContext.getTimestamp();
if (!localTimestamp.equals(urlTimestamp)) {
return false;
}
return true;
}
private boolean isStorageDataUpToDate(String localTimestamp, String urlTimestamp) {
if (localTimestamp.equals(urlTimestamp)) {
return true;
}
return false;
}
private boolean storageFilesExist() {
return PersistenceManager.filesExist();
}
private void createNewFiles() throws IOException {
PersistenceManager.createNewFiles();
}
}
Maybe it's another helpful information, that this ProcessController is invoked by my MainActivity's AsyncTask at the app setup:
public class InitializationTask extends AsyncTask<Void, Void, InitializationResult> {
private ProcessController processController = new ProcessController();
private ProgressDialog progressDialog;
private MainActivity mainActivity;
private final String TAG = this.getClass().getSimpleName();
public InitializationTask(MainActivity mainActivity) {
this.mainActivity = mainActivity;
}
#Override
protected void onPreExecute() {
super.onPreExecute();
ProcessNotification.setCancelled(false);
progressDialog = new ProgressDialog(mainActivity);
progressDialog.setMessage("Processing.\nPlease wait...");
progressDialog.setIndeterminate(true); //means that the "loading amount" is not measured.
progressDialog.setCancelable(true);
progressDialog.show();
};
#Override
protected InitializationResult doInBackground(Void... params) {
return processController.initializeData();
}
#Override
protected void onPostExecute(InitializationResult result) {
super.onPostExecute(result);
progressDialog.dismiss();
if (result.isValid()) {
mainActivity.finalizeSetup();
}
else {
AlertDialog.Builder dialog = new AlertDialog.Builder(mainActivity);
dialog.setTitle("Error on initialization");
dialog.setMessage(result.getReason());
dialog.setPositiveButton("Ok",
new DialogInterface.OnClickListener() {
#Override
public void onClick(DialogInterface dialog, int which) {
dialog.cancel();
mainActivity.finish();
}
});
dialog.show();
}
processController = null;
}
#Override
protected void onCancelled() {
super.onCancelled();
Log.i(TAG, "onCancelled executed");
Log.i(TAG, "set CancelNotification status to cancelled.");
ProcessNotification.setCancelled(true);
progressDialog.dismiss();
try {
Log.i(TAG, "clearing files");
PersistenceManager.clearFiles();
Log.i(TAG, "files cleared");
}
catch (IOException e) {
Log.e(TAG, "not able to clear files.");
}
processController = null;
mainActivity.finish();
}
}
Here is the body of the JSONParser. (UPDATE: I set the method none static but the problem persists.) I omit the object creations from the JSON objects since I don't think that this is the error:
public class JsonStringParser {
private static String TAG = JsonStringParser.class.getSimpleName();
public static synchronized List<Zone> parse(String jsonString) throws ParseException, InterruptedIOException {
JSONParser jsonParser = new JSONParser();
Log.i(TAG, "start parsing JSON String with length " + ((jsonString != null) ? jsonString.length() : "null"));
List<Zone> zones = new ArrayList<Zone>();
//does a lot of JSON parsing here
Log.i(TAG, "finished parsing JSON String");
jsonParser = null;
return zones;
}
}
Here is the heap dump which shows the problem:
This is the details list which shows that this problem has something to do with the arraylist.
Any ideas what's wrong here? Btw: I don't know what's the other leak since there is no details information.
Maybe important: This diagram show the status when I don't start and stop the application over and over again. It's a diagram of a clean start. But when I start and stop several times it could lead to problems due to lack of space.
Here is a diagram of a real crash. I started and stopped the app while initialization several times:
[UPDATE]
I narrowed it down a bit by not storing the Android context into my ApplicationContext class and making PersistenceManager non-static. The problem hasn't changed, so I'm absolutely sure that it is not related to the fact that I store the Android context globally. It's still "Problem Suspect 1" of the graph above. So I have to do something with this huge list, but what? I already tried to serialize it, but unseralizing this list takes much longer than 20secs, so this is not an option.
Now I tried something different. I kicked out the whole ApplicationContext so I don't have any static references anymore. I tried to hold the ArrayList of Zone objects in MainActivity. Although I refactored at least the parts I need to make the application run, so I didn't even pass the Array or the Activity to all classes where I need it, I still have the same problem in a different manner, so my guess is that the Zone objects itself are somehow the problem. Or I cannot read the heap dump properly. See the new graphs below. This is the result of a simple app start without interference.
[UPDATE]
I came to the conclusion that there is no memory leak, because "the memory is accumulated in one instance" doesn't sound like a leak. The problem is that starting and stopping over and over again starts new AsyncTasks, as seen on one graph, so the solution would be to not start new AsyncTask. I found a possible solution on SO but it doesn't work for me yet.
First of all, I have to agree with Emile:
The "..tedious passing references by constructor" is what helps avoid
issues like this. Honestly, using statics in this way is certainly one
way to create memory leaks like this, especially with a static
reference to your context.
This also applies to all those other static methods in your code. static methods are not really different from global functions. You are building a big spaghetti plate full of static methods there. Especially when they start sharing some state it will sooner or later crash or create obscure results which you wouldn't get with a proper design, especially in the presence of a highly multi-threadable platform as Android.
What also catched my eye is, please note that the onCancelled method of the AsyncTask will not be called before doInBackground has finished. So your global cancelation flag (ProcessNotification.isCancelled()) is more or less worthless (if only used in the shown code passages).
Also from the memory images you posted, the zones list contains "only" 31 items. How much is it supposed to hold? By how much does it increase? If it actually increases, the culprint might be in the JsonStringParser.parse method, which is again static. If it holds a list of items in some cache and the control logic is not working correctly (for example in the presence of multiple threads accessing it at the same time), it might add items to that cache each time it is called.
Guess 1: As the parsing method is static, this data is not (necessarily) cleaned when the application is shut down. statics are initialized once and for the purpose of this case never de-initialized until the (physical vm-)process is stopped. Android does not guarantee that the process is killed however, even if the application is stopped (see for example a wonderful explanation here). Therefore you might accumulate some data in some static part of your (maybe parsing) code.
Guess 2: Since you are re-starting your application several times, you have the background thread running several times in parallel (assumption: each time you restart the application a new thread is spawned. Note that your code shows no guards against this.) This the first parsing is still running, another one is started since the global zones variables still holds no values. The global function parse might not be thread-safe and put several data multiple times into the list which is eventually returned, yielding a bigger and bigger list. Again this is generally avoided by not having static methods (and be aware of multi-threading).
(The code is not complete, therefore guesses, there might even be other things lurking there.)
Inside your AsyncTask, you own a reference on a Context : MainActivity. When you start several AsyncTask, they are gonna be queued by an ExecutorService. So all the AsyncTask, if they are long running, will be "alive" (not garbage collected). And each of them will keep a reference on an Activity. Consequently, all you activities are gonna be kept alive as well.
This is a real memory leak as Android will want to garbage collect an Activity that is not displayed any more. And your AsyncTasks will prevent that. All the activities are kept in memory.
I encourage you to try RoboSpice Motivations to learn more about this problem. In this app we explain why you should not use AsyncTasks for long running operations. There are still a few work around that enable you to use them, but they are difficult to implement.
One way to get rid of this problem is to use WeakReference to point to your Activities inside your AsyncTask class. If you use them carefully, you can then avoid your activities not to be garbage collected.
Actually, RoboSpice is a library that allows to execute Network requests inside a service. This approach is quite interesting has it will create a context (a service) that is not linked to your activities. Thus, your request can take as long as they want and don't interfere with the garbage collection behavior of Android.
There are two modules of RoboSpice that you can use to deal with REST request. One for Spring Android and the other one for Google Http Java Client. Both libs will ease JSON parsing.
I assume you fixed the reference to MainActivity, but I'd like to mention another problem ...
You state that the parsing takes 20sec. And if you "interrupt" the app, this processing does not go away.
From the code you show here it seems 99% of that 20sec is spent inside JsonStringParser.parse().
If I look at your comment "does a lot of JSON parsing here", I assume your app makes a call into JSONParser.something() that stays away for 20sec. Even though JsonStringParser is static, each call to JsonStringParser.parse() creates a new copy of JSONParser() and my guess is that uses a lot of memory.
A background process that takes 20sec is a really big task, and in what I have seen with JSON parsers, in this time a lot of objects get created and destroyed and a lot of cycles get consumed.
So I think your root cause here is that you start a second (or third or fourth) copy of JSONParser.something(), because each of them will execute independently and try to allocate many chunks of memory, and stay running even longer than 20sec because they will have to share the CPU cycles. The combined memory allocation of multiple JSONParser objects is what kills your system.
To summarize:
Do not start another JsonStringParser.parse() until the first one
is killed or completed.
This means you must find a way to stop JsonStringParser.parse()
when you "interrupt" the app, or reuse the running copy when you
restart the app.
THink i see how it might be possible, my eyes have gone crossed eyed looking though.
Check that your not loading the data from your local storage, adding more data to it and then saving it back to local disk.
Something around the following methods in combination with other parts of your program.
If the following was called, and then you call getDatafromURL for some reason, then i believe you'd continually grow your data set.
That would be my starting point at least. Loading, appending and saving.
ApplicationContext.setData(getLocalTimestamp(), getLocalJsonData());
private List<Zone> getLocalJsonData() throws IOException, ParseException {
if (ProcessNotification.isCancelled()) {
throw new InterruptedIOException();
}
return JsonStringParser.parse(PersistenceManager.getFileData(FileType.JSONDATA));
}
Otherwise i think the problem lies in either your Parsing code, or perhaps one of the static classes your using to save the data.
MY FINAL SOLUTION
I found a solution on my own now. It runs stable and doesn't produce memory leaks when I start and stop the application a lot of times. Another advantage with this solution is that I was able to kick out all this ProcessNotification.isCancelled() parts.
The key is to hold a reference to my InitializationTask in my ApplicationContext. With this approach I can resume the running AsyncTask in a new MainActivity when I start a new one. This means that I never start more than one AsyncTask but I attach every new MainActivity instance to the currently running task. The old Activity will be detached. This looks like this:
new methods in ApplicationContext:
public static void register(InitializationTask initializationTask) {
ApplicationContext.initializationTask = initializationTask;
}
public static void unregisterInitializationTask() {
initializationTask = null;
}
public static InitializationTask getInitializationTask() {
return initializationTask;
}
MainActivity
(I have to put the progressDialog in here, otherwise it wouldn't be shown if I stop and start a new Activity):
#Override
protected void onStart() {
super.onStart();
progressDialog = new ProgressDialog(this);
progressDialog.setMessage("Processing.\nPlease wait...");
progressDialog.setIndeterminate(true); // means that the "loading amount" is not measured.
progressDialog.setCancelable(true);
progressDialog.show();
if (ApplicationContext.getInitializationTask() == null) {
initializationTask = new InitializationTask();
initializationTask.attach(this);
ApplicationContext.register(initializationTask);
initializationTask.execute((Void[]) null);
}
else {
initializationTask = ApplicationContext.getInitializationTask();
initializationTask.attach(this);
}
}
MainActivity's "onPause" contains initializationTask.detach(); and progressDialog.dismiss();. finalizeSetup(); dismisses the dialog too.
InitializationTask contains two more methods:
public void attach(MainActivity mainActivity) {
this.mainActivity = mainActivity;
}
public void detach() {
mainActivity = null;
}
onPostExecute of the task invokes ApplicationContext.unregisterInitializationTask();.
I wrote a lazy image downloader for my app using an ExecutorService. It gives me great control about how many downloads are running in parallel at what time and so on.
Now, the only problem that I have is that if I submit a task it ends up at the tail of the queue (FIFO).
Does anyone know how to change this to LIFO?
You can do it in two or three simple steps:
Create a LifoBlockingDeque class:
public class LifoBlockingDeque <E> extends LinkedBlockingDeque<E> {
#Override
public boolean offer(E e) {
// Override to put objects at the front of the list
return super.offerFirst(e);
}
#Override
public boolean offer(E e,long timeout, TimeUnit unit) throws InterruptedException {
// Override to put objects at the front of the list
return super.offerFirst(e,timeout, unit);
}
#Override
public boolean add(E e) {
// Override to put objects at the front of the list
return super.offerFirst(e);
}
#Override
public void put(E e) throws InterruptedException {
//Override to put objects at the front of the list
super.putFirst(e);
}
}
Create the executor:
mThreadPool = new ThreadPoolExecutor(THREAD_POOL_SIZE,
THREAD_POOL_SIZE, 0L,
TimeUnit.MILLISECONDS,
new LifoBlockingDeque<Runnable>());
LinkedBlockingDeque is supported only from API Level 9. To use it on earlier versions do the following:
Use the Java 1.6 implementation - download it from here.
Then change
implements BlockingDeque<E>
to
implements BlockingQueue<E>
To make it compile on Android. BlockingDeque is subtype of BlockingQueue, so no harm done.
And you're done!
You will need to specify the queue type that the ExecutorService is using.
Typically you might be retrieving an ExecutorService via the static methods in Executors. Instead you will need to instantiate one directly and pass in the Queue type that you want that provides LIFO.
EG, to create a LIFO thread pool executor, you could use the following constructor.
ThreadPoolExecutor(int corePoolSize, int maximumPoolSize, long keepAliveTime, TimeUnit unit, BlockingQueue<Runnable> workQueue)
and pass in a LIFO queue as the final parameter.
There is no LIFO queue in the java collections that I am aware of (please correct me if wrong), but you could easily just create an anonymous inner class that extends LinkedBlockingQueue and overrides the appropriate methods.
For example, (untested)
ThreadPoolExecutor executor = new ThreadPoolExecutor(4, 16, 1, TimeUnit.MINUTES, new LinkedBlockingQueue() {
#Override
public void put(Object obj) {
// override to put objects at the front of the list
super.addFirst(obj);
}
});
UPDATE in response to comments.
We can use a blocking queue that wraps a priority queue. We have to wrap because the Executor expects runnables but we need timestamps too.
// the class that will wrap the runnables
static class Pair {
long timestamp;
Runnable runnable;
Pair(Runnable r) {
this.timestamp = System.currentTimeMillis();
this.runnable = r;
}
}
ThreadPoolExecutor executor = new ThreadPoolExecutor(4, 16, 1, TimeUnit.MINUTES, new BlockingQueue<Runnable>() {
private Comparator comparator = new Comparator<Pair>() {
#Override
public int compare(Pair arg0, Pair arg1) {
Long t1 = arg0.timestamp;
Long t2 = arg1.timestamp;
// compare in reverse to get oldest first. Could also do
// -t1.compareTo(t2);
return t2.compareTo(t1);
}
};
private PriorityBlockingQueue<Pair> backingQueue = new PriorityBlockingQueue<Pair>(11, comparator);
#Override
public boolean add(Runnable r) {
return backingQueue.add(new Pair(r));
}
#Override
public boolean offer(Runnable r) {
return backingQueue.offer(new Pair(r));
}
#Override
public boolean offer(Runnable r, long timeout, TimeUnit unit) {
return backingQueue.offer(new Pair(r), timeout, unit);
}
// implement / delegate rest of methods to the backing queue
});
The ThreadPoolExecutor has a constructor which allows to specify the queue type to use. You can plug any BlockingQueue in there, and possibly a priority queue might be a good fit for you. You can configure the priority queue to sort based on a (creation) time stamp which you add to you download jobs, and the executor will execute the jobs in the desired order.
I had the same requirements: Lazy loading and LIFO for a better user experience. So I have used the ThreadPoolExecutor with a wrapped BlockingQueue (like mentioned before).
For easy backward compatibility I decided to go the easy way and for older devices I am simply using a fixed thread pool - wich means FIFO ordering. That's not perfect but for the first try okay. This looks like:
try {
sWorkQueue = new BlockingLifoQueue<Runnable>();
sExecutor = (ThreadPoolExecutor) Class.forName("java.util.concurrent.ThreadPoolExecutor").getConstructor(int.class, int.class, long.class, TimeUnit.class, BlockingQueue.class).newInstance(3, DEFAULT_POOL_SIZE, 10, TimeUnit.MINUTES, sWorkQueue);
if (BuildConfig.DEBUG) Log.d(LOG_TAG, "Thread pool with LIFO working queue created");
} catch (Exception e) {
if (BuildConfig.DEBUG) Log.d(LOG_TAG, "LIFO working queues are not available. Using default fixed thread pool");
sExecutor = (ThreadPoolExecutor) Executors.newFixedThreadPool(DEFAULT_POOL_SIZE);
}