Develop Reference

How to find all occurrences where main thread waited on lock held by background thread in a Android Profiler output

My application uses background thread to initialize some of the stuff required by main thread to load the hero activity. It has bunch of locks for synchronization. I am looking for a quick way to plot possible stack traces where main thread was waiting for lock held by background thread during app startup. I checked Thread Status Monitor. How do I debug this? What's causing it? but this is talking about point to time thread state where as I am looking for all such events that occurred during startup without knowing where those occured.
I do see that this information can be found by manually inspecting startup android profile trace and looking at various thread stacks to find these occurrences, but there is lot of data to go through. It would be great if the tool can just show me # of times main thread waited for lock held by other threads, total amount of time spent and places where it held those locks.
Q1) Do we know whether Android profiler can show this? I checked https://developer.android.com/studio/profile/cpu-profiler but couldn't find it.
Q2) If not, is there any other tool that can parse profiler exported trace and print this info?
Q3) If not, do you have any pointers on how to read exported trace file. It seems to be binary. I am going through https://github.com/JetBrains/android to figure it out. It seems to be using perfetto now, but I haven't been able to write any utility to read that data yet.
Update: I discovered that CPU profiling/Systrace option shows that my main thread does remain idle or waits for some resources, however it doesn't have information around what does it wait for or what methods run after it was idle for some time. Any pointers to how to marry java method traces and sys trace view?

Perfetto is able to show this information. Look for "Lock Contention" messages on the timeline for a given thread. It also prints the data which thread has obtained that lock.

Sudden spike of ANRs the last few days, trace doesn't have any of my code in it. How do I go about troubleshooting this?

I've had a sudden spike of ANRs lately, seems like October 5th is when they really started spiking up. They are affecting my latest release that is from around that date but they are also affecting my previous release that is from earlier in September. For that earlier release I can see that the ANRs really started on the 5th.
For my top two ANR traces, I don't see a single line that involves my code and the info in them isn't really useful. I'm including a full screen printout of one of the traces, they are all a bit different but they have some of the same stuff. Particularly I noticed most of them have a gms measurement call going on. While I haven't updated gms recently.
So my question is, how do I go about troubleshooting a trace like this?
Trace link https://imgur.com/JYaU9Uo
Thanks.

Here is are couple of clues from the Traces:
CountDown latch is used. So one of those threads, that is using latch is taking a long time to complete. That is causing the Main thread to be blocked.
Shared Preferences is being edited
A service is being stopped
Is there something of this phenomenon thats happening in your app?
"A service is being stopped, and shared preferences is being edited".
Countdown latch :
A java.util.concurrent.CountDownLatch is a concurrency construct that allows one or more threads to wait for a given set of operations to complete.
A CountDownLatch is initialized with a given count. This count is decremented by calls to the countDown() method. Threads waiting for this count to reach zero can call one of the await() methods. Calling await() blocks the thread until the count reaches zero.
#casolorz : Debugging ANR is a challenge. The ANR trace you have is just the state of all threads in your Application Process when the ANR has occurred. It is not as straight forwards as debugging a crash.
Here are some ways that you can debug
Method 1
See the state of the Main Thread.
If the main thread is Waiting, then figure out what is making it wait. It may be some other thread that is making it to wait. You can find out the culprit by tracking down the thread identifier. You might see something like "Waiting because of a resource held by Thread tid= ". tid should help you to navigate to the culprit.
With tid, you can follow the trail as long as possible till you reach a dead end.
If method 1 does not give you a headway, try this
Method 2 : Come up with a Hyphothesis.
Find out whats happening to the Main thread
In your case, the app is getting launched. You can see Zygote getting initialised, which confirms this hypothesis.
Once you know that the ANR is happening at App launch, see what is happening at app launch which would relate to I/O or Network.
This could be like accessing a SharedPreference which is quiet heavy(Say 1 mb which is a bad thing. You need to consider making the shared pref thin).
Or this could be an SDK that you are initialising. You can profile your code using Traces, and this would give you a fair idea of something that is taking a long time to execute.
Sometime this might be totally unrelated to your application. I have seen this occur.
Debugging ANR's are quiet challenging and its interesting too.

1.) It may be because of you are running SharedPreferences apply() method very frequently which are spanning so many threads and all threads are trying to write to the same shared preference file which are queued and while onPause() method is called all these operations are blocking to finish which is causing ANR.
2.) If you are using commit() for Shared Preferences then it will be blocking call on main thread for all previous apply() asynchronous operations for the same SharedPreference which may cause ANR on main thread.
3.) May be some big data is saved in Shared Preference because of which other commit()/apply() are placed in queue and waiting for their turn to write to the I/O file and can cause ANR once onPause() is called since all these commit() and apply() operations need to be finished before onPause() finished.

UI blocking loops behaviours differ( Oreo vs Mashmallow)

I have a small Android application which does a server call to post some User data to a server.
Following is the code :
private boolean completed = false;
public String postData( Data data){
new Thread(new Runnable() {
#Override
public void run() {
try{
String response = callApi(data);
completed = true;
}catch(Exception e){
Log.e("API Error",e.getMessage());
completed = true;
return;
}
}
}).start();
while(!completed){
// Log.i("Inside loop","yes");
}
return response.toString();
}
The above method calls the API to post data and returns the response received which works fine.
The loop at the bottom is a UI blocking loop which blocks the UI until a response is received or an error.
The problem :
I tried the same code for Marshmallow and Oreo device and the results were different.
For Marshmallow : Things moved in line with my expectation. :)
For Oreo (8.1.0) :
The very first API call works good enough after I open the App. However, the subsequent API calls after, cause the UI to block forever although an Error or Response is received from the Server(verified by logging and debugging).
However, on setting breakpoints(running in Debug mode) the App moves with much less trouble.
It seems the system is unable to exit the UI blocking loop although the condition is met.
The second behavior which was noticed is when I log a message in the UI blocking thread, the System is able to exit the loop and return from the Method though the API response is not logged.
Could someone help understand such inconsistency across these two flavors of Android and what could be the change introduced causing such a behavior for Oreo but not for Marshmallow?
Any insight would be extremely helpful.

It's more likely to be differences in the processor cache implementation in the two different hardware devices you're using. Probably not the JVM at all.
Memory consistency is a pretty complicated topic, I recommend checking out a tutorial like this for a more in-depth treatment. Also see this java memory model explainer for details on the guarantees that the JVM will provide, irrespective of your hardware.
I'll explain a hypothetical scenario in which the behavior you've observed could happen, without knowing the specific details of your chipset:
HYPOTHETICAL SCENARIO
Two threads: Your "UI thread" (let's say it's running on core 1), and the "background thread" (core 2). Your variable, completed, is assigned a single, fixed memory location at compile time (assume that we have dereferenced this, etc., and we've established what that location is). completed is represented by a single byte, initial value of "0".
The UI thread, on core 1, quickly reaches the busy-wait loop. The first time it tries to read completed, there is a "cache miss". Thus the request goes through the cache, and reads completed (along with the other 31 bytes in the cache line) out of main memory. Now that the cache line is in core 1's L1 cache, it reads the value, and it finds that it is "0". (Cores are not connected directly to main memory; they can only access it via their cache.) So the busy-wait continues; core 1 requests the same memory location, completed, again and again, but instead of a cache miss, L1 is now able to satisfy each request, and need no longer communicate with main memory.
Meanwhile, on core 2, the background thread is working to complete the API call. Eventually it finishes, and attempts to write a "1" to that same memory location, completed. Again, there is a cache miss, and the same sort of thing happens. Core 2 writes a "1" into appropriate location in its own L1 cache. But that cache line doesn't necessarily get written back to main memory yet. Even if it did, core 1 isn't referencing main memory anyway, so it wouldn't see the change. Core 2 then completes the thread, returns, and goes off to do work someplace else.
(By the time core 2 is assigned to a different process, its cache has probably been synchronized to main memory, and flushed. So, the "1" does make it back to main memory. Not that that makes any difference to core 1, which continues to run exclusively from its L1 cache.)
And things continue in this way, until something happens to suggest to core 1's cache that it is dirty, and it needs to refresh. As I mentioned in the comments, this could be a fence occurring as part of a System.out.println() call, debugger entry, etc. Naturally, if you had used a synchronized block, the compiler would've placed a fence in your own code.
TAKEAWAYS
...and that's why you always protect accesses to shared variables with a synchronized block! (So you don't have to spend days reading processor manuals, trying to understand the details of the memory model on the particular hardware you are using, just to share a byte of information between two threads.) A volatile keyword will also solve the problem, but see some of the links in the Jenkov article for scenarios in which this is insufficient.

What is the difference between ANR and crash in Android?

I have searched on the internet regarding what an ANR is. And I studied those references as well. But I don't get details regarding a crash in Android.
Can someone tell me the difference between ANR(Android not Responding) and a crash in Android?

ANR stands for Application Not Responding.
An ANR will occur if you are running a process on the UI thread which takes a long time, usually around 5 seconds. During this time the GUI (Graphical User Interface) will lock up which will result in anything the user presses will not be actioned. After the 5 seconds approx has occurred, if the thread still hasn't recovered then an ANR dialogue box is shown informing the user that the application is not responding and will give the user the choice to either wait, in the hope that the app will eventually recover, or to force close the app.
A crash is when an exception within the app has been thrown which has not been handled. For example, if you try to set the text of an EditText component, but the EditText is null and there is no try catch statement to catch the exception that your app will crash and will be force closed. The user will not see what caused the crash, they will be shown a dialogue telling that the app has force closed unexpectedly and will give them the option to send a bug report. In this example if you were to look in the bug report you would see the error caused by java.lang.NullPointerException.

ANR (Application Not Responding) is due to handling a long running task in the main (UI) thread. If the main thread is stopped for more than 5 seconds, the user will get an ANR.
Crashes are due to exceptions and errors like NullPointerException, ClassNotFoundException, typecasting or parsing errors, etc. ANR also causes a crash of the application.
Note: Never perform a long-running task on the UI thread.
Reference ANR

ANR and Crash Examples:
This question already has an accepted answer, but I am adding 2 simple examples to understand ANR and Crash better.
ANR:
// this will produce an ANR on your app
int i = 0;
while(true) {
i++;
}
Crash:
// this will crash your app : will produce java.lang.ArithmeticException
int value = 5, i = 0;
int result = value / i;

Application Not Responding (ANR):
ANR will display in the following conditions:
Response to the input event (such as key press or screen touch even) within 5 Sec.
A Broadcast Receiver hasn’t finished executing within 10 Sec.
How to avoid ANRs?
Create a different worker thread for long running operations like database operations, network operations etc.
Reinforce Responsiveness:
In android app usually, 100 to 200 ms is the threshold beyond which user will feel that app is slow. Following are the tips through which we can show application more responsive.
Show progress dialog whenever you are doing any background work and a user is waiting for the response.
For games specifically, do calculations for moves in the worker thread.
Show splash screen if your application has time-consuming initial setup.
Crash:
The crash is unhandled condition into the application and it will forcefully close our application. Some of the examples of crashes are like Nullpointer exception, Illegal state exception etc.

ANR stands for Application Not Responding, which means that your app does not register events on the UI Thread anymore because a long running operation is executed there

ANR: It is called when anything your application is doing in the UI thread that
takes a long time to complete (5 sec approx)
Reference: ANR
Crash: It is called when your Application gets some Error or Exception raised by the DVM

ANR also caused by-
No response to an input event (such as key press or screen touch events) within 5 seconds.
A BroadcastReceiver hasn't finished executing within 10 seconds.

ANR stands for Application Not Responding and its occurs when long operation takes place into Main thread......
Crash are due to exception and error like Nullpoint,

ANR stands for Application Not Responding.
It can occur due to many reasons like if an application blocks on some I/O operation on the UI thread so the system can’t process incoming user input events. Or perhaps the app spends too much time building an elaborate in-memory structure or computing the next move in the UI thread.
Blocking the main thread, won't result in a crash, but a popup will be displayed to let users kills the app after 5 seconds.
But For Crash, the main reason is the human errors.
Most of the time an app crashes is because of a coding/design error made by human
Human Errors
Lack of testing
Null Pointer exception
OutofMemory
Example :
This is common when a programmer makes a reference to an object or variable that does not exist, basically creating a null-pointer error.
If you have a bad connection, that can also make your apps crash. The app could also have memory management problems.
Please see my answer for the type of android specific exception which may cause the crash.
Android Specific Exception

ANR for ex: if You are downloading huge amount data in ui thread, meny other possibilities like insufficient memory etc it will come.. probably it leads to crashes in android , We can't say both are same one follows other

[ANR and Crash Different][1]
Android applications normally run entirely on a single thread by default the “UI thread” or
“main thread”. This means anything your application is doing in the UI thread that takes a long time to complete can trigger the ANR dialog because your application is not giving itself a chance to handle the input event or intent broadcasts.

ANR: Basically due to a long running task on main thread.
There are some common patterns to look for when diagnosing ANRs:
The app is doing slow operations involving I/O on the main thread.
The app is doing a long calculation on the main thread.
The main thread is doing a synchronous binder call to another process, and that other process is taking a long time to return.
The main thread is blocked waiting for a synchronized block for a long operation that is happening on another thread.
The main thread is in a deadlock with another thread, either in your process or via a binder call. The main thread is not just waiting for a long operation to finish, but is in a deadlock situation.
The following techniques can help you find out which of these causes is causing your ANRs.
CRASH:
Reason for crashs can be many. Some reasons are obvious, like checking for a null value or empty string, but others are more subtle, like passing invalid arguments to an API or even complex multithreaded interactions

This is good article at developer portal.
It gives clarity in detail about ANR.
https://developer.android.com/training/articles/perf-anr.html

App dies with "Sending signal." but no exception or other info

I'm working on an app that is recording data via Bluetooth, but it intermittently crashes after hours of collecting data (making it hard to track down the bug).
The logcat output isn't very helpful:
http://i.imgur.com/EalnX.png
There are no exceptions thrown and no clues for what caused the process to be terminated.
How can I figure out what went wrong? Is there an exception being thrown that isn't being shown by logcat? How can I track this bug down?

Signal 9 is SIGKILL, which will terminate a process immediately (no handlers inside the process will run). From the log line, the process is killing itself, so its not an external agent that is issuing the SIGKILL.
My guess (and its really a guess) is that the memory management code running inside your process (as part of the infrastructure, not code that you wrote) is deciding that you've exhausted some resource and the only recourse is to die. I would expect there to be more messages before this point is reached in the log, so it may be worth browsing the log history to see if there are useful warnings from the process before this point.
The line immediately before this is a GC log, which implies that some sort of memory resource is running low. But it looks like the heaps are not full, so failing allocations seems unlikely. You can still get allocation failures if the object being allocated was too large to fit on the heap, or fragmentation prevented it from being allocated. I'd expect to see more relevant log messages in this case, though.
I think capturing more of the log (perhaps filtering it by your app's PID if necessary) will help you make progress.

In my case there was no warnings or any clues in the log.
Eventually I found that my problem was that one of the activities I was going into (lets say Activity X) was registering to a broadcast receiver but never unregistered from it.
Therefor by closing the activity (Activity X) and coming back to it caused registering Again to the same broadcast receiver - which caused the mess!
Simply adding unregisterReceiver(mybroadcast); (in Activity X) solved it.
(I added mine to onDestroy. make sure you unregister in the right location).
And if you are super desperate I recommend seeing this slide share which explains Android crash debugging your errors.

this problem happens when using RXjava and not implement the onError callback method