I am developing an automated test suite to get the timing information for some Android applications (whose source code I do not have access to).
I haven't decided whether to use MonkeyRunner or Robotium yet. My main concern is after I perform an action on the UI (say typed an URL), how to determine when Android has fulfilled my request, all of the next activity's components are ready for use and I am ready to get the result and take the next action (say the page I requested is fully loaded, or email is fully opened).
For web-browser this is simple, I can just use onProgressChaged() or onPageFinished(). But I am looking for a more general way which works for all applications. I think Instrumentation.waitForIdleSync() or Instrumentation.waitForIdle() might be my best bet here.
However, as far as the documentation I read about MonkeyRunner and Robotium, none of them seem to integrate with waitForIdle well. In Robotium I could send some input and then get the output, but there doesn't seem to be a simple way for me to know when the output is ready, and maybe invoke a callback at that point. MonkeyRunner is similar in this aspect, too.
So I wonder is there a simple way for me to know what time my request has been fulfilled (as perceived by the user) without re-implementing Robotium functionality all by my own?
Thanks a lot.
This can be very tricky and entirely dependent on what exactly you asked monkeyrunner to do.
For example, if you have a monkeyrunner script, and issued a command to launch calculator app, you can have a python subprocess monitoring adb logcat -b events output to determine whether calculator app has been launched or not. If you are asking to press a button in the calculator, you can have a sleep of 1 or 2 seconds.
But there is no direct way to determine whether android has processed your event or not. Simply because, every operation differs and takes its own time.
You can put asserts in robotium and then use system.nanoseconds() before and after like a timer.
This might be a easy way to get timing information
Related
TL;DR
The app has tons of flows, but sometimes runs get passed faster than 2 mins...
Is there any way to keep it running until the timeout period (e.g. 1hr) is almost consumed? Attached a screenshot for a quick termination e.g.
Although the app is very big with tons of flows, sometimes runs get passed after 2min, 5mins but what is the critieria which decides that the running robo test should terminate now with a passed result? any idea what makes the recorded graph decides to go to this node? n.b. I assumed it's the terminal node
Why Robo tests get marked as passed so quickly?
It turns out that due to having a varying b.e. responses, the app journey gets changed. If there're 3 disconnected components (as in gif), what happens is sth like the app can start in any of the 3 flows resembling the 3 components. Which implies how long the journey will be
Is there any way to keep it running until the timeout period (e.g. 1hr) is almost consumed
Guiding the robo tests as explained here is a promising way to let journey bigger by following some sequence of actions which make the graph bigger
What is the criteria which decides that the running robo test should terminate now with a passed result?
Robo tests are simply applying flood fill on the app (as in gif). Where the graph nodes are represented by screens, e.g. onboarding screen, and edges are represented by actions, e.g. click on next button
Most likely it is always more or less the same time duration ...while the only difference may be the test's position in the queue (you're not the only user there, which is why it may appear as if the duration would vary). And that TerminatedActivity-33 only confirms that the Activity under test had been successfully terminated ...which is "The End" of the story.
For reasons of efficiency, the test will terminate as soon as possible - the timeout value can only be reached when it's stuck.
That the queue may also run in parallel might be another possible cause; while then, even if the real-time duration would indeed vary, the processing time (CPU shares) would still be about the same. Disclaimer: I have no clue how it internally works, just tried to apply some common sense.
I want to know how to detect when an external app runs one of this methods. I'm working with some classmates in a project where we want to examinate the response time of other applications. The idea is to measure the time between the run of each method to get an aproximation of the response time when opening the app.
Is this possible to achieve?
Android apps are sandboxed and only expose content that they intend to expose. The methods you name are part of components that cannot be accessed directly from the "outside" world. In other ways, if an app wanted you to know when those methods are being called, they will expose that information (i.e. sending a Broadcast or maybe storing the information in a ContentProvider). You can try and see if you can get some information out of the logcat, but I cannot assure how accurate and consistent it will be.
This is imprecise, but I would monitor logcat activity. Depending on the device/VM/AVD logcat is super active during transitions (such as back-grounding and foregrounding) and idle when an app is awaiting user input.
EDIT:
Other than that, if you can do your analysis off the device, perhaps look into using DDMS?
I am trying to develop an API for Android platform to measure the app performance. For example, I need the following things to measure of an android app. I need Expert suggestion
How much time app is taking to login? Login via back end system. User name and password is saved in database server and android app will check user name and password from database server. How much time app is taking to check? One common solution for this: starting time and ending time of user name/password checking function, then (ending time - starting time) will give the actual time which app needed to be login check. Any other suggestion from Expert?
Calculate the rendering time it takes for all these UI components (TextView,ImageView etc) to load successfully. ImageView image will be load from server. Need suggestion from Expert also. I can apply #1 ticks here, but like to hear from Experts :)
Thanks
Arefin
If you're concerned about local CPU usage, then don't use wall clock time (i.e., the time that would be shown by a clock on your wall). Android includes a Debug class with a static method, threadCpuTimeNanos() that would be much more appropriate for that purpose:
long startTime = Debug.threadCpuTimeNanos();
// your computation goes here
long endTime = Debug.threadCpuTimeNanos();
// If this comes out as 0, you probably don't need to worry about performance :-)
long cpuTimeInMilliseconds = (endTime - startTime) / 1000000;
To measure your app's performance, it's a fair choice to make use of a simple Stopwatch kind of utility. Or you could also make use of a standard tool like Android's traceview.
However, if you'd like to see how your app is performing amongst your users, then a solution like New Relic can come in very handy. It allows you to measure your app's performance for various HTTP requests.
Finally, to track how your app is performing when it comes to rendering objects, then it's best to use Android's very own Hierarchy Viewer.
I think (ending time - starting time) is a great solution. No point making it more complex than it is...
Can you take a look at: https://firebase.google.com/docs/perf-mon/get-started-android?
You can use either custom traces to measure performance of piece of code (by adding Trace.start(), Trace.stop() around that code) or use #AddTrace annotation to trace specific methods.
Once enabled, Firebase performance automatically measures App start time and network requests latency, request/response size etc.
I have developed an android app on which i want to run the robotium test cases. The first problem i have is, In my app i am using so many web service calls to interact with the server, Whenever the app hits the server i am displaying a progress dialog, Now i want to make wait the robotium until the app get response from the server but i can make wait for some time using waitForDialogToClose(), Actually i want to make it wait exactly till it get some response or error from the server. How to do it..??
And the Second problem is i am doing some uploading data(eg. file or image) to server and i am displaying progress bar for it, In this case i want to make wait robotium until my progress bar reaches it max value. Please help..
Regards,
Ram.
Robotium is a black box testing framework : you test an app from the UI layer and it can even run without any knowledge of the code of the app under test.
For this reason, from robotium you can't plug a listener in your model or rest client and wait till the response comes into your app. So, you have two options :
relaying on the UI as you do, waiting for a given state to appear in the UI, as you suggested with a dialog, or the content of a list, whatever.
wait for some time before continuing the test, a time at which you are pretty sure data will be there. But this is more risky and will fail more often.
Some unit tests would be nice to test your rest client (or equivalent), then if you could mock it and run your ui tests disconnected it would be nicer. But all this require a large amount of design work, worth it but complex for a small app and a beginner in java. Maybe you would be better testing only your UI and keeping in mind that your tests are not perfect.
I'm trying to make a stopwatch & countdown app for WindowPhone 7 using Silverlight for WindowPhone SDK and trying to make it run in background when it's tombstoned. In Android, I can use android.app.Service to run it in background. According to MS AppHub Quickstart, "The Windows Phone operating system doesn't allow any third-party applications to run in the background". Please help me if you have any idea for keeping the countdown running when a phone call is received or the phone goes to sleep. Thank you.
At the moment, once your app is tombstoned, your app cannot continue with any custom processes such as having your countdown continue. The Mango SDK coming out this month allows for a bit more freedom in this respect.
If, however, you want to run your app under a lock screen, this is possible. You can simply do this:
PhoneApplicationService.Current.ApplicationIdleDetectionMode = IdleDetectionMode.Disabled;
Jaime Rodriguez has a great post on running under a lock screen which you can read here.
Until Mango comes out, there's no way to have your program run in the background.
As keyboardP notes, you can't continue your process on the phone.
However you could use a server and Toast message popups to still notify the user. It requires more setup and a server to run against, but it will meet your requirements.
One thing I have seen with other people making apps like this is to store the time that app was tombstoned and then, when the app is reactivated to look at the difference between the current time and the saved time and deduct that from the countdown.
This may or may not be appropriate, depending on your requirements but may be something to consider.