By default,
benchmarkRule.measureRepeated {
runs the test for 50 times with warmups.
Is there any way we can reduce the run to 10 or something?
Found out we can configure instrumentation arguments with
testInstrumentationRunnerArgument 'androidx.benchmark.startupMode.enable' , 'true'
that captures 10 measurements.
but this doesn't want "measureRepeated" to be used,
we get:
java.lang.AssertionError: Error - multiple benchmarks in startup mode.
Only one benchmark may be run per 'am instrument' call, to ensure
result isolation.
Any idea on how this can be taken ahead?
I find they way how you can decrease count :
#Before
fun init() {
val field = androidx.benchmark.BenchmarkState::class.java.getDeclaredField("REPEAT_COUNT")
field.isAccessible = true
field.set(benchmarkRule, 1)
}
Where 1 is the number of repeat count.
but you need to be careful - this can greatly affect the stable work for library
Related
In android studio using kotlin, I created a class with the code below:
class Dice(val numSides:Int){
fun roll():Int{
return (1..numSides).random()
}
}
in ExampleUnitTest.kt file I have created the following code to test:
class ExampleUnitTest {
#Test
fun generates_number() {
val dice = Dice(4)
val rollResult = dice.roll()
assertTrue("The value of rollResult was not between 1 and 6", rollResult in 1..6)
}
}
The test should be passed only if Dice(6) as the result should be any number between 1 to 6 but as shown above when with Dice(4) or any number, the test passed. that why? thanks
Your test checks if the rolled number is between 1 and 6. With Dice(4) the result will be between 1 and 4, which means it's always going to be between 1 and 6 as well. Or to put it another way, it's impossible for you to get a value from Dice(4).roll() that isn't somewhere between 1 and 6 - it's never going to be 5 or 6, but that doesn't contradict anything!
What you probably want to test is whether Dice(4).roll()'s possible range of outputs is from 1 to 6. But there's no way to absolutely prove that from the outside, not the way the class is written anyway. All you can do is provide a value for numSides and then call roll() and check the output. That's all the class offers to you in terms of interactivity.
A better question is why you want to test this? Your test seems to be written to fail when your code is working - you should be testing that the behaviour is what you'd expect, so all your tests pass. So really, what you'd want to test is that Dice(4).roll() only produces values within the expected range, i.e. between 1 and 4.
Since it's random the only way to really do this is run the same test lots of times and make sure a bad number never comes up (or you eventually see a number you want), so you can say with a high degree of confidence that it's probably correct. Something like
#Test
fun generates_number() {
val dice = Dice(4)
val allGood = generateSequence { dice.roll() }.take(9999).all { it in 1..4 }
assertTrue("At least one dice roll was not between 1 and 4", allGood)
}
(or you could use a for loop with break to exit early if one of the values isn't valid, instead of the sequence + all)
I just picked 9999 at random but you could use statistics to pick a more suitable number. But the problem here is the random behaviour - because it's not predictable, you can't do a simple state in -> result out test with an expected result.
Edit, probably the better place to have posted this is on the appcenter forums (which I have now done):
https://github.com/microsoft/appcenter-cli/issues/1137
In short, an app I'm working on is built in appcenter and set to run unit tests, the problem I have though is that I'm unable to figure out what is making them fail (they won't fail locally).
I can't share the code that I'm having trouble with here, but to illustrate the problem I'm having, Suppose I defined the following kotlin function:
fun returnFooString() {
return "Foo "
}
and wrote the following test:
#Test
fun test_returnFooString_returns_foo() {
val foo = returnFooString()
assertThat("${foo} is equal to "Foo", "Foo", foo)
}
What I'd like to see is something like:
java.lang.AssertionError: Foo is equal to Foo
Expected: "Foo"
but: was "Foo "
Expected :Foo
Actual :Foo
However the only thing I would see in the appcenter logs for this failing test is:
com.mypackage.name.MyTest > test_returnFooString_returns_foo FAILED
java.lang.AssertionError at MyTest.kt:4
and so I have no clue what just happened. I'm still somewhat of a beginner to android development, and though I've searched google, I haven't been successful in finding something that looks relevant. Is there some setting that can be placed in the build.gradle to not suppress the assertion message, or some environment variable(s) I need to specify in appcenter, or something else to see what was expected and received?
Right now I'm seeing if I can throw exceptions in the test / code to shed some light on the issue but there must be a nicer way (especially since it takes about 10 minutes for a build)
Edit: throwing exceptions doesn't work, if I throw an exception logging out every variable I'm interested in, all I get back is something like:
com.mypackage.name.MyTest > test_returnFooString_returns_foo FAILED
java.lang.Exception at MyTest.kt:4
At this point it seems I either have to abandon the tests or do something crazy like writing 256 tests to figure out the first character my string has at position 0, then 10 minutes later another 256 to figure out the second ...
I turns out that logging can be enabled with the following:
testOptions {
unitTests.all {
testLogging {
showStandardStreams true
}
}
}
When using mockito 2, I see a huge performance boost when instantiating mocks inline vs using #Mock annotations or instantiating them in the setup method. This is super confusing to me because nowhere in the documentation is discouraged to instantiate mocks inline, despite the apparent performance boost. All times were taken from Android studio's test runner output that displays execution time.
I created a new project with a very simple test:
private val mockBanana: Banana
private lateinit var cut: Person
#Before
fun setup(){
cut = Person()
}
#Test
fun check(){
cut.banana = mockBanana
cut.eat()
verify(mockBanana).grow()
verify(mockBanana).harvest()
verify(mockBanana).peelOff()
verify(mockBanana).eat()
}
Then, I just changed the way the mock is created.
Slow method: call mock in setup. Takes 400ms
#Before
fun setup(){
mockBanana = mock(Banana::class.java)
cut = Person()
}
Slow method: mock annotation. Takes 430ms
#Mock
private lateinit var mockBanana: Banana
Fast method: calling mock inline. Takes 25ms
private val mockBanana: Banana = mock(Banana::class.java)
Initially I thought the performance boost was caused by the mocks only being created once per class and then shared between all tests, which would be a deal breaker. However, further testing showed without a doubt that that wasn't the cause (I used breakpoints and logs in the mockito code and the mocks were created multiple times. Also, the mock reference is always different between tests).
Before I start the huge endeavour it would be to refactor all the tests in my project, I'd like to understand the performance boost a little bit better to ensure nothing will break in the future.
Turns out the "performance increase" was just android studio being misleading with test execution times. I used JProfiler and saw that both inlining the method call and doing it in the setup method leads to a call to Mockito.mock() that takes the same time to execute. The only difference is where in the stack trace said call is performed, so I'm assuming android studio only starts counting after a certain point, which leads to the very low execution times.
I also double checked using gradle's built in profiler and observed the same results. Regardless of mock initialisation method, the test execution takes the same amount of time.
I am working with Espresso in android at the moment, and I have this test, which is clicking a button, populating my listView and then logging the execution time of the action.
#RunWith(AndroidJUnit4::class)
class Test {
#get:Rule
val mActivityActivityTestRule = ActivityTestRule(MainActivity::class.java)
#Test
#Throws(java.lang.Exception::class)
fun test_fetchData_populates_listView() {
//Click the get_rec_articles button.
onView(withId(R.id.get_rec_articles_btn)).perform(click())
val start = System.currentTimeMillis()
//check if the listView is created.
onView(withId(R.id.rec_articles_listview)).check(matches(isDisplayed()))
//check if items can be clicked.
onData(CoreMatchers.anything()).inAdapterView(withId(R.id.rec_articles_listview)).atPosition(0).perform(click())
val end = System.currentTimeMillis()
val loadTime = (end - start)
Log.d("###### LOADTIME FOR FETCHDATA FUNCTION:", "" + loadTime + " MILLISECS ######")
}
}
I would like to find a good way to call the test 50 times and then get the average load time. I could just run it manually 50 times and then calculate it, but I have several tests which I would like the average load times from.
I have read the question on the site about the same topic and tried the suggested solutions, but it didn't help.
I had a very similar issue and as a result I've created a library to run Android UI tests multiple times. Might be useful in your case: https://github.com/stepstone-tech/AndroidTestXRunner
As far as the load time goes, after running tests there's a JUnit report created with times of each test being placed in a generated JUNit report XML. You could get those values and calculate an average based on that.
I playing around with Kotlin and Coroutines in my demo android application.
Here's what I have:
fun testCoroutine3() = runBlocking {
var num = 0
val jobs = List(10_000) { // create a lot of coroutines and list their jobs.
launch(CommonPool) {
delay(1000L)
println(num++)
}
}
for(job in jobs) {
job.join() //wait for all jobs to finish
}
println("FINAL RESULT $num")
}
Basically I'm creating a list of 10,000 Coroutines that wait for 1 second and print a number then increment it.
Then when all jobs are done I print the final result.
(This demo is taken from the GitHub Documentation)
Now most of my test run fine, all the coroutines run almost simultaneously, and my final result is 10000
However in some rare occasions, I am getting the final result as 9,999
This become more obvious when I increase the number to 50,000 for example:
Is it possible that Kotlin is skipping some coroutines when there's a lot of them? on the 50,000, looks like it skipped 2
Or is something else happening here?
num++ consists of two operations: tmp = num + 1 and num = tmp. When dealing with multithreading like your example there are cases where some operations might overwrite the results of another thread, leading to cases like your example.
If you want to know more, research "race conditions" where the end result depends on a "race" between two seperate processes.