Develop Reference

How to avoid concurrency issues with Kotlin coroutines?

I am going to implement a chat feature in the android application. In order to do that, I fetch chat messages every five seconds from the server by a coroutine flow. The problem is when I want to send a message sometimes the server receives two concurrent requests and it returns an error. How should I make sure that these requests run sequentially in my chat repository? Here is my chat repository:
class ChatRepositoryImpl #Inject constructor(
private val api: ApolloApi,
private val checkTokenIsSetDataStore: CheckTokenIsSetDataStore
) : ChatRepository {
override fun chatMessages(
lastIndex: Int,
limit: Int,
offset: Int,
channelId: Int,
): Flow<Resource<ChatMessages>> = flow {
var token = ""
checkTokenIsSetDataStore.get.first {
token = it
true
}
while (true) {
val response = ChatMessagesQuery(
lastIndex = Input.fromNullable(lastIndex),
limit = Input.fromNullable(limit),
offset = Input.fromNullable(offset),
channelId
).let {
api.getApolloClient(token)
.query(it)
.await()
}
response.data?.let {
emit(
Resource.Success<ChatMessages>(
it.chatMessages
)
)
}
if (response.data == null)
emit(Resource.Error<ChatMessages>(message = response.errors?.get(0)?.message))
delay(5000L)
}
}.flowOn(Dispatchers.IO)
override fun chatSendText(channelId: Int, text: String): Flow<Resource<ChatSendText>> = flow {
var token = ""
checkTokenIsSetDataStore.get.first {
token = it
true
}
val response = ChatSendTextMutation(
channelId = channelId,
text = text
).let {
api.getApolloClient(token)
.mutate(it)
.await()
}
response.data?.let {
return#flow emit(
Resource.Success<ChatSendText>(
it.chatSendText
)
)
}
return#flow emit(Resource.Error<ChatSendText>(message = response.errors?.get(0)?.message))
}.flowOn(Dispatchers.IO)
}

One way to limit concurrency is to use utils like Mutex or Semaphore. We can very easily solve your problem with mutex:
class ChatRepositoryImpl ... {
private val apolloMutex = Mutex()
override fun chatMessages(...) {
...
apolloMutex.withLock {
api.getApolloClient(token)
.query(it)
.await()
}
...
}
override fun chatSendText(...) {
...
apolloMutex.withLock {
api.getApolloClient(token)
.mutate(it)
.await()
}
...
}
However, this problem should not be really fixed on the client side, but on the server side. Your attempted solution doesn't protect you against concurrent requests entirely. If for some reasons two instances of the application has the same token or if the user attempts to manipulate your application, it could still send concurrent requests.
If you can't easily fix the problem properly, you can apply the same fix on the server side that you intend to apply on the client side. Just handle requests or part of requests sequentially. It is more error-proof and also more performant, because this way only part of the whole request time has to be done sequentially.

Kotlin Flow: emitAll is never collected

I am trying to write a UnitTest for the kotlin-version of networkBoundResource that can be found on serveral sources with several features
Here is my version of it with marker-comments for the following question.
inline fun <ResultType, RequestType> networkBoundResource(
...
coroutineDispatcher: CoroutineDispatcher
) = flow {
emit(Resource.loading(null)) // emit works!
val data = queryDatabase().firstOrNull()
val flow = if (shouldFetch(data)) {
emit(Resource.loading(data)) // emit works!
try {
saveFetchResult(fetch())
query().map { Resource.success(it) }
} catch (throwable: Throwable) {
onFetchFailed(throwable)
query().map { Resource.error(throwable.toString(), it) }
}
} else {
query().map { Resource.success(it) }
}
emitAll(flow) // emitAll does not work!
}.catch { exception ->
emit(Resource.error("An error occurred while fetching data! $exception", null))
}.flowOn(coroutineDispatcher)
This is one of my UnitTests for this code. The code is edited a bit to focus on my question:
#get:Rule
val testCoroutineRule = TestCoroutineRule()
private val coroutineDispatcher = TestCoroutineDispatcher()
#Test
fun networkBoundResource_noCachedData_shouldMakeNetworkCallAndStoreUserInDatabase() = testCoroutineRule.runBlockingTest {
...
// When getAuthToken is called
val result = networkBoundResource(..., coroutineDispatcher).toList()
result.forEach {
println(it)
}
}
The problem is that println(it) is only printing the Resource.loading(null) emissions. But if you have a look at the last line of the flow {} block, you will see that there should be another emission of the val flow. But this emission never arrives in my UnitTest. Why?

I'm not too sure of the complete behaviour, but essentially you want to get a resource, and current flow is all lumped into the FlowCollector<T> which makes it harder to reason and test.
I have never used or seen the Google code before and if I'm honest only glanced at it. My main take away was it had poor encapsulation and seems to break separations of concern - it manages the resource state, and handles all io work one one class. I'd prefer to have 2 different classes to separate that logic and allows for easier testing.
As simple pseudo code I would do something like this :
class ResourceRepository {
suspend fun get(r : Request) : Resource {
// abstract implementation details network request and io
// - this function should only fulfill the request
// can now be mocked for testing
delay(3_000)
return Resource.success(Any())
}
}
data class Request(val a : String)
sealed class Resource {
companion object {
val loading : Resource get() = Loading
fun success(a : Any) : Resource = Success(a)
fun error(t: Throwable) : Resource = Error(t)
}
object Loading : Resource()
data class Success(val a : Any) : Resource()
data class Error(val t : Throwable) : Resource()
}
fun resourceFromRequest(r : Request) : Flow<Resource> =
flow { emit(resourceRepository.get(r)) }
.onStart { emit(Resource.loading) }
.catch { emit(Resource.error(it)) }
This allows you to massively simplify the actual testing of the resourceFromRequest() function as you only have to mock the repository and one method. This allows you to abstract and deal with the networking and io work elsewhere, independently which again can be tested in isolation.

As #MarkKeen suggested, I now created my own implementation and it works quite well. Compared to the code that is going around on SO, this version now injects the coroutineDispatcher for easier testing, it lets flow take care of error handling, it does not contain nested flows and is imho easier to read and understand, too. There is still the side-effect of storing updated data to the database, but I am too tired now to tackle this.
import kotlinx.coroutines.CoroutineDispatcher
import kotlinx.coroutines.flow.*
inline fun <ResultType, RequestType> networkBoundResource(
crossinline query: () -> Flow<ResultType?>,
crossinline fetch: suspend () -> RequestType,
crossinline saveFetchResult: suspend (RequestType) -> Unit,
crossinline shouldFetch: (ResultType?) -> Boolean = { true },
coroutineDispatcher: CoroutineDispatcher
) = flow<Resource<ResultType>> {
// check for data in database
val data = query().firstOrNull()
if (data != null) {
// data is not null -> update loading status
emit(Resource.loading(data))
}
if (shouldFetch(data)) {
// Need to fetch data -> call backend
val fetchResult = fetch()
// got data from backend, store it in database
saveFetchResult(fetchResult)
}
// load updated data from database (must not return null anymore)
val updatedData = query().first()
// emit updated data
emit(Resource.success(updatedData))
}.onStart {
emit(Resource.loading(null))
}.catch { exception ->
emit(Resource.error("An error occurred while fetching data! $exception", null))
}.flowOn(coroutineDispatcher)
One possible UnitTest for this inline fun, which is used in an AuthRepsitory:
#ExperimentalCoroutinesApi
class AuthRepositoryTest {
companion object {
const val FAKE_ID_TOKEN = "FAkE_ID_TOKEN"
}
#get:Rule
val testCoroutineRule = TestCoroutineRule()
private val coroutineDispatcher = TestCoroutineDispatcher()
private val userDaoFake = spyk<UserDaoFake>()
private val mockApiService = mockk<MyApi>()
private val sut = AuthRepository(
userDaoFake, mockApiService, coroutineDispatcher
)
#Before
fun beforeEachTest() {
userDaoFake.clear()
}
#Test
fun getAuthToken_noCachedData_shouldMakeNetworkCallAndStoreUserInDatabase() = testCoroutineRule.runBlockingTest {
// Given an empty database
coEvery { mockApiService.getUser(any()) } returns NetworkResponse.Success(UserFakes.getNetworkUser(), null, HttpURLConnection.HTTP_OK)
// When getAuthToken is called
val result = sut.getAuthToken(FAKE_ID_TOKEN).toList()
coVerifyOrder {
// Then first try to fetch data from the DB
userDaoFake.get()
// Then fetch the User from the API
mockApiService.getUser(FAKE_ID_TOKEN)
// Then insert the user into the DB
userDaoFake.insert(any())
// Finally return the inserted user from the DB
userDaoFake.get()
}
assertThat(result).containsExactly(
Resource.loading(null),
Resource.success(UserFakes.getAppUser())
).inOrder()
}
}

Unit Test for retrofit2.Call

Since Retrofit 2.6.0 has support for Coroutines and the library of kotlinCoroutinesAdapter is deprecated I started to refactor my API calls. I changed it and is working fine but I have problems with Unit Tests.
This is how I had my calls:
//Retrofit API
#GET("playlists/{playlistId}")
fun getPlaylistAsync(#Path("playlistId") id: String): Deferred<PlaylistData>
//DataSource
fun getPlaylistAsync(playlistId: String): Deferred<PlaylistData> = playlistApi.getPlaylistAsync(playlistId)
//Repository
override suspend fun getPlaylist(playlistId: String): PlaylistDomain {
val playlistDeferred = remoteDataSource.getPlaylistAsync(playlistId)
return playlistRemoteMapper.map(playlistDeferred.await())
}
And this is what I changed to make it work:
//Retrofit API
#GET("playlists/{playlistId}")
fun getPlaylistAsync(#Path("playlistId") id: String): Call<PlaylistData>
//DataSource
fun getPlaylistAsync(playlistId: String): Call<PlaylistData> = playlistApi.getPlaylistAsync(playlistId)
//Repository
override suspend fun getPlaylist(playlistId: String): PlaylistDomain {
val playlistDeferred = remoteDataSource.getPlaylistAsync(playlistId)
return playlistRemoteMapper.map(playlistDeferred.await())
}
Basically is the same... the await() used with Deferred and with Call is different but everything else is so similar.
As I said the problems comes with the test. Here I have what I did for the first example:
#Test
fun `getPlaylist() maps correctly the playlist and the favorite request when is favorite`() = runBlocking {
val playlist = PlaylistDataBuilder().getPlaylistData(id = PLAYLIST_ID)
every { playlistDataSource.getPlaylistAsync(PLAYLIST_ID) } answers { async { playlist } }
val response = playlistRepository.getPlaylist(PLAYLIST_ID)
assertTrue(response.isFavorite)
}
Now I want to change this to fit the second example... but async returns a Deferred and now I need Call to fit my needs. I tried to mock Call and do the following:
#Test
fun `getPlaylist() maps correctly the playlist and the favorite request when is favorite`() = runBlocking {
val playlist = PlaylistDataBuilder().getPlaylistData(id = PLAYLIST_ID)
every { playlistCall.await() }
every { playlistDataSource.getPlaylistAsync(PLAYLIST_ID) } answers { playlistCall }
val response = playlistRepository.getPlaylist(PLAYLIST_ID)
assertTrue(response.isFavorite)
}
The problem here is that playlistCall.await() must be into a suspend method so I cannot make it work this way. I would need a way to make a Unit Test for retrofit Call.

How to use Fuel with coroutines in Kotlin?

I want to get an API request and save request's data to a DB. Also want to return the data (that is written to DB). I know, this is possible in RxJava, but now I write in Kotlin coroutines, currently use Fuel instead of Retrofit (but a difference is not so large). I read How to use Fuel with a Kotlin coroutine, but don't understand it.
How to write a coroutine and methods?
UPDATE
Say, we have a Java and Retrofit, RxJava. Then we can write a code.
RegionResponse:
#AutoValue
public abstract class RegionResponse {
#SerializedName("id")
public abstract Integer id;
#SerializedName("name")
public abstract String name;
#SerializedName("countryId")
public abstract Integer countryId();
public static RegionResponse create(int id, String name, int countryId) {
....
}
...
}
Region:
data class Region(
val id: Int,
val name: String,
val countryId: Int)
Network:
public Single<List<RegionResponse>> getRegions() {
return api.getRegions();
// #GET("/regions")
// Single<List<RegionResponse>> getRegions();
}
RegionRepository:
fun getRegion(countryId: Int): Single<Region> {
val dbSource = db.getRegion(countryId)
val lazyApiSource = Single.defer { api.regions }
.flattenAsFlowable { it }
.map { apiMapper.map(it) }
.toList()
.doOnSuccess { db.updateRegions(it) }
.flattenAsFlowable { it }
.filter({ it.countryId == countryId })
.singleOrError()
return dbSource
.map { dbMapper.map(it) }
.switchIfEmpty(lazyApiSource)
}
RegionInteractor:
class RegionInteractor(
private val repo: RegionRepository,
private val prefsRepository: PrefsRepository) {
fun getRegion(): Single<Region> {
return Single.fromCallable { prefsRepository.countryId }
.flatMap { repo.getRegion(it) }
.subscribeOn(Schedulers.io())
}
}

Let's look at it layer by layer.
First, your RegionResponse and Region are totally fine for this use case, as far as I can see, so we won't touch them at all.
Your network layer is written in Java, so we'll assume it always expects synchronous behavior, and won't touch it either.
So, we start with the repo:
fun getRegion(countryId: Int) = async {
val regionFromDb = db.getRegion(countryId)
if (regionFromDb == null) {
return apiMapper.map(api.regions).
filter({ it.countryId == countryId }).
first().
also {
db.updateRegions(it)
}
}
return dbMapper.map(regionFromDb)
}
Remember that I don't have your code, so maybe the details will differ a bit. But the general idea with coroutines, is that you launch them with async() in case they need to return the result, and then write your code as if you were in the perfect world where you don't need to concern yourself with concurrency.
Now to the interactor:
class RegionInteractor(
private val repo: RegionRepository,
private val prefsRepository: PrefsRepository) {
fun getRegion() = withContext(Schedulers.io().asCoroutineDispatcher()) {
val countryId = prefsRepository.countryId
return repo.getRegion(countryId).await()
}
}
You need something to convert from asynchronous code back to synchronous one. And for that you need some kind of thread pool to execute on. Here we use thread pool from Rx, but if you want to use some other pool, so do.

After researching How to use Fuel with a Kotlin coroutine, Fuel coroutines and https://github.com/kittinunf/Fuel/ (looked for awaitStringResponse), I made another solution. Assume that you have Kotlin 1.3 with coroutines 1.0.0 and Fuel 1.16.0.
We have to avoid asynhronous requests with callbacks and make synchronous (every request in it's coroutine). Say, we want to show a country name by it's code.
// POST-request to a server with country id.
fun getCountry(countryId: Int): Request =
"map/country/"
.httpPost(listOf("country_id" to countryId))
.addJsonHeader()
// Adding headers to the request, if needed.
private fun Request.addJsonHeader(): Request =
header("Content-Type" to "application/json",
"Accept" to "application/json")
It gives a JSON:
{
"country": {
"name": "France"
}
}
To decode the JSON response we have to write a model class:
data class CountryResponse(
val country: Country,
val errors: ErrorsResponse?
) {
data class Country(
val name: String
)
// If the server prints errors.
data class ErrorsResponse(val message: String?)
// Needed for awaitObjectResponse, awaitObject, etc.
class Deserializer : ResponseDeserializable<CountryResponse> {
override fun deserialize(content: String) =
Gson().fromJson(content, CountryResponse::class.java)
}
}
Then we should create a UseCase or Interactor to receive a result synchronously:
suspend fun getCountry(countryId: Int): Result<CountryResponse, FuelError> =
api.getCountry(countryId).awaitObjectResponse(CountryResponse.Deserializer()).third
I use third to access response data. But if you wish to check for a HTTP error code != 200, remove third and later get all three variables (as Triple variable).
Now you can write a method to print the country name.
private fun showLocation(
useCase: UseCaseImpl,
countryId: Int,
regionId: Int,
cityId: Int
) {
GlobalScope.launch(Dispatchers.IO) {
// Titles of country, region, city.
var country: String? = null
var region: String? = null
var city: String? = null
val countryTask = GlobalScope.async {
val result = useCase.getCountry(countryId)
// Receive a name of the country if it exists.
result.fold({ response -> country = response.country.name }
, { fuelError -> fuelError.message })
}
}
val regionTask = GlobalScope.async {
val result = useCase.getRegion(regionId)
result.fold({ response -> region = response.region?.name }
, { fuelError -> fuelError.message })
}
val cityTask = GlobalScope.async {
val result = useCase.getCity(cityId)
result.fold({ response -> city = response.city?.name }
, { fuelError -> fuelError.message })
}
// Wait for three requests to execute.
countryTask.await()
regionTask.await()
cityTask.await()
// Now update UI.
GlobalScope.launch(Dispatchers.Main) {
updateLocation(country, region, city)
}
}
}
In build.gradle:
ext {
fuelVersion = "1.16.0"
}
dependencies {
...
implementation 'org.jetbrains.kotlinx:kotlinx-coroutines-android:1.0.0'
// Fuel.
//for JVM
implementation "com.github.kittinunf.fuel:fuel:${fuelVersion}"
//for Android
implementation "com.github.kittinunf.fuel:fuel-android:${fuelVersion}"
//for Gson support
implementation "com.github.kittinunf.fuel:fuel-gson:${fuelVersion}"
//for Coroutines
implementation "com.github.kittinunf.fuel:fuel-coroutines:${fuelVersion}"
// Gson.
implementation 'com.google.code.gson:gson:2.8.5'
}
If you want to work with coroutines and Retrofit, please, read https://medium.com/exploring-android/android-networking-with-coroutines-and-retrofit-a2f20dd40a83 (or https://habr.com/post/428994/ in Russian).

You should be able to significantly simplify your code. Declare your use case similar to the following:
class UseCaseImpl {
suspend fun getCountry(countryId: Int): Country =
api.getCountry(countryId).awaitObject(CountryResponse.Deserializer()).country
suspend fun getRegion(regionId: Int): Region =
api.getRegion(regionId).awaitObject(RegionResponse.Deserializer()).region
suspend fun getCity(countryId: Int): City=
api.getCity(countryId).awaitObject(CityResponse.Deserializer()).city
}
Now you can write your showLocation function like this:
private fun showLocation(
useCase: UseCaseImpl,
countryId: Int,
regionId: Int,
cityId: Int
) {
GlobalScope.launch(Dispatchers.Main) {
val countryTask = async { useCase.getCountry(countryId) }
val regionTask = async { useCase.getRegion(regionId) }
val cityTask = async { useCase.getCity(cityId) }
updateLocation(countryTask.await(), regionTask.await(), cityTask.await())
}
}
You have no need to launch in the IO dispatcher because your network requests are non-blocking.
I must also note that you shouldn't launch in the GlobalScope. Define a proper coroutine scope that aligns its lifetime with the lifetime of the Android activity or whatever else its parent is.

java.lang.RuntimeException: Failed to invoke public android.arch.lifecycle.LiveData() with no args

I'm trying to call API using Retrofit and Android architecture components but I'm getting this error
java.lang.RuntimeException: Failed to invoke public android.arch.lifecycle.LiveData() with no args
this is the data class for the response
data class ForecastResult(val city: City, val list: List<Forecast>)
Services Interface
interface ServicesApis { // using dummy data
#GET("data/2.5/forecast/")
fun getForecast(#Query("APPID") APPID: String = "xxxxxxxx"
, #Query("q") q: String = "94043", #Query("mode") mode: String = "json", #Query("units") units: String = "metric"
, #Query("cnt") cnt: String = "7"): Call<LiveData<ForecastResult>>
}
and the api implementation
class WeatherRepoImpl : WeatherRepo {
override fun getDailyForecast(): LiveData<Resource<ForecastResult>> {
val forecast: MutableLiveData<Resource<ForecastResult>> = MutableLiveData()
RestAPI.getAPIsrevice().getForecast().enqueue(object : Callback<LiveData<ForecastResult>> {
override fun onResponse(call: Call<LiveData<ForecastResult>>?, response: Response<LiveData<ForecastResult>>?) {
when {
response!!.isSuccessful -> {
forecast.postValue(Resource.success(response.body()?.value))
}
else -> {
val exception = AppException(responseBody = response.errorBody())
forecast.postValue(Resource.error(exception))
}
}
}
override fun onFailure(call: Call<LiveData<ForecastResult>>?, t: Throwable?) {
val exception = AppException(t)
forecast.postValue(Resource.error(exception))
}
})
return forecast
}
}
appreciate your help!

Remove LiveData from API call and create ViewModel class have contained in MutableLiveData Object.
For Example:
API Call Definition something like this: (Remove LiveData)
#GET("data/2.5/forecast/")
fun getForecast(#Query("APPID") APPID: String = "xxxxxxxx"
, #Query("q") q: String = "94043", #Query("mode") mode: String = "json", #Query("units") units: String = "metric"
, #Query("cnt") cnt: String = "7"): Call<ForecastResult>
Create One ViewModel Class:
class YourViewModel: ViewModel() {
var allObjLiveData = MutableLiveData<YOUROBJECT>()
fun methodForAPICall(){
mApiService?.getForecast(.....)?.enqueue(object : Callback<YOUROBJECT>
{
override fun onFailure(call: Call<YOUROBJECT>, t: Throwable) {
allObjLiveData.value = null
}
override fun onResponse(call: Call<YOUROBJECT>, response: Response<YOUROBJECT>) {
allObjLiveData.value=response.body() //Set your Object to live Data
}
})
}
}
Now in your activity or fragment initialization ViewModel class.
And observe to live data.
yourViewModelClassObj?.allObjLiveData?.observe(this, Observer {
//whenever object change this method call every time.
}
})
So You can use livedata in ViewModel class.

Yes so probably your API's response is not being correctly serialized.
Anyway, it doesn't make sense to wrap your LiveData to an API response. Just have your exposed object like so:
someLivedataObject: LiveData<YourApiResponseModel>
// So whenever the APIs response comes in:
someLivedataObject.postValue(YourApiResponseModel)
If for some reason doesn't work use
someLivedataObject.setValue(YourApiResponseModel)
You can read more about the difference between these on the documentation, notice that I'm calling the LiveData#setValue() method, do not use Kotlin's deference setter method.
And since your view is observing changes to your exposed someLivedataObject the UI is updated.
This is similar to the RxJava counterpart where API responses with Observable wrapper don't really make sense, it's not a continuous stream of data, hence makes more sense to use Single.
Take these suggestions with a grain of salt, I'm not fully aware of your application's use cases and there are exceptions to these rules.

I use LiveDataAdapter to convert Call to LiveData, that way you don't need wrap your Retrofit response as Call<LiveData<T>>.
This adapter is similar to Rx adapter by Jake.
Get the LiveDataAdapterFactory and LiveDataAdapter from here.
After adding the Adapter, you'll need to set it:
Retrofit.Builder()
.addConverterFactory(GsonConverterFactory.create(gson))
.addCallAdapterFactory(LiveDataCallAdapterFactory())
.baseUrl(BASE_URL)
.client(httpClient)
.build()
Self Promotion: I have made a video on the same over at: https://caster.io/lessons/android-architecture-components-livedata-with-retrofit