I am new with kotlin flow and I am working about this document. Kotlin Flows. In this code every five seconds datasource fetch data from api and emits it.
This is my example datasource class.
I am getting data and emitting it.
class RemoteDataSourceImpl #Inject constructor(
private val api:CryptoApi
): RemoteDataSource {
override suspend fun cryptoList(): Flow<List<CryptoCoinDto>> {
return flow {
while (true){
val data = api.getCoinList()
emit(data)
delay(5000L)
}
}
}
}
This is my example repository.
I am mapping data and saving it room database. I want to get data from room database and emit it because of single source of truth principle but I still have to return dataSource because if I open new flow{} I can't reach datasource's data. Of course I can fix the problem by using List instead of Flow<List> inside of RemoteDataSource class. But I want to understand this example. How can I apply here single source of truth.
class CoinRepositoryImpl #Inject constructor(
private val dataSource:RemoteDataSource,
private val dao: CryptoDao
):CoinRepository {
override fun getDataList(): Flow<List<CryptoCoin>> {
dataSource.cryptoList().map { dtoList ->
val entityList = dtoList.map { dto ->
dto.toCryptoEntity()
}
dao.insertAll(entityList)
}
return dataSource.cryptoList().map {
it.map { it.toCryptoCoin() }
}
}
This is actually more complicated than it seems. Flows were designed to support back-pressure which means that they usually only produce items on demand, when being consumed. They are passive, instead of pushing items, items are pulled from the flow.
(Disclaimer: this is all true for cold flows, not for hot flows. But cryptoList() is a cold flow.)
It was designed this way to greatly simplify cases when the consumer is slower than producer or nobody is consuming items at all. Then producer just stops producing and everything is fine.
In your case there are two consumers, so this is again more complicated. You need to decide what should happen if one consumer is slower than the other. For example, what should happen if nobody collects data from getDataList()? There are multiple options, each requires a little different approach:
Stop consuming the source flow and therefore stop updating the database.
Update the database all the time and queue items if nobody is collecting from getDataList(). What if there are more and more items in the queue?
Update the database all the time and discard items if nobody is collecting from getDataList().
Ad.1.
It can be done by using onEach():
return dataSource.cryptoList().onEach {
// update db
}.map {
it.map { it.toCryptoCoin() }
}
In this solution updating the database is a "side effect" of consuming the getDataList() flow.
Ad.2. and Ad.3.
In this case we can't passively wait until someone asks us for an item. We need to actively consume items from the source flow and push them to the downstream flow. So we need a hot flow: SharedFlow. Also, because we remain the active side in this case, we have to launch a coroutine that will do this in the background. So we need a CoroutineScope.
Solution depends on your specific needs: do you need a queue or not, what should happen if queue exceeded the size limit, etc., but it will be similar to:
return dataSource.cryptoList().onEach {
// update db
}.map {
it.map { it.toCryptoCoin() }
}.shareIn(scope, SharingStarted.Eagerly)
You can also read about buffer() and MutableSharedFlow - they could be useful to you.
Related
Suppose you have a list of users downloaded from a remote data source in your Android application, and for some reason you do not have a local DB. This list of users is then used throughout your entire application in multiple ViewModels to make other network requests, so you would surely like to have it cached for as long as the app lives and re-fetch it only on demand. This necessarily means you want to cache it inside the Data Layer, which is a Repository in my case, to then get it from your ViewModels.
It is easy to do in a state holder like a ViewModel - just make a StateFlow or whatever. But what if we want a Flow of List<User> (that is cached in RAM after every API request) available inside a repository to then collect from it from the UI Layer? What is the most testable, stable and right way of achieving this?
My initial idea led to this:
class UsersRepository #Inject constructor(
private val usersApi: UsersApi,
private val handler: ResponseHandler
) {
private val _usersFlow = MutableStateFlow<Resource<List<UserResponse>>>(Resource.Empty)
val usersFlow = _usersFlow.asStateFlow()
suspend fun fetchUserList() = withContext(Dispatchers.IO) {
_usersFlow.emit(Resource.Loading)
_usersFlow.emit(
handler {
usersApi.getUsers()
}
)
}
}
Where ResponseHandler is:
class ResponseHandler {
suspend operator fun <T> invoke(block: suspend () -> T) = try {
Resource.Success(block())
} catch (e: Exception) {
Log.e(javaClass.name, e.toString())
val errorCode = when (e) {
is HttpException -> e.code()
is SocketTimeoutException -> ErrorCodes.SocketTimeOut.code
is UnknownHostException -> ErrorCodes.UnknownHost.code
else -> Int.MAX_VALUE
}
Resource.Error(getErrorMessage(errorCode))
}
}
But while researching I found random guy on the internet telling that it is wrong:
Currently StateFlow is hot in nature so it’s not recommended to use in repository. For cold and reactive stream, you can use flow, channelFlow or callbackFlow in repository.
Is he right? If he is, how exactly do cold flows help in this situation, and how do we properly manage them?
If it helps, my UI Layer is written solely with Jetpack Compose
In the official "Guide to app architecture" from Google for Android:
About the source of true: ✅ The repository can contain an in-memory-cache.
The source of truth can be a data source—for example, the database—or even an in-memory cache that the repository might contain. Repositories combine different data sources and solve any potential conflicts between the data sources to update the single source of truth regularly or due to a user input event.
About the lifecycle: ✅ You can scope an instance of your repository to the Application class (but take care).
If a class contains in-memory data—for example, a cache—you might want
to reuse the same instance of that class for a specific period of
time. This is also referred to as the lifecycle of the class instance.
If the class's responsibility is crucial for the whole application,
you can scope an instance of that class to the Application class. This
makes it so the instance follows the application's lifecycle.
About the implementation: I recommend you to check the link directly.
class NewsRepository(
private val newsRemoteDataSource: NewsRemoteDataSource
) {
// Mutex to make writes to cached values thread-safe.
private val latestNewsMutex = Mutex()
// Cache of the latest news got from the network.
private var latestNews: List<ArticleHeadline> = emptyList()
suspend fun getLatestNews(refresh: Boolean = false): List<ArticleHeadline> {
if (refresh || latestNews.isEmpty()) {
val networkResult = newsRemoteDataSource.fetchLatestNews()
// Thread-safe write to latestNews
latestNewsMutex.withLock {
this.latestNews = networkResult
}
}
return latestNewsMutex.withLock { this.latestNews }
}
}
You should read the following page, I think it will answer a lot of your questions : https://developer.android.com/topic/architecture/data-layer
To make this work as a cache you will have to use this repository as a singleton. This effectively create a huge memory leak since you have no control over this memory. You cannot free it, you cannot bypass cache if you want (i mean you can, but it requires additional code outside the flow), you don't have any control over eviction. It's very dumb cache which acts like a memory leak. Not worth it.
Cold flow don't "help" in caching per se. They just give you control over each request that comes from the client. There you can check some outside memory cache if the entry is cached. If yes - is it correct or should be evicted? If it is evicted you can just a normal request. And all this is a single flow that gets disposed right after, so no memory leaks. The only part that have to be singleton is the cache. Although you can implement it as disk cache, it will be faster than network anyway
I have a MediatorLiveData that uses three LiveData sources. When any of them emits a new value and I have at least one of each, I use the three values to produce the output for the UI.
Two of the sources are user settings for how to sort and filter a list, and the third is the list data, pulled from a Room database Flow.
It looks something like this:
val thingsLiveData: LiveData<List<Thing>> = object: MediatorLiveData<List<Thing>>() {
var isSettingA: Boolean = true
var settingB: MySortingEnum = MySortingEnum.Alphabetical
var data: List<Thing>? = null
init {
addSource(myRepo.thingsFlow.asLiveData()) {
data = it
dataToValue()
}
addSource(settingALiveData) {
isSettingA= it
dataToValue()
}
addSource(settingBLiveData) {
settingB= it
dataToValue()
}
}
private fun dataToValue() {
data?.let { data ->
viewModelScope.launch {
val uiList = withContext(Dispatchers.Default) {
produceUiList(data, isSettingA, settingB)
}
value = listItems
}
}
}
}
I'm looking for a clean way to convert this to a SharedFlow, preferably without any #ExperimentalCoroutinesApi. The only SharedFlow builder function I've come across is callbackFlow, which isn't applicable. Are you intended to use flow { ... }.asSharedFlow(...) in most cases, and if so, what would that look like here?
The two settings LiveData I also plan to migrate to flows.
The source Flows can be combined using combine(), which creates a cold Flow that, when collected, will start collecting from its source Flows, which may be hot or cold.
I was originally thinking that I must be missing something and that there should be some way to directly combine hot Flows into a combined hot Flow. But I realized it makes sense that the operators should only return cold Flows and leave it up to you to convert it back to a hot Flow if that's what you need.
In many cases, such as mine, it's perfectly fine to leave it cold. I only collect this Flow from one place in my UI, so it doesn't matter that it only starts combining the sources when it's collected. The source hot Flows don't care whether something is currently collecting them or not...they just keep emitting regardless.
If I collected this Flow from multiple places or multiple times, then it might make sense to use shareIn on the combined Flow to make it hot, which would avoid redundant work of combining the sources. The potential downside would be that it would combine those sources even when nothing is collecting, which would be wasted work.
val thingsFlow: Flow<List<Thing>> = combine(
myRepo.thingsFlow,
settingALiveData.asFlow(),
settingBLiveData.asFlow()
) { data, isSettingA, settingB -> produceUiList(data, isSettingA, settingB) }
// where produceUiList is now a suspend function that wraps
// blocking code using withContext
The company I just started working at uses a so called Navigator, which I for now interpreted as a stateless ViewModel. My Navigator receives some usecases, with each contains 1 suspend function. The result of any of those usecases could end up in a single LiveData. The Navigator has no coroutine scope, so I pass the responsibility of scoping suspending to the Fragment using fetchValue().
Most current code in project has LiveData in the data layer, which I tried not to. Because of that, their livedata is linked from view to dao.
My simplified classes:
class MyFeatureNavigator(
getUrl1: getUrl1UseCase,
getUrl1: getUrl1UseCase
) {
val url = MediatorLiveData<String>()
fun goToUrl1() {
url.fetchValue { getUrl1() }
}
fun goToUrl2() {
url.fetchValue { getUrl2() }
}
fun <T> MediatorLiveData<T>.fetchValue(provideValue: suspend () -> T) {
val liveData = liveData { emit(provideValue()) }
addSource(liveData) {
removeSource(liveData)
value = it
}
}
}
class MyFeatureFragment : Fragment {
val viewModel: MyFeatureViewModel by viewModel()
val navigator: MyFeatureNavigator by inject()
fun onViewCreated() {
button.setOnClickListener { navigator.goToUrl1() }
navigator.url.observe(viewLifecycleOwner, Observer { url ->
openUrl(url)
})
}
}
My two questions:
Is fetchValue() a good way to link a suspend function to LiveData? Could it leak? Any other concerns?
My main reason to only use coroutines (and flow) in the data layer, is 'because Google said so'. What's a better reason for this? And: what's the best trade off in being consistent with the project and current good coding practices?
Is fetchValue() a good way to link a suspend function to LiveData?
Could it leak? Any other concerns?
Generally it should work. You probably should remove the previous source of the MediatorLiveData before adding new one, otherwise if you get two calls to fetchValue in a row, the first url can be slower to fetch, so it will come later and win.
I don't see any other correctness concerns, but this code is pretty complicated, creates a couple of intermediate objects and generally difficult to read.
My main reason to only use coroutines (and flow) in the data layer,
is 'because Google said so'. What's a better reason for this?
Google has provided a lot of useful extensions to use coroutines in the UI layer, e.g. take a look at this page. So obviously they encourage people to use it.
Probably you mean the recommendation to use LiveData instead of the Flow in the UI layer. That's not a strict rule and it has one reason: LiveData is a value holder, it keeps its value and provides it immediately to new subscribers without doing any work. That's particularly useful in the UI/ViewModel layer - when a configuration change happens and activity/fragment is recreated, the newly created activity/fragment uses the same view model, subscribes to the same LiveData and receives the value at no cost.
At the same time Flow is 'cold' and if you expose a flow from your view model, each reconfiguration will trigger a new flow collection and the flow will be to execute from scratch.
So e.g. if you fetch data from db or network, LiveData will just provide the last value to new subscriber and Flow will execute the costly db/network operation again.
So as I said there is no strict rule, it depends on the particular use-case. Also I find it very useful to use Flow in view models - it provides a lot of operators and makes the code clean and concise. But than I convert it to a LiveData with help of extensions like asLiveData() and expose this LiveData to the UI. This way I get best from both words - LiveData catches value between reconfigurations and Flow makes the code of view models nice and clean.
Also you can use latest StateFlow and SharedFlow often they also can help to overcome the mentioned Flow issue in the UI layer.
Back to your code, I would implement it like this:
class MyFeatureNavigator(
getUrl1: getUrl1UseCase,
getUrl1: getUrl1UseCase
) {
private val currentUseCase = MutableStateFlow<UseCase?>(null)
val url = currentUseCase.filterNotNull().mapLatest { source -> source.getData()}.asLiveData()
fun goToUrl1() {
currentUseCase.value = getUrl1
}
fun goToUrl2() {
currentUseCase.value = getUrl2
}
}
This way there are no race conditions to care about and code is clean.
And: what's the best trade off in being consistent with the project
and current good coding practices?
That's an arguable question and it should be primarily team decision. In most projects I participated we adopted this rule: when fixing bugs, doing maintenance of existing code, one should follow the same style. When doing big refactoring/implementing new features one should use latest practices adopted by the team.
I'm trying out the new coroutine's flow, my goal is to make a simple repository that can fetch data from a web api and save it to db, also return a flow from the db.
I'm using room and firebase as the web api, now everything seems pretty straight forward until i try to pass errors coming from the api to the ui.
Since i get a flow from the database which only contains the data and no state, what is the correct approach to give it a state (like loading, content, error) by combining it with the web api result?
Some of the code i wrote:
The DAO:
#Query("SELECT * FROM users")
fun getUsers(): Flow<List<UserPojo>>
The Repository:
val users: Flow<List<UserPojo>> = userDao.getUsers()
The Api call:
override fun downloadUsers(filters: UserListFilters, onResult: (result: FailableWrapper<MutableList<UserApiPojo>>) -> Unit) {
val data = Gson().toJson(filters)
functions.getHttpsCallable("users").call(data).addOnSuccessListener {
try {
val type = object : TypeToken<List<UserApiPojo>>() {}.type
val users = Gson().fromJson<List<UserApiPojo>>(it.data.toString(), type)
onResult.invoke(FailableWrapper(users.toMutableList(), null))
} catch (e: java.lang.Exception) {
onResult.invoke(FailableWrapper(null, "Error parsing data"))
}
}.addOnFailureListener {
onResult(FailableWrapper(null, it.localizedMessage))
}
}
I hope the question is clear enough
Thanks for the help
Edit: Since the question wasn't clear i'll try to clarify. My issue is that with the default flow emitted by room you only have the data, so if i were to subscribe to the flow i would only receive the data (eg. In this case i would only receive a list of users). What i need to achieve is some way to notify the state of the app, like loading or error. At the moment the only way i can think of is a "response" object that contains the state, but i can't seem to find a way to implement it.
Something like:
fun getUsers(): Flow<Lce<List<UserPojo>>>{
emit(Loading())
downloadFromApi()
if(downloadSuccessful)
return flowFromDatabase
else
emit(Error(throwable))
}
But the obvious issue i'm running into is that the flow from the database is of type Flow<List<UserPojo>>, i don't know how to "enrich it" with the state editing the flow, without losing the subscription from the database and without running a new network call every time the db is updated (by doing it in a map transformation).
Hope it's clearer
I believe this is more of an architecture question, but let me try to answer some of your questions first.
My issue is that with the default flow emitted by room you only have
the data, so if i were to subscribe to the flow i would only receive
the data
If there is an error with the Flow returned by Room, you can handle it via catch()
What i need to achieve is some way to notify the state of the app,
like loading or error.
I agree with you that having a State object is a good approach. In my mind, it is the ViewModel's responsibility to present the State object to the View. This State object should have a way to expose errors.
At the moment the only way i can think of is a "response" object that
contains the state, but i can't seem to find a way to implement it.
I have found that it is easier to have the State object that the ViewModel controls be responsible for errors instead of an object that bubbles up from the Service layer.
Now with these questions out of the way, let me try to propose one particular "solution" to your issue.
As you mention, it is common practice to have a Repository that handles retrieving data from multiple data sources. In this case, the Repository would take the DAO and an object that represents getting data from the network, let's call it Api. I am assuming that you are using FirebaseFirestore, so the class and method signature would look something like this:
class Api(private val firestore: FirebaseFirestore) {
fun getUsers() : Flow<List<UserApiPojo>
}
Now the question becomes how to turn a callback based API into a Flow. Luckily, we can use callbackFlow() for this. Then Api becomes:
class Api(private val firestore: FirebaseFirestore) {
fun getUsers() : Flow<List<UserApiPojo> = callbackFlow {
val data = Gson().toJson(filters)
functions.getHttpsCallable("users").call(data).addOnSuccessListener {
try {
val type = object : TypeToken<List<UserApiPojo>>() {}.type
val users = Gson().fromJson<List<UserApiPojo>>(it.data.toString(), type)
offer(users.toMutableList())
} catch (e: java.lang.Exception) {
cancel(CancellationException("API Error", e))
}
}.addOnFailureListener {
cancel(CancellationException("Failure", e))
}
}
}
As you can see, callbackFlow allows us to cancel the flow when something goes wrong and have someone donwnstream handle the error.
Moving to the Repository we would now like to do something like:
val users: Flow<List<User>> = Flow.concat(userDao.getUsers().toUsers(), api.getUsers().toUsers()).first()
There are a few caveats here. first() and concat() are operators you will have to come up with it seems. I did not see a version of first() that returns a Flow; it is a terminal operator (Rx used to have a version of first() that returned an Observable, Dan Lew uses it in this post). Flow.concat() does not seem to exist either. The goal of users is to return a Flow that emits the first value emitted by any of the source Flows. Also, note that I am mapping DAO users and Api users to a common User object.
We can now talk about the ViewModel. As I said before, the ViewModel should have something that holds State. This State should represent data, errors and loading states. One way that can be accomplished is with a data class.
data class State(val users: List<User>, val loading: Boolean, val serverError: Boolean)
Since we have access to the Repository the ViewModel can look like:
val state = repo.users.map {users -> State(users, false, false)}.catch {emit(State(emptyList(), false, true)}
Please keep in mind that this is a rough explanation to point you in a direction, there are many ways to accomplish state management and this is by no means a complete implementation. It may not even make sense to turn the API call into a Flow, for example.
The answer from Emmanuel is really close to answering what i need, i need some clarifications about some of it.
It may not even make sense to turn the API call into a Flow
You are totally right, in fact i only want to actually make it a coroutine, i don't really need it to be a flow.
If there is an error with the Flow returned by Room, you can handle it via catch()
Yes i discovered this after posting the question. But my problem is more something like:
I'd like to call a method, say "getData", this method should return the flow from db, start the network call to update the db (so that i'm going to be notified when it's done via the db flow) and somewhere in here, i would need to let the ui know if db or network errored, right?. Or should i maybe do a separate "getDbFlow" and "updateData" and get the errors separately for each one?
val users: Flow> = Flow.concat(userDao.getUsers().toUsers(), api.getUsers().toUsers()).first()
This is a good idea, but i'd like to keep the db as the single source of truth, and never return to the ui any data directly from the network
In my application I'm using RxJava2 and new class from Architecture Components ViewModel. In my case, I need to push SQL clause to ViewModel, which will do some magic and return Observable that will give me the data I need. Everything works fine, but I am not sure if I am using RX in the best way.
My data flow:
ViewModel has PublishSubject on which I am pushing SQL's. ViewModel has also Observable which is created by mapping subject. Also, I used distinctUntilChanged on Subject, to prevent from executing the same query again.
To cache data I used replay(1).autoconnect(1) on Observable, but that approach had a flaw. Sometimes my Subject pushed Sql when Observable wasn't yet connect, and my data never arrived to me. Should I use BehaviourSubject? Or maybe I shouldn't use replay(1).autoconnect(1) in the first place? Or maybe my whole flow is wrong? Example:
val listSubject: Subject<RawSql> = PublishSubject.create()
val sqlListEmitter: Observable<List<T>> =
listSubject
.subscribeOn(Schedulers.computation())
.map { // SOME MAGIC HERE }
.replay(1).autoConnect(1, { compositeDisposable.add(it) })
In your case autoConnect() just waits for the first subscription to connect() to your stream. Since your subject and your stream build an inherent entity, you might not want to wait for it at all and instead connect it directly.
val listSubject: Subject<RawSql> = PublishSubject.create()
val sqlListEmitter: Observable<List<T>> =
listSubject
.observeOn(Schedulers.computation())
.map { // SOME MAGIC HERE }
.replay(1)
.let {
it.connect(compositeDisposable::add)
it.publish()
}
Also you might need to change subscribeOn() to observeOn(). The subject emits on the same thread as the data is pushed to it and does not consider the thread it's subscribed on.