I need to make a small app that needs to parse from a raw JSON file around 200K cities.
I can't use any DB, but the user needs to be able to perform a city search.
The problem with the approach is that parsing such a big JSON file takes a long time, but I can't think of any other approach. I have a couple of requirements:
Optimise for fast searches. Loading time of the app is not so important.
Time efficiency for filter algorithm should be better than linear.
The list can be preprocessed into any other representation that you consider more efficient for searches and display. Provide information of why that representation is more efficient in the comments of the code.
In my repository I have something like:
class CitiesRepository(private val application: Application) {
fun getCities(): List<City> {
val bufferReader = application.assets.open("cities.json").bufferedReader()
val data = bufferReader.use {
it.readText()
}
val gson = GsonBuilder().create()
val type: Type = object : TypeToken<ArrayList<City?>?>() {}.type
return gson.fromJson(data, type)
}
}
and my ViewModel consists of:
class CitiesListViewModel(val app: Application) : AndroidViewModel(app) {
private val repository: CitiesRepository = CitiesRepository(app)
val allCities: LiveData<List<City>> =
liveData(context = viewModelScope.coroutineContext + Dispatchers.IO) {
val cities = repository.getCities()
withContext(Dispatchers.Main) {
emit(cities)
}
}
}
Is there anything else (better) I can do?
Thanks in advance!
Related
I have response like this :
{
"response":{"numFound":5303,"start":0,"maxScore":6.5102634,"docs":[
{
"id":"10.1371/journal.pone.0000290",
"journal":"PLoS ONE",
"eissn":"1932-6203",
"publication_date":"2007-03-14T00:00:00Z",
"article_type":"Research Article",
"author_display":["Rayna I. Kraeva",
"Dragomir B. Krastev",
"Assen Roguev",
"Anna Ivanova",
"Marina N. Nedelcheva-Veleva",
"Stoyno S. Stoynov"],
"abstract":["Nucleic acids, due to their structural and chemical properties, can form double-stranded secondary structures that assist the transfer of genetic information and can modulate gene expression. However, the nucleotide sequence alone is insufficient in explaining phenomena like intron-exon recognition during RNA processing. This raises the question whether nucleic acids are endowed with other attributes that can contribute to their biological functions. In this work, we present a calculation of thermodynamic stability of DNA/DNA and mRNA/DNA duplexes across the genomes of four species in the genus Saccharomyces by nearest-neighbor method. The results show that coding regions are more thermodynamically stable than introns, 3′-untranslated regions and intergenic sequences. Furthermore, open reading frames have more stable sense mRNA/DNA duplexes than the potential antisense duplexes, a property that can aid gene discovery. The lower stability of the DNA/DNA and mRNA/DNA duplexes of 3′-untranslated regions and the higher stability of genes correlates with increased mRNA level. These results suggest that the thermodynamic stability of DNA/DNA and mRNA/DNA duplexes affects mRNA transcription."],
"title_display":"Stability of mRNA/DNA and DNA/DNA Duplexes Affects mRNA Transcription",
"score":6.5102634},
Now in this I want to get the 'abstract' field. For this I had specified it as String but it gave me error that it the array and can not convert to string.
Now I am not sure how to create object for this which array type I should specify.
I checked that we can use the Type Converters but not able to write the converter for the same.
Following is my object and converter which I tried.
DAO
#Entity(tableName = "news_table")
data class NewsArticles(
#PrimaryKey var id: String = "",
#SerializedName("article_type") var title: String? = null,
#SerializedName("abstract") var description: Array<String>,
#SerializedName("publication_date") var publishedAt: String? = null
)
Type Converter
class Converters {
#TypeConverter
fun fromTimestamp(value: Array<String>?): String? {
return value?.let { String(it) } //error
}
#TypeConverter
fun dateToTimestamp(array: Array<String>): String? {
return array.toString()
}
}
Its giving me error for return line that none of the following functions can be called with arguments supplied.
EDIT :
now I changed defination to ArrayList
#SerializedName("abstract") var description: ArrayList,
and converter to this
class ArrayConverters {
#TypeConverter
fun fromArray(value: ArrayList<String>?): String? {
return value?.let { arrayToString(it) }
}
#TypeConverter
fun arrayToString(array: ArrayList<String>): String? {
return array.toString()
}
}
Now its showing this error : error: Multiple methods define the same conversion. Conflicts with these: CustomTypeConverter
Please help. Thank you.
EDIT 2:
As per answer of richard slond, I have added the converter as
class ArrayConverters {
#TypeConverter
fun to(array: Array<String>): String {
return array.joinToString(" ")
}
#TypeConverter
fun from(value: String): List<String> {
return value.split(" ")
}
}
and added in the database as
#Database(entities = [NewsArticles::class], version = 2, exportSchema = false)
#TypeConverters(ArrayConverters::class)
abstract class AppDatabase : RoomDatabase() {
abstract fun newsArticlesDao(): NewsArticlesDao
}
Also in the news article module
#Entity(tableName = "news_table")
#TypeConverters(ArrayConverters::class)
data class NewsArticles(
#PrimaryKey var id: String = "",
#SerializedName("article_type") var title: String? = null,
#SerializedName("abstract") var description: String? = null,
#SerializedName("publication_date") var publishedAt: String? = null
)
Here for descriptionn variable if i have added string I am getting error as the field is begin with array.
and if i have specified as the arraylist it gives the error as can not add this type to the database please try using type converter.
What's missing??
The easiest way to store Collection (like Array, List) data into database is to convert them to String In JSON format, and GSON library (developed by Google) is designed for this situation.
How to Use:
String jsonString;
toJsonButton.setOnClickListener(new View.OnClickListener() {
#Override
public void onClick(View view) {
Gson gson = new Gson();
jsonString = gson.toJson(student); //object -> json
}
});
toObjectButton.setOnClickListener(new View.OnClickListener() {
#Override
public void onClick(View view) {
Gson gson = new Gson();
Student currentStudent = gson.fromJson(jsonString, Student.class); //json -> object
}
});
Reminder: try to put your Collection into an Object (as a member variable), otherwise you may need following extra works:
Get the Type for Your Collection:
TypeToken<List<Integer>> typeToken = new TypeToken<List<Integer>>(){};
List<Integer> retrievedNumbers = gson.fromJson(numbersJsonString, typeToken.getType());
If you really want to do that, you need to find a way to represent an array of strings using a primitive type. The easiest way is to use JSON format. For that reason, in your converter, you need to serialize and deserialize your string array.
As quick solution (which I do not recommend) is the following:
#TypeConverter
fun to(array: Array<String>): String {
return array.joinToString(",")
}
#TypeConverter
fun from(value:String): Array<String> {
return value.split(",")
}
Please be aware, following this path, your strings cannot include commas - but you can use another not so common character as separator
I have a ViewModel that has a MutableLiveData of an arraylist of class Course
private var coursesList: MutableLiveData<ArrayList<Course>> = MutableLiveData()
This coursesList is filled with data got from an API (by Retrofit): coursesList.postValue(response.body())
Now, a user can search for a course by its name. The function that I have for searching is that I iterate through the elements of the coursesList and check if its name is equal to what a user typed. It returns an arrayList with the courses that start with the name typed (this list is later sent to a fragment which passes it to an adapter to be shown in a recyclerview):
fun getCoursesList(): MutableLiveData<ArrayList<Course>> {
return coursesList
}
fun searchCourses(searchString: String): ArrayList<Course> {
val resultsList: ArrayList<Course> = ArrayList()
if (getCoursesList().value == null) return resultsList
if (getCoursesList().value!!.size > 0) {
for (course in getCoursesList().value!!.iterator()) {
if (course.name.toLowerCase(Locale.ROOT).startsWith(searchString)) {
resultsList.add(course)
}
}
}
resultsList.sortBy { it.price }
return resultsList
}
This function works and all but my instructor asked me to use LiveData for searching without giving any additional hints on how to do that.
So my question is how to use LiveData for searching? I tried to search for answers, I saw that some used LiveDataTransformations.switchMap but they were all using RoomDAOs and I couldn't adapt it to the code that I have.
Any help would be appreciated very much. Thanks in advance.
Maybe that can help you a little bit,
class YourViewModel(
private val courcesRepository: CourcesRepository
) : ViewModel() {
// Private access - mutableLiveData!
private val _coursesList = MutableLiveData<ArrayList<Course>>()
// Public access - immutableLiveData
val coursesList: LiveData<ArrayList<Course>>
get() = _coursesList
init {
// mutableLiveData initialize, automatic is immutable also initialize
_coursesList.postValue(getCourses())
}
// Here you get your data from repository
private fun getCourses(): ArrayList<Course> {
return courcesRepository.getCources()
}
// Search function
fun searchCourses(searchString: String) {
// you hold your data into this methode
val list: ArrayList<Course> = getCources()
if (searchString.isEmpty()) {
// here you reset the data if search string is empty
_coursesList.postValue(list)
} else {
// here you can search the list and post the new one to your LiveData
val filterList = list.filter {
it.name.toLowerCase(Locale.ROOT).startsWith(searchString)
}
filterList.sortedBy { it.price }
_coursesList.postValue(filterList)
}
}
}
The first tip is you should use LiveData like below, that is also recommended from google's jet pack team. The reason is so you can encapsulate the LivaData.
The second tip is you should use kotlin's idiomatic way to filter a list. Your code is readable and faster.
At least is a good idea to make a repository class to separate the concerns in your app.
And some useful links for you:
https://developer.android.com/jetpack/guide
https://developer.android.com/topic/libraries/architecture/livedata
I hope that's helpful for you
Ii is hard to guess the desired outcome, but a possible solution is to use live data for searched string also. And then combine them with coursesList live data into live data for searched courses, like this for example.
val searchStringLiveData: MutableLiveData<String> = MutableLiveData()
val coursesListLiveData: MutableLiveData<ArrayList<Course>> = MutableLiveData()
val searchedCourses: MediatorLiveData<ArrayList<Course>> = MediatorLiveData()
init {
searchedCourses.addSource(searchStringLiveData) {
searchedCourses.value = combineLiveData(searchStringLiveData, coursesListLiveData)
}
searchedCourses.addSource(coursesListLiveData) {
searchedCourses.value = combineLiveData(searchStringLiveData, coursesListLiveData)
}
}
fun combineLiveData(searchStringLiveData: LiveData<String>, coursesListLiveData: LiveData<ArrayList<Course>> ): ArrayList<Course> {
// your logic here to filter courses
return ArrayList()
}
I haven't run the code so I am not 100% sure that it works, but the idea is that every time either of the two live data changes value, searched string or courses, the combine function is executed and the result is set as value of the searchedCourses mediator live data. Also I omitted the logic of the filtering for simplicity.
I am writing a simple Android app in Kotlin, which will show prayers divided into categories to user. There are 5 JSON files in assets folder, each of them has just around 10 KiB.
I use Klaxon for parsing the JSON files into those two data classes:
data class Prayer(val prayerName: String, val verseTitle: String, val verseBody: String,
val prayerLine: String, val prayerBody: String, val prayerEnding: String)
data class PrayerCategory(val title: String, val bgImage: String, val headerImage: String,
val prayers : List<Prayer>)
Here is the code I use for parsing the prayers:
private fun loadPrayerNames(jsonFile: String) {
val millis = measureTimeMillis {
val input = assets.open("${jsonFile}.json")
val prayerCategory = Klaxon().parse<PrayerCategory>(input)
if (prayerCategory != null) {
for (prayer in prayerCategory.prayers) {
val prayerName = prayer.prayerName
prayersMap[prayerName] = prayer
}
}
}
println("Loading prayer category took $millis ms.")
}
As you can see, there is just one access to assets. No assets.list(), no bullshit.
And as you noticed, I have measured the time.. make your guesses.. Here is the debug output:
Loading prayer category took 3427 ms.
Yup, that's right. Loading and parsing 10KiB big JSON took 3.5 SECONDS! I repeat. No rocket science involved. Just parsing 10 KiB JSON. 3.5 seconds..... Hmm..
Btw, I am testing it on Nokia 6.1, which is a pretty snappy phone.
So.. my questions:
What causes this behaviour?
Is there any way how to speed this up other than building a database for storing about 50 prayers?
I will be very thankful for your help!
Android assets seem to have bad reputation when it comes to performance. However, my testing proved that in this situation, it was Klaxon library who was responsible.
After finding major performance problem in Klaxon (see https://github.com/cbeust/klaxon/issues/154), which is still not fixed, I have tried recommended alternative: Moshi (https://github.com/square/moshi).
Moshi did improve performance, but parsing my JSON still took about 1 second.
After these experiments, I have resorted to the good old fashioned parsing using JSONObject:
data class Prayer(val prayerName: String, val verseTitle: String, val verseBody: String,
val prayerLine: String, val prayerBody: String, val prayerEnding: String) {
companion object {
fun parseJson(json: JSONObject) : Prayer = Prayer(json.getString("prayerName"),
json.getString("verseTitle"), json.getString("verseBody"),
json.getString("prayerLine"), json.getString("prayerBody"),
json.getString("prayerEnding"))
}
}
data class PrayerCategory(val title: String, val bgImage: String, val headerImage: String,
val prayers : List<Prayer>) {
companion object {
fun parseJson(json: JSONObject): PrayerCategory {
val prayers = ArrayList<Prayer>()
val prayersArray = json.getJSONArray("prayers")
for(i in 0 until prayersArray.length()) {
prayers.add(Prayer.parseJson(prayersArray.getJSONObject(i)))
}
return PrayerCategory(json.getString("title"), json.getString("bgImage"),
json.getString("headerImage"), prayers)
}
}
}
This has reduced parsing time from 3427 ms to 13ms. Case closed. ;-)
I am using Moshi to deserialize json from our server but I have come across an issue I’m sure has a solution, I just can’t see it. Over the socket, we are send json that, at the top level, has three fields:
{
"data_type": "<actual_data_type>",
"data_id": "<actual_data_id>",
"data": <data_object>
}
The issue is that the data can actually be several different objects based on what data_type is can I’m not sure how to pass that information into the adaptor for Data. I’ve tried a couple different things, but it just gets closer and closer to me parsing the whole thing myself, which seems to defeat the point. Is there a way to pass information from one adaptor to another?
For anyone who wants to do something similar, I took the basic shape of a generic factory from here: https://github.com/square/moshi/pull/264/files (which also what #eric cochran is recommending in his comment) and made it more specific to fit my exact case.
class EventResponseAdapterFactory : JsonAdapter.Factory {
private val labelKey = "data_type"
private val subtypeToLabel = hashMapOf<String, Class<out BaseData>>(
DataType.CURRENT_POWER.toString() to CurrentPower::class.java,
DataType.DEVICE_STATUS_CHANGED.toString() to DeviceStatus::class.java,
DataType.EPISODE_EVENT.toString() to EpisodeEvent::class.java,
DataType.APPLIANCE_INSTANCE_UPDATED.toString() to ApplianceInstanceUpdated::class.java,
DataType.RECURRING_PATTERNS.toString() to RecurringPatternOccurrence::class.java,
DataType.RECURRING_PATTERN_UPDATED.toString() to RecurringPatternUpdated::class.java
)
override fun create(type: Type, annotations: Set<Annotation>, moshi: Moshi): JsonAdapter<*>? {
if (!annotations.isEmpty() || type != EventResponse::class.java) {
return null
}
val size = subtypeToLabel.size
val labelToDelegate = LinkedHashMap<String, JsonAdapter<EventResponse<BaseData>>>(size)
for (entry in subtypeToLabel.entries) {
val key = entry.key
val value = entry.value
val parameterizedType = Types.newParameterizedType(EventResponse::class.java, value)
val delegate = moshi.adapter<EventResponse<BaseData>>(parameterizedType, annotations)
labelToDelegate.put(key, delegate)
}
return EventResponseAdapter(
labelKey,
labelToDelegate
)
}
private class EventResponseAdapter internal constructor(
private val labelKey: String,
private val labelToDelegate: LinkedHashMap<String, JsonAdapter<EventResponse<BaseData>>>
) : JsonAdapter<EventResponse<BaseData>>() {
override fun fromJson(reader: JsonReader): EventResponse<BaseData>? {
val raw = reader.readJsonValue()
if (raw !is Map<*, *>) {
throw JsonDataException("Value must be a JSON object but had a value of $raw of type ${raw?.javaClass}")
}
val label = raw.get(labelKey) ?: throw JsonDataException("Missing label for $labelKey")
if (label !is String) {
throw JsonDataException("Label for $labelKey must be a string but had a value of $label of type ${label.javaClass}")
}
val delegate = labelToDelegate[label] ?: return null
return delegate.fromJsonValue(raw)
}
// Not used
override fun toJson(writer: JsonWriter, value: EventResponse<BaseData>?) {}
}
}
The only thing to watch out for is that the RuntimeJsonAdapterFactory in the link uses Types.getRawType(type) to get the type with the generics stripped away. We, of course, don't want that because once the specific generic type has been found, we want the normal Moshi adapters to kick in and do the proper parsing for us.
I've been playing around with Kotlin for Android. I have a mutable list which is a list of objects. Now I want to persist them, but I don't know what's the best way to do it. I think it can be done with SharedPreferences but I don't know how to parse the objects to a plain format or something. The objects are actually coming from a data class, maybe that can be useful.
Thanks
It is very easy to persist any data within SharedPreferences. All you need to do is get Gson implementation 'com.squareup.retrofit2:converter-gson:2.3.0' and then create class you would like to persist like this:
class UserProfile {
#SerializedName("name") var name: String = ""
#SerializedName("email") var email: String = ""
#SerializedName("age") var age: Int = 10
}
and finally in your SharedPreferences
fun saveUserProfile(userProfile: UserProfile?) {
val serializedUser = gson.toJson(userProfile)
sharedPreferences.edit().putString(USER_PROFILE, serializedUser).apply()
}
fun readUserProfile(): UserProfile? {
val serializedUser = sharedPreferences.getString(USER_PROFILE, null)
return gson.fromJson(serializedUser, UserProfile::class.java)
}
You can also use Firebase too for data persistence, you can use SharedPreferences, but personally I find it more easy and comfortable using Firebase.
I will leave a link here so you can take a look at disk persistence behavior of Firebase.
Hope this helps you!
For lists it think using sqlite will provide you with better control of your data.
Check out anko's sqlite wiki for more info.
Shared Preferences are mostly used in storing data like settings or configs, it's not meant for storing large data though it can do it as long as you implement Parcable. If you think your data contained in your MutableList is large, best way is to make a database.
Feel free to use this Kotlin extension functions to store and edit your list to SharedPreferences. I think they are realy useful.
inline fun <reified T> SharedPreferences.addItemToList(spListKey: String, item: T) {
val savedList = getList<T>(spListKey).toMutableList()
savedList.add(item)
val listJson = Gson().toJson(savedList)
edit { putString(spListKey, listJson) }
}
inline fun <reified T> SharedPreferences.removeItemFromList(spListKey: String, item: T) {
val savedList = getList<T>(spListKey).toMutableList()
savedList.remove(item)
val listJson = Gson().toJson(savedList)
edit {
putString(spListKey, listJson)
}
}
fun <T> SharedPreferences.putList(spListKey: String, list: List<T>) {
val listJson = Gson().toJson(list)
edit {
putString(spListKey, listJson)
}
}
inline fun <reified T> SharedPreferences.getList(spListKey: String): List<T> {
val listJson = getString(spListKey, "")
if (!listJson.isNullOrBlank()) {
val type = object : TypeToken<List<T>>() {}.type
return Gson().fromJson(listJson, type)
}
return listOf()
}