Parsing tiny JSON from android assets is incredibly slow - android

I am writing a simple Android app in Kotlin, which will show prayers divided into categories to user. There are 5 JSON files in assets folder, each of them has just around 10 KiB.
I use Klaxon for parsing the JSON files into those two data classes:
data class Prayer(val prayerName: String, val verseTitle: String, val verseBody: String,
val prayerLine: String, val prayerBody: String, val prayerEnding: String)
data class PrayerCategory(val title: String, val bgImage: String, val headerImage: String,
val prayers : List<Prayer>)
Here is the code I use for parsing the prayers:
private fun loadPrayerNames(jsonFile: String) {
val millis = measureTimeMillis {
val input = assets.open("${jsonFile}.json")
val prayerCategory = Klaxon().parse<PrayerCategory>(input)
if (prayerCategory != null) {
for (prayer in prayerCategory.prayers) {
val prayerName = prayer.prayerName
prayersMap[prayerName] = prayer
}
}
}
println("Loading prayer category took $millis ms.")
}
As you can see, there is just one access to assets. No assets.list(), no bullshit.
And as you noticed, I have measured the time.. make your guesses.. Here is the debug output:
Loading prayer category took 3427 ms.
Yup, that's right. Loading and parsing 10KiB big JSON took 3.5 SECONDS! I repeat. No rocket science involved. Just parsing 10 KiB JSON. 3.5 seconds..... Hmm..
Btw, I am testing it on Nokia 6.1, which is a pretty snappy phone.
So.. my questions:
What causes this behaviour?
Is there any way how to speed this up other than building a database for storing about 50 prayers?
I will be very thankful for your help!

Android assets seem to have bad reputation when it comes to performance. However, my testing proved that in this situation, it was Klaxon library who was responsible.
After finding major performance problem in Klaxon (see https://github.com/cbeust/klaxon/issues/154), which is still not fixed, I have tried recommended alternative: Moshi (https://github.com/square/moshi).
Moshi did improve performance, but parsing my JSON still took about 1 second.
After these experiments, I have resorted to the good old fashioned parsing using JSONObject:
data class Prayer(val prayerName: String, val verseTitle: String, val verseBody: String,
val prayerLine: String, val prayerBody: String, val prayerEnding: String) {
companion object {
fun parseJson(json: JSONObject) : Prayer = Prayer(json.getString("prayerName"),
json.getString("verseTitle"), json.getString("verseBody"),
json.getString("prayerLine"), json.getString("prayerBody"),
json.getString("prayerEnding"))
}
}
data class PrayerCategory(val title: String, val bgImage: String, val headerImage: String,
val prayers : List<Prayer>) {
companion object {
fun parseJson(json: JSONObject): PrayerCategory {
val prayers = ArrayList<Prayer>()
val prayersArray = json.getJSONArray("prayers")
for(i in 0 until prayersArray.length()) {
prayers.add(Prayer.parseJson(prayersArray.getJSONObject(i)))
}
return PrayerCategory(json.getString("title"), json.getString("bgImage"),
json.getString("headerImage"), prayers)
}
}
}
This has reduced parsing time from 3427 ms to 13ms. Case closed. ;-)

Related

Json file is not serialized properly

I'm trying to get data from API. My app crashes and gives me this message:
kotlinx.serialization.json.internal.JsonDecodingException: Expected
start of the array '[', but had 'EOF' instead
JSON input: .....f2afd0-71e8-5229-9ac9-17d602cd35e4"}],"meta":{"hits":3934}}}
I'm facing this error since a week and I can't find out a solution for it. I need help.
This is the json rsponse:
{"copyright":"Copyright (c) 2022 The New York Times Company. All Rights Reserved.","response":{"docs":[{"abstract":"A career in health journalism has taught me that when it comes to living well, it’s the inner workout that counts the most.","web_url":"https://www.nytimes.com/2022/03/31/well/mind/health-fitness-lessons.html","snippet":"A career in health journalism has taught me that when it comes to living well, it’s the inner workout that counts the most.","lead_paragraph":"When I first started writing about health more than 20 years ago, my columns mostly focused on the physical body: A healthy diet, exercise and screening for disease were regular topics.","source":"The New York Times","multimedia":[{"rank":0,"subtype":"xlarge","caption":null,"credit":null,"type":"image","url":"images/2022/04/05/well/31Well-NL-InnerFitness/31Well-NL-InnerFitness-articleLarge.jpg","height":400,"width":600,"subType":"xlarge","crop_name":"articleLarge","legacy":{"xlarge":"images/2022/04/05/well/31Well-NL-InnerFitness/31Well-NL-InnerFitness-articleLarge.jpg","xlargewidth":600,"xlargeheight":400}},{"rank":0,"subtype":"jumbo","caption":null,"credit":null,"type":"image","url":"images/2022/04/05/well/31Well-NL-InnerFitness/31Well-NL-InnerFitness-jumbo.jpg","height":682,"width":1024,"subType":"jumbo","crop_name":"jumbo","legacy":{}},{"rank":0,"subtype":"superJumbo","caption":null,"credit":null,"type":"image","url":"images/2022/04/05/well/31Well-NL-InnerFitness/31Well-NL-InnerFitness-superJumbo.jpg","height":1166,"width":1750,"subType":"superJumbo","crop_name":"superJumbo","legacy":{}},{"rank":0,"subtype":"thumbnail","caption":null,"credit":null,"type":"image","url":"images/2022/04/05/well/31Well-NL-InnerFitness/31Well-NL-InnerFitness-thumbStandard.jpg","height":75,"width":75,"subType":"thumbnail","crop_name":"thumbStandard","legacy":{"thumbnail":"images/2022/04/05/well/31Well-NL-InnerFitness/31Well-NL-InnerFitness-thumbStandard.jpg","thumbnailwidth":75,"thumbnailheight":75}},{"rank":0,"subtype":"thumbLarge","caption":null,"credit":null,"type":"image","url":"images/2022/04/05/well/31Well-NL-InnerFitness/31Well-NL-InnerFitness-thumbLarge.jpg","height":150,"width":150,"subType":"thumbLarge","crop_name":"thumbLarge","legacy":{}}],"headline":{"main":"For Better Health, Try Fitness From the Inside Out","kicker":null,"content_kicker":null,"print_headline":"","name":null,"seo":null,"sub":null},"keywords":[{"name":"subject","value":"Content Type: Service","rank":1,"major":"N"},{"name":"subject","value":"Anxiety and Stress","rank":2,"major":"N"},{"name":"subject","value":"Meditation","rank":3,"major":"N"},{"name":"subject","value":"Exercise","rank":4,"major":"N"}],"pub_date":"2022-04-01T00:00:08+0000","document_type":"article","news_desk":"Well","section_name":"Well","subsection_name":"Mind","byline":{"original":"By Tara Parker-Pope","person":[{"firstname":"Tara","middlename":null,"lastname":"Parker-Pope","qualifier":null,"title":null,"role":"reported","organization":"","rank":1}],"organization":null},"type_of_material":"News","_id":"nyt://article/0115da20-943c-52dc-8992-817faa170aa1","word_count":1235,"uri":"nyt://article/0115da20-943c-52dc-8992-817faa170aa1"},{"abstract":"He was famous for his portraits of supermodels and his close relationship with Princess Diana. His images are part of the fashion canon.","web_url":"https://www.nytimes.com/2022/03/31/style/patrick-demarchelier-dead.html","snippet":"He was famous for his portraits of supermodels and his close relationship with Princess Diana. His images are part of the fashion canon.","lead_paragraph":"Patrick Demarchelier, a photographer whose work helped define fashion and celebrity in the late 20th and early 21st centuries, died on Thursday. He was 78.","print_section":"A","print_page":"26","source":"The New York Times","multimedia":[{"rank":0,"subtype":"xlarge","caption":null,"credit":null,"type":"image","url":"images/2022/04/03/obituaries/02DEMARCHELIER-print2/00DEMARCHELIER-diana-articleLarge.jpg","height":817,"width":600,"subType":"xlarge","crop_name":"articleLarge","legacy":{"xlarge":"images/2022/04/03/obituaries/02DEMARCHELIER-print2/00DEMARCHELIER-diana-articleLarge.jpg","xlargewidth":600,"xlargeheight":817}},{"rank":0,"subtype":"jumbo","caption":null,"credit":null,"type":"image","url":"images/2022/04/03/obituaries/02DEMARCHELIER-print2/00DEMARCHELIER-diana-jumbo.jpg","height":1024,"width":752,"subType":"jumbo","crop_name":"jumbo","legacy":{}},{"rank":0,"subtype":"superJumbo","caption":null,"credit":null,"type":"image","url":"images/2022/04/03/obituaries/02DEMARCHELIER-print2/00DEMARCHELIER-diana-superJumbo.jpg","height":2048,"width":1504,"subType":"superJumbo","crop_name":"superJumbo","legacy":{}},{"rank":0,"subtype":"thumbnail","caption":null,"credit":null,"type":"image","url":"images/2022/04/03/obituaries/02DEMARCHELIER-print2/00DEMARCHELIER-diana-thumbStandard-v2.jpg","height":75,"width":75,"subType":"thumbnail","crop_name":"thumbStandard","legacy":{"thumbnail":"images/2022/04/03/obituaries/02DEMARCHELIER-print2/00DEMARCHELIER-diana-thumbStandard-.....
These are some of the data classes
#Serializable
data class Article(
#SerialName("copyright")
val copyright: String?,
#SerialName("response")
val response: Response?
)
#Serializable
data class Response(
#SerialName("docs")
val docs: List<Doc>?,
#SerialName("meta")
val meta: Meta?
)
#Serializable
data class Doc(
#SerialName("abstract")
val `abstract`: String?,
#SerialName("byline")
val byline: Byline?,
#SerialName("document_type")
val documentType: String?,
#SerialName("headline")
val headline: Headline?,
#SerialName("_id")
val id: String?,
#SerialName("keywords")
val keywords: List<Keyword>?,
#SerialName("lead_paragraph")
val leadParagraph: String?,
#SerialName("multimedia")
val multimedia: List<Multimedia>?,
#SerialName("news_desk")
val newsDesk: String?,
#SerialName("print_page")
val printPage: String?,
#SerialName("print_section")
val printSection: String?,
#SerialName("pub_date")
val pubDate: String?,
#SerialName("section_name")
val sectionName: String?,
#SerialName("snippet")
val snippet: String?,
#SerialName("source")
val source: String?,
#SerialName("subsection_name")
val subsectionName: String?,
#SerialName("type_of_material")
val typeOfMaterial: String?,
#SerialName("uri")
val uri: String?,
#SerialName("web_url")
val webUrl: String?,
#SerialName("word_count")
val wordCount: Int?
)
This is the provide function for the retrofit instance:
#ExperimentalSerializationApi
#Provides
#Singleton
fun provideRetrofitInstance(okHttpClient: OkHttpClient): Retrofit {
val contentType = "application/json".toMediaType()
return Retrofit.Builder()
.baseUrl(BASE_URL)
.client(okHttpClient)
.addConverterFactory(json.asConverterFactory(contentType))
.build()
}
you should be using this plugin
this plugin makes your work easy
after installing
and paste all JSON
before using read all documents Link
I recommend using Gson Or moshi to Convert Json Response.
Add dependency in build.Gradle (moshi or gson)
configure retrofit .addConverterFactory(MoshiConverterFactory.create(moshi)) for moshi
Annotate data class with #Json for moshi or #SerializableName for Gson.
Hope it helps !

Room very slow read operation

I have a room database and I'm trying to read data from it, when reading 16k records, the operation takes 15seconds.
this is my code
#Entity(tableName = "reading_table")
data class DatabaseReading(
#PrimaryKey(autoGenerate = true)
val readingId: Int = 0,
val readingNum: Int,
val deviceId: String,
val readingTime: Long,
val topRh: Double,
val topTemp: Double,
val botRh: Double,
val botTemp: Double,
)
this is my DAO
#Query("SELECT * FROM reading_table WHERE deviceId = :id ORDER BY readingTime ASC")
suspend fun getDeviceReadings(id: String): List<DatabaseReading>
then from my viewmodel I'm running the query
fun getReadings() {
Timber.d("get readings start")
viewModelScope.launch {
withContext(Dispatchers.IO) {
val time = measureTimeMillis {
devicesRepository.database.deviceDatabaseDao.getDeviceReadings(
deviceID
)
}
Timber.d("get readings time $time")
}
}
}
getReadings is taking 15563ms for 16155 records.
how can I improve this?
it turns out that the issue was caused by the database inspector plugin on android studio.
as long as I don't use it everything runs normally.
I mean for a mobile device that's a pretty sizable amount of records, chances are you dont need and probably wont use all 16k of them.
You should probably use the Paging library for room to only load chunks of data instead of everything

Android: parsing huge JSON file and search optimization

I need to make a small app that needs to parse from a raw JSON file around 200K cities.
I can't use any DB, but the user needs to be able to perform a city search.
The problem with the approach is that parsing such a big JSON file takes a long time, but I can't think of any other approach. I have a couple of requirements:
Optimise for fast searches. Loading time of the app is not so important.
Time efficiency for filter algorithm should be better than linear.
The list can be preprocessed into any other representation that you consider more efficient for searches and display. Provide information of why that representation is more efficient in the comments of the code.
In my repository I have something like:
class CitiesRepository(private val application: Application) {
fun getCities(): List<City> {
val bufferReader = application.assets.open("cities.json").bufferedReader()
val data = bufferReader.use {
it.readText()
}
val gson = GsonBuilder().create()
val type: Type = object : TypeToken<ArrayList<City?>?>() {}.type
return gson.fromJson(data, type)
}
}
and my ViewModel consists of:
class CitiesListViewModel(val app: Application) : AndroidViewModel(app) {
private val repository: CitiesRepository = CitiesRepository(app)
val allCities: LiveData<List<City>> =
liveData(context = viewModelScope.coroutineContext + Dispatchers.IO) {
val cities = repository.getCities()
withContext(Dispatchers.Main) {
emit(cities)
}
}
}
Is there anything else (better) I can do?
Thanks in advance!

Fold with Kotlin in a property of a data class model in an Android app

I'm not sure the title is clarifying enough, so I'll explain here:
I have four models:
This model is the one I use for parsing the JSON from a file
#JsonClass(generateAdapter = true)
data class Account(
val account: String,
val balance: String,
val transactions: List<Transaction>
)
This Account has a list of Transactions:
#JsonClass(generateAdapter = true)
data class Transaction(
val id: String,
val amount: String,
val description: String,
val otherAccount: String,
val date: String
)
For the UI layer, I have some equivalent models:
data class AccountEntity(
val account: String,
val balance: String,
val transactions: List<TransactionEntity>
)
and
data class TransactionEntity(
var amount: BigDecimal,
var balanceBefore: BigDecimal,
var balanceAfter: BigDecimal,
val description: String,
val otherAccount: String,
val date: Date
)
The tricky thing I want to do is, that an Account has a balance and I need to show in every TransactionData the balanceBefore and the balanceAfter which are respectively, the balance of the account BEFORE this transaction was made, and the balance of the account AFTER this transaction was made.
For me this is a bit tricky not only to understand but also to implement.
I thought of maybe implementing some sort of fold operation in every transaction, but I'm not sure how to apply it only in the balanceBefore and the balanceAfter properties.
Any other idea?
Thanks a lot in advance!
Well, it's some sort of reduce/fold function that also allows you to map your Transaction objects.
What this function do is to track the current balance in an accumulator and transform each Transaction object:
inline fun <R> List<Transaction>.map(
initial: Int,
transform: (transaction: Transaction, beforeBalance: Int, afterBalance: Int) -> R
): List<R> {
var accumulator = initial
val destination = ArrayList<R>()
for (transaction in this) {
val after = accumulator - transaction.amount
destination.add(transform(transaction, accumulator, after))
accumulator = after
}
return destination
}
And you can use it like:
account.transactions.map(account.balance, {t, b, a-> TransactionEntity(...)})
To be more generic and functional, you can move this line accumulator - transaction.amount to the function parameter as a fold input, and call the function foldMap or whatever which reduce your current transaction and also map it.

Retrofit - Android - Handling multiple type keys and values with gson

So I've been having some trouble with an api that I want to consume from my application when dealing with multiple types from the keys and values of the response json.
Let me show you the json response:
{
"error":[
],
"result":{
"field1":[
[
1544258160,
"57.15",
"57.15",
"57.15",
"57.15",
"0.00",
"0.00000000",
0
],
[
1544258220,
"56.89",
"56.89",
"56.89",
"56.89",
"56.89",
"2.94406281",
1
]
],
"field2":1544301240
}
}
and here is the representation of the pojo class:
data class Response(val error: List<String>, val result: LinkedTreeMap<String, List<List<Result>>>)
data class Result(
val time: Double,
val open: String,
val high: String,
val low: String,
val close: String,
val vwap: String,
val volume: String,
val count: Double
)
I know that the current structure fails to represent the json format. but I have run out of ideas.
btw the stack error is saying this:
Expected BEGIN_OBJECT but was NUMBER
edit: adding a bit more context
I'm using Gsonconverter for the retrofit builder.
val retrofit = Retrofit.Builder().baseUrl(API_URL).client(client)
.addConverterFactory(GsonConverterFactory.create()).build()
You could build a Gson JsonDeserializer for the filed1, filed2 types.
It is more manually written code, but this way you can check the type of the filed and call the right deserializer.
yeah,what make the error is your response field(result),it's different type.however.you just use the first type Response(val error: List<String>, val result:LinkedTreeMap<String, List<List<Result>>> receive all the json.
in my view,you can use below ways to solve it.
First,redefine you receive model,maybe you can use the Gson JsonObject and when deal with value remember check the type.
Second,discuss with your background server engineer,agree on the every type response.may be the "field2" can be "field2:[[]]"
Next,change your model.may be you define below it
#Serialized("field1")
LinkedTreeMap<String, List<List<Result>>> field;
#Serialized("field2")
String fields;
hope to help you.
Your problem is here:
"field2":1544301240
You defined like below:
val result: LinkedTreeMap<String, List<List<Result>>>
but instead of that you get number!!!
EDIT
Just try this:
data class Response(val error: List<String>, val result: YourModel)
data class YourModel(val field1: LinkedTreeMap<String, List<List<Result>>>, val field2: Double)
data class Result(
val time: Double,
val open: String,
val high: String,
val low: String,
val close: String,
val vwap: String,
val volume: String,
val count: Double
)
it has been a while but still i already came to a conclusion of what i need.
data class OHLCResponse(
private val error: List<String>,
val result: LinkedTreeMap<String, Any>)
turns out that on the LinkedTreeMap class i just needed to pass the second class type parameter as Any and it can be down casted to anything that you need.
also i leave here a nice tool that can help you map your json responses into kotlin plain objects/DTOs:
https://app.quicktype.io/
thanks all for your responses.
I solved the problem with such a solution in adapter class .
override fun getItemCount(): Int = leagueTableList[0].size
override fun onBindViewHolder(holder: LeagueTableViewHolder, position: Int) {
val gs = Gson()
val js = gs.toJson(leagueTableList[0][position])
val standing = gs.fromJson(js, Standing::class.java)
holder.view.table = standing
holder.bind(standing,onItemClick)
}

Categories

Resources