i am trying to parse a big json file (>76K character) and read two regions out of it for mapping those into a WebView.
My problem is that i receive JSONExceptions when parsing it and i cant reconstruct the error.
This is my callVM class which includes the functionality.
fun callVM()
var url: String = getString(R.string.app_name)//R.string.myHtml
try {
//read url via getTermsString
url = getTermsString()
val gson = Gson()
var decodedHtml = unescape(url)
readJson(decodedHtml)
wv_map.loadData(decodedHtml, "text/html; charset=UTF-8", null);
} catch (e: Exception) {
Log.e("FAIL", "Initializing failed")
e.printStackTrace()
}
}
The getTermsString()-function reads the "url", in this case its my api_response_jreal.json and no url. The naming is old.
The api_response_jreal.json is in my raw folder.
private fun getTermsString(): String {
Log.e("gts", "Start getTermsString()")
val ist: InputStreamReader
try {
Log.e("Reader", "Start reader")
//R.raw.x - x equals the html file
ist = InputStreamReader(resources.openRawResource(R.raw.api_response_jreal))
val theString = IOUtils.toString(ist)
Log.e("gtsfin", "getTermsString() finished")
ist.close()
return theString
} catch (e: IOException) {
System.err.println(e.printStackTrace())
}
return "gts did not load anything."
}
My unescape function is there for replacing the escape character. I also tried to call the readJson-function without unescaping, but this is not working either...
fun unescape(str: String) : String
{
var strVar = str
strVar = strVar
// ADDED AFTER THE FIRST EXCEPTION.replace("\\u0","\\u003d")
.replace("\\u003d","=")
.replace("\\u003c","<")
.replace("\\u003e",">")
.replace("\\u0027","'")
.replace("\\\"", "")
.replace("\\t", " ")
.replace("\\u0023", "#")
.replace("\\u0024", "$")
.replace("\\u0025", "%")
.replace("\\u0026", "&")
.replace("\\u0028", "(")
.replace("\\u0029", ")")
.replace("\\u002A", "*")
.replace("\\u002B", "+")
.replace("\\u002C", ",")
.replace("\\u002D", "-")
.replace("\\u002E", ".")
.replace("\\u002F", "/")
.replace("\\u003A", ":")
.replace("\\u003B", ";")
.replace("\\u003F", "?")
.replace("\\u0040", "#")
.replace("/\\n/g", "\\n")
.replace("/\\'/g", "\\'")
.replace("/\\&/g", "\\&")
.replace("/\\r/g", "\\r")
.replace("/\\t/g", "\\t")
.replace("/\\b/g", "\\b")
.replace("/\\f/g", "\\f")
.replace("\\s+", " ")
.replace("\\\n","")
.replace("\\n","")
return strVar
}
Last but not least i try to transform my unescaped json to an Object to access the needed values (i just need "regionsToSvgMapping)
fun readJson(url : String){
val data = JSONObject(url)
Log.e("readjsn obj", "MyFile: " + data)
}
Now two exceptions appear.
The first was "Invalid escape sequence: 0"...
I copied the whole string in Notepad++ and take a look at character "3051"..
<polygon id\u0
03d12
Then i tried something stupid and thought the dumbest way would work and added the first .replace value in my unescape function. Well the dumbest way is not working ... ;)
Now its the time i dont have a solution for that problem and asking stackoverflow community.
I also tried to use a code beautifier to check if the json is complete so, this should be fine.
Thank you guys for help!
The solution was using StringBuilder.
When working with Strings >10k numbers and digits my Android studio has split the large string into multiple strings with around 1k numbers and digits.
To solve this i have used the stringbuilder like this,..
fun unescape(str: String) : String
{
var stringBuilder = StringBuilder()
var strVar = str
strVar = strVar
.replace("\\u003d","=")
.replace("\u003d","=")
.replace("\\u003c","<")
.... (see above)
stringBuilder.append(strVar)
return stringBuilder.toString()
}
Related
I am testing with MockWebServer.
And I need a lot of json files for request and response data.
Hard coded json values seem messy and I want to create json files instead.
So, I created json files in resources(test). And I tried to read file with these methods.
object TestHelper {
fun read(fileName: String): String {
val resource = javaClass.classLoader?.getResource(fileName)
return resource?.readText() ?: ""
}
fun readJson(fileName: String): String {
val byteArray = readBinaryFileFromResources(fileName)
val sb = StringBuilder("")
byteArray.forEach {
println("byte: $it")
sb.append(it as Char)
}
return sb.toString()
}
#Throws(IOException::class)
fun readBinaryFileFromResources(fileName: String): ByteArray {
var inputStream: InputStream? = null
val byteStream = ByteArrayOutputStream()
try {
inputStream = javaClass.classLoader?.getResourceAsStream(fileName)
var nextValue = inputStream?.read() ?: -1
while (nextValue != -1) {
byteStream.write(nextValue)
nextValue = inputStream?.read() ?: -1
}
return byteStream.toByteArray()
} catch (e: Exception) {
println(e.stackTraceToString())
return byteStream.toByteArray()
} finally {
inputStream?.close()
byteStream.close()
}
}
}
None of them seems work. What's the problem with this code?
I've had trouble with this before, and I believe it has to do with getting the correct classLoader from the call site, as well as having resources in the src/test/resources not being accessible properly. I eventually got it to work by passing in the calling test class as a reified type parameter:
inline fun <reified T> loadFileText(
caller: T,
filePath: String
): String =
T::class.java.getResource(filePath)?.readText() ?: throw IllegalArgumentException(
"Could not find file $filePath. Make sure to put it in the correct resources folder for $caller's runtime."
)
For my setup I have a separate shared module :testtools that I use testImplementation to include in my :app's gradle build so they don't get compiled into the production APK. I have my test resources in:
/testtools/src/main/resources/customfolder
And then calling this from a unit test class in :app like so:
class UnitTestClass {
#Test
fun myTest() {
loadFileText(this, "/customfolder/file_name.txt")
}
}
You might have some luck putting your resources straight into /app/src/test/resources/customfolder, I haven't tried in a while.
I am trying to create an android app with kotlin, this app need to have a mini download manager as I will need to download files from 100MB to 8GB and user can pause and resume download later when the server supports the pause, searching I found the Ktor library and reading the documentation plus some videos on youtube, I managed to write a base code where I could download the files and make the process of stopping the download and keep going all right when one of mine tests gave error there are files whose url pattern is: http://server.com/files?file=/10/55/file.zip
The problem is that I put this link, but Ktor converts to http://server.com/files?file=%2F10%2F55%2Ffile.zip this generate an error response on the server, as I don't have access to the server to change this rule I need to send the right url without encoding. Does anyone know how to do this? Prevent Ktor from doing a URL_encode in the url parameters, I couldn't find anything in the documentation
My code is this:
ktor-client version 1.6.7
fun startDownload(url: String, auth: String = "", userAgentS: String = "", fileName: String = ""){
val client = HttpClient(CIO)
val path = Environment.getExternalStoragePublicDirectory(Environment.DIRECTORY_DOWNLOADS)
val file = File.createTempFile("File", "index", path)
runBlocking {
client.get<HttpStatement>(url){
headers {
append(HttpHeaders.Authorization, auth)
append(HttpHeaders.UserAgent, userAgentS)
append(HttpHeaders.Range, "bytes=${file.length()}-")
}
}
.execute { httpResponse ->
val channel: ByteReadChannel = httpResponse.receive()
while (!channel.isClosedForRead) {
val packet = channel.readRemaining(DEFAULT_BUFFER_SIZE.toLong())
while (!packet.isEmpty) {
val bytes = packet.readBytes()
file.appendBytes(bytes)
println("Received ${(file.length())} bytes from ${httpResponse.contentLength()}")
}
}
val pathF = Environment.getExternalStoragePublicDirectory(Environment.DIRECTORY_DOWNLOADS + "/${fileName}")
file.renameTo(pathF)
println("A file saved to ${file.path}")
}
}
}
Can anyone help me solve this problem with ktor, if there is no solution, can someone tell me another way to achieve the same goal? Need to be with Kotlin.
update 2022-02-17
Thanks to Aleksei Tirman's help I managed to solve the problem, thank you very much. And the base code looks like this:
fun startDownload(url: String, auth: String = "", userAgentS: String = "", fileName: String = ""){
val client = HttpClient(CIO)
val path = Environment.getExternalStoragePublicDirectory(Environment.DIRECTORY_DOWNLOADS)
val file = File.createTempFile("File", "index", path)
runBlocking {
client.get<HttpStatement>(url){
url {
parameters.urlEncodingOption = UrlEncodingOption.NO_ENCODING
}
headers {
append(HttpHeaders.Authorization, auth)
append(HttpHeaders.UserAgent, userAgentS)
append(HttpHeaders.Range, "bytes=${file.length()}-")
}
}
.execute { httpResponse ->
val channel: ByteReadChannel = httpResponse.receive()
while (!channel.isClosedForRead) {
val packet = channel.readRemaining(DEFAULT_BUFFER_SIZE.toLong())
while (!packet.isEmpty) {
val bytes = packet.readBytes()
file.appendBytes(bytes)
println("Received ${(file.length())} bytes from ${httpResponse.contentLength()}")
}
}
val pathF = Environment.getExternalStoragePublicDirectory(Environment.DIRECTORY_DOWNLOADS + "/${fileName}")
file.renameTo(pathF)
println("A file saved to ${file.path}")
}
}
}
You can disable query parameters encoding by assigning the UrlEncodingOption.NO_ENCODING value to the urlEncodingOption property of the ParametersBuilder. Here is an example:
val requestBuilder = HttpRequestBuilder()
requestBuilder.url {
protocol = URLProtocol.HTTP
host = "httpbin.org"
path("get")
parameters.urlEncodingOption = UrlEncodingOption.NO_ENCODING
parameters.append("file", "/10/55/file.zip")
}
val response = client.get<String>(requestBuilder)
I am currently making an android app that needs to read quotes in a text file. I have code to read everything from my file and display it through a toast, but I do not know how I can just read a specific line from it (eg. only displaying line 5 in a toast).
Here is my code:
var string: String? = ""
val stringBuilder = StringBuilder()
val `is`: InputStream = this.resources.openRawResource(R.raw.quotes)
val reader = BufferedReader(InputStreamReader(`is`))
while (true) {
try {
if (reader.readLine().also { string = it } == null) break
} catch (e: IOException) {
e.printStackTrace()
}
stringBuilder.append(string).append("\n")
Log.d("strings", stringBuilder.toString())
}
`is`.close()
Toast.makeText(baseContext, stringBuilder.toString(),
Toast.LENGTH_LONG).show()
Any help would be greatly appreciated, and thank you for any help in advance!
You can use useLines, which lets you work with a Sequence of the lines from within the lambda and automatically closes the stream afterwards:
val fifthLine = resources.openRawResource(R.raw.quotes)
.bufferedReader().useLines { it.elementAtOrNull(4) ?: "" }
on Android, using com.google.gson:gson:2.8.5,
when passing a josnString and when the json is large (noticed for example when string length is 669304), it got
com.google.gson.JsonSyntaxException: com.google.gson.stream.MalformedJsonException:
Unterminated string at line 2902 column 564904 path $.items.result[10].quoted_status.full_text
or different errors:
the json has array of sub json strings, structured like,
{
"items": {
"result": [
{subJsonString...},
{subJsonString...},
...
]
}
}
and the json string is stored in the res/raw/jsonstring.txt, and read in at runtime before give to Gson fro deserialization.
if reduce the sub json elements in the array (i.e. 10 items or less in the array) it works fine, and the individual json elements string all working fine. But when the array has more items it starts to throw.
update: it seems is problem from reading the json string from res/raw
after further look,
the log shows the output json string read from the res/raw/jsonString.txt are corrupted, when there are more items. Here after adding the 11th sub item into the json string array, it shows 000000000... at certain point (the 11th item who encountered issue are just a copy from the 1st item, so the string should be fine)
here is the code how the raw string is read, how it logs it
val jsonStr = getJsonFromFile(context, "res/raw/jsoninraw.txt")
returnedModel = gson.fromJson<T>(jsonStr, dataClassType)
......
fun getJsonFromFile(context: Context?, fileName: String): String? {
val json: String
try {
val inputStream: InputStream = context.getAssets().open(fileName)
val size = inputStream.available()
val buffer = ByteArray(size)
inputStream.use { it.read(buffer) }
json = String(buffer)
} catch (ioException: IOException) {
ioException.printStackTrace()
return null
}
return json
.also {
logString(it)
}
}
fun logString(jsonStr: String) {
val chunkSize = 512
var i = 0
while (i < jsonStr.length) {
Log.e("+++", jsonStr.substring(i, Math.min(jsonStr.length, i + chunkSize)))
i += chunkSize
}
}
what are the better way to deserialize the json string into model?
Found the problem that the ByteArray has 64k limit, so anything beyond is corrupted.
the updated the getJsonFromFile() works.
fun getJsonFromFile(context: Context?, fileName: String?): String? {
val returnString = StringBuilder()
var inputStream: InputStream? = null
var isr: InputStreamReader? = null
var input: BufferedReader? = null
try {
inputStream = if (context == null) {
val cLoader = this.javaClass.classLoader
cLoader.getResourceAsStream(fileName)
} else {
context.getAssets().open(fileName)
}
// either #1:
// returnString.append(inputStream?.readBytes()?.toString(Charsets.UTF_8))
// or #2:
inputStream?.let {
val inputString = inputStream.bufferedReader().use { it.readText() }
returnString.append(inputString)
}
// or #3:
// isr = InputStreamReader(inputStream)
// input = BufferedReader(isr)
// var line: String? = ""
// while (input.readLine().also { line = it } != null) {
// returnString.append(line)
// }
} catch (e: Exception) {
e.printStackTrace()
return null
} finally {
try {
isr?.close()
inputStream?.close()
input?.close()
} catch (e2: Exception) {
e2.message
}
}
return returnString.toString()
}
There is a function param() in jQuery library
It serializes JS-object into a string set of HTTP parameters
Example: object {a:'x',b:{c:[0,1]}}
$.param({a:'x',b:{c:[0,1]}})
"a=x&b%5Bc%5D%5B%5D=0&b%5Bc%5D%5B%5D=1"
This string is encoded by encodeURIComponent()
It could be decoded by decodeURIComponent()
decodeURIComponent($.param({a:'x',b:{c:[0,1]}}))
"a=x&b[c][]=0&b[c][]=1"
We see that the object is converted into a set of parameters:
a=x
b[c][]=0
b[c][]=1
I need to do the same serialization in my Android application
I have to send data in the same format as is being done by jQuery
So the question is:
Is there some function in Android SDK which is an analogue of jQuery.param() ?
While people stays silent I offer my handmade solution on Kotlin:
fun serialize(data: JSONObject): String {
var result = arrayListOf<String>()
fun parse(value: Any, name: String, alt: String? = null) {
if (value is JSONObject) {
val keys = value.keys()
while (keys.hasNext())
keys.next().let {
parse(value[it], "$name[$it]")
}
} else if (value is JSONArray) {
for (i in 0 until value.length())
parse(value[i], "$name[$i]", "$name[]")
} else {
result.add(Uri.encode(alt ?: name) + "=" + Uri.encode("$value"))
}
}
val keys = data.keys()
while (keys.hasNext())
keys.next().let {
parse(data[it], it)
}
return result.joinToString("&")
}
This variant works identically to jQuery. It would be nice to avoid such homework though :-)