Related
Background
I wanted to get information about APK files(including split APK files), even if they are inside compressed zip files (without de-compressing them). In my case, this include various things, such as package name, version-code, version-name, app-label, app-icon, and if it's a split APK file or not.
Do note that I want to do it all inside an Android app, not using a PC, so some tools might not be possible to be used.
The problem
This means I can't use the getPackageArchiveInfo function, as this function requires a path to the APK file, and works only on non-split-apk files.
In short, there is no framework function to do it, so I have to find a way of how to do it by going into the zipped file, using the InputStream as input for parsing it in a function.
There are various solutions online, including outside of Android, but I don't know of one that is stable and works for all cases. Many might be good even for Android (example here), but might fail parsing and might require a file path instead of Uri/InputStream.
What I've found&tried
I've found this on StackOverflow, but sadly according to my tests, it always generates content, but in some rare cases it's not a valid XML content.
So far, I've found these apps package names and their version codes that the parser fails to parse, as the output XML content is invalid:
com.farproc.wifi.analyzer 139
com.teslacoilsw.launcherclientproxy 2
com.hotornot.app 3072
android 29 (that's the "Android System" system app itself)
com.google.android.videos 41300042
com.facebook.katana 201518851
com.keramidas.TitaniumBackupPro 10
com.google.android.apps.tachyon 2985033
com.google.android.apps.photos 3594753
Using an XML viewer and XML validator, here are the issues with these apps:
For #1,#2, I got a very weird content, starting with <mnfs .
For #3, it doesn't like the "&" in <activity theme="resourceID 0x7f13000b" label="Features & Tests" ...
For #4, it missed the end tag of "manifest" in the end.
For #5, it missed multiple end tags, at least of "intent-filter","receiver" and "manifest". Maybe more.
For #6, it got "allowBackup" attribute twice in the "application" tag for some reason.
For #7, it got a value without attribute in the manifest tag: <manifest versionCode="resourceID 0xa" ="1.3.2".
For #8, it missed a lot of content after getting some "uses-feature" tags, and didn't have an ending tag for "manifest".
For #9, it missed a lot of content after getting some "uses-permission" tags, and didn't have an ending tag for "manifest"
Surprisingly, I didn't find any issue with split APK files. Only with main APK files.
Here's the code (also available here) :
MainActivity .kt
class MainActivity : AppCompatActivity() {
override fun onCreate(savedInstanceState: Bundle?) {
super.onCreate(savedInstanceState)
setContentView(R.layout.activity_main)
thread {
val problematicApkFiles = HashMap<ApplicationInfo, HashSet<String>>()
val installedApplications = packageManager.getInstalledPackages(0)
val startTime = System.currentTimeMillis()
for ((index, packageInfo) in installedApplications.withIndex()) {
val applicationInfo = packageInfo.applicationInfo
val packageName = packageInfo.packageName
// Log.d("AppLog", "$index/${installedApplications.size} parsing app $packageName ${packageInfo.versionCode}...")
val mainApkFilePath = applicationInfo.publicSourceDir
val parsedManifestOfMainApkFile =
try {
val parsedManifest = ManifestParser.parse(mainApkFilePath)
if (parsedManifest?.isSplitApk != false)
Log.e("AppLog", "$packageName - parsed normal APK, but failed to identify it as such")
parsedManifest?.manifestAttributes
} catch (e: Exception) {
Log.e("AppLog", e.toString())
null
}
if (parsedManifestOfMainApkFile == null) {
problematicApkFiles.getOrPut(applicationInfo, { HashSet() }).add(mainApkFilePath)
Log.e("AppLog", "$packageName - failed to parse main APK file $mainApkFilePath")
}
if (Build.VERSION.SDK_INT >= Build.VERSION_CODES.LOLLIPOP)
applicationInfo.splitPublicSourceDirs?.forEach {
val parsedManifestOfSplitApkFile =
try {
val parsedManifest = ManifestParser.parse(it)
if (parsedManifest?.isSplitApk != true)
Log.e("AppLog", "$packageName - parsed split APK, but failed to identify it as such")
parsedManifest?.manifestAttributes
} catch (e: Exception) {
Log.e("AppLog", e.toString())
null
}
if (parsedManifestOfSplitApkFile == null) {
Log.e("AppLog", "$packageName - failed to parse main APK file $it")
problematicApkFiles.getOrPut(applicationInfo, { HashSet() }).add(it)
}
}
}
val endTime = System.currentTimeMillis()
Log.d("AppLog", "done parsing. number of files we failed to parse:${problematicApkFiles.size} time taken:${endTime - startTime} ms")
if (problematicApkFiles.isNotEmpty()) {
Log.d("AppLog", "list of files that we failed to get their manifest:")
for (entry in problematicApkFiles) {
Log.d("AppLog", "packageName:${entry.key.packageName} , files:${entry.value}")
}
}
}
}
}
ManifestParser.kt
class ManifestParser{
var isSplitApk: Boolean? = null
var manifestAttributes: HashMap<String, String>? = null
companion object {
fun parse(file: File) = parse(java.io.FileInputStream(file))
fun parse(filePath: String) = parse(File(filePath))
fun parse(inputStream: InputStream): ManifestParser? {
val result = ManifestParser()
val manifestXmlString = ApkManifestFetcher.getManifestXmlFromInputStream(inputStream)
?: return null
val factory: DocumentBuilderFactory = DocumentBuilderFactory.newInstance()
val builder: DocumentBuilder = factory.newDocumentBuilder()
val document: Document? = builder.parse(manifestXmlString.byteInputStream())
if (document != null) {
document.documentElement.normalize()
val manifestNode: Node? = document.getElementsByTagName("manifest")?.item(0)
if (manifestNode != null) {
val manifestAttributes = HashMap<String, String>()
for (i in 0 until manifestNode.attributes.length) {
val node = manifestNode.attributes.item(i)
manifestAttributes[node.nodeName] = node.nodeValue
}
result.manifestAttributes = manifestAttributes
}
}
result.manifestAttributes?.let {
result.isSplitApk = (it["android:isFeatureSplit"]?.toBoolean()
?: false) || (it.containsKey("split"))
}
return result
}
}
}
ApkManifestFetcher.kt
object ApkManifestFetcher {
fun getManifestXmlFromFile(apkFile: File) = getManifestXmlFromInputStream(FileInputStream(apkFile))
fun getManifestXmlFromFilePath(apkFilePath: String) = getManifestXmlFromInputStream(FileInputStream(File(apkFilePath)))
fun getManifestXmlFromInputStream(ApkInputStream: InputStream): String? {
ZipInputStream(ApkInputStream).use { zipInputStream: ZipInputStream ->
while (true) {
val entry = zipInputStream.nextEntry ?: break
if (entry.name == "AndroidManifest.xml") {
// zip.getInputStream(entry).use { input ->
return decompressXML(zipInputStream.readBytes())
// }
}
}
}
return null
}
/**
* Binary XML doc ending Tag
*/
private var endDocTag = 0x00100101
/**
* Binary XML start Tag
*/
private var startTag = 0x00100102
/**
* Binary XML end Tag
*/
private var endTag = 0x00100103
/**
* Reference var for spacing
* Used in prtIndent()
*/
private var spaces = " "
/**
* Parse the 'compressed' binary form of Android XML docs
* such as for AndroidManifest.xml in .apk files
* Source: http://stackoverflow.com/questions/2097813/how-to-parse-the-androidmanifest-xml-file-inside-an-apk-package/4761689#4761689
*
* #param xml Encoded XML content to decompress
*/
private fun decompressXML(xml: ByteArray): String {
val resultXml = StringBuilder()
// Compressed XML file/bytes starts with 24x bytes of data,
// 9 32 bit words in little endian order (LSB first):
// 0th word is 03 00 08 00
// 3rd word SEEMS TO BE: Offset at then of StringTable
// 4th word is: Number of strings in string table
// WARNING: Sometime I indiscriminently display or refer to word in
// little endian storage format, or in integer format (ie MSB first).
val numbStrings = lew(xml, 4 * 4)
// StringIndexTable starts at offset 24x, an array of 32 bit LE offsets
// of the length/string data in the StringTable.
val sitOff = 0x24 // Offset of start of StringIndexTable
// StringTable, each string is represented with a 16 bit little endian
// character count, followed by that number of 16 bit (LE) (Unicode) chars.
val stOff = sitOff + numbStrings * 4 // StringTable follows StrIndexTable
// XMLTags, The XML tag tree starts after some unknown content after the
// StringTable. There is some unknown data after the StringTable, scan
// forward from this point to the flag for the start of an XML start tag.
var xmlTagOff = lew(xml, 3 * 4) // Start from the offset in the 3rd word.
// Scan forward until we find the bytes: 0x02011000(x00100102 in normal int)
run {
var ii = xmlTagOff
while (ii < xml.size - 4) {
if (lew(xml, ii) == startTag) {
xmlTagOff = ii
break
}
ii += 4
}
} // end of hack, scanning for start of first start tag
// XML tags and attributes:
// Every XML start and end tag consists of 6 32 bit words:
// 0th word: 02011000 for startTag and 03011000 for endTag
// 1st word: a flag?, like 38000000
// 2nd word: Line of where this tag appeared in the original source file
// 3rd word: FFFFFFFF ??
// 4th word: StringIndex of NameSpace name, or FFFFFFFF for default NS
// 5th word: StringIndex of Element Name
// (Note: 01011000 in 0th word means end of XML document, endDocTag)
// Start tags (not end tags) contain 3 more words:
// 6th word: 14001400 meaning??
// 7th word: Number of Attributes that follow this tag(follow word 8th)
// 8th word: 00000000 meaning??
// Attributes consist of 5 words:
// 0th word: StringIndex of Attribute Name's Namespace, or FFFFFFFF
// 1st word: StringIndex of Attribute Name
// 2nd word: StringIndex of Attribute Value, or FFFFFFF if ResourceId used
// 3rd word: Flags?
// 4th word: str ind of attr value again, or ResourceId of value
// TMP, dump string table to tr for debugging
//tr.addSelect("strings", null);
//for (int ii=0; ii<numbStrings; ii++) {
// // Length of string starts at StringTable plus offset in StrIndTable
// String str = compXmlString(xml, sitOff, stOff, ii);
// tr.add(String.valueOf(ii), str);
//}
//tr.parent();
// Step through the XML tree element tags and attributes
var off = xmlTagOff
var indent = 0
// var startTagLineNo = -2
while (off < xml.size) {
val tag0 = lew(xml, off)
//int tag1 = LEW(xml, off+1*4);
// val lineNo = lew(xml, off + 2 * 4)
//int tag3 = LEW(xml, off+3*4);
// val nameNsSi = lew(xml, off + 4 * 4)
val nameSi = lew(xml, off + 5 * 4)
if (tag0 == startTag) { // XML START TAG
// val tag6 = lew(xml, off + 6 * 4) // Expected to be 14001400
val numbAttrs = lew(xml, off + 7 * 4) // Number of Attributes to follow
//int tag8 = LEW(xml, off+8*4); // Expected to be 00000000
off += 9 * 4 // Skip over 6+3 words of startTag data
val name = compXmlString(xml, sitOff, stOff, nameSi)
//tr.addSelect(name, null);
// startTagLineNo = lineNo
// Look for the Attributes
val sb = StringBuffer()
for (ii in 0 until numbAttrs) {
// val attrNameNsSi = lew(xml, off) // AttrName Namespace Str Ind, or FFFFFFFF
val attrNameSi = lew(xml, off + 1 * 4) // AttrName String Index
val attrValueSi = lew(xml, off + 2 * 4) // AttrValue Str Ind, or FFFFFFFF
// val attrFlags = lew(xml, off + 3 * 4)
val attrResId = lew(xml, off + 4 * 4) // AttrValue ResourceId or dup AttrValue StrInd
off += 5 * 4 // Skip over the 5 words of an attribute
val attrName = compXmlString(xml, sitOff, stOff, attrNameSi)
val attrValue = if (attrValueSi != -1)
compXmlString(xml, sitOff, stOff, attrValueSi)
else
"resourceID 0x" + Integer.toHexString(attrResId)
sb.append(" $attrName=\"$attrValue\"")
//tr.add(attrName, attrValue);
}
resultXml.append(prtIndent(indent, "<$name$sb>"))
indent++
} else if (tag0 == endTag) { // XML END TAG
indent--
off += 6 * 4 // Skip over 6 words of endTag data
val name = compXmlString(xml, sitOff, stOff, nameSi)
resultXml.append(prtIndent(indent, "</$name>")) // (line $startTagLineNo-$lineNo)
//tr.parent(); // Step back up the NobTree
} else if (tag0 == endDocTag) { // END OF XML DOC TAG
break
} else {
// println(" Unrecognized tag code '" + Integer.toHexString(tag0)
// + "' at offset " + off
// )
break
}
} // end of while loop scanning tags and attributes of XML tree
// println(" end at offset $off")
return resultXml.toString()
} // end of decompressXML
/**
* Tool Method for decompressXML();
* Compute binary XML to its string format
* Source: Source: http://stackoverflow.com/questions/2097813/how-to-parse-the-androidmanifest-xml-file-inside-an-apk-package/4761689#4761689
*
* #param xml Binary-formatted XML
* #param sitOff
* #param stOff
* #param strInd
* #return String-formatted XML
*/
private fun compXmlString(xml: ByteArray, #Suppress("SameParameterValue") sitOff: Int, stOff: Int, strInd: Int): String? {
if (strInd < 0) return null
val strOff = stOff + lew(xml, sitOff + strInd * 4)
return compXmlStringAt(xml, strOff)
}
/**
* Tool Method for decompressXML();
* Apply indentation
*
* #param indent Indentation level
* #param str String to indent
* #return Indented string
*/
private fun prtIndent(indent: Int, str: String): String {
return spaces.substring(0, min(indent * 2, spaces.length)) + str
}
/**
* Tool method for decompressXML()
* Return the string stored in StringTable format at
* offset strOff. This offset points to the 16 bit string length, which
* is followed by that number of 16 bit (Unicode) chars.
*
* #param arr StringTable array
* #param strOff Offset to get string from
* #return String from StringTable at offset strOff
*/
private fun compXmlStringAt(arr: ByteArray, strOff: Int): String {
val strLen = (arr[strOff + 1] shl (8 and 0xff00)) or (arr[strOff].toInt() and 0xff)
val chars = ByteArray(strLen)
for (ii in 0 until strLen) {
chars[ii] = arr[strOff + 2 + ii * 2]
}
return String(chars) // Hack, just use 8 byte chars
} // end of compXmlStringAt
/**
* Return value of a Little Endian 32 bit word from the byte array
* at offset off.
*
* #param arr Byte array with 32 bit word
* #param off Offset to get word from
* #return Value of Little Endian 32 bit word specified
*/
private fun lew(arr: ByteArray, off: Int): Int {
return (arr[off + 3] shl 24 and -0x1000000 or ((arr[off + 2] shl 16) and 0xff0000)
or (arr[off + 1] shl 8 and 0xff00) or (arr[off].toInt() and 0xFF))
} // end of LEW
private infix fun Byte.shl(i: Int): Int = (this.toInt() shl i)
// private infix fun Int.shl(i: Int): Int = (this shl i)
}
The questions
How come I get an invalid XML content for some APK manifest files (hence getting the XML parsing fail for them) ?
How can I get it to work, always?
Is there a better way to parse the manifest file into a valid XML ? Maybe a better alternative, that could work with all kinds of APK files, including inside zipped files, without decompressing them?
Likely you'd have to handle all of the special cases you've already identified.
Aliases & hexadecimal references might confuse it; these would need to be resolved.
For example, to fall-back from manifest to mnfs would at least solve one issue:
fun getRootNode(document: Document): Node? {
var node: Node? = document.getElementsByTagName("manifest")?.item(0)
if (node == null) {
node = document.getElementsByTagName("mnfs")?.item(0)
}
return node
}
"Features & Tests" would require TextUtils.htmlEncode() for & or another parser configuration.
Making it parse single AndroidManifest.xml files would make it easier to test, because with each other package there may be more unexpected input - until it comes close to the manifest parser which the OS uses (the source code might help). As one can see, it may set cookies for reading it. Take this list of package names and set up a test case for each of them, then the issues are rather isolated. But the main issue is that these cookies are most likely not available to 3rd party applications.
It seems that ApkManifestFetcher doesn't handle all cases such as text (between tags) and name space declarations and, maybe, a few other things. Below is a rework of ApkManifestFetcher that handles all 300+ APKs on my phone except for the Netflix APK which is coming up with some blank attributes.
I no longer believe that the files that start with <mnfs have anything to do with obfuscation but are encoded using UTF-8 rather than UTF-16 which the app assumes (16 bits vs 8 bits). The reworked app handles UTF-8 encoding and can parse these files.
As mentioned above, name spaces are not handled correctly by the original class or this rework although the rework can skip past them. Comments in the code describe this a little.
That said, the code below may be good enough for certain applications. The better, although longer, course of action would be to use code from apktool which seems to be able to handle all APKs.
ApkManifestFetcher
object ApkManifestFetcher {
fun getManifestXmlFromFile(apkFile: File) =
getManifestXmlFromInputStream(FileInputStream(apkFile))
fun getManifestXmlFromFilePath(apkFilePath: String) =
getManifestXmlFromInputStream(FileInputStream(File(apkFilePath)))
fun getManifestXmlFromInputStream(ApkInputStream: InputStream): String? {
ZipInputStream(ApkInputStream).use { zipInputStream: ZipInputStream ->
while (true) {
val entry = zipInputStream.nextEntry ?: break
if (entry.name == "AndroidManifest.xml") {
return decompressXML(zipInputStream.readBytes())
}
}
}
return null
}
/**
* Binary XML name space starts
*/
private const val startNameSpace = 0x00100100
/**
* Binary XML name space ends
*/
private const val endNameSpace = 0x00100101
/**
* Binary XML start Tag
*/
private const val startTag = 0x00100102
/**
* Binary XML end Tag
*/
private const val endTag = 0x00100103
/**
* Binary XML text Tag
*/
private const val textTag = 0x00100104
/*
* Flag for UTF-8 encoded file. Default is UTF-16.
*/
private const val FLAG_UTF_8 = 0x00000100
/**
* Reference var for spacing
* Used in prtIndent()
*/
private const val spaces = " "
// Flag if the manifest is in UTF-8 but we don't really handle it.
private var mIsUTF8 = false
/**
* Parse the 'compressed' binary form of Android XML docs
* such as for AndroidManifest.xml in .apk files
* Source: http://stackoverflow.com/questions/2097813/how-to-parse-the-androidmanifest-xml-file-inside-an-apk-package/4761689#4761689
*
* #param xml Encoded XML content to decompress
*/
private fun decompressXML(xml: ByteArray): String {
val resultXml = StringBuilder()
/*
Compressed XML file/bytes starts with 24x bytes of data
9 32 bit words in little endian order (LSB first):
0th word is 03 00 (Magic number) 08 00 (header size words 0-1)
1st word is the size of the compressed XML. This should equal size of xml array.
2nd word is 01 00 (Magic number) 1c 00 (header size words 2-8)
3rd word is offset of byte after string table
4th word is number of strings in string table
5th word is style count
6th word are flags
7th word string table offset
8th word is styles offset
[string index table (little endian offset into string table)]
[string table (two byte length followed by text for each entry UTF-16, nul)]
*/
mIsUTF8 = (lew(xml, 24) and FLAG_UTF_8) != 0
val numbStrings = lew(xml, 4 * 4)
// StringIndexTable starts at offset 24x, an array of 32 bit LE offsets
// of the length/string data in the StringTable.
val sitOff = 0x24 // Offset of start of StringIndexTable
// StringTable, each string is represented with a 16 bit little endian
// character count, followed by that number of 16 bit (LE) (Unicode) chars.
val stOff = sitOff + numbStrings * 4 // StringTable follows StrIndexTable
// XMLTags, The XML tag tree starts after some unknown content after the
// StringTable. There is some unknown data after the StringTable, scan
// forward from this point to the flag for the start of an XML start tag.
var xmlTagOff = lew(xml, 3 * 4) // Start from the offset in the 3rd word.
// Scan forward until we find the bytes: 0x02011000(x00100102 in normal int)
run {
var ii = xmlTagOff
while (ii < xml.size - 4) {
if (lew(xml, ii) == startTag) {
xmlTagOff = ii
break
}
ii += 4
}
}
/*
XML tags and attributes:
Every XML start and end tag consists of 6 32 bit words:
0th word: 02011000 for startTag and 03011000 for endTag
1st word: a flag?, like 38000000
2nd word: Line of where this tag appeared in the original source file
3rd word: 0xFFFFFFFF ??
4th word: StringIndex of NameSpace name, or 0xFFFFFF for default NS
5th word: StringIndex of Element Name
(Note: 01011000 in 0th word means end of XML document, endDocTag)
Start tags (not end tags) contain 3 more words:
6th word: 14001400 meaning??
7th word: Number of Attributes that follow this tag(follow word 8th)
8th word: 00000000 meaning??
Attributes consist of 5 words:
0th word: StringIndex of Attribute Name's Namespace, or 0xFFFFFF
1st word: StringIndex of Attribute Name
2nd word: StringIndex of Attribute Value, or 0xFFFFFFF if ResourceId used
3rd word: Flags?
4th word: str ind of attr value again, or ResourceId of value
Text blocks consist of 7 words
0th word: The text tag (0x00100104)
1st word: Size of the block (28 bytes)
2nd word: Line number
3rd word: 0xFFFFFFFF
4th word: Index into the string table
5th word: Unknown
6th word: Unknown
startNameSpace blocks consist of 6 words
0th word: The startNameSpace tag (0x00100100)
1st word: Size of the block (24 bytes)
2nd word: Line number
3rd word: 0xFFFFFFFF
4th word: Index into the string table for the prefix
5th word: Index into the string table for the URI
endNameSpace blocks consist of 6 words
0th word: The endNameSpace tag (0x00100101)
1st word: Size of the block (24 bytes)
2nd word: Line number
3rd word: 0xFFFFFFFF
4th word: Index into the string table for the prefix
5th word: Index into the string table for the URI
*/
// Step through the XML tree element tags and attributes
var off = xmlTagOff
var indent = 0
while (off < xml.size) {
val tag0 = lew(xml, off)
val nameSi = lew(xml, off + 5 * 4)
when (tag0) {
startTag -> {
val numbAttrs = lew(xml, off + 7 * 4) // Number of Attributes to follow
off += 9 * 4 // Skip over 6+3 words of startTag data
val name = compXmlString(xml, sitOff, stOff, nameSi)
// Look for the Attributes
val sb = StringBuffer()
for (ii in 0 until numbAttrs) {
val attrNameSi = lew(xml, off + 1 * 4) // AttrName String Index
val attrValueSi = lew(xml, off + 2 * 4) // AttrValue Str Ind, or 0xFFFFFF
val attrResId = lew(xml, off + 4 * 4) // AttrValue ResourceId or dup AttrValue StrInd
off += 5 * 4 // Skip over the 5 words of an attribute
val attrName = compXmlString(xml, sitOff, stOff, attrNameSi)
val attrValue = if (attrValueSi != -1)
compXmlString(xml, sitOff, stOff, attrValueSi)
else
"resourceID 0x" + Integer.toHexString(attrResId)
sb.append(" $attrName=\"$attrValue\"")
}
resultXml.append(prtIndent(indent, "<$name$sb>"))
indent++
}
endTag -> {
indent--
off += 6 * 4 // Skip over 6 words of endTag data
val name = compXmlString(xml, sitOff, stOff, nameSi)
resultXml.append(prtIndent(indent, "</$name>")
)
}
textTag -> { // Text that is hanging out between start and end tags
val text = compXmlString(xml, sitOff, stOff, lew(xml, off + 16))
resultXml.append(text)
off += lew(xml, off + 4)
}
startNameSpace -> {
//Todo startNameSpace and endNameSpace are effectively skipped, but they are not handled.
off += lew(xml, off + 4)
}
endNameSpace -> {
off += lew(xml, off + 4)
}
else -> {
Log.d(
"Applog", " Unrecognized tag code '" + Integer.toHexString(tag0)
+ "' at offset " + off
)
}
}
}
return resultXml.toString()
}
/**
* Tool Method for decompressXML();
* Compute binary XML to its string format
* Source: Source: http://stackoverflow.com/questions/2097813/how-to-parse-the-androidmanifest-xml-file-inside-an-apk-package/4761689#4761689
*
* #param xml Binary-formatted XML
* #param sitOff
* #param stOff
* #param strInd
* #return String-formatted XML
*/
private fun compXmlString(
xml: ByteArray, #Suppress("SameParameterValue") sitOff: Int,
stOff: Int,
strInd: Int
): String? {
if (strInd < 0) return null
val strOff = stOff + lew(xml, sitOff + strInd * 4)
return compXmlStringAt(xml, strOff)
}
/**
* Tool Method for decompressXML();
* Apply indentation
*
* #param indent Indentation level
* #param str String to indent
* #return Indented string
*/
private fun prtIndent(indent: Int, str: String): String {
return spaces.substring(0, min(indent * 2, spaces.length)) + str
}
/**
* Tool method for decompressXML()
* Return the string stored in StringTable format at
* offset strOff. This offset points to the 16 bit string length, which
* is followed by that number of 16 bit (Unicode) chars.
*
* #param arr StringTable array
* #param strOff Offset to get string from
* #return String from StringTable at offset strOff
*/
private fun compXmlStringAt(arr: ByteArray, strOff: Int): String {
var start = strOff
var charSetUsed: Charset = Charsets.UTF_16LE
val byteLength = if (mIsUTF8) {
charSetUsed = Charsets.UTF_8
start += 2
arr[strOff + 1].toInt() and 0xFF
} else { // UTF-16LE
start += 2
((arr[strOff + 1].toInt() and 0xFF shl 8) or (arr[strOff].toInt() and 0xFF)) * 2
}
return String(arr, start, byteLength, charSetUsed)
}
/**
* Return value of a Little Endian 32 bit word from the byte array
* at offset off.
*
* #param arr Byte array with 32 bit word
* #param off Offset to get word from
* #return Value of Little Endian 32 bit word specified
*/
private fun lew(arr: ByteArray, off: Int): Int {
return (arr[off + 3] shl 24 and -0x1000000 or ((arr[off + 2] shl 16) and 0xff0000)
or (arr[off + 1] shl 8 and 0xff00) or (arr[off].toInt() and 0xFF))
}
private infix fun Byte.shl(i: Int): Int = (this.toInt() shl i)
}
In Kotlin, as in Java, there is no overflow error in arithmetic operations. I know that there are special Java operations that test overflow and throw exceptions that need to be handled.
I would want a simpler way. So I thought of a model, which is not so efficient, but it is very simple and effective.
Suppose someone wants to test a 2 long numbers multiplication: a * b
I use
if ( a.doDouble()* b.toDouble() - a*b != 0.0 )
println("Overflow")
else
println("Ok")
The justification is simple. Within the universe of Long the difference between a number and its Double is always 0, even at extreme values, when the Double does not reach all precision. In this case, adding or subtracting a small number does not even change the equality test:.
var l1= -Long.MAX_VALUE
var d1 = l1.toDouble()
if (d1-l1==0.0) println("-MaxLong")
if (d1+100-l1==0.0) println("it still -MaxLong")
var l2= Long.MAX_VALUE
var d2 =l2.toDouble()
if (d2-l2==0.0) println("MaxLong")
if (d2+100-l2==0.0) println("it still MaxLong")
This generates the output:
-MaxLong
it still -MaxLong
MaxLong
it still MaxLong
Is it correct or I'm missing something?
Even if it's correct, is there any other solution better than this?
Update 1: Notice that other possibility is testing if Double calculation is greater that longValue.MAXVALUE. However, it fails!
var n1= Long.MAX_VALUE/2+1
var n2= Long.MAX_VALUE/2+1
println((n1.toDouble()+n2.toDouble()) -
Long.MAX_VALUE.toDouble()==0.0)
println((n1.toDouble()+n2.toDouble()) > Long.MAX_VALUE.toDouble())
It prints:
true
false
Update 2: Although my solution seems to work, it doesn't!
Alexey Romanov, points me in his accepted answer the following situation:
val lo1 = Long.MAX_VALUE - 600
val lo2 = 100L
var do1: Double = lo1.toDouble()
var do2:Double = lo2.toDouble()
var d= do1+do2
var l=lo1+lo2
println(d-l==0.0)
As the result is inside Long range, it should gives true, but it gives false, because Double calculation is not exact!
As he said, the best way is really using special functions like multiplyExact encapsulated in an user function.
Unfortunately, its resources only can be used in Android from API 24 onwards, so it rests the other solution from Alexey Romanov, that consists in test the inverse operation.
So, for instance, in the multiplication one should do:
var a = Long.MIN_VALUE
var b = -1L
var c = a*b
if (b!=0 && c/b != a)
println("overflow $c")
else
println("ok $c")
It prints overflow -9223372036854775808
Among traditional operations, there are usually concerns with addition, subtraction, and multiplication, which are the object of the functions addExact, subtractExact, multipyExact functions, that are easily emulated using inverse operations, as cited.
Negation (inv()) also has the negateExact function to deal with the negation of Long.MIN_VALUE, which is invalid as it has no positive counterpart. Less commented is the division, which has no specialized function in Java to lead with overflow. However it gives problem in a single case: Long.MIN_VALUE / -1 is invalid.
Within the universe of Long the difference between a number and its Double is always 0
No, not really.
println(Long.MAX_VALUE)
println(BigDecimal(Long.MAX_VALUE.toDouble()))
prints
9223372036854775807
9223372036854775808
You tried to check this:
var l2= Long.MAX_VALUE
var d2 =l2.toDouble()
if (d2-l2==0.0) println("MaxLong")
But the problem is that arithmetic operations on JVM (and in most languages, really) can only work on values of the same type, so the compiler inserts toDouble() and you really calculate d2 - l2.toDouble().
If you want a simple test, you can do
val product = a*b
if ((b != 0 && product/b != a) || (a == Long.MIN_VALUE && b == -1)) {
println("Overflow")
} else {
// can use product here
println("OK")
}
but really, using multiplyExact instead of doing it manually makes more sense. Or use Kotlin's nullable types and define
fun multiplyExact(x: Long, y: Long): Long? =
try { java.math.multiplyExact(x, y) } catch (e: ArithmeticException) { null }
EDIT: to demonstrate a fault in your test, consider addition (I am pretty sure it's wrong for multiplication as well, but it's harder to find suitable numbers):
val largeNumber = Long.MAX_VALUE - 600
val smallNumber = 100L
// prints true, even though there's no overflow
println((largeNumber.toDouble() + smallNumber.toDouble()) - (largeNumber + smallNumber) != 0.0)
The reason is that largeNumber.toDouble() + smallNumber.toDouble() == largeNumber.toDouble() while (largeNumber + smallNumber).toDouble() == Long.MAX_VALUE.toDouble().
You should know that Long DataType has a fixed number of bytes Oracle Docs
The long data type is a 64-bit signed two's complement
integer. It has a minimum value of -9,223,372,036,854,775,808 and a
maximum value of 9,223,372,036,854,775,807 (inclusive). Use this data
type when you need a range of values wider than those provided by int.
//if it is not within the range then its an overflow (infinity/undefined)
if(a*b < Long.MIN_VALUE || a*b > Long.MAX_VALUE)
println("Overflow")
else
println("Ok")
Edit
Truly and unfortunately the above method is not reliable. See below table from a run test on android studio with JDK 8
##### Overflow Test #########
Long.MAX_VALUE = 9223372036854775807
Long.MIN_VALUE = -9223372036854775808
Long.MAX_VALUE - 2 = 9223372036854775805
Long.MAX_VALUE - 1 = 9223372036854775806
Long.MAX_VALUE - 0 = 9223372036854775807
Long.MAX_VALUE + 0 = 9223372036854775807
Long.MAX_VALUE + 1 = -9223372036854775808
Long.MAX_VALUE + 2 = -9223372036854775807
Long.MAX_VALUE * 2 = -2
Long.MAX_VALUE / 2 = 4611686018427387903
Long.MIN_VALUE - 2 = 9223372036854775806
Long.MIN_VALUE - 1 = 9223372036854775807
Long.MIN_VALUE - 0 = -9223372036854775808
Long.MIN_VALUE + 0 = -9223372036854775808
Long.MIN_VALUE + 1 = -9223372036854775807
Long.MIN_VALUE + 2 = -9223372036854775806
Long.MIN_VALUE * 2 = 0
Long.MIN_VALUE / 2 = -4611686018427387904
Long.MIN_VALUE + Long.MAX_VALUE = -1
Long.MAX_VALUE - Long.MIN_VALUE = -1
Long.MAX_VALUE * Long.MIN_VALUE = -9223372036854775808
Long.MAX_VALUE / Long.MIN_VALUE = 0
Long.MIN_VALUE / Long.MAX_VALUE = -1
Long.MAX_VALUE + Long.MAX_VALUE = -2
Long.MIN_VALUE + Long.MIN_VALUE = 0
Double.MAX_VALUE = 1.7976931348623157E308
Double.MAX_VALUE * 2 = Infinity
Double.MAX_VALUE + Double.MAX_VALUE = Infinity
Long.MAX_VALUE * Double.MAX_VALUE = Infinity
Double.MAX_VALUE > Long.MAX_VALUE = true
Double.MIN_VALUE < Long.MIN_VALUE = true
Looking at the log you would notice anytime Long.MAX_VALUE reaches its peak instead of hitting Infinity like Double.MAX_VALUE, the bit is switched and its next value becomes Long.MIN_VALUE and it goes on and on like that.
So now we see why the above method isn't reliable. Hence we can assume that in java Long is a DataType with zero Infinity.
Method modified introducing floating point constants in-between
//using floating points forces larger memory allocation
//this prevents bit switch after crossing max or min value of Long
if(a * 1.0 * b < Long.MIN_VALUE || a * 1.0 * b > Long.MAX_VALUE)
println("Either a Double or Long Overflow")
else
println("Ok")
I am working on a C++ Android NDK project that relies on a SQLite3 database as a data repository. The database has three tables; column counts are 6, 8, and 6; row counts are 240, ~12.5 million, and ~4 million.
The SQLite driver is being compiled, unmodified, directly into my native library from the SQLite3 amalgamation source code, version 3.19.3.
My problem is that when running the native library, the database query results in "SQL logic error or missing database" . I know the database is where I expect, and is accessible. If I execute the same query against the same database in a desktop environment (rather than on the mobile device), I see the expected results. Furthermore, if I execute the same query on the mobile device using a database housing a subset of the data in the repository (~300 total records), I see the expected results.
Any ideas?
For reference, here is the query I'm using:
WITH
weather_distance AS
(SELECT latitude, longitude, temperature, humidity, time, (ABS((latitude - $lat)) + ABS((longitude - $lon))) AS distance FROM Weather WHERE time BETWEEN $start AND $end),
min_weather_distance AS
(SELECT latitude, longitude, temperature, humidity, time FROM weather_distance WHERE distance = (SELECT MIN(distance) FROM weather_distance)),
solar_distance AS
(SELECT latitude, longitude, ghi, (time - 1800) AS time, (ABS((latitude - $lat)) + ABS((longitude - $lon))) AS distance FROM SolarData WHERE time BETWEEN $start AND $end),
min_solar_distance AS
(SELECT latitude, longitude, ghi, time FROM solar_distance WHERE distance = (SELECT MIN(distance) FROM solar_distance))
SELECT s.time, s.ghi, w.temperature, w.humidity FROM min_weather_distance w INNER JOIN min_solar_distance s ON w.time = s.time ORDER BY s.time ASC
and the code (C++) I'm using to make the query:
const char* getEnvQuery =
"WITH "
"weather_distance AS "
"(SELECT latitude, longitude, temperature, humidity, time, (ABS((latitude - $lat)) + ABS((longitude - $lon))) AS distance FROM Weather WHERE time BETWEEN $start AND $end), "
"min_weather_distance AS "
"(SELECT latitude, longitude, temperature, humidity, time FROM weather_distance WHERE distance = (SELECT MIN(distance) FROM weather_distance)), "
"solar_distance AS "
"(SELECT latitude, longitude, ghi, (time - 1800) AS time, (ABS((latitude - $lat)) + ABS((longitude - $lon))) AS distance FROM SolarData WHERE time BETWEEN $start AND $end), "
"min_solar_distance AS (SELECT latitude, longitude, ghi, time FROM solar_distance WHERE distance = (SELECT MIN(distance) FROM solar_distance)) "
"SELECT s.time, s.ghi, w.temperature, w.humidity FROM min_weather_distance w INNER JOIN min_solar_distance s ON w.time = s.time ORDER BY s.time ASC;\0";
sqlite3_stmt* getEnvStmt;
prepareSqlStatement(getEnvQuery, &getEnvStmt, envdbhandle, "Error preparing SQL statement to retrieve environmental data. SQLite return code: ");
sqlite3_bind_double(getEnvStmt, 1, iter->latitude); //iter is defined above quoted code
sqlite3_bind_double(getEnvStmt, 2, iter->longitude);
sqlite3_bind_double(getEnvStmt, 3, iter->startTime);
sqlite3_bind_double(getEnvStmt, 4, iter->endTime);
__android_log_print(ANDROID_LOG_DEBUG, "SPP/getEnvironment", "Bound parameters: lat=%f, lon=%f, start=%ld, end=%ld", iter->latitude, iter->longitude, iter->startTime, iter->endTime);
int rc = sqlite3_step(getEnvStmt);
__android_log_print(ANDROID_LOG_DEBUG, "SPP/getEnvironment", "step(getEnvStmt) = %d", rc);
int errCode = sqlite3_extended_errcode(envdbhandle);
__android_log_print(ANDROID_LOG_DEBUG, "SPP/getEnvironment", "Most recent SQLITE error code: %s. Message: %s", sqlite3_errstr(errCode), sqlite3_errmsg(envdbhandle));
while(rc == SQLITE_ROW)
{
EnvironmentDatum envData;
int dbTime = sqlite3_column_int(getEnvStmt, 0);
envData.UnixTime = timeconvert::secondsOfYearToUNIXTime(dbTime, year);
__android_log_print(ANDROID_LOG_DEBUG, "SPP/getEnvironment", "EnvironmentDatum dbTime=%d, UnixTime=%f", dbTime, envData.UnixTime);
envData.GHI = sqlite3_column_double(getEnvStmt, 1);
envData.Temperature = sqlite3_column_double(getEnvStmt, 2);
envData.Humidity = sqlite3_column_double(getEnvStmt, 3);
envCollection.push_back(envData);
rc = sqlite3_step(getEnvStmt);
}
sqlite3_finalize(getEnvStmt);
Important debugging info:
sqlite3_stmt* verQueryStmt;
prepareSqlStatement("select sqlite_version();\0", &verQueryStmt, envdbhandle, "Error getting driver version. Error code:");
sqlite3_step(verQueryStmt);
std::string sqliteVersion = parseSqliteStringColumn(verQueryStmt, 0);
sqlite3_finalize(verQueryStmt);
__android_log_print(ANDROID_LOG_DEBUG, "SPP/buildScenario", "sqlite version=%s", sqliteVersion.c_str()); // outputs "sqlite version=3.19.3"
__android_log_print(ANDROID_LOG_DEBUG, "SPP/buildScenario", "env db readability=%s", (sqlite3_db_readonly(envdbhandle, "main") == 1 ? "READONLY" : (sqlite3_db_readonly(envdbhandle, "main") == 0 ? "READ/WRITE" : "NOT CONNECTED"))); // outputs "READ/WRITE"
Per request, here is prepareStatement:
static int prepareSqlStatement(const char* query, sqlite3_stmt** statement, sqlite3* db, const char* failMsg)
{
int returnCode = sqlite3_prepare(db, query, -1, statement, NULL);
if(returnCode != SQLITE_OK || statement == NULL)
{
int errCode = sqlite3_extended_errcode(dbhandle);
std::cout << "Most recent SQLITE error code: " << sqlite3_errstr(errCode) << ". Message: " << sqlite3_errmsg(dbhandle) << std::endl;
reportError(failMsg, -1 * returnCode);
}
return returnCode;
}
And in anticipation, here is reportError:
static void reportError(const char* message, int errorCode)
{
std::stringstream ss;
ss << message << errorCode;
throw std::runtime_error(ss.str());
}
When intermediate results of the query become too large, the database must swap out some data into a temporary file.
6410 is SQLITE_IOERR_GETTEMPPATH, which means that none of the temporary file storage locations are accessible.
Android doesn't have any of the default Unix paths. The built-in database framework compiles its copy of the SQLite library with SQLITE_TEMP_STORE = 3. If you want to get actual temporary files instead, you should put them into whatever directory is returned by Context.getCacheDir(); this would require setting the SQLITE_TMPDIR environment variable.
I have problem in dealing with scientific notation. In order to simplifeid problem, I used constants instead of variables, and the statement as follows :
double temp = 211.751 * 110 * 500 * 0.9971;
Log.e("Temp : ", Double.toString(temp));
the result is : 1.1612530715499999E7
what I want is : 11,612,530.72
My question is how to convert the result into normal expression and formatted to 2 decimal places.
Double.parseDouble("9.78313E+2");
gives you
978.313
and then
NumberFormat formatter = new DecimalFormat("#0.00");
System.out.println(formatter.format(number));
try it out
// try this
double temp = 211.751 * 110 * 500 * 0.9971;
DecimalFormat format = new DecimalFormat("#00,000,000.00");
Log.e("Temp : ",format.format(temp));
I have to convert an int to an hex value. This is for example the int value:
int_value = -13516;
To convert to a hex value i do:
hex_value = Integer.toHexString(int_value);
The value that I should get is : -34CC (I don't know if i should make it positive).
The thing is that doing the conversion that way, the value that I get is: ffff cb34
Can't I use this function to make this conversion?
Documentation says Integer.toHexString returns the hexadecimal representation of the int as an unsigned value.
I believe Integer.toString(value, 16) will accomplish what you want.
public static int convert(int n) {
return Integer.valueOf(String.valueOf(n), 16);
}
// in onstart:
Log.v("TAG", convert(20) + ""); // 32
Log.v("TAG", convert(54) + ""); // 84
From: Java Convert integer to hex integer
Both Integer.toHexString, as well as String.format("%x") do not support signs. To solve the problem, you can use a ternary expression:
int int_value = -13516;
String hex_value = int_value < 0
? "-" + Integer.toHexString(-int_value)
: Integer.toHexString(int_value);
String.format("#%06X", (0xFFFFFF & colorYellow));
Output: #FFC107
Go through following code for Integer to hex and Hex to integer Conversion
public class MainActivity extends Activity {
int number;
#Override
protected void onCreate(Bundle savedInstanceState) {
super.onCreate(savedInstanceState);
setContentView(R.layout.activity_main);
number = 678668;
Log.i("ACT", "Integer Number " + number);
/**
* Code for convert integer number to hex number. two mwthods.
*/
Log.i("ACT", String.format("#%x", number)); // use lower case x for
// lowercase hex
Log.i("ACT", "#" + Integer.toHexString(number));
/**
* Code for convert hex number to integer number
*/
String hex = Integer.toHexString(number).replace("/^#/", "");
int intValue = Integer.parseInt(hex, 16);
Log.i("ACT", "Integer Number " + intValue);
}
}
I don't think the above answers would give you the exact value for the signed bits. For example the value of 11 is 0B but the value of -11 would be F5 and not -B since 2's complement gets into the game to solve this i have modified the above answer
int int_value = -11;
String hex_value = int_value < 0
? Integer.toHexString(int_value+65536) : Integer.toHexString(int_value);
String shortHexString = hex_value.substring(2);
where, 65536 is 2^16 now you can get the expected results . Happy coding :)
List item