Develop Reference

Display name is empty for tensor flow lite model generated using teachable machine

I am using teachable machine to do generate a tensorflow-lite model and using that model in my android app. The result from tensor flow lite model shows display name as empty. Below is a screenshot of my teachable-machine process. I have added Cats and Dogs as name though other values like index, label, score are displayed properly. Here is a sample of the output after detection
<Category "1" (displayName= score=0.99609375 index=1)>
Here is my code snippet
class ImageClassifierHelper(
var threshold: Float = 0.5f,
var numThreads: Int = 2,
var maxResults: Int = 2,
var currentDelegate: Int = 0,
var currentModel: Int = 0,
val context: Context,
val imageClassifierListener: ClassifierListener?
) {
private var imageClassifier: ImageClassifier? = null
init {
setupImageClassifier()
}
fun clearImageClassifier() {
imageClassifier = null
}
private fun setupImageClassifier() {
val optionsBuilder = ImageClassifier.ImageClassifierOptions.builder()
.setScoreThreshold(threshold)
.setMaxResults(maxResults)
val baseOptionsBuilder = BaseOptions.builder().setNumThreads(numThreads)
when (currentDelegate) {
DELEGATE_CPU -> {
// Default
}
DELEGATE_GPU -> {
if (CompatibilityList().isDelegateSupportedOnThisDevice) {
baseOptionsBuilder.useGpu()
} else {
imageClassifierListener?.onError("GPU is not supported on this device")
}
}
DELEGATE_NNAPI -> {
baseOptionsBuilder.useNnapi()
}
}
optionsBuilder.setBaseOptions(baseOptionsBuilder.build())
val modelName =
when (currentModel) {
MODEL_MOBILENETV1 -> "model.tflite"
MODEL_EFFICIENTNETV0 -> "model.tflite"
MODEL_EFFICIENTNETV1 -> "model.tflite"
MODEL_EFFICIENTNETV2 -> "model.tflite"
else -> "model.tflite"
}
try {
imageClassifier =
ImageClassifier.createFromFileAndOptions(context, modelName, optionsBuilder.build())
} catch (e: IllegalStateException) {
imageClassifierListener?.onError(
"Image classifier failed to initialize. See error logs for details"
)
Log.e(TAG, "TFLite failed to load model with error: " + e.message)
}
}
fun classify(image: Bitmap, rotation: Int) {
if (imageClassifier == null) {
setupImageClassifier()
}
// Inference time is the difference between the system time at the start and finish of the
// process
var inferenceTime = SystemClock.uptimeMillis()
// Create preprocessor for the image.
// See https://www.tensorflow.org/lite/inference_with_metadata/
// lite_support#imageprocessor_architecture
val imageProcessor =
ImageProcessor.Builder()
.build()
// Preprocess the image and convert it into a TensorImage for classification.
val tensorImage = imageProcessor.process(TensorImage.fromBitmap(image))
val imageProcessingOptions = ImageProcessingOptions.builder()
.setOrientation(getOrientationFromRotation(rotation))
.build()
val results = imageClassifier?.classify(tensorImage, imageProcessingOptions)
inferenceTime = SystemClock.uptimeMillis() - inferenceTime
imageClassifierListener?.onResults(
results,
inferenceTime
)
}
// Receive the device rotation (Surface.x values range from 0->3) and return EXIF orientation
// http://jpegclub.org/exif_orientation.html
private fun getOrientationFromRotation(rotation: Int) : ImageProcessingOptions.Orientation {
return when (rotation) {
Surface.ROTATION_270 ->
ImageProcessingOptions.Orientation.BOTTOM_RIGHT
Surface.ROTATION_180 ->
ImageProcessingOptions.Orientation.RIGHT_BOTTOM
Surface.ROTATION_90 ->
ImageProcessingOptions.Orientation.TOP_LEFT
else ->
ImageProcessingOptions.Orientation.RIGHT_TOP
}
}
interface ClassifierListener {
fun onError(error: String)
fun onResults(
results: List<Classifications>?,
inferenceTime: Long
)
}
companion object {
const val DELEGATE_CPU = 0
const val DELEGATE_GPU = 1
const val DELEGATE_NNAPI = 2
const val MODEL_MOBILENETV1 = 0
const val MODEL_EFFICIENTNETV0 = 1
const val MODEL_EFFICIENTNETV1 = 2
const val MODEL_EFFICIENTNETV2 = 3
private const val TAG = "ImageClassifierHelper"
}
}
I don't think issue is with my android app code but with the model generated from teachable machine.

The model only outputs as numbers, each of the unique number representing a category.
In the above example, you need to create a map that assigns a Category to display name: {'0': 'Cat', '1': 'dog'}.

How to test ViewModel + Flow

I'm doing a small project to learn flow and the latest Android features, and I'm currently facing the viewModel's testing, which I don't know if I'm performing correctly. can you help me with it?
Currently, I am using a use case to call the repository which calls a remote data source that gets from an API service a list of strings.
I have created a State to control the values in the view model:
data class StringItemsState(
val isLoading: Boolean = false,
val items: List<String> = emptyList(),
val error: String = ""
)
and the flow:
private val stringItemsState = StringtemsState()
private val _stateFlow = MutableStateFlow(stringItemsState)
val stateFlow = _stateFlow.asStateFlow()
and finally the method that performs all the logic in the viewModel:
fun fetchStringItems() {
try {
_stateFlow.value = stringItemsState.copy(isLoading = true)
viewModelScope.launch(Dispatchers.IO) {
val result = getStringItemsUseCase.execute()
if (result.isEmpty()) {
_stateFlow.value = stringItemsState
} else {
_stateFlow.value = stringItemsState.copy(items = result)
}
}
} catch (e: Exception) {
e.localizedMessage?.let {
_stateFlow.value = stringItemsState.copy(error = it)
}
}
}
I am trying to perform the test following the What / Where / Then pattern, but the result is always an empty list and the assert verification always fails:
private val stringItems = listOf<String>("A", "B", "C")
#Test
fun `get string items - not empty`() = runBlocking {
// What
coEvery {
useCase.execute()
} returns stringItems
// Where
viewModel.fetchStringItems()
// Then
assert(viewModel.stateFlow.value.items == stringItems)
coVerify(exactly = 1) { viewModel.fetchStringItems() }
}
Can someone help me and tell me if I am doing it correctly? Thanks.

I got different result for same input on junit tests and on Android (ART) at runtime

I have a one static method which finds matching texts on target text, but it returns different results for same input, results in Junit tests and result in Android at runtime is different.
private const val REGEX = "GE[a-zA-Z0-9]{2}\\s?([a-zA-Z0-9]{2})([0-9]{2}\\s?)([0-9]{4}\\s?){3}([0-9]{2})\\s?"
private val PATTERN: Pattern = Pattern.compile(REGEX, Pattern.MULTILINE or Pattern.CASE_INSENSITIVE)
fun find(text: String): List<String> {
val textFormatted = text.replace("\\s".toRegex(), "")
val list = ArrayList<String>()
val matcher = matcher(textFormatted)
when {
matcher.matches() -> {
list.add(textFormatted)
}
matcher.find() -> {
val partitions = findGroupMatches(matcher)
list.addAll(partitions)
}
}
return list
}
private fun findGroupMatches(matcher: Matcher): List<String> {
val partitions = mutableListOf<String>()
for (i in 0 until matcher.groupCount()) {
val partition = matcher.group(i)
if (partition != null && PATTERN.matcher(partition).matches()) {
partitions.add(partition.replace("\\s".toRegex(), ""))
}
}
return partitions
}
And the magic (imho) happens here
On JVM tests, it returns emptyList
#Test
fun `find with not valid text returns emptyList`(){
val targets = PatternUtils.find("GE03TB 7433311450666666300008")
assertTrue(targets.isEmpty()) // success
}
PatternUtils.find("GE03TB 7433311450666666300008") on Android inside `onCreate()` returns 1 size list("GE03TB743331145066666") //
Why it works like that? Is there any bug? or am I missing something?

Reason was that Pattern.java implementation by Oracle and by Google are different.
In order to work similarly I had to create new Matcher instead of reusing it. (Matcher.java has some methods with side effect)
fun find(text: String): List<String> {
val textFormatted = text.replace("\\s".toRegex(), "")
val list = ArrayList<String>()
val matcher = createMatcher(textFormatted)
when {
createMatcher(textFormatted).matches() -> {
list.add(textFormatted)
}
matcher.find() -> {
val partitions = findGroupMatches(matcher)
list.addAll(partitions)
}
}
return list
}

Issues with the latest CameraX and Barcode Scanning on some devices

So I'm working on an app that requires QR scanner as a main feature. Previously I was using camerax-alpha06 with Firebase ML vision 24.0.3 and they were working fine for months, no customer complaints about scanning issues.
Then about two weeks ago I had to change Firebase ML vision to MLKit barcode scanning (related to the Crashlytics migration - out of topic) and now some of the users who could scan in the previous version now could not. Some sample devices be Samsung Tab A7 (Android 5.1.1) and Vivo 1919 (Android 10)
This is my build.gradle section that involves this feature
def camerax_version = "1.0.0-beta11"
implementation "androidx.camera:camera-core:${camerax_version}"
implementation "androidx.camera:camera-camera2:${camerax_version}"
implementation "androidx.camera:camera-lifecycle:${camerax_version}"
implementation "androidx.camera:camera-view:1.0.0-alpha18"
implementation "androidx.camera:camera-extensions:1.0.0-alpha18"
implementation 'com.google.android.gms:play-services-mlkit-barcode-scanning:16.1.2'
This is my camera handler file
class ScanQRCameraViewHandler(
private val fragment: ScanQRDialogFragment,
private val previewView: PreviewView
) {
private val displayLayout get() = previewView
companion object {
private const val RATIO_4_3_VALUE = 4.0 / 3.0
private const val RATIO_16_9_VALUE = 16.0 / 9.0
}
private val analyzer = GMSMLKitAnalyzer(onFoundQR = { extractedString ->
fragment.verifyExtractedString(extractedString)
}, onNotFoundQR = {
resetStateToAllowNewImageStream()
})
private var cameraProviderFuture: ListenableFuture<ProcessCameraProvider>? = null
private var camera: Camera? = null
private var isAnalyzing = false
internal fun resetStateToAllowNewImageStream() {
isAnalyzing = false
}
internal fun setTorceEnable(isEnabled: Boolean) {
camera?.cameraControl?.enableTorch(isEnabled)
}
internal fun initCameraProviderIfHasNot() {
if (cameraProviderFuture == null) {
fragment.context?.let {
cameraProviderFuture = ProcessCameraProvider.getInstance(it)
val executor = ContextCompat.getMainExecutor(it)
cameraProviderFuture?.addListener({
bindPreview(cameraProviderFuture?.get(), executor)
}, executor)
}
}
}
private fun bindPreview(cameraProvider: ProcessCameraProvider?, executor: Executor) {
val metrics = DisplayMetrics().also { displayLayout.display.getRealMetrics(it) }
val screenAspectRatio = aspectRatio(metrics.widthPixels, metrics.heightPixels)
val preview = initPreview(screenAspectRatio)
val imageAnalyzer = createImageAnalyzer()
val imageAnalysis = createImageAnalysis(executor, imageAnalyzer, screenAspectRatio)
val cameraSelector = createCameraSelector()
cameraProvider?.unbindAll()
camera = cameraProvider?.bindToLifecycle(
fragment as LifecycleOwner,
cameraSelector, imageAnalysis, preview
)
}
private fun createCameraSelector(): CameraSelector {
return CameraSelector.Builder()
.requireLensFacing(CameraSelector.LENS_FACING_BACK)
.build()
}
private fun createImageAnalysis(
executor: Executor, imageAnalyzer: ImageAnalysis.Analyzer, screenAspectRatio: Int
): ImageAnalysis {
val rotation = displayLayout.rotation
val imageAnalysis = ImageAnalysis.Builder()
// .setTargetRotation(rotation.toInt())
// .setTargetAspectRatio(screenAspectRatio)
.setBackpressureStrategy(ImageAnalysis.STRATEGY_KEEP_ONLY_LATEST)
.build()
imageAnalysis.setAnalyzer(executor, imageAnalyzer)
return imageAnalysis
}
private fun createImageAnalyzer(): ImageAnalysis.Analyzer {
return ImageAnalysis.Analyzer {
isAnalyzing = true
analyzer.analyze(it)
}
}
private fun initPreview(screenAspectRatio: Int): Preview {
val preview: Preview = Preview.Builder()
//.setTargetResolution(Size(840, 840))
// .setTargetAspectRatio(screenAspectRatio)
// .setTargetRotation(displayLayout.rotation.toInt())
.build()
preview.setSurfaceProvider(previewView.surfaceProvider)
return preview
}
fun unbindAll() {
cameraProviderFuture?.get()?.unbindAll()
}
private fun aspectRatio(width: Int, height: Int): Int {
val previewRatio = width.coerceAtLeast(height).toDouble() / width.coerceAtMost(height)
if (kotlin.math.abs(previewRatio - RATIO_4_3_VALUE) <= kotlin.math.abs(previewRatio - RATIO_16_9_VALUE)) {
return AspectRatio.RATIO_4_3
}
return AspectRatio.RATIO_16_9
}
}
And my analyzer
internal class GMSMLKitAnalyzer(
private val onFoundQR: (String) -> Unit,
private val onNotFoundQR: () -> Unit
) :
ImageAnalysis.Analyzer {
private val options = BarcodeScannerOptions.Builder()
.setBarcodeFormats(Barcode.FORMAT_QR_CODE).build()
#SuppressLint("UnsafeExperimentalUsageError")
override fun analyze(imageProxy: ImageProxy) {
imageProxy.image?.let { mediaImage ->
val image = InputImage.fromMediaImage(mediaImage, imageProxy.imageInfo.rotationDegrees)
val scanner = BarcodeScanning.getClient(options)
CoroutineScope(Dispatchers.Main).launch {
val result = scanner.process(image).await()
result.result?.let { barcodes ->
barcodes.find { it.rawValue != null }?.rawValue?.let {
onFoundQR(it)
} ?: run { onNotFoundQR() }
}
imageProxy.close()
}
} ?: imageProxy.close()
}
}
The commented out lines are what I've tried to add and didn't help, some even caused issues on other (used-to-be-working) devices.
I am unsure if I misconfigure anything or not, so I would like any suggestions that would help me find the solution.
Thank you
P.S. This is my first post so if I've done anything wrong or missed something please advise.

BarcodeScanning does not work on some devices running with camera-camera2:1.0.0-beta08 version or later. You can use an earlier version of camera-camera2 to bypass this issue. For example:
See: https://developers.google.com/ml-kit/known-issues
We are working on fix internally in MLKit for the next SDK release.

Update your ML barcode scan plugin above 16.1.1
This issue was fixed in 'com.google.mlkit:barcode-scanning:16.1.1'

ML Kit Firebase translate text null Kotlin Android

I'm a newbie in android so sorry in advance, I'm inside a fun that use the user's input and translate it before sending the data to firebase Database, I think I'm doing something wrong with the variables because in the loop I can get the language text "en" but just after its null so the translation can't start..
println("SAVING DATABASE")
var theLangue:String? = null
var detect:String? = null
val languageIdentifier = LanguageIdentification.getClient()
languageIdentifier.identifyPossibleLanguages(userText)
.addOnSuccessListener { identifiedLanguages ->
for (identifedLanguage in identifiedLanguages) {
detect = identifedLanguage.languageTag
val confidence = identifedLanguage.confidence
println(" LANGUAGE DETECTED: $detect , LANGUAGE CONFIDENCE: $confidence")
}
theLangue = detect
}.addOnFailureListener {
println("problem cant translate!")
}
println("Translate pre build ${theLangue.toString()} ++ $theLangue") //todo NULL
val options = TranslatorOptions.Builder()
.setSourceLanguage(theLangue.toString()) // todo NULL
.setTargetLanguage(Locale.getDefault().displayLanguage)
.build()
val theTranslator = Translation.getClient(options)
val conditions = DownloadConditions.Builder()
.requireWifi()
.build()
theTranslator.downloadModelIfNeeded(conditions)
.addOnSuccessListener {
println("succes downloading models languages.. going to translate wait..")
theTranslator.translate(userText)
.addOnSuccessListener {
// Translation successful.
println("Succes translated text")
}
.addOnFailureListener { exception ->
// Error.
// ...
println("there is a problem failed to transalte !")
}
}
.addOnFailureListener { exception ->
// Model couldn’t be downloaded or other internal error.
// ...
println(exception.message)
println(exception.localizedMessage)
println("cant download languages models !")
}

The .addOnSuccessListener is an async call, which may not be executed right away.
A quick fix is to move the translate logic into the addOnSuccessListener callback in language identification.
`
...
languageIdentifier.identifyPossibleLanguages(userText)
.addOnSuccessListener { identifiedLanguages ->
for (identifedLanguage in identifiedLanguages) {
detect = identifedLanguage.languageTag
val confidence = identifedLanguage.confidence
println(" LANGUAGE DETECTED: $detect , LANGUAGE CONFIDENCE: $confidence")
}
theLangue = detect
val options = TranslatorOptions.Builder()
.setSourceLanguage(theLangue.toString()) // todo NULL
.setTargetLanguage(Locale.getDefault().displayLanguage)
.build()
val theTranslator = Translation.getClient(options)
...
}.addOnFailureListener {
println("problem cant translate!")
}
...
`
To improve the code readability, you could use the continueWithTask.