I would like to implement ARCore with Twilio's video call. The documentation says this is possible but I could not figure out how to do it. Can Anyone tell me what I'm doing wrong?
This is my activity:
class MixActivity : AppCompatActivity() {
private lateinit var mArFragment: ArFragment
private lateinit var mVideoView: ArSceneView
private var mScreenVideoTrack: LocalVideoTrack? = null
override fun onCreate(savedInstanceState: Bundle?) {
super.onCreate(savedInstanceState)
setContentView(R.layout.activity_mix)
mArFragment = ar_fragment as ArFragment
mVideoView = mArFragment.arSceneView
mScreenVideoTrack = LocalVideoTrack.create(this, true,
ViewCapturer(mVideoView)
)
} }
This is view:
<?xml version="1.0" encoding="utf-8"?>
<RelativeLayout xmlns:android="http://schemas.android.com/apk/res/android"
android:id="#+id/container"
android:layout_width="match_parent"
android:layout_height="match_parent">
<fragment
android:id="#+id/ar_fragment"
android:name="com.google.ar.sceneform.ux.ArFragment"
android:layout_width="match_parent"
android:layout_height="match_parent"/>
</RelativeLayout>
And VideoCapture:
internal class ViewCapturer(private val view: View) : VideoCapturer, PixelCopy.OnPixelCopyFinishedListener {
private val handler = Handler(Looper.getMainLooper())
private var videoCapturerListener: VideoCapturer.Listener? = null
private val started = AtomicBoolean(false)
private lateinit var mViewBitmap: Bitmap
private val viewCapturer = object : Runnable {
override fun run() {
val dropFrame = view.width == 0 || view.height == 0
// Only capture the view if the dimensions have been established
if (!dropFrame) {
// Draw view into bitmap backed canvas
val measuredWidth = View.MeasureSpec.makeMeasureSpec(
view.width,
View.MeasureSpec.EXACTLY
)
val measuredHeight = View.MeasureSpec.makeMeasureSpec(
view.height,
View.MeasureSpec.EXACTLY
)
view.measure(measuredWidth, measuredHeight)
view.layout(0, 0, view.measuredWidth, view.measuredHeight)
mViewBitmap = Bitmap.createBitmap(
view.width, view.height,
Bitmap.Config.ARGB_8888
)
val viewCanvas = Canvas(mViewBitmap)
view.draw(viewCanvas)
// Extract the frame from the bitmap
val bytes = mViewBitmap.byteCount
val buffer = ByteBuffer.allocate(bytes)
mViewBitmap.copyPixelsToBuffer(buffer)
val array = buffer.array()
val captureTimeNs = TimeUnit.MILLISECONDS.toNanos(SystemClock.elapsedRealtime())
// Create video frame
val dimensions = VideoDimensions(view.width, view.height)
val videoFrame = VideoFrame(
array,
dimensions, VideoFrame.RotationAngle.ROTATION_0, captureTimeNs
)
// Notify the listener
if (started.get()) {
videoCapturerListener!!.onFrameCaptured(videoFrame)
}
}
// Schedule the next capture
if (started.get()) {
handler.postDelayed(this, VIEW_CAPTURER_FRAMERATE_MS.toLong())
}
}
}
/**
* Returns the list of supported formats for this view capturer. Currently, only supports
* capturing to RGBA_8888 bitmaps.
*
* #return list of supported formats.
*/
override fun getSupportedFormats(): List<VideoFormat> {
val videoFormats = ArrayList<VideoFormat>()
val videoDimensions = VideoDimensions(view.width, view.height)
val videoFormat = VideoFormat(videoDimensions, 30, VideoPixelFormat.RGBA_8888)
videoFormats.add(videoFormat)
return videoFormats
}
/**
* Returns true because we are capturing screen content.
*/
override fun isScreencast(): Boolean {
return true
}
/**
* This will be invoked when it is time to start capturing frames.
*
* #param videoFormat the video format of the frames to be captured.
* #param listener capturer listener.
*/
override fun startCapture(videoFormat: VideoFormat, listener: VideoCapturer.Listener) {
// Store the capturer listener
this.videoCapturerListener = listener
this.started.set(true)
// Notify capturer API that the capturer has started
val capturerStarted = handler.postDelayed(
viewCapturer,
VIEW_CAPTURER_FRAMERATE_MS.toLong()
)
this.videoCapturerListener!!.onCapturerStarted(capturerStarted)
}
/**
* Stop capturing frames. Note that the SDK cannot receive frames once this has been invoked.
*/
override fun stopCapture() {
this.started.set(false)
handler.removeCallbacks(viewCapturer)
}
override fun onPixelCopyFinished(i: Int) {
// Extract the frame from the bitmap
val bytes = mViewBitmap.getByteCount()
val buffer = ByteBuffer.allocate(bytes)
mViewBitmap.copyPixelsToBuffer(buffer)
val array = buffer.array()
val captureTimeNs = TimeUnit.MILLISECONDS.toNanos(SystemClock.elapsedRealtime())
// Create video frame
val dimensions = VideoDimensions(view.width, view.height)
val videoFrame = VideoFrame(
array,
dimensions, VideoFrame.RotationAngle.ROTATION_0, captureTimeNs
)
// Notify the listener
if (started.get()) {
videoCapturerListener?.onFrameCaptured(videoFrame)
}
if (started.get()) {
handler.postDelayed(viewCapturer, VIEW_CAPTURER_FRAMERATE_MS.toLong())
}
}
companion object {
private val VIEW_CAPTURER_FRAMERATE_MS = 100
}
}
The ARCore part works but the Twilio part does not work.
I referred to another post that talked about it but it was incomplete:
Streaming CustomView ARcore with Twilio video
Related
I'm trying to run a YoloV4 model on Android that's been converted to .tflite. My input shape seems to be fine [1, 224, 224, 4] but the app crashes on my output shape. I'm using code from a Udacity course on tflite.
I get the above error when I run the following code:
class TFLiteObjectDetectionAPIModel private constructor() : Classifier {
override val statString: String
get() = TODO("not implemented") //To change initializer of created properties use File | Settings | File Templates.
private var isModelQuantized: Boolean = false
// Config values.
private var inputSize: Int = 0
// Pre-allocated buffers.
private val labels = Vector<String>()
private var intValues: IntArray? = null
// outputLocations: array of shape [Batchsize, NUM_DETECTIONS,4]
// contains the location of detected boxes
private var outputLocations: Array<Array<FloatArray>>? = null
// outputClasses: array of shape [Batchsize, NUM_DETECTIONS]
// contains the classes of detected boxes
private var outputClasses: Array<FloatArray>? = null
// outputScores: array of shape [Batchsize, NUM_DETECTIONS]
// contains the scores of detected boxes
private var outputScores: Array<FloatArray>? = null
// numDetections: array of shape [Batchsize]
// contains the number of detected boxes
private var numDetections: FloatArray? = null
private var imgData: ByteBuffer? = null
private var tfLite: Interpreter? = null
override fun recognizeImage(bitmap: Bitmap): List<Classifier.Recognition> {
// Log this method so that it can be analyzed with systrace.
Trace.beginSection("recognizeImage")
Trace.beginSection("preprocessBitmap")
// Preprocess the image data from 0-255 int to normalized float based
// on the provided parameters.
bitmap.getPixels(intValues, 0, bitmap.width, 0, 0, bitmap.width, bitmap.height)
imgData!!.rewind()
for (i in 0 until inputSize) {
for (j in 0 until inputSize) {
val pixelValue = intValues!![i * inputSize + j]
if (isModelQuantized) {
// Quantized model
imgData!!.put((pixelValue shr 16 and 0xFF).toByte())
imgData!!.put((pixelValue shr 8 and 0xFF).toByte())
imgData!!.put((pixelValue and 0xFF).toByte())
} else { // Float model
imgData!!.putFloat(((pixelValue shr 16 and 0xFF) - IMAGE_MEAN) / IMAGE_STD)
imgData!!.putFloat(((pixelValue shr 8 and 0xFF) - IMAGE_MEAN) / IMAGE_STD)
imgData!!.putFloat(((pixelValue and 0xFF) - IMAGE_MEAN) / IMAGE_STD)
}
}
}
Trace.endSection() // preprocessBitmap
// Copy the input data into TensorFlow.
Trace.beginSection("feed")
outputLocations = Array(1) { Array(NUM_DETECTIONS) { FloatArray(4) } }
outputClasses = Array(1) { FloatArray(NUM_DETECTIONS) }
outputScores = Array(1) { FloatArray(NUM_DETECTIONS) }
numDetections = FloatArray(1)
val inputArray = arrayOf<Any>(imgData!!)
val outputMap = ArrayMap<Int, Any>()
outputMap[0] = outputLocations!!
outputMap[1] = outputClasses!!
outputMap[2] = outputScores!!
outputMap[3] = numDetections!!
Trace.endSection()
// Run the inference call.
Trace.beginSection("run")
tfLite!!.runForMultipleInputsOutputs(inputArray, outputMap)
Trace.endSection()
// Show the best detections.
// after scaling them back to the input size.
val recognitions = ArrayList<Classifier.Recognition>(NUM_DETECTIONS)
for (i in 0 until NUM_DETECTIONS) {
val detection = RectF(
outputLocations!![0][i][1] * inputSize,
outputLocations!![0][i][0] * inputSize,
outputLocations!![0][i][3] * inputSize,
outputLocations!![0][i][2] * inputSize)
// SSD Mobilenet V1 Model assumes class 0 is background class
// in label file and class labels start from 1 to number_of_classes+1,
// while outputClasses correspond to class index from 0 to number_of_classes
val labelOffset = 1
recognitions.add(
Classifier.Recognition(
"" + i,
labels[outputClasses!![0][i].toInt() + labelOffset],
outputScores!![0][i],
detection))
}
Trace.endSection() // "recognizeImage"
return recognitions
}
override fun enableStatLogging(debug: Boolean) {
//Not implemented
}
override fun close() {
//Not needed.
}
override fun setNumThreads(numThreads: Int) {
if (tfLite != null) tfLite!!.setNumThreads(numThreads)
}
override fun setUseNNAPI(isChecked: Boolean) {
if (tfLite != null) tfLite!!.setUseNNAPI(isChecked)
}
companion object {
// Only return this many results.
private const val NUM_DETECTIONS = 3087
// Float model
private const val IMAGE_MEAN = 128.0f
private const val IMAGE_STD = 128.0f
/** Memory-map the model file in Assets. */
#Throws(IOException::class)
private fun loadModelFile(assets: AssetManager, modelFilename: String): MappedByteBuffer {
val fileDescriptor = assets.openFd(modelFilename)
val inputStream = FileInputStream(fileDescriptor.fileDescriptor)
val fileChannel = inputStream.channel
val startOffset = fileDescriptor.startOffset
val declaredLength = fileDescriptor.declaredLength
return fileChannel.map(FileChannel.MapMode.READ_ONLY, startOffset, declaredLength)
}
/**
* Initializes a native TensorFlow session for classifying images.
*
* #param assetManager The asset manager to be used to load assets.
* #param modelFilename The filepath of the model GraphDef protocol buffer.
* #param labelFilename The filepath of label file for classes.
* #param inputSize The size of image input
* #param isQuantized Boolean representing model is quantized or not
*/
#Throws(IOException::class)
fun create(
assetManager: AssetManager,
modelFilename: String,
labelFilename: String,
inputSize: Int,
isQuantized: Boolean): Classifier {
val d = TFLiteObjectDetectionAPIModel()
val labelsInput: InputStream?
val actualFilename = labelFilename.split("file:///android_asset/".toRegex())
.dropLastWhile { it.isEmpty() }.toTypedArray()[1]
labelsInput = assetManager.open(actualFilename)
val br: BufferedReader?
br = BufferedReader(InputStreamReader(labelsInput!!))
while (br.readLine()?.let { d.labels.add(it) } != null);
br.close()
d.inputSize = inputSize
try {
val options = Interpreter.Options()
options.setNumThreads(4)
d.tfLite = Interpreter(loadModelFile(assetManager, modelFilename), options)
} catch (e: Exception) {
throw RuntimeException(e)
}
d.isModelQuantized = isQuantized
// Pre-allocate buffers.
val numBytesPerChannel: Int = if (isQuantized) {
1 // Quantized
} else {
4 // Floating point
}
d.imgData = ByteBuffer.allocateDirect(1 * d.inputSize * d.inputSize * 3 * numBytesPerChannel)
d.imgData!!.order(ByteOrder.nativeOrder())
d.intValues = IntArray(d.inputSize * d.inputSize)
d.outputLocations = Array(1) { Array(NUM_DETECTIONS) { FloatArray(2) } }
d.outputClasses = Array(1) { FloatArray(NUM_DETECTIONS) }
d.outputScores = Array(1) { FloatArray(NUM_DETECTIONS) }
d.numDetections = FloatArray(1)
return d
}
}
When I change the outputLocation to
outputLocations = Array(1) { Array(NUM_DETECTIONS) { FloatArray(2) } }
I get the following error Cannot copy from a TensorFlowLite tensor (Identity) with shape [1, 3087, 4] to a Java object with shape [1, 3087, 2]
What is Identity and Identity_1? I've looked at my model on Netron and can see both but I'm not sure how to understand the model.
Can anyone help? Is there anything else I can change or is my model just not suitable for mobile platforms?
Cannot copy from a TensorFlowLite tensor (Identity) with shape [1, 25200, 8] to a Java object with shape [1, 80, 80, 255].
I have encountered similar problems and haven't found a solution yet
I have added CameraX to my ongoing development app a while ago. I know it was in alpha but I was ready to make the change when beta or final release will be available.
So I started working on it today. I have updated from
implementation 'androidx.camera:camera-core:1.0.0-alpha04'
implementation 'androidx.camera:camera-camera2:1.0.0-alpha04'
to this:
implementation 'androidx.camera:camera-core:1.0.0-beta01'
implementation 'androidx.camera:camera-camera2:1.0.0-beta01'
implementation 'androidx.camera:camera-lifecycle:1.0.0-beta01'
My Previous Working Code (alpha-04):
class ScannerX : AppCompatActivity() {
private lateinit var context: Context
var isOtpAuthCode = true
private val immersiveFlagTimeout = 500L
private val flagsFullscreen = View.SYSTEM_UI_FLAG_LOW_PROFILE or View.SYSTEM_UI_FLAG_FULLSCREEN or View.SYSTEM_UI_FLAG_LAYOUT_STABLE or
View.SYSTEM_UI_FLAG_IMMERSIVE_STICKY or View.SYSTEM_UI_FLAG_LAYOUT_HIDE_NAVIGATION or View.SYSTEM_UI_FLAG_HIDE_NAVIGATION
private var preview: Preview? = null
private var lensFacing = CameraX.LensFacing.BACK
private var imageAnalyzer: ImageAnalysis? = null
private lateinit var analyzerThread: HandlerThread
override fun onCreate(savedInstanceState: Bundle?) {
super.onCreate(savedInstanceState)
setContentView(R.layout.activity_scanner_x)
context = this
btnCancel.setOnClickListener {
finish()
}
analyzerThread = if (GoogleApiAvailability.getInstance().isGooglePlayServicesAvailable(context) == ConnectionResult.SUCCESS) {
HandlerThread("BarcodeFirebaseAnalyzer").apply { start() }
} else {
HandlerThread("BarcodeZxingAnalyzer").apply { start() }
}
Dexter.withActivity(this)
.withPermissions(Manifest.permission.CAMERA)
.withListener(object : MultiplePermissionsListener {
override fun onPermissionsChecked(report: MultiplePermissionsReport?) {
textureView.post {
val metrics = DisplayMetrics().also { textureView.display.getRealMetrics(it) }
val screenAspectRatio = Rational(metrics.widthPixels, metrics.heightPixels)
val previewConfig = PreviewConfig.Builder().apply {
setLensFacing(lensFacing)
// We request aspect ratio but no resolution to let CameraX optimize our use cases
setTargetAspectRatio(screenAspectRatio)
// Set initial target rotation, we will have to call this again if rotation changes
// during the lifecycle of this use case
setTargetRotation(textureView.display.rotation)
}.build()
val analyzerConfig = ImageAnalysisConfig.Builder().apply {
setLensFacing(lensFacing)
// Use a worker thread for image analysis to prevent preview glitches
setCallbackHandler(Handler(analyzerThread.looper))
// In our analysis, we care more about the latest image than analyzing *every* image
setImageReaderMode(ImageAnalysis.ImageReaderMode.ACQUIRE_LATEST_IMAGE)
// Set initial target rotation, we will have to call this again if rotation changes
// during the lifecycle of this use case
setTargetRotation(textureView.display.rotation)
}.build()
preview = AutoFitPreviewBuilder.build(previewConfig, textureView)
imageAnalyzer = ImageAnalysis(analyzerConfig).apply {
analyzer = if (GoogleApiAvailability.getInstance().isGooglePlayServicesAvailable(context) == ConnectionResult.SUCCESS) {
BarcodeFirebaseAnalyzer { qrCode ->
if (isOtpAuthCode) {
if (qrCode.startsWith("otpauth")) {
toAddAuth(qrCode)
}
} else {
toAddAuth(qrCode)
}
}
} else {
BarcodeZxingAnalyzer { qrCode ->
if (isOtpAuthCode) {
if (qrCode.startsWith("otpauth")) {
toAddAuth(qrCode)
}
} else {
toAddAuth(qrCode)
}
}
}
}
// Apply declared configs to CameraX using the same lifecycle owner
CameraX.bindToLifecycle(this#ScannerX, preview, imageAnalyzer)
}
}
override fun onPermissionRationaleShouldBeShown(permissions: MutableList<PermissionRequest>?, token: PermissionToken?) {
//
}
}).check()
}
override fun onStart() {
super.onStart()
// Before setting full screen flags, we must wait a bit to let UI settle; otherwise, we may
// be trying to set app to immersive mode before it's ready and the flags do not stick
textureView.postDelayed({
textureView.systemUiVisibility = flagsFullscreen
}, immersiveFlagTimeout)
}
override fun onDestroy() {
analyzerThread.quit()
super.onDestroy()
}
private fun toAddAuth(scannedCode: String) {
if (CameraX.isBound(imageAnalyzer)) {
CameraX.unbind(imageAnalyzer)
}
val intent = Intent()
intent.putExtra("scanResult", scannedCode)
setResult(RESULT_OK, intent)
finish()
}
companion object {
private const val RESULT_OK = 666
}
}
And the code I have changed is as follows (beta-01):
class ScannerX : AppCompatActivity() {
private lateinit var context: Context
var isOtpAuthCode = true
private val immersiveFlagTimeout = 500L
private val flagsFullscreen = View.SYSTEM_UI_FLAG_LOW_PROFILE or View.SYSTEM_UI_FLAG_FULLSCREEN or View.SYSTEM_UI_FLAG_LAYOUT_STABLE or
View.SYSTEM_UI_FLAG_IMMERSIVE_STICKY or View.SYSTEM_UI_FLAG_LAYOUT_HIDE_NAVIGATION or View.SYSTEM_UI_FLAG_HIDE_NAVIGATION
private var preview: Preview? = null
private var lensFacing = CameraSelector.DEFAULT_BACK_CAMERA
private var imageAnalyzer: ImageAnalysis? = null
private lateinit var analysisExecutor: ExecutorService
private lateinit var processCameraProvider: ListenableFuture<ProcessCameraProvider>
override fun onCreate(savedInstanceState: Bundle?) {
super.onCreate(savedInstanceState)
setContentView(R.layout.activity_scanner_x)
context = this
btnCancel.setOnClickListener {
finish()
}
Dexter.withActivity(this)
.withPermissions(Manifest.permission.CAMERA)
.withListener(object : MultiplePermissionsListener {
override fun onPermissionsChecked(report: MultiplePermissionsReport?) {
textureView.post {
analysisExecutor = Executors.newSingleThreadExecutor()
processCameraProvider = ProcessCameraProvider.getInstance(context)
preview = Preview.Builder()
.setTargetAspectRatio(AspectRatio.RATIO_16_9)
.setTargetRotation(textureView.display.rotation)
.build()
imageAnalyzer = ImageAnalysis.Builder()
.setBackpressureStrategy(ImageAnalysis.STRATEGY_KEEP_ONLY_LATEST)
.setTargetRotation(textureView.display.rotation)
.build()
if (GoogleApiAvailability.getInstance().isGooglePlayServicesAvailable(context) == ConnectionResult.SUCCESS) {
imageAnalyzer?.apply {
setAnalyzer(analysisExecutor, BarcodeFirebaseAnalyzer { qrCode ->
if (isOtpAuthCode) {
if (qrCode.startsWith("otpauth")) {
toAddAuth(qrCode)
}
} else {
toAddAuth(qrCode)
}
})
}
} else {
imageAnalyzer?.apply {
setAnalyzer(analysisExecutor, BarcodeZxingAnalyzer { qrCode ->
if (isOtpAuthCode) {
if (qrCode.startsWith("otpauth")) {
toAddAuth(qrCode)
}
} else {
toAddAuth(qrCode)
}
})
}
}
processCameraProvider.get().bindToLifecycle(this#ScannerX, lensFacing, imageAnalyzer)
}
}
override fun onPermissionRationaleShouldBeShown(permissions: MutableList<PermissionRequest>?, token: PermissionToken?) {
//
}
}).check()
}
override fun onStart() {
super.onStart()
// Before setting full screen flags, we must wait a bit to let UI settle; otherwise, we may
// be trying to set app to immersive mode before it's ready and the flags do not stick
textureView.postDelayed({
textureView.systemUiVisibility = flagsFullscreen
}, immersiveFlagTimeout)
}
override fun onDestroy() {
if (!analysisExecutor.isShutdown) {
analysisExecutor.shutdown()
}
super.onDestroy()
}
private fun toAddAuth(scannedCode: String) {
/*if (CameraX.isBound(imageAnalyzer)) {
CameraX.unbind(imageAnalyzer)
}*/
val intent = Intent()
intent.putExtra("scanResult", scannedCode)
setResult(RESULT_OK, intent)
finish()
}
companion object {
private const val RESULT_OK = 666
}
}
After I upgraded there were so many changes in library and now I cant make it work.
I also cant use Google Provided AutoFitPreview Class along with initial alpha release of this library. This was not necessary even with alpha04 since the only problem without this class was camera view little bit stretched out but scanning and analyzing worked properly.
/**
* Builder for [Preview] that takes in a [WeakReference] of the view finder and [PreviewConfig],
* then instantiates a [Preview] which automatically resizes and rotates reacting to config changes.
*/
class AutoFitPreviewBuilder private constructor(config: PreviewConfig, viewFinderRef: WeakReference<TextureView>) {
/** Public instance of preview use-case which can be used by consumers of this adapter */
val useCase: Preview
/** Internal variable used to keep track of the use case's output rotation */
private var bufferRotation: Int = 0
/** Internal variable used to keep track of the view's rotation */
private var viewFinderRotation: Int? = null
/** Internal variable used to keep track of the use-case's output dimension */
private var bufferDimens: Size = Size(0, 0)
/** Internal variable used to keep track of the view's dimension */
private var viewFinderDimens: Size = Size(0, 0)
/** Internal variable used to keep track of the view's display */
private var viewFinderDisplay: Int = -1
/** Internal reference of the [DisplayManager] */
private lateinit var displayManager: DisplayManager
/**
* We need a display listener for orientation changes that do not trigger a configuration
* change, for example if we choose to override config change in manifest or for 180-degree
* orientation changes.
*/
private val displayListener = object : DisplayManager.DisplayListener {
override fun onDisplayAdded(displayId: Int) = Unit
override fun onDisplayRemoved(displayId: Int) = Unit
override fun onDisplayChanged(displayId: Int) {
val viewFinder = viewFinderRef.get() ?: return
if (displayId == viewFinderDisplay) {
val display = displayManager.getDisplay(displayId)
val rotation = getDisplaySurfaceRotation(display)
updateTransform(viewFinder, rotation, bufferDimens, viewFinderDimens)
}
}
}
init {
// Make sure that the view finder reference is valid
val viewFinder = viewFinderRef.get() ?:
throw IllegalArgumentException("Invalid reference to view finder used")
// Initialize the display and rotation from texture view information
viewFinderDisplay = viewFinder.display.displayId
viewFinderRotation = getDisplaySurfaceRotation(viewFinder.display) ?: 0
// Initialize public use-case with the given config
useCase = Preview(config)
// Every time the view finder is updated, recompute layout
useCase.onPreviewOutputUpdateListener = Preview.OnPreviewOutputUpdateListener {
val viewFinderI = viewFinderRef.get() ?: return#OnPreviewOutputUpdateListener
Log.d(TAG, "Preview output changed. " +
"Size: ${it.textureSize}. Rotation: ${it.rotationDegrees}")
// To update the SurfaceTexture, we have to remove it and re-add it
val parent = viewFinderI.parent as ViewGroup
parent.removeView(viewFinderI)
parent.addView(viewFinderI, 0)
// Update internal texture
viewFinderI.surfaceTexture = it.surfaceTexture
// Apply relevant transformations
bufferRotation = it.rotationDegrees
val rotation = getDisplaySurfaceRotation(viewFinderI.display)
updateTransform(viewFinderI, rotation, it.textureSize, viewFinderDimens)
}
// Every time the provided texture view changes, recompute layout
viewFinder.addOnLayoutChangeListener { view, left, top, right, bottom, _, _, _, _ ->
val viewFinderII = view as TextureView
val newViewFinderDimens = Size(right - left, bottom - top)
Log.d(TAG, "View finder layout changed. Size: $newViewFinderDimens")
val rotation = getDisplaySurfaceRotation(viewFinderII.display)
updateTransform(viewFinderII, rotation, bufferDimens, newViewFinderDimens)
}
// Every time the orientation of device changes, recompute layout
// NOTE: This is unnecessary if we listen to display orientation changes in the camera
// fragment and call [Preview.setTargetRotation()] (like we do in this sample), which will
// trigger [Preview.OnPreviewOutputUpdateListener] with a new
// [PreviewOutput.rotationDegrees]. CameraX Preview use case will not rotate the frames for
// us, it will just tell us about the buffer rotation with respect to sensor orientation.
// In this sample, we ignore the buffer rotation and instead look at the view finder's
// rotation every time [updateTransform] is called, which gets triggered by
// [CameraFragment] display listener -- but the approach taken in this sample is not the
// only valid one.
displayManager = viewFinder.context
.getSystemService(Context.DISPLAY_SERVICE) as DisplayManager
displayManager.registerDisplayListener(displayListener, null)
// Remove the display listeners when the view is detached to avoid holding a reference to
// it outside of the Fragment that owns the view.
// NOTE: Even though using a weak reference should take care of this, we still try to avoid
// unnecessary calls to the listener this way.
viewFinder.addOnAttachStateChangeListener(object : View.OnAttachStateChangeListener {
override fun onViewAttachedToWindow(view: View?) =
displayManager.registerDisplayListener(displayListener, null)
override fun onViewDetachedFromWindow(view: View?) =
displayManager.unregisterDisplayListener(displayListener)
})
}
/** Helper function that fits a camera preview into the given [TextureView] */
private fun updateTransform(textureView: TextureView?, rotation: Int?, newBufferDimens: Size, newViewFinderDimens: Size) {
// This should not happen anyway, but now the linter knows
val textureViewI = textureView ?: return
if (rotation == viewFinderRotation &&
Objects.equals(newBufferDimens, bufferDimens) &&
Objects.equals(newViewFinderDimens, viewFinderDimens)) {
// Nothing has changed, no need to transform output again
return
}
if (rotation == null) {
// Invalid rotation - wait for valid inputs before setting matrix
return
} else {
// Update internal field with new inputs
viewFinderRotation = rotation
}
if (newBufferDimens.width == 0 || newBufferDimens.height == 0) {
// Invalid buffer dimens - wait for valid inputs before setting matrix
return
} else {
// Update internal field with new inputs
bufferDimens = newBufferDimens
}
if (newViewFinderDimens.width == 0 || newViewFinderDimens.height == 0) {
// Invalid view finder dimens - wait for valid inputs before setting matrix
return
} else {
// Update internal field with new inputs
viewFinderDimens = newViewFinderDimens
}
val matrix = Matrix()
Log.d(TAG, "Applying output transformation.\n" +
"View finder size: $viewFinderDimens.\n" +
"Preview output size: $bufferDimens\n" +
"View finder rotation: $viewFinderRotation\n" +
"Preview output rotation: $bufferRotation")
// Compute the center of the view finder
val centerX = viewFinderDimens.width / 2f
val centerY = viewFinderDimens.height / 2f
// Correct preview output to account for display rotation
matrix.postRotate(-viewFinderRotation!!.toFloat(), centerX, centerY)
// Buffers are rotated relative to the device's 'natural' orientation: swap width and height
val bufferRatio = bufferDimens.height / bufferDimens.width.toFloat()
val scaledWidth: Int
val scaledHeight: Int
// Match longest sides together -- i.e. apply center-crop transformation
if (viewFinderDimens.width > viewFinderDimens.height) {
scaledHeight = viewFinderDimens.width
scaledWidth = (viewFinderDimens.width * bufferRatio).roundToInt()
} else {
scaledHeight = viewFinderDimens.height
scaledWidth = (viewFinderDimens.height * bufferRatio).roundToInt()
}
// Compute the relative scale value
val xScale = scaledWidth / viewFinderDimens.width.toFloat()
val yScale = scaledHeight / viewFinderDimens.height.toFloat()
// Scale input buffers to fill the view finder
matrix.preScale(xScale, yScale, centerX, centerY)
// Finally, apply transformations to our TextureView
textureViewI.setTransform(matrix)
}
companion object {
private val TAG = AutoFitPreviewBuilder::class.java.simpleName
/** Helper function that gets the rotation of a [Display] in degrees */
fun getDisplaySurfaceRotation(display: Display?) = when(display?.rotation) {
Surface.ROTATION_0 -> 0
Surface.ROTATION_90 -> 90
Surface.ROTATION_180 -> 180
Surface.ROTATION_270 -> 270
else -> null
}
/**
* Main entry point for users of this class: instantiates the adapter and returns an instance
* of [Preview] which automatically adjusts in size and rotation to compensate for
* config changes.
*/
fun build(config: PreviewConfig, viewFinder: TextureView) =
AutoFitPreviewBuilder(config, WeakReference(viewFinder)).useCase
}
}
Please Help
I need to take as many frames as possible from the preview of the camera.
I'm doing this to start the camera using CameraX:
private fun startCamera() {
// Create configuration object for the viewfinder use case
val metrics = DisplayMetrics().also { view_finder.display.getRealMetrics(it) }
// define the screen size
val screenSize = Size(metrics.widthPixels, metrics.heightPixels)
val screenAspectRatio = Rational(metrics.widthPixels, metrics.heightPixels)
val previewConfig = PreviewConfig.Builder()
.setLensFacing(CameraX.LensFacing.BACK) // defaults to Back camera
.setTargetAspectRatio(screenAspectRatio)
.setTargetResolution(screenSize)
.build()
// Build the viewfinder use case
val preview = Preview(previewConfig)
// Every time the viewfinder is updated, recompute layout
preview.setOnPreviewOutputUpdateListener {
// To update the SurfaceTexture, we have to remove it and re-add it
val parent = view_finder.parent as ViewGroup
parent.removeView(view_finder)
parent.addView(view_finder, 0)
view_finder.surfaceTexture = it.surfaceTexture
updateTransform()
}
val analyzerConfig = ImageAnalysisConfig.Builder().apply {
// Use a worker thread for image analysis to prevent glitches
val analyzerThread = HandlerThread("AnalysisThread").apply {
start()
}
setLensFacing(CameraX.LensFacing.BACK) // defaults to Back camera
setTargetAspectRatio(screenAspectRatio)
setMaxResolution(Size(600, 320))
setCallbackHandler(Handler(analyzerThread.looper))
setImageReaderMode(ImageAnalysis.ImageReaderMode.ACQUIRE_LATEST_IMAGE)
}.build()
val analyzerUseCase = ImageAnalysis(analyzerConfig).apply {
analyzer = context?.let { FrameCapturer(it) }
}
//====================== Image Analysis Config code End==========================
CameraX.bindToLifecycle(this, preview, analyzerUseCase)
}
My FrameCapturer class is:
class FrameCapturer(context: Context): ImageAnalysis.Analyzer {
private var mListener = context as FrameCapturerListener
override fun analyze(image: ImageProxy?, rotationDegrees: Int) {
val buffer = image?.planes?.get(0)?.buffer
// Extract image data from callback object
val data = buffer?.toByteArray()
// Convert the data into an array of pixel values
//val pixels = data?.map { it.toInt() and 0xFF }
mListener.updateFps()
}
private fun ByteBuffer.toByteArray(): ByteArray {
rewind() // Rewind the buffer to zero
val data = ByteArray(remaining())
get(data) // Copy the buffer into a byte array
return data // Return the byte array
}
interface FrameCapturerListener {
fun updateFps()
}
And my updateFps function is:
fun updateFps() {
fps += 1
if (!isCountDownStarted) {
var timer = object : CountDownTimer(1000, 1000) {
override fun onTick(millisUntilFinished: Long) {
}
override fun onFinish() {
Log.d("CameraFragment", fps.toString())
fps = 0
isCountDownStarted = false
}
}.start()
isCountDownStarted = true
}
}
I'm around 16-22 fps, even if I don't convert the image to an array of byte in the FrameCapturer class. So, it seems like the Analyzer take only 20 image per second. Is there a way to increase the fps? I need to take at least 40-60 image per second, because I need to do post-processing with machine learning for each frame, so they will probably drop to 20-30 after ML analysis.
EDIT: I discover that with the Pixel 2xl I got 60fps without any drop..with my device (Xiaomi Redmi Note 5 Pro) I got only 25 fps...Can I optimize the code in any way to increase FPS?
Okay, I went through different posts and find out that depending on mobile manufacturers there can be a complications such as capture images get rotated, so you have to be aware of that. What I did was:
fun rotateBitmap(bitmap: Bitmap): Bitmap? {
val matrix = Matrix()
when (getImageOrientation(bitmap)) {
ExifInterface.ORIENTATION_NORMAL -> return bitmap
ExifInterface.ORIENTATION_FLIP_HORIZONTAL -> matrix.setScale(-1f, 1f)
ExifInterface.ORIENTATION_ROTATE_270 -> matrix.setRotate(-90f)
ExifInterface.ORIENTATION_ROTATE_180 -> matrix.setRotate(180f)
ExifInterface.ORIENTATION_ROTATE_90 -> matrix.setRotate(90f)
ExifInterface.ORIENTATION_FLIP_VERTICAL -> {
matrix.setRotate(180f)
matrix.postScale(-1f, 1f)
}
ExifInterface.ORIENTATION_TRANSPOSE -> {
matrix.setRotate(90f)
matrix.postScale(-1f, 1f)
}
ExifInterface.ORIENTATION_TRANSVERSE -> {
matrix.setRotate(-90f)
matrix.postScale(-1f, 1f)
}
else -> return bitmap
}
This worked. But then I noticed something really weird and that might be related with how I configured Camera X configuration.
With the same device I get differently rotated Bitmaps (well, this should not happen. If devices rotates image weirdly, it should rotate images in both modes - in ImageAnalysesUseCase and ImageCaptureUseCase).
So, why is this happening and how can I fix it?
Code implementation:
Binding camera X to life-cycle:
CameraX.bindToLifecycle(
this,
buildPreviewUseCase(),
buildImageAnalysisUseCase(),
buildImageCaptureUseCase()
)
Preview use case:
private fun buildPreviewUseCase(): Preview {
val previewConfig = PreviewConfig.Builder()
.setTargetAspectRatio(config.aspectRatio)
.setTargetResolution(config.resolution)
.setTargetRotation(Surface.ROTATION_0)
.setLensFacing(config.lensFacing)
.build()
return AutoFitPreviewBuilder.build(previewConfig, cameraTextureView)
}
Capture use case:
private fun buildImageCaptureUseCase(): ImageCapture {
val captureConfig = ImageCaptureConfig.Builder()
.setTargetAspectRatio(config.aspectRatio)
.setTargetRotation(Surface.ROTATION_0)
.setTargetResolution(config.resolution)
.setCaptureMode(config.captureMode)
.build()
val capture = ImageCapture(captureConfig)
manualModeTakePhotoButton.setOnClickListener {
capture.takePicture(object : ImageCapture.OnImageCapturedListener() {
override fun onCaptureSuccess(imageProxy: ImageProxy, rotationDegrees: Int) {
viewModel.onManualCameraModeAnalysis(imageProxy, rotationDegrees)
}
override fun onError(useCaseError: ImageCapture.UseCaseError?, message: String?, cause: Throwable?) {
//
}
})
}
return capture
}
Analysis use case:
private fun buildImageAnalysisUseCase(): ImageAnalysis {
val analysisConfig = ImageAnalysisConfig.Builder().apply {
val analyzerThread = HandlerThread("xAnalyzer").apply { start() }
analyzerHandler = Handler(analyzerThread.looper)
setCallbackHandler(analyzerHandler!!)
setTargetAspectRatio(config.aspectRatio)
setTargetRotation(Surface.ROTATION_0)
setTargetResolution(config.resolution)
setImageReaderMode(config.readerMode)
setImageQueueDepth(config.queueDepth)
}.build()
val analysis = ImageAnalysis(analysisConfig)
analysis.analyzer = ImageRecognitionAnalyzer(viewModel)
return analysis
}
AutoFitPreviewBuilder:
class AutoFitPreviewBuilder private constructor(config: PreviewConfig,
viewFinderRef: WeakReference<TextureView>) {
/** Public instance of preview use-case which can be used by consumers of this adapter */
val useCase: Preview
/** Internal variable used to keep track of the use-case's output rotation */
private var bufferRotation: Int = 0
/** Internal variable used to keep track of the view's rotation */
private var viewFinderRotation: Int? = null
/** Internal variable used to keep track of the use-case's output dimension */
private var bufferDimens: Size = Size(0, 0)
/** Internal variable used to keep track of the view's dimension */
private var viewFinderDimens: Size = Size(0, 0)
/** Internal variable used to keep track of the view's display */
private var viewFinderDisplay: Int = -1
/** Internal reference of the [DisplayManager] */
private lateinit var displayManager: DisplayManager
/**
* We need a display listener for orientation changes that do not trigger a configuration
* change, for example if we choose to override config change in manifest or for 180-degree
* orientation changes.
*/
private val displayListener = object : DisplayManager.DisplayListener {
override fun onDisplayAdded(displayId: Int) = Unit
override fun onDisplayRemoved(displayId: Int) = Unit
override fun onDisplayChanged(displayId: Int) {
val viewFinder = viewFinderRef.get() ?: return
if (displayId == viewFinderDisplay) {
val display = displayManager.getDisplay(displayId)
val rotation = getDisplaySurfaceRotation(display)
updateTransform(viewFinder, rotation, bufferDimens, viewFinderDimens)
}
}
}
init {
// Make sure that the view finder reference is valid
val viewFinder = viewFinderRef.get() ?:
throw IllegalArgumentException("Invalid reference to view finder used")
// Initialize the display and rotation from texture view information
viewFinderDisplay = viewFinder.display.displayId
viewFinderRotation = getDisplaySurfaceRotation(viewFinder.display) ?: 0
// Initialize public use-case with the given config
useCase = Preview(config)
// Every time the view finder is updated, recompute layout
useCase.onPreviewOutputUpdateListener = Preview.OnPreviewOutputUpdateListener {
val viewFinder =
viewFinderRef.get() ?: return#OnPreviewOutputUpdateListener
// To update the SurfaceTexture, we have to remove it and re-add it
val parent = viewFinder.parent as ViewGroup
parent.removeView(viewFinder)
parent.addView(viewFinder, 0)
viewFinder.surfaceTexture = it.surfaceTexture
bufferRotation = it.rotationDegrees
val rotation = getDisplaySurfaceRotation(viewFinder.display)
updateTransform(viewFinder, rotation, it.textureSize, viewFinderDimens)
}
// Every time the provided texture view changes, recompute layout
viewFinder.addOnLayoutChangeListener { view, left, top, right, bottom, _, _, _, _ ->
val viewFinder = view as TextureView
val newViewFinderDimens = Size(right - left, bottom - top)
val rotation = getDisplaySurfaceRotation(viewFinder.display)
updateTransform(viewFinder, rotation, bufferDimens, newViewFinderDimens)
}
// Every time the orientation of device changes, recompute layout
displayManager = viewFinder.context
.getSystemService(Context.DISPLAY_SERVICE) as DisplayManager
displayManager.registerDisplayListener(displayListener, null)
// Remove the display listeners when the view is detached to avoid
// holding a reference to the View outside of a Fragment.
// NOTE: Even though using a weak reference should take care of this,
// we still try to avoid unnecessary calls to the listener this way.
viewFinder.addOnAttachStateChangeListener(object : View.OnAttachStateChangeListener {
override fun onViewAttachedToWindow(view: View?) {
displayManager.registerDisplayListener(displayListener, null)
}
override fun onViewDetachedFromWindow(view: View?) {
displayManager.unregisterDisplayListener(displayListener)
}
})
}
/** Helper function that fits a camera preview into the given [TextureView] */
private fun updateTransform(textureView: TextureView?, rotation: Int?, newBufferDimens: Size,
newViewFinderDimens: Size) {
// This should not happen anyway, but now the linter knows
val textureView = textureView ?: return
if (rotation == viewFinderRotation &&
Objects.equals(newBufferDimens, bufferDimens) &&
Objects.equals(newViewFinderDimens, viewFinderDimens)) {
// Nothing has changed, no need to transform output again
return
}
if (rotation == null) {
// Invalid rotation - wait for valid inputs before setting matrix
return
} else {
// Update internal field with new inputs
viewFinderRotation = rotation
}
if (newBufferDimens.width == 0 || newBufferDimens.height == 0) {
// Invalid buffer dimens - wait for valid inputs before setting matrix
return
} else {
// Update internal field with new inputs
bufferDimens = newBufferDimens
}
if (newViewFinderDimens.width == 0 || newViewFinderDimens.height == 0) {
// Invalid view finder dimens - wait for valid inputs before setting matrix
return
} else {
// Update internal field with new inputs
viewFinderDimens = newViewFinderDimens
}
val matrix = Matrix()
// Compute the center of the view finder
val centerX = viewFinderDimens.width / 2f
val centerY = viewFinderDimens.height / 2f
// Correct preview output to account for display rotation
matrix.postRotate(-viewFinderRotation!!.toFloat(), centerX, centerY)
// Buffers are rotated relative to the device's 'natural' orientation: swap width and height
val bufferRatio = bufferDimens.height / bufferDimens.width.toFloat()
val scaledWidth: Int
val scaledHeight: Int
// Match longest sides together -- i.e. apply center-crop transformation
if (viewFinderDimens.width > viewFinderDimens.height) {
scaledHeight = viewFinderDimens.width
scaledWidth = Math.round(viewFinderDimens.width * bufferRatio)
} else {
scaledHeight = viewFinderDimens.height
scaledWidth = Math.round(viewFinderDimens.height * bufferRatio)
}
// Compute the relative scale value
val xScale = scaledWidth / viewFinderDimens.width.toFloat()
val yScale = scaledHeight / viewFinderDimens.height.toFloat()
// Scale input buffers to fill the view finder
matrix.preScale(xScale, yScale, centerX, centerY)
// Finally, apply transformations to our TextureView
textureView.setTransform(matrix)
}
companion object {
/** Helper function that gets the rotation of a [Display] in degrees */
fun getDisplaySurfaceRotation(display: Display?) = when(display?.rotation) {
Surface.ROTATION_0 -> 0
Surface.ROTATION_90 -> 90
Surface.ROTATION_180 -> 180
Surface.ROTATION_270 -> 270
else -> null
}
/**
* Main entrypoint for users of this class: instantiates the adapter and returns an instance
* of [Preview] which automatically adjusts in size and rotation to compensate for
* config changes.
*/
fun build(config: PreviewConfig, viewFinder: TextureView) =
AutoFitPreviewBuilder(config, WeakReference(viewFinder)).useCase
}
}
If configuration is correct (it looks okay to me), then next idea was that maybe converting captured images objects to bitmap might be faulty. Below you can see implementation.
Capture mode uses this function:
fun imageProxyToBitmap(image: ImageProxy): Bitmap {
val buffer: ByteBuffer = image.planes[0].buffer
val bytes = ByteArray(buffer.remaining())
buffer.get(bytes)
return BitmapFactory.decodeByteArray(bytes, 0, bytes.size)
}
Analysis mode uses this function:
fun toBitmapFromImage(image: Image?): Bitmap? {
try {
if (image == null || image.planes[0] == null || image.planes[1] == null || image.planes[2] == null) {
return null
}
val yBuffer = image.planes[0].buffer
val uBuffer = image.planes[1].buffer
val vBuffer = image.planes[2].buffer
val ySize = yBuffer.remaining()
val uSize = uBuffer.remaining()
val vSize = vBuffer.remaining()
val nv21 = ByteArray(ySize + uSize + vSize)
/* U and V are swapped */
yBuffer.get(nv21, 0, ySize)
vBuffer.get(nv21, ySize, vSize)
uBuffer.get(nv21, ySize + vSize, uSize)
val yuvImage = YuvImage(nv21, ImageFormat.NV21, image.width, image.height, null)
val out = ByteArrayOutputStream()
yuvImage.compressToJpeg(Rect(0, 0, yuvImage.width, yuvImage.height), 50, out)
val imageBytes = out.toByteArray()
return BitmapFactory.decodeByteArray(imageBytes, 0, imageBytes.size)
} catch (e: IllegalStateException) {
Log.e("IllegalStateException", "#ImageUtils.toBitmapFromImage(): Can't read the image file.")
return null
}
}
So, weirdly, on few devices toBitmapFromImage() sometimes comes up upwards, but at the same time (same device) imageProxyToBitmap() returns image in correct rotation - it has to be the image to bitmap functions fault, right?Why is this happening (because capture mode returns image normally) and how to fix this?
Inside onImageCaptureSuccess, get the rotationDegrees and rotate your bitmap by that degree to get the correct orientation.
override fun onImageCaptureSuccess(image: ImageProxy) {
val capturedImageBitmap = image.image?.toBitmap()?.rotate(image.imageInfo.rotationDegrees.toFloat())
mBinding.previewImage.setImageBitmap(capturedImageBitmap)
showPostClickViews()
mCurrentFlow = FLOW_CAMERA
}
toBitmap() and rotate() are extension functions.
fun Image.toBitmap(): Bitmap {
val buffer = planes[0].buffer
buffer.rewind()
val bytes = ByteArray(buffer.capacity())
buffer.get(bytes)
return BitmapFactory.decodeByteArray(bytes, 0, bytes.size)
}
fun Bitmap.rotate(degrees: Float): Bitmap =
Bitmap.createBitmap(this, 0, 0, width, height, Matrix().apply { postRotate(degrees) }, true)
CameraX returns the captured image with a rotation value in the callback, which can be used to rotate the image.
https://developer.android.com/reference/androidx/camera/core/ImageCapture.OnImageCapturedListener.html#onCaptureSuccess(androidx.camera.core.ImageProxy,%20int)
For Analyzer UseCases, you have to get rotationDegree coming through analyze method of ImageAnalysis.Analyzer and work accordingly.
Hope it helps!
Recently, I was creating a test app to familiarize myself with RecyclerViewand the Android Palette library when I came across this semantic error in my fragment that deals with Palette. When I take a picture in the fragment, it saves the photo in the File for the current orientation, for the landscape orientation but when I rotate my phone back to portrait, the File resets back to null. I have discovered this based off my Log tests and reading stack traces.
Currently I've wrapped the null absolute path in a null check to prevent further errors but I'm not sure how to proceed. Below is my Kotlin file.
class PicFragment : Fragment() {
private var imgFile: File? = null
private lateinit var cameraPic: ImageView
private lateinit var cycleLayout: View
private var swatchIndex: Int = 0
override fun onCreateView(inflater: LayoutInflater?, container: ViewGroup?, savedInstanceState: Bundle?): View? {
val view: View? = inflater?.inflate(R.layout.camera_fragment, container, false)
// init
val cameraButton: ImageButton = view!!.findViewById(R.id.click_pic)
val colorCycler: ImageButton = view.findViewById(R.id.color_clicker)
cameraPic = view.findViewById(R.id.camera_pic)
cycleLayout = view.findViewById(R.id.color_selector)
val swatchDisplay: ImageView = view.findViewById(R.id.main_color)
val swatchName: TextView = view.findViewById(R.id.main_color_name)
// restoring the picture taken if it exists
if(savedInstanceState != null){
val path: String? = savedInstanceState.getString("imageFile")
swatchIndex = savedInstanceState.getInt("swatchIndex")
if(path != null) {
val bm: Bitmap = BitmapFactory.decodeFile(path)
cameraPic.setImageBitmap(bm)
animateColorSlides(cycleLayout, duration = 500)
}
}
// taking the picture (full size)
cameraButton.setOnClickListener { _ ->
val intent = Intent(MediaStore.ACTION_IMAGE_CAPTURE)
if (intent.resolveActivity(context.packageManager) != null){
imgFile = createFileName()
val photoURI = FileProvider.getUriForFile(context, "com.github.astronoodles.provider", imgFile)
grantUriPermissions(intent, photoURI)
intent.putExtra(MediaStore.EXTRA_OUTPUT, photoURI)
startActivityForResult(intent, 3)
}
}
// Palette Button (click to go through color values)
colorCycler.setOnClickListener { _ ->
if(cameraPic.drawable is BitmapDrawable){
val img: Bitmap = (cameraPic.drawable as BitmapDrawable).bitmap
Palette.from(img).generate { palette ->
val swatches = palette.swatches
Log.d(MainActivity.TAG, "Swatch Size: ${swatches.size}")
Log.d(MainActivity.TAG, "Counter: $swatchIndex")
val hexCode = "#${Integer.toHexString(swatches[swatchIndex++ % swatches.size].rgb)}"
swatchName.text = hexCode
animateColorDrawableFade(context, swatchDisplay, hexCode)
}
} else Log.e(MainActivity.TAG, "No bitmap found! Cannot cycle images...")
}
return view
}
override fun onSaveInstanceState(outState: Bundle?) {
super.onSaveInstanceState(outState)
outState?.putString("imageFile", imgFile?.absolutePath)
outState?.putInt("swatchIndex", swatchIndex)
}
/**
* Animates the color of an ImageView using its image drawable
* #author Michael + StackOverflow
* #since 6/24/18
* #param ctx Context needed to load the animations
* #param target Target ImageView for switching colors
* #param hexCode The hex code of the colors switching in
*/
private fun animateColorDrawableFade(ctx: Context, target: ImageView, hexCode: String){
val fadeOut = AnimationUtils.loadAnimation(ctx, android.R.anim.fade_out)
val fadeIn = AnimationUtils.loadAnimation(ctx, android.R.anim.fade_in)
fadeOut.setAnimationListener(object: Animation.AnimationListener {
override fun onAnimationStart(animation: Animation?) {}
override fun onAnimationRepeat(animation: Animation?) {}
override fun onAnimationEnd(animation: Animation?) {
target.setImageDrawable(ColorDrawable(Color.parseColor(hexCode)))
target.startAnimation(fadeIn)
}
})
target.startAnimation(fadeOut)
}
/**
* Helper method for animating a layout's visibility from invisible and visible
* #author Michael
* #param layout The layout to animate
* #param duration The length of the alpha animation.
*/
private fun animateColorSlides(layout: View, duration: Long){
layout.alpha = 0f
layout.visibility = View.VISIBLE
layout.animate().alpha(1f).setListener(null).duration = duration
}
/**
* Creates an unique name for the file as suggested here using a SimpleDateFormat
* #author Michael
* #returns A (temporary?) file linking to where the photo will be saved.
*/
private fun createFileName(): File {
val timeStamp: String = SimpleDateFormat("yyyyMd_km", Locale.US).format(Date())
val jpegTitle = "JPEG_${timeStamp}_"
val directory: File = context.getExternalFilesDir(Environment.DIRECTORY_PICTURES)
try {
return File.createTempFile(jpegTitle, ".png", directory)
} catch (e: IOException) {
e.printStackTrace()
}
return File(directory, "$jpegTitle.jpg")
}
/**
* Grants URI permissions for the file provider to successfully save the full size file. <br>
* Code borrowed from https://stackoverflow.com/questions/18249007/how-to-use-support-fileprovider-for-sharing-content-to-other-apps
* #param intent The intent to send the photo
* #param uri The URI retrieved from the FileProvider
* #author Michael and Leszek
*/
private fun grantUriPermissions(intent: Intent, uri: Uri){
val intentHandleList: List<ResolveInfo> = context.packageManager.queryIntentActivities(intent,
PackageManager.MATCH_DEFAULT_ONLY)
intentHandleList.forEach {
val packageName: String = it.activityInfo.packageName
context.grantUriPermission(packageName, uri, Intent.FLAG_GRANT_WRITE_URI_PERMISSION or
Intent.FLAG_GRANT_READ_URI_PERMISSION)
}
}
override fun onActivityResult(requestCode: Int, resultCode: Int, data: Intent?) {
if(requestCode == 3 && resultCode == Activity.RESULT_OK){
val bitmap: Bitmap = BitmapFactory.decodeFile(imgFile!!.absolutePath)
cameraPic.setImageBitmap(bitmap)
animateColorSlides(cycleLayout, duration = 2000)
}
}
}
I also have my WRITE_EXTERNAL_STORAGE permission in the manifest if that helps.
Thanks.
From the Android activity lifecycle documentation, this is the relevant part:
If you override onSaveInstanceState(), you must call the superclass implementation if you want the default implementation to save the state of the view hierarchy
Which will give you something like this:
override fun onSaveInstanceState(outState: Bundle?) {
outState?.putString("imageFile", imgFile?.absolutePath)
outState?.putInt("swatchIndex", swatchIndex)
// Always call the superclass so it can save the view hierarchy state
super.onSaveInstanceState(outState)
}