I did lot of reading tried so many different methods available. CameraX is producing yuv_420_888 format Image object and provides it to the ImageAnalysis.
However, there is no way to convert this to a bytebuffer in order to scale, convert to bitmap and run detection operations. I tried following and numerous other proposed techniques.
Converting ImageProxy to Bitmap
All those created grayscale (even after using all 3 planes) and some overlay color shade image. It also created glitchy outputs in-between frames sometime which I could not figure out a reason.
What’s the proper way to get a simple byte array so that it can be converted to bitmap later?
Also how to get cameraX authors attention?
fun imageProxyToByteArray(image: ImageProxy): ByteArray {
val yuvBytes = ByteArray(image.width * (image.height + image.height / 2))
val yPlane = image.planes[0].buffer
val uPlane = image.planes[1].buffer
val vPlane = image.planes[2].buffer
yPlane.get(yuvBytes, 0, image.width * image.height)
val chromaRowStride = image.planes[1].rowStride
val chromaRowPadding = chromaRowStride - image.width / 2
var offset = image.width * image.height
if (chromaRowPadding == 0) {
uPlane.get(yuvBytes, offset, image.width * image.height / 4)
offset += image.width * image.height / 4
vPlane.get(yuvBytes, offset, image.width * image.height / 4)
} else {
for (i in 0 until image.height / 2) {
uPlane.get(yuvBytes, offset, image.width / 2)
offset += image.width / 2
if (i < image.height / 2 - 2) {
uPlane.position(uPlane.position() + chromaRowPadding)
}
}
for (i in 0 until image.height / 2) {
vPlane.get(yuvBytes, offset, image.width / 2)
offset += image.width / 2
if (i < image.height / 2 - 1) {
vPlane.position(vPlane.position() + chromaRowPadding)
}
}
}
return yuvBytes
}
You can use this class ripped from Mlkit Pose Detection.
Mlkit pose detection: BitmapUtils.java
object ImageProxyUtils {
fun getByteArray(image: ImageProxy): ByteArray? {
image.image?.let {
val nv21Buffer = yuv420ThreePlanesToNV21(
it.planes, image.width, image.height
)
return ByteArray(nv21Buffer.remaining()).apply {
nv21Buffer.get(this)
}
}
return null
}
private fun yuv420ThreePlanesToNV21(
yuv420888planes: Array<Plane>,
width: Int,
height: Int
): ByteBuffer {
val imageSize = width * height
val out = ByteArray(imageSize + 2 * (imageSize / 4))
if (areUVPlanesNV21(yuv420888planes, width, height)) {
yuv420888planes[0].buffer[out, 0, imageSize]
val uBuffer = yuv420888planes[1].buffer
val vBuffer = yuv420888planes[2].buffer
vBuffer[out, imageSize, 1]
uBuffer[out, imageSize + 1, 2 * imageSize / 4 - 1]
} else {
unpackPlane(yuv420888planes[0], width, height, out, 0, 1)
unpackPlane(yuv420888planes[1], width, height, out, imageSize + 1, 2)
unpackPlane(yuv420888planes[2], width, height, out, imageSize, 2)
}
return ByteBuffer.wrap(out)
}
private fun areUVPlanesNV21(planes: Array<Plane>, width: Int, height: Int): Boolean {
val imageSize = width * height
val uBuffer = planes[1].buffer
val vBuffer = planes[2].buffer
val vBufferPosition = vBuffer.position()
val uBufferLimit = uBuffer.limit()
vBuffer.position(vBufferPosition + 1)
uBuffer.limit(uBufferLimit - 1)
val areNV21 =
vBuffer.remaining() == 2 * imageSize / 4 - 2 && vBuffer.compareTo(uBuffer) == 0
vBuffer.position(vBufferPosition)
uBuffer.limit(uBufferLimit)
return areNV21
}
private fun unpackPlane(
plane: Plane,
width: Int,
height: Int,
out: ByteArray,
offset: Int,
pixelStride: Int
) {
val buffer = plane.buffer
buffer.rewind()
val numRow = (buffer.limit() + plane.rowStride - 1) / plane.rowStride
if (numRow == 0) {
return
}
val scaleFactor = height / numRow
val numCol = width / scaleFactor
var outputPos = offset
var rowStart = 0
for (row in 0 until numRow) {
var inputPos = rowStart
for (col in 0 until numCol) {
out[outputPos] = buffer[inputPos]
outputPos += pixelStride
inputPos += plane.pixelStride
}
rowStart += plane.rowStride
}
}
}
You just need to use imageProxy.image?.toBitmap() to convert imageProxy and then convert bitmap to bytearray as follow:
Here's an example:
private fun takePhoto() {
camera_capture_button.isEnabled = false
// Get a stable reference of the modifiable image capture use case
val imageCapture = imageCapture ?: return
imageCapture.takePicture(
ContextCompat.getMainExecutor(this),
object : ImageCapture.OnImageCapturedCallback() {
#SuppressLint("UnsafeExperimentalUsageError")
override fun onCaptureSuccess(imageProxy: ImageProxy) {
val bitmapImage = imageProxy.image?.toBitmap()
val stream = ByteArrayOutputStream()
bitmapImage.compress(Bitmap.CompressFormat.PNG, 90, stream)
val image = stream.toByteArray()
}
override fun onError(exception: ImageCaptureException) {
super.onError(exception)
}
})
}
Background
We record a video of the user's face, and usually the face is located at the upper half of the video.
Later we wish to view the video, but the aspect ratio of the PlayerView might be different than the one of the video, so there needs to be some scaling and cropping.
The problem
The only way I've found to scale the PlayerView so that it will be shown in the entire space it has but keeping the aspect ratio (which will result in cropping when needed, of course) , is by using app:resize_mode="zoom" . Here's a sample of how it works with center-crop: http://s000.tinyupload.com/?file_id=00574047057406286563 . The more the Views that show the content have a similar aspect ratio, the less cropping is needed.
But this is only for the center, meaning it takes a point of 0.5x0.5 of the video, and scale-crops from that point. This causes many cases of losing the important content of the video.
For example, if we have a video that was taken in portrait, and we have a square PlayerView and want to show the top area, this is the part that will be visible:
Of course, if the content itself is square, and the views are also square, it should show the entire content, without cropping.
What I've tried
I've tried searching over the Internet, StackOverflow (here) and on Github, but I couldn't find how to do it. The only clue I've found is about AspectRatioFrameLayout and AspectRatioTextureView, but I didn't find how to use them for this task, if it's even possible.
I was told (here) that I should use a normal TextureView , and provide it directly to SimpleExoPlayer using SimpleExoPlayer.setVideoTextureView. And to set a special transformation to it using TextureView.setTransform.
After a lot of trying what is best to use (and looking at video-crop repository , SuperImageView repository , and JCropImageView repository which have examples of scale/crop of ImageView and video), I've published a working sample that seems to show the video correctly, but I'm still not sure about it, as I also use an ImageView that's shown on top of it before it starts playing (to have a nicer transition instead of black content).
Here's the current code:
class MainActivity : AppCompatActivity() {
private val imageResId = R.drawable.test
private val videoResId = R.raw.test
private val percentageY = 0.2f
private var player: SimpleExoPlayer? = null
override fun onCreate(savedInstanceState: Bundle?) {
window.setBackgroundDrawable(ColorDrawable(0xff000000.toInt()))
super.onCreate(savedInstanceState)
if (cache == null) {
cache = SimpleCache(File(cacheDir, "media"), LeastRecentlyUsedCacheEvictor(MAX_PREVIEW_CACHE_SIZE_IN_BYTES))
}
setContentView(R.layout.activity_main)
// imageView.visibility = View.INVISIBLE
imageView.setImageResource(imageResId)
imageView.doOnPreDraw {
imageView.imageMatrix = prepareMatrixForImageView(imageView, imageView.drawable.intrinsicWidth.toFloat(), imageView.drawable.intrinsicHeight.toFloat())
// imageView.imageMatrix = prepareMatrix(imageView, imageView.drawable.intrinsicWidth.toFloat(), imageView.drawable.intrinsicHeight.toFloat())
// imageView.visibility = View.VISIBLE
}
}
override fun onStart() {
super.onStart()
playVideo()
}
private fun prepareMatrix(view: View, contentWidth: Float, contentHeight: Float): Matrix {
var scaleX = 1.0f
var scaleY = 1.0f
val viewWidth = view.measuredWidth.toFloat()
val viewHeight = view.measuredHeight.toFloat()
Log.d("AppLog", "viewWidth $viewWidth viewHeight $viewHeight contentWidth:$contentWidth contentHeight:$contentHeight")
if (contentWidth > viewWidth && contentHeight > viewHeight) {
scaleX = contentWidth / viewWidth
scaleY = contentHeight / viewHeight
} else if (contentWidth < viewWidth && contentHeight < viewHeight) {
scaleY = viewWidth / contentWidth
scaleX = viewHeight / contentHeight
} else if (viewWidth > contentWidth)
scaleY = viewWidth / contentWidth / (viewHeight / contentHeight)
else if (viewHeight > contentHeight)
scaleX = viewHeight / contentHeight / (viewWidth / contentWidth)
val matrix = Matrix()
val pivotPercentageX = 0.5f
val pivotPercentageY = percentageY
matrix.setScale(scaleX, scaleY, viewWidth * pivotPercentageX, viewHeight * pivotPercentageY)
return matrix
}
private fun prepareMatrixForVideo(view: View, contentWidth: Float, contentHeight: Float): Matrix {
val msWidth = view.measuredWidth
val msHeight = view.measuredHeight
val matrix = Matrix()
matrix.setScale(1f, (contentHeight / contentWidth) * (msWidth.toFloat() / msHeight), msWidth / 2f, percentageY * msHeight) /*,msWidth/2f,msHeight/2f*/
return matrix
}
private fun prepareMatrixForImageView(view: View, contentWidth: Float, contentHeight: Float): Matrix {
val dw = contentWidth
val dh = contentHeight
val msWidth = view.measuredWidth
val msHeight = view.measuredHeight
// Log.d("AppLog", "viewWidth $msWidth viewHeight $msHeight contentWidth:$contentWidth contentHeight:$contentHeight")
val scalew = msWidth.toFloat() / dw
val theoryh = (dh * scalew).toInt()
val scaleh = msHeight.toFloat() / dh
val theoryw = (dw * scaleh).toInt()
val scale: Float
var dx = 0
var dy = 0
if (scalew > scaleh) { // fit width
scale = scalew
// dy = ((msHeight - theoryh) * 0.0f + 0.5f).toInt() // + 0.5f for rounding
} else {
scale = scaleh
dx = ((msWidth - theoryw) * 0.5f + 0.5f).toInt() // + 0.5f for rounding
}
dy = ((msHeight - theoryh) * percentageY + 0.5f).toInt() // + 0.5f for rounding
val matrix = Matrix()
// Log.d("AppLog", "scale:$scale dx:$dx dy:$dy")
matrix.setScale(scale, scale)
matrix.postTranslate(dx.toFloat(), dy.toFloat())
return matrix
}
private fun playVideo() {
player = ExoPlayerFactory.newSimpleInstance(this#MainActivity, DefaultTrackSelector())
player!!.setVideoTextureView(textureView)
player!!.addVideoListener(object : VideoListener {
override fun onVideoSizeChanged(width: Int, height: Int, unappliedRotationDegrees: Int, pixelWidthHeightRatio: Float) {
super.onVideoSizeChanged(width, height, unappliedRotationDegrees, pixelWidthHeightRatio)
Log.d("AppLog", "onVideoSizeChanged: $width $height")
val videoWidth = if (unappliedRotationDegrees % 180 == 0) width else height
val videoHeight = if (unappliedRotationDegrees % 180 == 0) height else width
val matrix = prepareMatrixForVideo(textureView, videoWidth.toFloat(), videoHeight.toFloat())
textureView.setTransform(matrix)
}
override fun onRenderedFirstFrame() {
Log.d("AppLog", "onRenderedFirstFrame")
player!!.removeVideoListener(this)
// imageView.animate().alpha(0f).setDuration(5000).start()
imageView.visibility = View.INVISIBLE
}
})
player!!.volume = 0f
player!!.repeatMode = Player.REPEAT_MODE_ALL
player!!.playRawVideo(this, videoResId)
player!!.playWhenReady = true
// player!!.playVideoFromUrl(this, "https://sample-videos.com/video123/mkv/240/big_buck_bunny_240p_20mb.mkv", cache!!)
// player!!.playVideoFromUrl(this, "https://sample-videos.com/video123/mkv/720/big_buck_bunny_720p_1mb.mkv", cache!!)
// player!!.playVideoFromUrl(this#MainActivity, "https://sample-videos.com/video123/mkv/720/big_buck_bunny_720p_1mb.mkv")
}
override fun onStop() {
super.onStop()
player!!.setVideoTextureView(null)
// playerView.player = null
player!!.release()
player = null
}
companion object {
const val MAX_PREVIEW_CACHE_SIZE_IN_BYTES = 20L * 1024L * 1024L
var cache: com.google.android.exoplayer2.upstream.cache.Cache? = null
#JvmStatic
fun getUserAgent(context: Context): String {
val packageManager = context.packageManager
val info = packageManager.getPackageInfo(context.packageName, 0)
val appName = info.applicationInfo.loadLabel(packageManager).toString()
return Util.getUserAgent(context, appName)
}
}
fun SimpleExoPlayer.playRawVideo(context: Context, #RawRes rawVideoRes: Int) {
val dataSpec = DataSpec(RawResourceDataSource.buildRawResourceUri(rawVideoRes))
val rawResourceDataSource = RawResourceDataSource(context)
rawResourceDataSource.open(dataSpec)
val factory: DataSource.Factory = DataSource.Factory { rawResourceDataSource }
prepare(LoopingMediaSource(ExtractorMediaSource.Factory(factory).createMediaSource(rawResourceDataSource.uri)))
}
fun SimpleExoPlayer.playVideoFromUrl(context: Context, url: String, cache: Cache? = null) = playVideoFromUri(context, Uri.parse(url), cache)
fun SimpleExoPlayer.playVideoFile(context: Context, file: File) = playVideoFromUri(context, Uri.fromFile(file))
fun SimpleExoPlayer.playVideoFromUri(context: Context, uri: Uri, cache: Cache? = null) {
val factory = if (cache != null)
CacheDataSourceFactory(cache, DefaultHttpDataSourceFactory(getUserAgent(context)))
else
DefaultDataSourceFactory(context, MainActivity.getUserAgent(context))
val mediaSource = ExtractorMediaSource.Factory(factory).createMediaSource(uri)
prepare(mediaSource)
}
}
I had various issues on trying this till I got to the current situation, and I've updated this question multiple times accordingly. Now it even works with the percentageY I talked about, so I could set it to be from 20% of the top of the video, if I wish. However, I still think that it has a big chance that something is wrong, because when I tried to set it to 50% , I've noticed that the content might not fit the entire View.
I even looked at the source code of ImageView (here), to see how center-crop is used. When applied to the ImageView, it still worked as center-crop, but when I used the same technique on the video, it gave me a very wrong result.
The questions
My goal here was to show both ImageView and the video so that it will smoothly transition from a static image to a video. All that while having both have the top-scale-crop of 20% from the top (for example). I've published a sample project here to try it out and share people of what I've found.
So now my questions are around why this doesn't seem to work well for the imageView and/or video :
As it turns out, none of the matrix creations that I've tried work well for either ImageView or the video. What's wrong with it exactly? How can I change it for them to look the same? To scale-crop from the top 20%, for example?
I tried to use the exact matrix for both, but it seems each need it differently, even though both have the exact same size and content size. Why would I need a different matrix for each?
EDIT: after this question was answered, I've decided to make a small sample of how to use it (Github repository available here) :
import android.content.Context
import android.graphics.Matrix
import android.graphics.PointF
import android.net.Uri
import android.os.Bundle
import android.view.TextureView
import android.view.View
import androidx.annotation.RawRes
import androidx.appcompat.app.AppCompatActivity
import androidx.core.view.doOnPreDraw
import com.google.android.exoplayer2.ExoPlayerFactory
import com.google.android.exoplayer2.Player
import com.google.android.exoplayer2.SimpleExoPlayer
import com.google.android.exoplayer2.source.ExtractorMediaSource
import com.google.android.exoplayer2.source.LoopingMediaSource
import com.google.android.exoplayer2.trackselection.DefaultTrackSelector
import com.google.android.exoplayer2.upstream.*
import com.google.android.exoplayer2.upstream.cache.Cache
import com.google.android.exoplayer2.upstream.cache.CacheDataSourceFactory
import com.google.android.exoplayer2.upstream.cache.LeastRecentlyUsedCacheEvictor
import com.google.android.exoplayer2.upstream.cache.SimpleCache
import com.google.android.exoplayer2.util.Util
import com.google.android.exoplayer2.video.VideoListener
import kotlinx.android.synthetic.main.activity_main.*
import java.io.File
// https://stackoverflow.com/questions/54216273/how-to-have-similar-mechanism-of-center-crop-on-exoplayers-playerview-but-not
class MainActivity : AppCompatActivity() {
companion object {
private val FOCAL_POINT = PointF(0.5f, 0.2f)
private const val IMAGE_RES_ID = R.drawable.test
private const val VIDEO_RES_ID = R.raw.test
private var cache: Cache? = null
private const val MAX_PREVIEW_CACHE_SIZE_IN_BYTES = 20L * 1024L * 1024L
#JvmStatic
fun getUserAgent(context: Context): String {
val packageManager = context.packageManager
val info = packageManager.getPackageInfo(context.packageName, 0)
val appName = info.applicationInfo.loadLabel(packageManager).toString()
return Util.getUserAgent(context, appName)
}
}
private var player: SimpleExoPlayer? = null
override fun onCreate(savedInstanceState: Bundle?) {
super.onCreate(savedInstanceState)
setContentView(R.layout.activity_main)
if (cache == null)
cache = SimpleCache(File(cacheDir, "media"), LeastRecentlyUsedCacheEvictor(MAX_PREVIEW_CACHE_SIZE_IN_BYTES))
// imageView.visibility = View.INVISIBLE
imageView.setImageResource(IMAGE_RES_ID)
}
private fun prepareMatrix(view: View, mediaWidth: Float, mediaHeight: Float, focalPoint: PointF): Matrix? {
if (view.visibility == View.GONE)
return null
val viewHeight = (view.height - view.paddingTop - view.paddingBottom).toFloat()
val viewWidth = (view.width - view.paddingStart - view.paddingEnd).toFloat()
if (viewWidth <= 0 || viewHeight <= 0)
return null
val matrix = Matrix()
if (view is TextureView)
// Restore true media size for further manipulation.
matrix.setScale(mediaWidth / viewWidth, mediaHeight / viewHeight)
val scaleFactorY = viewHeight / mediaHeight
val scaleFactor: Float
var px = 0f
var py = 0f
if (mediaWidth * scaleFactorY >= viewWidth) {
// Fit height
scaleFactor = scaleFactorY
px = -(mediaWidth * scaleFactor - viewWidth) * focalPoint.x / (1 - scaleFactor)
} else {
// Fit width
scaleFactor = viewWidth / mediaWidth
py = -(mediaHeight * scaleFactor - viewHeight) * focalPoint.y / (1 - scaleFactor)
}
matrix.postScale(scaleFactor, scaleFactor, px, py)
return matrix
}
private fun playVideo() {
player = ExoPlayerFactory.newSimpleInstance(this#MainActivity, DefaultTrackSelector())
player!!.setVideoTextureView(textureView)
player!!.addVideoListener(object : VideoListener {
override fun onVideoSizeChanged(videoWidth: Int, videoHeight: Int, unappliedRotationDegrees: Int, pixelWidthHeightRatio: Float) {
super.onVideoSizeChanged(videoWidth, videoHeight, unappliedRotationDegrees, pixelWidthHeightRatio)
textureView.setTransform(prepareMatrix(textureView, videoWidth.toFloat(), videoHeight.toFloat(), FOCAL_POINT))
}
override fun onRenderedFirstFrame() {
// Log.d("AppLog", "onRenderedFirstFrame")
player!!.removeVideoListener(this)
imageView.animate().alpha(0f).setDuration(2000).start()
// imageView.visibility = View.INVISIBLE
}
})
player!!.volume = 0f
player!!.repeatMode = Player.REPEAT_MODE_ALL
player!!.playRawVideo(this, VIDEO_RES_ID)
player!!.playWhenReady = true
// player!!.playVideoFromUrl(this, "https://sample-videos.com/video123/mkv/240/big_buck_bunny_240p_20mb.mkv", cache!!)
// player!!.playVideoFromUrl(this, "https://sample-videos.com/video123/mkv/720/big_buck_bunny_720p_1mb.mkv", cache!!)
// player!!.playVideoFromUrl(this#MainActivity, "https://sample-videos.com/video123/mkv/720/big_buck_bunny_720p_1mb.mkv")
}
override fun onStart() {
super.onStart()
imageView.doOnPreDraw {
val imageWidth: Float = imageView.drawable.intrinsicWidth.toFloat()
val imageHeight: Float = imageView.drawable.intrinsicHeight.toFloat()
imageView.imageMatrix = prepareMatrix(imageView, imageWidth, imageHeight, FOCAL_POINT)
}
playVideo()
}
override fun onStop() {
super.onStop()
if (player != null) {
player!!.setVideoTextureView(null)
// playerView.player = null
player!!.release()
player = null
}
}
override fun onDestroy() {
super.onDestroy()
if (!isChangingConfigurations)
cache?.release()
}
fun SimpleExoPlayer.playRawVideo(context: Context, #RawRes rawVideoRes: Int) {
val dataSpec = DataSpec(RawResourceDataSource.buildRawResourceUri(rawVideoRes))
val rawResourceDataSource = RawResourceDataSource(context)
rawResourceDataSource.open(dataSpec)
val factory: DataSource.Factory = DataSource.Factory { rawResourceDataSource }
prepare(LoopingMediaSource(ExtractorMediaSource.Factory(factory).createMediaSource(rawResourceDataSource.uri)))
}
fun SimpleExoPlayer.playVideoFromUrl(context: Context, url: String, cache: Cache? = null) = playVideoFromUri(context, Uri.parse(url), cache)
fun SimpleExoPlayer.playVideoFile(context: Context, file: File) = playVideoFromUri(context, Uri.fromFile(file))
fun SimpleExoPlayer.playVideoFromUri(context: Context, uri: Uri, cache: Cache? = null) {
val factory = if (cache != null)
CacheDataSourceFactory(cache, DefaultHttpDataSourceFactory(getUserAgent(context)))
else
DefaultDataSourceFactory(context, MainActivity.getUserAgent(context))
val mediaSource = ExtractorMediaSource.Factory(factory).createMediaSource(uri)
prepare(mediaSource)
}
}
Here's a solution for ImageView alone, if needed:
class ScaleCropImageView(context: Context, attrs: AttributeSet?) : AppCompatImageView(context, attrs) {
var focalPoint = PointF(0.5f, 0.5f)
set(value) {
field = value
updateMatrix()
}
private val viewWidth: Float
get() = (width - paddingLeft - paddingRight).toFloat()
private val viewHeight: Float
get() = (height - paddingTop - paddingBottom).toFloat()
init {
scaleType = ScaleType.MATRIX
}
override fun onSizeChanged(w: Int, h: Int, oldw: Int, oldh: Int) {
super.onSizeChanged(w, h, oldw, oldh)
updateMatrix()
}
override fun setImageDrawable(drawable: Drawable?) {
super.setImageDrawable(drawable)
updateMatrix()
}
#Suppress("MemberVisibilityCanBePrivate")
fun updateMatrix() {
if (scaleType != ImageView.ScaleType.MATRIX)
return
val dr = drawable ?: return
imageMatrix = prepareMatrix(
viewWidth, viewHeight,
dr.intrinsicWidth.toFloat(), dr.intrinsicHeight.toFloat(), focalPoint, Matrix()
)
}
private fun prepareMatrix(
viewWidth: Float, viewHeight: Float, mediaWidth: Float, mediaHeight: Float,
focalPoint: PointF, matrix: Matrix
): Matrix? {
if (viewWidth <= 0 || viewHeight <= 0)
return null
var scaleFactor = viewHeight / mediaHeight
if (mediaWidth * scaleFactor >= viewWidth) {
// Fit height
matrix.postScale(scaleFactor, scaleFactor, -(mediaWidth * scaleFactor - viewWidth) * focalPoint.x / (1 - scaleFactor), 0f)
} else {
// Fit width
scaleFactor = viewWidth / mediaWidth
matrix.postScale(scaleFactor, scaleFactor, 0f, -(mediaHeight * scaleFactor - viewHeight) * focalPoint.y / (1 - scaleFactor))
}
return matrix
}
}
The question is how to manipulate an image like ImageView.ScaleType.CENTER_CROP but to shift the focus from the center to another location that is 20% from the top of the image. First, let's look at what CENTER_CROP does:
From the documentation:
CENTER_CROP
Scale the image uniformly (maintain the image's aspect ratio) so that both dimensions (width and height) of the image will be equal to or larger than the corresponding dimension of the view (minus padding). The image is then centered in the view. From XML, use this syntax: android:scaleType="centerCrop".
In other words, scale the image without distortion such that either the width or height of the image (or both width and height) fit within the view so that the view is completely filled with the image (no gaps.)
Another way to think of this is that the center of the image is "pinned" to the center of the view. The image is then scaled to meet the criteria above.
In the following video, the white lines mark the center of the image; the red lines mark the center of the view. The scale type is CENTER_CROP. Notice how the center points of the image and the view coincide. As the view changes size, these two points continue to overlap and always appear at the center of the view regardless of the view size.
So, what does it mean to have center crop-like behavior at a different location such as 20% from the top? Like center crop, we can specify that the point that is 20% from the top of the image and the point that 20% from the top of the view will be "pinned" like the 50% point is "pinned" in center crop. The horizontal location of this point remains at 50% of the image and view. The image can now be scaled to satisfy the other conditions of center crop which specify that either the width and/or height of the image will fit the view with no gaps. (Size of view is understood to be the view size less padding.)
Here is a short video of this 20% crop behavior. In this video, the white lines show the middle of the image, the red lines show the pinned point in the view and the blue line that shows behind the horizontal red line identifies 20% from the top of the image. (Demo project is on GitHub.
Here is the result showing the full image that was supplied and the video in a square frame that transition from the still image. .
MainActivity.kt
prepareMatrix() is the method that does the work to determine how to scale/crop the image. There is some additional work to be done with the video since it appears that the video is made to fit the TextureViewas a scale type "FIT_XY" when it is assigned to the TextureView. Because of this scaling, the media size must be restored before prepareMatrix() is called for the video
class MainActivity : AppCompatActivity() {
private val imageResId = R.drawable.test
private val videoResId = R.raw.test
private var player: SimpleExoPlayer? = null
private val mFocalPoint = PointF(0.5f, 0.2f)
override fun onCreate(savedInstanceState: Bundle?) {
window.setBackgroundDrawable(ColorDrawable(0xff000000.toInt()))
super.onCreate(savedInstanceState)
if (cache == null) {
cache = SimpleCache(File(cacheDir, "media"), LeastRecentlyUsedCacheEvictor(MAX_PREVIEW_CACHE_SIZE_IN_BYTES))
}
setContentView(R.layout.activity_main)
// imageView.visibility = View.INVISIBLE
imageView.setImageResource(imageResId)
imageView.doOnPreDraw {
imageView.scaleType = ImageView.ScaleType.MATRIX
val imageWidth: Float = ContextCompat.getDrawable(this, imageResId)!!.intrinsicWidth.toFloat()
val imageHeight: Float = ContextCompat.getDrawable(this, imageResId)!!.intrinsicHeight.toFloat()
imageView.imageMatrix = prepareMatrix(imageView, imageWidth, imageHeight, mFocalPoint, Matrix())
val b = BitmapFactory.decodeResource(resources, imageResId)
val d = BitmapDrawable(resources, b.copy(Bitmap.Config.ARGB_8888, true))
val c = Canvas(d.bitmap)
val p = Paint()
p.color = resources.getColor(android.R.color.holo_red_dark)
p.style = Paint.Style.STROKE
val strokeWidth = 10
p.strokeWidth = strokeWidth.toFloat()
// Horizontal line
c.drawLine(0f, imageHeight * mFocalPoint.y, imageWidth, imageHeight * mFocalPoint.y, p)
// Vertical line
c.drawLine(imageWidth * mFocalPoint.x, 0f, imageWidth * mFocalPoint.x, imageHeight, p)
// Line in horizontal and vertical center
p.color = resources.getColor(android.R.color.white)
c.drawLine(imageWidth / 2, 0f, imageWidth / 2, imageHeight, p)
c.drawLine(0f, imageHeight / 2, imageWidth, imageHeight / 2, p)
imageView.setImageBitmap(d.bitmap)
imageViewFull.setImageBitmap(d.bitmap)
}
}
fun startPlay(view: View) {
playVideo()
}
private fun getViewWidth(view: View): Float {
return (view.width - view.paddingStart - view.paddingEnd).toFloat()
}
private fun getViewHeight(view: View): Float {
return (view.height - view.paddingTop - view.paddingBottom).toFloat()
}
private fun prepareMatrix(targetView: View, mediaWidth: Float, mediaHeight: Float,
focalPoint: PointF, matrix: Matrix): Matrix {
if (targetView.visibility != View.VISIBLE) {
return matrix
}
val viewHeight = getViewHeight(targetView)
val viewWidth = getViewWidth(targetView)
val scaleFactorY = viewHeight / mediaHeight
val scaleFactor: Float
val px: Float
val py: Float
if (mediaWidth * scaleFactorY >= viewWidth) {
// Fit height
scaleFactor = scaleFactorY
px = -(mediaWidth * scaleFactor - viewWidth) * focalPoint.x / (1 - scaleFactor)
py = 0f
} else {
// Fit width
scaleFactor = viewWidth / mediaWidth
px = 0f
py = -(mediaHeight * scaleFactor - viewHeight) * focalPoint.y / (1 - scaleFactor)
}
matrix.postScale(scaleFactor, scaleFactor, px, py)
return matrix
}
private fun playVideo() {
player = ExoPlayerFactory.newSimpleInstance(this#MainActivity, DefaultTrackSelector())
player!!.setVideoTextureView(textureView)
player!!.addVideoListener(object : VideoListener {
override fun onVideoSizeChanged(width: Int, height: Int, unappliedRotationDegrees: Int, pixelWidthHeightRatio: Float) {
super.onVideoSizeChanged(width, height, unappliedRotationDegrees, pixelWidthHeightRatio)
val matrix = Matrix()
// Restore true media size for further manipulation.
matrix.setScale(width / getViewWidth(textureView), height / getViewHeight(textureView))
textureView.setTransform(prepareMatrix(textureView, width.toFloat(), height.toFloat(), mFocalPoint, matrix))
}
override fun onRenderedFirstFrame() {
Log.d("AppLog", "onRenderedFirstFrame")
player!!.removeVideoListener(this)
imageView.animate().alpha(0f).setDuration(2000).start()
imageView.visibility = View.INVISIBLE
}
})
player!!.volume = 0f
player!!.repeatMode = Player.REPEAT_MODE_ALL
player!!.playRawVideo(this, videoResId)
player!!.playWhenReady = true
// player!!.playVideoFromUrl(this, "https://sample-videos.com/video123/mkv/240/big_buck_bunny_240p_20mb.mkv", cache!!)
// player!!.playVideoFromUrl(this, "https://sample-videos.com/video123/mkv/720/big_buck_bunny_720p_1mb.mkv", cache!!)
// player!!.playVideoFromUrl(this#MainActivity, "https://sample-videos.com/video123/mkv/720/big_buck_bunny_720p_1mb.mkv")
}
override fun onStop() {
super.onStop()
if (player != null) {
player!!.setVideoTextureView(null)
// playerView.player = null
player!!.release()
player = null
}
}
companion object {
const val MAX_PREVIEW_CACHE_SIZE_IN_BYTES = 20L * 1024L * 1024L
var cache: com.google.android.exoplayer2.upstream.cache.Cache? = null
#JvmStatic
fun getUserAgent(context: Context): String {
val packageManager = context.packageManager
val info = packageManager.getPackageInfo(context.packageName, 0)
val appName = info.applicationInfo.loadLabel(packageManager).toString()
return Util.getUserAgent(context, appName)
}
}
fun SimpleExoPlayer.playRawVideo(context: Context, #RawRes rawVideoRes: Int) {
val dataSpec = DataSpec(RawResourceDataSource.buildRawResourceUri(rawVideoRes))
val rawResourceDataSource = RawResourceDataSource(context)
rawResourceDataSource.open(dataSpec)
val factory: DataSource.Factory = DataSource.Factory { rawResourceDataSource }
prepare(LoopingMediaSource(ExtractorMediaSource.Factory(factory).createMediaSource(rawResourceDataSource.uri)))
}
fun SimpleExoPlayer.playVideoFromUrl(context: Context, url: String, cache: Cache? = null) = playVideoFromUri(context, Uri.parse(url), cache)
fun SimpleExoPlayer.playVideoFile(context: Context, file: File) = playVideoFromUri(context, Uri.fromFile(file))
fun SimpleExoPlayer.playVideoFromUri(context: Context, uri: Uri, cache: Cache? = null) {
val factory = if (cache != null)
CacheDataSourceFactory(cache, DefaultHttpDataSourceFactory(getUserAgent(context)))
else
DefaultDataSourceFactory(context, MainActivity.getUserAgent(context))
val mediaSource = ExtractorMediaSource.Factory(factory).createMediaSource(uri)
prepare(mediaSource)
}
}
you can use app:resize_mode="zoom" in com.google.android.exoplayer2.ui.PlayerView
I had a similar problem and solved it by applying transformations on the TextureView whose Surface is used by ExoPlayer:
player.addVideoListener(object : VideoListener {
override fun onVideoSizeChanged(
videoWidth: Int,
videoHeight: Int,
unappliedRotationDegrees: Int,
pixelWidthHeightRatio: Float,
) {
removeVideoListener(this)
val viewWidth: Int = textureView.width - textureView.paddingStart - textureView.paddingEnd
val viewHeight: Int = textureView.height - textureView.paddingTop - textureView.paddingBottom
if (videoWidth == viewWidth && videoHeight == viewHeight) {
return
}
val matrix = Matrix().apply {
// TextureView makes a best effort in fitting the video inside the View. The first transformation we apply is for reverting the fitting.
setScale(
videoWidth.toFloat() / viewWidth,
videoHeight.toFloat() / viewHeight,
)
}
// This algorithm is from ImageView's CENTER_CROP transformation
val offset = 0.5f // the center in CENTER_CROP but you probably want a different value here
val scale: Float
val dx: Float
val dy: Float
if (videoWidth * viewHeight > viewWidth * videoHeight) {
scale = viewHeight.toFloat() / videoHeight
dx = (viewWidth - videoWidth * scale) * offset
dy = 0f
} else {
scale = viewWidth.toFloat() / videoWidth
dx = 0f
dy = (viewHeight - videoHeight * scale) * offset
}
setTransform(matrix.apply {
postScale(scale, scale)
postTranslate(dx, dy)
})
}
})
player.setVideoTextureView(textureView)
player.prepare(createMediaSource())
Note that unless you're using DefaultRenderersFactory you need to make sure that your video Renderer actually calls onVideoSizeChanged by for instance creating the factory like so:
val renderersFactory = RenderersFactory { handler, videoListener, _, _, _, _ ->
// Allows other renderers to be removed by R8
arrayOf(
MediaCodecVideoRenderer(
context,
MediaCodecSelector.DEFAULT,
DefaultRenderersFactory.DEFAULT_ALLOWED_VIDEO_JOINING_TIME_MS,
handler,
videoListener,
-1,
),
MediaCodecAudioRenderer(context, MediaCodecSelector.DEFAULT),
)
}
I am implementing a custom KeyboardView in my app and it's all working at the moment, however, when I attempt to press a key on the keyboard using Espresso ViewAction, I am getting an exception saying:
android.support.test.espresso.PerformException:
Error performing 'single click - At Coordinates: 1070, 2809 and
precision: 16, 16' on view 'with id:
com.example.app.mvpdemo:id/keyboardLayout'.
The code throwing the exception is:
#Test
fun enter100AsPriceShouldDisplay120ForA20PercentTip(){
onView(withId(R.id.editTextCheckAmount))
.perform(typeText("100"), closeSoftKeyboard())
val appContext = InstrumentationRegistry.getTargetContext()
val displayMetrics = appContext.resources.displayMetrics
onView(withId(R.id.keyboardLayout)).perform(clickXY(displayMetrics.widthPixels - 10, displayMetrics.heightPixels - 10))
onView(withText("$120.00")).check(matches(isDisplayed()))
}
and the click XY function which came from this post
private fun clickXY(x: Int, y: Int): ViewAction {
return GeneralClickAction(
Tap.SINGLE,
CoordinatesProvider { view ->
val screenPos = IntArray(2)
view.getLocationOnScreen(screenPos)
val screenX = (screenPos[0] + x).toFloat()
val screenY = (screenPos[1] + y).toFloat()
floatArrayOf(screenX, screenY)
},
Press.FINGER, 0, 0)
}
Here is my keyboard layout (pinned to the bottom of the screen inside a ConstraintLayout):
Does anyone know why? Any help is appreciated.
Answering my own question after determining a flexible solution:
First attempt - get DisplayMetrics of the root View and subtract an arbitrary number to attempt to hit the Keyboard.Key
this didn't work because clickXY function uses the position of the view
this ended up being the reason for the exception since the view is smaller than the DisplayMetrics values and adding to the Views on screen position would give a very high number for the x and y.
So I tried again,
Second attempt - use check method on the ViewMatcher to check the KeyBoardView.
by doing so I was able to get access to the KeyboardView's position x
then I was able to get the KeyboardView's width and height
by performing some math, I was able to figure out target index for x & y
the math:
take the widthPercent for the Keyboard.Key (in my case 33.3%)
take the rowCount of the keyboard.xml (in my case 3)
use (viewWidth * widthPercent) / 4 to get relativeButtonX
use (viewHeight / rowCount) / 2 to get relativeButtonY
then for targetY, I took viewHeight - relativeButtonY
finally, for targetX, I took (viewPosX + viewWidth) - relativeButtonX
So enough explanation, here is the code:
#Test
fun enter100AsPriceShouldDisplay120ForA20PercentTip() {
onView(withId(R.id.editTextCheckAmount))
.perform(typeText("100"), closeSoftKeyboard())
// call the function to get the targets
val (viewTargetY, viewTargetX) = getTargetXAndY()
// perform the action
onView(withId(R.id.keyboardLayout)).perform(clickXY(viewTargetX.toInt(), viewTargetY))
onView(withText("Tip: $20.00")).check(matches(isDisplayed()))
onView(withText("Total: $120.00")).check(matches(isDisplayed()))
}
and the helper method with all the math:
private fun getTargetXAndY(): Pair<Int, Double> {
var viewHeight = 0
var viewWidth = 0
var viewPosX = 0F
val viewMatcher = onView(withId(R.id.keyboardLayout))
viewMatcher.check { view, _ ->
viewWidth = view.width
viewHeight = view.height
viewPosX = view.x
}
val keyboardKeyWidthPercent = 0.333
val keyboardRowsCount = 3
val keyboardButtonWidthQuarter = (viewWidth * keyboardKeyWidthPercent) / 4
val keyboardButtonHeightHalf = (viewHeight / keyboardRowsCount) / 2
val viewTargetY = viewHeight - keyboardButtonHeightHalf
val viewTargetX = (viewPosX + viewWidth) - keyboardButtonWidthQuarter
return Pair(viewTargetY, viewTargetX)
}
Now, the click is not perfectly centered but it clicks the button pretty close to the center.
Background
I have a small live wallpaper app, that I want to add support for it to show GIF animations.
For this, I've found various solutions. There is the solution of showing a GIF animation in a view (here), and there is even a solution for showing it in a live wallpaper (here).
However, for both of them, I can't find how to fit the content of the GIF animation nicely in the space it has, meaning any of the following:
center-crop - fits to 100% of the container (the screen in this case), cropping on sides (top&bottom or left&right) when needed. Doesn't stretch anything. This means the content seems fine, but not all of it might be shown.
fit-center - stretch to fit width/height
center-inside - set as original size, centered, and stretch to fit width/height only if too large.
The problem
None of those is actually about ImageView, so I can't just use the scaleType attribute.
What I've found
There is a solution that gives you a GifDrawable (here), which you can use in ImageView, but it seems it's pretty slow in some cases, and I can't figure out how to use it in LiveWallpaper and then fit it.
The main code of the LiveWallpaper GIF handling is as such (here) :
class GIFWallpaperService : WallpaperService() {
override fun onCreateEngine(): WallpaperService.Engine {
val movie = Movie.decodeStream(resources.openRawResource(R.raw.cinemagraphs))
return GIFWallpaperEngine(movie)
}
private inner class GIFWallpaperEngine(private val movie: Movie) : WallpaperService.Engine() {
private val frameDuration = 20
private var holder: SurfaceHolder? = null
private var visible: Boolean = false
private val handler: Handler = Handler()
private val drawGIF = Runnable { draw() }
private fun draw() {
if (visible) {
val canvas = holder!!.lockCanvas()
canvas.save()
movie.draw(canvas, 0f, 0f)
canvas.restore()
holder!!.unlockCanvasAndPost(canvas)
movie.setTime((System.currentTimeMillis() % movie.duration()).toInt())
handler.removeCallbacks(drawGIF)
handler.postDelayed(drawGIF, frameDuration.toLong())
}
}
override fun onVisibilityChanged(visible: Boolean) {
this.visible = visible
if (visible)
handler.post(drawGIF)
else
handler.removeCallbacks(drawGIF)
}
override fun onDestroy() {
super.onDestroy()
handler.removeCallbacks(drawGIF)
}
override fun onCreate(surfaceHolder: SurfaceHolder) {
super.onCreate(surfaceHolder)
this.holder = surfaceHolder
}
}
}
The main code for handling GIF animation in a view is as such:
class CustomGifView #JvmOverloads constructor(context: Context, attrs: AttributeSet? = null, defStyleAttr: Int = 0) : View(context, attrs, defStyleAttr) {
private var gifMovie: Movie? = null
var movieWidth: Int = 0
var movieHeight: Int = 0
var movieDuration: Long = 0
var mMovieStart: Long = 0
init {
isFocusable = true
val gifInputStream = context.resources.openRawResource(R.raw.test)
gifMovie = Movie.decodeStream(gifInputStream)
movieWidth = gifMovie!!.width()
movieHeight = gifMovie!!.height()
movieDuration = gifMovie!!.duration().toLong()
}
override fun onMeasure(widthMeasureSpec: Int, heightMeasureSpec: Int) {
setMeasuredDimension(movieWidth, movieHeight)
}
override fun onDraw(canvas: Canvas) {
val now = android.os.SystemClock.uptimeMillis()
if (mMovieStart == 0L) { // first time
mMovieStart = now
}
if (gifMovie != null) {
var dur = gifMovie!!.duration()
if (dur == 0) {
dur = 1000
}
val relTime = ((now - mMovieStart) % dur).toInt()
gifMovie!!.setTime(relTime)
gifMovie!!.draw(canvas, 0f, 0f)
invalidate()
}
}
}
The questions
Given a GIF animation, how can I scale it in each of the above ways?
Is it possible to have a single solution for both cases?
Is it possible to use GifDrawable library (or any other drawable for the matter) for the live wallpaper, instead of the Movie class? If so, how?
EDIT: after finding how to scale for 2 kinds, I still need to know how to scale according to the third type, and also want to know why it keeps crashing after orientation changes, and why it doesn't always show the preview right away.
I'd also like to know what's the best way to show the GIF animation here, because currently I just refresh the canvas ~60fps (1000/60 waiting between each 2 frames), without consideration of what's in the file.
Project is available here.
If you have Glide in your project, You can easily load Gifs, as it provides drawing GIFs to your ImageViews and does support many scaling options (like center or a given width and ...).
Glide.with(context)
.load(imageUrl or resourceId)
.asGif()
.fitCenter() //or other scaling options as you like
.into(imageView);
OK I think I got how to scale the content. Not sure though why the app still crashes upon orientation change sometimes, and why the app doesn't show the preview right away sometimes.
Project is available here.
For center-inside, the code is:
private fun draw() {
if (!isVisible)
return
val canvas = holder!!.lockCanvas() ?: return
canvas.save()
//center-inside
val scale = Math.min(canvas.width.toFloat() / movie.width().toFloat(), canvas.height.toFloat() / movie.height().toFloat());
val x = (canvas.width.toFloat() / 2f) - (movie.width().toFloat() / 2f) * scale;
val y = (canvas.height.toFloat() / 2f) - (movie.height().toFloat() / 2f) * scale;
canvas.translate(x, y)
canvas.scale(scale, scale)
movie.draw(canvas, 0f, 0f)
canvas.restore()
holder!!.unlockCanvasAndPost(canvas)
movie.setTime((System.currentTimeMillis() % movie.duration()).toInt())
handler.removeCallbacks(drawGIF)
handler.postDelayed(drawGIF, frameDuration.toLong())
}
For center-crop, the code is:
private fun draw() {
if (!isVisible)
return
val canvas = holder!!.lockCanvas() ?: return
canvas.save()
//center crop
val scale = Math.max(canvas.width.toFloat() / movie.width().toFloat(), canvas.height.toFloat() / movie.height().toFloat());
val x = (canvas.width.toFloat() / 2f) - (movie.width().toFloat() / 2f) * scale;
val y = (canvas.height.toFloat() / 2f) - (movie.height().toFloat() / 2f) * scale;
canvas.translate(x, y)
canvas.scale(scale, scale)
movie.draw(canvas, 0f, 0f)
canvas.restore()
holder!!.unlockCanvasAndPost(canvas)
movie.setTime((System.currentTimeMillis() % movie.duration()).toInt())
handler.removeCallbacks(drawGIF)
handler.postDelayed(drawGIF, frameDuration.toLong())
}
for fit-center, I can use this:
val canvasWidth = canvas.width.toFloat()
val canvasHeight = canvas.height.toFloat()
val bitmapWidth = curBitmap.width.toFloat()
val bitmapHeight = curBitmap.height.toFloat()
val scaleX = canvasWidth / bitmapWidth
val scaleY = canvasHeight / bitmapHeight
scale = if (scaleX * curBitmap.height > canvas.height) scaleY else scaleX
x = (canvasWidth / 2f) - (bitmapWidth / 2f) * scale
y = (canvasHeight / 2f) - (bitmapHeight / 2f) * scale
...
Change the width and the height of the movie:
Add this code in onDraw method before movie.draw
canvas.scale((float)this.getWidth() / (float)movie.width(),(float)this.getHeight() / (float)movie.height());
or
canvas.scale(1.9f, 1.21f); //this changes according to screen size
Scale to fill and scale to fit:
There's already a good answer on that:
https://stackoverflow.com/a/38898699/5675325