I want to make a waveform drawing for an audio recorder in Android. The usual one with lines/bars, like this one:
More importantly, I want it live, while the song is being recorded. My app already computes the RMS through AudioRecord. But I am not sure which is the best approach for the actual drawing in terms of processing, resources, battery, etc.
The Visualizer does not show anything meaningful, IMO (are those graphs more or less random stuff??).
I've seen the canvas approach and the layout approach (there are probably more?). In the layout approach you add thin vertical layouts in a horizontal layout. The advantage is that you don't need to redraw the whole thing each 1/n secs, you just add one layout each 1/n secs... but you need hundreds of layouts (depending on n). In the canvas layout, you need to redraw the whole thing (right??) n times per second. Some even create bitmaps for each drawing...
So, which is cheaper, and why? Is there anything better nowadays? How much frequency update (i.e., n) is too much for generic low end devices?
EDIT1
Thanks to the beautiful trick #cactustictacs taught me in his answer, I was able to implement this with ease. Yet, the image is strangely rendered kind of "blurry by movement":
The waveform runs from right to left. You can easily see the blur movement, and the left-most and right-most pixels get "contaminated" by the other end. I guess I can just cut both extremes...
This renders better if I make my Bitmap bigger (i.e., making widthBitmap bigger), but then the onDraw will be heavier...
This is my full code:
package com.floritfoto.apps.ave;
import android.content.Context;
import android.graphics.Bitmap;
import android.graphics.Canvas;
import android.graphics.Paint;
import android.graphics.Rect;
import android.util.AttributeSet;
import java.util.Arrays;
public class Waveform extends androidx.appcompat.widget.AppCompatImageView {
//private float lastPosition = 0.5f; // 0.5 for drawLine method, 0 for the others
private int lastPosition = 0;
private final int widthBitmap = 50;
private final int heightBitmap = 80;
private final int[] transpixels = new int[heightBitmap];
private final int[] whitepixels = new int[heightBitmap];
//private float top, bot; // float for drawLine method, int for the others
private int aux, top;
//private float lpf;
private int width = widthBitmap;
private float proportionW = (float) (width/widthBitmap);
Boolean firstLoopIsFinished = false;
Bitmap MyBitmap = Bitmap.createBitmap(widthBitmap, heightBitmap, Bitmap.Config.ARGB_8888);
//Canvas canvasB = new Canvas(MyBitmap);
Paint MyPaint = new Paint();
Paint MyPaintTrans = new Paint();
Rect rectLbit, rectRbit, rectLdest, rectRdest;
public Waveform(Context context, AttributeSet attrs) {
super(context, attrs);
MyPaint.setColor(0xffFFFFFF);
MyPaint.setStrokeWidth(1);
MyPaintTrans.setColor(0xFF202020);
MyPaintTrans.setStrokeWidth(1);
Arrays.fill(transpixels, 0xFF202020);
Arrays.fill(whitepixels, 0xFFFFFFFF);
}
public void drawNewBar() {
// For drawRect or drawLine
/*
top = ((1.0f - Register.tone) * heightBitmap / 2.0f);
bot = ((1.0f + Register.tone) * heightBitmap / 2.0f);
// Using drawRect
//if (firstLoopIsFinished) canvasB.drawRect(lastPosition, 0, lastPosition+1, heightBitmap, MyPaintTrans); // Delete last stuff
//canvasB.drawRect(lastPosition, top, lastPosition+1, bot, MyPaint);
// Using drawLine
if (firstLoopIsFinished) canvasB.drawLine(lastPosition, 0, lastPosition, heightBitmap, MyPaintTrans); // Delete previous stuff
canvasB.drawLine(lastPosition ,top, lastPosition, bot, MyPaint);
*/
// Using setPixel (no tiene sentido, mucho mejor setPixels.
/*
int top = (int) ((1.0f - Register.tone) * heightBitmap / 2.0f);
int bot = (int) ((1.0f + Register.tone) * heightBitmap / 2.0f);
if (firstLoopIsFinished) {
for (int i = 0; i < top; ++i) {
MyBitmap.setPixel(lastPosition, i, 0xFF202020);
MyBitmap.setPixel(lastPosition, heightBitmap - i-1, 0xFF202020);
}
}
for (int i = top ; i < bot ; ++i) {
MyBitmap.setPixel(lastPosition,i,0xffFFFFFF);
}
//System.out.println("############## "+top+" "+bot);
*/
// Using setPixels. Works!!
top = (int) ((1.0f - Register.tone) * heightBitmap / 2.0f);
if (firstLoopIsFinished)
MyBitmap.setPixels(transpixels,0,1,lastPosition,0,1,heightBitmap);
MyBitmap.setPixels(whitepixels, top,1, lastPosition, top,1,heightBitmap-2*top);
lastPosition++;
aux = (int) (width - proportionW * (lastPosition));
rectLbit.right = lastPosition;
rectRbit.left = lastPosition;
rectLdest.right = aux;
rectRdest.left = aux;
if (lastPosition >= widthBitmap) { firstLoopIsFinished = true; lastPosition = 0; }
}
#Override
protected void onSizeChanged(int w, int h, int oldw, int oldh) {
super.onSizeChanged(w, h, oldw, oldh);
width = w;
proportionW = (float) width/widthBitmap;
rectLbit = new Rect(0, 0, widthBitmap, heightBitmap);
rectRbit = new Rect(0, 0, widthBitmap, heightBitmap);
rectLdest = new Rect(0, 0, width, h);
rectRdest = new Rect(0, 0, width, h);
}
#Override
protected void onDraw(Canvas canvas) {
super.onDraw(canvas);
drawNewBar();
canvas.drawBitmap(MyBitmap, rectLbit, rectRdest, MyPaint);
canvas.drawBitmap(MyBitmap, rectRbit, rectLdest, MyPaint);
}
}
EDIT2
I was able to prevent the blurring just using null as Paint in the canvas.drawBitmap:
canvas.drawBitmap(MyBitmap, rectLbit, rectRdest, null);
canvas.drawBitmap(MyBitmap, rectRbit, rectLdest, null);
No Paints needed.
Your basic custom view approach would be to implement onDraw and redraw your current data each frame. You'd probably keep some kind of circular Buffer holding your most recent n amplitude values, so each frame you'd iterate over those, and use drawRect to draw the bars (you'd calculate things like width, height scaling, start and end positions etc in onSizeChanged, and use those values when defining the coordinates for the Rects).
That in itself might be fine! The only way you can really tell how expensive draw calls are is to benchmark them, so you could try this approach out and see how it goes. Profile it to see how much time it takes, how much the CPU spikes etc.
There are a few things you can do to make onDraw as efficient as possible, mostly things like avoiding object allocations - so watch out for loop functions that create Iterators, and in the same way you're supposed to create a Paint once instead of creating them over and over in onDraw, you could reuse a single Rect object by setting its coordinates for each bar you need to draw.
Another approach you could try is creating a working Bitmap in your custom view, which you control, and calling drawBitmap inside onDraw to paint it onto the Canvas. That should be a pretty inexpensive call, and it can easily be stretched as required to fit the view.
The idea there, is that very time you get new data, you paint it onto the bitmap. Because of how your waveform looks (like blocks), and the fact you can scale it up, really all you need is a single vertical line of pixels for each value, right? So as the data comes in, you paint an extra line onto your already-existing bitmap, adding to the image. Instead of painting the entire waveform block by block every frame, you're just adding the new blocks.
The complication there is when you "fill" the bitmap - now you have to "shift" all the pixels to the left, dropping the oldest ones on the left side, so you can draw the new ones on the right. So you'll need a way to do that!
Another approach would be something similar to the circular buffer idea. If you don't know what that is, the idea is you take a normal buffer with a start and an end, but you treat one of the indices as your data's start point, wrap around to 0 when you hit the last index of the buffer, and stop when you hit the index you're calling your end point:
Partially filled buffer:
|start
123400
|end
Data: 1234
Full buffer:
|start
123456
|end
Data: 123456
After adding one more item:
|start
723456
|end
Data: 234567
See how once it's full, you shift the start and end one step "right", wrapping around if necessary? So you always have the most recent 6 values added. You just have to handle reading from the correct index ranges, from start -> lastIndex and then firstIndex -> end
You could do the same thing with a bitmap - start "filling" it from the left, increasing end so you can draw the next vertical line. Once it's full, start filling from the left by moving end there. When you actually draw the bitmap, instead of drawing the whole thing as-is (723456) you draw it in two parts (23456 then 7). Make sense? When you draw a bitmap to the canvas, there's a call that takes a source Rect and a destination one, so you can draw it in two chunks.
You could always redraw the bitmap from scratch each frame (clear it and draw the vertical lines), so you're basically redrawing your whole data buffer each time. Probably still faster than the drawRect approach for each value, but honestly not much easier than the "treat the bitmap as another circular buffer" method. If you're already managing one circular buffer, it's not much more work - since the buffer and the bitmap will have the same number of values (horizontal pixels in the bitmap's case) you can use the same start and end values for both
You would never do this with layouts. Layouts are for premade components. They're high level combinations of components and you don't want to dynamically add or remove views from it frequently. For this, you use a custom view with a canvas. Layouts aren't even an option for something like this.
Update 1
I have an idea what inRange function does. But I don't want to apply mask and show the new image with skin color. What I want to do is to know if the image contains skin color and cover larger area.
What I want to do
I want to capture a picture whenever finger is detected inside a boundary. Its dimensions are known.
Struggling points
Manipulate image data in native code.
Detecting skin in live camera, so whenever that particular area is focused and skin is detected, snap should be taken
What I have done
I am using JNI Layer to perform the operation. I am able to get Mat from image data using this tutorial, but don't know how to manipulate poutPixels. The format is NV21 and I am not sure how to do operations on it.
I need to crop image and then detect if there's skin present in the image. I have successfully cropped the image to the desired dimension, but has no clue to move forward to detect skin. I want this method to return true or false.
Here is the code:
jbyte * pNV21FrameData = env->GetByteArrayElements(NV21FrameData, 0);
jint * poutPixels = env->GetIntArrayElements(outPixels, 0);
Mat mNV(height, width, CV_8UC3, (unsigned char*)pNV21FrameData);
Mat finalImage(height, width, CV_8UC3, (unsigned char*) poutPixels);
jfloat wScale = (float) width/screenWidth;
jfloat hScale = (float) height/screenHeight;
float temp = rectX * wScale;
int x = (int) temp;
temp = rectY * hScale;
int y = (int) temp;
int cW = (int) (width * wScale);
int cH = (int) (height * hScale);
cH = cH/2;
Rect regionToCrop(x, y, cW, cH);
mNV = mNV(regionToCrop);
finalImage = finalImage(regionToCrop);
//detect skin and return true or false
I have read about inRange function, but I don't know how to check whether there's skin or not.
Questions
Am I on the right path to proceed further?
The image format I am getting is NV21. Is it a 8UC1 or it can be 8UC3 too?
How to proceed from here to start detecting skin?
Any help is appreciated.
I have solved my problem by extracting skin color range and making all pixels equal to zero. Below are the steps.
Convert the image to HSV
First convert image to HSV.
Mat mHsv = new Mat(rows, cols, CvType.CV_8UC3);
Imgproc.cvtColor(mRgba, mHsv, Imgproc.COLOR_RGB2HSV);
Get range of skin color
Skin color range may vary, but this one is working fine for me.
Mat output = new Mat();
Core.inRange(mHsv, new Scalar(0, 0.18*255, 0), new Scalar(25, 0.68*255, 255), output);
Extract this Skin Range channel
Now extract this channel while making skin pixels equal to zero
Mat mExtracted = new Mat();
Core.extractChannel(output, mExtracted, 0);
Now you have mExtracted matrix, in which skin colored pixels are 0 and rests are 255 (or skin color, I am not sure).
Get count of zeros
Since 0 now is actually skin color area, what you can do is to define a threshold which suits your need. According to my need, I want skin to cover more than half of the area, so I made my logic accordingly.
int n = Core.countNonZero(mExtracted);
int check = (mExtracted.rows() * mExtracted.cols())/2;
if(n >= check && isFocused) {
//Take picture
}
I'm currently new to Android and I am having a bit of difficulty.
Currently the image just draws on screen at the size of the bitmap, however what I am looking to do is that everytime the application is launched (or ran in the emulator) the bitmap will draw at a random height and width between a defined maximum and minimum value.
I have no idea on how to go about this.
Any help would be great.
You are looking for something like this:
Random r = new Random();
int myRandomWidth = r.nextInt(maxWidthValue - minWidthValue) + minWidthValue;
int myRandomHeight = r.nextInt(maxHeightValue - minHeightValue) + minHeightValue;
Bitmap b = ContextCompat.getDrawable(this,R.drawable.myBitmap);
b = Bitmap.createScaledBitmap(b, myRandomWidth, myRandomHeight, false);
myImageView.setImageBitmap(b);
I copied the code from this question and it's working nearly perfectly, the problem is that the author sets the heights of the list items to fixed values, and it isn't displaying correctly on my device.
The code sets the height of the list items to 150, and I thought "Hey, I'll just measure the height of my text" and so I call
Paint paint = new Paint(Paint.ANTI_ALIAS_FLAG | Paint.LINEAR_TEXT_FLAG);
Rect bounds = new Rect();
paint.getTextBounds("April 3, 2016", 0, 13, bounds);
Log.d("Fragment.toggle", "Text height is " + bounds.height());
and for text that is slightly larger than the 150 fixed value, the text height returned from getTextBounds() is 12. Since 12 is not "slightly larger" than 150, I've clearly done something wrong.
I've written a lot of software in my life, but this is my first day doing Android development, so I am probably doing something obviously stupid. I suspect that the two values (150 and 12) are in different units, but I've been poking around in the documentation and can't figure it out.
How can I calculate the height of my text, in whichever units AbsListView.LayoutParams is expecting?
I finally beat this question into submission.
The first part of the answer is that the return code from getTextBounds() is in points (wouldn't it be nice if the documentation said that?). Dividing that answer by 72 gives my text height in inches, multiplying that by DisplayMetrics.densityDpi gives the text height in pixels.
Paint paint = new Paint(Paint.ANTI_ALIAS_FLAG | Paint.LINEAR_TEXT_FLAG);
Rect bounds = new Rect(0, 0, 10000, 10000);
paint.getTextBounds("April 3, 2016", 0, 13, bounds);
DisplayMetrics dm = getActivity().getResources().getDisplayMetrics();
float textHeightInches = bounds.height() / 72.0f;
float textHeightPixels = textHeightInches * dm.densityDpi;
The second part of the answer is that the list item has padding on the top and bottom that must be included when calculating the total height.
LayoutInflater vi = (LayoutInflater) getActivity().getSystemService(Context.LAYOUT_INFLATER_SERVICE);
LinearLayout listItemLayout = (LinearLayout)vi.inflate(R.layout.list_item, null);
int collapsedHeight = (int)(textHeightPixels + 0.5f) + listItemLayout.getPaddingTop() + listItemLayout.getPaddingBottom();
Now, when I create the ListItem objects, I can pass in my calculated height instead of a fixed number.
I am using the code below to get pixel by supplying the bitmap
public int[] foo(Bitmap bitmapFoo) {
int[] pixels;
// Bitmap bitmapFoo ;
int height = bitmapFoo.getHeight();
int width = bitmapFoo.getWidth();
pixels = new int[height * width];
bitmapFoo.getPixels(pixels, 0, width, 1, 1, width - 1, height - 1);
return pixels;
}
now how do I compare it to similar image??
int[] i = foo(img2);
int[] i2 = foo(img1);
if (i==i2) {
txt.setText("same");
}else{
txt.setText("different");
}
Even if the image is similar not same it still shows different .How to avoid it ??
Is the comparison correct ?? or I am doing something wrong ??
I believe the problem is that you are comparing objects. When you create two objects independently they are not equal.
If you have i.equals(i2) instead it should work as intended.
.equals compares values rather than testing to see if the objects are actually the same object
EDIT: I forgot that you may have to override .equals to achieve the desired result.
This question may help you.
Why do we have to override the equals() method in Java?