Develop Reference

Render FBO to same FBO

I am trying to create a ghost-like camera filter. This requires mixing the previous frame to current one. I use one FBO to make the mixing and a second one to simply put the context to the screen.
My implementation works on 4 out of 5 devices I have tried. On the fifth (Samsung galaxy S7) I get some random pixels.
The simpler shader to reproduce the error is the following (the frame counter and cropping is just for debugging). The result is that I get on the center of the screen on line gradually going up.
uniform samplerExternalOES camTexture;
uniform sampler2D fbo;
uniform int frame_no;
varying vec2 v_CamTexCoordinate;
void main ()
{
vec2 uv = v_CamTexCoordinate;
if(frame_no<10){
gl_FragColor = texture2D(camTexture, uv);
}else{
if(uv.y>0.2 && uv.y<0.8 && uv.x>0.2 && uv.x<0.8)
gl_FragColor = texture2D(fbo, uv + vec2(0.0, +0.005));
else
gl_FragColor = texture2D(camTexture, uv);
}
}
But on the Samsung I get some correct pixels and some random ones as the following sample. Some black and other random pixels going up together with the camera's pixels. Any idea of what might be wrong?
Fault sample
Correct sample

Point sprite alpha blending issue (Android / OpenGL ES 2.0)

I've recently started looking into OpenGL ES for Android and am working on a drawing app. I've implemented some basics such as point sprites, path smoothing and FBO for double buffering. At the moment I am playing around with the glBlendFunc, more specifically when I put two textures close to each other with the same color/alpha values, the alpha gets added so it appears darker at the intersection of the sprites. This is a problem because the stroke opacity is not preserved if a lot of points are close together, as the color tends to more opaque rather than staying with the same opacity. Is there a way to make the textures have the same color on the intersection, i.e. have the same alpha value for the intersecting pixels, but keep the alpha values for the rest of the pixels?
Here's how I've done the relevant parts of the app:
for drawing the list of point sprites I use blending like this:
GLES20.glEnable(GLES20.GL_BLEND);
GLES20.glBlendFunc(GLES20.GL_ONE, GLES20.GL_ONE_MINUS_SRC_ALPHA);
the app uses an FBO with a texture, where it renders each brush stroke first and then this texture is rendered to the main screen. The blending func there is:
GLES20.glEnable(GLES20.GL_BLEND);
GLES20.glBlendFunc(GLES20.GL_SRC_ALPHA, GLES20.GL_ONE_MINUS_SRC_ALPHA);
OpenGL ES 2.0 doesn't support alpha masking;
there is no DEPTH_TEST function used anywhere in the app;
the textures for the point sprites are PNGs with transparent backgrounds;
the app supports texture masking which means one texture is used for the shape and one texture is used for the content;
my fragment shader looks like this:
precision mediump float;
uniform sampler2D uShapeTexture;
uniform sampler2D uFillTexture;
uniform float vFillScale;
varying vec4 vColor;
varying float vShapeRotation;
varying float vFillRotation;
varying vec4 vFillPosition;
vec2 calculateRotation(float rotationValue) {
float mid = 0.5;
return vec2(cos(rotationValue) * (gl_PointCoord.x - mid) + sin(rotationValue) * (gl_PointCoord.y - mid) + mid,
cos(rotationValue) * (gl_PointCoord.y - mid) - sin(rotationValue) * (gl_PointCoord.x - mid) + mid);
}
void main() {
// Calculations.
vec2 rotatedShape = calculateRotation(vShapeRotation);
vec2 rotatedFill = calculateRotation(vFillRotation);
vec2 scaleVector = vec2(vFillScale, vFillScale);
vec2 positionVector = vec2(vFillPosition[0], vFillPosition[1]);
// Obtain colors.
vec4 colorShape = texture2D(uShapeTexture, rotatedShape);
vec4 colorFill = texture2D(uFillTexture, (rotatedFill * scaleVector) + positionVector);
gl_FragColor = colorShape * colorFill * vColor;
}
my vertex shader is this:
attribute vec4 aPosition;
attribute vec4 aColor;
attribute vec4 aJitter;
attribute float aShapeRotation;
attribute float aFillRotation;
attribute vec4 aFillPosition;
attribute float aPointSize;
varying vec4 vColor;
varying float vShapeRotation;
varying float vFillRotation;
varying vec4 vFillPosition;
uniform mat4 uMVPMatrix;
void main() {
// Sey position and size.
gl_Position = uMVPMatrix * (aPosition + aJitter);
gl_PointSize = aPointSize;
// Pass values to fragment shader.
vColor = aColor;
vShapeRotation = aShapeRotation;
vFillRotation = aFillRotation;
vFillPosition = aFillPosition;
}
I've tried playing around with the glBlendFunc parameters but I can't find the right combination to draw what I want. I've attached some images showing what I would like to achieve and what I have at the moment. Any suggestions?
The Solution
Finally managed to get this working properly with a few lines thanks to # Rabbid76. First of all I had to configure my depth test function before I draw to the FBO:
GLES20.glEnable(GLES20.GL_DEPTH_TEST);
GLES20.glDepthFunc(GLES20.GL_LESS);
// Drawing code for FBO.
GLES20.glDisable(GLES20.GL_DEPTH_TEST);
Then in my fragment shader I had to make sure that any pixels with alpha < 1 in the mask are discarded like this:
...
vec4 colorMask = texture2D(uMaskTexture, gl_PointCoord);
if (colorMask.a < 1.0)
discard;
else
gl_FragColor = calculatedColor;
And the result is (flickering is due to Android emulator and gif capture tool):

If you set the glBlendFunc
with the functions (GL_SRC_ALPHA, GL_ONE_MINUS_SRC_ALPHA) and you use
glBlendEquation with the equation GL_FUNC_ADD then the destination color is
calculated as follows:
C_dest = C_src * A_src + C_dest * (1-A_src)
If you blend for example C_dest = 1 with C_src = 0.5 and A_src = 0.5 then:
C_dest = 0.75 = 1 * 0.5 + 0.5 * 0.5
If you repeat blending the same color C_src = 0.5 and A_src = 0.5 then the destination color becomes darker:
C_dest = 0.625 = 0.75 * 0.5 + 0.5 * 0.5
Since the new target color is always a function of the original target color and the source color, the color can not remain equel when blending 2 times, because the target color has already changed after the 1st time blending (except GL_ZERO).
You have to avoid that any fragment is blended twice. If all fragments are drawn to the same depth (2D) then you can use the depth test for this:
glEnable( GL_DEPTH_TEST );
glDepthFunc( GL_LESS );
// do the drawing with the color
glDisable( GL_DEPTH_TEST );
Or the stencil test can be used. For example, the stencil test can be set to pass only when the stencil buffer is equal to 0.
Every time a fragment is to be written the stencil buffer is incremented:
glClear( GL_STENCIL_BUFFER_BIT );
glEnable( GL_STENCIL_TEST );
glStencilOp( GL_KEEP, GL_KEEP, GL_INCR );
glStencilFunc( GL_EQUAL, 0, 255 );
// do the drawing with the color
glDisable( GL_STENCIL_TEST );
Extension to the answer
Note that you can discard fragments which should not be drawn.
If the fragment in your sprite texture has an alpha channel of 0 you should discard it.
Note, if you discard a fragment neither the color nor the depth and stencil buffer will be written.
Fragment shaders also have access to the discard command. When executed, this command causes the fragment's output values to be discarded. Thus, the fragment does not proceed on to the next pipeline stages, and any fragment shader outputs are lost.
Fragment shader
if ( color.a < 1.0/255.0 )
discard;

It's not possible to do this using the fixed-function blending in OpenGL ES 2.0, because what you want isn't actually alpha blending. What you want is a logical operation (e.g. max(src, dst)) which is rather different to how OpenGL ES blending works.
If you want to do path / stroke / fill rendering with pixel-exact edges you might get somewhere with using stencil masks and stencil tests, but you can't do transparency in this case - just boolean operators.

GLSL - fragment shaders for image manipulation/blending

I'm trying to build a shader that allows you to combine two images by applying a gradient opacity mask to the one on top, like you do in photoshop. I've gotten to the point where I can overlay the masked image over the other but as a newbie am confused about a few things.
It seems that the images inside the shader are sometimes skewed to fit the canvas size, and they always start at position 0,0. I have played around with a few snippets I have found to try and scale the textures, but always end up with unsatisfactory results.
I am curious if there is a standard way to size, skew, and translate textures within a view, or if images in GLSL are necessarily limited in some way that will stop me from accomplishing my goal.
I'm also unsure of how I am applying the gradient/mask and if it is the right way to do it, because I do not have a lot of control over the shape of the gradient at the moment.
Here's what I have so far:
precision highp float;
varying vec2 uv;
uniform sampler2D originalImage;
uniform sampler2D image;
uniform vec2 resolution;
void main(){
float mask;
vec4 result;
vec2 position = gl_FragCoord.xy / ((resolution.x + resolution.y) * 2.0 );
mask = vec4(1.0,1.0,1.0,1.0 - position.x).a;
vec4 B = texture2D(image,uv);
vec4 A = texture2D(originalImage,uv) * (mask);
result = mix(B, A, A.a);
gl_FragColor = result;
}
Which produces an image like this:
What I would like to be able to do is change the positions of the images independently and also make sure that they conform to their proper dimensions.
I have tried naively shifting positions like this:
vec2 pos = uv;
pos.y = pos.y + 0.25;
texture2D(image, pos)
Which does shift the texture, but leads to a bunch of strange lines dragging:
I tried to get rid of them like this:
gl_FragColor = uv.y < 0.25 ? vec4(0.0,0.0,0.0,0.0) : result;
but it does nothing

You really need to decide what you want to happen when images are not the same size. What you probably want is for it to appear there's no image so check your UV coordinates and use 0,0,0,0 when outside of the image
//vec4 B = texture2D(image,uv);
vec4 getImage(sampler2D img, vec2 uv) {
if (uv.x < 0.0 || uv.x > 1.0 || uv.y < 0.0 || uv.y > 1.0) {
return vec4(0);
}
return texture2D(img, uv);
}
vec4 B = getImage(image, uv);
As for a standard way to size/skew/translate images use a matrix
uniform mat4 u_imageMatrix;
...
vec2 newUv = u_imageMatrix * vec4(uv, 0, 1).xy;
An example of implementing canvas 2d's drawImage using a texture matrix.
In general though I don't think most image manipulations programs/library would try to do everything in the shader. Rather they'd build up the image with very very simple primitives. My best guess would be they use a shader that's just A * MASK then draw B followed by A * MASK with blending on.
To put it another way, if you have 30 layers in photoshop they wouldn't generate a single shader that computes the final image in one giant shader taking in all 30 layers at once. Instead each layer would be applied on its own with simpler shaders.
I also would expect them to create an texture for the mask instead of using math in the shader. That way the mask can be arbitrarily complex, not just a 2 stop ramp.
Note I'm not saying you're doing it wrong. You're free to do whatever you want. I'm only saying I suspect that if you want to build a generic image manipulation library you'll have more success with smaller building blocks you combine rather than trying to do more complex things in shaders.
ps: I think getImage can be simplified to
vec4 getImage(sampler2D img, vec2 uv) {
return texture2D(img, uv) * step(0.0, uv) * step(-1.0, -uv);
}

Why doesn't my simple fragment shader work on iOS but not Android under Cocos2d-x 3.2?

I have modified a working grey_scale fragment shader to change all of the non-transparent pixels to purple. For some reason it works great on iOS but on Android the transparent parts of the image are visible (although mostly transparent). Can anybody see what I am doing wrong?
The working grey_scale shader contains the lines that are commented out. I added the last line.
#ifdef GL_ES
precision mediump float;
#endif
varying vec4 v_fragmentColor;
varying vec2 v_texCoord;
void main(void)
{
vec4 c = texture2D(CC_Texture0, v_texCoord);
//gl_FragColor.xyz = vec3(0.2126*c.r + 0.7152*c.g + 0.0722*c.b);
//gl_FragColor.w = c.w;
gl_FragColor = vec4(0.5, 0.0, 0.4, c.w);
}

It turns out that I need to apply the alpha to all of the colors:
gl_FragColor = vec4(0.5*c.w, 0.0*c.w, 0.4*c.w, c.w);
I am sure why the old method didn't work. The image uses premultiplied alpha, so I guess the cocos render assumes it (or was somehow told to use it by TexturePacker). So the shader needs to re-multiply the color values by the alpha in order to behave the same way?

A screenshot will be worth a thousands of words.
Are you sure GL_ES is defined? If not, you will have unspecified precision for float (according to specs, it is unspecified for fragment shaders) which can lead even to compilation errors on certain OpenGL ES drivers. I'd play around with that float precision first and see the difference on Android.
I'm not sure about vec3() w/ single parameter. Does the following notation work:
float a = 0.2126*c.r + 0.7152*c.g + 0.0722*c.b;
gl_FragColor.xyz = vec3(a, a, a);
Or this, with a single assignment of gl_FragColor:
float a = 0.2126*c.r + 0.7152*c.g + 0.0722*c.b;
gl_FragColor.xyz = vec4(a, a, a, c.w);
On a side note, you may want to declare numeric literals as constants in order not to consume uniform space. More info: Declaring constants instead of literals in vertex shader. Standard practice, or needless rigor?
Like this:
const float COEFF_R = 0.2126;
const float COEFF_G = 0.7152;
const float COEFF_B = 0.0722;
float a = COEFF_R*c.r + COEFF_G*c.g + COEFF_B*c.b;
gl_FragColor.xyz = vec4(a, a, a, c.w);

Tear when not expected after re-sizing the texture

In certain circumstances, there is a desire to expand a texture when the user is zooming into an image until the shortest edge fills the entire screen (as seen with standard galleries). However, the input to the OpenGL shader is a matrix from an ImageView, so certain translations need to be put into place as well as corrective calculations for different spaces.
EDIT:
The vertex shader code is simplistic and is as follows:
private final String vertexShader_ =
"attribute vec4 a_position;\n" +
"attribute vec4 a_texCoord;\n" +
"varying vec2 v_texCoord;\n" +
"void main() {\n" +
" gl_Position = a_position;\n" +
" v_texCoord = a_texCoord.xy;\n" +
"}\n";
where a_position references verticesData_ below. When loading onto the screen, the position of the image is accurate based upon calculations from the display width and a portion of the screen that is occupies.
Additionally, I have the following fragment shader code:
private final String fragmentShader_ =
"precision mediump float;\n" +
"varying vec2 v_texCoord;\n" +
"uniform sampler2D texture;\n" +
"uniform mat3 transform;\n" +
"void main() {\n" +
" vec2 uv = (transform * vec3(v_texCoord, 1.0)).xy;\n" +
" gl_FragColor = texture2D( texture, uv );\n" +
"}\n";
where the input of the mat3 transform is input that is coming from an ImageView. Essentially, there is an ImageView underneath the OpenGLSurfaceView. It is a much lower resolution image and when the user swipes, the SurfaceView is hidden and the ImageView is beneath it at the same position that the user was in the SurfaceView.
However, later on when I want to expand upon this texture, I am finding myself will unexpected results.
When panning back and forth, the screen moves where it is expected to move. So, the translation in the x and y coordinates from the matrix are coming across ok. However, when zooming in, the image is tearing. It is only tearing when the bounds of the texture have grown beyond the screen dimensions. As can be seen below, the height is not introducing extra pixels when growing, but the width is tearing as it progresses.
In order to pass an appropriate matrix, the values from the ImageView to the OpenGL SurfaceView were inverted and transposed, since the space required it for an appropriate conversion. A scale factor is passed into the a listener that the Activity is listening in on.
#Override
public void onTranslation(float imageScaleF, float[] matrixTranslationF)
{
currentScaleForImage_ = imageScaleF;
//If we are currently using the entire screen, then there is not need anymore to resize the texture itself.
if(surfaceViewWidth_ * imageScaleF > displayViewWidth_ && surfaceViewHeight_ * imageScaleF > displayViewHeight_)
{
}
else //Expand the size of the texture to be displayed on the screen.
{
maximumScaleForImage_ = imageScaleF;
photoViewRenderer_.updateTextureSize(imageScaleF);
photoSurfaceView_.requestRender();
}
matrixTranslationF = updateTranslationBasedOnTextureSize(matrixTranslationF);
photoViewRenderer_.updateTranslationOfTexture(matrixTranslationF);
photoSurfaceView_.requestRender();
}
But, with the following code above, the portion of the image that scrolls is always cut short, and if I attempt to correct it based upon the scale by uncommenting the line there for updating the Translation, it causes tears. So, it seems as though at this point, the user below has gotten me into a better position with their input, but I am still a single step away and I think that it is within this function. The following now updated code below has made the corrections necessary in order to provide an appropriate translation between the matricies in the ImageView and the OpenGL texture/coordinates.
private float[] updateTranslationBasedOnTextureSize(float[] matrixTranslationsF)
{
if(scaleDirection_ == ScaleDirection.WIDTH)
{
matrixTranslationsF[0] = matrixTranslationsF[0] * maximumScaleForImage_;
}
else if(scaleDirection_ == ScaleDirection.HEIGHT)
{
matrixTranslationsF[4] = matrixTranslationsF[4] * maximumScaleForImage_;
}
return matrixTranslationsF;
}
I was causing tearing in the photographs in the first iteration as seen below. Or it was clipping the photograph due to improper conversions between the two spaces.
The width seems to be maintained, so it suggests to me that maybe something happens with values outside the standard bounds, but I am not sure what to try.

This streaking is happening because you have the texture wrap set to clamp to edge. You have something like:
glTexParameteri( GL_TEXTURE_2D, GL_TEXTURE_WRAP_S, GL_CLAMP_TO_EDGE );
glTexParameteri( GL_TEXTURE_2D, GL_TEXTURE_WRAP_T, GL_CLAMP_TO_EDGE );
GL_REPEAT is the default and will repeat the image. But you're looking to not have any "effect" at the bottom. This bottom is happening because of your MATH. You are getting the streaking only on the bottom and not the sides. As someone said in the comment you're vertex range isn't correct. The min and max are 0 and 1.. not -1 and 1. So at the bottom of your picture it's streaking because you're calculating a texture coordinate number out of the 0-1 bounds and probably getting in to the <0 area or the >1 area. When the texture2D(u_textureSampler, v_fragmentTexCoord0) gets a texture coordinate outside of the 0-1 range it's going to fall on that WRAP_S / WRAP_T setting to know what value to give. It's repeating the edge pixel because you have GL_CLAMP_TO_EDGE.
Your actual problem though isn't this setting. It's that you are calculating an S or T outside of the range to begin with. That's not good form cause even if you were writing black in there it's wasted cycles on the fragment shader and will slow down your app. Basically, your formulas are wrong.
As for your zooming in math... it's math. You'll have to work it out for whatever you're trying to do. (Or ask for help.) You could do something like this, which is a simplified version of what I'm doing in my new photo editing app:
GLfloat aspect = photo.width / photo.height;
if (aspect > 1) {
displayWidth = panelWidth;
displayHeight = panelWidth/aspect;
}
else {
displayHeight = panelHeight;
displayWidth = panelHeight*destAspect;
}
Where panelWidth and panelHeight are the size of the area you're displaying your photo - which could be the full screen or something smaller. Then build your vertex based on the display height and width. If you zoom in beyond the size of the panel you'll need to then do a crop to keep the image inside the panel, both by reducing the display width or height (which ever one is too large) and also changing your texture coordinates to match. This is very rough but something like this:
GLfloat sSize = (destSMax-destSMin) / panelZoom;
GLfloat tSize = (destTMax-destTMin) / panelZoom;
displaySMin = destSMin + panelPanX/coreTextureSize;
displayTMin = destTMin + panelPanY/coreTextureSize;
displaySMax = displaySMin + sSize;
displayTMax = displayTMin + tSize;
Then of course build your vertex data, use matrix math to move the square where you want it, and draw it.
I recommend rethinking your approach to using something more like what I'm doing above. Don't remove a percentage of the photo when it goes off screen. Instead calculate what is going to be seen and build your data based on that.
EDIT - Also, I realized my code is ios code and this is an android question. Same concept applies though. The glTexParameteri call might be slightly different and it might be float instead of GLfloat.