Develop Reference

Rendering from GL_TEXTURE_EXTERNAL_OES to GL_TEXTURE_2D but only one color flashes

I'm pretty new to opengl-es and currently have a problem rendering my video output on a screen in Unity.
I was developing a video player project with Unity. I bought the EasyMovieTexture plugin and replaced the video player module with another open-source video player (Ijkplayer) years ago which worked fine all the time.
Now I want to replace it with newer VLC using libvlcjni. I compiled and just replaced the old Ijkplayer but it didn't work as I expected. The screen just started flashing one color from every video frame but the video was going and the audio track was normally playing too.
Screenshot - A test scene with a screen, sorry there's mistake with texcoord but only color flashing
I'd like to provide some further information hope that I can find some help (Sorry if I have some mistakes or misunderstandings):
As far as I know, these video player modules need a Surface (Or SurfaceTexture) in the Android layer and the video decoder will work as a data producer. The SurfaceTexture consumes data from the producer and converts it to the texture with the type of GL_TEXTURE_EXTERNAL_OES which can be directly consumed and displayed with components like TextureView. But this texture data cannot be consumed in the Unity layer unless I use a GLSL shader directly sampling my OES texture data, like this:
// Unity shader with GLSL
GLSLPROGRAM
#pragma only_renderers gles3
#include "UnityCG.glslinc"
// ...
// Ignoring vertex shader
// ...
in vec2 textureCoord;
layout(binding = 0) uniform samplerExternalOES _MainTex;
out vec4 fragColor;
void main()
{
fragColor = texture(_MainTex, textureCoord);
}
ENDGLSL
My approach was to convert the texture to GL_TEXTURE_2D with the native library which came along with the EasyMovieTexture plugin. Here I cannot provide the source code of this .so library but I've decompiled it in IDAPro and I know it can work along with GLES and render the external texture data to another 2d texture using FrameBuffer Object and external shader program.
Here is a random example to explain the procedure, it is NOT the accurate code from the binary: FilterFBOTexture.java
Though I cannot edit the .so file, luckily it was reading two external files as the shader program:
// vertex shader
attribute highp vec3 inVertex;
attribute mediump vec3 inNormal;
attribute mediump vec2 inTexCoord;
uniform highp mat4 MVPMatrix;
uniform mediump vec2 TexCoordMove;
varying mediump vec2 TexCoord;
void main()
{
highp vec4 vPos = vec4(0,0,0,1);
vPos.x = ( inTexCoord.x * 2.0 - 1.0 );
vPos.y = ( inTexCoord.y * 2.0 - 1.0 );
gl_Position = vPos;
mediump vec4 vec4Temp = vec4(inTexCoord.x - TexCoordMove.x,inTexCoord.y - TexCoordMove.y,0,1.0);
vec4Temp = MVPMatrix * vec4Temp;
vec4Temp.xyz = vec4Temp.xyz / vec4Temp.w;
TexCoord = vec4Temp.xy;
}
// fragment shader
#extension GL_OES_EGL_image_external : require
uniform samplerExternalOES sTexture;
uniform lowp float AlphaValue;
varying mediump vec2 TexCoord;
void main()
{
lowp vec4 color = texture2D(sTexture, TexCoord) ;
color = vec4(color.rgb, color.a * AlphaValue);
gl_FragColor = color;
}
I don't know whether I have to check the vertex shader or just dive into the source code of libvlcjni so that I can correctly render my video output. Any idea will be grateful, thanks.
Update 2022-11-4:
I turned around and began to use VLC for Android.
Big appreciation to #mfkl, I created an issue days ago on the VLC repo.
https://code.videolan.org/videolan/vlc-unity/-/issues/164
The problem still remains but at least I can work for something now.

GL_OES_EGL_image_external_essl3 extension in compute shaders does not work properly

I use the GL_OES_EGL_image_external_essl3 extension to access camera pictures in GLSL. For fragment shaders it works fine. Here is my simple fragment shader:
#version 320 es
#extension GL_OES_EGL_image_external_essl3 : require
precision mediump float;
uniform samplerExternalOES cameraTexture;
in vec2 v_TexCoordinate;
out vec4 fragmentColor;
void main() {
fragmentColor = texture(cameraTexture, v_TexCoordinate);
}
I can see the picture from the camera.
However when I insert a simple compute shader stage in pipeline that only copies data from that external image to a new texture which I display, I can see only black screen. I also generate a red line in the compute shader for debugging.
Here is the code of that compute shader:
#version 320 es
#extension GL_OES_EGL_image_external_essl3 : require
precision mediump float;
layout(local_size_x = LOCAL_SIZE, local_size_y = LOCAL_SIZE) in;
layout(binding=1, rgba8) uniform mediump writeonly image2D outputImage;
uniform samplerExternalOES cameraTexture;
void main() {
ivec2 position = ivec2(gl_GlobalInvocationID.xy);
vec4 cameraColor = texture(cameraTexture, vec2(gl_GlobalInvocationID.xy)/1024.);
imageStore(outputImage, position, cameraColor);
// generate a red line to see that in general the texture
// that is produced by the compute shader is displayed on the
// screen
if (position.x == 100) imageStore(outputImage, position, vec4(1,0,0,1));
}
So it seems like it accesses the texture in the same way, but vec(0,0,0,1) is returned by texture() instead. So the screen is black.
In both cases I bind the texture like this:
glActiveTexture(GL_TEXTURE0)
glBindTexture(GLES11Ext.GL_TEXTURE_EXTERNAL_OES, mCameraTextureId)
glUniform1i(cameraUniformHandle, 0)
Why does this extension not work properly in my compute shader? Is it supposed to work in compute shaders at all?
My platform is Samsung Galaxy S7 (Mali GPU).

Standard derivatives from Fragment Shader (dFdx, dFdy), don't run correctly in Android 4.4

I'm using a fragment shader that uses dFdy dFdx functions to calculate the normal of the
face to view in a flat appearance. This shader has been running ok in gles 2.0 and 3.0. Inexplicably, shader don't work in Android 4.4 ( KitKat - gles3.0 ).
(Solved!!.. individual derivatives for each component, solve the problem).
In order check error, i prepared these shaders :
//Vertex Shader
#version 300 es
precision highp float;
precision highp int;
uniform mat4 PMatrix; //Projection Matrix (varies according to camera)
uniform mat4 MVMatrix; //Model View Matrix (no change)
in vec3 vPosition;
out vec3 vPos;
main()
{
gl_Position=PMatrix * MVMatrix * vec4(vPosition.xyz,1.0);
vPos = (MVMatrix * vec4(vPosition.xyz,1.0)).xyz;
}
// Fragment shader
#version 300 es
#extension GL_OES_standard_derivatives : enable
precision highp float;
precision highp int;
in vec3 vPos;
main()
{
// don't run correctly
// vec3 fdx = dFdx(vPos);
// vec3 fdy = dFdy(vPos);
// ***Solved!*** this run correctly in KitKat
vec3 fdx = vec3(dFdx(vPos.x),dFdx(vPos.y),dFdx(vPos.z));
vec3 fdy = vec3(dFdy(vPos.x),dFdy(vPos.y),dFdy(vPos.z));
vec3 N = normalize(cross(fdx,fdy));
fragColor = vec4(N,1.0);
}
Drawing a cube, in Android<4.4 colors remain fixed for each side, independently of camera position (correct, the color identifies each normal-face). In Android 4.4 , colors vary if you move the camera.
Analyzing each derived separately :
1.- fragColor=vec4(normalize( fdx ), 1.0); colors are constantly changing (wrong)
2.- fragColor=vec4(normalize( fdy ), 1.0); the colors remain quasi-stable (quasi-ok)
A bug in the implementation of these features in Android 4.4?
We are doing something wrong?

You said that this was originally a GLES2 shader, and you are using highp in a fragment shader unconditionally? That is a disaster waiting to happen because GLES2 implementations are not required to support highp in fragment shaders. Likewise, support for dFdx (...), dFdy (...) and fwidth (...) is optional.
You need to check for GL_OES_standard_derivatives and GL_FRAGMENT_PRECISION_HIGH in the GLES2 implementation of this fragment shader.
To that end, you might consider the accuracy hint for derivatives:
GL_FRAGMENT_SHADER_DERIVATIVE_HINT_OES (GLES2, if the extension is supported)
GL_FRAGMENT_SHADER_DERIVATIVE_HINT (GLES3)

Individual derivatives in each component solve the problem in KitKat:
// Replace vec3 fdx = dFdx(vPos) by:
vec3 fdx = vec3(dFdx(vPos.x),dFdx(vPos.y),dFdx(vPos.z));
// Replace vec3 fdy = dFdy(vPos) by:
vec3 fdy = vec3(dFdy(vPos.x),dFdy(vPos.y),dFdy(vPos.z));

I've seen similar things on some AMD desktop setups.
Where dFdy( .xyz ) worked fine on NVIDIA/intel I had to do the derivate per-component to get it correctly for some AMD cards.

OpenGL Shadar Language in Android

I'm learning OpenGL through a great book that uses OpenGL version 3.3 in the Premake 4 plataform, but I actually want to learn OpenGL for Android, which supports OpenGL version 2.0. It's pretty much the same, as the concepts are the same, but I'm having some trouble with the Shader Language code.
For example, in the book the autor implements a Vertex Shader with the code:
#version 330
layout(location = 0) in vec4 position;
void main(){
gl_Position = position;
}
After a lot of research I was able to implement this exactly Vertex Shader code in Android with the code:
String vertexShaderCode =
"attribute vec4 vPosition;" +
"void main() {" +
" gl_Position = vPosition;" +
"}";
But I can't figure out how can I implement this Vertex Shader below:
#version 330
layout (location = 0) in vec4 position;
layout (location = 1) in vec4 color;
smooth out vec4 theColor;
void main() {
gl_Position = position;
theColor = color;
}
The book is http://arcsynthesis.org/gltut/Basics/Tut02%20Vertex%20Attributes.html

You cannot implement the shader as it is written in OpenGL ES 2.0. This shader is written against the OpenGL 3.3 GLSL specification, OpenGL ES 2.0 uses syntax roughly equivalent to GLSL 120 (OpenGL 2.1).
The smooth qualifier is used to control interpolation between shader stages, GLSL ES does not support this, all vertex shader outputs are smoothly interpolated in GLSL ES.
layout (...) is unsupported by GLSL ES, you must use glBindAttribLocation (...) to set the location.
in and out are not supported by GLSL ES, you must use attribute for vertex shader inputs and varying for vertex shader outputs and fragment shader inputs.
And last, #version 330 is invalid. In OpenGL ES 2.0, there is only one GLSL ES specification, version 1.0 (so a compliant OpenGL ES 2.0 shader should begin with #version 100).
To answer your question about implementing that particular vertex shader:
#version 100
attribute vec4 position;
attribute vec4 color;
varying vec4 theColor;
void main() {
gl_Position = position;
theColor = color;
}
This is of course only half of the equation, you need to pair it with a fragment shader that also uses OpenGL ES 2.0-compliant syntax.
A minimal fragment shader that would work with this vertex shader:
#version 100
varying vec4 theColor; // In GLSL 330 this would be written as: in vec4 theColor;
void main (void) {
gl_FragColor = theColor;
}

OpenGL ES shader degrades too much performance

I'm optimizing a game that works for both, iPhone and Android. I'm using 4 shader to draw the scene and I noticed that if I change one of them to another the fps goes from 32 to 42, even though there's only 1 sprite being drawn with that shader, and the only difference in this 2 shaders is just a product in the fragmente shader.
These are the shaders:
default-2d-tex.shader
#ifdef GL_ES
precision highp float;
precision lowp int;
#endif
#ifdef VERTEX
uniform mat4 umvp;
attribute vec4 avertex;
attribute vec2 auv;
varying vec2 vuv;
void main()
{
// Pass the texture coordinate attribute to a varying.
vuv = auv;
// Here we set the final position to this vertex.
gl_Position = umvp * avertex;
}
#endif
#ifdef FRAGMENT
uniform sampler2D map0;
uniform vec4 ucolor;
varying vec2 vuv;
void main()
{
gl_FragColor = texture2D(map0, vuv) * ucolor;
}
#endif
default-2d-tex-white.shader
#ifdef GL_ES
precision highp float;
precision lowp int;
#endif
#ifdef VERTEX
uniform mat4 umvp;
attribute vec4 avertex;
attribute vec2 auv;
varying vec2 vuv;
void main()
{
// Pass the texture coordinate attribute to a varying.
vuv = auv;
// Here we set the final position to this vertex.
gl_Position = umvp * avertex;
}
#endif
#ifdef FRAGMENT
uniform sampler2D map0;
varying vec2 vuv;
void main()
{
gl_FragColor = texture2D(map0, vuv);
}
#endif
Again,
If I modify default-2d-tex.shader and remove the product "* ucolor", the fps goes from 32 to 42, and I'm using it for just one sprite in the scene!
Is this normal? Why is this shader being so slow and how can I improve it?
EDIT:
I see this performance slowdown on both iPod and Android in an equal ratio. Both are PowerVr SGX GPUs (iPod 3rd gen and Samsung Galaxy SL -PowerVR SGX 530-). iOS version is 4.1 and Android is 2.3.3
The sprite I'm drawing is scaled to fill the screen (scaled to 4x) and I'm drawing it once per frame. It's taken from a texture map so the texture is actually larger (1024x1024) but the portion taken is 80x120. Alpha blending is enabled.
EDIT 2
I made a mistake. The sprite is scaled 11x: its 32x48.
If I don't draw that sprite at all, fps goes to 45. I'm drawing a lot of sprites in the scene, why is that one taking so much time? Could it be because it's scaled so much?

When you remove "* ucolor" from the abovementioned code many things happen:
uniform ucolor becomes unused
GLSL compiler removes this uniform from the active uniforms set
when you have you program linked the ID of the missing uniform
becomes -1
when you do glUniform4fv( -1, value ) for the removed uniform it
just exits instantly, without any update
This is pretty much less work compared to having "* ucolor" in place.
And additionally, this is one less operation per fragment in the fragment shader.

Your problem is in your fragment shader in default-2d-tex.shader. You have
uniform vec4 ucolor;
That means, that each of your color's components (RGBA) will be converted to 32-bits float value. And this drops performance heavily.
Should be:
uniform lowp vec4 ucolor;