Forums > Jitter

slab feedback with shader in CG

May 4, 2009 | 6:23 pm

Hi everybody,

I need you help, I am trying to make a hsflow patch with shader written in CG.

Despite the limited documentation about jitter and Cg I manage to put up something togheter, but I have one problem that I can’t solve. It alsways output 0.0…

Can somebody look at it? I hope someone will see where is my mistake, because I am clueless

And by the way does anybody know a where i can find a good resume of the doc about Cg I am specialy looking for the explanation of the semantics (not sure if it is the right name for them), in examples the auto named variable like state.matrix.mvp or state.matrix.texture[0]…

Also does anybody know if the Cgfx file are supported or know a way to have technic and passes? It would saved alot of texture memory since slab always have a input texture and an output texture…

thanks

Mani


May 13, 2009 | 7:48 am

Hi everybody,

Sorry don’t want to be rude,

I still haven’t solved my problem…

The patch isnot the final one, it is only the part that ihave identified that isin’t behaving accordingly. When you strat it, it should at least not output black.

Can Anybody help? I have seen this kind of feed back to the slab often but. I don’t get why it dosn’t work.

I’ll try to explain the shader maybe you would understand it better… Basically it is a optical flow more text book that Adrew did… I kow it can work I have doe it in hlsl in a game at school. And I want to recreate the same effect…

Everything esential is in the pixel shader, unless I have done a stupid mistake in the vertex shader, that I have’t seenin a week..

So I basicaly sample the first texture in the inlet 0. this should be the gradient (part of a more complex patch)

Then I samble the texture that contain the last Flow that is feedback in the inlet 1. I samplearound the current textcoord to make a mean.

The It is the equation for the Flow that I put in he variable optFlowUV and I output it…

This is the paper I where i took the operation.
http://www.ece.iastate.edu/~namrata/EE527/OpticalFlow.pdf

I really hope someone can help me with this one…

thanks


May 13, 2009 | 7:51 am

Just to tease a bit. This is the effect the more complex will acheive, I hope. It is a screenshot from the game we made.

thanks


May 13, 2009 | 12:31 pm

I’ve recently been looking at cg shaders and can comment on a few issues I see in your example. However, my comments relate to the jitter format and not to the details of your shader. My graphics card doesn’t support textures in the vertex shader so I haven’t tried to take this any further. But hopefully this will help.

You need to expose the parameters that Jitter will use in your jxs file, this includes the state parameters. So, for example, the modelview projection matrix is exposed with the following line:

where matViewProj is the name you provide for the parameter.

You will also bind the parameter to the shader (again in your jxs file) with the line:

For samplers the format is similar:

The entry point to the cg shader also needs to be detailed in the jxs file. So something like this:

pelado


May 13, 2009 | 8:27 pm

Hi Pelado,

Thank you very much for your answer!

I am also only startng CG shader, but i have some experience with hlsl and GLSL. I see in you post that you are using glsl «king» of semantic Am-I right? I read a little bit the CG user manual and they suggest to use CG specific semantics name to stay portable I guess. So MODELVIEW_PROJECTION_MATRIX become state.matrix.mvp.

My next question about your post is, is it an absolute must,i mean by that, that the shader will not work… Because I am looking now at an other shader that I did in cg, it does a gradient in x, y and t and I didn’t declare it in the jsx file but I have declare it in the cg vertex shader this way.

uniform float4x4 matViewProj: state.matrix.mvp;

And the compiler didn’t complain and it was working.

As for the jsx file I though it was only for the parameters that you wnat to be access from the patcher.

An other question that I had is for the uv coordinates when you have 2 textures (one in each inlet) do I have really to use:

uniform float4x4 matTexture: state.matrix.texture[0];
uniform float4x4 matTexture1: state.matrix.texture[1];

and then multiply

OUT.uv = mul(matTexture, float4(IN.uv, 0.0, 1.0)).xy;
OUT.uv1 = mul(matTexture1, float4(IN.uv1, 0.0, 1.0)).xy;

OUT is declare this way:

struct vpOutput
{
float4 position : POSITION;
float2 uv : TEXCOORD0;
float2 uv1 : TEXCOORD1;
float3 objectPos : TEXCOORD2;
};

Same thing for the entry point it wasn’t declare in the jsx file of my gradient shader and the compiler don’t complain.

And finaly, sorry for the many questions…

I don’t get why you are talking about vertex texture fetching.. all the texture sempler are done in the pixel shader…

Again thanks so much for you answer… I saw alot of your post around in the forum you rock…

ciao

I’ll post again the patch with the complete exaustive jsx file, just to be on the safe side…


May 13, 2009 | 9:01 pm

Here is the same patch with the more complete jsx But still only black… Sad

Any help to find why the shader is not working is appreciated… I tried to send the output of the feedback slab to a texture and to show it in a pwindow but this is also always black…

thanks

Mani


May 14, 2009 | 12:15 am

Hello,
In your JXS file you need to specify your texture inputs using the

and tags. Otherwise, the jit.gl.slab object won’t pass the textures to the shader. Look at the shaders in the jitter-shaders folder for an example.

Andrew B.


May 14, 2009 | 7:41 am

Hi Mani
I changed your shader to a format that I’m getting familiar with. That’s not to say your way is wrong, I don’t know, but I find it easier to read this way. Anyway, it still doesn’t work! I see I was wrong about using textures in the vertex shader, from what I can gather the vertex shader doesn’t do anything but state the position and pass on the texture coordinates to the fragment shader. I don’t think you need to compute the texture coordinates using the texture matrix as these are provided by openGL, so I took that part out.

I also changed the fragment shader to use the same format, but haven’t done anything else because I don’t understand what you’re trying to do well enough to be able to suggest anything. I know I haven’t helped, but here’s what I ended up with. (By the way, its all one jxs file).

One basic thing that I’m not sure is correct is the binding of the second texture unit for the right inlet to the slab. I tried using default values like in GLSL but I’m not convinced that the shader is even receiving the texture. Perhaps someone else can comment?

pelado


Compute the optical flow of a texture by the Horn Schunk technic





< ![CDATA[

void opticalFlow_vert(
float4 position : POSITION,
float2 uv : TEXCOORD0,
float2 uv1 : TEXCOORD1,
out float4 oPosition : POSITION,
out float2 oUv : TEXCOORD0,
out float2 oUv1 : TEXCOORD1,
uniform float4x4 matViewProj
)
{
oPosition = mul(matViewProj, position);
}
]]>

< ![CDATA[

void opticalFlow_frag(
float2 oUv : TEXCOORD0,
float2 oUv1 : TEXCOORD1,
out float4 colour : COLOR,
uniform samplerRECT inGrad,
uniform samplerRECT inUV,
uniform float lambda,
uniform float feedback
)
{
float alpha = 4.0*lambda;

float3 grad = texRECT(inGrad, oUv).xyz;

float2 UV0 = texRECT(inUV, oUv1).xy*100000.0;
float2 UVx1 = texRECT(inUV, oUv1 + float2(1.0, 0.0)).xy * feedback;
float2 UVx2 = texRECT(inUV, oUv1 + float2(-1.0, 0.0)).xy * feedback;
float2 UVy1 = texRECT(inUV, oUv1 + float2(0.0, 1.0)).xy * feedback;
float2 UVy2 = texRECT(inUV, oUv1 + float2(0.0, -1.0)).xy * feedback;

float2 moyUV = (UVx1 + UVx2 + UVy1 + UVy2)/4.0;

float division = 0.0;

if( grad.x != 0.0 || grad.y != 0.0 || grad.z != 0.0)
division = (grad.x*moyUV.x + grad.y*moyUV.y + grad.z) / (alpha + pow(grad.x, 2.0) + pow(grad.y, 2.0));

float2 optFlowUV;
optFlowUV.x = moyUV.x – grad.x * division;
optFlowUV.y = moyUV.y – grad.y * division;

// what it should return
colour = float4(optFlowUV.x, optFlowUV.y, 0.0, 1.0);
}

]]>


May 14, 2009 | 8:41 pm

Thanks andrew, but in the last post I corrected this…

Also thanks alot Pelado, I see you have a very different way of writting shaders. But I understand it, so I’ll comment more my code, and maybe you’ll understand it better. As I am starting to suspect, there is something wrong with my code, or probably an assumption that I am doing and is wrong.

The two first assumptions I want to confirm is that, that first time a frame enter the shader, the texture bind to the second inlet, was cleared to 0? Is there a way to clear a texture, or the default texture for a slab is one filled with 0? ( I wish those kind of deep information was documented somewhere…)

The second is if I declare the type of the slab a float16, will it pass the negative value? or not? Usually when you render to a texture, (Not sure if this is the roght term), in a float16_r or float16_rgb it pass the negative value.

Because I know that the very first texture the shader will output, if the second inlet texture is at 0 by default will be negative because of this substraction in the shader:

float2 optFlowUV;
optFlowUV.x = moyUV.x – grad.x * division;
optFlowUV.y = moyUV.y – grad.y * division;

moyUV will all be 0 because it originate from look up the the second texture and so the substraction will be negative.


May 14, 2009 | 8:51 pm

Ok here is the shader code From pelado with comments, you’ll see there is no magic there. As I said in my last post I think one of my two assumption could be wrong and it could mybe explain it…

And again thanks… by the way Andrew I like your flow patch, you posted exacly at the same time I decided to do this projet for a class for our game. For those interested, this shader was written at first as a compositor in Ogre3D and I decided to converted it in cg, to learn cg.


Compute the optical flow of a texture by the Horn Schunk technic





< ![CDATA[

void opticalFlow_vert(
float4 position : POSITION,
float2 uv : TEXCOORD0,
float2 uv1 : TEXCOORD1,
out float4 oPosition : POSITION,
out float2 oUv : TEXCOORD0,
out float2 oUv1 : TEXCOORD1,
uniform float4x4 matViewProj
)
{
oPosition = mul(matViewProj, position);
}
]]>

< ![CDATA[

void opticalFlow_frag(
float2 oUv : TEXCOORD0,
float2 oUv1 : TEXCOORD1,
out float4 colour : COLOR,
uniform samplerRECT inGrad,
uniform samplerRECT inUV,
uniform float lambda,
uniform float feedback
)
{
float alpha = 4.0*lambda;

// look up in the first inlet texture now it is only > 0.0 but in the end it could be bigger or smaller than 0.
float3 grad = texRECT(inGrad, oUv).xyz;

// look up in the second inlet texture the first pass should be all 0 but then it could be bigger or smaller than 0.
float2 UVx1 = texRECT(inUV, oUv1 + float2(1.0, 0.0)).xy * feedback;
float2 UVx2 = texRECT(inUV, oUv1 + float2(-1.0, 0.0)).xy * feedback;
float2 UVy1 = texRECT(inUV, oUv1 + float2(0.0, 1.0)).xy * feedback;
float2 UVy2 = texRECT(inUV, oUv1 + float2(0.0, -1.0)).xy * feedback;

// compute the mean of the precedent look up, again the first will be 0 all the next one ….
float2 moyUV = (UVx1 + UVx2 + UVy1 + UVy2)/4.0;

float division = 0.0;
//I don’t want to divide by 0
if( grad.x != 0.0 || grad.y != 0.0 || grad.z != 0.0)
division = (grad.x*moyUV.x + grad.y*moyUV.y + grad.z) / (alpha + pow(grad.x, 2.0) + pow(grad.y, 2.0)); // This is a term in the equation to compute the optical flow even the first frame shouldn’t be 0 because of the grad.z which will be none 0.

// Computing the optical flow in x and y nothing really magic here. Beside again the first frame MoyUV will all be 0.
float2 optFlowUV;
optFlowUV.x = moyUV.x – grad.x * division;
optFlowUV.y = moyUV.y – grad.y * division;

// what it should return
colour = float4(optFlowUV.x, optFlowUV.y, 0.0, 1.0);
}

]]>


May 15, 2009 | 6:51 pm

i’ve made some modifications to your cg shader, which i think brings it closer (although still not there). you have to give it a negative lambda to see something, but at least you know pixels are getting drawn. i think it’s just a matter of re-checking your equations.




Compute the optical flow of a texture by the Horn Schunk technic












	
	
	
	
	
	
	

	 

< ![CDATA[

void opticalFlow_vert(
	float4 position : POSITION,
	float4 uv : TEXCOORD0,
	float4 uv1 : TEXCOORD1,
	out float4 oPosition : POSITION,
	out float2 oUv : TEXCOORD0,
	out float2 oUv1 : TEXCOORD1,
	uniform float4x4 matViewProj,
	uniform float4x4 texMatrix,
	uniform float4x4 texMatrix1
	)
{
	oPosition = mul(matViewProj, position);
	oUv = mul(texMatrix, uv).xy;
	oUv1 = mul(texMatrix1, uv1).xy;
	//oUv = uv;
	//oUv1 = uv1;
}
]]>



< ![CDATA[

void opticalFlow_frag(
	float2 oUv : TEXCOORD0,
	float2 oUv1 : TEXCOORD1,
	out float4 colour : COLOR,
	uniform samplerRECT inGrad,
	uniform samplerRECT inUV,
	uniform float lambda,
	uniform float feedback
	)
{
	float alpha = 4.0*lambda;

	float3 grad = texRECT(inGrad, oUv).xyz;

	float2 UV0 = texRECT(inUV, oUv1).xy*100000.0;
	float2 UVx1 = texRECT(inUV, oUv1 + float2(1.0, 0.0)).xy * feedback;
	float2 UVx2 = texRECT(inUV, oUv1 + float2(-1.0, 0.0)).xy * feedback;
	float2 UVy1 = texRECT(inUV, oUv1 + float2(0.0, 1.0)).xy * feedback;
	float2 UVy2 = texRECT(inUV, oUv1 + float2(0.0, -1.0)).xy * feedback;

	float2 moyUV = (UVx1 + UVx2 + UVy1 + UVy2)/4.0;

	float division = 1.0;

	if( grad.x != 0.0 || grad.y != 0.0 || grad.z != 0.0)
		division = (grad.x*moyUV.x + grad.y*moyUV.y + grad.z) / (alpha + pow(grad.x, 2.0) + pow(grad.y, 2.0));

	float2 optFlowUV;
	optFlowUV.x = moyUV.x - grad.x * division;
	optFlowUV.y = moyUV.y - grad.y * division;

	// what it should return
	colour = float4(optFlowUV.x, optFlowUV.y, 0.0, 1.0);
	//colour = texRECT(inGrad, oUv);
}

]]>



May 15, 2009 | 8:35 pm

Ah, that clears up a few things. It doesn’t surprise me that the colour is yellowish because of the following line in the shader:

colour = float4(optFlowUV.x, optFlowUV.y, 0.0, 1.0);

Nice one Rob.
Yeah, Mani you were right about the texture matrix.

pelado


May 18, 2009 | 2:56 am

Thanks, you are ight, it kind of work if the lambda is negative, which confirm that one of my assumption is false, a slab can’t send out a negative value, which I know I can do in a float16_r or float16_rgb. Does anybody know how to do this? When I had done this I was rendering to a texture, is it what a slab do? or not?

also thanks alot everybody for your help.. I won’t be able to answer for the next 6 week as I will be doing a trip in Thailand. But I’ll come back with this in july.

thks

Mani


May 18, 2009 | 10:04 pm

jit.gl.slab @type float32 will give you unclamped output.

wes


June 30, 2009 | 8:35 am

Hi all,

I am back and I am glad to announce you that I mange to get this working!!! thanks wes! you got the answer! It should be documented somewhere actually which type of gl texture are the type float, float16 and float32. Since in HLSL I had the same shader and a FLOAT16_RGB and it was signed output.. well

I have done a little app and included the source file of the shaders. They are not perfect and a little bit dirty but I think they will help people who wants to make shader in CG.

Since robtherich and i have different way of coding the shaders (both are working and produce the same result, well it should) I have included both file. So that people can compare.

so start the render context, grab the camera and chose one version of the optical flow…

I hope you’ll like it and that it will help somepeople with cg shaders.

And ofcourse thanks to you all!!

Mani



Ad.
March 1, 2010 | 9:45 am

Sorry to update this solved post, but I’m still have problem with this horn shader. I try to run the andrew’s patch under 4.6, jitter 1.6.3, mac 10.5 and still have some troubles, even when I subsitute "et.optflow3.jxs" to rob’s shader…

• error: jit.gl.shader: jit.gl.shader: error creating CG program : CG ERROR : The compile returned an error.
• error: — START CG INFO LOG –
(23) : warning C1057: too little data in initialization
(23) : error C1057: too little data in initialization
• error: — END CG INFO LOG –
• error: jit.gl.shader: jit.gl.shader: error creating CG program : CG ERROR : The compile returned an error.
• error: — START CG INFO LOG –
(20) : warning C1057: too little data in initialization
(20) : error C1057: too little data in initialization
• error: — END CG INFO LOG –



Ad.
March 5, 2010 | 9:25 am

Anyone could give me some tips ?
-Ad-



Ad.
April 8, 2010 | 8:00 pm

no one ?
Ad


April 9, 2010 | 6:08 pm

it’s possible your graphics card doesn’t like the shader, but who knows.
if you want to post a zip file of all the files you are using, someone can take a look and see if it’s simply a syntax error.



Ad.
April 9, 2010 | 8:04 pm

You’re right, here it is !

Ad.



Ad.
April 9, 2010 | 8:06 pm

Ad.

Attachments:
  1. Optical_FLOW.zip

April 9, 2010 | 10:36 pm

you had a couple errors with context names, but i assume those were typos (your window, and the feedback slab both had a context name of "combined" instead of "flow").

after fixing those, i didn’t received any errors, and did get some output, so it’s possibly related to your graphics card, or some max 4 weirdness.



Ad.
April 10, 2010 | 3:29 am

Sorry, for the render name…
In Max 5, it does work actually…But I need to Work under Max 4.6…maybe I’ll send the optical.flow output through net.send localhost…

Thx for your time anyway !
/Ad.


Viewing 23 posts - 1 through 23 (of 23 total)