Monami

I need you help, I am trying to make a hsflow patch with shader written in CG.

Despite the limited documentation about jitter and Cg I manage to put up something togheter, but I have one problem that I can't solve. It alsways output 0.0...

Can somebody look at it? I hope someone will see where is my mistake, because I am clueless

And by the way does anybody know a where i can find a good resume of the doc about Cg I am specialy looking for the explanation of the semantics (not sure if it is the right name for them), in examples the auto named variable like state.matrix.mvp or state.matrix.texture[0]...

Also does anybody know if the Cgfx file are supported or know a way to have technic and passes? It would saved alot of texture memory since slab always have a input texture and an output texture...

I need you help, I am trying to make a hsflow patch with shader written in CG. 

Can somebody look at it? I hope someone will see where is my mistake, because I am clueless 

slab-feedback-with-shader-in-cg

The patch isnot the final one, it is only the part that ihave identified that isin't behaving accordingly. When you strat it, it should at least not output black.

Can Anybody help? I have seen this kind of feed back to the slab often but. I don't get why it dosn't work.

I'll try to explain the shader maybe you would understand it better... Basically it is a optical flow more text book that Adrew did... I kow it can work I have doe it in hlsl in a game at school. And I want to recreate the same effect...

Everything esential is in the pixel shader, unless I have done a stupid mistake in the vertex shader, that I have't seenin a week..

So I basicaly sample the first texture in the inlet 0. this should be the gradient (part of a more complex patch)

Then I samble the texture that contain the last Flow that is feedback in the inlet 1. I samplearound the current textcoord to make a mean.

The It is the equation for the Flow that I put in he variable optFlowUV and I output it...

This is the paper I where i took the operation.

www.ece.iastate.edu/~namrata/EE527/OpticalFlow.pdf

I really hope someone can help me with this one...

This is the paper I where i took the operation.
www.ece.iastate.edu/~namrata/EE527/OpticalFlow.pdf

Just to tease a bit. This is the effect the more complex will acheive, I hope. It is a screenshot from the game we made.

I've recently been looking at cg shaders and can comment on a few issues I see in your example. However, my comments relate to the jitter format and not to the details of your shader. My graphics card doesn't support textures in the vertex shader so I haven't tried to take this any further. But hopefully this will help.

You need to expose the parameters that Jitter will use in your jxs file, this includes the state parameters. So, for example, the modelview projection matrix is exposed with the following line:

where matViewProj is the name you provide for the parameter.

You will also bind the parameter to the shader (again in your jxs file) with the line:

The entry point to the cg shader also needs to be detailed in the jxs file. So something like this:

I've recently been looking at cg shaders and can comment on a few issues I see in your example.  However, my comments relate to the jitter format and not to the details of your shader.  My graphics card doesn't support textures in the vertex shader so I haven't tried to take this any further.  But hopefully this will help.

You need to expose the parameters that Jitter will use in your jxs file, this includes the state parameters.  So, for example, the modelview projection matrix is exposed with the following line:

You will also bind the parameter to the shader (again in your jxs file) with the line:
For samplers the format is similar:

The entry point to the cg shader also needs to be detailed in the jxs file. So something like this:
	 

I am also only startng CG shader, but i have some experience with hlsl and GLSL. I see in you post that you are using glsl «king» of semantic Am-I right? I read a little bit the CG user manual and they suggest to use CG specific semantics name to stay portable I guess. So MODELVIEW_PROJECTION_MATRIX become state.matrix.mvp.

My next question about your post is, is it an absolute must,i mean by that, that the shader will not work... Because I am looking now at an other shader that I did in cg, it does a gradient in x, y and t and I didn't declare it in the jsx file but I have declare it in the cg vertex shader this way.

uniform float4x4 matViewProj: state.matrix.mvp;

And the compiler didn't complain and it was working.

As for the jsx file I though it was only for the parameters that you wnat to be access from the patcher.

An other question that I had is for the uv coordinates when you have 2 textures (one in each inlet) do I have really to use:

uniform float4x4 matTexture: state.matrix.texture[0];

uniform float4x4 matTexture1: state.matrix.texture[1];

OUT.uv = mul(matTexture, float4(IN.uv, 0.0, 1.0)).xy;

OUT.uv1 = mul(matTexture1, float4(IN.uv1, 0.0, 1.0)).xy;

Same thing for the entry point it wasn't declare in the jsx file of my gradient shader and the compiler don't complain.

And finaly, sorry for the many questions...

I don't get why you are talking about vertex texture fetching.. all the texture sempler are done in the pixel shader...

Again thanks so much for you answer... I saw alot of your post around in the forum you rock...

I'll post again the patch with the complete exaustive jsx file, just to be on the safe side...

My next question about your post is, is it an absolute must,i mean by that, that the shader will not work... Because I am looking now at an other shader that I did in cg, it does a gradient in x, y  and t and I didn't declare it in the jsx file but I have declare it in the cg vertex shader this way.

uniform float4x4 matTexture: state.matrix.texture[0];
uniform float4x4 matTexture1: state.matrix.texture[1];

   OUT.uv = mul(matTexture, float4(IN.uv, 0.0, 1.0)).xy;
   OUT.uv1 = mul(matTexture1, float4(IN.uv1, 0.0, 1.0)).xy;

struct vpOutput
{
   float4 position : POSITION;
   float2 uv : TEXCOORD0;
   float2 uv1 : TEXCOORD1;
   float3 objectPos : TEXCOORD2;
};

I don't get why you are talking about vertex texture fetching.. all the texture sempler are done in the pixel shader... 

I'll post again the patch with the complete exaustive jsx file, just to be on the safe side...


Here is the same patch with the more complete jsx But still only black...

Any help to find why the shader is not working is appreciated... I tried to send the output of the feedback slab to a texture and to show it in a pwindow but this is also always black...

Here is the same patch with the more complete jsx But still only black... 

In your JXS file you need to specify your texture inputs using the and tags. Otherwise, the jit.gl.slab object won't pass the textures to the shader. Look at the shaders in the jitter-shaders folder for an example.

Hello,
In your JXS file you need to specify your texture inputs using the  and  tags.  Otherwise, the jit.gl.slab object won't pass the textures to the shader.  Look at the shaders in the jitter-shaders folder for an example.

I changed your shader to a format that I'm getting familiar with. That's not to say your way is wrong, I don't know, but I find it easier to read this way. Anyway, it still doesn't work! I see I was wrong about using textures in the vertex shader, from what I can gather the vertex shader doesn't do anything but state the position and pass on the texture coordinates to the fragment shader. I don't think you need to compute the texture coordinates using the texture matrix as these are provided by openGL, so I took that part out.

I also changed the fragment shader to use the same format, but haven't done anything else because I don't understand what you're trying to do well enough to be able to suggest anything. I know I haven't helped, but here's what I ended up with. (By the way, its all one jxs file).

One basic thing that I'm not sure is correct is the binding of the second texture unit for the right inlet to the slab. I tried using default values like in GLSL but I'm not convinced that the shader is even receiving the texture. Perhaps someone else can comment?

Compute the optical flow of a texture by the Horn Schunk technic

Hi Mani
I changed your shader  to a format that I'm getting familiar with.  That's not to say your way is wrong, I don't know, but I find it easier to read this way.  Anyway, it still doesn't work!  I see I was wrong about using textures in the vertex shader, from what I can gather the vertex shader doesn't do anything but state the position and pass on the texture coordinates to the fragment shader.  I don't think you need to compute the texture coordinates using the texture matrix as these are provided by openGL, so I took that part out.

I also changed the fragment shader to use the same format, but haven't done anything else because I don't understand what you're trying to do well enough to be able to suggest anything.  I know I haven't helped, but here's what I ended up with.  (By the way, its all one jxs file).

One basic thing that I'm not sure is correct is the binding of the second texture unit for the right inlet to the slab.  I tried using default values like in GLSL but I'm not convinced that the shader is even receiving the texture.  Perhaps someone else can comment?

Thanks andrew, but in the last post I corrected this...

Also thanks alot Pelado, I see you have a very different way of writting shaders. But I understand it, so I'll comment more my code, and maybe you'll understand it better. As I am starting to suspect, there is something wrong with my code, or probably an assumption that I am doing and is wrong.

The two first assumptions I want to confirm is that, that first time a frame enter the shader, the texture bind to the second inlet, was cleared to 0? Is there a way to clear a texture, or the default texture for a slab is one filled with 0? ( I wish those kind of deep information was documented somewhere...)

The second is if I declare the type of the slab a float16, will it pass the negative value? or not? Usually when you render to a texture, (Not sure if this is the roght term), in a float16_r or float16_rgb it pass the negative value.

Because I know that the very first texture the shader will output, if the second inlet texture is at 0 by default will be negative because of this substraction in the shader:

optFlowUV.x = moyUV.x - grad.x * division;

optFlowUV.y = moyUV.y - grad.y * division;

moyUV will all be 0 because it originate from look up the the second texture and so the substraction will be negative.

The two first assumptions I want to confirm is that, that first time a frame enter the shader, the texture bind to the second inlet, was cleared to 0? Is there a way to clear a texture, or the default texture for a slab is one filled with 0? ( I wish those kind of  deep information was documented somewhere...)

float2 optFlowUV;
optFlowUV.x = moyUV.x - grad.x * division;
optFlowUV.y = moyUV.y - grad.y * division;

moyUV will all be 0 because it originate from look up the the second texture and so the substraction will be negative.


Ok here is the shader code From pelado with comments, you'll see there is no magic there. As I said in my last post I think one of my two assumption could be wrong and it could mybe explain it...

And again thanks... by the way Andrew I like your flow patch, you posted exacly at the same time I decided to do this projet for a class for our game. For those interested, this shader was written at first as a compositor in Ogre3D and I decided to converted it in cg, to learn cg.

i've made some modifications to your cg shader, which i think brings it closer (although still not there). you have to give it a negative lambda to see something, but at least you know pixels are getting drawn. i think it's just a matter of re-checking your equations.




Compute the optical flow of a texture by the Horn Schunk technic












    
    
    
    
    
    
    

     








 


i've made some modifications to your cg shader, which i think brings it closer (although still not there).  you have to give it a negative lambda to see something, but at least you know pixels are getting drawn. i think it's just a matter of re-checking your equations.




Compute the optical flow of a texture by the Horn Schunk technic












	
	
	
	
	
	
	

	 








 


Ah, that clears up a few things. It doesn't surprise me that the colour is yellowish because of the following line in the shader:

    colour = float4(optFlowUV.x, optFlowUV.y, 0.0, 1.0);    

Yeah, Mani you were right about the texture matrix.

	colour = float4(optFlowUV.x, optFlowUV.y, 0.0, 1.0);	

Nice one Rob.
Yeah, Mani you were right about the texture matrix.

Thanks, you are ight, it kind of work if the lambda is negative, which confirm that one of my assumption is false, a slab can't send out a negative value, which I know I can do in a float16_r or float16_rgb. Does anybody know how to do this? When I had done this I was rendering to a texture, is it what a slab do? or not?

also thanks alot everybody for your help.. I won't be able to answer for the next 6 week as I will be doing a trip in Thailand. But I'll come back with this in july.

Thanks, you are ight, it kind of work if the lambda is negative, which confirm that one of my assumption is false, a slab can't send out a  negative value, which I know I can do in a float16_r or float16_rgb.  Does anybody know how to do this? When I had done this I was rendering to a texture, is it what a slab do? or not?

jit.gl.slab @type float32 will give you unclamped output.

I am back and I am glad to announce you that I mange to get this working!!! thanks wes! you got the answer! It should be documented somewhere actually which type of gl texture are the type float, float16 and float32. Since in HLSL I had the same shader and a FLOAT16_RGB and it was signed output.. well

I have done a little app and included the source file of the shaders. They are not perfect and a little bit dirty but I think they will help people who wants to make shader in CG.

Since robtherich and i have different way of coding the shaders (both are working and produce the same result, well it should) I have included both file. So that people can compare.

so start the render context, grab the camera and chose one version of the optical flow...

I hope you'll like it and that it will help somepeople with cg shaders.

I have done a little app and included the source file of the shaders. They are not perfect and a little bit dirty but I think they will help people who wants to make shader in CG. 

Sorry to update this solved post, but I'm still have problem with this horn shader. I try to run the andrew's patch under 4.6, jitter 1.6.3, mac 10.5 and still have some troubles, even when I subsitute "et.optflow3.jxs" to rob's shader…

• error: jit.gl.shader: jit.gl.shader: error creating CG program : CG ERROR : The compile returned an error.

(23) : warning C1057: too little data in initialization

(23) : error C1057: too little data in initialization

(20) : warning C1057: too little data in initialization

(20) : error C1057: too little data in initialization

• error: jit.gl.shader: jit.gl.shader: error creating CG program : CG ERROR : The compile returned an error.
• error: -- START CG INFO LOG --
(23) : warning C1057: too little data in initialization
(23) : error C1057: too little data in initialization
• error: -- END CG INFO LOG --
• error: jit.gl.shader: jit.gl.shader: error creating CG program : CG ERROR : The compile returned an error.
• error: -- START CG INFO LOG --
(20) : warning C1057: too little data in initialization
(20) : error C1057: too little data in initialization
• error: -- END CG INFO LOG --


it's possible your graphics card doesn't like the shader, but who knows.

if you want to post a zip file of all the files you are using, someone can take a look and see if it's simply a syntax error.

it's possible your graphics card doesn't like the shader, but who knows.
if you want to post a zip file of all the files you are using, someone can take a look and see if it's simply a syntax error.


you had a couple errors with context names, but i assume those were typos (your window, and the feedback slab both had a context name of "combined" instead of "flow").

after fixing those, i didn't received any errors, and did get some output, so it's possibly related to your graphics card, or some max 4 weirdness.

In Max 5, it does work actually…But I need to Work under Max 4.6…maybe I'll send the optical.flow output through net.send localhost…

Sorry, for the render name…
In Max 5, it does work actually…But I need to Work under Max 4.6…maybe I'll send the optical.flow output through net.send localhost…

slab feedback with shader in CG