Forums Jitter

kinectimage to opengl?


Per

Per
May 15, 2012 at 11:17am

kinectimage to opengl?

hi!

im using a kinect to mask out an image (a person, see attached image)
i then want to transform this to a openGl object and place it in a jit.phys.world object.

the thing i want to do is to:
take only the image (and mask out the background) and to get it appear as an opengl object that can interact with other objects in jit.phys.world

so:
how do I transform the person in the object to interact in the jit.phys.world?

– Pasted Max Patch, click to expand. –

[attachment=193818,3870]

Attachments:
  1. kinect.tiff
#63602

Rob Ramirez

Rob Ramirez
May 15, 2012 at 3:07pm

check out this thread: http://cycling74.com/forums/topic.php?id=39036

#229235

lembert.dome

lembert.dome
May 15, 2012 at 3:58pm

You can also try the last patch in this thread:

http://cycling74.com/forums/topic.php?id=38963

If you add a jit.phys.world and a jit.phys.body you will have an interactive object in jit.gl.

I have an adaptation of that patch set up for the kinect, doing just what you are looking for, but I won’t be able to access it for a day or so. I’ll post that later.

#229236

Per

Per
May 15, 2012 at 8:46pm

okej, thanks. post it when you got it!

thanks

one more question. Im trying meanwhile with jit.gl.nurbs

but how can I get rid of the nurbs that are zero (think I explained correctly), how can I get rid of the nurbs that not are bended or used?

– Pasted Max Patch, click to expand. –
#229237

lembert.dome

lembert.dome
May 15, 2012 at 9:40pm

That was the problem I was trying to solve while working on the patch I cited. I have tried various approaches to this, but basically here is the scoop:

When working with any matrix, be it a geometry matrix or something else, you are essentially working with an (x,y) grid whose dimensions cannot be changed dynamically, frame by frame. In addition to this, to draw a shape, the vertices must be in a specific order. This means that the matrix will always need to have the dimensions of (the widest part of the object, the tallest part of the object). The workaround is to set the values of “unused” cells to something else, either to the closest “used” cell or to some neutral value.

It is possible to make the “unused” cells invisible, but that would not help in a jit.phys.world.

There seems to be a limit to the size of the matrix which can be sent to a jit.phys.body for use as its shape; I have been able to do so with few vertices but not with the output of a kinect. I will be investigating this soon.

In the patch posted at http://cycling74.com/forums/topic.php?id=38963 , I set all of the “unused” cells to 0. The only difference between that patch and the adaptation for the Kinect is that the Z-value of all “used” cells is the same, making it flat. (I am away from home and won’t be near a Kinect for another day, hence the wait.) It is also possible to set the “unused” values to the nearest “used” values with a couple of xray.jit objects, but that sends the CPU use up significantly.

#229238

Per

Per
May 19, 2012 at 9:06pm

It is possible to make the “unused” cells invisible, but that would not help in a jit.phys.world.

how do I do this?? if using jit.gl.nurbs as above? how do I make unused cells invisible. being trying different ways but Im kind of beginner in openGL and cant figure it out …

– Pasted Max Patch, click to expand. –
#229239

zthomas

zthomas
May 22, 2012 at 5:23pm

“There seems to be a limit to the size of the matrix which can be sent to a jit.phys.body for use as its shape; I have been able to do so with few vertices but not with the output of a kinect. I will be investigating this soon.”

Any insight into this yet? I am dealing with the same problem.

#229240

lembert.dome

lembert.dome
May 22, 2012 at 7:51pm

Hi. I got swept away with another project and haven’t plugged a Kinect in for a couple of weeks. I’ll give it a try later tonight and post what I find. One hopeful thing I noticed, though, is the patch posted here: http://cycling74.com/forums/topic.php?id=39036 uses a jit.phys.body collision shape with a large matrix. I believe the shape type is set to “convexhull” rather than “dynamicmesh” Could that be the answer?

#229241

zthomas

zthomas
May 23, 2012 at 8:22pm

This seems like it could work! Downsampling the matrix and setting the shape to convexhull allows for a collision shape to be rendered without doing much harm to the fps. The problem remains as you mentioned above, in setting the unused values to the nearest used values. The phys.body that I’m getting is unfortunately just a box-shaped point cloud.

#229242

lembert.dome

lembert.dome
May 23, 2012 at 9:56pm

I worked on this a bunch this morning, and got some things to work without being completely satisfied. I used the CameraToPhys2d patch mentioned earlier, exchanged the camera for the Kinect and set the jit.phys.body shape to dynamicmesh.
I had to reduce the size of the matrix from the Kinect before the dynamicmesh would work, so there is a reduction-matrix between the gen object generating geometry and the jit.phys.object.
Even so, it is a bit strange. This seems to come from the way the collision detection works, but this patch is a great way to look at that.
The convexhull idea does not work well because it is essentially a mesh with very few dimensions, and this causes a complex shape such as a person to create large, unpersonlike shapes in the hull.

Here is the patch:

– Pasted Max Patch, click to expand. –

As for hiding the unused points, I left that out because it would confuse what is happening to the dynamicmesh. I have to go to work now, but I’ll post that tonight.

#229243

lembert.dome

lembert.dome
May 24, 2012 at 2:38am
– Pasted Max Patch, click to expand. –

That’s the basic idea, in any case.

#229244

zthomas

zthomas
May 24, 2012 at 6:10pm

Interesting, works pretty well! I’m guessing the determining of the closest cell with a z-value > 0 must be done outside of gen then? I can’t think of anyway to query the entire matrix like that within gen. Those coords would then replace the x/y center param and use the corresponding z-value? I apologize for not being of more help, gen is still quite new to me.

#229245

lembert.dome

lembert.dome
May 24, 2012 at 8:19pm

I was looking for the same thing about a month ago. Wesley Smith pointed me in the right direction and assured me that no, Gen cannot do that sort of look-up. (http://cycling74.com/forums/topic.php?id=38963)

I did find what I was looking for with xray.jit objects; I believe xray.jit.cellcoord was one of the two objects needed. This does exactly what you describe; finding the closest cell with a z-value>0. It returns a matrix with all cells set to either the left or right edges of a blob, whichever is closer. However, this introduces a bit of latency and eats CPU. That is why, when I first got Gen and Max6, I thought there would be some way of doing this with jit.gl.pix to put it on the GPU. If there is, I couldn’t find it; and that post was a query to the Max world if anyone knows other methods. All in all, the xray.jit library is probably the fastest way of doing this.

Of course, my quick solution of simply setting all cells where z-plane=0 to x=0 and y=0 is not optimal in most cases. You will want (at least) to set those coordinates to the centroid of the blob, and I want to try some sort of matrix-offset-wrap which could make a z-plane symmetrical blob by setting z-plane=0 cells to a submatrix whose offset is equal to the coordinates of the first non-zero cell. That, however, might just be the same as the xray.jit solution.

All of this is just important if the draw-mode is not points. With draw-mode points, one stray point is not much of a problem. It only becomes a problem when all the edges of a blob are drawn to connect to 0.

Please post if you discover anything else in this vein!

#229246

zthomas

zthomas
May 24, 2012 at 11:03pm

I was looking into a javascript solution earlier today but with little luck. I’ll make a more thorough attempt in the next few days and post if I achieve anything. Thanks for pointing toward the xray library, I will look into that as well. Unfortunately the stray points are problematic to the phys.world. The project I’m working on currently uses Synapse to animate a semi-humanoid figure to interact with the physical environment, but a mesh which adapts to the physical characteristics of the user would be much more interesting.

#229247

lembert.dome

lembert.dome
May 25, 2012 at 12:46am

I have not used JavaScript with Max much, nor have I used it for speed-dependent applications, but I think you might find it to present a speed bottleneck. An object in C++ will be faster, hence xray.jit. I’ll dig that patch out if I can find it and post it here.
When thinking about this today, I realized that I am not entirely clear on what the ideal behavior would be. If the Kinect was looking at a person, would I want a convex, tri-grid representation of them with a flat back? No back? An approximation of a rounded back?
As for the jit.gl.world, the dynamicmesh is drawn with tri-grid as the draw-mode. This means that it is not enough to have the unused points invisible, they have to be somehow within the 3D shape. Using the method I posted last but setting the unused cell values to x,y and z centroids, you will essentially get just that, but with some strange artifacts (any edge to the blob will be drawn back to one central point.)

I’ll try to find that patch…

#229248

lembert.dome

lembert.dome
May 26, 2012 at 4:26pm

I found the patch. It was jit.xray.minmax which works the best. The other method is a way to have a variable sized matrix which, I believe, was more costly in terms of CPU.

The patch I am posting here is the same as before, but sets the 0-cells to the value of the nearest non-zero cells. Due to the adaptations I made to fit the Kinect input to the 2DphysWorld patch, the outline matrix and the mesh are drawn opposite to each other and therefore not yet fully useful. I’m going to simplify the whole patch and take out the physWorld components in order to make a concise Kinect-to-GL patch.

In the patch as it is, I have a mesh being drawn which is a representation of the outline in OpenGL.

– Pasted Max Patch, click to expand. –
#229249

You must be logged in to reply to this topic.

Forums Jitter