That old problem: Kinect to surface (mesh or nurbs?) with data filtering

What is your end goal? Technically or artistically, what do you want?
I'm not able to figure your goal from your patches.

diablodale, this is sort of what i'm looking for. an image is attached and the patch is bellow:

i want to generate those surfaces from a face, but at the same time i want to be able to filter the background below a certain depth. the only way i could do it was with xray.jit.sift and jit.gl.mesh, but them the "tri_grid" draw method loses the correlation between the nearest points and the triangles are draw in sequential order (left to right, top to bottom), not a grid.

To try and clarify, are you wanting a tri-grid image of just a face? If so, that's a lot of hand work because you'll need to edit out the heck and chest, and other parts of the body.

Are you wanting a pointcloud of whatever is in the camera's view minus points below/above a depth value? Drawn as points? Or drawn as a tri-grid in your picture?

Must it be a pointcloud? Or can it be a 3d face model of 60 or maybe up to 1000 points?

And you write "below a certain depth". Below would mean the points on the nose would start to disappear first, then eyes, then ears, etc. Do you really mean below -or- do you mean to filter the background for those points who have a distance above a certain distance?

Diablodale, sorry I couldn't explain properly. Explaining complex things in english is not my forte ;) I will try my best.

I'm helping a friend which is also an artist to develop a system to generate portraits. The main concept is to draw it with polygons that can be edited, creating deformations. A realistic representation is not the focus. But first I'm thinking about the capture part of the project and leaving the editing for later.

My first thought was that using facetracking and a 3d face model would be too "general". It would not properly capture the differences between each face, even if the differences captured by the kinect are "low res". Am I wrong in this respect? So I thought that it would be better to generate a pointcloud and draw the triangles (hence the tri_grid) based on those points.

Editing the rest of the body is not an issue. My only concern is to filter the background, removing points below a certain threshold in the z plane (depth, in terms of proximity to the camera, not distance). I could do it fairly easily with xray.jit.sift and I was able to generate the pointcloud with no issues, but now I can't create the tri_grids, as the points lost their x and y position in the matrix.

I thought about doing some sort of analysis of the x and y planes of the coordinates matrix and create an index matrix to feed the jit.gl.mesh, by finding the nearest point in each direction, but I wasn't successful. Also tried getting a similar result with nurbs, as it creates surfaces, but I couldn't filter the z coordinate (I needed to transform the matrix with jit.scanwrap in a single line / row, and jit.gl.nurbs didn't worked).

Is there a way to reconstruct the tri_grid after the points have been filtered in the z coordinate? As the resulting matrix is one dimensional with variable length, I can't figure out a way to find the corresponding indexes of the triangles.

Or maybe facetracking is the way to go? I got interested in this 1.000 points face model you quoted. Would it be able to capture ones facial features?

use geometry shaders

SPA, thank you for your answer. To be honest, this is not something I was familiar with and I got really interested. Since yesterday I'm trying to learn more about it by looking at the few examples in Max as well here in the forum. But as my experience with custom shaders are really limited (just used a few for extremely basic stuff and switched to gen as soon as I could), I'm having a hard time wrapping my head around it and understanding how can I use it to achieve what I want. Could you explain a little more or indicate something some reference about it?

I also saw a 2012 post here in which you say that you achieved exactly what I described. Could you share your shader? It may be a valuable way to learn more about that kind of stuff.

Using a geometry shader might not do all you need. The geometry shader will be applied during the render process. Therefore, you will not get a dataset output which allows you to then edit by hand. Using SPA's approach would be
1) collect pointcloud data from Kinect
2) hand edit point cloud to make the artistic deforms or changes you want
3) feed points into rendering engine (for example, jit.gl.mesh)
4) geometry shader runs to remove triangles further than xxx meters
5) you see the visual

Step (2) could also remove the background; then you don't need the (4) geometry shader. There are some open source code/tools that allow changing/editing of pointcloud data into a mesh. I have not used any of them. Here are some of them:
https://www.google.de/search?q=pointcloud+mesh+editing&oq=pointcloud+mesh+editing&aqs=chrome..69i57.3461j0j7&sourceid=chrome&es_sm=122&ie=UTF-8

Another option is to use my dp.kinect2 object (runs on windows), you can get a high definition face model that has 1347 vertices and 2630 triangles. It is only the face; no neck and no shoulders.

Diablodale, thank you so much for you complete answer.

I thought that what SPA suggested was a little different from what you proposed. Something like this:

1) collect point cloud
2) remove points further then x meters
3) deformations in remaining points
4) rendering engine (jit.gl.mesh) -> as I described before, it should be rendered as points, not as triangles,
[so far I could do all of this fairly easily]
5) use a geometry shader to render the point cloud as triangles, "detecting" nearest points to connect.

Could a geometry shader do this kind of thing? Thought my research, I found that it might be possible (as the old post from SPA I quoted or others like this one: https://cycling74.com/forums/first-steps-to-generate-polygon-mesh-and-line-mesh-from-video-input-2), but I don't not quite exactly how.

I'm really interested in using your object. I have read a lot about it, but I didn't know that it could deliver that many points from a face. Seems to be exactly what I need. The problem is that I'm using a Kinect 1 and, although a Kinect 2 is pretty easy to find, the adaptor for USB is extremely rare here in Brazil. I'm trying to figure out a solution to buy the adaptor and as soon as I get my hands on it, I will get dp.kinect2.

A geometry shader runs in OpenGL and operates on "primitives". Primitives are things like points or triangles.
http://www.informit.com/articles/article.aspx?p=2120983&seqNum=2

You do not have access to the entire pointcloud in a geometry shader. Without access to all of them...you can't search for the nearest one. Perhaps you could use a displacement map and send the data into the OpenGL pipeline as a texture. And then search through the texture for the nearest neighbor. However, this quickly becomes complex.

You only have access to the vertices in the primitive. For example, if you are drawing points, then you only have access to the single vertex. If you are drawing triangles, then you have access to the 3 vertices of the triangle. You can create new/change the primitive type. For example you could receive points and output triangles. However, you still only have access to the single vertex (point input) or the 3 vertices (triangle input).

If you only need the visual and don't need the data after the culling (filtering out the background), then try using a far clip (no need for shader, its built into the max opengl objects as an attribute). @far_clip removes primitives beyond the given distance. If you don't like this effect, then you can write a geometry shader. The link above has some good examples you could extend.