Using jit.freenect, some cv.jit, GLSL, Java, and JavaScript… here’s a Kinect tracker patch/app that I’ve been using lately as a server for controlling other patches/apps. Some pretty cool features in this, IMO, that others might find useful…

best,
Zachary

demo video: http://youtu.be/bx02WIG7ooU

download app and max patches at: http://www.vis.kaust.edu.sa/tools/KVL_Kinect_Tracker/

-documentation is non-existent aside from video, and comments in JSON config file, but there’s probably enough info there for a start.

FEATURES:
1. 3D background subtraction

2. custom culling of Kinect depth matrix for carving out ideal tracking space

3. reports blob location data via OSC to variable number of listening clients, as follows:
— /kinect/blobs label1 pixelX1 pixelY1 x1 y1 z1 label2 pixelX2 pixelY2 x2 y2 z2… etc. for as many blobs as are present (message length will be in multiples of 6 element list)

4. by default blobs are sorted and labeled in ascending order nearest to farthest from camera (labels start at 1)
— pixel x/y coords are blob centroid in pixel (camera frame) space, normalized from -1. to 1., with origin at center of frame
— x/y/z coords are real-space 3D blob centroid coordinates in meters (right-handed coordinate system, with z pointing out from camera, y pointing up)

5. optional depth-sensitive tracking – attempts to consistently associate labels with correct blobs, regardless of overlap, etc.
— like cv.jit.blobs.sort, but considering depth as well

@dtr, I’ve been wanting to implement a kinect-based cv.jit.blobs.sort for a while now. Thinking about some other tweaks that could be added… like right now with tracking enabled, if a blob disappears for one or more frame, it’s label gets freed, and then when that same blob reappears it gets assigned the lowest available label. It probably makes sense to make that behavior flexible, and allow a blob to disappear for a custom-defined number of frames before its label is stripped…

http://youtu.be/4HvooLGSKwY

in v19 of app/patch, here: http://www.vis.kaust.edu.sa/tools/KVL_Kinect_Tracker/

Now you have a buffer parameter that defines how long a blob’s location/label will be remembered by the tracker after it has disappeared from the frame.

Default is 0, meaning no frames of memory (when a blob disappears, it’s forgotten). This is the original behavior from previous version.

Value of -1 sets memory to infinite. When a blob disappears, it’s coord/label remains indefinitely. When a new blob appears within that specified distance threshold, the tracker interprets it as that old blob.

And more practical… a value of 30-300 (1-10 sec., assuming 30fps) will remember a blob’s coord/label for a short amount of time. This may be good if, for example, a person turns sideways and is momentarily too small to register as a blob.

When launching the patch, Max asks for a “project.osc.recv” which it doesn’t find. What is this object ?

Can’t get no information about it.

Thank you

(works perfectly in Max 5, not at all in Max 6 – no errors, but turning the camera on just outputs a single frame – if I toggle the kinect view window, each time it opens it contains the newest frame, but no motion)

@pseudostereo, Thanks! I don’t have Max6 at my work, where this was made, but I do have it at home. I’ll figure out what’s happening there soon. In Max5, you have to do some funky things to make gl objects work without a visible window, and that’s what I’m doing. Maybe if you remove the visible-then-invisible window stuff from the patch, and just make it visible always, the problem goes away? I’ll check when I can.

I made a few minor changes to ensure that the patch works correctly in Max6.

Redownload v19 for the update. I haven’t extensively checked the patch in Max6, just a few minutes of testing. So let me know if any strange behavior pops up.

Actually, I don’t understand why any changes were necessary… If anyone form Cycling is out there listening, maybe they can give insight.

Here are the details:

1. In the Max5 version, I wasn’t erasing the render context before banging it, since I didn’t see the need in this case. But that erase is required in Max6 (haven’t had time to thoroughly figure out why).

2. In the Max5 version, I was setting the gl.render object to @transform_reset 1, and then the gl.videoplane was correspondingly effected. But in Max6, setting @transform_reset to 1 for the gl.render object doesn’t seem to effect the gl.videoplane at all. So I explicitly set the gl.videoplane attribute. This is a surprising one to me, and it means that lots of my old patches that global set @transform_reset may be broken in Max6…

Anyways, it’s all working in 5 and 6 now.

best,
Zachary

Thank you

Caroline

zseldess at gmail

as it is based on jit.freenect.grab, i suppose there’s no problem in using 2 kinects & two of your patches at the same time ?

Here are the things which would be necessary :
- “open 1″ & “open 2″ messages in [jit.freenect.grab]
- distinguishing the send&receive & OSC ports in each patch
- change the ports and make a 2nd kinect_tracker_config.json

do i forget something ?

That sounds like everything, although I won’t know for sure until I get back to my kinects in a week or so. If you use the standalone, all that should be necessary to change is the open message, I think. I’ll rebuild to allow that to be selected in the interface and via the json config in the near future.

Also, just fyi, I’m planning to update this to make it possible to carve out multiple tracking spaces with one camera, but need to think it through some more. I’m also going to be working in the next few months on a version as tracking server for 3D nav in cave environments…

best,
Zachary

Another small update to KVL Kinect Tracker (v20), based on tep’s last post. I’ve added a camera index field in the config file (and in the gui) which lets you select from one of multiple connected Kinect devices.

You can get newest version (and change log) here:

http://www.vis.kaust.edu.sa/tools/KVL_Kinect_Tracker/

For now, this will work out the box when running multiple versions of the standalone app.

If you want to run as two versions of the patch, you’ll need to make some mods:

1. duplicate the patch in two directories
2. make unique send/receives for each patch
3. make unique argument for all z.jsonio objects in each patch
4. modify the JSON config file uniquely for each patch, if needed

best,
Zachary

one question – when i have tracking on and set the frame buffer to -1 the blobs hold their respective labels in the video window but when i look at the unpacked OSC the labels jump.

for example: i have 2 blobs. when the one labeled “2″ moves closer to kinect it stays as “2″ in the window but it is now coming out of the first outlet from OSC-route -> unpack.

any ideas?
thanks, david

Thanks for the feedback. I think that is the expected behavior, although I’ll have to check tomorrow when I get access to the Kinect.

Blobs will always be reordered nearest to farthest from the camera, whether or not “tracking” of the blobs is enabled, as in your settings (seems at least as sensible as left to right or other sorting logic, since it’s in 3D).

But since you have tracking on, and the buffer set to -1 (i.e. infinite), the label should be correct. So if it’s blob two that you’re interested in, regardless of it’s nearness to the camera, I’d recommend passing the OSC list through a [zl iter 6] to chop it into separate 6 elements per blob, then pass that to a [route desiredBlobLabel].

Let me know if that helps, or doesn’t make any sense at all…

best,
Zachary

thanks for the reply. i will give that a try.
sounds like it should sort the problem out.

cheers
david

Thank you for sharing this, it will help me immensely with a project (I’ll share the results soon).

I have a question though as I’m new to the OSC route object – how do one actually unpacks the blob data within the patch? Or do I have to open a new patch? Sorry – maybe it’s an obvious thing, but I cannot figure it out.

Thanks!