Cheng Chien-Wen

I found this performance on youtube, and it is really awesome. I am so curioius about how the video tracking work in the performance.

http://www.youtube.com/watch?v=pS1WALmBqUw

As far as I know, they use infrared camera. But I think that is still hard to differentiate the human body from the projected animation or background stuff. In this performance, they created some animation only around the dancers' body. I was wondering how the video camera "found" the shape of the human body and ignored all the other images projected on the ground. I have tried some cv.jit, but still in vain.

Any ideas about the way to create something like the posted youtube video ?

As far as I know, they use infrared camera. But I think that is still hard to differentiate the human body from the projected animation or background stuff. In this performance, they created some animation only around the dancers' body. I was wondering how the video camera "found" the shape of the human body and ignored all the other images projected on the ground. I have tried some cv.jit, but still in vain. 

Can Jitter do something like this ?
Any ideas about the way to create something like the posted youtube video ?

how-does-the-video-tracking-work-in-this-video

although I didn't see the production myself, I do know the developer of the software tracking system (Frieder Weiss) did not use maxmsp/jitter to create this. My understanding is that he designed the system from scratch so that it could effectively deal with the demands for realtime tracking. Having said that, the software is susceptible to changes lighting conditions and therefore placed certain restrictions during the development of the work. Also check out chunky's work 'Glow'.

this is the dynamic alpha mask from frame differencing technique. I worked on that show and use this same technique. check out the camera data article that popped up here on C74 recently and check the silhouettes patch. that is part of the technique. Freider works in Delphi now which is a RAD for C++. I was told it was because of inherent issues with the Max scheduler. The major difference between what your trying and what they do is that they only track in the infrared spectrum so as not to disrupt the tracking with lighting and projection. you could use the tutorial patch and camera filters and achieve the mask. Frieder's animation algo's take it from there and you will have to write your own shaders that react to the mask coming in. I use one machine for tracking and another for drawing for best performance. Jitter will handle the realtime mask pretty well. Background loading is my biggest complaint with Jitter. The sick sick magic of Mortal Engine is Freider's program that takes the tracked alpha mask and then wraps that into two audio signals that when driving the scanners on the laser create the realtime outline on the dancers body......naaaaaaaaaaaaaassssssssstyyyyyyy!

this is the dynamic alpha mask from frame differencing technique.  I worked on that show and use this same technique.  check out the camera data article that popped up here on C74 recently and check the silhouettes patch.  that is part of the technique.  Freider works in Delphi now which is a RAD for C++.  I was told it was because of inherent issues with the Max scheduler. The major difference between what your trying and what they do is that they only track in the infrared spectrum so as not to disrupt the tracking with lighting and projection.  you could use the tutorial patch and camera filters and achieve the mask.  Frieder's animation algo's take it from there and you will have to write your own shaders that react to the mask coming in.  I use one machine for tracking and another for drawing for best performance.  Jitter will handle the realtime mask pretty well.  Background loading is my biggest complaint with Jitter.  The sick sick magic of Mortal Engine is Freider's program that takes the tracked alpha mask and then wraps that into two audio signals that when driving the scanners on the laser create the realtime outline on the dancers body......naaaaaaaaaaaaaassssssssstyyyyyyy!


Wow that video has to be one of the most amazing things I've seen in terms of interaction and mapping.

Cheng Chien - Wen - if you make any progress post it here, I am also shooting for something very similar.

plwolf thanks a lot for the post - it would be great if you could provide more in depth info about the process.

Wow that video has to be one of the most amazing things I've seen in terms of interaction and mapping. 

plwolf thanks a lot for the post - it would be great  if you could provide more in depth info about the process.


notice there is no other changing lighting. BW or modded "IR" cameras are actually only near-IR. Video projectors and lasers have very little near-IR light, so with the visable lighting filtered out the camera doesn't see the projections. However the lights normally used to light a stage have a lot of near-IR. This is a problem I am currently confronting for a similar work. I am having to compose the lighting changes and presence of dancers so that when I am tracking the camera re-calibrates after each lighting transition.

notice there is no other changing lighting.  BW or modded "IR" cameras are actually only near-IR.  Video projectors and lasers have very little near-IR light, so with the visable lighting filtered out the camera doesn't see the projections.  However the lights normally used to light a stage have a lot of near-IR.  This is a problem I am currently confronting for a similar work.  I am having to compose the lighting changes and presence of dancers so that when I am tracking the camera re-calibrates after each lighting transition.

I don't suppose anyone can offer directions to a MAX 4.6 equivalent to the silhouettes patch & article referenced above? (The latter is MAX 5.)

How does the video tracking work in this video ?