seven point motion tracking

    gecko
    May 10 2006 | 11:54 am
    hi all another tracking problem we are trying to build an instalation where a shodow is projected onto a semi transparent screen, behind wich lies a grid of amplifiers, and behind the, the camera that grabs the video for tracking. basically the user would notice that his or hers shadow would control the volume/pan of the amplifiers behind the screen. Having explored the excellent jit.cv, i decide not to use shape analyses since i need specific coordinates of the users shadow extremities (seven in all, head, shoulders hands and knees) two questions i was hoping you could help follow: 1) is there a way to effect a "foreground subtraction"? because the grid of amplifiers is between the screen with thetrackable shadow, and the camera 2) in order tfor he tracking points to stick where we need them to be, is there any alternative to binarized the image? i ask because currently we get a lot of noise with binarized images... using a mac, btw thanks for the time and patiente
    see you
    gck
    • Jean-Marc Pelletier
      May 11 2006 | 12:26 am
      For "foreground subtraction", let me sure I'm getting this straight. Your setup is something like this:
      projector -> people -> screen -> amplifiers -> camera
      If the position of the camera and the amplifiers doesn't change, you can just make a "cut-out" mask of the amplifiers. You could do this with jit.lcd, but a better way would be to just take a single frame, export it to Photoshop or a similar program and create a binary image where the amplifier pixels are black and the rest are white. Multiply your incoming video data with this mask and your amplifiers are gone. You're still left with holes, though, but that's inevitable.
      For the tracking, if all you're seeing are shadows, the optical-flow-based algorithm of cv.jit.track is probably not going to work very well. It likes well-textured greyscale patterns.
      Is there going to be only one person standing in front of the screen at a time?
    • gecko
      May 11 2006 | 11:10 am
      yep, you got the setup straight
      the point of having the camera behind the screen and amplifiers has to do with the author's intention to have the user (one single person at a time, btw) stand next to the screen, in order for the shadows to become more defined and the tracking->sound engine to start working.
      also he doesnt want the user to be aware of the camera, and therefore, the tracking mechanism of the instalation. we are on a tight budget (what with the 24 amplifiers and all), and furthermore , the instalation is site specific for the phaeno museum in wolfsburg, so presently we are not sure we are able to drill some holes in oerder to hide the camera.
      I think this problem could easily be solved by using two cameras in a setup more or less like this:
      projector/camera -> people -> screen -> amplifiers, with an extra camera set lateraly perpendicular to the screen, so that it would track the user's proximity to the screen. as soon as he was close enough, the seven point track system would begin...
      but, like i said, untill we can be sure we're able to buy and hide an extra camera, i need to study the first setup to exhaustion. I'll try doing the cutout approach you mentioned. what other tracking objects do you recomend for shadows, besides your loving firstborn cv.jit.track? softVNS?
      thank you
      gck
    • Jean-Marc Pelletier
      May 12 2006 | 1:29 am
      You don't need two cameras. I'd just put one right on the projector. That was, you don't have a problem with the amps blocking your view. This setup has been used very succesfully in a number of pieces. For instance, Mesa di Voce: http://tmema.org/messa/
      Hiding the camera shouldn't be too much of a problem. You can easily make a cache for both the camera and projector.
      Using a second camera for proximity sensing is overkill. I'd use a proximity sensor instead, simpler and probably more accurate.
      I haven't used SoftVNS in a long time. Nowadays, I do a lot of stuff straight in C/C++. If you're on Windows, check out EyesWeb if you haven't already. They have modules for segmenting body parts that may be of interest to you.
    • gecko
      May 15 2006 | 2:37 pm
      sorry for the late reply
      unfortunately, EyesWeb or any other windows application is not an option right now.
      I managed to find a proximity sensor and i'm setting it up. I tend to go for the overkill solution too often.
      In the meantime, i thought of usinf the grid tool in Cyclops to determine the boundaries of the amplifier grid in regard to the user's shadow... Duon't really know if it will work..
      Thank you very much for all the help
      gck