Specific Motion Tracking Situation

Sep 25, 2006 at 9:26am

Specific Motion Tracking Situation

Hello. I have a technical question about motion tracking that I’ll like to ‘share’ in this forum.
I’m working in an interactive installation where i would like visitors to interact with a video projection. The idea in fact, is to make an interactive floor (something like ‘the lighting district’ from time’s up).
The tracking system is a fire-i with an IR filter, and a video projector, both with fish eye, mounted in the ceiling, and ‘pointing’ to the floor.
To simplify the understanding of the interaction, we can imagine that the installation works like this: the video projection creates an image of a grid in the floor, when people enter each rectangle; the motion tracking system detects it, and gives an audiovisual feedback…
My question is: What is the smartest way to conceive and program an ‘auto calibration system’?.
In other words: Who can the fire-i recognize the squares, or interactive zones, that are projected? (Don’t forget, the fire-i is only sensible to IR).
Or, how can the motion tracking zones automatically correspond to the video projection zones?.
The installation is going to be permanent, so i need the system to recalibrate itself (The different motion tracking zones must fit in the image grid).

I thought of different solutions. For example using reflective 3M tape to mark the floor according to a fixed video projection area, so that the IR camera can recognize it and adjust the motion tracking grid.
Or maybe i could use another fire-i without an IR filter, to do the calibration work.
Or …

I hope my question is clear, and that it will contribute the discussions about motion tracking in the forum (I haven’t found any info about solutions to this specific situation).
Thanks!.
Felix

#27781
Sep 25, 2006 at 9:43am

>
>My question is: What is the smartest way to conceive and program an
>’auto calibration system?.
>In other words: Who can the fire-i recognize the squares, or
>interactive zones, that are projected? (Don’t forget, the fire-i is
>only sensible to IR).
>Or, how can the video projection zones automatically, correspond to
>the motion tracking zones?.
>The installation is going to be permanent, so i need the system to
>recalibrate itself (The different motion tracking zones must fit in
>the image grid).
>
>I thought of different solutions. For example using reflective 3M
>tape to mark the floor according to a fixed video projection area,
>so that the IR camera can recognize it and adjust the motion
>tracking grid.
>Or maybe i could use another fire-i without an IR filter, to do the
>calibration work.
>Or …

reading quickly through your questions, it makes me think that it
sounds like a job for softVNS – square zones (defined in vns) are
the b-a ba of the program you can define as many as you want – for a
somehow similar project(s) i often used grids forming 64 zones – each
of them being then used (or not) to control a different parameter (or
sets of parameters of course)

you can also hand-draw the zones (in any shape) in the soft itself
and even have zones overlap

__as for the auto-calibration, i have done such a patch in softVNS –
i could send it to you, (don’t have it right now, there) but I
remember it was not difficult to make – and it worked (stayed a few
days in galleries, where it had to deal by itself with day light,
electric lights, etc etc

so yes, check the soft VNS demo

best

kasper

Kasper T. Toeplitz
noise, composition, bass, computer

http://www.sleazeArt.com

#84506
Sep 25, 2006 at 10:11am

Hello Kasper,
Thank you for your reply!.
I haven’t tried SoftVns for the moment. I was patching with cv.jit.
Good to know that there is a vns object for tracking square zones!.
Yes, I’ll love to see your patch to understand how you auto-calibrate the motion tracking ‘system’. I hope you can send it to me once you find it … ;-)
The auto-calibration function must respond to the situation: What happens if the camera is moved a little bit?, or if anyone changes parameters from the projector (like for example the zoom) ???. The motion tracking grid must automatically re-adjusts to this situations.
What I don’t really get is: How can the IR camera detect the position of the projected grid in the floor and re-adjusts, if she is ‘Blind’ to normal light?.
Thanks,
Felix

#84507
Sep 25, 2006 at 10:29am

Quote: Felix _ OtherSounds wrote on Mon, 25 September 2006 04:11

> The auto-calibration function must respond to the situation: What happens if the camera is moved a little bit?, or if anyone changes parameters from the projector (like for example the zoom) ???. The motion tracking grid must automatically re-adjusts to this situations.

I know this might not be the answer you’re looking for, but: make sure no one touches the camera or the projector. Not everything is best done in software and good image analysis starts with a good physical setup.

What you want to do is much more in SoftVNS’ alley than cv.jit. However, you can do the analysis part using standard Jitter objects only. The method works this way: prepare a “map” image of your zones. This can be done in a drawing program or using jit.lcd. The map is a greyscale image where each zone has different-valued pixels. Make sure to use a lossless format when saving and that there is no anti-aliasing.

Then, identify the foreground pixels using the method of your choice. Take the resulting binary image and multiply it with the map image. That way, all foreground pixels in “zone 1″ are going to have the value 1. Using jit.histogram, you can then easily find how many foreground pixels are in each zone.

#84508
Sep 25, 2006 at 10:48am

>Hello Kasper,
>Thank you for your replay!.
>I haven’t tried SoftVns for the moment. I was patching with cv.jit.
>God to know that there is a vns object for tracking square zones!.
>Yes, I’ll love to see your patch to understand how you
>auto-calibrate the motion tracking ‘system’. I hope you
>can send it to me once you find it…
>The auto-calibration function, must respond to the situation: What
>happens if the camera is moved a little bit?, or if anyone changes
>parameters from the projector (like for example the zoom) ???. The
>motion tracking grid must automatically re-adjusts to this
>situations.
>What I don’t really get is: How can the IR camera detect the
>position of the projected grid in the floor and re-adjusts, if she
>is ‘Blind’ to normal light?.

in what i did the grid is not on the floor/wall, but in the software

for what you say (if somebody chages the setting, moves the camera,
changes the zoom etc, the best would be to hire a guard to prevent it
!!!!

then maybe with what is called “head-tracking” in softvns??? in any
case, in my works, i just made sure no one would touch the camera –
should be easy specially if you hang it on the ciling…

the auto-calibration of course depends of the nature of the tracking
system you choose and the general way of conceiving the patch…

best

kasper

#84509
Sep 25, 2006 at 11:29am

if you just need a 8by8 grid consider doing so with simple floor sensors. this could prove more relaible.

#84510
Sep 25, 2006 at 11:34am

Hello Jean-Marc!
Thank you for your reply.
> I know this might not be the answer you’re looking for, but: make >sure no one touches > the camera or the projector. Not everything is > best done in software and good image
> analysis starts with a good physical setup.

You are totally right, but I wanted to think about the worst scenarios …
So, to adjust the projected video image, to the motion tracking ‘map’, you and Kasper seems to agree that the best way is to bee sure the physical setup will stay perfectly fixed.

> What you want to do is much more in SoftVNS’ alley than cv.jit.

Thanks, if you say so … now I’m sure I need SoftVNS

>However, you can do the analysis part using standard Jitter objects >only. The method works this way: prepare a “map” image of your >zones. This can be done in a drawing program or using jit.lcd. The >map is a greyscale image where each zone has different-valued >pixels.

Do you mean that each zone should have a different grayscale value?

>Make sure to use a lossless format when saving and that there is no >anti-aliasing.
>Then, identify the foreground pixels using the method of your >choice.

This way I’ll make the Background elimination?

> Take the resulting binary image and multiply it with the map >image. That way, all >foreground pixels in “zone 1″ are going to >have the value 1. Using jit.histogram, you >can then easily find how >many foreground pixels are in each zone.

Ok. If I understand, this way I adjust the motion tracking grid to the foreground and they become a ‘value 1 zone’, so that I can detect ‘presence’ in every zone.
Am I right?

Thank you both for your answers, they are of great help.

Of course, I’ll appreciate any other comments on the subject.

F

#84511
Sep 25, 2006 at 11:38am

Hello Yair!.
Do you mean, piezo sensors?
I’have worked with sensitive floors in the past.
They are not super solid for permanent installations…
Thanks.
F

#84512
Sep 25, 2006 at 12:16pm

peizo sensors are an overkill, if what you need is onoff pulses there are
many of-the-shelf industrial contact mats, but you can build one for
yourself that will satisfy your needs.
2 sheets of Plexiglas with a heavy duty aluminum foil glued to the
sides+some kind of foam to keep the circuit open. i recommand a midi input
board, as they are fast and expect up to 128 digi inputs per unit, per 1ms~
, cheap at midibox.de/
if you are short you can use a hacked keyboard . if you plan to make the
installation outdoor this can get tricky, but not more then (for one)
dealing with changing lighting in a cv tracking setup….

2006/9/25, Felix :
>
>
> Do you mean, piezo sensors?
> I’have worked with sensitive floors in the past.
> They are not super solid for permanent installations…
> F
> –
> Felix Luque
> http://www.othersounds.net
> felix@othersounds.net
>

#84513
Sep 25, 2006 at 9:08pm

>
> if you are short you can use a hacked keyboard .

no no no…

it was mentioned before, but hacked computer keyboards only allow few
keys to be pressed simultaneously (try it yourself using Apple
Keyboard viewer). I learned it the hard way.

Also setting up the installation when your guinea pigs are typing
into any selected object might be frustrating – i;m doing this often,
but it is way more satisfying to use hacked game controller, as they
don’t interfere with normal operation of computer.

klif

#84514
Sep 26, 2006 at 3:06am

> Do you mean that each zone should have a different grayscale value?

Yes. Fill in the pixels that belong to zone 1, with a greyscale value of 1, 2 for zone 2, and so on. Using a paint program is good because your zones can be of any shape you want. You can load in a still picture taken by your camera and paint the zones on top of it to make sure the map image matches your setup perfectly.

> This way I?ll make the Background elimination?

You need to distinguish people (foreground) from floor (background). How you do this depends on your particular setup, but in the end pixels that correspond to people should end up with greyscale values of 255, and floor pixels should be at 0.

> Ok. If I understand, this way I adjust the motion tracking grid to the foreground and they become a ?value 1 zone?, so that I can detect ?presence? in every zone.

With a histogram you can count the number of “people pixels” for each zone. The binary foreground image you calculated becomes a mask. Look at the patch fragment below. It’s not functional as-is, there’s some stuff you’ll need to fill in, but it shows you the workflow.

#P window setfont “Sans Serif” 9.;
#P window linecount 2;
#P comment 56 355 100 196617 Presence in each zone.;
#P window linecount 1;
#P comment 56 224 100 196617 Zone labeling;
#P user jit.cellblock 162 394 627 427 3 9 256 1 45 17 0 1 1 0 1 1 1 1 1 0 0 0 255 255 255 0 0 0 0 0 0 191 191 191 0 0 0 215 215 240 1 1 1 0 4 0 0 0;
#P comment 189 329 149 196617 9 regions + 1 (0 doesn’t count);
#P newex 162 352 182 196617 jit.histogram 1 long 256 @autoclear 1;
#P window linecount 2;
#P comment 364 224 100 196617 … or whatever you saved your map as.;
#P window linecount 1;
#P message 257 224 103 196617 importmovie map.gif;
#P button 217 230 15 0;
#P newex 217 256 150 196617 jit.matrix map 1 char 160 120;
#P newex 162 280 65 196617 jit.op @op *;
#P comment 56 109 100 196617 Foreground detection;
#P number 215 157 35 9 0 0 0 3 0 0 0 221 221 221 222 222 222 0 0 0;
#P newex 162 179 63 196617 jit.op @op >;
#P newex 162 138 175 196617 jit.op @op absdiff;
#P newex 327 107 182 196617 jit.matrix background 1 char 160 120;
#P newex 162 108 154 196617 jit.matrix input 1 char 160 120;
#P comment 253 159 100 196617 Threshold;
#P connect 12 0 14 0;
#P connect 7 0 12 0;
#P fasten 10 0 8 0 262 249 222 249;
#P connect 9 0 8 0;
#P connect 8 0 7 1;
#P connect 4 0 7 0;
#P connect 5 0 4 1;
#P connect 3 0 4 0;
#P connect 2 0 3 1;
#P connect 1 0 3 0;
#P window clipboard copycount 17;

#84515
Oct 2, 2006 at 2:55pm

Hello Jean-Marc.
Thank you for your help!.
Sorry for the delay of this reply, but lately i didn’t have any time to work in max/jitter.
I have been doing some ‘quick colage’ with your jit.histogram example, and a background subtraction patch that you post in the forum sometime ago (it woks pretty well with my setup).
In this ‘colage patch’ (see below), I first get the mask (255 for foreground, 0 for background), then i multiply it with the map (that I did as you explain to me: an image divided in 9 square zones, each one with a different greyscale value (values 1 to 9)). the output goes to the jit.histogram object.
Then it gets complicated for me. I don’t understand how can i distinguish, from the output of the jit.histogram, the number of foreground pixels of each of the 9 different zones of the map image (with greyscale values from 1 to 9).

Histogram gives back 256 values, it creates automatically a jit.cellblock of 1 row and 256 columns. The only values that ‘change’ are row 1 – column 1 and row 1 – column 255.
It seems to me that in the jit.cellblock, row 1 – column 1 value is the number of ‘black’ pixels, and that row 1 – column 255 value is the number of ‘white pixels’ or foreground pixels of the entire image.

I’m sorry for my ‘incompetency’, but I’m new to jitter.
Thanks again,
Felix

max v2;
#N vpatcher 39 44 1230 1022;
#P origin -122 86;
#P window setfont “Sans Serif” 9.;
#P window linecount 2;
#P comment 85 421 153 196617 average color components and work with greyscale difference;
#P window linecount 1;
#P newex 1 450 235 196617 jit.rgb2luma @rscale 0.3 @gscale 0.3 @bscale 0.3;
#P message 164 81 45 196617 settings;
#P message 122 81 32 196617 close;
#P comment 167 564 100 196617 mask;
#P comment 145 474 100 196617 threshold;
#P user jit.pwindow 0 549 162 122 0 0 0 0 1 0;
#P flonum 102 474 35 9 0 0 0 3 0 0 0 221 221 221 222 222 222 0 0 0;
#P newex 1 496 111 196617 jit.op @op > @val 0.15;
#P window linecount 2;
#P comment 211 180 162 196617 background averaging coefficient (larger values smoother);
#P flonum 168 184 35 9 0 0 0 3 0 0 0 221 221 221 222 222 222 0 0 0;
#P window linecount 1;
#P message 168 209 133 196617 slide_up $1 , slide_down $1;
#P newex 79 230 171 196617 jit.slide @slide_up 2 @slide_down 2;
#P toggle 79 160 15 0;
#P newex 79 180 27 196617 gate;
#P newex 1 255 88 196617 jit.op @op absdiff;
#P message 79 81 29 196617 open;
#P flonum 35 61 35 9 0.5 0 1 3 0 0 0 221 221 221 222 222 222 0 0 0;
#P toggle 1 61 15 0;
#P newex 1 81 44 196617 metro 2;
#P newex 1 119 98 196617 jit.qt.grab 160 120;
#P user jit.pwindow 0 284 162 122 0 0 0 0 1 0;
#P comment 169 295 100 196617 difference;
#P window linecount 2;
#P comment 110 148 176 196617 to set background , open gate for some period of time and then close gate;
#P window linecount 1;
#P newex 196 760 32 196617 print;
#P window linecount 2;
#P comment 204 811 100 196617 Presence in each zone.;
#P window linecount 1;
#P comment 12 687 100 196617 Zone labeling;
#P user jit.cellblock 1 849 409 908 3 9 256 1 45 17 0 1 1 0 0 0 1 1 1 0 0 0 255 255 255 0 0 0 0 0 0 191 191 191 0 0 0 215 215 240 1 1 1 1 4 0 0 0;
#P comment 28 786 149 196617 9 regions + 1 (0 doesn’t count);
#P newex 1 809 161 196617 jit.histogram 1 long @autoclear 1;
#P window linecount 2;
#P comment 249 706 100 196617 … or whatever you saved your map as.;
#P window linecount 1;
#P message 96 706 128 196617 importmovie greyscale.tif;
#P button 56 712 15 0;
#P newex 56 738 150 196617 jit.matrix map 1 char 160 120;
#P newex 1 762 65 196617 jit.op @op *;
#P connect 16 0 15 0;
#P fasten 15 0 14 0 6 112 6 112;
#P fasten 18 0 14 0 84 107 6 107;
#P fasten 31 0 14 0 127 107 6 107;
#P fasten 32 0 14 0 169 107 6 107;
#P connect 14 0 19 0;
#P connect 19 0 13 0;
#P connect 13 0 33 0;
#P connect 33 0 26 0;
#P connect 26 0 28 0;
#P connect 28 0 0 0;
#P connect 0 0 5 0;
#P connect 5 0 7 0;
#P connect 17 0 15 1;
#P fasten 3 0 1 0 101 731 61 731;
#P connect 2 0 1 0;
#P connect 1 0 0 1;
#P connect 21 0 20 0;
#P connect 20 0 22 0;
#P fasten 23 0 22 0 173 227 84 227;
#P connect 22 0 19 1;
#P fasten 14 0 20 1 6 144 101 144;
#P connect 27 0 26 1;
#P connect 24 0 23 0;
#P connect 1 1 10 0;
#P pop;

#84516
Oct 3, 2006 at 12:24am

> Histogram gives back 256 values, it creates automatically a jit.cellblock of 1 row and 256 columns. The only values that ‘change’ are row 1 – column 1 and row 1 – column 255.
> It seems to me that in the jit.cellblock, row 1 – column 1 value is the number of ‘black’ pixels, and that row 1 – column 255 value is the number of ‘white pixels’ or foreground pixels of the entire image.

I just tried it, it works fine for me. Are you sure your map image is loaded properly? When I try it, I get changing values for columns 0 to 9, as expected.

#84517
Oct 3, 2006 at 8:57am

Hello!.
I think I do everything right. I use exactly the patch and map image I send you. And I use the exact message “importmovie greyscale160_120.tif” to load the file. The file is in the same folder as the patch file, and when I load it, the print object send me back:
“print: importmovie greyscale160_120.tif 1”. This looks ok to me …

Are you saying that you get different results using the exactly same patch and map image that I send you?. You have columns 0-9 changing in a “jit.cellblock” of 256 columns?.

I only get values changing in column 0 and in column 255. When I move in the room, the mask changes perfectly according to the motion tracking. But values only change in these two columns. I think column 0 is the total number of black pixels and column 255 is the total number of White pixels, because when I reduce the threshold, this two values change proportionally.

I’m using Max/Msp 4.5.5 and jitter 1.5.0 in a dual G5 with OSX 10.4.4.

Thanks for your help and time …
Felix

#84518
Oct 3, 2006 at 9:23am

This is weird, I tried again and at first it didn’t work, as you described. I sent another bang, and now it’s working. So, try to see if banging the map matrix again doesn anything.

#84519
Oct 3, 2006 at 9:34am

No, it doesn’t …
Weird!.
I’m going to try the patch in a Windows XP computer right now, and i’ll tell you if it works.
thanks.
Felix

#84520
Oct 3, 2006 at 10:51am

Weird, it must be a bug…
I tried it in a Windows XP computer with max 4.5.5 and jitter 1.5.1 and it works.
But I only get changing values in 8 columns (8 +1). Do you get 9 (9+1)?.
For the OSX bug, do you think that it could be solve with a max/jitter upgrade?
I think this patch could work pretty well, how could I implement the analysis?.
Could I take advantage of any softVNS object??.
(Kasper talk about “square zones (defined in vns)” and “head-tracking”).
Thanks,
Felix

#84521
Oct 3, 2006 at 2:56pm

Finally.
I change the map image, giving RGB values from 2-10 to the 9 different square zones. Now everything looks ok. I get changing values in 9 columns (9 +1).
….
I think this patch could work pretty well, how could I implement the analysis?.
Could I take advantage of any softVNS object??.
(Kasper talk about “square zones (defined in vns)” and “head-tracking”).

Thanks,
Felix

#84522

You must be logged in to reply to this topic.