Forums > Jitter

eyes reconition

March 5, 2008 | 1:45 pm


March 5, 2008 | 2:09 pm

Have a look at the Machine Perception Toolbox that does an eye
blinking detection:

http://mplab.ucsd.edu/grants/project1/free-software/mptwebsite/API/

I started an external called jit.op.sarko but I did never have the
time to do finish it yet. GRRrrr

best,

O./////


March 6, 2008 | 7:47 pm

maybe you could use an IR (infrared) array and track the reflecting light from the eyes with a Wii-controller

maybe…


March 6, 2008 | 11:03 pm

"cvEyeTracker is eyetracking software aimed at doing simple dark-pupil eye
tracking from a firewire camera that monitors the eye. The software also
supports input from a second camera to provide a frame of reference of where
the eye is looking."

http://thirtysixthspan.com/openEyes/

http://thirtysixthspan.com/openEyes/cvEyeTracker-1.2.5.tar.gz

they modified a unibrain firewire camera with sensor separated from the cam
body which unibrain later commercialised.

http://unibrain.com/Products/VisionImg/Fire_i_BC_remoteCCD.htm

yair reshef
On Thu, Mar 6, 2008 at 9:47 PM, Dieter_Laser wrote:

>
> maybe you could use an IR (infrared) array and track the reflecting light
> from the eyes with a Wii-controller
>
> maybe…
>


March 7, 2008 | 12:50 pm

We have been doing some work here (IAMAS) with blink detection and I’ve devised a number of patches for head tracking, eye detection and blink detection that work fairly well using only standard cv.jit objects. I’ve included one of the better patches below. It works well with dark eyes, but I think blue-eyed people might be its weakness. Unfortunately, being in Japan, I don’t have access to a whole lot of blue eyes for testing!

cv.jit.faces is for face detection and ideally once a face is detected, it is better to use another method for tracking. Here, using the bounding box returned by cv.jit.faces, I make a rough guess as to the eyes’ positions. I then initialize cv.jit.track to features in that area and then track the face that way. The eyes are a feature-rich region of the face and are easy to track solidly.

There is another big advantage to identifying features: the eye region is surrounded by very uniform regions: the forehead, the cheeks and the nose ridge. This means that if I have a list of features roughly centered around where the eyes should be, calculating the orientation of this distribution will give me the orientation of the head.

From this information, I can make a much better guess as to the eyes’ positions.

I further refine this guess by using the new external cv.jit.shift to "hone in" on the darkest area in a submatrix that contains an eye. In most cases, this will accurately find the pupil.

In test conditions, we’ve had eye and blink detection ratios of close to 100%. However, when actual interactive pieces were deployed, performance dropped to around 80%. Odd eyeglasses and haircuts accounted for a small part of the problem. However, the major issue was that extra processing running on the same computer to play video and sound slowed down the frame rate to the point where it was too slow to capture rapid blinks.

Here is the patch, it’s rather big and not all that well documented but it should work. (Built-in iSights on Mac laptops work especially well in good lighting.)

Warning: there’s an important loadbang that needs to fire, so if you just c&p it probably won’t work. Save and re-open.

Jean-Marc

#P user jit.pwindow 53 574 102 68 0 1 0 0 1 0;
#P window setfont "Sans Serif" 9.;
#P window linecount 2;
#P comment 566 321 216 196617 Adjust this thresholding value so that the eyes appear as clearly as possible.;
#P window linecount 4;
#P comment 630 232 100 196617 Reinitialize tracking by pressing ‘r’ or by sending a bang to "reset_face".;
#P window linecount 3;
#P comment 630 188 100 196617 You must press the ‘R’ key to find a face.;
#P window linecount 1;
#P comment 630 169 100 196617 Important:;
#P window linecount 2;
#P comment 143 199 100 196617 Press esc for fullscreen display.;
#P window linecount 3;
#P comment 363 41 100 196617 Press the space bar to start/stop the grabber.;
#P window linecount 1;
#P newex 55 494 38 196617 zl join;
#P newex 83 471 81 196617 vexpr $i2 – $i1;
#P newex 55 445 51 196617 zl slice 2;
#P newex 54 536 70 196617 jit.submatrix;
#P message 54 515 213 196617 offset $1 $2 , dim $3 $4 , jit_matrix rightbin;
#P newex 568 515 38 196617 zl join;
#P newex 596 492 81 196617 vexpr $i2 – $i1;
#P newex 568 466 51 196617 zl slice 2;
#P user jit.pwindow 566 580 102 68 0 1 0 0 1 0;
#P newex 567 557 70 196617 jit.submatrix;
#P message 567 536 207 196617 offset $1 $2 , dim $3 $4 , jit_matrix leftbin;
#N vpatcher 10 59 786 559;
#P window setfont "Sans Serif" 9.;
#P newex 50 307 114 196617 vexpr ($f1 + $f2)*0.5;
#P newex 50 273 114 196617 zl slice 2;
#P newex 384 307 114 196617 vexpr ($f1 + $f2)*0.5;
#P newex 384 273 114 196617 zl slice 2;
#P newex 384 253 84 196617 vexpr $f1 + $f2;
#P message 459 235 67 196617 $3 $4 $3 $4;
#P newex 384 194 85 196617 t l b;
#P newex 459 214 51 196617 v offsets;
#P newex 384 170 85 196617 vexpr $f1 * $f2;
#P message 459 152 67 196617 $1 $2 $1 $2;
#P newex 384 111 85 196617 t l b;
#P newex 459 50 48 196617 loadbang;
#P message 459 71 151 196617 0.02 0.030303 0.02 0.030303;
#P newex 384 89 85 196617 vexpr $f1 * $f2;
#P newex 459 131 51 196617 v eyedim;
#P newex 50 253 84 196617 vexpr $f1 + $f2;
#P message 125 235 67 196617 $1 $2 $1 $2;
#P newex 50 194 85 196617 t l b;
#P newex 125 214 51 196617 v offsets;
#P newex 50 170 85 196617 vexpr $f1 * $f2;
#P message 125 152 67 196617 $1 $2 $1 $2;
#P newex 50 111 85 196617 t l b;
#P newex 125 50 48 196617 loadbang;
#P message 125 71 151 196617 0.02 0.030303 0.02 0.030303;
#P newex 50 89 85 196617 vexpr $f1 * $f2;
#P newex 125 131 51 196617 v eyedim;
#P inlet 50 69 15 0;
#P inlet 384 69 15 0;
#P outlet 50 353 15 0;
#P outlet 384 353 15 0;
#P connect 3 0 5 0;
#P connect 5 0 8 0;
#P connect 8 0 10 0;
#P connect 10 0 12 0;
#P connect 12 0 14 0;
#P connect 14 0 28 0;
#P connect 28 0 29 0;
#P connect 29 0 1 0;
#P connect 13 0 14 1;
#P connect 7 0 6 0;
#P connect 6 0 5 1;
#P connect 8 1 4 0;
#P connect 4 0 9 0;
#P connect 9 0 10 1;
#P connect 12 1 11 0;
#P connect 11 0 13 0;
#P connect 28 1 29 1;
#P connect 2 0 16 0;
#P connect 16 0 19 0;
#P connect 19 0 21 0;
#P connect 21 0 23 0;
#P connect 23 0 25 0;
#P connect 25 0 26 0;
#P connect 26 0 27 0;
#P connect 27 0 0 0;
#P connect 24 0 25 1;
#P connect 18 0 17 0;
#P connect 17 0 16 1;
#P connect 19 1 15 0;
#P connect 15 0 20 0;
#P connect 20 0 21 1;
#P connect 23 1 22 0;
#P connect 22 0 24 0;
#P connect 26 1 27 1;
#P pop;
#P newobj 721 446 98 196617 p find_eye_position;
#P newex 721 467 98 196617 zl join;
#P newex 721 488 63 196617 v two_eyes;
#N vpatcher 10 59 610 459;
#P inlet 347 53 15 0;
#P window setfont "Sans Serif" 9.;
#P newex 205 50 48 196617 loadbang;
#P message 205 73 122 196617 brgb 0 0 0 , frgb 0 255 0;
#P message 50 73 143 196617 framerect $1 $2 $3 $4 , bang;
#P newex 50 94 99 196617 jit.lcd 4 char 50 33;
#P inlet 50 53 15 0;
#P outlet 50 116 15 0;
#P connect 1 0 3 0;
#P fasten 4 0 2 0 210 91 55 91;
#P connect 3 0 2 0;
#P connect 6 0 2 0;
#P connect 2 0 0 0;
#P connect 5 0 4 0;
#P pop;
#P newobj 430 456 73 196617 p draw_pupil;
#P user jit.pwindow 429 478 102 68 0 1 0 0 1 0;
#P user jit.pwindow 285 478 102 68 0 1 0 0 1 0;
#N vpatcher 10 59 610 459;
#P inlet 347 53 15 0;
#P window setfont "Sans Serif" 9.;
#P newex 205 50 48 196617 loadbang;
#P message 205 73 122 196617 brgb 0 0 0 , frgb 0 255 0;
#P message 50 73 143 196617 framerect $1 $2 $3 $4 , bang;
#P newex 50 94 99 196617 jit.lcd 4 char 50 33;
#P inlet 50 53 15 0;
#P outlet 50 116 15 0;
#P connect 1 0 3 0;
#P connect 6 0 2 0;
#P connect 3 0 2 0;
#P fasten 4 0 2 0 210 91 55 91;
#P connect 2 0 0 0;
#P connect 5 0 4 0;
#P pop;
#P newobj 286 456 74 196617 p draw_pupil;
#N vpatcher 10 59 610 459;
#P window setfont "Sans Serif" 9.;
#P newex 236 74 67 196617 r reset_face;
#P message 236 146 89 196617 rect 15 15 33 33;
#P newex 180 212 67 196617 prepend rect;
#P newex 50 214 27 196617 t l l;
#P newex 50 187 149 196617 cv.jit.shift @rect 15 15 33 33;
#P inlet 50 30 15 0;
#P outlet 50 236 15 0;
#P connect 5 0 2 0;
#P connect 4 0 2 0;
#P connect 1 0 2 0;
#P connect 2 0 3 0;
#P connect 3 0 0 0;
#P connect 3 1 4 0;
#P connect 6 0 5 0;
#P pop;
#P newobj 430 408 62 196617 p find_pupil;
#P newex 560 233 67 196617 s reset_face;
#P newex 560 212 44 196617 sel 114;
#P newex 560 190 40 196617 key;
#N vpatcher 10 59 610 459;
#P window setfont "Sans Serif" 9.;
#P newex 236 74 67 196617 r reset_face;
#P message 236 146 89 196617 rect 15 15 33 33;
#P newex 180 212 67 196617 prepend rect;
#P newex 50 214 27 196617 t l l;
#P newex 50 187 149 196617 cv.jit.shift @rect 15 15 33 33;
#P inlet 50 30 15 0;
#P outlet 50 236 15 0;
#P connect 1 0 2 0;
#P connect 4 0 2 0;
#P connect 5 0 2 0;
#P connect 2 0 3 0;
#P connect 3 0 0 0;
#P connect 3 1 4 0;
#P connect 6 0 5 0;
#P pop;
#P newobj 286 408 62 196617 p find_pupil;
#N vpatcher 10 59 610 459;
#P window setfont "Sans Serif" 9.;
#P newex 67 50 51 196617 t l l l;
#N vpatcher 500 44 1261 896;
#P window setfont "Sans Serif" 9.;
#P window linecount 1;
#P newex 180 509 40 196617 t l l b;
#P newex 340 534 56 196617 v centroid;
#P newex 166 96 40 196617 t l b l;
#P newex 219 125 56 196617 v centroid;
#N vpatcher 539 231 1139 631;
#P window setfont "Sans Serif" 9.;
#P window linecount 1;
#P newex 194 190 51 196617 – 3.1415;
#P newex 194 168 90 196617 split 1.57 3.1416;
#P flonum 194 212 58 9 0 0 0 3 0 0 0 221 221 221 222 222 222 0 0 0;
#P window linecount 0;
#P newex 194 234 90 196617 v eye_orientation;
#N vpatcher 10 59 610 459;
#P window setfont "Sans Serif" 9.;
#P newex 50 98 40 196617 change;
#P newex 50 146 44 196617 del 200;
#P message 50 168 14 196617 1;
#P newex 50 122 32 196617 sel 1;
#P newex 87 50 33 196617 > 10.;
#P newex 50 232 67 196617 s reset_face;
#P toggle 104 98 15 0;
#P newex 50 71 47 196617 ||;
#P newex 50 50 33 196617 > 20.;
#P inlet 50 30 15 0;
#P inlet 87 30 15 0;
#P connect 1 0 2 0;
#P connect 2 0 3 0;
#P connect 3 0 10 0;
#P connect 10 0 7 0;
#P connect 7 0 9 0;
#P connect 9 0 8 0;
#P connect 0 0 6 0;
#P connect 6 0 3 1;
#P connect 3 0 4 0;
#P pop;
#P newobj 50 132 74 196617 p check_shape;
#P newex 370 76 54 196617 pack 0. 0.;
#P newex 370 119 56 196617 v centroid;
#P flonum 114 110 35 9 0 0 0 3 0 0 0 221 221 221 222 222 222 0 0 0;
#P flonum 50 110 35 9 0 0 0 3 0 0 0 221 221 221 222 222 222 0 0 0;
#N vpatcher 10 59 610 459;
#P window setfont "Sans Serif" 9.;
#P newex 50 50 114 196617 pack 0. 0. 0. 0. 0. 0. 0.;
#P newex 50 73 87 196617 cv.jit.orientation;
#P inlet 84 30 15 0;
#P inlet 67 30 15 0;
#P inlet 50 30 15 0;
#P outlet 50 95 15 0;
#P connect 1 0 5 0;
#P connect 5 0 4 0;
#P connect 4 0 0 0;
#P connect 2 0 5 1;
#P connect 3 0 5 2;
#P pop;
#P newobj 194 97 67 196617 p orientation;
#P flonum 194 121 61 9 0 0 0 3 0 0 0 221 221 221 222 222 222 0 0 0;
#P flonum 306 85 35 9 0 0 0 3 0 0 0 221 221 221 222 222 222 0 0 0;
#P flonum 278 120 35 9 0 0 0 3 0 0 0 221 221 221 222 222 222 0 0 0;
#N vpatcher 576 162 1331 858;
#P window setfont "Sans Serif" 9.;
#P number 359 442 62 9 0 0 0 3 0 0 0 221 221 221 222 222 222 0 0 0;
#P number 359 461 62 9 0 0 0 3 0 0 0 221 221 221 222 222 222 0 0 0;
#P number 359 481 62 9 0 0 0 3 0 0 0 221 221 221 222 222 222 0 0 0;
#P number 28 459 62 9 0 0 0 3 0 0 0 221 221 221 222 222 222 0 0 0;
#P number 28 430 62 9 0 0 0 3 0 0 0 221 221 221 222 222 222 0 0 0;
#P number 28 408 62 9 0 0 0 3 0 0 0 221 221 221 222 222 222 0 0 0;
#P window linecount 0;
#P newex 226 406 29 196617 * 1.;
#P newex 184 406 29 196617 * 1.;
#P newex 143 406 29 196617 * 1.;
#P newex 44 67 40 196617 t b l b;
#P newex 141 101 41 196617 jit.iter;
#P message 434 116 33 196617 set 0;
#P message 391 116 14 196617 1;
#P outlet 296 547 15 0;
#P outlet 254 547 15 0;
#P outlet 209 547 15 0;
#P outlet 167 548 15 0;
#P outlet 126 548 15 0;
#P inlet 44 45 15 0;
#P window linecount 1;
#P newex 209 514 27 196617 / 1.;
#P newex 167 514 27 196617 / 1.;
#P newex 126 514 27 196617 / 1.;
#P newex 363 350 27 196617 *;
#N counter;
#X flags 0 0;
#P newobj 363 173 66 196617 counter;
#P newex 141 136 232 196617 t l b;
#P newex 296 327 27 196617 / 1.;
#P newex 254 327 27 196617 / 1.;
#P newex 296 268 40 196617 accum;
#P newex 254 268 40 196617 accum;
#P newex 209 268 40 196617 accum;
#P newex 167 268 40 196617 accum;
#P newex 126 268 40 196617 accum;
#P newex 209 430 27 196617 – 0.;
#P newex 167 430 27 196617 – 0.;
#P newex 126 430 27 196617 – 0.;
#P newex 224 223 29 196617 * 1.;
#P newex 182 223 29 196617 * 1.;
#P newex 141 223 29 196617 * 1.;
#P newex 141 164 58 196617 unpack 0 0;
#P connect 7 0 33 0;
#P connect 8 0 34 0;
#P connect 9 0 35 0;
#P connect 20 0 29 0;
#P connect 27 0 7 0;
#P lcolor 15;
#P connect 29 0 7 0;
#P lcolor 7;
#P connect 7 0 4 0;
#P connect 4 0 17 0;
#P connect 17 0 21 0;
#P fasten 29 1 28 0 64 92 146 92;
#P connect 28 0 14 0;
#P connect 14 0 0 0;
#P connect 0 0 1 0;
#P connect 1 0 7 1;
#P connect 10 0 30 0;
#P connect 30 0 4 1;
#P connect 16 0 17 1;
#P connect 0 0 1 1;
#P connect 12 0 30 1;
#P connect 27 0 8 0;
#P lcolor 15;
#P connect 29 0 8 0;
#P lcolor 7;
#P connect 8 0 5 0;
#P connect 5 0 18 0;
#P connect 18 0 22 0;
#P connect 0 1 2 0;
#P connect 2 0 8 1;
#P connect 11 0 31 0;
#P connect 31 0 5 1;
#P connect 16 0 18 1;
#P connect 0 1 2 1;
#P connect 13 0 31 1;
#P connect 27 0 9 0;
#P lcolor 15;
#P connect 29 0 9 0;
#P lcolor 7;
#P connect 9 0 6 0;
#P connect 6 0 19 0;
#P connect 19 0 23 0;
#P connect 0 0 3 0;
#P connect 3 0 9 1;
#P connect 10 0 32 0;
#P connect 32 0 6 1;
#P connect 16 0 19 1;
#P connect 0 1 3 1;
#P connect 13 0 32 1;
#P connect 27 0 10 0;
#P lcolor 15;
#P connect 29 0 10 0;
#P lcolor 7;
#P connect 10 0 12 0;
#P connect 12 0 24 0;
#P connect 0 0 10 1;
#P fasten 15 0 12 1 368 299 276 299;
#P connect 27 0 11 0;
#P lcolor 15;
#P connect 29 0 11 0;
#P lcolor 7;
#P connect 11 0 13 0;
#P connect 13 0 25 0;
#P connect 0 1 11 1;
#P fasten 15 0 13 1 368 311 318 311;
#P connect 30 0 38 0;
#P connect 31 0 37 0;
#P connect 32 0 36 0;
#P connect 14 1 15 0;
#P connect 15 0 16 0;
#P fasten 15 0 16 1 368 270 385 270;
#P fasten 29 2 26 0 79 87 396 87;
#P connect 26 0 15 2;
#P fasten 29 2 27 0 79 87 439 87;
#P pop;
#P newobj 194 50 125 196617 p moments;
#P inlet 194 30 15 0;
#P fasten 1 0 6 0 199 88 55 88;
#P connect 6 0 10 0;
#P fasten 1 1 7 0 227 88 119 88;
#P connect 7 0 10 1;
#P connect 0 0 1 0;
#P connect 1 0 5 0;
#P connect 5 0 4 0;
#P connect 4 0 13 0;
#P connect 13 0 14 0;
#P connect 13 1 12 0;
#P connect 14 0 12 0;
#P connect 12 0 11 0;
#P connect 1 1 5 1;
#P connect 1 2 5 2;
#P connect 1 3 2 0;
#P connect 1 4 3 0;
#P fasten 1 3 9 0 283 71 375 71;
#P connect 9 0 8 0;
#P fasten 1 4 9 1 311 71 419 71;
#P pop;
#P newobj 396 142 137 196617 p calc_moments_orientation;
#P flonum 247 416 35 9 0 0 0 3 0 0 0 221 221 221 222 222 222 0 0 0;
#P newex 180 447 29 196617 * 1.;
#P newex 247 386 27 196617 + 1.;
#P newex 247 364 35 196617 * 0.5;
#P newex 247 342 37 196617 abs 0.;
#P message 180 765 67 196617 $3 $4 $1 $2;
#P newex 180 643 27 196617 t l l;
#P newex 197 714 27 196617 + 1;
#P newex 197 692 54 196617 < ;
#P newex 143 741 47 196617 gate 2 1;
#P newex 197 668 76 196617 unpack 0 0 0 0;
#P message 372 326 55 196617 0 $1 0 $1;
#P newex 372 297 35 196617 * 0.5;
#P newex 180 619 84 196617 vexpr $f1 + $f2;
#P newex 180 586 96 196617 zl join;
#P newex 180 561 83 196617 vexpr $f2 – $f1;
#P newex 266 561 84 196617 vexpr $f1 + $f2;
#P newex 180 488 54 196617 pack 0. 0.;
#P newex 180 468 54 196617 poltocar;
#P newex 180 292 54 196617 t f b;
#P newex 224 315 90 196617 v eye_orientation;
#P flonum 234 272 50 9 0 0 0 3 0 0 0 221 221 221 222 222 222 0 0 0;
#P newex 234 250 29 196617 sqrt;
#P newex 180 228 64 196617 unpack 0. 0.;
#P flonum 180 272 50 9 0 0 0 3 0 0 0 221 221 221 222 222 222 0 0 0;
#P newex 166 203 53 196617 jit.3m;
#P newex 166 181 65 196617 jit.op @op *;
#P newex 180 250 29 196617 sqrt;
#P newex 166 148 63 196617 jit.op @op -;
#P outlet 180 813 15 0;
#P inlet 166 30 15 0;
#P window linecount 0;
#P comment 278 126 100 196617 Eyes center;
#P connect 24 0 22 0;
#P connect 1 0 34 0;
#P connect 34 0 3 0;
#P connect 3 0 5 0;
#P connect 5 0 6 0;
#P connect 6 1 8 0;
#P connect 8 0 4 0;
#P connect 4 0 7 0;
#P connect 7 0 12 0;
#P connect 12 0 30 0;
#P connect 30 0 13 0;
#P connect 13 0 14 0;
#P connect 14 0 36 0;
#P connect 36 0 16 0;
#P connect 16 0 17 0;
#P connect 17 0 18 0;
#P connect 18 0 25 0;
#P connect 25 0 22 1;
#P connect 22 1 26 0;
#P fasten 22 0 2 0 148 796 185 796;
#P connect 26 0 2 0;
#P connect 25 1 21 0;
#P connect 21 0 23 0;
#P connect 23 0 24 0;
#P connect 29 0 30 1;
#P fasten 34 1 33 0 186 121 224 121;
#P connect 33 0 3 1;
#P fasten 3 0 5 1 171 171 226 171;
#P connect 12 1 11 0;
#P connect 11 0 13 1;
#P connect 13 1 14 1;
#P connect 8 1 9 0;
#P connect 9 0 10 0;
#P connect 21 2 23 1;
#P connect 11 0 27 0;
#P connect 27 0 28 0;
#P connect 28 0 29 0;
#P connect 29 0 31 0;
#P fasten 35 0 16 1 345 554 258 554;
#P connect 20 0 18 1;
#P fasten 36 1 15 0 200 556 271 556;
#P connect 15 0 17 1;
#P fasten 36 2 35 0 215 530 345 530;
#P connect 35 0 15 1;
#P fasten 10 0 19 0 239 292 377 292;
#P connect 19 0 20 0;
#P fasten 34 2 32 0 201 117 401 117;
#P pop;
#P newobj 67 175 62 196617 p find_eyes;
#N vpatcher 639 152 998 642;
#P window setfont "Sans Serif" 9.;
#P flonum 151 260 35 9 0 0 0 3 0 0 0 221 221 221 222 222 222 0 0 0;
#P newex 50 47 67 196617 r reset_face;
#N vpatcher 537 328 1137 728;
#P inlet 247 37 15 0;
#P window setfont "Sans Serif" 9.;
#P window linecount 1;
#P newex 129 238 27 196617 + 0.;
#P number 358 258 35 9 0 0 0 3 0 0 0 221 221 221 222 222 222 0 0 0;
#P outlet 51 301 15 0;
#P window linecount 0;
#P newex 51 265 66 196617 pack 0 0 0 0;
#P newex 167 207 27 196617 + 0.;
#P newex 133 207 27 196617 + 0.;
#P newex 92 207 30 196617 !- 0.;
#P newex 51 207 30 196617 !- 0.;
#P newex 184 165 27 196617 + 0.;
#P newex 150 165 27 196617 + 0.;
#P newex 92 132 35 196617 * 0.5;
#P newex 51 132 35 196617 * 0.5;
#P newex 92 166 35 196617 * 0.5;
#P newex 51 166 35 196617 * 0.5;
#P newex 84 107 27 196617 !-;
#P newex 52 106 27 196617 !-;
#P newex 55 50 76 196617 unpack 0 0 0 0;
#P inlet 201 37 15 0;
#P inlet 55 30 15 0;
#P connect 3 0 7 0;
#P connect 7 0 5 0;
#P connect 5 0 11 0;
#P connect 11 0 15 0;
#P connect 15 0 16 0;
#P fasten 2 0 3 0 57 68;
#P connect 0 0 2 0;
#P connect 2 2 3 1;
#P connect 12 0 15 1;
#P connect 9 0 11 1;
#P connect 1 0 5 1;
#P connect 2 1 4 0;
#P connect 13 0 15 2;
#P connect 4 0 8 0;
#P connect 8 0 6 0;
#P connect 6 0 12 0;
#P connect 2 3 4 1;
#P connect 18 0 15 3;
#P connect 10 0 12 1;
#P connect 19 0 6 1;
#P connect 12 0 18 0;
#P connect 5 0 13 0;
#P connect 6 0 18 1;
#P connect 7 0 9 0;
#P connect 9 0 13 1;
#P connect 2 0 9 1;
#P connect 6 0 14 0;
#P connect 8 0 10 0;
#P connect 10 0 14 1;
#P connect 2 1 10 1;
#P connect 6 0 17 0;
#P pop;
#P newobj 50 281 70 196617 p adjust_size;
#P flonum 110 260 35 9 0 0 0 3 0 0 0 221 221 221 222 222 222 0 0 0;
#P message 50 68 14 196617 1;
#P toggle 50 103 15 0;
#P newex 50 226 29 196617 t l 0;
#P newex 50 136 47 196617 gate 1 1;
#N vpatcher 293 221 893 621;
#P outlet 61 192 15 0;
#P window setfont "Sans Serif" 9.;
#P newex 61 167 29 196617 gate;
#P newex 97 140 27 196617 != 0;
#P newex 97 117 27 196617 +;
#P newex 141 94 27 196617 +;
#P newex 97 94 27 196617 +;
#P newex 97 72 76 196617 unpack 0 0 0 0;
#P newex 80 50 27 196617 t l l;
#P inlet 80 30 15 0;
#P fasten 6 0 7 0 102 162 66 162;
#P connect 7 0 8 0;
#P connect 0 0 1 0;
#P connect 1 0 7 1;
#P connect 1 1 2 0;
#P connect 2 0 3 0;
#P connect 3 0 5 0;
#P connect 5 0 6 0;
#P connect 2 1 3 1;
#P connect 4 0 5 1;
#P connect 2 2 4 0;
#P connect 2 3 4 1;
#P pop;
#P newobj 50 201 68 196617 p filter_zero;
#P newex 50 180 41 196617 jit.iter;
#P newex 50 159 62 196617 cv.jit.faces;
#P inlet 87 116 15 0;
#P outlet 50 309 15 0;
#P connect 11 0 8 0;
#P connect 8 0 7 0;
#P fasten 6 1 7 0 74 250 139 250 139 88 55 88;
#P connect 7 0 5 0;
#P connect 5 0 2 0;
#P connect 2 0 3 0;
#P connect 3 0 4 0;
#P connect 4 0 6 0;
#P connect 6 0 10 0;
#P connect 10 0 0 0;
#P connect 9 0 10 1;
#P connect 1 0 5 1;
#P connect 12 0 10 2;
#P pop;
#P newobj 326 85 81 196617 p find_one_face;
#P newex 87 109 249 196617 cv.jit.features2track @threshold 0.05 @distance 0.1;
#P newex 67 137 162 196617 cv.jit.track @npoints 1 @radius 7;
#P inlet 67 30 15 0;
#P outlet 67 197 15 0;
#P connect 1 0 6 0;
#P fasten 3 0 2 0 92 131 72 131;
#P connect 6 0 2 0;
#P connect 2 0 5 0;
#P connect 5 0 0 0;
#P connect 6 1 3 0;
#P fasten 6 2 4 0 112 76 331 76;
#P connect 4 0 3 1;
#P pop;
#P newobj 286 191 60 196617 p find_face;
#P newex 445 159 32 196617 print;
#P window linecount 2;
#P newex 430 350 103 196617 jit.op @op < @val 35 @out_name leftbin;
#P window linecount 1;
#P newex 217 100 32 196617 sel 3;
#P number 523 326 35 9 0 0 0 3 0 0 0 221 221 221 222 222 222 0 0 0;
#P window linecount 2;
#P newex 286 350 103 196617 jit.op @op < @val 35 @out_name rightbin;
#P button 268 100 15 0;
#P window linecount 1;
#N vpatcher 133 67 733 467;
#P window setfont "Sans Serif" 9.;
#P window linecount 1;
#P message 493 169 17 196617 0.;
#P inlet 50 30 15 0;
#P window linecount 0;
#P message 445 189 48 196617 theta $1;
#P newex 248 50 27 196617 t l b;
#P newex 248 213 226 196617 jit.rota @anchor_x 25 @anchor_y 16 @interp 1;
#P newex 50 187 125 196617 cv.jit.resize @size 50 33;
#P newex 248 187 125 196617 cv.jit.resize @size 50 33;
#P newex 248 166 80 196617 jit.matrix right;
#P newex 50 213 233 196617 jit.rota @anchor_x 25 @anchor_y 16 @interp 1;
#P newex 408 164 47 196617 cartopol;
#P newex 408 141 81 196617 vexpr $i1 – $i2;
#P newex 408 111 51 196617 zl slice 2;
#P newex 408 88 63 196617 v two_eyes;
#P inlet 248 30 15 0;
#P outlet 50 235 15 0;
#P outlet 248 235 15 0;
#P connect 14 0 10 0;
#P fasten 13 0 7 0 450 208 55 208;
#P connect 10 0 7 0;
#P connect 7 0 1 0;
#P connect 2 0 12 0;
#P connect 12 0 8 0;
#P connect 8 0 9 0;
#P connect 9 0 11 0;
#P connect 13 0 11 0;
#P connect 11 0 0 0;
#P connect 3 0 4 0;
#P connect 4 0 5 0;
#P connect 5 0 6 0;
#P connect 15 0 13 0;
#P connect 6 1 13 0;
#P fasten 4 1 5 1 454 135 484 135;
#P pop;
#P newobj 286 320 154 196617 p normalize;
#P newex 171 333 43 196617 print w;
#P user jit.pwindow 429 242 102 68 0 1 0 0 1 0;
#P button 217 123 15 0;
#N vpatcher 282 139 1072 680;
#P window setfont "Sans Serif" 9.;
#P window linecount 1;
#P newex 218 400 51 196617 v eyedim;
#P newex 313 360 68 196617 jit.normalize;
#P newex 50 360 68 196617 jit.normalize;
#P newex 50 399 80 196617 jit.matrix right;
#P newex 313 396 74 196617 jit.matrix left;
#P newex 156 400 51 196617 v offsets;
#P newex 156 377 38 196617 zl join;
#P window linecount 0;
#P message 313 315 199 196617 offset $1 $2 , dim $3 $4 , jit_matrix grey;
#P newex 313 292 38 196617 zl join;
#P newex 313 263 81 196617 vexpr $i1 – $i2;
#P newex 313 335 70 196617 jit.submatrix;
#P message 50 313 199 196617 offset $1 $2 , dim $3 $4 , jit_matrix grey;
#P newex 50 290 38 196617 zl join;
#P newex 50 261 81 196617 vexpr $i1 – $i2;
#P newex 121 238 73 196617 vexpr $i1 / 2;
#P newex 121 214 30 196617 pack;
#P newex 141 191 41 196617 * 0.66;
#P newex 50 333 70 196617 jit.submatrix;
#P newex 141 133 47 196617 cartopol;
#P newex 141 107 47 196617 unpack;
#P newex 141 84 81 196617 vexpr $i2 – $i1;
#P newex 141 50 51 196617 zl slice 2;
#P inlet 141 30 15 0;
#P outlet 313 431 15 0;
#P outlet 50 430 15 0;
#P fasten 3 0 11 0 146 74 55 74;
#P connect 11 0 12 0;
#P connect 12 0 13 0;
#P connect 13 0 7 0;
#P connect 7 0 22 0;
#P connect 22 0 21 0;
#P connect 21 0 0 0;
#P fasten 9 0 12 1 126 235 208 235 208 283 83 283;
#P fasten 6 0 9 0 146 169 126 169;
#P connect 9 0 10 0;
#P connect 10 0 11 1;
#P connect 2 0 3 0;
#P connect 3 0 4 0;
#P connect 4 0 5 0;
#P connect 5 0 6 0;
#P connect 6 0 8 0;
#P connect 8 0 9 1;
#P connect 11 0 18 0;
#P connect 18 0 19 0;
#P connect 5 1 6 1;
#P connect 15 0 18 1;
#P fasten 3 1 4 1 187 76 217 76;
#P connect 9 0 24 0;
#P fasten 3 1 15 0 187 75 318 75;
#P connect 15 0 16 0;
#P connect 16 0 17 0;
#P connect 17 0 14 0;
#P connect 14 0 23 0;
#P connect 23 0 20 0;
#P connect 20 0 1 0;
#P fasten 9 0 16 1 126 236 355 236 355 289 346 289;
#P fasten 10 0 15 1 126 258 389 258;
#P pop;
#P newobj 286 221 154 196617 p two_eyes;
#P user jit.pwindow 285 242 102 68 0 1 0 0 1 0;
#P message 159 253 68 196617 fullscreen $1;
#P toggle 142 253 15 0;
#P newex 184 228 35 196617 sel 27;
#P newex 142 228 40 196617 key;
#P newex 123 309 58 196617 jit.window;
#N vpatcher 530 46 1410 570;
#P window setfont "Sans Serif" 9.;
#P window linecount 1;
#P newex 470 190 27 196617 + 5;
#P newex 437 190 27 196617 + 5;
#P newex 404 190 27 196617 – 5;
#P newex 371 190 27 196617 – 5;
#P newex 239 211 246 196617 pack 0 0 0 0 0 0 0 0;
#P newex 338 190 27 196617 + 5;
#P newex 305 190 27 196617 + 5;
#P newex 272 190 27 196617 – 5;
#P newex 239 190 27 196617 – 5;
#P newex 239 162 88 196617 unpack 0. 0. 0. 0.;
#P message 239 231 294 196617 frgb 255 0 0 , frameoval $1 $2 $3 $4 , frameoval $5 $6 $7 $8;
#P newex 239 134 63 196617 v two_eyes;
#P window linecount 0;
#P newex 50 50 56 196617 t b b l l;
#P newex 658 128 64 196617 prepend dim;
#P newex 658 105 53 196617 route dim;
#P newex 658 82 72 196617 jit.matrixinfo;
#P newex 65 399 69 196617 jit.lcd 4 char;
#P inlet 50 30 15 0;
#P outlet 65 421 15 0;
#P connect 1 0 6 0;
#P connect 8 0 2 0;
#P fasten 6 2 2 0 85 74 639 74 639 152 70 152;
#P fasten 6 0 2 0 55 134 70 134;
#P fasten 5 0 2 0 663 153 70 153;
#P connect 2 0 0 0;
#P connect 6 1 7 0;
#P connect 7 0 9 0;
#P connect 9 0 10 0;
#P connect 10 0 14 0;
#P connect 14 0 8 0;
#P connect 9 1 11 0;
#P connect 11 0 14 1;
#P connect 9 0 12 0;
#P connect 12 0 14 2;
#P connect 9 1 13 0;
#P connect 13 0 14 3;
#P connect 9 2 15 0;
#P connect 15 0 14 4;
#P connect 9 3 16 0;
#P connect 16 0 14 5;
#P connect 9 2 17 0;
#P connect 17 0 14 6;
#P connect 9 3 18 0;
#P connect 18 0 14 7;
#P fasten 6 3 3 0 100 73 663 73;
#P connect 3 0 4 0;
#P connect 4 0 5 0;
#P pop;
#P newobj 123 278 63 196617 p draw_line;
#P newex 314 79 38 196617 sel 32;
#P number 314 58 35 9 0 0 0 3 0 0 0 221 221 221 222 222 222 0 0 0;
#P newex 314 38 40 196617 key;
#P number 617 81 35 9 0 0 0 3 0 0 0 221 221 221 222 222 222 0 0 0;
#P message 617 104 47 196617 input $1;
#P message 544 104 68 196617 dim 320 240;
#P message 471 104 68 196617 dim 640 480;
#P message 420 104 46 196617 settings;
#P message 384 104 33 196617 close;
#P message 347 104 30 196617 open;
#P newex 286 159 144 196617 jit.rgb2luma @out_name grey;
#P newex 286 133 169 196617 jit.qt.grab @vmode 2 @unique 1;
#P toggle 286 81 15 0;
#P newex 286 100 46 196617 metro 2;
#P connect 55 0 51 0;
#P connect 51 0 52 0;
#P connect 52 0 62 0;
#P fasten 33 0 53 0 291 435 60 435;
#P connect 53 0 55 0;
#P connect 53 0 54 0;
#P connect 54 0 55 1;
#P fasten 3 0 14 0 291 185 128 185;
#P connect 19 0 15 0;
#P connect 14 0 15 0;
#P fasten 17 0 18 0 189 250 147 250;
#P connect 53 1 54 1;
#P fasten 18 0 19 0 158 268 158 253;
#P connect 15 1 24 0;
#P fasten 16 0 17 0 183 245 183 230;
#P fasten 12 0 29 0 319 76 222 76;
#P connect 29 0 22 0;
#P connect 13 0 1 0;
#P connect 1 0 0 0;
#P connect 8 0 2 0;
#P connect 7 0 2 0;
#P connect 0 0 2 0;
#P connect 4 0 2 0;
#P connect 5 0 2 0;
#P connect 6 0 2 0;
#P connect 9 0 2 0;
#P connect 26 0 2 0;
#P connect 2 0 3 0;
#P connect 22 0 3 0;
#P connect 3 0 32 0;
#P connect 32 0 21 0;
#P connect 21 0 20 0;
#P connect 20 0 25 0;
#P connect 25 0 27 0;
#P connect 27 0 33 0;
#P connect 33 0 38 0;
#P connect 38 0 39 0;
#P connect 11 0 12 0;
#P connect 12 0 13 0;
#P fasten 27 0 38 1 291 399 355 399;
#P fasten 28 0 27 1 528 345 384 345;
#P connect 21 1 23 0;
#P connect 23 0 25 1;
#P connect 25 1 30 0;
#P connect 30 0 37 0;
#P connect 37 0 41 0;
#P connect 41 0 40 0;
#P connect 2 1 31 0;
#P fasten 30 0 41 1 435 399 498 399;
#P connect 28 0 30 1;
#P connect 34 0 35 0;
#P connect 35 0 36 0;
#P connect 50 0 45 0;
#P connect 45 0 46 0;
#P connect 46 0 47 0;
#P fasten 37 0 48 0 435 445 573 445;
#P connect 48 0 50 0;
#P connect 48 0 49 0;
#P connect 49 0 50 1;
#P connect 10 0 9 0;
#P connect 48 1 49 1;
#P fasten 33 0 44 0 291 438 726 438;
#P connect 44 0 43 0;
#P connect 43 0 42 0;
#P fasten 37 0 44 1 435 430 814 430;
#P connect 44 1 43 1;
#P window clipboard copycount 63;


March 10, 2008 | 5:39 pm

JM: this patch is great, but I find it jumps to my eyebrows instead of focusing on the pupil after about 7 seconds or so. Refreshing seems to fix it but maybe there’s a more elegant solution? Thanks for sharing.


March 11, 2008 | 2:32 am

Quote: cap10subtext wrote on Mon, 10 March 2008 11:39
—————————————————-
> JM: this patch is great, but I find it jumps to my eyebrows instead of focusing on the pupil after about 7 seconds or so. Refreshing seems to fix it but maybe there’s a more elegant solution? Thanks for sharing.
—————————————————-

Play with the threshold value, put it as low as possible. When I last tried it the default value (35, I think) was a bit too high. This is likely what causes it to latch on eyebrows instead.

Also, it helps to refresh periodically. I didn’t include this function in this patch because the timing is likely to be application-dependent. You might want to reset every five or ten seconds, or after a blink, or when the person is still…

Jean-Marc


July 18, 2010 | 4:35 pm

hello!

sorry to dig this up..!

I was just wondering if it would be a big deal in modification if I were to detect 2 further pair of eyes :) having 2 people in the picture..

would it suffice to merely double everything from [p find_face] on – respectively adapting [value]‘s "addressees" and bringing them together in [p draw_line]..?

even a "yes" or a "no" at this point would be of great help before I start messing around for nothing (my speciality)..

cheers!

-jonas

– Pasted Max Patch, click to expand. –

July 18, 2010 | 10:29 pm

that patch is so cool!! jean paul you are such a cool guy!…you might be interested in this: http://text20.net/node/14


July 19, 2010 | 7:44 am

this is a great patch, although…

i have to take my glasses off to get it to work, hahaha. i have the worst eye-sight also so i cant actually see what is happening without them. but still cool none-the-less.

lewis edwards
——
smokingbunny.co.uk



pid
July 19, 2010 | 8:09 am

lewis, you must have missed the "@spectacles $1" attribute…. must be your eyesight…


July 23, 2010 | 5:12 pm

sorry for asking before having made any practical attempts, but I’m slightly dizzy from stripping down this patch, after quite some hours.. I’ll continue tomorrow, with or without your help :)

until then, if someone can support me in the following assumption, I’d be quite happy to know if I’m on the right track:

in order to actually visualize the detection of 2 pair of eyes, I feel to have found the key in [p find_eyes] (tucked in [p find face]

am I possibly right..? slicing what [p find_eyes] gives me shouldn’t be the problem. I’ll just have to get it to extract "eye data" equal to the number of faces detected.

increasing cv.jit.track’s npoints ([p find_eyes] upstream object) hasn’t helped much either, so it seems I have to concentrate on this 1 subpatcher alone.

I’ll let you know if I’ve succeeded. tomorrow. :) until then

all the best

-jonas


July 24, 2010 | 6:09 pm

done :) for anybody who’s interested in detecting as many eyes as possible (whereas I haven’t actually messed around with its possible limits, this modification only goes for 2 pairs of eyes), here you go

thanks again, jean-marc, for this incredible starting point

– Pasted Max Patch, click to expand. –

July 25, 2010 | 10:41 am

sorry, slight revision in [p draw_line] (the clear messages quite dispensable..):

– Pasted Max Patch, click to expand. –

April 18, 2011 | 12:34 am

Did someone arrive to get the blinking information?
i tried with jit.3m but …

i really looking forward for blinking detection, if someone got something to share!!


January 16, 2014 | 4:27 pm

Its been awhile, so maybe this is due to a max version change, or a problem with an international keyboard or something, but when I pasted JMP’s code into max, I had to find and replace all the "–" (Ascii extended code 150dec, not [minus]) with "-" minus, and all the "#1", "#2", etc. with "$1", "$2".

After fixing these obvious glitches, I was still having trouble making it work, so I started reverse engineering it and have discovered these fixes…so far:

The sub-patch "two_eyes" would not pass signal through the jit.submatrix object until I added another inlet and fed it the matrix from jit.rgb2luma object. I also had to add the embedded message "@dim 50 33" to the jit.submatrix objects to get them to be the right size.

The sub-patch "draw_pupil" needed some editing to the message box feeding the jit.lcd object. change to, "frgb 0 255 0 255, framerect $1 $2 $3 $4"

The sub-patch "draw_line" : There is something wrong with the final message feeding the jit.lcd object. The message has a packed list with 6 ints being fed to it, but it is only sending "frgb 255 0 0" I haven’t figured out what those 6 values are for. panitarc? ("paintarc $1 $2 $3 $4 $5 $6" superimposes a half circle roughly placed over the pupil)

And finally the jit.pwindows at the very bottom of the patch, which I believe should be a isolation of detected pupil portion of the video stream, won’t display anything unless you feed it a matrix from the somewhere (probably from the left outlets of top two pwindows. You also have to set a dimensions of the sub matrix objects. I can’t tell if these should be the same as the dimensions in the "two_eyes" sub patch or whether these should be dynamically updated from the "eyedim" value object.

AND NOW SOME QUESTIONS:

Has anybody used this for gaze tracking? Do you have a calibration patch with final screen coordinates that you would like to share?

Is this even worth it, or should I use external eye tracking software and bring the data into max?

Attachments:
  1. Eye-Tracker.maxpat

January 21, 2014 | 9:52 am

I found a complete version with everything working on a different thread. I am posting it here too, in case anybody finds this thread and wants it.

Attachments:
  1. Eyetracker2.maxpat

January 22, 2014 | 1:12 am

Awesome patches, but Gaze detection is another matter. You need external hardware if you are interested in that.

The Pupil and The EyeWriter projects are both excellent and reasonably affordable open-source options:

http://pupil-labs.com/pupil/

http://www.instructables.com/id/The-EyeWriter-20/

The Pupil allows you to track gaze on a head-cam, essentially letting you move around while it tracks your gaze, while the EyeWriter requires you to maintain a relatively fixed head position.


Viewing 18 posts - 1 through 18 (of 18 total)