We have been doing some work here (IAMAS) with blink detection and I've devised a number of patches for head tracking, eye detection and blink detection that work fairly well using only standard cv.jit objects. I've included one of the better patches below. It works well with dark eyes, but I think blue-eyed people might be its weakness. Unfortunately, being in Japan, I don't have access to a whole lot of blue eyes for testing!
cv.jit.faces is for face detection and ideally once a face is detected, it is better to use another method for tracking. Here, using the bounding box returned by cv.jit.faces, I make a rough guess as to the eyes' positions. I then initialize cv.jit.track to features in that area and then track the face that way. The eyes are a feature-rich region of the face and are easy to track solidly.
There is another big advantage to identifying features: the eye region is surrounded by very uniform regions: the forehead, the cheeks and the nose ridge. This means that if I have a list of features roughly centered around where the eyes should be, calculating the orientation of this distribution will give me the orientation of the head.
From this information, I can make a much better guess as to the eyes' positions.
I further refine this guess by using the new external cv.jit.shift to "hone in" on the darkest area in a submatrix that contains an eye. In most cases, this will accurately find the pupil.
In test conditions, we've had eye and blink detection ratios of close to 100%. However, when actual interactive pieces were deployed, performance dropped to around 80%. Odd eyeglasses and haircuts accounted for a small part of the problem. However, the major issue was that extra processing running on the same computer to play video and sound slowed down the frame rate to the point where it was too slow to capture rapid blinks.
Here is the patch, it's rather big and not all that well documented but it should work. (Built-in iSights on Mac laptops work especially well in good lighting.)
Warning: there's an important loadbang that needs to fire, so if you just c&p it probably won't work. Save and re-open.
JM: this patch is great, but I find it jumps to my eyebrows instead of focusing on the pupil after about 7 seconds or so. Refreshing seems to fix it but maybe there's a more elegant solution? Thanks for sharing.
> JM: this patch is great, but I find it jumps to my eyebrows instead of focusing on the pupil after about 7 seconds or so. Refreshing seems to fix it but maybe there's a more elegant solution? Thanks for sharing.
Play with the threshold value, put it as low as possible. When I last tried it the default value (35, I think) was a bit too high. This is likely what causes it to latch on eyebrows instead.
Also, it helps to refresh periodically. I didn't include this function in this patch because the timing is likely to be application-dependent. You might want to reset every five or ten seconds, or after a blink, or when the person is still...
done :) for anybody who's interested in detecting as many eyes as possible (whereas I haven't actually messed around with its possible limits, this modification only goes for 2 pairs of eyes), here you go
thanks again, jean-marc, for this incredible starting point
Its been awhile, so maybe this is due to a max version change, or a problem with an international keyboard or something, but when I pasted JMP's code into max, I had to find and replace all the "–" (Ascii extended code 150dec, not [minus]) with "-" minus, and all the "#1", "#2", etc. with "$1", "$2".
After fixing these obvious glitches, I was still having trouble making it work, so I started reverse engineering it and have discovered these fixes...so far:
The sub-patch "two_eyes" would not pass signal through the jit.submatrix object until I added another inlet and fed it the matrix from jit.rgb2luma object. I also had to add the embedded message "@dim 50 33" to the jit.submatrix objects to get them to be the right size.
The sub-patch "draw_pupil" needed some editing to the message box feeding the jit.lcd object. change to, "frgb 0 255 0 255, framerect $1 $2 $3 $4"
The sub-patch "draw_line" : There is something wrong with the final message feeding the jit.lcd object. The message has a packed list with 6 ints being fed to it, but it is only sending "frgb 255 0 0" I haven't figured out what those 6 values are for. panitarc? ("paintarc $1 $2 $3 $4 $5 $6" superimposes a half circle roughly placed over the pupil)
And finally the jit.pwindows at the very bottom of the patch, which I believe should be a isolation of detected pupil portion of the video stream, won't display anything unless you feed it a matrix from the somewhere (probably from the left outlets of top two pwindows. You also have to set a dimensions of the sub matrix objects. I can't tell if these should be the same as the dimensions in the "two_eyes" sub patch or whether these should be dynamically updated from the "eyedim" value object.
AND NOW SOME QUESTIONS:
Has anybody used this for gaze tracking? Do you have a calibration patch with final screen coordinates that you would like to share?
Is this even worth it, or should I use external eye tracking software and bring the data into max?