[sharing is touching] Kinect based multi touch surface

I recently made a touch screen led wall for the Code Control Festival in Leicester, I was lucky to receive a Catalyst Award for it - thanks Cycling '74, Leicester Phoenix, Sean, Arts Council etc.

Anyway building the wall took me ages and I had hardly any time left to do the software but I thought I'd share the kinect based touch system which is largely copy and pasted from the forums anyway.

The cameras were mounted directly above the wall looking down. I used 2 cameras in my installation but to keep it simple the attached patch is for one.

I discarded all the depth image except for the few cm adjacent to the touch surface and discarded all depth data not within a min and max threshold level.

I then applied Dieter Vandoren's undistortion shader and used a combinaltion of a virtual camera in gl world and jit.gl.cornerpin to get a virtual image of my touchable surface.

I fed this to cv.jit.touches to get the touch data.

I had a few issues with calibration, using 2 cameras, and getting the kinect in the right plane but I was super impressed with the data coming out of the camera and the accuracy of cv.jit.touches (one caveat being I suspect a memory leak).

I think there is some mileage in the patch for someone anyway and there is definitely room for improvement. I don't really understand the jit.gl stuff. I just managed to get it to work!

Hopefully it is understandable but if not give me a prod and I'll try to explain it/sort it our.