Tech stack for kinect finger motion tracking to make an "air" synth in Live?
I'd like to make a setup similiar to using a kaosillator pad except instead of pressing on the kaossilator xy pad with your finger to play a synth, you move your finger in the air and a kinect motion tracker is used to make midi events for a synth in live.
What would be the easiest software stack to make this happen?
1.) Motion tracker hardware - Kinect
2.) Convert depth data to finger point data - FingerTracker Processing library http://makematics.com/code/FingerTracker/
3.) Read or send Processing data into max4live - How to do this?
Is there an easier/better way to do this, maybe some way of getting finger data without needing Processing at all? How do you get a stream of data from Processing into max4live? Maybe with a local port connection or something?
Well, if you have not yet bought a Kinect, you may be interested in buying a Leap Motion controller. There's a couple of Max externals (like o.io.leap ) that will give you finger data directly in Max.
Instead, if you want to go the Kinect way, you'll want to use OSC to communicate between Processing and Max (check the oscP5 library for Processing http://www.sojamo.de/code/ and the odot externals for Max https://github.com/CNMAT/CNMAT-odot/releases)
yeah, the Leap is ideal for this. Fun too, and there's none of the whole "stand like a cactus" thing going on.
Leap motion with the o.io.leap external and you'll be rocking in 5 minutes flat, the kinect just couldn't come close to the accuracy of the leap.
Thanks for the replies, has anyone actually used a leap to do this? From what I can tell it seems to have a lot of latency and got generally poor reviews on amazon. For instance, most of the videos of it in action have a lot of delay between finger motion and cursor movement, such as in this gif:
The only instrument I could find that someone had made which uses depth tracking without the use of infrared leds is this virtual harp which uses a fairly expensive intel camera:
This worries me because it looks like they deliberately chose an instrument with a long decay and put it in an echoy, reverby room in order to mask the latency.
Am I not looking in the right place? Is there a proof of concept that uses stereo cameras to generate low latency midi data?
I'm thinking what might be more playable will be to use a wii remote and infrared pen with a pressure sensitive tip. The tip should probably be able to make very low latency on/off information and if there's a delay on the location of infrared pen it should not matter quite so much since the pitches will be usually be quantized to notes.
Something like this:
Am I wrong about the latency for stereoscopic cameras? Or did I overlook an instrument someone made which is fast enough to be used in a real setting? I know the leap has a very low published latency value but I don't see anyone getting those latency numbers in practice?
One of my friends used a leap for this type of work and it was very responsive (no perceivable latency issue).