Sonification
Hello,
I want to transmit video signals from a patcher to a hardware synthesizer for live processing. How can I do this?
Thanks for any tips and tricks.
Wolfgang
Depends on which characteristics of the video you want to be sonified. Here are a few ideas:
vertical/horizontal movement tracking
average colors/brightness/saturation in different areas or globally
blob tracking using cv.jit
object detection and tracking using Mediapipe
You can map those extracted data to MIDI CC or notes in many ways, depending on the kind of output you want.
Thank you, I don't want to edit the video any further. I've already applied various effects. Ultimately, I want to extract the video matrix from the Max Patch and the Mac and process it further in an external audio device.
I'm not suggesting to edit the video, I give you ideas about how you can make your video to "speak" with your external audio device. Unless I'm missing something, you cannot straight up send matrices/textures to an audio device, you need a translation layer to convert your video to something your audio device can understand, typically MIDI or audio. The examples I gave are a first step in that direction.
Another example is to use [jit.peek~] to convert matrices to an audio signal.
If you were more specific about your final goal maybe we could provide more on-point examples.