Image to Spectrum to Audio
Hello everyone,
I have a question that I was hoping to get some clarity to;
I was hoping to be able to create an audiovisual piece for an exhibition later this year, which I was wanting to build around the representation of sound and the relation between what is seen / what is heard.
The idea would be transform frames from a video into a detailed spectrum, that would then be played as audio. I was hoping to get some interesting sounds and images out of this, and I would love to have some tips on where to begin and where to start looking!
Best, and thanks,
J
Basic Jitter analysis objects are jit.3m, jit.histogram, and jit.fft. For converting a matrix (e.g. loaded with image file data) into an MSP buffer, check out jit.peek~
Not a lot of documentation for jit.peek~ but search the File browser for the 2 example patches: jit.peek~-additivesynth.maxpat and jit.peek~-osctrack.maxpat
For additional image analysis install the cv.jit package , e.g. for focussing only on prominent image features by isolating via cv.jit.features or cv.jit.keypoints.
Search the forums and web on any of those objects and you'll probably dig up some interesting stuff as well.
e.g. a google search for jit.peek~ turned up these interesting links:
https://www.researchgate.net/figure/A-minimal-Max-implementation-for-generating-the-sonification-from-video-input-The_fig4_233751591
https://www.youtube.com/watch?v=gB0rhswQlKE
Also, for easier instances of this idea, make sure you have a look at Evgeny Murzin and the ANS synthesizer (conversion of image to sound via additive synthesis, with the spectrum represented on a logarithmic scale).
Amen to Jean-Françcois Charles suggestion for investigating the ANS synthesizer. It is an remains a touchstone for those of us who have followed in its wake.
Although this page is in Russian, I expect you'll recognize some of the names here (Coil, Edward Artemeiev's work on "Solaris," etc.
https://snowman-john.livejournal.com/23776.html
Thank you all for your inputs and suggestions!
It's really helpful – thank you.
Any further suggestions on artists/composers that has implemented such techniques, other than those mentioned in this thread and its links?
Thank you,
J
Just an ulterior idea here's a patch of mine to transform columns of pixels of an image into samples:
https://youtu.be/t9AwM0Br_HE
Thanks Frederico, I've been watching a lot of your videos (and that one, too) and it is definately food for thought in this project!
Anyone has any ideas how to collapse (or add together) a 400x400 px matrix into a single column matrix (1x400)?
I would like to get the average value of each row from the 400x400 matrix presented as a single pixel in the 1x400 matrix.
Any ideas? :)
Best,
J