Image to Spectrum to Audio

Joni950's icon

Hello everyone,
I have a question that I was hoping to get some clarity to;
I was hoping to be able to create an audiovisual piece for an exhibition later this year, which I was wanting to build around the representation of sound and the relation between what is seen / what is heard.

The idea would be transform frames from a video into a detailed spectrum, that would then be played as audio. I was hoping to get some interesting sounds and images out of this, and I would love to have some tips on where to begin and where to start looking!

Best, and thanks,
J

Rob Ramirez's icon

Basic Jitter analysis objects are jit.3m, jit.histogram, and jit.fft. For converting a matrix (e.g. loaded with image file data) into an MSP buffer, check out jit.peek~

Not a lot of documentation for jit.peek~ but search the File browser for the 2 example patches: jit.peek~-additivesynth.maxpat and jit.peek~-osctrack.maxpat

For additional image analysis install the cv.jit package , e.g. for focussing only on prominent image features by isolating via cv.jit.features or cv.jit.keypoints.

Search the forums and web on any of those objects and you'll probably dig up some interesting stuff as well.

Jean-Francois Charles's icon

Also, for easier instances of this idea, make sure you have a look at Evgeny Murzin and the ANS synthesizer (conversion of image to sound via additive synthesis, with the spectrum represented on a logarithmic scale).

Gregory Taylor's icon

Amen to Jean-Françcois Charles suggestion for investigating the ANS synthesizer. It is an remains a touchstone for those of us who have followed in its wake.

Although this page is in Russian, I expect you'll recognize some of the names here (Coil, Edward Artemeiev's work on "Solaris," etc.

https://snowman-john.livejournal.com/23776.html

Joni950's icon

Thank you all for your inputs and suggestions!
It's really helpful – thank you.

Any further suggestions on artists/composers that has implemented such techniques, other than those mentioned in this thread and its links?

Thank you,
J

Federico-AmazingMaxStuff's icon

Just an ulterior idea here's a patch of mine to transform columns of pixels of an image into samples:
https://youtu.be/t9AwM0Br_HE

Joni950's icon

Thanks Frederico, I've been watching a lot of your videos (and that one, too) and it is definately food for thought in this project!

Joni950's icon

Anyone has any ideas how to collapse (or add together) a 400x400 px matrix into a single column matrix (1x400)?

I would like to get the average value of each row from the 400x400 matrix presented as a single pixel in the 1x400 matrix.

Any ideas? :)

Best,
J