jit.fft as a sound filter

Hello,

As part of my final project for university, I'm developing a musical synth in which the input is only dynamic image (video). You can load videos, process them, and it triggers sound events as well as it functions as an oscillator.

To transform the video into a VCO, I'm using jit.peek and having it run through all coordinates of a frame (320x240 at 30fps means 76,800 samples per 33.3... ms. Because this drastically fails the nyquist frequency at any sample rate below 4,608,000Hz, I'll have to lower the quality of the video drastically to a point where I lose a lot of details, unfortunately). Since jit.peek functions for each plane individually, it creates a sort of triple oscillator, which each plane functioning independently.

I can also use motion detection as sort of a "VCA", with the amount of difference between two frames dictating the volume.

Now I'm looking for a way to implement a filter, and, for that, I would like to use the fft of an image (it is bidimensional so it would be cool to choose which dimension I want to serve as a filter or maybe the combination of both, or maybe one dimensions for one plane, the other for another plane, and the combination for the third plane. idk, i just wanna make sure this is possible) and have it shape the spectrum of the signal that the video itself creates.

This way, the video would be a VCO, VCA and VCF on its own.

The documentation for jit.fft only shows me that it does the transform and then does the inverse. I'm also struggling to find useful exercises with this object and ways to join both the video fft with an audio fft.

If anyone out there can help with this, I would be very grateful!
Cheers, James