for those of you who dont know what it is, jsut have a look at the link above.
I have a question that might be a bit over my head, but I am trying to create a similar patch in Max for a school project. im going to leave out the video part of it, but how would I go about ‘searching the database’ for similar sounds?
im think I have an idea of how to do the rest, but scanning the buffer in real time for a similar waveform/spectrum is the place where I am stumped.

any suggestions?

(sorry to thread jack)

. . . . . . . . . . . .
Http://www.EstateSound.com
Http://ideasforstuff.blogspot.com
. . . . . . . . . . . .

So the steps would be…
1. Process source soundfile and save FFT data using sfrecord~.
2. Load source FFT data in buffer~. Use index~ to access it.
Calculate FFT distributions and build data structure for fast
access of FFT data. Store this data structure in a coll. You
would also save the index of index~. These values could then
be ordered for faster searching.
4. Analyze incoming sound, calculate FFT distribution, lookup
closest value in Col.

I am trying to keep this as simple as possible, no jitter, no DB.
This method would consume a lot of RAM though. Step 2 is the step
that needs to be figured out. When I say distribution, I mean some
equation that takes a set of values and then calculates some
identifying value. I am not a mathematician, I was wondering if
someone out there could offer some suggestions. Any ideas?

Anthony

first of all, he makes it sound like he’s matching the full samples
live. this is impossible because of the age old “you dont know the
future rule” in other words, he cant say CHAAH and expect to get a
CHAAAH live because at the instant it selects the slice, all its
heard is CH.

also, I’ve thrown this structure our in an earlier thread but it
seems applicable. forgive the repetition. you’ve rekindled my
interest in it.

STORE
A: analyze each slice with as much as much metadata as possible…
brightness~,fiddle~, loudness~, try and find some formants, max
pitch, min pitch, etc…
B: save an xml file for each transient (or in this case song with
multiple transient descriptions)
C: upon load, go find all the xml files and load them into a database.

now rather than 512 fft bands which is unorganized data, you have 30
pieces of useful information.

so to search you might analyze the input and find the closest match.
ie average difference of values is closest to zero.

this is just what I’m working on, I don’t know if its how he’s doing it.
-matt

Anthony

sorry. in creativity world I’m a lonely guy, I get excited when
people even approach being in phase with me.
-matt

Anthony

it could just rigged like a trade show computer to perform well under
x circumstances.

I draw no conclusions.I’m leaning toward fft for him, but its still
metadata for me.
you guys rock.
-matt

Anthony

Number of Zero Crossings – General noisiness, existence of transients
Root Mean Square – (RMS) Mean acoustic energy (loudness)
Spectral Centroid – Mean frequency of total spectral energy
Spectra Dropoff – Frequency below which 85% of energy exists
Harmonicity Deviation – from a harmonic spectra
Pitch – Estimate of fundamental frequency

I doubt I could do all these in real time on my laptop. But I
might be able to get good results if I choose one. Which one
would be the best one to use? Does any one know of an external
that will do any of these? If not could someone please point
me in the direction of what algorithm to use.

Anthony

Peter McCulloch

So the storage would need to be sparse in k * j space, I would
recommend a list to start. Maybe make a buffer for the imported sound
file and address a chunk by the beginning and ending sample. The
lookup will be faster if you restrict the database size, you can
optimize later. So imagine you’ve got 32 chunks, for each incoming
chunk (that you’re mapping to a sample in the database), you just
iterate through the list and find the chunk that is of least scalar
distance.

To make this simple: imagine that you only have fft bins per fft
window, and 3 windows per chunk, and 8 chunks per bar and one bar of
input sample. Each sample would have 3 * 3 = 9 coordinates per
chunk. So each saved chunk would be an entry in a list that looks
like [1.1 1.2 1.3 2.1 2.2 2.3 3.1 3.2 3.3 begin_sample end_sample].
Then you need to write an abstraction that takes a 9 element list (the
incoming chunk’s coord’s) and produces a 2 element list (the outgoing
chunk’s position in the saved buffer).

This entire process would of course be optimized later, but you could
probably get away with only using a small number of fft bins.

Does this make any sense? Or am I being stupid?

_Mark

Anthony

You could probably deal with the bark~ latency by compensating with
delay~, and use the scheduler in audio interrupt (and using fftout~ to
convert your buffers back into real-world audio), though. How are you
doing the correlation? (looking at bin delta?) (and is this in
real-time, or NRT?)

Peter McCulloch

I think he is doing something a little more complicated
when it comes to comparing the spectra, I don�t think scalar
distance is a meaningful comparison. For now I am trying to
keep things simple. So I have k bins per window and j windows
per chunk. Lets say K is a window size of 1024, j would vary depending
on
the size of the original file. I am not using a DB so this could
potentially
use a lot of RAM. I would run my source sound file through fft~ and
save the FFT data to disk. I then load this FFT file into buffer~, and
use index~ to get to a specific window FFT data. Lets just say for now
I am going
to do a linear search (later on I know I will have to optimize this).
What I am missing is the piece that takes as input an FFT window and
spits out some meaningful value that I can use to compare. Like wise
I would do this same process on the FFT window of the incoming signal.
I then compare values and keep repeating this process until I find the
closest match. I then insert the matching FFT window into the output
audio stream. Perhaps there is some easy way to compare things that
I am just not thinking of. But it sounds like I need something more
specific.
Is there any source out there that I could perhaps use to calculate
Pitch,
Loudness, Brightness given FFT input?

Anthony

The SDIF file sounds fascinating, does Max support SDIF files?
I was looking to do this in real-time. So that is why I am trying
to use existing Max objects like sfrecord~, fft~, buffer~, etc…
I also need to easily do an ifft~ to produce the output audio
stream. So using the native Max representation of fft data is
much easier.

Trying to sync up bark~ with delay~ sounds complicated. If I
use audio interrupt scheduling, will I be assured
that each output from bark~ would correspond to a fft frame?
It would be nice to not have to worry about that. Again I am
trying to keep things simple. But If I could not find
anything else that worked, I could try it.

Anthony

Number of Zero Crossings – General noisiness, existence of transients
Root Mean Square – (RMS) Mean acoustic energy (loudness)
Spectral Centroid – Mean frequency of total spectral energy
Spectra Dropoff – Frequency below which 85% of energy exists
Harmonicity Deviation – from a harmonic spectra
Pitch – Estimate of fundamental frequency

you will have a vector of length 6. With a bunch of vectors, you are
basically populating a 6 dimensional space with data. If you have a
ton of data, this is going to be a lengthy search as that space is
massive compared to how much data you have. There are a few ways to
do fast searching if you consider your data to be distributed in
specific ways. For instance, if you assume your data to be gaussian
dsitributed, you could use principle component analysis to compare
things in a smaller dimensional space. Or, you could use something
like an approximate nearest neighbor search:
http://www.cs.umd.edu/~mount/ANN/ and find your seult very quickly.
That would be a good way to query a database imho.

best,
wes

Anthony

Peter McCulloch