[SHARE] Sonographic Sound Processing Diss.
Finally the time has come to give something back to this precious forum…
I wrote a dissertation in October 2011 entitled "STFT Analysis Driven Sonographic Sound Processing in Real-Time using Max/MSP and Jitter". I am sharing here the paper and all the accompanying patches (around 40 patches):
And here you can watch an excerpt from the performance I had in February this year (video and especially sound quality are extremely bad). Almost everything beside 3D version of a spectrogram is covered in the dissertation and the patches.
Table of Contents
1.1 Summary of problem 9
1.2 Aim of project 9
1.3 Project scope 10
1.4 Methods 10
1.5 Chapter overview 11
2.1 Graphical Modifications of Visually Presented Audio Data in the Context of Sound Synthesis 13
2.2 Placing the Analysis Driven Sonographic Sound Processing in Wider Context That Made It Possible 16
2.3 History of Analysis Driven Sonographic Sound Procesing and Overview of Its Most Important Implementations 20
2.4 The Reasons Behind Choosing Max/MSP and Jitter 22
From Fourier Transform to Phase Vocoder and Sonogram 24
3.1 Introduction to the Concepts of Fourier Analysis and Its Offspring 24
3.2 Explaining the Fourier Analysis 27
3.2.1 Fourier Series 27
3.2.2 Fourier Transform 28
3.2.3 The Link Between Complex Numbers, Circular Functions and Exponential Functions 30
3.2.4 Magnitude Detection in Fourier Transform 36
3.2.5 Magnitude Detection in Special Cases 36
3.2.6 Maximum Magnitude Values 39
3.2.7 Fourier Integral of the Incoming Sine Wave 41
3.2.8 Magnitude Detection in Harmonic Signals 43
3.2.9 Phases 45
3.2.10 DFT and FFT 48
3.2.11 STFT 56
3.2.12 Phase Vocoder and Spectrogram in Max/MSP 63
3.2.13 Interacting With Sonogram 70
Max/MSP/Jitter Patch of PV With Spectrogram as a Spectral Data Storage and User Interface 78
4.1 Main Patch – Time Domain 79
4.2 Creating the Spectral Matrix – Inside the object pfft~ – Frequency Domain 82
4.3 Reading the Transformed Spectral Matrix – Inside the Object pfft~ – Frequency Domain 84
4.4 Mapping the Display Window to Spectral Matrix 86
4.5 Default Horizontal Speed of Reading the Spectrogram 88
Implementation and Critical Evaluation of Existing and Unexisting Real-time ADSSP Techniques in Max/MSP 90
5.1 Optimization 91
5.1.1 Moving Area of ADSSP According to Reading Position 91
5.1.2 Horizontal Blur 94
5.1.3 Frame Interpolation 96
5.1.4 Interpolate – Concatenate – Blur (ICB) 99
5.1.5 Poking Holes in Interpolated Spectrum 101
5.2 Global Transformations 103
5.2.1 Blur/Water Effect 103
5.2.2 Spectral Smear 111
5.2.3 Lock to Grid 112
5.2.4 Spectrum Shift, Scale and Rotation with jit.rota 112
5.2.5 Time Scramble 114
5.2.6 Slice Fade 116
5.2.7 Sound Rider 117
5.2.8 Saturation / Limiter 118
5.2.9 Denoiser 119
5.2.10 Compression 120
5.3 Local Transformations 121
5.3.1 Masks 121
18.104.22.168 Filter Masks 122
22.214.171.124 Applying a Mask to a Spectrum 133
126.96.36.199 Rectangular Masks 137
188.8.131.52 Arbitrary Masks 143
5.3.2 Repositions 146
184.108.40.206 Creating Spatial Maps with Multislider Object 146
220.127.116.11 Creating Spatial Maps with Itable Object 148
18.104.22.168.1 Frequency Warp 152
22.214.171.124.2 Spectral Delay 158
126.96.36.199.3 Spectral Rolling Pin 165
5.4 Spectral Interactions 170
5.4.1 Cross-Synthesis Using Masks 170
5.4.2 Centroid Based Spectral Morphing 174
5.4.3 Indirect Spectral Interaction 176
5.4.4 Target-Based Concatenative Synthesis 177
5.5 FFT Data Processing on Graphic Cards 181
Conclusions and Future Work 185
6.1 Future Work 186
And special thanks to J.F.Charles that made this dissertation possible with his priceless article in Computer Music Journal entitled "A tutorial on spectral sound processing using Max/MSP and Jitter".
Thanks so much for sharing this, I just had a peek and it looks fascinating.
> And here you can watch an excerpt from the performance I had in February this year (video and especially sound quality are extremely bad). Almost everything beside 3D version of a spectrogram is covered in the dissertation and the patches.
great! i’ve been doing related spectrum-to-3d stuff lately. will have an in-depth look at your system. tanx!
This is a fabulous resource– may I suggest you publish it through a print-on-demand website– it would be something good to buy and have on the bookshelf… although I imagine there are probably embargo issues with your university…
Thank you for your replies! Print-on-demand release would actually be a good idea, but probably I would add a chapter about 3D spectrograms before I do that. Also a poof reading is needed.
Hopefully I will do that soon and in the meantime, please let me know if you find any mistakes or nonsenses…
Thanks t this looks incredible.
What a great work !!
It seems that the link expired. Could you please post the files again?
Best regards and many thanks in advance,
Thanks a lot for your feedback! Here is the link again:
Thanks for the new link! I have already start to read your dissertation and I am very impressed, congratulations!!!
Would you mind uploading the link once more? It is expired again. I am really interested and would really appreciate it.
I just finished reading your thesis and going thru all the accompanying Max patch files. Well done and thanks! It has greatly aided my postgraduate research.
Whoa, this is phenomenal. Thanks for sharing!
Thanks for sharing. It looks really helpful.
Especially because I’m looking into FFT techniques right now.
Sorry to bring this topic up again, but I moved the paper and the patches to my site, so here is the new link:
Also I wrote an article on that subject, that could be found here:
Great! This is amazing! Thanks for sharing.
Hey T, would you give me a bit of advice on this issue? http://cycling74.com/forums/topic/sonify-matrix-through-pfft-please-clarify/
I’d send this over PM but those are still not activated on the new forum…
Nice work, thanks for sharing :)
Thanks so much. Great work !!!
Its fine now thanks… !!!