aletheia

I'm wondering if you guys can share with me your techniques for synching your Jitter visuals/videos with a DJ or producer's music in a live setting?

My partner and I are a live audiovisual act. He uses either Ableton or Tractor Scratch pro. For the most part, my source material is original video clips and photographs. I have been learning Jitter for the last few months and still haven't quite come up with a good solution to get my visuals to respond to his music in a live performance. I would really appreciate to hear some of your ideas/techniques/solutions to help me get started.

I know this is a more general question, and that I'm not giving you a lot of info about what I personally want to do, but I'm leaving it open in hopes that I hear a range of thoughts.

I'm wondering if you guys can share with me your techniques for synching your Jitter visuals/videos with a DJ or producer's music in a live setting?  

My partner and I are a live audiovisual act.  He uses either Ableton or Tractor Scratch pro.  For the most part, my source material is original video clips and photographs.  I have been learning Jitter for the last few months and still haven't quite come up with a good solution to get my visuals to respond to his music in a live performance.  I would really appreciate to hear some of your ideas/techniques/solutions to help me get started.  

I know this is a more general question, and that I'm not giving you a lot of info about what I personally want to do, but I'm leaving it open in hopes that I hear a range of thoughts.  

liveon-the-fly-sync-of-audio-with-video

Hmm. Not too sure why, but I don't ever get a response from anyone on this forum whenever I post. Is my question too open? Is it a stupid question??

I'm really looking for different ways that I can sync my visuals up with someone using Ableton or Traktor Scratch Pro. I don't have a specific issue, as I'm at the beginning of the process and am trying to figure out the best way to synchronize the visual with the audio.

Please, can anyone share their ideas/techniques with me?

Hmm.  Not too sure why, but I don't ever get a response from anyone on this forum whenever I post.  Is my question too open?  Is it a stupid question??  

I'm really looking for different ways that I can sync my visuals up with someone using Ableton or Traktor Scratch Pro.  I don't have a specific issue, as I'm at the beginning of the process and am trying to figure out the best way to synchronize the visual with the audio.  

Please, can anyone share their ideas/techniques with me?  

All answers will be greatly appreciated!


yeah, questions that are too general or broad tend to not get answered around here. not sure why that is, but it is.

the more specific you can make your question, the better chance you'll get a useful answer.

for example your topic says sync audio and video, but your body says "respond to his music", which to me are two different things. sync generally involves sending some kind of absolute or relative timing information between two applications or devices.

you seem to be asking for general audio/video synesthesia tips.

searching around the jitter forum should reveal something to get you started.

in addition, andrew benson's jitter recipes are a great place for intermediate jitter users to get more training. 

https://cycling74.com/story/2006/2/6/114353/4118

https://cycling74.com/story/2006/2/14/161736/108

https://cycling74.com/story/2006/2/17/153427/406

for starters, check out recipe 26 videosynth3

other than that, if you can narrow down your question and post a patch showing your first attempts, we can be of further assistance.

e.g. "i want to change the color of my video based on the frequency spectrum of some input audio" or "i want to map the dynamic range of some input audio to displace vertices on an opengl mesh"

yeah, questions that are too general or broad tend to not get answered around here.  not sure why that is, but it is.

you seem to be asking for general audio/video synesthesia tips.
searching around the jitter forum should reveal something to get you started.
in addition, andrew benson's jitter recipes are a great place for intermediate jitter users to get more training. https://cycling74.com/story/2006/2/6/114353/4118https://cycling74.com/story/2006/2/14/161736/108https://cycling74.com/story/2006/2/17/153427/406
for starters, check out recipe 26 videosynth3

other than that, if you can narrow down your question and post a patch showing your first attempts, we can be of further assistance.
e.g. "i want to change the color of my video based on the frequency spectrum of some input audio" or "i want to map the dynamic range of some input audio to displace vertices on an opengl mesh"

hi, i tend to find that about this forum too, but when help comes its very helpful. (probably not my post) but here is goes anyway

this is what i use, its a auto beat detected, its not perfect 

not sure if this is the sort of thing your after, but it might help. those andrew benson patch are amazing

this is what i use, its a auto beat detected, its not perfect 
not sure if this is the sort of thing your after, but it might help. those andrew benson patch are amazing 

nice patch strimbob... you inspired me to play around with that patch a bit, i added in a low pass filter sidechain filter for the detector and a tempo calculator. i changed the adc to a file player so i could load some loops to test it... it works very well, the sidechain really helps with tempo detection when theres a consistent kick drum. i thought you might be interested to see.

hey guys - first, i'm stoked to have some answers! thanks so much!

@ robtherich: cool, thanks for your thoughts, they were helpful. i was indeed just looking for tips, but i will make my questions more specific now. as for the recipes, i thought that i had gone through them before, but am now realizing that i must have just skimmed them. i will get on to reading them right away. thanks for pointing them out to me again.

@ strimbob: this is the kind of thing i'm looking for. i'm a beginner at this, so i was unaware of both adc~ and peakamp. this definitely helps me.

@ the_man361: awesome. this patch looks great, but i'll have to spend some time wrapping my head around how it works. i'm going to spend some time with it, but i may have some questions for you soon if that's cool.

hmm here's a question: do you guys know if there is a way for jitter to analyze incoming audio (via line-in) such that it can separate the different bands of audible frequencies? so, for example, could i separate the frequencies into 4 (or more) different bands (low, low-mid, high-mid, high) so that i can then link each band to a different jitter effect? the last part i know i could figure out, but i'm just not sure if there is a way to get jitter to analyze incoming audio beyond the amplitude.

for my last live set, i made a patch (with help from a friend) that took incoming MIDI notes and sent that data to different effects in jitter (contrast, brightness, saturation, etc.). using this method meant that my partner had to create a separate MIDI track in Ableton that was then sent to me. i would rather not work this way, because i'm then always relying on him to create MIDI tracks for me. i'd rather be able to do everything on my side. does that make sense? any suggestions for a better way to do this? (again, though, i still need to go through the jitter recipes, so perhaps my answer is in there.)

please note that the patch is kinda messy, but it looks nice + pretty in presentation mode

hey guys - first, i'm stoked to have some answers!  thanks so much!  

@ robtherich:  cool, thanks for your thoughts, they were helpful.  i was indeed just looking for tips, but i will make my questions more specific now. as for the recipes, i thought that i had gone through them before, but am now realizing that i must have just skimmed them.  i will get on to reading them right away.  thanks for pointing them out to me again.  

@ strimbob:  this is the kind of thing i'm looking for. i'm a beginner at this, so i was unaware of both adc~ and peakamp. this definitely helps me.  

@ the_man361:  awesome.  this patch looks great, but i'll have to spend some time wrapping my head around how it works.  i'm going to spend some time with it, but i may have some questions for you soon if that's cool.  

hmm here's a question:  do you guys know if there is a way for jitter to analyze incoming audio (via line-in) such that it can separate the different bands of audible frequencies?  so, for example, could i separate the frequencies into 4 (or more) different bands (low, low-mid, high-mid, high) so that i can then link each band to a different jitter effect?  the last part i know i could figure out, but i'm just not sure if there is a way to get jitter to analyze incoming audio beyond the amplitude.  

for my last live set, i made a patch (with help from a friend) that took incoming MIDI notes and sent that data to different effects in jitter (contrast, brightness, saturation, etc.).  using this method meant that my partner had to create a separate MIDI track in Ableton that was then sent to me.  i would rather not work this way, because i'm then always relying on him to create MIDI tracks for me.  i'd rather be able to do everything on my side.  does that make sense?  any suggestions for a better way to do this?  (again, though, i still need to go through the jitter recipes, so perhaps my answer is in there.)

please note that the patch is kinda messy, but it looks nice + pretty in presentation mode    

To get your audio into max look at [adc~]. To split the audio by frequency you should use a filter, have a look at [svf~] and [biquad~]. In the bottom right corner of the helpfiles will be a menu with a list of other filters you could use, there is also a list 

. You might also want to look into using [pfft~] to create a steeper slope if you want absolute sections. There are many objects to analyse the audio, it all depends on what type of music you are using and what you want to do with it. Emmanuel Jourdan's 

 could be useful, as could Tristan Jehans 

. Once you've got your data you might want to use max to do some sort of averaging/filtering, 

 are some examples of good methods. Then you can map your data to the desired controller range, have a look at [map] and [scale], you could even write your own equation with [expr] or use a lookup [table]. I hope this points you in the right direction.

To get your audio into max look at [adc~]. To split the audio by frequency you should use a filter, have a look at [svf~] and [biquad~]. In the bottom right corner of the helpfiles will be a menu with a list of other filters you could use, there is also a list here. You might also want to look into using [pfft~] to create a steeper slope if you want absolute sections. There are many objects to analyse the audio, it all depends on what type of music you are using and what you want to do with it. Emmanuel Jourdan's zsa.descriptors could be useful, as could Tristan Jehans MSP externals. Once you've got your data you might want to use max to do some sort of averaging/filtering, here are some examples of good methods. Then you can map your data to the desired controller range, have a look at [map] and [scale], you could even write your own equation with [expr] or use a lookup [table]. I hope this points you in the right direction.

nicely done the_man361 ! will be putting that in, it works so well. 

will have a tinker with it over the week and post if i come up with anything good, 

i think with tristan Jehans analyzer object teamed with this could be great

as for working with ableton but you could also use soundflower to rout up to 16 audio tracks from live in to you adc

had a quick look at your patch look good, will have a good look later on

nicely done the_man361 ! will be putting that in, it works so well. 
will have a tinker with it over the week and post if i come up with anything good, 
i think with tristan Jehans analyzer object teamed with this could be great  

aletheia
 as for working with ableton but you could also use soundflower to rout up to 16 audio tracks from live in to you adc 

i thought, maybe rather than just using the svf as a low pass filter, having selectable either lowpass, bandpass and high pass would be more useful for soloing out a single sound to key it.

I'm glad to see some productive discussion of this. In my experience, there are seemingly endless ways that you can create connections between audio and video. Deciding how you want to do it is most of the fun. That said, there are definitely some tried and true techniques that you can start with. 

If you are able to hook into the control stream of the audio performer somehow (MIDI, OSC, whatever) that is often easier to work with and latency-compensate than analyzing audio after the fact. Because the control stream is already abstracted, and displays intentionality, you don't have to teach your patch to hear music. The trouble with this is that it can be a little bit overly correlated.

One technique is to use the audio wave itself and draw into a matrix with it, using jit.poke~ or jit.catch~. This can be pretty fun, and with enough signal manipulation can produce very interesting results evoking early video synthesis. There are several Recipes that do this in different ways.

Another technique is to do simple frequency analysis and use that to drive various parameters of your patch. A simple and direct approach would be to use the fffb~ object to seperate out different bands and hang a peakamp~ from each outlet. A more complex approach might involve using objects like analyzer~ or the zsa.descriptors to gather musically relevant information and detect transients and such. If you have access to individual tracks of audio, you will be able to get far more precise information because you won't have to filter out various layers of audio.

Ultimately, what makes any of these techniques effective is the particular way that you choose to map the data to different qualities of your video. I encourage you to think about ways that you can complicate these relationships so that it's something more than "when the sound gets loud, the torus gets big". Also, there is really something to be said for a human finger on a knob, guided by an ear and a brain.

Hello,
I'm glad to see some productive discussion of this.  In my experience, there are seemingly endless ways that you can create connections between audio and video. Deciding how you want to do it is most of the fun. That said, there are definitely some tried and true techniques that you can start with. 
If you are able to hook into the control stream of the audio performer somehow (MIDI, OSC, whatever) that is often easier to work with and latency-compensate than analyzing audio after the fact.  Because the control stream is already abstracted, and displays intentionality, you don't have to teach your patch to hear music.  The trouble with this is that it can be a little bit overly correlated.
One technique is to use the audio wave itself and draw into a matrix with it, using jit.poke~ or jit.catch~.  This can be pretty fun, and with enough signal manipulation can produce very interesting results evoking early video synthesis. There are several Recipes that do this in different ways.
Another technique is to do simple frequency analysis and use that to drive various parameters of your patch.  A simple and direct approach would be to use the fffb~ object to seperate out different bands and hang a peakamp~ from each outlet.  A more complex approach might involve using objects like analyzer~ or the zsa.descriptors to gather musically relevant information and detect transients and such.  If you have access to individual tracks of audio, you will be able to get far more precise information because you won't have to filter out various layers of audio.
Ultimately, what makes any of these techniques effective is the particular way that you choose to map the data to different qualities of your video.  I encourage you to think about ways that you can complicate these relationships so that it's something more than "when the sound gets loud, the torus gets big".  Also, there is really something to be said for a human finger on a knob, guided by an ear and a brain.  

interesting reply andrew, and a very educative read! it'd be great to get some more discussion going on this area, as though im a novice jitter user and intermediate with max/msp, i find the concept of 'live' video generation by means of a musical source very interesting. obviously, it all hangs on how well you manage to isolate or handle your input and map it to the video parameters.

i like the concept of going a bit deeper than directly mapping one parameter to a visual, as in maybe incorporating some feedback loops into the data or having some crosstalk with the separate parameters, having one value effect the other (say, data of patterns derived from low frequency hit points influencing the strength or style of data derived from something quite different, such as the volume level of a different frequency band)

i think the simplicity of having a direct control coming from midi data is obviously easy to handle, though i think this would introduce a need for the patch to be tailored to a specific performer, if using much more than note on/off, velocity and standard CC data. the scope for using a small amount of quite simple but meaningful midi as an aid to data from audio analysis is quite large though, and it could be possible to get rid of the overly correlated nature of the midi data if it were to be modified by data captured from an audio stream.

andrewb@cycling74.com wrote on Wed, 21 October 2009 11:39

Also, there is really something to be said for a human finger on a knob, guided by an ear and a brain.

andrewb@cycling74.com wrote on Wed, 21 October 2009 11:39Also, there is really something to be said for a human finger on a knob, guided by an ear and a brain.

I'm usually one for the human element. Instead of looking mechanical, you can make things seem very natural. I don't use automatic beat matching, and one of the most common comments for my visuals is how I get them matching up so well, or they think I worked with the artist to specifically set things up. That and that they look very "analog".

What I usually do is to have dials set to the beginning and end of each video loop, I usually don't mess with playback speed. I then am constantly changing the loop length, as well as things like contrast and luminance which makes elements more or less prominent, as well as making things less boring, since there isn't as much of an obvious loop going on. When there is big bass, sometimes it can look good to have prominent large portions of the screen changing somewhat with the beat, and when something like a high hat comes in, mixing in another layer of video with smaller, brighter elements.

I could see playing around with having some more subtle elements being controlled by the computer, but more than that stuff tends to get "mechanical". But then again, I'm more a fan of playing for music that's being created in a more "live" way as well, and maybe with some rock influence. Solid 4-4 techno might be a better match for beat-matching your visuals.

I'm usually one for the human element.  Instead of looking mechanical, you can make things seem very natural.  I don't use automatic beat matching, and one of the most common comments for my visuals is how I get them matching up so well, or they think I worked with the artist to specifically set things up.  That and that they look very "analog".

What I usually do is to have dials set to the beginning and end of each video loop, I usually don't mess with playback speed.  I then am constantly changing the loop length, as well as things like contrast and luminance which makes elements more or less prominent, as well as making things less boring, since there isn't as much of an obvious loop going on.  When there is big bass, sometimes it can look good to have prominent large portions of the screen changing somewhat with the beat, and when something like a high hat comes in, mixing in another layer of video with smaller, brighter elements.

I could see playing around with having some more subtle elements being controlled by the computer, but more than that stuff tends to get "mechanical".  But then again, I'm more a fan of playing for music that's being created in a more "live" way as well, and maybe with some rock influence.  Solid 4-4 techno might be a better match for beat-matching your visuals.

I also go for the non-automated approach. I had a gigantic audio analysis patch that used analyzer~ and had a huge decision matrix to alter video based on things like loudness, loudness changes, noisiness, tempo, etc etc and just linked it to my video performance patch, but it quickly grew out of hand and I reached sort of the limits of my programming ability...and the results were eh.

For my current performance practice, instead of concise loops, I load a 7 or 8 minute 640x480 motion jpeg movie, dump out the duration, and assign the controls on my trigger finger to jump to random points in time in that duration...and I have a quick key to randomize which points in time my buttons go to.

It gives me a great sense of chance when I'm sort of picking clips at random in a performance or improvisation with others. I can just let the movie play from whatever point, or jump around to different points rapidly, creating a sort of tension for myself because I don't even know where my program is going to take me next.

I've also done work with some video analysis of my 7-8min clips. I use simple tricks to analyze them beforehand for things like luminance, level of detail and motion, and save all of these values into a coll. Then instead of random points in time, i can order my control buttons from brightest to darkest, or from still to quickly moving clips.

Having that human element is extremely important for a visualist...it shouldn't be your job to be background visuals/mood machine for the musician..audio/video should be working together to create something strong together

whew..this is my ongoing mfa thesis work..so i have a lot to say, haha

I hope it's not as hard as it sounds....


i do a greed with that a human element is important, but i do use automated, not for everything, but i click it in and out when i want, i think i workes great when using opengl stuff

i like having the visual lock on, but i like when the visuals have no relation to the music at all, and i use both, like going from reality to dream scape,

i tend to have master buttons in which can over ride the automated,

giving some control away to the computer helps me focus on composition and colour, and gives me the time to make my visuals look the way i intend them to look.

i do a greed with that a human element is important, but i do use automated, not for everything, but i click it in and out when i want, i think i workes great when using opengl stuff
 i like having the visual lock on, but i like when the visuals  have no relation to the music at all, and i use both, like going from reality to dream scape,

i tend to have master buttons in which can over ride the automated,
giving some control away to the computer helps me focus on composition and colour, and gives me the time to make my visuals look the way i intend them  to look.


Live/"On the Fly" Sync of Audio with Video