Live video capture looping
Hi!
I want to loop a live video capture that is in sync with ableton’s looper. Could you please help me understand how to do this or if this is possible now?
In Live, i can use the looper device to capture and loop an audio source. I want to loop the video along with the audio, so that i have an in-sync video and audio loop. Is there any way that i can use Max for Live or resolume or some other software with the Ableton looper to sync up with the looper and loop video along with the audio?
For example, I hit a midi button that starts both the video and audio loop. I play a drum riff for 5 seconds, during which Ableton captures live audio and Ableton, max or Resolume captures live video. When i hit the midi button again, the audio will replay in Ableton on loop. The video will replay in the Resolume clip as a loop.
I’m thinking that it would create a new clip in resolume for each loop, and that i could loop multiple layers of audio loops and have multiple clips of video loops for each audio loop in resolume. But i’m open to whatever solution that can make this work and figure its probably easier to focus on just one loop first than multiple layers of loops.
Does that make sense? Is that possible to do ?
what have you tried? jit.matrixset is a good object for making capturing loops from video (or the equivalent jit.gl.textureset.js for gl textures)
Thanks Rob, i haven't tried anything yet. Just starting to figure it out now.
I just looked into jit.matrixset and am confused, I'm new to Max and don't have much programming experience.
Do you think that jit.matrixset can achieve everything that i'm looking to do? If so, how easy/difficult would it be for a beginner to implement?
Or are there any other solutions that i should look into?
you're going to have to learn how to program Max to do this. I'd recommend starting with the first 10 or so Max tutorials, the first 10 or so Jitter tutorials, and follow that up with the Video and Graphics tutorials (all accessible via the Reference Window).
there's lots of 3rd party resources to check out, including lots of video tutorials. I put together a list of some of these here: https://docs.google.com/document/d/1pIty79tq4_-kfUpO6i_VDM01hVBZKL4zbqcEcJZnUqQ
here's a basic example of using matrixset / textureset as a buffer of frames that can be used as a video looper:
update to this texture-looper patch using jit.gl.textureset (available with Max 8.2 and up). also ensure jit.world is outputing at 30 fps by setting sync 0.
Rob, you are a genius... thank you, works perfectly!!!
Now the next step is for the video loop to sync with the looping audio signal associated with the video (live television in my case). I already have a patch that takes the live audio signal, records it to a circular buffer, which is synced to the transport, loops it with groove~, which is also synced to the transport. Where in the jitter patch would I bring this all together?
The idea is to have a talking head on cable news, and then all of a sudden the pundit seamlessly starts looping and glitching uncontrollably, sort of like that crazy scene with Steve Carrell in Anchorman.
Randall
One more question: I have one computer setup with 8.2 beta, but my production computer has not been upgraded because I need the stability. Is it possible to use the jit.gl.textureset external with v 8.1.11?
Thanks!
it's not a simple answer, you will have to map from milliseconds of your groove~ loops to frames in the texturebuffer, based on whatever fps you are using. if you post an attempt I'd be happy to further assist getting the wrinkles out.
it's possible you could use the jit.gl.textureset in 8.1.11, but there's no support if anything goes wrong. you can try simply copying the Jitter Tools package from your 8.2 installation (inside the app bundle if on Mac, Resources/c74/packages/Jitter Tools) to your user Packages folder, and modify the package-info.json to change the max_version_min to 8.1 or whatever. might work.
Rob, here is my updated & vastly simplified & hopelessly clumsy audio-video looper patch:
It's still in prototype mode, so I reduced the interaction to one toggle, just above the record~ object, which starts / restarts the whole looping scheme. The audio recording moves forward in time, and the video recording captures the last two seconds. I have sent the live signal to the dac so that you can hear the sync between the original and the looped audio. (Very important.) I had difficulty for some reason getting video out of jit.world to a jit.pwindow, so I have two jit.pwindows, one for the live and one for the looped video. I am also syncing the transport with the start of the audio recording to get transport and groove in sync.
So with all this effort the result is that audio and video do not start together, and they are out of sync in terms of speed. I believe the video is a bit slowed down. (the best layed plans...) I set everything at 30 fps according to your advice but I would prefer 60fps because that is how I am working.
So if you can wrap your head around what I have done and find a way to pull the transport together with the groove~ and video looping, while at the same time the recording can transition seamlessly from the original live into the loop, that would be totally mind bending.
Once the timing is right, it would be great to phase shift the audio-video loop while maintaining sync between the two. That is why I am working with mc audio objects.
Thanks!
Randall
here's the changes I would suggest:
first you were double banging your jit.grab output, likely the cause of the slowdown in playback. by default jit.grab is in automatic mode, so it outputs whenever the render context draws. either disable automatic mode or remove the extra bang from the jit.world. I would also disable unique mode, to ensure the output is exactly in sync with the jit.world (probably not an issue at 30fps, but maybe will be at higher fps, or whenever the output fps of your device can't keep up with the render fps)
second, i would add an extra second or so of frames to your texture buffer for "slop"
third, whenever a new loop recording is started, simply start writing at index 0, this will make it much easier to sync. this may not work for your particular needs, e.g. if overwriting frames from existing recording is undesirable, but makes much easier to work with at this stage. and in fact it might be advisable to create 2 video buffers and ping-pong between them if necessary for your design.
last, to sync take the sync~ output of groove~ and simply multiply it by the number of frames that were written to the buffer, and use that value to trigger the videobuffer output. use a snapshot~ triggered by jit.world render bang to keep everything in good sync.
Beyond Awesome... it's perfect. Thank you Rob! I'll post results here when I have the patch ready to go. And by the way, this is part of a project called Post Reality TV, which will begin 24/7 live streaming via the Third Space Network on Halloween. https://www.thirdspacenetwork.com. I'll be sure to credit you and stay tuned!
I wanted to share another revision:
What needed to be added is the ability to switch between the original live audio-video signal state to looping, and then back again. So I created an uber live/loop switch. I have to say the transition from live to looping is quite seamless. It's interesting to use very short loop lengths as short as .125 ( I should probably convert milliseconds to beat time) to get a stutter effect. I needed to also pass the video loop length to the audio buffer length as well as the delay length that shuts off the looper toggle.
There is also a simple metro that can be turned on that triggers a new loop rhythmically.
Finally, I am going to add a trigger that rhythmically changes the loop length so the effect is like changing dubstep stutter, similar to the Ableton beat repeat.
Thanks again Rob!!