MBroach

I've been working on a video step sequencer -- one that works exactly like an audio step sequence, but uses short video clips instead of audio samples. There are up to 8 possible slots for loading videos -- and four different ways to arrange those videos in the window (full screen, vertical stripes, horizontal stripes, and random rectangles).

I've attached a version without any video footage along with this post -- to load footage into the sequencer, drop a couple folders with video clips in them into the footage folder; the sequencer should then load them automatically when it next opens. If you'd rather just jump in and work with some pre-supplied video, you can download this version from my webspace:

http://www.aphasiac.org/download/SequencerDemo.zip

In general, the functionality is there, but I find there's more lag in loading/playing back the video files than I would like. Does anyone have any suggestions as to how I can speed up playback? It becomes clear, particular when there are multiple videos going, that playback is laggy and irregular. Is it just an issue of having 8 QTs/Videoplanes going, and there's no way around it? Or are there some steps I could take to streamline the process a bit? I'd love whatever thoughts you might have.

Hey all-
I've been working on a video step sequencer -- one that works exactly like an audio step sequence, but uses short video clips instead of audio samples.  There are up to 8 possible slots for loading videos -- and four different ways to arrange those videos in the window (full screen, vertical stripes, horizontal stripes, and random rectangles).  

I've attached a version without any video footage along with this post -- to load footage into the sequencer, drop a couple folders with video clips in them into the footage folder; the sequencer should then load them automatically when it next opens.  If you'd rather just jump in and work with some pre-supplied video, you can download this version from my webspace:
http://www.aphasiac.org/download/SequencerDemo.zip

In general, the functionality is there, but I find there's more lag in loading/playing back the video files than I would like.  Does anyone have any suggestions as to how I can speed up playback?  It becomes clear, particular when there are multiple videos going, that playback is laggy and irregular.  Is it just an issue of having 8 QTs/Videoplanes going, and there's no way around it?  Or are there some steps I could take to streamline the process a bit?  I'd love whatever thoughts you might have.

video-sequencer

Maybe you could use matrices to store the video? Then it would be uncompressed in ram which should give you an advantage.

I guess Ram becomes an issue as there is a 3GB (or is it 2GB?) limit. Not sure how much video that is in minutes but maybe not enough for you. But maybe you could load it into the matrix asynchronously.

In my latest (m4l) project I store my video on a Ram Disk and frame dump it into a jit.matrixset when it is loaded.

If it is triggered before it is all dumped I use jit.qt.movie but once all the frames are dumped and the matrixset is full I use that as the performance seemed better (for scratching).

Seems a bit overcomplicated but its the best performance I could get. The framedumping was so I could load a video whilst playing another without the dreaded stutter - maybe this isn't so much of an issue for you as the videos can be loaded in advance.

Similarly I found a ram disk avoided the stutter that 'loadram' was causing on my system. But not sure this is an issue for you.

I downloaded your file - it looks good! I couldn't notice the lag myself as its by nature quite glitchy anyway. Maybe you don't need to worry about it!

Or maybe you could combine all the videos together and just extract the relevant bits?

In my latest (m4l) project I store my video on a Ram Disk and frame dump it into a jit.matrixset when it is loaded.
If it is triggered before it is all dumped I use jit.qt.movie but once all the frames are dumped and the matrixset is full I use that as the performance seemed better (for scratching).

Seems a bit overcomplicated but its the best performance I could get. The framedumping was so I could load a video whilst playing another without the dreaded stutter - maybe this isn't so much of an issue for you as the videos can be loaded in advance.
Similarly I found a ram disk avoided the stutter that 'loadram' was causing on my system. But not sure this is an issue for you.

instead of using 8 snippets you could have these 8 videos in one file and just jump to the right positions.

if something causes a delay a when triggering it, so that it plays late, you can solve this problem by

implementing a logical delay into your sequencer routine, in order to compensate the time.

not sure how to do this for video, but eventually you´d just add 12 frames before each snippet (which

could be copies of frame 1) and start actually playing the video always 0.5 seconds before they are sent

to the display: think "tracks", and not "sampler".

if something causes a delay a when triggering it, so that it plays late, you can solve this problem by
implementing a logical delay into your sequencer routine, in order to compensate the time.
not sure how to do this for video, but eventually you´d just add 12 frames before each snippet (which
could be copies of frame 1) and start actually playing the video always 0.5 seconds before they are sent
to the display: think "tracks", and not "sampler".

gavspav -- the framedump/RAM disk idea is interesting, but the problem is sync sound -- I'm using both the audio and the video from the clips to create audio/visual rhythms. This is why accurate triggering is so important -- I'm aiming to create on the fly audio/visual "beats" in the same way that you might with a normal audio step sequencer.

Roman -- given that I'm working with a huge bank of video clips, I think combining them all into a single clip might be cumbersome. The other problem is that this wouldn't allow me to play multiple videos at once, the way the current version of the sequencer does. However, the delay idea seems worth exploring -- I'd have to create a second set of clips with the extra space, but jumping in a file that's already playing might cut down a fair amount of the lag time. I'd be curious to know if anyone had played with this idea in video/Jitter before, and how well it worked.

gavspav -- the framedump/RAM disk idea is interesting, but the problem is sync sound -- I'm using both the audio and the video from the clips to create audio/visual rhythms.  This is why accurate triggering is so important -- I'm aiming to create on the fly audio/visual "beats" in the same way that you might with a normal audio step sequencer.

Roman -- given that I'm working with a huge bank of video clips, I think combining them all into a single clip might be cumbersome.  The other problem is that this wouldn't allow me to play multiple videos at once, the way the current version of the sequencer does.  However, the delay idea seems worth exploring -- I'd have to create a second set of clips with the extra space, but jumping in a file that's already playing might cut down a fair amount of the lag time.  I'd be curious to know if anyone had played with this idea in video/Jitter before, and how well it worked.

Yes I'm using sound too. I load the sound into a buffer object which can be driven with phasor~ or groove~.

I take the playhead info, divide it by the sample and frame rates, pass it through a change object and use this as the index of the jit.matrixset.

I would suggest that if you are aiming for optimum triggering performance then you need to separate audio and video files anyway.

Definitely you should add a @vol 0. to your jit.qt.movies and deal with the sounds on a parallel circuit.

Even sfplay~ objects would work better. But useless to try using spigot~ for your project.

You could also try these optimisations methods using the @colormode uyvy and the @unique 1:

Definitely you should add a @vol 0. to your jit.qt.movies and deal with the sounds on a parallel circuit.
Even sfplay~ objects would work better. But useless to try using spigot~ for your project.

You could also try these optimisations methods using the @colormode uyvy and the @unique 1:


Florent -- Thanks for your suggestions. I'll definitely give those a try, see how well it speeds things up.

gavspav -- do you have a demo patch that shows your method? I'm sure I could figure it out with a little trial-and-error, but if you've got something rigged up already, I'd love to see it. Definitely sounds promising.

Florent -- Thanks for your suggestions.  I'll definitely give those a try, see how well it speeds things up.  

gavspav -- do you have a demo patch that shows your method?  I'm sure I could figure it out with a little trial-and-error, but if you've got something rigged up already, I'd love to see it.  Definitely sounds promising.


I've just put this together to at least let you know what I'm on about.

There is no framedumping here, just the a/v sync.

You need to load the video and corresponding audio/wav files manually I'm afraid.

For the demo I just used a pwindow - sorry!

FWIW I made a glitchy m4l video shuffler a few years ago before I knew this method!. It just uses jit.qt.movie as I remember.

https://forum.ableton.com/viewtopic.php?f=35&t=132447

I've just put this together to at least let you know what I'm on about.
There is no framedumping here, just the a/v sync.
You need to load the video and corresponding audio/wav files manually I'm afraid.
For the demo I just used a pwindow - sorry!

FWIW I made a glitchy m4l video shuffler a few years ago before I knew this method!. It just uses jit.qt.movie as I remember.
https://forum.ableton.com/viewtopic.php?f=35&t=132447


Oh I just found the basis of the framedump method in my email archives from Dave@Cycling74 Support.

Oh I just found the basis of the framedump method in my email archives from Dave@Cycling74 Support.
Hope all this stuff is some use.


gavspav -- it took me awhile to get around to it, but I finally had a chance to implement your jit.matrixset and audio synch methods -- and it works so, so much better now. Much less notable lag in trigger -- particularly on the audio side, where it would be more noticeable, anyway. Thanks so much for your help!

gavspav -- it took me awhile to get around to it, but I finally had a chance to implement your jit.matrixset and audio synch methods -- and it works so, so much better now.  Much less notable lag in trigger -- particularly on the audio side, where it would be more noticeable, anyway.  Thanks so much for your help!


Great. If you turn it into an M4L device and share it, let me know.

Video Sequencer