karl krach

i'm quite new to max and only started with it for a specific but rather demanding project. as by now, i'm done with the preliminary work (testing and interface programming) and i'm getting to the core of my little project and start wondering what would be the best approach...?

first, i want to capture a live video input to a buffer, so it continually overwrites itself (pretty much like a surveillance system capturing always, let's say, the last 30 min). simultaneously i want to play the buffer content back and be able to shuttle around (fast forward/rewind) in it as i could do in a quicktime movie using "rate $1" and "jump $1" in jit.qt.movie.

on top if it i would need to modify some stuff in the final output but ok, i guess i'll save that for later...

- now, what commands should i be looking at? jit.qt.grab? jit.matrixset? ...?

- are there any suggestions on what codec to use? i did quite a bit of testing but my results are rather incoherent. the only sure way, i managed to get really smooth shuttling at high speeds (32x) was, to preload small (640x360) uncompressed or almost uncompressed (animation@100) quicktimemovies into ram...

- since i might not have THAT much ram available and i'm looking at quite a large buffer: might it be possible not to save this to ram but use a ssd-drive instead? i don't own one yet but afaik these are quite fast, especially when it comes to seek time which i guess might be important for my setup...?

any help will be much appreciated, really!

hi everybody,
i'm quite new to max and only started with it for a specific but rather demanding project. as by now, i'm done with the preliminary work (testing and interface programming) and i'm getting to the core of my little project and start wondering what would be the best approach...?

first, i want to capture a live video input to a buffer, so it continually overwrites itself (pretty much like a surveillance system capturing always, let's say, the last 30 min). simultaneously i want to play the buffer content back and be able to shuttle around (fast forward/rewind) in it as i could do in a quicktime movie using "rate $1" and "jump $1" in jit.qt.movie.
on top if it i would need to modify some stuff in the final output but ok, i guess i'll save that for later...

- now, what commands should i be looking at? jit.qt.grab? jit.matrixset? ...?
- are there any suggestions on what codec to use? i did quite a bit of testing but my results are rather incoherent. the only sure way, i managed to get really smooth shuttling at high speeds (32x) was, to preload small (640x360) uncompressed or almost uncompressed (animation@100) quicktimemovies into ram...
- since i might not have THAT much ram available and i'm looking at quite a large buffer: might it be possible not to save this to ram but use a ssd-drive instead? i don't own one yet but afaik these are quite fast, especially when it comes to seek time which i guess might be important for my setup...?

any help will be much appreciated, really!
karl


capture-video-to-a-buffer-then-shuttle-around-in-it

jit.matrixset is the object to explore further. search the jitter examples folder, as well as the forum.

however, as you are aware, the amount of frames you can store is RAM limited.

another possibility is to write your frames to disk as an image sequence, as opposed to a quicktime movie. that way you can continuously read and write the, and are not limited by RAM.

however, as you are aware, the amount of frames you can store is RAM limited.
another possibility is to write your frames to disk as an image sequence, as opposed to a quicktime movie. that way you can continuously read and write the, and are not limited by RAM.


thank you for your answer - saving the whole stuff as an image sequence is quite an interesting thought. is it possible to read an image sequence continually? i mean, i can open it as a movie via quicktime but how do i get it to update itself?

another thought which occured to me in the last hour: would it be a good idea to work with a ram disk (from os x) in the meantime? then i would have all the speed of ram but could change the chaching location to a ssd-drive later without setting the whole stuff up anew... any experiences with that anyone?

yeah, and i got a jit.matrixset-buffer to work but how the heck do i get it to behave it like a quicktime-mov (rate and jump...)?!? any ideas?

thank you for your answer - saving the whole stuff as an image sequence is quite an interesting thought. is it possible to read an image sequence continually? i mean, i can open it as a movie via quicktime but how do i get it to update itself?
another thought which occured to me in the last hour: would it be a good idea to work with a ram disk (from os x) in the meantime? then i would have all the speed of ram but could change the chaching location to a ssd-drive later without setting the whole stuff up anew... any experiences with that anyone?
yeah, and i got a jit.matrixset-buffer to work but how the heck do i get it to behave it like a quicktime-mov (rate and jump...)?!? any ideas?
tanks again...
karl


I believe a 3d matrix is a more efficient video buffer than jit.matrixset. Just set the z dimension to the number of frames you want in the buffer and use a 2d jit.submatrix to pull individual frames with the offset being the frame number.

I believe a 3d matrix is a more efficient video buffer than jit.matrixset.  Just set the z dimension to the number of frames you want in the buffer and use a 2d jit.submatrix to pull individual frames with the offset being the frame number.


3d matrix, mhm, i see. i tried to get some understanding of what you mean today but i'm not sure wether i got it right so let me try to put this in my own words to see wether this is, what you meant:

the basic idea is to use opengl because of it's hardware acceleration, right? for the output i could use something like jit.gl.videoplane (which works :) but i didn't find anything about a 3d buffer...?!? what objects should i be looking for? how do i make my buffer?

and how can i control the playback via "rate" and jump"?

3d matrix, mhm, i see. i tried to get some understanding of what you mean today but i'm not sure wether i got it right so let me try to put this in my own words to see wether this is, what you meant:
the basic idea is to use opengl because of it's hardware acceleration, right? for the output i could use something like jit.gl.videoplane (which works :) but i didn't find anything about a 3d buffer...?!? what objects should i be looking for? how do i make my buffer?
and how can i control the playback via "rate" and jump"?

Basically you would take the output of your jit.qt.movie, and feed it into a jit.matrix 4 char 320 240 10 (the last 10 is how many 320x240 frames you would want to 'buffer'). So for a 640x480 movie, that you want to buffer a continuous 100 frames in, you would make a jit.matrix 4 char 640 480 100 (see how that is 3D?). You then add slices to the end of it, and read slices from the front of it (or vice versa, etc). The output of that matrix then goes to GL land like normal.

See? Now build it. Its a good exercise to understand how jitter matrices work.

You can then have fun by doing all sorts of src dim and dest dim madness to make slit scans etc.

ah! ok, i got it. i guess this'll keep me busy for a little... :)

ah! ok, i got it. i guess this'll keep me busy for a little... :)
i'll get back once i got it set up.
thank all of you so much for your help!
karl


for the records: i didn't manage to figure out how to use a normal matrix as buffer so i went with the very convenient jit.matrixset which actually does quite well in terms of performance (buffering about 5000 1 char 720x576 matrices)...

if anyone wants to try using a normal jit.matrix as storage solution though, i think this is how to do it: 

https://cycling74.com/articles/a-noisy-matrix/

 (that's the stuff i didn't manage to understand... ;)

if anyone wants to try using a normal jit.matrix as storage solution though, i think this is how to do it: https://cycling74.com/articles/a-noisy-matrix/ (that's the stuff i didn't manage to understand... ;)


For the slitscanning vade was talking about, look at /examples/jitter-examples/video/matrix/jit.matrix-slitscan.maxpat and jit.matrix-slitscan-multi.maxpat.

it's that simple?!? i just read the article i posted the link above and that had me completely mystified... thank you!

however today i discovered either way i have a REALLY BIG PROBLEM:

for the last three weeks i spent day and night programming my patch on a macbook pro with 3 GB RAM. on that system i could buffer up to 6000 frames (= 4 min 720x576, 1 plane greyscale video) into my matrixset. after that i started to get pageouts so i didn't try any further.

today however i moved my patch to a bigger system with 12 GB RAM and i tried to make my matrixset bigger but as soon as total memory consumption (monitored in the activity monitor) went past 3,02 GB Max would quit instantly.

since from the former posts i gathered a 3d jit.matrix might perform better i did some testing with your patch above but in my experience the performance was actually worse: as soon as i tried to make a single jit.matrix with more then about 2550 instances (720 576 2550) i got the error message “could not allocate matrix”. i then tried to combine multiple jit.matrices but in no case i got past the 3GB (about 6500 frames) plus cpu-load went for the stars. as well, the whole thing started to behave quite incoherently - sometimes it would let me create a matrix with 2500 frames, other times with only 500 frames it would tell me it couldn’t allocate it.

can anyone tell me what’s happening here? for getting my installation to work i need to buffer at least about 30 minutes (= 45.000 frames = about 18 GB RAM).

since i do need to be able to rapidly shuttle around in the video using compression is not an option. and if i don’t get this to work one way or another i spent the complete last month for nothing…

someone with a better understanding of this please give me some idea of what's happening here!

thank you so very much for the loads of help i get here!!!

for the last three weeks i spent day and night programming my patch on a macbook pro with 3 GB RAM. on that system i could buffer up to 6000 frames (= 4 min 720x576, 1 plane greyscale video) into my matrixset. after that i started to get pageouts so i didn't try any further.
today however i moved my patch to a bigger system with 12 GB RAM and i tried to make my matrixset bigger but as soon as total memory consumption (monitored in the activity monitor) went past 3,02 GB Max would quit instantly.
wtf?!?
since from the former posts i gathered a 3d jit.matrix might perform better i did some testing with your patch above but in my experience the performance was actually worse: as soon as i tried to make a single jit.matrix with more then about 2550 instances (720 576 2550) i got the error message “could not allocate matrix”. i then tried to combine multiple jit.matrices but in no case i got past the 3GB (about 6500 frames) plus cpu-load went for the stars. as well, the whole thing started to behave quite incoherently - sometimes it would let me create a matrix with 2500 frames, other times with only 500 frames it would tell me it couldn’t allocate it.

can anyone tell me what’s happening here? for getting my installation to work i need to buffer at least about 30 minutes (= 45.000 frames = about 18 GB RAM).
since i do need to be able to rapidly shuttle around in the video using compression is not an option. and if i don’t get this to work one way or another i spent the complete last month for nothing…

PLEASE PLEASE PLEASE!!!
someone with a better understanding of this please give me some idea of what's happening here!
thank you so very much for the loads of help i get here!!!


I've had the "could not allocate matrix" error before with much smaller buffers than you're trying. I think maximum matrix size is limited regardless of your RAM because Jitter can only do 32bit addresssing, but someone more knowledgeable than me will have to comment on that. I am pretty sure you'll be able to make it larger with char than float32, though.

I would have to know more about what you're trying to do with the buffer here, but I have to think there's a better solution than buffering the entire 30 minutes of video. If you don't need to pull frames from every part of it at once maybe you can use several buffers that get loaded to and from disk. This might cause serious latency issues though; I've never actually tried it. I also remember someone proposing running multiple patches in runtime to get around the memory space issue, but once again- not tested.

Finally, if you don't need it to be completely live you *can* scrub through compressed video from jit.qt.movie with a decent framerate (plus loadram really helps, although you'll want to specify only a portion of your total 30 min). You just stop playback and send it individual frame messages. There are at least a few examples of that on the forums.

I've had the "could not allocate matrix" error before with much smaller buffers than you're trying.  I think maximum matrix size is limited regardless of your RAM because Jitter can only do 32bit addresssing, but someone more knowledgeable than me will have to comment on that.  I am pretty sure you'll be able to make it larger with char than float32, though. 

I would have to know more about what you're trying to do with the buffer here, but I have to think there's a better solution than buffering the entire 30 minutes of video.  If you don't need to pull frames from every part of it at once maybe you can use several buffers that get loaded to and from disk.  This might cause serious latency issues though; I've never actually tried it.  I also remember someone proposing running multiple patches in runtime to get around the memory space issue, but once again- not tested. 

Finally, if you don't need it to be completely live you *can* scrub through compressed video from jit.qt.movie with a decent framerate (plus loadram really helps, although you'll want to specify only a portion of your total 30 min).  You just stop playback and send it individual frame messages.  There are at least a few examples of that on the forums.


what i'm trying to do is the following: it's an interactive installation where a camera is filming the audience. this live input gets buffered in a loop (pretty much like in a surveillance system), is projected and can be controlled by the audience via a sort of jog/shuttle so they can play, rewind, fast forward and so forth. the maximum shuttle speed is 32x/-32x so in terms of speed i have to be able to load 32 minutes of video footage in one minute of real time. there are some modifications to the live image (i.e. simple keying with fixed matte keys, all on the gpu-side) plus sometimes the image gets replaced by prerecorded footage without the audience being able to realize the switch (no delays while swiching!). to realize this, i'm working with two matrixset-buffers, one big one for the live input and a smaller one for the prerecorded stuff. as soon as a prerecorded scene is shown, in the background the second buffer adjusts its size to the next prerecorded scene and loads that into the matrixset replacing the last one. since the whole thing is about playing with time the fluidity of moving and shuttling is key issue.

the cool thing with this system right now is, that the navigation is always performed with the framenumbers of the live-buffer so in terms of navigation the image switch is not perceivable.

up to the 3gb ram limit my system performs quite well not doing any glitches when switching while most of the time having quite a low cpu-load. the central enginge (the two buffers) i attached below...

what i was thinking about in the meantime: if it really shouldn't be possible to adress more than 3 GB in max maybe i could do this with still images being saved to a stripeset or maybe even ssd-drives? my absolute data rate isn't that high really (PAL playback), i guess the problems will be more about seek time and cpu-load for the quicktime engine. could this maybe be done with .jxf (like a big .jxf-image-sequence) avoiding quicktime altogether?

thank you for your quick reply!
what i'm trying to do is the following: it's an interactive installation where a camera is filming the audience. this live input gets buffered in a loop (pretty much like in a surveillance system), is projected and can be controlled by the audience via a sort of jog/shuttle so they can play, rewind, fast forward and so forth. the maximum shuttle speed is 32x/-32x so in terms of speed i have to be able to load 32 minutes of video footage in one minute of real time. there are some modifications to the live image (i.e. simple keying with fixed matte keys, all on the gpu-side) plus sometimes the image gets replaced by prerecorded footage without the audience being able to realize the switch (no delays while swiching!). to realize this, i'm working with two matrixset-buffers, one big one for the live input and a smaller one for the prerecorded stuff. as soon as a prerecorded scene is shown, in the background the second buffer adjusts its size to the next prerecorded scene and loads that into the matrixset replacing the last one. since the whole thing is about playing with time the fluidity of moving and shuttling is key issue.
the cool thing with this system right now is, that the navigation is always performed with the framenumbers of the live-buffer so in terms of navigation the image switch is not perceivable.
up to the 3gb ram limit my system performs quite well not doing any glitches when switching while most of the time having quite a low cpu-load. the central enginge (the two buffers) i attached below...

you're going to be limited by 32-bit memory addressing for really long video buffers. For longer sequences, I highly recommend a disk-based approach. Check out the patch below, which writes and reads jit.matrix JXF files. The playback can be controlled the same as jit.matrixset, but it all resides on disk. Using an external FW800 drive, I'm able to get easy 30fps with this method.

you're going to be limited by 32-bit memory addressing for really long video buffers.  For longer sequences, I highly recommend a disk-based approach.  Check out the patch below, which writes and reads jit.matrix JXF files.  The playback can be controlled the same as jit.matrixset, but it all resides on disk.  Using an external FW800 drive, I'm able to get easy 30fps with this method.

i just checked your patch and yep, that seems to be the way to go.

since i need to do the shuttling i think i'll try it with ssd-drives for they have no seek time... i'll post the outcome.

thank you so very much for your help again. i'll sleep much better tonight!

i just checked your patch and yep, that seems to be the way to go.
a .jxf-image-sequence.

since i need to do the shuttling i think i'll try it with ssd-drives for they have no seek time... i'll post the outcome.
thank you so very much for your help again. i'll sleep much better tonight!
:)


i'm running Andrew Benson's patch in a macbook pro and I'm only getting frame rates of 6 fps. any idea why this happens? does a an external FW800 drive works better?

i'm running Andrew Benson's patch in a macbook pro and I'm only getting frame rates of 6 fps. any idea why this happens?  does a an external FW800 drive works better?

I'm running Andrew Benson's patch and I'm just wondering how I can change it to capture video in video format (mov,mpeg, etc.) instead of image sequence. Also, can I just add stop message to the jit.matrix to stop the recording instead of toggling the whole "write to disk"?

The patch I posted above is really specific to the needs of the OP (very long video buffers with the ability to have continuous recording and random access). Unless you find yourself in a similarly difficult situation, I'd recommend sticking to jit.matrixset or jit.qt.record for recording.

Hi piyo240,
The patch I posted above is really specific to the needs of the OP (very long video buffers with the ability to have continuous recording and random access).  Unless you find yourself in a similarly difficult situation, I'd recommend sticking to jit.matrixset or jit.qt.record for recording.


At the risk of ressurecting the old thread, have to ask: is there a new way to do this in Max 8? I.e. continuously record a video buffer and read certain segments from it?

At the risk of ressurecting the old thread, have to ask: is there a new way to do this in Max 8? I.e. continuously record a video buffer and read certain segments from it?
Thanks!

i would still recommend matrixset for short buffers (limited only by the amount of memory available on your machine), and image or binary matrix sequences for longer durations (preferably writing and reading from a fast SSD). Both techniques allows for maximum flexibility with playback of the buffer, but require writing your own logic and timing structures.

No good way of doing via GL? And could you please elaborate a bit more on " image or binary matrix sequences"?

No good way of doing via GL? And could you please elaborate a bit more on " image or binary matrix sequences"?
Many thanks!

I think Rob is talking about the use of jxf files, as described in previous answers above.

yeah the result of a write message to jit.matrix. for GL you can look at jit.gl.textureset, but again buffer length will be limited by cpu and gpu resources

Thanks! I will look into both options - the patch of Andrew works, but after a bit it creates some problems with writing/reading (MacBook Pro from 2016) - the whole system starts lagging.

On a side note: I don't write here often and am astouned how helpful you all are! Thanks!!

Thanks! I will look into both options - the patch of Andrew works, but after a bit it creates some problems with writing/reading  (MacBook Pro from 2016) - the whole system starts lagging.

On a side note: I don't write here often and am astouned how helpful you all are! Thanks!!

capture video to a buffer, then shuttle around in it?