capture video to a buffer, then shuttle around in it?
i’m quite new to max and only started with it for a specific but rather demanding project. as by now, i’m done with the preliminary work (testing and interface programming) and i’m getting to the core of my little project and start wondering what would be the best approach…?
first, i want to capture a live video input to a buffer, so it continually overwrites itself (pretty much like a surveillance system capturing always, let’s say, the last 30 min). simultaneously i want to play the buffer content back and be able to shuttle around (fast forward/rewind) in it as i could do in a quicktime movie using "rate $1" and "jump $1" in jit.qt.movie.
on top if it i would need to modify some stuff in the final output but ok, i guess i’ll save that for later…
- now, what commands should i be looking at? jit.qt.grab? jit.matrixset? …?
- are there any suggestions on what codec to use? i did quite a bit of testing but my results are rather incoherent. the only sure way, i managed to get really smooth shuttling at high speeds (32x) was, to preload small (640×360) uncompressed or almost uncompressed (animation@100) quicktimemovies into ram…
- since i might not have THAT much ram available and i’m looking at quite a large buffer: might it be possible not to save this to ram but use a ssd-drive instead? i don’t own one yet but afaik these are quite fast, especially when it comes to seek time which i guess might be important for my setup…?
any help will be much appreciated, really!
jit.matrixset is the object to explore further. search the jitter examples folder, as well as the forum.
however, as you are aware, the amount of frames you can store is RAM limited.
another possibility is to write your frames to disk as an image sequence, as opposed to a quicktime movie. that way you can continuously read and write the, and are not limited by RAM.
thank you for your answer – saving the whole stuff as an image sequence is quite an interesting thought. is it possible to read an image sequence continually? i mean, i can open it as a movie via quicktime but how do i get it to update itself?
another thought which occured to me in the last hour: would it be a good idea to work with a ram disk (from os x) in the meantime? then i would have all the speed of ram but could change the chaching location to a ssd-drive later without setting the whole stuff up anew… any experiences with that anyone?
yeah, and i got a jit.matrixset-buffer to work but how the heck do i get it to behave it like a quicktime-mov (rate and jump…)?!? any ideas?
I believe a 3d matrix is a more efficient video buffer than jit.matrixset. Just set the z dimension to the number of frames you want in the buffer and use a 2d jit.submatrix to pull individual frames with the offset being the frame number.
3d matrix, mhm, i see. i tried to get some understanding of what you mean today but i’m not sure wether i got it right so let me try to put this in my own words to see wether this is, what you meant:
the basic idea is to use opengl because of it’s hardware acceleration, right? for the output i could use something like jit.gl.videoplane (which works :) but i didn’t find anything about a 3d buffer…?!? what objects should i be looking for? how do i make my buffer?
and how can i control the playback via "rate" and jump"?
thank you very much again!!
Just use a jit.matrix, no need for GL.
Basically you would take the output of your jit.qt.movie, and feed it into a jit.matrix 4 char 320 240 10 (the last 10 is how many 320×240 frames you would want to ‘buffer’). So for a 640×480 movie, that you want to buffer a continuous 100 frames in, you would make a jit.matrix 4 char 640 480 100 (see how that is 3D?). You then add slices to the end of it, and read slices from the front of it (or vice versa, etc). The output of that matrix then goes to GL land like normal.
See? Now build it. Its a good exercise to understand how jitter matrices work.
You can then have fun by doing all sorts of src dim and dest dim madness to make slit scans etc.
ah! ok, i got it. i guess this’ll keep me busy for a little… :)
i’ll get back once i got it set up.
thank all of you so much for your help!
for the records: i didn’t manage to figure out how to use a normal matrix as buffer so i went with the very convenient jit.matrixset which actually does quite well in terms of performance (buffering about 5000 1 char 720×576 matrices)…
if anyone wants to try using a normal jit.matrix as storage solution though, i think this is how to do it: http://cycling74.com/2006/06/29/a-noisy-matrix/ (that’s the stuff i didn’t manage to understand… ;)
It works like this:
----------begin_max5_patcher---------- 940.3oc2X00aaBCE84jeEVnsmVZD9K9XpOz86XppBBNItJ.ofQMaU8+9.aXg 1RRLDha1TThKtF6y8bO2601uLchUX5NVtE36feBlL4koSlH6ppiI0OOwJNX2 hMA4xgYsHMNlkHrlo9eB1Ngr+au8VP95hkK2v.qYYrlArLMQjy+MqZPPzb65 tSJh4IaXB4jBaM1jfX4Xs9QFOXSyrrMPrXMOY0CYrEBEboNvxYC.cPxFpcUC p7Wv86WizBQyhzrx7H4zmF93MPhUUeuNcZ0OyzjAx2viXYMHSsBheskofkkU 60+i1XWVhsizDfMVP4eSZaH+k+rssayfsrNXGVGZHVW4zFdDqiW45AVgAIq1 anWNebMyn7wmzEi5fDfChDXOW9xePi+HWL+YdRT5yfkooGjijjyr1JgKFAg8 7p3DBR13g6OA4OD9IlkmGrh8ABJc4xbl.XW94KPMhPzgWPmgvAikBGWWoJx6 f7RGQOD6QV3jWDFGHx36.2EwiAXGa.wy9fjT4q7fZ7lUJg8wp7oN8WKg8FYN qlvBKVd0wSHehTZQG.M4NhzzWkUENZ9ZiEtgnRsCEdBNoivMryHxIe6ZhSPR 1f3M.Ng9+RAbjJlYHEvwjQTXby0gl.54KS0hlS6slvdLqWGkKJqEkKBxZJZO Cn5ikDAvTe.w0+StTNjnRm3qZvv9WKG47OefTCKP8GZfDZLCjDfMfvCqJLGu 3oNtjusVATcQKnQjVVjVjHXYkQR999mR4z7s2odnmi.p9jBjSHfHcvTvK1t6 .Dvh0AYMaJVV9FbWQNSkLB.ud19mRvgIp8843OfijNtG4ZKK4yL8LVE.BUm2 Z.GzxcjUUOIluJKHD3PtxNfUcpJphirc6uxgNpYvCORF7lqvvrYwqEQj9yLC p1lHc0pML81Rnt2kmxRHn8WkWO2e2H5heJlIxRAX7wcyW9bDJRQcTYGZ+Ik2 T3Q9ZVa3Iu+dqkKbU+ukoxSKxVzXvpnnYf8KdDKWvSBD7zjViobWAsGzZdTD Ko8sHGyi1lVpPpw.39Nca5BIhFPhZTD4N9HB6Ws6rpKpurwa+CmIRgHMfJDZ T16cKW2Xp7nblDSXcboXOiq6gmBRt8CRPahTZQTES7c+6SmKApiNyrtTOMfj uY8n1W.UFERqbhXWYpiZWp7oyk.0IwKwvwoN5vfDyhIcpY9NfewwjVJMyV0D oUsHyFQhbtBwjNwcHypmP5TyFQLud5TEHQFduM5jx7LPT4CuN8O.lgx69A -----------end_max5_patcher-----------
For the slitscanning vade was talking about, look at /examples/jitter-examples/video/matrix/jit.matrix-slitscan.maxpat and jit.matrix-slitscan-multi.maxpat.
it’s that simple?!? i just read the article i posted the link above and that had me completely mystified… thank you!
however today i discovered either way i have a REALLY BIG PROBLEM:
for the last three weeks i spent day and night programming my patch on a macbook pro with 3 GB RAM. on that system i could buffer up to 6000 frames (= 4 min 720×576, 1 plane greyscale video) into my matrixset. after that i started to get pageouts so i didn’t try any further.
today however i moved my patch to a bigger system with 12 GB RAM and i tried to make my matrixset bigger but as soon as total memory consumption (monitored in the activity monitor) went past 3,02 GB Max would quit instantly.
since from the former posts i gathered a 3d jit.matrix might perform better i did some testing with your patch above but in my experience the performance was actually worse: as soon as i tried to make a single jit.matrix with more then about 2550 instances (720 576 2550) i got the error message "could not allocate matrix". i then tried to combine multiple jit.matrices but in no case i got past the 3GB (about 6500 frames) plus cpu-load went for the stars. as well, the whole thing started to behave quite incoherently – sometimes it would let me create a matrix with 2500 frames, other times with only 500 frames it would tell me it couldn’t allocate it.
can anyone tell me what’s happening here? for getting my installation to work i need to buffer at least about 30 minutes (= 45.000 frames = about 18 GB RAM).
since i do need to be able to rapidly shuttle around in the video using compression is not an option. and if i don’t get this to work one way or another i spent the complete last month for nothing…
PLEASE PLEASE PLEASE!!!
someone with a better understanding of this please give me some idea of what’s happening here!
thank you so very much for the loads of help i get here!!!
I’ve had the "could not allocate matrix" error before with much smaller buffers than you’re trying. I think maximum matrix size is limited regardless of your RAM because Jitter can only do 32bit addresssing, but someone more knowledgeable than me will have to comment on that. I am pretty sure you’ll be able to make it larger with char than float32, though.
I would have to know more about what you’re trying to do with the buffer here, but I have to think there’s a better solution than buffering the entire 30 minutes of video. If you don’t need to pull frames from every part of it at once maybe you can use several buffers that get loaded to and from disk. This might cause serious latency issues though; I’ve never actually tried it. I also remember someone proposing running multiple patches in runtime to get around the memory space issue, but once again- not tested.
Finally, if you don’t need it to be completely live you *can* scrub through compressed video from jit.qt.movie with a decent framerate (plus loadram really helps, although you’ll want to specify only a portion of your total 30 min). You just stop playback and send it individual frame messages. There are at least a few examples of that on the forums.
thank you for your quick reply!
what i’m trying to do is the following: it’s an interactive installation where a camera is filming the audience. this live input gets buffered in a loop (pretty much like in a surveillance system), is projected and can be controlled by the audience via a sort of jog/shuttle so they can play, rewind, fast forward and so forth. the maximum shuttle speed is 32x/-32x so in terms of speed i have to be able to load 32 minutes of video footage in one minute of real time. there are some modifications to the live image (i.e. simple keying with fixed matte keys, all on the gpu-side) plus sometimes the image gets replaced by prerecorded footage without the audience being able to realize the switch (no delays while swiching!). to realize this, i’m working with two matrixset-buffers, one big one for the live input and a smaller one for the prerecorded stuff. as soon as a prerecorded scene is shown, in the background the second buffer adjusts its size to the next prerecorded scene and loads that into the matrixset replacing the last one. since the whole thing is about playing with time the fluidity of moving and shuttling is key issue.
the cool thing with this system right now is, that the navigation is always performed with the framenumbers of the live-buffer so in terms of navigation the image switch is not perceivable.
up to the 3gb ram limit my system performs quite well not doing any glitches when switching while most of the time having quite a low cpu-load. the central enginge (the two buffers) i attached below…
what i was thinking about in the meantime: if it really shouldn’t be possible to adress more than 3 GB in max maybe i could do this with still images being saved to a stripeset or maybe even ssd-drives? my absolute data rate isn’t that high really (PAL playback), i guess the problems will be more about seek time and cpu-load for the quicktime engine. could this maybe be done with .jxf (like a big .jxf-image-sequence) avoiding quicktime altogether?
----------begin_max5_patcher---------- 2494.3oc6cs0baabE9YoeEXzjGRSjY1ydeyCclbQdpl1lLMwI4g1LdnjfjQJ IHGRvFmlI+2KvtK3EIBfcA.WAHW6wRdoHMNmu8b+b10+94mcwMKde75Kh97n +YzYm86me1Y5Wp3ENyt9rKlO882Na5Z8a6hz3ecwM+xEWZ9QYwuOS+xqhVNa 5usdYb7ck+rzMySRmEmo+bH6KtbZ1suKI8g2tJ91LyyUgISPWFAfp3aB8BLZ BJ5mselj6zOh7G6qHH5d+yuXSV4+9f8UueQZV5z4w5OwWrJY5rx2u4Mm8aKi MO1KtX6Cn3CsN4+p+A.dBp3U+iyOu3KW9rAKDjPiGBWPEXjgJYS+2Y9BHbkU NQV7MhnNDgxHCSDIawCOLK9nrNTCqyU6ohTvz0v5p547mveIo4Oket+1ceXZ 1wYPbyLHkq2ao0xfxQlv9ra7UTWh0x3XhQsuVQchTMpviWEAS7U7.PBsHAnX Z4C7DVMxGTQuAH2OawzrdEUlGud8zGheBr.9iITn.LjzcPRgowiBIjAp4vpP Cj+nA3NZPgWRNGJYc.4h2A7Xz8vVVz30uA+CX13RXGC7VXQjoQDLFuCQpTfG SGYp+SZqDRIdTuA.LdT4xb8xYIYQnIEr0QAFhCpNbsqSyWqV0oBKi31557xS gOza1jksH0W6jkAUwbwLIv7zL4MSSenWsSlGmjuJABi4Q.nMGkDQh5Mc.iKh StZPa7bnP1McjCNNXrO.RiVgMx+XlKoQyeIEojBg2i0avB.SDp.k5sTBXlJj fsQI0PPv7W3oDv.oGnw.U0+g0+ZRNm4qWeFwHbK0B7lDEyQfJ3cpmR5sSL+1 EymGml8jM1u+mt9Me0eI5Ku5M+zUW8MQ+sq+wqh9hu4qi9G+v0e0e8MW+2u5 6K+HyRRiucwlzr84dWU9YH5DpDj.tHXH0D.gPDQ9e2XMfVojQEoNiZTxnOr7 uY9Mwq7NRGfWxRWFwZHPGZuEwWgAvK2FIT+ABUI37k+vqe8UeWzq+1u6HxJN 6SfvKDK3RUNHw2IUvaHGS9yoTwwiG3S8N3HkIT.F3PU0vhwV7h2ilrYVVx77 vFeq1rQ7pHXRDJ+2hCpR9NvRTWbSFIBIyjNkpV4CREQWS5PhTZEqSgBV6rxn nZHfPz9XD0quPfSqYlrUIquc5L86AMQcLrAlvT4wDH5wXRxoskaxxEvVk79n Ox630JQPg1DsBUaMbHm7ZVzA.pJSzyhSeH6cQKtOZx7E+mnjzn6WkSiqiJ90 e9n90wdJFJHvVC3D1VC31NEPqBMoOi1uaY8KHlB4Xbt2P1KjvU+hpTNxeSsP mnrFFTSTYh50I3CXchpwkByEZ481fORa6u0VcsNipDeDiP7II8t31H4X6grE YX75QlwUSjKCeo5HWXM2bciIQITugiJ5ODs0NpK+S0Q80Agn1E5RonBRWuDS RxU2NYzXNzkJDn9kjrIFCQqiyhDLTDDc66ltJRfQQLAukHJEYT6DxZgTjp2f zbF4sFFQKm87GVLHTtl7MoptLLfR9tB4mOwWqyDiAYLfZNWSrfNby0rUBEkb OwLii01DNLmOVkJ9rVJU.RGp.AlQGaUfXYxRu6XkjueUqqetOgQ2ne4cJxZX PRbngl.YrId7IsS1.DbWzVjiM33Uda8fZF1ChC3AAge4iG38JHWChGRwKLWr V6l1olu13tvR5X0C6m1NYBLyIUD1fUlnpBLp6V3Od8We02FYZGj2QlgUaaK3 98+gVaLZ.m7Bn+ODy31CHGbuB3g67B0eSQrI2MfCMOx.fh9AvzBk6mvHhfad Zg.XbUTsJ5OH3amAKwH6P3R30ARL3EeiAs3A1DlZ8MFDyE++FCdDmRlI3zff 02XPLWMfKxeuZLxDKixASQ3g5zsUUbL11hljFcyl6uOd0+J8iy+bSh34wi7m 5oNkJ2EoiyMJEW0osa.2nzRu36l12ZDTZ5jQMXaTpxNq2nl6SJdPO6.ml9jB 1gZxb9oquOoX3Co9jZKOfw2R8sIEC3QZaRUJuaSpAW.hCcIECnO.5RpszAJd yMIECvK9ljV3vricI0fnTSMprtoptDCzv1jzife52nNniGcGgn4shW+PPc8h Mqts7gt895HZG+cW75rjzoYI4d028tzmXybns7c8tj6tKNc+nLlmb2xE4BFV xnB2u8LUULCmAkp.Gnph7GCHQouHMZFpjgknTtRSnvQSRmDoTgknzOObSTkm Dk.TSjDBAHEmkJc7mbt8z2t+J6jJxERcczJV0Y9A4DJCAFkQtfxEC6gOTEfj jBfiXuYCLSMvgqr00WlmMLytpqLSwb96nsQR.MCwblpB8FOz+a7H4ywFOdHB wX5fjpbRbDGX6PXvIpBGXkDlydFCnAEhaXkvu.aTLilKyTuA.LcV6vkXJ01S E71kckc.WTSzm4+.t0+3mWUa87mgfhZjpng1WBe.F.o9xEqYpxWObcEqnNgU LgeTEGY5.tQakapo5gq.jdkpHYY6pNGYia7RXyPVOg1NniPOHv2S+9ta1SBb let4JgDX2tLgyxUCNMWJMvTEa.RU5B.5HU4gkAp1zE0LeSEWJBnGuBaN8FLF RaNTH6g71cJ2BTnqNhXPRUxgHUAtkGlmtbkXl1Qp4JNUnDaWEH5Mrl+HfKFZ zG+EeToAoNqCyAejZN4QGr.aNVVjxNNz8pEXnwF4DdXCsWeDQbDeGb65r9dW GL891tqqW0QNgANyHCO3U1A3E3ncvqcAvk6csqpW0UwW3vppVGmDRkJWbHom o9A2tt9PA4UsWOXWO2PocQXjQIjgnlCH5BFxQ8DFZzNZbGGEVW5XmBs8wz9o mpHNSUgqnqOVR53T0iy6oY4MiHFEY9FrcQWIWt5TPtDBojBQ6StcW+.eJH2N OUCNoevCa5L5wN2AaIgslM5CkmCZsAlp.mh5SJBrbka9tfAnbk9rT4SrnEyf WQakP5V.SyC0T7zklKyOh4h3nXQWq+jxsT7B69tiTUviF0I+sAtFv56G.GiX BBnUNm5UInFfXkwJmWMDVWCUaFi1Kd5CVQL09hsaQPh86w5QCC8VcAFndmpA HMGxCvNM6GrBauVvnlZK1CFFAoqrR.wWLxUhJbA76V8hBrGDrS8fJ3Yrw4NG QJLz71F3x75X+57s++Lr3Q8++IqNA8+Gbq2idVUixACzbqXvs+e9vgqL2mnL i2niuxhHEq57voob9j.L3jj7stbk3l4RDmSrt82ekwGUtDjtnExtm9uqsVGZ 2PwTR7vSWUxJ39iUba37BciYICQpBCtLCx3POQEN0ZcLMrS0jaMEl1xlXKs8 CxL195kgAF0S+LukvX9h+37+GvoJmvB -----------end_max5_patcher-----------
you’re going to be limited by 32-bit memory addressing for really long video buffers. For longer sequences, I highly recommend a disk-based approach. Check out the patch below, which writes and reads jit.matrix JXF files. The playback can be controlled the same as jit.matrixset, but it all resides on disk. Using an external FW800 drive, I’m able to get easy 30fps with this method.
----------begin_max5_patcher---------- 1239.3oc2YssjahCD8Y6uBUtxidcgZj3x9T1uisRkBajYXBFb.4Z7tox+9Jj fwDGyXIvRYx9fQk.Uxm9zs59HousbwpsUmYMqP+I5uQKV7skKVHeU6KVz0ew pCIm2UjzHG1pcUGNvJ4qVq9FmclKeeCqfsii1WUjxp6+59pRdYxAlbD+UcdR Q+WJOcHurfwkSJdvvax+W4vwvFutWeLgu6o7xrOWK9GTXkDDI9LBSHsMgAsO AuMdnOcY9qNw6+C5mo7TIRp197ef8W09tuubY6i0yz5eoNmyP7JTZdyWtu0e rl0HllDddU4.qxGjFBFKa75dLzllCmA9RxBGBRNCLmydnTVMKI05wIAwJa1K T1P8L2ngoXzGXMMIYrexn8LzhgIXwpXHeZaiOrgJ5DMpAiugWNrGJpww+miL 0buZE5S+VQFj.kCGfISFANhL1UT0vrNg.DYXQrujOhkQJlwGjGLe7bNeywWx KSqd4MWyeCiAikFCgJa.R3qo2FwdfaXO9iaOqmpMsunR7Ga8jaXpxMJiwuap saY73QMdgIjvaYfsIkYN2yF3ICM8ihmtmEhsfms0l1erI6Tt08tTPkAORZ7Q R2qefQLPjEXfR1Kh49mRe0RLGR304mQDztmRpEBL7PjHO6Wh2KRkbmpp6Ea9 5.X7j7B65yJ6Z5T1Xo7aU.g9.dMRtFy5BCTo9AUjTD07b+viVX.uJKqfYbxg .XflN3d48tkgLdQrbgR0G45hrDt8qp2yHJA9pmFxHz2EtVrZmb3P7jcs9++x 0hiCGvHSx0hevt1QnicUmJ4rZjmPJhoY8oSHj2WsiNYx+n6wKjavKduYjx5t hjqe7ANhcDuMujgDl0SHQwEAks44y6Qe7XRZpvLE8Ejn32GE0cxxX0MHvPBk L4xCT0YoDGYdYT7u4Jt.rZQl5rklhhKrMTbMl7gpirR6ugQufAaXLzbQC3P2 j9gi1hJr+Vt5RG2wGxCTvvEIiWoUICzwUbwJKYxEbwtSK0WOv30UnKaY2dUc UrR.8xSCYE3s8xO5sd8U9lr5jsNaSWcDDFHWIE2fEBXqtoq2EpTHfhXTWSwT TofeuqRwkZT.hRURnJOUzDznXCIJaOw4Ukll4kP5tBK+ol5Mxc4XZ0ZjJbjU Yxa3y92uWG4DJahAyczAuoi1JIg+0b9W8pVIdS97un+RN9K0cl9ArCtzCUhh XYgb0glalHVcuzC4LtpPj17p6TWhp12+i7TS0o5c8SU+0qft.rTVCOuTdMwC FTakyAC5o7zTV4vau7Pd5wJQ8fNP.Th7huvJsq.7ZuQ7m5h1q.xsQaqbMSPq KvT6YhfvtCSg1vqhoVxqBV.syDSTcfTjaCzn+PLzswD0oPRmjGTSycD1FW46 qToHpqz2alXs8NWt+5zXmxe5gI2FlA5j63JfeWLEnpH3ikEFCvgu1alnUGBD anSknBAArLCWGzk8lIXiz.rAtMmhVk+A2F.pSAUraSzA95Vj2gXRGeG3VdRq R4.0LwPcRMhka6HHJnuyLwpOV2XeraqweWL4YXIUOIkA.tsg3eo2bYv.cQK3 NFzSCLE6THEqAhHNcYZflbzTJZ5KaFzatNTcnuqDsX8Tud5J6vfnrfvta2Qp 2Hh38Zu4tLUKEctcYJP0ESS0qJ578k+2BgCAC -----------end_max5_patcher-----------
i just checked your patch and yep, that seems to be the way to go.
since i need to do the shuttling i think i’ll try it with ssd-drives for they have no seek time… i’ll post the outcome.
thank you so very much for your help again. i’ll sleep much better tonight!
i’m running Andrew Benson’s patch in a macbook pro and I’m only getting frame rates of 6 fps. any idea why this happens? does a an external FW800 drive works better?
I’m running Andrew Benson’s patch and I’m just wondering how I can change it to capture video in video format (mov,mpeg, etc.) instead of image sequence. Also, can I just add stop message to the jit.matrix to stop the recording instead of toggling the whole "write to disk"?
The patch I posted above is really specific to the needs of the OP (very long video buffers with the ability to have continuous recording and random access). Unless you find yourself in a similarly difficult situation, I’d recommend sticking to jit.matrixset or jit.qt.record for recording.