karl krach

i have had this problem for quite a while and i am really getting desperate since i already got complete new hardware and still no change... PLEASE PLEASE PLEASE, someone please help me!!! (yes, i AM desperate, sorry.)

i work on something like a video recorder: i write live camera input to a .jxf-file-image-sequence on a ssd drive - it's quite a large buffer and i can easily store about half an hour of footage (720p50 greyscale, that comes down to 95.000 .jxf files, each 922 KB).

at the same time i read out that sequence (at random points) and render it to a jit.window with some overlays too it, almost all rendered on the gpu side.

as the disk buffer fills and writes the files for the first time i get full 50 frames read and write (yeah!).

however, once the disk buffer is filled completely and max starts overwriting old .jxf-files, the speed drops to half, resulting in about 25 fps.

- the system is based on fast thunderbolt-ssds which are capable of writing/reading a tenfold of the required data rate and faster ssds actually do not make much of a difference. it does work however if i write to a ramdisk (with a much smaller buffer though).

- i use a single qmetro at 2 ms that feeds into the jit.qt.grab (@unique) followed by a trigger which sends the timing to jit.gl.render and all other parts of the patch.

- the total buffer size does not make any difference - as long as it is bigger then my 16 gb of ram, otherwise the system caches to ram and all works fine.

- i see in the activity monitor, that the data read actually doubles as soon as overwriting starts. but as said, it's not even close to maxing out these drives and cpu load is minimal. it seems like max is reading out every file which gets overwritten first and only then writes to it in a separate process thereby cutting the framerate in half.

if helpful i'll gladly send along a stripped down version of my patch.

again, i'm really fighting with this for ages and already tried zillions of things - if anyone can help me out, i'll get him a crate of beer of wine or whatever!

i work on something like a video recorder: i write live camera input to a .jxf-file-image-sequence on a ssd drive - it's quite a large buffer and i can easily store about half an hour of footage (720p50 greyscale, that comes down to 95.000 .jxf files, each 922 KB).
at the same time i read out that sequence (at random points) and render it to a jit.window with some overlays too it, almost all rendered on the gpu side.

the problem:
as the disk buffer fills and writes the files for the first time i get full 50 frames read and write (yeah!).
however, once the disk buffer is filled completely and max starts overwriting old .jxf-files, the speed drops to half, resulting in about 25 fps.
WHY?!?

some more background info:
- the system is based on fast thunderbolt-ssds which are capable of writing/reading a tenfold of the required data rate and faster ssds actually do not make much of a difference. it does work however if i write to a ramdisk (with a much smaller buffer though).
- i use a single qmetro at 2 ms that feeds into the jit.qt.grab (@unique) followed by a trigger which sends the timing to jit.gl.render and all other parts of the patch.
- the total buffer size does not make any difference - as long as it is bigger then my 16 gb of ram, otherwise the system caches to ram and all works fine.
- i see in the activity monitor, that the data read actually doubles as soon as overwriting starts. but as said, it's not even close to maxing out these drives and cpu load is minimal. it seems like max is reading out every file which gets overwritten first and only then writes to it in a separate process thereby cutting the framerate in half.

if helpful i'll gladly send along a stripped down version of my patch.
again, i'm really fighting with this for ages and already tried zillions of things - if anyone can help me out, i'll get him a crate of beer of wine or whatever!
karl


timing-of-jit-qt-grab-when-reading-and-writing-matrixes

one more thing: it seems to be related to the jit.qt.grab object too: if i use a 50 fps camera, i end up with 25 fps. if i use a 25 fps camera, i end up with 12,5.

one more thing: it seems to be related to the jit.qt.grab object too: if i use a 50 fps camera, i end up with 25 fps. if i use a 25 fps camera, i end up with 12,5.
best
k


i just spent another day trying to understand this and actually, the mac os activity monitor (disk tab) does show the problem: it's not the data rates but the "reads in/sec":

- the "writes out/sec" are as expected and stay at the speed coming from the metro (50 fps = 50 writes/sec).

- the "reads in/sec" however, go bonkers: if i only read and not write my .jxf. files, they are between 0 (i think this is when automatic macos ram caching is at work) to a maximum of 500. to my understanding it should be 50 (since i read at 50 fps) but regardless the patch works fine at a solid 50 fps.

- but when i write to disk (and even if i only write and do not read!), "reads in/sec" shoots up to up to 8500 - which is when my patch slows down.

- all this is exclusively caused by sending "write" to a jit.matrix 50 times/sec. if i gate the write-command temporarily, the "reads in/sec" in the activity monitor instantly drop to 0 (or the 500 tops of the reading process when enabled).

- as well i tried to first cache ten frames to a jit.matrixset and then writing this to disk so i would get less write operations and bigger files but that makes things only worse: same problem as above but overall lower performance and strong stutter at regular intervals.

come on guys, don't let me sit here alone in the dark, sobbing quietly.... someone has to know something about this!

is this a bug and i should file for support?!?

- as mentioned above, the problem only happens when overwriting .jxf-files but it does not always happen when doing so. sometimes the files get overwritten smoothly at 50fps and i do not have any "Reads in/sec", only the expected 50 "Writes out/sec". other times the reads shoot up to a couple of thousands as described above, slowing my patch down.

- i can't make it happen on purpose however: running the patch, it occurs for sure after a while but i can't track when overwriting happens smoothly and when not. it just starts randomly while running (me not touching anything): slows down for a couple of minutes, then speeds up again, then slows down again etc.

- i tried that on different mac systems and with max 5 and 6. the result is always the same.

- as well i tried to emulate my overwriting pattern with other programs but did not get anything close to this behaviour. it really seems to be max specific.

come on guys, don't let me sit here alone in the dark, sobbing quietly.... someone has to know something about this!
is this a bug and i should file for support?!?

some more observations:
- as mentioned above, the problem only happens when overwriting .jxf-files but it does not always happen when doing so. sometimes the files get overwritten smoothly at 50fps and i do not have any "Reads in/sec", only the expected 50 "Writes out/sec". other times the reads shoot up to a couple of thousands as described above, slowing my patch down.

doesn't sound like there's much else you can do besides lower you FPS.

if the problem only occurs when overwriting then why do you need to overwrite?

you're doing something that probably not a lot of people have tried before, and probably pushing the limits of your machine.

doesn't sound like there's much else you can do besides lower you FPS.
why 50 instead of 30?

you're doing something that probably not a lot of people have tried before, and probably pushing the limits of your machine.


hello rob, so many thanks for answering!!

the machine is pretty powerful and easily capable of doing what i need - if there would be the 50 reads/sec i request and not the 8000+ max seems to generate randomnly when doing a simple overwrite. as well i can easily capture and play back several high def video streams simultaneously with other programs and my patch only uses 720p grayscale.

if i run my patch with a thirty frame camera, i get 15 in the end, it's just all the same - when overwriting starts, max goes bonkers after some time.

do you have any idea or educated guess about what might be causing this? where these thousands of reads/sec come from?

hello rob, so many thanks for answering!! 

the machine is pretty powerful and easily capable of doing what i need - if there would be the 50 reads/sec i request and not the 8000+ max seems to generate randomnly when doing a simple overwrite. as well i can easily capture and play back several high def video streams simultaneously with other programs and my patch only uses 720p grayscale.
if i run my patch with a thirty frame camera, i get 15 in the end, it's just all the same - when overwriting starts, max goes bonkers after some time.

do you have any idea or educated guess about what might be causing this? where these thousands of reads/sec come from?


I've just read this and I see you mention having a SSD so a thought jumped to my mind.

I am not sure which ssd you have and how this one handles this exactly, but:

SSD's work in a way that you cannot simply overwrite single files. Instead all data in a certain block (the one including the file to be overwritten) has to be erased and rewritten with the new / now changed information. All depending on the situation.

Perhaps this is why you see a drop in FPS (and a spike in your disk I/O) the moment you start overwriting your frames.

I've just read this and I see you mention having a SSD so a thought jumped to my mind.
I am not sure which ssd you have and how this one handles this exactly, but:

SSD's work in a way that you cannot simply overwrite single files. Instead all data in a certain block (the one including the file to be overwritten) has to be erased and rewritten with the new / now changed information. All depending on the situation.
see: http://www.anandtech.com/show/2738/5

Perhaps this is why you see a drop in FPS (and a spike in your disk I/O) the moment you start overwriting your frames.


thanks for the link - it's quite an interesting read but definitely not related to my problem.

again: it's not the hardware, it's max. i can run similar in type but way more demanding tasks on this machine with no sweat whatsoever.

it is some weird thing going on inside max, or more specifically the write message sent to a jit.matrix generating all these "reads" at seemingly random times (because as outlined above, sometimes it does work as expected at least for some time).

thanks for the link - it's quite an interesting read but definitely not related to my problem.
again: it's not the hardware, it's max. i can run similar in type but way more demanding tasks on this machine with no sweat whatsoever.

it is some weird thing going on inside max, or more specifically the write message sent to a jit.matrix generating all these "reads" at seemingly random times (because as outlined above, sometimes it does work as expected at least for some time). 

I've got some experience with writing and reading files, for a perforamce called KKQQ ( 

http://2bcompany.ch/2bcompany/kkqqvideo.html

 ). I'had to record three different streams of video (WI-FI streams) and three streams of sound (one by actor) on SSD. The difficulty was each one played his partition at a different rate, the first one played in front of the camera at half the speed, the second one 0.66 the speed and the third at 0.75 the speed. When the performance began they were still playing in their boxes since 45min for the first, 30min for the second and 15min for the third. The performance began with the projection and they appear synchronized, technically the first plays at the rate of 2, the second at a rate of 1.66666 and the third at 1.33333. Voices are pitched even if they've tried to speak with lower voices (& slow motion gestures) at recording. So as a result at the beginnig of the performance I'm recording 3 video streams, 3 sound streams ( because they continue to record during the whole play until the projection catch the live action after 45min) and reading 3 video streams and 3 sound streams. I've recorded on little movie each 300 frames (something like 210 movie per actor) same for sound. Naming (001.mov -> 210.mov ), recording and writing was automatic after the 300 frames there was a switch to a second recorder ready to record. On the other hand for projection there were two players with selecting movies or sound, reading and playing in a similar automatic way with a switch betwen the players.

1. I've tried recording matrices but i've found that it was better to record little movies or chunks to avoid write and read operations (~ 2500 cumulated in a 90 minute perf between read and write are a lot i thought.)

2. I've forced the jit.qt.record to grab images to be synchronized with the sound with the little attached patch and these msp setting (48000khz with I/O vector size and signal vector size both set to 128 overdrive on). That give me one bang exactly every 40ms. In the other hand the projection was handled with a jit.qball after the little patch because forcing with the samee clock freeze Max and the little décalages or delays (glitch) were found funny by the audience.

Hi karl,
I've got some experience with writing and reading files, for a perforamce called KKQQ ( http://2bcompany.ch/2bcompany/kkqqvideo.html ). I'had to record three different streams of video (WI-FI streams) and three streams of sound (one by actor) on SSD. The difficulty was each one played his partition at a different rate, the first one played in front of the camera at half the speed, the second one 0.66 the speed and the third at 0.75 the speed. When the performance began they were still playing in their boxes since 45min for the first, 30min for the second and 15min for the third. The performance began with the projection and they appear synchronized, technically the first plays at the rate of 2, the second at a rate of 1.66666 and the third at 1.33333. Voices are pitched even if they've tried to speak with lower voices (& slow motion gestures) at recording. So as a result at the beginnig of the performance I'm recording 3 video streams, 3 sound streams ( because they continue to record during the whole play until the projection catch the live action after 45min) and reading 3 video streams and 3 sound streams. I've recorded on little movie each 300 frames (something like 210 movie per actor) same for sound. Naming (001.mov -> 210.mov ), recording and writing was automatic after the 300 frames there was a switch to a second recorder ready to record. On the other hand for projection there were two players with selecting movies or sound, reading and playing in a similar automatic way with a switch betwen the players. 

1. I've tried recording matrices but i've found that it was better to record little movies or chunks to avoid write and read operations (~ 2500 cumulated in a 90 minute perf between read and write are a lot i thought.)
2. I've forced the jit.qt.record to grab images to be synchronized with the sound with the little attached patch and these msp setting  (48000khz with I/O vector size and signal vector size both set to 128 overdrive on). That give me one bang exactly every 40ms. In the other hand the projection was handled with  a jit.qball after the little patch because forcing with the samee clock freeze Max and the little décalages or delays (glitch) were found funny by the audience. 

thank you very much for your detailed post! unfortunately using movies is no option for me since i record to a buffer - meaning i constantly overwrite the oldest recording - and since i can't overwrite only single bits of a mov file i can't use movs.

as well, if i would write eg. always ten frames into a movie file, i would get a whole lot of other problems since i am reading out individual images at random patterns - which would be a nightmare to cpu if all these were wrapped into individual quicktime movs.

working with image sequences is acutally pretty common, eg. in post production using dpx-sequences and since i do not deal with audio, keeping things sync is actually not a problem. i just need to keep it fast and smooth.

does noone have any idea where my initial problem comes from? like why all these "reads" are generated (and what the heck gets read there)???

all of you, thank you so much for your involvement!

hi filippo,
thank you very much for your detailed post! unfortunately using movies is no option for me since i record to a buffer - meaning i constantly overwrite the oldest recording - and since i can't overwrite only single bits of a mov file i can't use movs.
as well, if i would write eg. always ten frames into a movie file, i would get a whole lot of other problems since i am reading out individual images at random patterns - which would be a nightmare to cpu if all these were wrapped into individual quicktime movs.
working with image sequences is acutally pretty common, eg. in post production using dpx-sequences and since i do not deal with audio, keeping things sync is actually not a problem. i just need to keep it fast and smooth.

does noone have any idea where my initial problem comes from? like why all these "reads" are generated (and what the heck gets read there)???
all of you, thank you so much for your involvement!
k


I don't know how big your buffer has to be but I have had a good experience using a RAM disk (on mac os). You create the ram disk before opening max with a little free utility, then you can write to it from Max as if it is a normal disk. The write speeds are faster then a SSD and the size depends on how much RAM you have installed minus about 4 GB for the os and max.

Interesting, MrMaarten, how are you proceeding exactly, wich OS are you on, wich free utility are you using, are there precautions to take?

i've been running my patch with a ram disk already - that does work no problem. it's just to little space for what i need.

it is quite useful however, in macos you can just do it via terminal...

hi,
i've been running my patch with a ram disk already - that does work no problem. it's just to little space for what i need.
it is quite useful however, in macos you can just do it via terminal...


here is how you do it - i put it in a zipped text file since backticks are interpreted as text formatting tags in the forum...

if you use that a lot, make yourself an applescript to automate it. if you add this to your login items, you have it every time you start up.

here is how you do it - i put it in a zipped text file since backticks are interpreted as text formatting tags in the forum...
if you use that a lot, make yourself an applescript to automate it. if you add this to your login items, you have it every time you start up.


hi, the ram disk is deleted on restart (because it resides in ram). So anything you want to keep after the performance: back it up from the ram disk.

I've used it without issues in live performance on mac os 10.6.6. It seems to be very stable, and working across OS X versions: just tried it on 10.7.5 and it also seems to work fine.

I used this free utility: http://boredzo.org/make-ram-disk/

revisiting the ram disk approach, i did that again with the stuff i know by now and gathered some more observations:

when using an 8 gb ramdisk (4 x 2,2gb ramdisks striped), i can only buffer a very limited time but all works fine. since it is mounted as a "disk“, read/write access as well is shown in the activity monitor:

the behaviour is pretty much the same except that the "writes" are always about 200 (instead of the 50 with the ssd) - i think this is caused by the fact that i actually mount a 4-ramdisk stripeset (see post above). and again, when overwriting starts, the reads shoot up just the same, to about 11-12000/sec. the framerate in my patch however keeps steady at 50 fps so this seems where greedy max seems to hit the ceiling.

another thing i tried is to disable virtual memory completely (which is something i would not recommend at all - you can seriously fuck up your system with this sort of stuff) - however, regarding the behaviour of max and my patch that did not make any difference.

last but not least i tried to go in the opposite direction: i switched my patch to 1080p25 greyscale (that results in about 2,1 mb per jxf-file). in that case i have always about a 50 write-outs/sec (the double of my 25 fps) and if i switch it to 1080p50 i get 100 writes (the double of my 50 fps). again, until i start overwriting, my patch runs at a steady 25 or even 50 fps in full 1080p.

once overwriting starts, i get the same 7-8000 reads/sec and the patch bogs down to a mere 12 fps (regardless wether i run it at 25 or 50 fps).

mhm… so, in this setup, is one ssd maybe still just to slow and i could improve things by striping 2-3 ssds?

it still does not make sense though since as outlined above: if i do the same type of operation with other programs (as well simultaneously reading and writing high def video) all works blazingly fast, even if i use full colour 1080p and not the 1 plane greyscale 720p i use in my patch…

revisiting the ram disk approach, i did that again with the stuff i know by now and gathered some more observations:
when using an 8 gb ramdisk (4 x 2,2gb ramdisks striped), i can only buffer a very limited time but all works fine. since it is mounted as a "disk“, read/write access as well is shown in the activity monitor:
the behaviour is pretty much the same except that the "writes" are always about 200 (instead of the 50 with the ssd) - i think this is caused by the fact that i actually mount a 4-ramdisk stripeset (see post above). and again, when overwriting starts, the reads shoot up just the same, to about 11-12000/sec. the framerate in my patch however keeps steady at 50 fps so this seems where greedy max seems to hit the ceiling.

last but not least i tried to go in the opposite direction: i switched my patch to 1080p25 greyscale (that results in about 2,1 mb per jxf-file). in that case i have always about a 50 write-outs/sec (the double of my 25 fps) and if i switch it to 1080p50 i get 100 writes (the double of my 50 fps). again, until i start overwriting, my patch runs at a steady 25 or even 50 fps in full 1080p.
once overwriting starts, i get the same 7-8000 reads/sec and the patch bogs down to a mere 12 fps (regardless wether i run it at 25 or 50 fps).

mhm… so, in this setup, is one ssd maybe still just to slow and i could improve things by striping 2-3 ssds?
it still does not make sense though since as outlined above: if i do the same type of operation with other programs (as well simultaneously reading and writing high def video) all works blazingly fast, even if i use full colour 1080p and not the 1 plane greyscale 720p i use in my patch…


or: can anyone enlighten me on what these "reads in/sec" and "writes in/sec" i see in the activity monitor exactly mean? and how and why these are generated by max in these excessive numbers?

my office partner (who knows nothing of max) just gave me this brilliant idea and i still have to do some more testing and debugging but it seems to work fine if i run a separate deletion pass before writing the jxf-files: i just delete (via remove external) all my jxf files ten frames ahead of writing new ones thereby avoiding the jit.matrix to overwrite already existing files.

i am still fighting with occasional (very short) drops in speed but i do not get these 8000+ reads any more and my patch seems to run overally smooth up to full 1080p greyscale at 40 fps - regardless of overwriting or not.

i will post again when having examined this closer... i really hope i have not cheered too early.

for the time being thank all of you so much for your support!

my office partner (who knows nothing of max) just gave me this brilliant idea and i still have to do some more testing and debugging but it seems to work fine if i run a separate deletion pass before writing the jxf-files: i just delete (via remove external) all my jxf files ten frames ahead of writing new ones thereby avoiding the jit.matrix to overwrite already existing files.
i am still fighting with occasional (very short) drops in speed but i do not get these 8000+ reads any more and my patch seems to run overally smooth up to full 1080p greyscale at 40 fps - regardless of overwriting or not.
i will post again when having examined this closer... i really hope i have not cheered too early.
for the time being thank all of you so much for your support!
k


Karl - your solution suggests that it's the block-erase behaviour of the SSD that is the culprit. What is the current status of TRIM support on OS X?

What is the capacity of the SSD, and what controller does it use (Sandforce?)?

Also - depending on how file-caching works in your OS - RAMdisks can be problematic, to say the least. In Windows (for instance) it forces the OS to keep three copies of the same data in RAM.

It might be worth investigating how RAMdisk usage in OS X affects hard page-faults generated by other processes.

on macos, trimming is only enabled for internal drives by default. there are apps however, which can enable trimming on external drives too but in my case that did not make any difference.

as well, the same behaviour happened with conventional harddrives so i do not think trimming is the problem.

i do not use any ramdisks currently either.

about the ssd: to be frank, i do not now the controller, it's an external thunderbolt lacie 128 gb.

regardless, the solution outlined above works well and really did solve my problem.

only seen this now...
on macos, trimming is only enabled for internal drives by default. there are apps however, which can enable trimming on external drives too but in my case that did not make any difference.
as well, the same behaviour happened with conventional harddrives so i do not think trimming is the problem.
i do not use any ramdisks currently either.
about the ssd: to be frank, i do not now the controller, it's an external thunderbolt lacie 128 gb.

regardless, the solution outlined above works well and really did solve my problem.


timing of jit.qt.grab when reading and writing matrixes