karl krach

i know it's sorta lengthy and complicated but... can one of you masters of max please give me some idea of what is happening here? because, to be frank, i am completely puzzled. i do not even know wether it's actually max or the system or just the gods of whatever who are playing around with me...

the short version and two main questions:

1. evidently, max has to read a .jxf-file completely before it can overwrite it. why? can this be switched off somehow?

2. the results below indicate that some automatic RAM buffering is going on in the background... how does this work and how can it be controlled?

i build some sort of surveillance-system-like installation where a camera films a room and this data is fed into a matrix (greyscale, 1 char) and then saved as .jxf data to a ssd drive.

while this image buffer is continously refreshed, one can shuttle around in it and play forward and backward - which means that the system has to continously (over-)write AND read the .jxf-files at 25 fps.

since obviously the data storage is the bottleneck i did quite some testing (imac 3,4 ghz i7, 256 gb ssd, 16 gb ram, using its built in camera). the size of the image buffer was set to 90.000 frames which equals one hour of video at 25 fps.

- if i use a ram disk all is - as expected - super fast and nice (smaller buffer though since macs only allow ramdisks up to 8 gb).

- if i use the ssd and run an image resolution of 1280x720 (= 926 KB/.jxf file) it starts up at 50 fps but as soon as the buffer is filled once and files start to get overwritten the performance drops to 12-20 fps.

simultaneously i notice in the activity monitor, that the disk data read/s (which up to now, was equal to the data written) doubles - it seems that max has to read every file completely before it can overwrite it.

- if i switch the whole system (matrixes, jit.qt.grab etc.) to a resolution of 1024x576 (= 594 KB/.jxf), on startup, the activity monitor indicates zero data read/s (although i still continously shuttle around and do see the images) and the system is blazingly fast.

once the buffer is filled and overwriting starts, the disk read data rate bumps up and then equals the writing data rate again - everything is still working fast and nicely.

however, if the installation runs for longer time (so the files get overwritten more often) this setup starts to behave pretty much like the one above, only in this case some data still seems to be automatically buffered, resulting in the data read/s jumping between the single and double amount of the data written/s and the framerate oscillating between 14 and 25 fps.

wel... can anyone please enlighten me? if this is of any help, i would be lucky to post my patch in a stripped down version.

with some breaks, i spent over a year on this project already and it does freak me quite out that i still do not have any idea what's happening here...

and thank you already for reading so far! :)

hi,
i know it's sorta lengthy and complicated but... can one of you masters of max please give me some idea of what is happening here? because, to be frank, i am completely puzzled. i do not even know wether it's actually max or the system or just the gods of whatever who are playing around with me...

the short version and two main questions:
1. evidently, max has to read a .jxf-file completely before it can overwrite it. why? can this be switched off somehow?

the lengthy background:
i build some sort of surveillance-system-like installation where a camera films a room and this data is fed into a matrix (greyscale, 1 char) and then saved as .jxf data to a ssd drive.
while this image buffer is continously refreshed, one can shuttle around in it and play forward and backward - which means that the system has to continously (over-)write AND read the .jxf-files at 25 fps.

the results:
- if i use a ram disk all is - as expected - super fast and nice (smaller buffer though since macs only allow ramdisks up to 8 gb).

- if i use the ssd and run an image resolution of 1280x720 (= 926 KB/.jxf file) it starts up at 50 fps but as soon as the buffer is filled once and files start to get overwritten the performance drops to 12-20 fps.
simultaneously i notice in the activity monitor, that the disk data read/s (which up to now, was equal to the data written) doubles - it seems that max has to read every file completely before it can overwrite it.

- if i switch the whole system (matrixes, jit.qt.grab etc.) to a resolution of 1024x576 (= 594 KB/.jxf), on startup, the activity monitor indicates zero data read/s (although i still continously shuttle around and do see the images) and the system is blazingly fast.
once the buffer is filled and overwriting starts, the disk read data rate bumps up and then equals the writing data rate again - everything is still working fast and nicely.
however, if the installation runs for longer time (so the files get overwritten more often) this setup starts to behave pretty much like the one above, only in this case some data still seems to be automatically buffered, resulting in the data read/s jumping between the single and double amount of the data written/s and the framerate oscillating between 14 and 25 fps.

wel... can anyone please enlighten me? if this is of any help, i would be lucky to post my patch in a stripped down version.
with some breaks, i spent over a year on this project already and it does freak me quite out that i still do not have any idea what's happening here...

and thank you already for reading so far! :)
karl


jxf-readwrite-behaviour-ram-caching-and-complete-bewilderment

i just noticed this thread got somehow deleted... it used to be pretty long, for reference the result: there is a bug in jit.matrix when it needs to overwrite jxf-files. if you should run into this problem: run a separate deletion pass first (via external "remove"), do not let jit.matrix overwrite anything.

i just noticed this thread got somehow deleted... it used to be pretty long, for reference the result: there is a bug in jit.matrix when it needs to overwrite jxf-files. if you should run into this problem: run a separate deletion pass first (via external "remove"), do not let jit.matrix overwrite anything.
k


.jxf-read/write behaviour, ram caching and complete bewilderment