karl krach

i am looking for some advice on high speed data compression of matrix data (video):

in my setup, i write and read video as single frames to a harddrive, currently as uncompressed uyvy binary .jxf data - which is fast but requires really fast disks as well. which is why i am now looking at compression.

- what sort of compression would be needed in my case? i tried with jpg/png/etc. but all this is way too slow. a friend recommended having a look at snappy/lzf/lzo - would that make sense? i found an implementation of lzo (sadam external) but did not get that to really compress anything...

- how would i read and write the then compressed data to disk? since matrices only support int or float i guess i would need something else to do that. something that would just dump a list into a file...?

hello,
i am looking for some advice on high speed data compression of matrix data (video):
in my setup, i write and read video as single frames to a harddrive, currently as uncompressed uyvy binary .jxf data - which is fast but requires really fast disks as well. which is why i am now looking at compression.

as always, thanks a lot for any help!!
k


high-speed-compression-of-matrix-data

the uyvy data you reference is likely in a (jit.matrix 4 char width height). Correct? Whatever is outputting your video stream defines this. You can also check it with a jit.fpsgui

There are built in objects like jit.qt.record which can receive your matrix and write it to the hardhard drive in a compressed format. See help for details. Though...caution...Apple continues down the path to kill Quicktime.

Another options is a 3rd party external which can do the compression. One is 

 and its supports A LOT of compression formats; some formats which are specialized to compress YUV data (rather than RGBA).

Another options is a 3rd party external which can do the compression. One is http://ben.musicsmiths.us/vipr.phtml and its supports A LOT of compression formats; some formats which are specialized to compress YUV data (rather than RGBA).


yes, my matrixdata is a 4 char jit.matrix (but encoded 4:2:2 uyvy not argb so it's only half the size of the actual video resolution).

i tried with quicktime but due to the quicktime overhead the performance is just way too slow.

as well i already checked ffmpeg/vipr but unfortunately it won't help me much either since i have to use single frame file sequences (so it needs to be an intraframe codec) which i write to disk (afaik vipr can only stream).

this is why currently i am looking for some sort of really lightweight and fast (binary) compression such as lzf - a ratio of 2:1 would already be largely sufficient.

if this helps explaining, i attached a patch which shows the basic workings of my patch.

ps.: by the way, i know your dp.kinect/jit.openni, awesome!

hey diablodale!
thank you very much for your reply!
yes, my matrixdata is a 4 char jit.matrix (but encoded 4:2:2 uyvy not argb so it's only half the size of the actual video resolution).
i tried with quicktime but due to the quicktime overhead the performance is just way too slow.
as well i already checked ffmpeg/vipr but unfortunately it won't help me much either since i have to use single frame file sequences (so it needs to be an intraframe codec) which i write to disk (afaik vipr can only stream).

this is why currently i am looking for some sort of really lightweight and fast (binary) compression such as lzf - a ratio of 2:1 would already be largely sufficient.
if this helps explaining, i attached a patch which shows the basic workings of my patch.

ps.: by the way, i know your dp.kinect/jit.openni, awesome!


I see Deutsch in your patch! :-) I'm an Ami living in Berlin.

One possibility is to keep everything in memory; not write to disk unless you need it. I have 16gb mem in my Windows laptop. If you resolution is 640x480 with uyvy then 20,000 frames is about 11.4 GB of data. This would fit entirely in memory and be super fast. I tried and its worked for me. I did have an issue when I changed it to 1920x1080 res, after a long delay, Windows and Max both complained about low memory and lead to a forced crash. STill...have you considered this approach? I used jit.matrixset.

Have you tested the exportimage message of jit.matrix? It does support some lossless and lossy codecs.

Because you are choosing the frames randomly, the caching abilities of virtual memory or individual files on disk will be less successful. Therefore, using an SSD can help in both cases to maintain a high frame rate like 50-60fps; or some hardware caching that's large enough to cache all your compressed frames.

FYI, the jit.qt.movie object can output uyvy directly; saving the mem/processing time of a separate jit.argb2uyvy.

I see Deutsch in your patch! :-)  I'm an Ami living in Berlin.

FYI, the jit.qt.movie object can output uyvy directly; saving the mem/processing time of a separate jit.argb2uyvy.


Also some very relevant older posts you can see at 

https://cycling74.com/search-results/?q=jit.matrixset%20large%20memory#gsc.tab=0&gsc.q=jit.matrixset%20large%20memory&gsc.page=1

Also some very relevant older posts you can see at https://cycling74.com/search-results/?q=jit.matrixset%20large%20memory#gsc.tab=0&gsc.q=jit.matrixset%20large%20memory&gsc.page=1


i'm in berlin too! just wondering: fancy meeting for lunch one of these days? could be fun!

apart: your suggestions are good but i already tried literally all of these solutions you outlined - which is how i ended up with my current setup (wich gives me 1280x720 at a solid 30 fps running on a samsung 256gb 840 pro ssd...). any sort of export to a proper image format as eg. jpg (from jit.qt objects as well as directly from jit.matrix) was always wayyy slower.

i keep wondering: what about lzf/lzo? i'm no programmer but what i read is, that performance with this sort of compression is actually faster then just saving the uncompressed data so i could trade some harddrive- for cpu-load (which i still have ample)... or does lzf/lzo not work with video data in the first place?

hey!
i'm in berlin too! just wondering: fancy meeting for lunch one of these days? could be fun!

Sure, I'm up for lunch. Drop me a message using dale theroundatsymbol hidale.com

You ask questions about lzf, lzo (also caould be snappy, lz4, or any other lempel-ziv related one) may be interesting academic inquires but until you have C code running and can test it...I don't think it helps you. Any lossless compression format (like lz4, zlib, lzf, etc.) will work with any data input. However, it might not perform as well...or perform better. It needs to be tested.

The main issue is that you write you have tested all native Max objects and known 3rd party externals and none of them meet your needs. If you know how to write Max externals, then I recommend you implement these other algorithms and test them. Or find/hire someone that knows how to write them.

FYI, PNG which is built into jit.matrix uses deflate compression which is what zlib uses. Some articles like... 

http://blog.erdemagaoglu.com/post/4605524309/lzo-vs-snappy-vs-lzf-vs-zlib-a-comparison-of

FYI, PNG which is built into jit.matrix uses deflate compression which is what zlib uses. Some articles like... http://blog.erdemagaoglu.com/post/4605524309/lzo-vs-snappy-vs-lzf-vs-zlib-a-comparison-of


i did not test all and everything but certainly A LOT - actually i just tried the exportimage/png approach again, just in case i missed that one but this only seems to use one core which gives me about 8 fps on my system (compared to about 30 with uncompressed jxf).

anyway: i am certainly no programmer so i guess i'll forget about lzf/etc. and just get some ssd-raid or really use ram (which works but it would require like 128 gb+).

in any case: thank you so much for answering all my questions!!!

hello again,
thanks for claryfying!
i did not test all and everything but certainly A LOT - actually i just tried the exportimage/png approach again, just in case i missed that one but this only seems to use one core which gives me about 8 fps on my system (compared to about 30 with uncompressed jxf).

anyway: i am certainly no programmer so i guess i'll forget about lzf/etc. and just get some ssd-raid or really use ram (which works but it would require like 128 gb+).
in any case: thank you so much for answering all my questions!!!

high speed compression of matrix data?