high speed compression of matrix data?
i am looking for some advice on high speed data compression of matrix data (video):
in my setup, i write and read video as single frames to a harddrive, currently as uncompressed uyvy binary .jxf data – which is fast but requires really fast disks as well. which is why i am now looking at compression.
- what sort of compression would be needed in my case? i tried with jpg/png/etc. but all this is way too slow. a friend recommended having a look at snappy/lzf/lzo – would that make sense? i found an implementation of lzo (sadam external) but did not get that to really compress anything…
- how would i read and write the then compressed data to disk? since matrices only support int or float i guess i would need something else to do that. something that would just dump a list into a file…?
as always, thanks a lot for any help!!
bump… : )
the uyvy data you reference is likely in a (jit.matrix 4 char width height). Correct? Whatever is outputting your video stream defines this. You can also check it with a jit.fpsgui
There are built in objects like jit.qt.record which can receive your matrix and write it to the hardhard drive in a compressed format. See help for details. Though…caution…Apple continues down the path to kill Quicktime.
Another options is a 3rd party external which can do the compression. One is http://ben.musicsmiths.us/vipr.phtml and its supports A LOT of compression formats; some formats which are specialized to compress YUV data (rather than RGBA).
thank you very much for your reply!
yes, my matrixdata is a 4 char jit.matrix (but encoded 4:2:2 uyvy not argb so it’s only half the size of the actual video resolution).
i tried with quicktime but due to the quicktime overhead the performance is just way too slow.
as well i already checked ffmpeg/vipr but unfortunately it won’t help me much either since i have to use single frame file sequences (so it needs to be an intraframe codec) which i write to disk (afaik vipr can only stream).
this is why currently i am looking for some sort of really lightweight and fast (binary) compression such as lzf – a ratio of 2:1 would already be largely sufficient.
if this helps explaining, i attached a patch which shows the basic workings of my patch.
again, thanks for any help!
ps.: by the way, i know your dp.kinect/jit.openni, awesome!
----------begin_max5_patcher---------- 2968.3oc4bs0aaibE9Y6eECD1GRP0xv4Bukmh6FklTjVWXazs.MEAThijXhH oBuXYkE6+8dlYHkocDonTrFZmfEqCuNyY9ly8yg5ON8jAiStgmM.8Rz+EcxI +wombh7RhKbR44mLHx+lIK7yjO1fIIQQ737ACU2KmeSt75Ab3NKS4YYgIwnr 4IEKBPy8WtjGilyS4UuvhvX3IKhkuEo7hSShyyB+JWbMLwvr5xoyFKtj4lqD WDEFufmKoEb4EW5mOYdX7rOlxmjqVKddTCqgHlikgqmmmMcHBSsfgAQoFln+ W4aFFHo8jwe5Ww1CpQLw9QRhYvYog9KFb6rmTjWM8l0ffIIKRRUSMVLKla9C VLchG7OO8TweF9chyOlPYFwyvQ.ytLCOSWho.lsLaElwOMf4Tte.JLxeFevw B8b7LrcXdtt.0yj7lNth+RL2NzYYs2P2A.Hw7Uvj8M3wmByM7gkJoX80q2Aj nnf70K4pEp3k+XjedZ3MCFhFLXyhqqHE1BafKgJhYGfJh69BUjC.ph.wvZLH 2g2YHZre7rc.Tim8sLzVlXOabct5lP0FvQR2vwRVNaINhca.GI6KNheXY49R tQTx0gbzq7C7Wliv8IqG1QBYXBsMdOGcv50jZqkK7WO1exmQoEwYHwQq7SCx P4IH+qSBCPYqyx4QnT+HzDe4hUyFls7jlG7bjPocq1j8v5PiWSXYdHHfiFWL cJOEsJMLOGLxJL+BSFewZjls0pfJaFS3XCFSaC3bHZwTQQzXd59JQFJfZz.o 5w8VbrBEHRTP4pQSBh1r8VRbyDCxGvlb5G4w9iWvefsklsbQXNxDgAOS8NP3 S7OaG8ns3wA1QI7QMnkdE630pKGj9TY1u+1ytBc0ae2kn+0YW8auE85yGc4G h+Pb97vLjb4gfC7GCSOZBXzMGVsHezBvOENZQ30bz0gA7jRQXCD.5vyKjjEO X9b3g7yxkKdSSzTwddFJYp7NSfSR8QgwKgA+YIovQH475ONKO0eRtv+aeTFL RK3JGEetTKKZZxh.PgQg3VU2WN1nL9WJ3wSfIAFLiOcyzecZRJXbBsZdH7PY gQEKx8i4IEYf5EgWDhgPpNGHoLXXfGS9x9nnjTNBHK3BYf173fjHzT+r4.YY 7g3spZhczTMIccAyj9k4HUL4v1NCEi0mZzE3GZRQZJbyRDdvQNBTKamMxZXP 7yChShY0rDG04GTM2U3AyqttmVPBlYeq9tirQktFL3Ha42xUIl0ElH7O1l+s rUJYZk+A67Xk+oz9UpzbTRLvBAZoEwRltFETvUlQFWLSYygKYv3nw749WGlT jVYhRDejJzFDrdAL5kvfMgKuWPX1mqbcEljoEKVf.iDHoQGoUjrbAA.S0LdN J4ZdZIa7P46KMXk5CS6pP3UCRSVBt9FmA1SSAHYMJKIIFFagEH3o.iqeHddx JNLNCQgR5asbjC3BukCTFsx3BnEFzkoIhEL5SEf82w7oBiY0VN20JcDP47UC QhAectfi.sJI8yvpBrtYXXfdFLgqSJPq7ikqnLNWYqFFWwMRy3KlplqrUgB2 FDKw7jYBSyISmNDdSv2.wSe20NLVxUt.4f6DiltHbovMBoM4j3marUCsX5wR UfvuWA+OXqsRIJwjohfG2Pn.8ZLTTCzkWc1EWUmFl5OgWec9viRdlBECd1xj SRrsaMJAq9DetXzYuF82+OuA8l289QWdzLg35QjXfCdCuRKN860mHxqG89QW MRgGn+5n2b9EiPm+uGcwuew6tZzfimfUc.hPZMvRK29DfTZTqzZVo4l+g3mI MkH0qmorDbWC.a7T44aUq0QSokiI8VOVJy6CE9GSOrkrFMMfxZITAkUfAs4s xd6khMUkiK7FwrV3kraHOp5IUDy.yb6qGb6edmssu0m0RTgxZEUbzQdmaLuC BUMZPqL01otVYOqVgjd0NNAriCJl+sqPu472+5QWft5bjBlfCd14+SzkW95m qUi7kf2Fi7sxOw5UM1XCz6OGry+t+wY+sQf2pwY4hpKBtwWOKS5E9T4roB8r ZMKfLmdtvikgEzW0.RgUjxzbga04ILySGoLsAvJkGAND7.oPe2dXWut+VsBK r81K6pYuVERMMbLg4xSTbTGKSJlINxFhFRTvzCsC.Z.JySxVGMNYgF.SUPc2 ALaWdz9oFXNIIVjpaXwOGE6mKRM+3jj7tWAMZK.9vC0MMEtSAGPs87HX.Jbc M7bw1tvgT6lJIBsGEv+K6u1ulwFxtqTjMcC1PIsksMpo4CCSoksmir8GbMIt NNkLk1Vzi.SYzXfwC8hqSVTDwydw3Eg4e8EnuxkUmSTfFzqV5GHqBiI7eVnW AFQlMimlgHGRLTak2sgMG1tUX.wrVs4fUZLZJVJLUKckvwHXpxTWTutAsGQE 05mfHp7.4y6UMk1inh5piHpNFL.Np9Ij4s6foojeJ15qgG6XSm0mgQmHZJ0x pgG.yXxrBM2epT.qXk1ywah3ostSk1mwIppyinnCgYe93UNQlp0gvp9njzJC TeBGxVvcZZRjpZVe2I3730SaDGChH0VlpFCTNnMxiokVyRD07xUgwAIqFT2M qnj.07UuMd2G2R1sh5xhKvLM.cTdN1kcd+VYunOb9dzGq3JeSbusGO5z5l3c zCTnVZSXnIy8S6srmTAR0pYIl4zZ2aaenUt+aBkvlTFJgCqLTBaSOxgGJQas 7M5Wvcpque.7AnDSYt21HetVs1H2NOUc+yyztyESgv9Yv+uR.oVZKZ2KPh0S 08dpqJCMXVG16o+Dr2yLcqAH6XaG2ie6FRmt.Oyvems4b0+2L6i0tkTbT0b1 rciN6c9qXO7ohRlApW1ZJnL2YRnHZIITUlfnt2ZVW45WiAW0mUFIGMFs24xW YL+Pb5oRyUckzsl+Trk9SP222me1gkycUgiJCFmP1Dd9lC2dQ05Uc6eAF6zD D4fXe1+lknDhbJ652Vq1H4QfN9dTCeY+.Wpg2qUE73mHJ3ebTigx1qfY5caM FH31pwfVztOtHOOIdvAHws6VIYOUJ41tNoocR45CGGlHaqprrJy45fCfK46C 8bkNa40Z9Ds6QO.d7jQDwG+pn3MDOEaWqpsrdxkKDUZr+E7wObnJkT1rMJor nskFDsTOjk9w7Ea0GKSCuMZ.91ip6sUmStlqJnXakykNpBC60zWeGo8ehMDe VCyRACMAOEV61pJXTs1Ylst1so+Hs1oLK4htZs6XvJ6u1FA.GyCE.jilzIj6 8KEjjvDW+tnRVRQ5jJo5RWCP2RSA7r7PQW.AFxu8YX24YlGFDviqSdAgYBW6 CZ129tRMtcfZr0F03zApg3pMxwtCji2iJpQ1PVsROQgAKSf3KJ4doN1RgFhi nRMfMvgHaJ8tW514TqqEr2AtTj59.EJaN6Hr.r5BqprOvzD2gUmXObzG8bON wF1io5jh5LAQzBAw5xVlnYg0D.w5BGj1nFZWLMQzm5WpUWj30mwIZW1tnT8Q OcQ7hpOOatux2lLXpMBhzEFZ4mLgtHnN4tk979iX2UILrdnmtHwSzmIdRWj3 I5SCcmrvKp7ptnmtnARe9abuUdCx6Zb6pK5CYlOtnGYP.5wcLbWhLmnO9GIq A9Qi+XXqtp8AuGwro9gPhpxUG0prBV0OyQ8SULFqxo08Nkp97xwlk0Xdqm5Q kUQTd1QHVvNgLX8IXceeHZwUL7iHBRpubO3crJ2TUcAjk5az5tmYp9PfTcaf 7riw1emLKum4kwwTUWLUeDaQ1xYDU4v25YT0qWIdYZ1eqbr97v99YSnASb5y iVL4Awk+syaT9icjc02Ga8yvDYAHUhHxSNFa+lcL+nZx9sfbn6JW1r8Cqsrj 5Nr7JK2OayY8EfJygG9wChZoQwotPODldomcJdendLppzj+xkWySyJGSIoLH x+SppoYOTdZXr5TYMxFjxuNr54UOfe5j4g47I4Eoph7diq8fSEyyed5+GL8Z CuA -----------end_max5_patcher-----------
I see Deutsch in your patch! :-) I’m an Ami living in Berlin.
One possibility is to keep everything in memory; not write to disk unless you need it. I have 16gb mem in my Windows laptop. If you resolution is 640×480 with uyvy then 20,000 frames is about 11.4 GB of data. This would fit entirely in memory and be super fast. I tried and its worked for me. I did have an issue when I changed it to 1920×1080 res, after a long delay, Windows and Max both complained about low memory and lead to a forced crash. STill…have you considered this approach? I used jit.matrixset.
Have you tested the exportimage message of jit.matrix? It does support some lossless and lossy codecs.
Because you are choosing the frames randomly, the caching abilities of virtual memory or individual files on disk will be less successful. Therefore, using an SSD can help in both cases to maintain a high frame rate like 50-60fps; or some hardware caching that’s large enough to cache all your compressed frames.
FYI, the jit.qt.movie object can output uyvy directly; saving the mem/processing time of a separate jit.argb2uyvy.
Also some very relevant older posts you can see at http://cycling74.com/search-results/?q=jit.matrixset%20large%20memory#gsc.tab=0&gsc.q=jit.matrixset%20large%20memory&gsc.page=1
i’m in berlin too! just wondering: fancy meeting for lunch one of these days? could be fun!
apart: your suggestions are good but i already tried literally all of these solutions you outlined – which is how i ended up with my current setup (wich gives me 1280×720 at a solid 30 fps running on a samsung 256gb 840 pro ssd…). any sort of export to a proper image format as eg. jpg (from jit.qt objects as well as directly from jit.matrix) was always wayyy slower.
i keep wondering: what about lzf/lzo? i’m no programmer but what i read is, that performance with this sort of compression is actually faster then just saving the uncompressed data so i could trade some harddrive- for cpu-load (which i still have ample)… or does lzf/lzo not work with video data in the first place?
Sure, I’m up for lunch. Drop me a message using dale theroundatsymbol hidale.com
You ask questions about lzf, lzo (also caould be snappy, lz4, or any other lempel-ziv related one) may be interesting academic inquires but until you have C code running and can test it…I don’t think it helps you. Any lossless compression format (like lz4, zlib, lzf, etc.) will work with any data input. However, it might not perform as well…or perform better. It needs to be tested.
The main issue is that you write you have tested all native Max objects and known 3rd party externals and none of them meet your needs. If you know how to write Max externals, then I recommend you implement these other algorithms and test them. Or find/hire someone that knows how to write them.
FYI, PNG which is built into jit.matrix uses deflate compression which is what zlib uses. Some articles like… http://blog.erdemagaoglu.com/post/4605524309/lzo-vs-snappy-vs-lzf-vs-zlib-a-comparison-of
thanks for claryfying!
i did not test all and everything but certainly A LOT – actually i just tried the exportimage/png approach again, just in case i missed that one but this only seems to use one core which gives me about 8 fps on my system (compared to about 30 with uncompressed jxf).
anyway: i am certainly no programmer so i guess i’ll forget about lzf/etc. and just get some ssd-raid or really use ram (which works but it would require like 128 gb+).
in any case: thank you so much for answering all my questions!!!