Optical soundtrack reader
I’m trying to recreate an optical soundtrack reader in Jitter. I’ve used max for quite a while but only just starting using jitter, but essentially I want to convert video/images to sound by recreating the technique used in optical soundtrack reading.
In theory I should be able to do this by splitting each frame into 1764 rows( at 44100 sample rate) and by taking the average brightness of each of these rows and transposing this to 1 to -1 an audio signal can be output.
So this reading of 1764 rows has to occur within 40ms, added to a buffer and output.
To do this Im looking for advice on how to:
- Split video into a grid of 1764 rows, I’m using rgb2luma so all videos will only have 1 plane. I was thinking I could convert all videos to have a height resolution of 1764 and then read through each row, but this could be really processor heavy and I’m not sure how to do this?
- Read row of cells as one to obtain average brightness easily, rather than having to cycle through and add each cell individually, as this would not work as quick as required.
- any other objects, techniques or tutorials that could be useful for me to look at.
I’m not sure if you’ve already looked at jit.peek~ – that might be what you’re looking for, or maybe you can work from that?
Other objects that might help if you do need to unroll the matrix – jit.spill and jit.iter … but hopefully that wouldn’t be necessary.
Other objects that might be randomly useful: jit.catch~ jit.graph, jit.poke~, jit.buffer…
Hope that helps,
Something like this?? Should be a good start… This is from the always amazing Jitter Recipes Books (Book 1, recipe 11):
----------begin_max5_patcher---------- 1190.3oc2YszjahCD9r8uBUT6QudQh24zj6YqZqZOt0VSICJ1ZV.4HjGONoR 9su5ADic.ElI.dR7A.IKZ08m59SsZ9zxENaXOQpb.uA7OfEK9zxEKzco5XQc 6ENE3mRywU5g4jxJJHkBmUl+SPdRn6+uxwm.Gohc.wNBnfH3LP0dBICHX.b1 CGpDxmJI+Ae2Iwthl2u7PAsLmHzBGV2464a2nZ6t1soGVonDWPzS0a4TbdKA vNHZjPy32iEo6nkaumSREFyKvWJM.xOQeyyScCFs1E7u0uCMSKc1lG9cnSq4 sh9Q87JeQUmed4R0kU+f306X3LvI1ANnf8HkL+3gOTCDgpqQI1PCzziF3xLN 43l6ROklKUyH+0xQN+XhmuRbfPiyQXjMTwa7PkBRUEdK4aPkJAlK5DEPOSaF 1uMCQZa1X40gEAcax9MB2HYwo8DiLbb95vmRzfse5AC8ZdfgjHwFXDLKfQuz szz+yPeTkhKkwL2.FDuHM4ADgT2hsxgDNdQKR8aCg2HucJHIkky3Fkxccbja Pb7pNeB1R65FnFHrz3ysY6kycn7WzpNep8b2O2CDpwQu.anYTu9dTkuBvYCt bqSqYiKMIAgeOoDuImzdc8p0iF+LNU5WUOx0IiYTLOMiVnY1.tfeCtBX5gTl A7fIxdl5X7Ff1qld22VTd7rDk+.Urd+QZYF6n0v3t8A6hGq1phglcxzQnAtc aiI8aiqz14qCKRGQ341xffHXO4u4NAlD4iY3zuX06bvTmZWODJ97dtpqcZJP mWBGI4n7k+lfO0pBmjSvUju.tqfkQ.PmwjNrOSEFa7GcCsl1Mp2ksJ51R0ja Itqt2V7wZmeIAbhKLIT8ja8SZ2lQFV2tAken.OOvIT64DZGM85EMkJ78EXII +SVQzwEgLSH.BdeNCK7P.OjK3tCUDyF.x+3tK2b.p9+K2cP107.vA5ad07m8 gv92.DVhdRa4WtzeLHc8oPrl7Cr+Lu0dVuty+giEjYHIGCbFZNZGxVFNvva5 o53Db1TCFAnyfgYO2dAinYALlzXXYWzBoz0ttA27.a+gGXG+yPfsfscaN4Yj TK76rWS8wQBr5WlX8De1AjwaabSYcQSc7ZTKtKeqUlr+77uzQYRSr4CxbaT0 vUmPCxedxPIwjmhq05Xif+nInLCIU2WIt96Tb4ednhldypCrGrUcf66.ZnNK ONx+BuL8K5nJS2UerE8jqKe2E.TE6.OsYopg..bd9yHUBZIVPYksFjJbn0f1 QyxHksM0LZkhUHqeZggpO9CQcfeG0oflsmI4upZJ9Zhb+JPrwyNr44yyxno8 ASg16ENSZ+UKyij5q03D2yZeh6jn8pTrFcsODNSXuJa4QW6SbmKOG3.zdUcC lIVjqlpdzG+YSehGC04pEWOeMqFz78mMKu5FSg24PnkUUYatVeQCQefym9DO zcQguxvG3ykIGAapWdSioveaHajdEk4jhmCw+O4krwX8me.F680VSAdNH9P2 WjA3W+4tL4zdQq5ZhD6q8UhmDSaH65G+BcTLYWi2u+QBupVjZEQdPiGLmaIb 0RSoQLM0RzgSdj1L93kJo84k+OD1n9sA -----------end_max5_patcher-----------