Resizing jitter resolution (in jit.gl), downsampling using average instead of center of pixel
Hi all,
I am trying to resize a video using jit.gl to a 3x2 pixel size. I have a working solution using a jit.gl.node object with attributes @adapt 0. and @dim 3 2, but I'm not satisfied with the way it is changing the resolution.
For the values of the new pixels in the 3x2 matrix, jitter uses the value of one pixel in the original matrix, that coincides with the position of the center of the new pixel. For a small change in resolution this would be fine, but for changing to such a small resolution I loose too much visual information.
I would like to use the average of the whole area of the original video that coincides with the new pixel, rather than just the value of one pixel. In other words, I would like to divide the original video into a number of areas corresponding with the new resolution, and then take the average values of those areas as the value for the corresponding pixel in the new matrix.
I could think of some ways to do this using regular jitter objects, but I am blank on how to make this work using jit.gl objects. Does anyone have some ideas where I could start?
For some background, I am trying to control a 3x2 LED matrix (will be a 10x10 matrix later on), using video input. I am performing some video editing using jit.gl objects both before and after resizing the video content, which is why I would prefer to keep the resizing on the GPU rahter than using CPU based jitter objects.
Interesting. Since you will ultimately go to CPU, you could still try if you get the efficiency you want with [cv.jit.resize] (cv.jit package).
I don't know how you would do that in GL.
I guess one approach in GL would be to first fill a matrix the original size with the average values (all the pixels of each area would have the same value), then apply your resolution change as you currently do.
I guess one approach in GL would be to first fill a matrix the original size with the average values (all the pixels of each area would have the same value), then apply your resolution change as you currently do.
How do you mean this? All pixels in my resutling matrix would then also be the same value right?
Also how would I do that?
Basically I guess I would need to know how to slice a GL texture into 6 smaller textures, calculate the averages, and merge into a single GL texture.
I suppose I could do the slicing before going to GPU, but averaging and merging would need to be performed after doing my jit.gl video editing.
You could do it in CPU or GPU... You would need to use a better downsample algorithm than what Jitter typically uses. For example, you could simply resize the image to a 3x2 proportion like 192x128 and then use a sequence of 6 average downsampling processes to reach the 3x2 resolution.
See Joshua Kit Clayton's explanation here: https://cycling74.com/forums/bug-matrix-interpolation-doesnt-work-properly/
Pedro, this is a very good example, thanks!
Hi Pedro,
Thanks for your help! I tried to implement it in my patch, but haven't got it working properly yet. Will look into it a bit more tomorrow, might be back with some questions then.
Ok, back again. I got the jit.gl.slab @dimscale that Pedro suggested working.
While it does seem to handle the resizing in a better way, I am not sure it is actually using the average as I intended it. I guess it is better than my original solution, but if someone has a solution that comes closer to my intentions that would be appreciated.
Here is a screenshot to illustrate what I mean. The top pwindow is the incoming matrix, in this case with a 6x6 resolution for testing purpose.
The left window shows the normal way of resizing (in this case simply a 3x3 jit.matrix connected to the 6x6 input) - in this case it seems to be taking the top left pixel value of the 4 pixels that correspond with the position of the new pixel.
The right window shows the resizing through a slab with dimscale 0.5 0.5. I did also correct with a brcosa object to compensate for the loss in brightness. It seems to work fine in the top and bottom left pixel; though if it were using an average, I would have expected at least the middle pixel to also be grey, rather than black.
It might seem like nitpicking, but for my purpose I am actually trying to retain a bit of the smooth movement in my original video, despite the very pixelated resolution. The current solutions give a rather jumpy result, with spikes in brightness/black. I am normally using a gen patch to smoothen out the end result, but that does not bring back the visual information that got lost in the first place. For the dynamic light effect I am trying to create this is important.
For now this is a better solution than my previous one, so it will do with a little smoothening at the end, but if someone knows of a better solution please share!
for matrices this small, you can just iterate them and perform whatever operation you want on the data, and pack the results in a new matrix of appropriate size. you could use JS, or even jit.gen for this type of thing (my rec would be JS).
Hi Rob,
I guess iterating a jit.gl.pix @gen to average and then resizing would do the trick. However, that would only work for powers of 2, or at least for how I would imagine that operation it would; or am I missing a solution there? In any case, I suppose that could be worked around by resizing the incoming video to a size that works just using a jit.matrix with defined dimensions, before going to GPU. I will give that a try and see how it performs.
Ok, thanks for the inspiration everyone! Got it working as I want now. Below a screenshot of the result. In the sub-patch is a jit.gl.pix with a genpatch calculating average over 4 pixels, a jit.gl.slab @dimscale 0.5 0.5 and a brcosa to correct the brightness, repeated 8 times. Incoming matrix dimensions are set based on desired output dimensions (dimOUT).
Left pwindow is edited incoming video, middle is original way of resizing as compared to new way on the right pwindow :) Very happy with the result. Just need to fiddle a bit with the right brcosa correction.
