How to get the average brightness of a video using jit.gl
Hi all,
I'm working on a patch which is processing video using jit.gl. As the title says, I want to know the average brightness of the video. I know how to do this using normal jit matrices, but I want to stay with jit.gl for this and I've got no clue how to accomplish it. Anyone who can point me in the right direction?
typically these types of analyses are done on the CPU using the jit.3m object. You can help this process along by downsampling on the GPU to a small enough size, readback to matrix using jit.gl.asyncread (in it's default mode, the most efficient way to readback a texture), and then send along to jit.3m (or whatever analysis algo you want to use).
However, if using gl3 engine, you can use @rectangle 0 textures with @mipmap trilinear and a simple custom shader that samples a specific mip level via textureLod. This allows you to get your average in a single pass, and then readback a single cell to a matrix. I believe these results are more accurate as well.
The attached example shows using both techniques to retrieve average color, but you could convert to HSL to get brightness instead (either via jit.gl.pix rgb2hsl operator, or via jit.rgb2hsl).
Thanks. How exactly does the second solution work? I'm not sure I fully understand what is happening in the patch. What does the mipmap attribute do?
Your first solution with jit.3m is more or less what I was doing already. It works, but I always learned that going back and forth between CPU and GPU isn't a good idea for performance - and my framerate is dropping to 10fps with this solution.
Using the mipmap solution my framerate does stay close to 30fps. I can basically feed the 1x1 texture into the second inlet of a jit.gl.pix for further processing (before I was using the mean value from jit.3m as a param in the same jit.gl.pix), so I don't have to use any CPU objects in the process. But I have no clue what I'm doing with mipmap and this shader, so also no clue whether the measured mean brightness is accurate or not.
Thanks. How exactly does the second solution work? I'm not sure I fully understand what is happening in the patch. What does the mipmap attribute do?
https://en.wikipedia.org/wiki/Mipmap
https://www.khronos.org/registry/OpenGL-Refpages/gl4/html/textureLod.xhtml
Using the mipmap solution my framerate does stay close to 30fps.
Great!
But I have no clue what I'm doing with mipmap and this shader, so also no clue whether the measured mean brightness is accurate or not.
if you want accuracy then iterate the cells and do the calculation on the CPU with full control over the algorithm, if you want speed then use the GPU. If you're curious about the accuracy of the GPU, then compare the results to a CPU result and see how divergent they are.
Thanks! Those links do clarify a lot. I'll do a comparison to see if there's a significant difference or not.
edit: a quick comparison shows a maximum difference of 0.006 (float) between the two solutions, so that is negligible.