Alpha Channel Manipulation in OpenGL and Advice

I've found the best way is to make an open gl object into the shape that you want the mask, then apply the video as a texture onto the shape. The texture wrapping method is important, depending on what you want to do. Pasted below is a fairly simple example, with a little documentation included to explain what it does.

Let me know if I'm not correctly understanding your goal.

THANK YOU for the quick reply! Your example was very clear and the notes were VERY helpful. I spent some time playing around with your example and found that using 'tex_map 1' gets me very, very close to what I want. One thing that's throwing me off a bit, though, is having to set the s and t texture coordinates according to the radius of the circle (or ellipse). My destination video sizes aren't uniform. I don't suppose there's an autoscaling option I missed?

One thing I didn't mention in my original description was that I applied blurring to the oval alpha mask in order to soften and feather the edges of the small videos. I suppose I can still use an alpha mask to do this.

Thanks again for taking the time to answer. You've been very helpful.

-Ed

Rereading your above post, it seems like what you're wanting to do is have the whole video appear in the circle with just the corners rounded off? Is that right? With non-uniform destination sizes, that makes things trickier. I honestly don't understand the math of the texture mappings well enough to figure out how to convert the radius into the appropriate number for the tex_plane parameters. I'll keep playing with it and come back to it if I get it figured out.

The feathering is actually quite tricky if you use my above-suggested method with open gl objects. As you might see from my other recent post today, feathering is something I've been spending a good bit of time with recently. From what I've learned, there is no way to actually feather the edges of an open-gl object (if you feather the texture, you just end up with black around the edges of your object). That might be incorrect, but I'm pretty sure it's true. In other words, if you need to feather, you'll have to completely scrap what I posted above and start over.

There are a few ways to work around this. However, from your original post it sounds like you have the basic idea. Fundamentally, you need to come up with a feathered alphamask, then use alphaglue to get your video onto the mask, and alphablend to combine that with the background material.

The most processor friendly way I've found to create the alphamask is to use jit.gl.sketch to make a plane, capture than plane to a texture, pass that texture through a fader (edgeblend for a rectangle/square, or Andrew Benson's spotmask for a circle), pop that texture onto a videoplane, and capture the output of the videoplane to yet another texture. From there, my method has been to convert the texture to a CPU readable matrix, use dstdim to position the resultant alphamask, use alphaglue to get your video into your alphamask, and finally alphablend to get your background behind it.

The worst part about that method (besides its odiousness), is having to convert the texture to a CPU-readable matrix in order to use dstdim, which can take a dig into your framerate if you're making multiple circles. This is the biggest flaw I can see in my method, so let me know if you find a better way to position the alphamask.

Basically what I’m trying to say is that this will work to do what you’re asking, but boy is it a silly way to have to do it.

Pasted below is another example patch demonstrating this method. If anyone can post a more efficient way of doing this, I also would be very grateful!

Let me know how it works for ya, Ed.

(note, for below patch to work, you must have andrew benson's spotmask shader from his Jitter Recipes Book #3)

"Rereading your above post, it seems like what you're wanting to do is have the whole video appear in the circle with just the corners rounded off? Is that right?"

Yes, exactly.

You know, I tried using alphaglue with a mask at one point but it spit out tons of errors for some reason.

I'll check out your patch tomorrow but it does sound awfully convoluted. I'll also check out your other thread.

Thanks again!

-Ed

Having slept on the problem, I've made another adjustment. I got rid of the jit.matrix objects which were used for usedstdim and instead used a td.rota.jxs shader for sizing and positioning the mask. This is a far more efficient method, as the matrix never has to be converted back to the CPU.

This is most likely the best way to accomplish your goal.

See below for the patch.

Thanks for the code! There's something deifnitely screwy with alphaglue. I took your example from last night and got rid of the black borders by using jit.op in place of alphaglue and setting the alpha mask to only one plane. This is exactly how my matrix based Jitter patcher creates round feathered borders.

By the way, you're trying to do this with arbitrary shapes, right? Have you tried extracting the luma channel, setting a threshold so points are either 100% black or 100% white, using the erode shader then adding a gaussian blur to create a feathered alpha mask?

----------------------------------------------------------
[MOD - removed uncompressed patch]

First of all, when copying patches into the forum, using Copy Compressed instead of just copy saves a huge amount of space. In fact, the patch you pasted in got cropped off by the forum, so I can't see it. If you could repost it with copy compressed that would be great!

Second, what exactly is happening for you with alphaglue? It's working perfectly on my computer (windows 7 here, although I doubt it makes a difference). You said before it spits out a lot of errors? If anyone else is trolling this thread, could you chime it to let us know how these patches working for you?

That's a good idea with luma and erode. I've posted below a version of what I think you mean (still using alphaglue since it works for me, though you can switch in jit.op in its place). Extrapolating that idea, I bet that the way to accomplish the feathering of any shape (as I posted about in another thread) would be to create that shape in jit.gl.sketch, then pass it through the erode and gaussian blur shaders like you said. I'll give that a shot and post with what I come up with.

Here's the code for what I think you suggested above.

Sorry about that. The compressed version of the code that makes the black edges go away is below.

Alphaglue spits out the following error message for EVERY frame on my Vista machine:

0(9) : warning C7531: global type sampler2DRect requires "#extension GL_ARB_texture_rectangle : enable" before use
0(25) : error C1011: cannot index a non-array value
-- END GLSL INFO LOG: C:/Program Files (x86)/Cycling '74/Max 5.0/Cycling '74/jitter-shaders/color/cc.alphaglue.jxs --
-- START GLSL INFO LOG: C:/Program Files (x86)/Cycling '74/Max 5.0/Cycling '74/jitter-shaders/color/cc.alphaglue.jxs --
Fragment info

"Extrapolating that idea, I bet that the way to accomplish the feathering of any shape .... would be to create that shape in jit.gl.sketch, then pass it through the erode and gaussian blur shaders..."

Yes, the code you included was essentially what I meant. I added a contrast shader to set a threshold and also to invert the results. I also added a planemap shader to copy the red channel to the alpha channel in the mask before using alphaglue.