slab compositing and colormodes

Oct 14, 2010 at 9:41pm

slab compositing and colormodes

Hi, hoping someone can help out here. I am having some issue with slab compositing. The patch below illustrates my issue – some composite .jxs work fine (such as average and normal and others) while others like additive, subtractive and others, don’t. On my machine it ends up looking like a colormode issue, but I cant seem to get it right. It will work with both sources as ARGB, but not with both sources as UYVY.

I’ve distilled a much larger patch down to what you see below. Im running a MacBookPro i7 / OS 10.6.4 / NVIDIA330M 512

– Pasted Max Patch, click to expand. –
#52756
Oct 15, 2010 at 7:43am

– Pasted Max Patch, click to expand. –
#189605
Oct 15, 2010 at 3:18pm

well, yes that works OK, but I loose half the resolution (vertical) oversampling by setting the dim@ 1440 480 does the trick, but it works the cpu too hard… – I’ve reworded my issue and posted it here:

http://cycling74.com/forums/topic.php?id=28971

I need to maintain full quality, yet Id like to use the uyvy colormode and Im not sure why the slab doesn’t seem to work correctly….the jit.colormode works, but it is pretty heavy too, i was trying to use the GPU version of course…

-matthew

#189606
Oct 15, 2010 at 8:26pm

I think there is some misunderstanding about colormodes here. colormode UYVY primarily exists as a way to get data quickly to the GPU, where the colorspace conversion and upsampling to full resolution RGBA can be handled by the graphics card. Once on the GPU, *everything* is processed as RGBA. There is no concept of UYVY on the GPU or in any shader except for ones custom programmed to work with that sort of data *masquerading* as RGBA.

The cc.uyvy2rgba.jxs and cc.rgba2uyvy.jxs shaders are provided as ways to perform the conversion of UYVY data to RGBA and back for the transfer to and from the GPU.

The quick answer here, is that in all likelihood, you want to avoid using UYVY at any point beyond an initial move to the GPU. And at that point all processing on the GPU should be in RGBA.

OS X supports a UYVY texture type, which undergoes a shader based conversion to RGBA as it is loaded into the GPU, but it doesn’t do chroma smoothing or other color corrections, and Windows does not provide any. This is why cc.uyvy2rgba.jxs exists, and it should *always* be used with @dimscale 2 1 like the example jjit.gl.slab-uyvy-gpu.maxpat example, to convert from half chroma horizontal resolution to full horizontal resolution.

As for cc.rgba2uyvy.jxs, it is an example shader to be used for transfer back to the GPU in the half data UYVY format and should *always* used with the @dimscale 0.5 1 , to convert from full horizontal resolution to half chroma horizontal resolution. Chances are you will never need to use this.

I believe that Yoann’s post above is what you want to do. However, where Yoann’s post was incorrect, was in using cc.uyvy2rgba.jxs *without* @dimscale 2. 1. cc.uyvy2rgba.jxs is tricky (UYVY data masquerading as RGBA data), and requires this manual upsampling for the shader to work properly, otherwise there is still half horizontal resolution output. This is probably what you noticed.

Alternately, if you are on OS X, you can just use jit.gl.texture @colormode uyvy or jit.gl.slab @colormode uyvy for the very first time the data moves from CPU to GPU (and *nowhere else down the slab processing chain*), and you do not need to use cc.uyvy2rgba.jxs. However, for the chroma smoothing and some nice color correction along the way, I think cc.uyvy2rgba.jxs and cc.uyvy2rgba.exp.jxs provide better results than Apple’s default UYVY texture -> RGBA conversion. Play around with them in jit.gl.slab-uyvy-gpu.maxpat

I hope this helps. I guess we need to have a more detailed article on this subject.

-Joshua

#189607
Oct 18, 2010 at 7:11pm

Huuuge thanks for the in depth explanation of this Joshua. Clears allot up for me, I was driving my self more than a little crazy!

#189608
Oct 18, 2010 at 7:45pm

also a huge thanks to Joshua.

i have still one question to the last option using uyvy on mac osx.
if the matrix data is coming from an jit.qt.movie object.
do i have to set the @colormode uyvy at the jit.qt.movie object
to speed up or not ?

best,
tom

#189609
Oct 18, 2010 at 8:27pm

Sure thing. Glad it helps. We’ll look at making the docs and or an article about this in the future.

@tom: Yes. jit.qt.movie must use @colormode uyvy in order to output UYVY data.

-Joshua

#189610
Oct 20, 2010 at 7:45pm

Merci Joshua
For this enlighting explanation
I was sending matrices from 3 webcams via jit.net.send in uyvy because I thought matrices were twice smaller than in RGBA ….
best.
Filippo

#189611
Oct 22, 2010 at 6:54pm

Fillipo, what you describe is a perfectly appropriate reason to use UYVY as output from jit.qt.grab. It is 2x smaller when in RAM. Once on the GPU, things are expanded and processed as RGBA.

#189612
Feb 2, 2011 at 2:10pm

Hi joshua,
I’ve recorded movies (chunks of 15 seconds) using [jit.qt.record 160 240 @colormode uyvy] with the images received via jit.net.send as explained in my last post. Now if I want to record the movies back and recover the dim 320 240 and fullquality I must upload to GPU (with cc.uyvy2argb.jxs) dim 2 1 and use jit.gl.asyncread. Is there another way to do it?
thanx filippo

#189613
Feb 2, 2011 at 7:49pm

you can convert back without GPU using jit.uyvy2argb

#189614
Feb 2, 2011 at 9:02pm

Hi robert,
the fact is matrices were still in uyvy, recording movies with size 160 240 and attriute colormode set on uyvy compress a second time the images: the result is a movie with colormode argb and size 160 240, I’ve lost half matrices. I’m so stupid !
I don’t think there is a way to recover. As stated by joshua and generalized, once a matrix is in uyvy do not set it a second time in the chain processing ( GPU or CPU )
Have a good night
Filippo

#189615
Feb 5, 2011 at 11:11am

Well I had always wondered what the difference between using cc.uyvy2arbg.jxs and colormode uyvy was! Thanks very much for explaining Joshua :)

If anyone is interested, I made a small patch to see for myself the difference between these two modes. I was really surprised when that horrible red pixelation that you see on so many videos pretty much just disappeared! Finding Nemo is a great test video as you constantly see this around the clown fish against the blue of the sea.

DiGiTaLFX

– Pasted Max Patch, click to expand. –
#189616
May 29, 2012 at 5:02pm

Hi ,

I just came across this post, and I would love to get more detailed info on the usage with jit.qt.movie.

If I set @colormode uyvy when reading the QT movie in, can I pretty much ignore any worry about uyvy vs rgba if I process entirely on the GPU from there? If I understand, once the data gets to the GPU, it will be converted and processed as RGBA from there (unless I explicitly wanted to do an RGBA to UYVY conversion for some reason).

I am currently mapping the qt movie onto textures on .obj’s (jit.gl.model) etc. or onto videoplanes then rendering out to jit.window.

I just want to be sure I’m getting this right. Is there anywhere else in that chain that I need to explicitly set a colormode attribute or convert, or is it converted from the jit.qt.movie uyvy automatically by the GPU once it takes over the processing? And is there anywhere in that chain that I need to set dim attributes if the source data is from that QT movie?

Thanks! It is all a bit confusing. I am starting to understand, but there are a few pieces of missing info =)

Valentin

#189617
May 29, 2012 at 5:25pm

the common usage of colormode uyvy is to read from the disk to the gpu using uyvy.
once on the gpu, all texutres should be rgba.
for the users, all you need to do is set @colormode uyvy on the qt.movie, and send directly to a cc.uyvy2rgba.jxs @dimscale 2 1 slab.

after that initial conversion, you do not need to worry about the colormode.

– Pasted Max Patch, click to expand. –
#189618
May 30, 2012 at 12:29am

I think that answers the question perfectly. Thanks!

V

#189619
Jun 4, 2012 at 2:16pm

hello,

i am a bit confused.

on my mac i do not need the cc.uyvy2rgba.jxs shader with @dimscale 2 1.
i simply can do this:

– Pasted Max Patch, click to expand. –

or this:

– Pasted Max Patch, click to expand. –

so which is the most efficient method for loading moviedata to the gpu on a mac ? do we need the cc.uyvy2rgba.jxs or not ?

best,
l

#189620
Jun 4, 2012 at 7:01pm

hi lee. quoting from Joshua’s post above:

“Alternately, if you are on OS X, you can just use jit.gl.texture @colormode uyvy or jit.gl.slab @colormode uyvy for the very first time the data moves from CPU to GPU (and *nowhere else down the slab processing chain*), and you do not need to use cc.uyvy2rgba.jxs. However, for the chroma smoothing and some nice color correction along the way, I think cc.uyvy2rgba.jxs and cc.uyvy2rgba.exp.jxs provide better results than Apple’s default UYVY texture -> RGBA conversion. Play around with them in jit.gl.slab-uyvy-gpu.maxpat ”

#189621

You must be logged in to reply to this topic.