mbiederman

Hi, hoping someone can help out here. I am having some issue with slab compositing. The patch below illustrates my issue - some composite .jxs work fine (such as average and normal and others) while others like additive, subtractive and others, don't. On my machine it ends up looking like a colormode issue, but I cant seem to get it right. It will work with both sources as ARGB, but not with both sources as UYVY.

I've distilled a much larger patch down to what you see below. Im running a MacBookPro i7 / OS 10.6.4 / NVIDIA330M 512

Hi, hoping someone can help out here.  I am having some issue with slab compositing.  The patch below illustrates my issue - some composite .jxs work fine (such as average and normal and others) while others like additive, subtractive and others, don't.  On my machine it ends up looking like a colormode issue, but I cant seem to get it right.  It will work with both sources as ARGB, but not with both sources as UYVY.

I've distilled a much larger patch down to what you see below. Im running a MacBookPro i7 / OS 10.6.4 / NVIDIA330M 512


slab-compositing-and-colormodes

well, yes that works OK, but I loose half the resolution (vertical) oversampling by setting the dim@ 1440 480 does the trick, but it works the cpu too hard... - I've reworded my issue and posted it here:

https://cycling74.com/forums/jit-qt-grab-uyvy-quality-and-jit-gl-processing

I need to maintain full quality, yet Id like to use the uyvy colormode and Im not sure why the slab doesn't seem to work correctly....the jit.colormode works, but it is pretty heavy too, i was trying to use the GPU version of course...

I think there is some misunderstanding about colormodes here. colormode UYVY primarily exists as a way to get data quickly to the GPU, where the colorspace conversion and upsampling to full resolution RGBA can be handled by the graphics card. Once on the GPU, *everything* is processed as RGBA. There is no concept of UYVY on the GPU or in any shader except for ones custom programmed to work with that sort of data *masquerading* as RGBA.

The cc.uyvy2rgba.jxs and cc.rgba2uyvy.jxs shaders are provided as ways to perform the conversion of UYVY data to RGBA and back for the transfer to and from the GPU.

The quick answer here, is that in all likelihood, you want to avoid using UYVY at any point beyond an initial move to the GPU. And at that point all processing on the GPU should be in RGBA.

OS X supports a UYVY texture type, which undergoes a shader based conversion to RGBA as it is loaded into the GPU, but it doesn't do chroma smoothing or other color corrections, and Windows does not provide any. This is why cc.uyvy2rgba.jxs exists, and it should *always* be used with @dimscale 2 1 like the example jjit.gl.slab-uyvy-gpu.maxpat example, to convert from half chroma horizontal resolution to full horizontal resolution.

As for cc.rgba2uyvy.jxs, it is an example shader to be used for transfer back to the GPU in the half data UYVY format and should *always* used with the @dimscale 0.5 1 , to convert from full horizontal resolution to half chroma horizontal resolution. Chances are you will never need to use this.

I believe that Yoann's post above is what you want to do. However, where Yoann's post was incorrect, was in using cc.uyvy2rgba.jxs *without* @dimscale 2. 1. cc.uyvy2rgba.jxs is tricky (UYVY data masquerading as RGBA data), and requires this manual upsampling for the shader to work properly, otherwise there is still half horizontal resolution output. This is probably what you noticed.

Alternately, if you are on OS X, you can just use jit.gl.texture @colormode uyvy or jit.gl.slab @colormode uyvy for the very first time the data moves from CPU to GPU (and *nowhere else down the slab processing chain*), and you do not need to use cc.uyvy2rgba.jxs. However, for the chroma smoothing and some nice color correction along the way, I think cc.uyvy2rgba.jxs and cc.uyvy2rgba.exp.jxs provide better results than Apple's default UYVY texture -> RGBA conversion. Play around with them in jit.gl.slab-uyvy-gpu.maxpat

I hope this helps. I guess we need to have a more detailed article on this subject.

OS X supports a UYVY texture type, which undergoes a shader based conversion to RGBA as it is loaded into the GPU, but it doesn't do chroma smoothing or other color corrections, and Windows does not provide any. This is why  cc.uyvy2rgba.jxs exists, and it should *always* be used with @dimscale 2 1 like the example jjit.gl.slab-uyvy-gpu.maxpat example, to convert from half chroma horizontal resolution to full horizontal resolution.

Huuuge thanks for the in depth explanation of this Joshua. Clears allot up for me, I was driving my self more than a little crazy!

Huuuge thanks for the in depth explanation of this Joshua.  Clears allot up for me, I was driving my self more than a little crazy!


i have still one question to the last option using uyvy on mac osx.

if the matrix data is coming from an jit.qt.movie object.

do i have to set the @colormode uyvy at the jit.qt.movie object

i have still one question to the last option using uyvy on mac osx.
if the matrix data is coming from an jit.qt.movie object.
do i have to set the @colormode uyvy at the jit.qt.movie object
to speed up or not ?

Sure thing. Glad it helps. We'll look at making the docs and or an article about this in the future.

@tom: Yes. jit.qt.movie must use @colormode uyvy in order to output UYVY data.

I was sending matrices from 3 webcams via jit.net.send in uyvy because I thought matrices were twice smaller than in RGBA ....

Merci Joshua
For this enlighting explanation
I was sending matrices from 3 webcams via jit.net.send in uyvy because I thought matrices were twice smaller than in RGBA ....
best.
Filippo


Fillipo, what you describe is a perfectly appropriate reason to use UYVY as output from jit.qt.grab. It is 2x smaller when in RAM. Once on the GPU, things are expanded and processed as RGBA.

I've recorded movies (chunks of 15 seconds) using [jit.qt.record 160 240 @colormode uyvy] with the images received via jit.net.send as explained in my last post. Now if I want to record the movies back and recover the dim 320 240 and fullquality I must upload to GPU (with cc.uyvy2argb.jxs) dim 2 1 and use jit.gl.asyncread. Is there another way to do it?

Hi joshua,
I've recorded movies (chunks of 15 seconds) using [jit.qt.record 160 240 @colormode uyvy] with the images received via jit.net.send as explained in my last post. Now if I want to record the movies back and recover the dim 320 240 and fullquality I must upload to GPU (with cc.uyvy2argb.jxs) dim 2 1  and use jit.gl.asyncread. Is there another way to do it?
thanx filippo


you can convert back without GPU using jit.uyvy2argb

the fact is matrices were still in uyvy, recording movies with size 160 240 and attriute colormode set on uyvy compress a second time the images: the result is a movie with colormode argb and size 160 240, I've lost half matrices. I'm so stupid !

I don't think there is a way to recover. As stated by joshua and generalized, once a matrix is in uyvy do not set it a second time in the chain processing ( GPU or CPU )

Hi robert,
the fact is matrices were still in uyvy, recording movies with size 160 240 and attriute colormode set on uyvy compress a second time the images: the result is  a movie with colormode argb and size 160 240, I've lost half matrices. I'm so stupid !
I don't think there is a way to recover. As stated by joshua and generalized, once a matrix is in uyvy do not set it a second time in the chain processing ( GPU or CPU )
Have a good night
Filippo


Well I had always wondered what the difference between using cc.uyvy2arbg.jxs and colormode uyvy was! Thanks very much for explaining Joshua :)

If anyone is interested, I made a small patch to see for myself the difference between these two modes. I was really surprised when that horrible red pixelation that you see on so many videos pretty much just disappeared! Finding Nemo is a great test video as you constantly see this around the clown fish against the blue of the sea.

I just came across this post, and I would love to get more detailed info on the usage with jit.qt.movie.

If I set @colormode uyvy when reading the QT movie in, can I pretty much ignore any worry about uyvy vs rgba if I process entirely on the GPU from there? If I understand, once the data gets to the GPU, it will be converted and processed as RGBA from there (unless I explicitly wanted to do an RGBA to UYVY conversion for some reason).

I am currently mapping the qt movie onto textures on .obj's (jit.gl.model) etc. or onto videoplanes then rendering out to jit.window.

I just want to be sure I'm getting this right. Is there anywhere else in that chain that I need to explicitly set a colormode attribute or convert, or is it converted from the jit.qt.movie uyvy automatically by the GPU once it takes over the processing? And is there anywhere in that chain that I need to set dim attributes if the source data is from that QT movie?

Thanks! It is all a bit confusing. I am starting to understand, but there are a few pieces of missing info =)

I just want to be sure I'm getting this right. Is there anywhere else in that chain that I need to explicitly set a colormode attribute or convert, or is it converted from the jit.qt.movie uyvy automatically by the GPU once it takes over the processing?  And is there anywhere in that chain that I need to set dim attributes if the source data is from that QT movie?

Thanks!  It is all a bit confusing. I am starting to understand, but there are a few pieces of missing info =)

the common usage of colormode uyvy is to read from the disk to the gpu using uyvy.

once on the gpu, all texutres should be rgba.

for the users, all you need to do is set @colormode uyvy on the qt.movie, and send directly to a cc.uyvy2rgba.jxs @dimscale 2 1 slab.

after that initial conversion, you do not need to worry about the colormode.

the common usage of colormode uyvy is to read from the disk to the gpu using uyvy.
once on the gpu, all texutres should be rgba.
for the users, all you need to do is set @colormode uyvy on the qt.movie, and send directly to a cc.uyvy2rgba.jxs @dimscale 2 1 slab.

after that initial conversion, you do not need to worry about the colormode.


I think that answers the question perfectly. Thanks!

on my mac i do not need the cc.uyvy2rgba.jxs shader with @dimscale 2 1.

so which is the most efficient method for loading moviedata to the gpu on a mac ? do we need the cc.uyvy2rgba.jxs or not ?

on my mac i do not need the cc.uyvy2rgba.jxs  shader with @dimscale 2 1.
i simply can do this:

hi lee. quoting from Joshua's post above:

"Alternately, if you are on OS X, you can just use jit.gl.texture @colormode uyvy or jit.gl.slab @colormode uyvy for the very first time the data moves from CPU to GPU (and *nowhere else down the slab processing chain*), and you do not need to use cc.uyvy2rgba.jxs. However, for the chroma smoothing and some nice color correction along the way, I think cc.uyvy2rgba.jxs and cc.uyvy2rgba.exp.jxs provide better results than Apple's default UYVY texture -> RGBA conversion. Play around with them in jit.gl.slab-uyvy-gpu.maxpat "

slab compositing and colormodes